Multimodal LLM Illusion Issue Reduced by 30%! Innovative 'Woodpecker' Method from USTC

ScreenSpot-Pro: A Multimodal LLM Benchmark Tool Designed for High-Resolution Environments!

Financial Analysis Multimodal LLM FinTral: A Groundbreaking Large Language Model

Innovation in the financial sector: FinTral employs a multimodal approach to handle text, numbers, tables, and visual data, outperforming GPT-4. Model features: Based on the Mistral-7B model, it has undergone domain-specific pre-training, fine-tuning, and AI feedback to achieve precise financial analysis through visual data processing. Strong performance validation: The FinTral-DPO model approaches the average score of GPT-4, demonstrating excellent performance in financial tasks. Experimental results confirm that FinTral excels in financial tasks.

Peking University and Stanford Collaborate with Pika to Launch RPG Framework, Achieving New Breakthroughs in Text-to-Image Research

Peking University, Stanford, and Pika Labs have released a new open-source text-to-image framework called RPG, successfully addressing two major challenges using multimodal LLM. The framework excels at handling complex text prompts, enhancing the generalization ability of text-to-image models. The RPG framework employs MLLM for global planning, breaking down the image generation process into multiple simpler tasks, outperforming existing models. Researchers proposed three core strategies: decomposing text prompts using MLLM, dividing the image space, and independently generating sub-region images.

LeCun Exposes Major Defects in Multimodal LLMs, Proposes Interleaved-MoF to Significantly Enhance Visual Understanding

['Recently, research teams from New York University and UC Berkeley have made significant breakthroughs in the realm of multimodal large language models, successfully capturing major deficiencies in visual understanding.', 'Researchers found that current MLLMs often perform worse than random guessing in specific scenarios, including some image problems that are easily recognized by humans.', 'Paper link: https://arxiv.org/pdf/2401.06209.pdf', 'The research team proposed Interleaved Feature Mixing (Inte'

ByteDance and USTC Join Forces to Break Resolution Limits, Launch Multi-Modal Document Large Model

ByteDance has collaborated with the University of Science and Technology of China for the first time to launch the high-resolution multi-modal document large model, DocPedia. The model has been uploaded to arXiv, solving the problem of previous models being unable to parse high-resolution document images. With a resolution of 2560×2560, DocPedia exhibits significant superiority in areas such as image-text understanding and visual question answering. The model enhances performance by addressing resolution issues through the frequency domain, creating a new technical breakthrough.

USTC and the Fengshenbang Team Release ChiMed-GPT, a Large Model in the Medical Field

USTC and the Fengshenbang team of the IDEA Institute have released ChiMed-GPT, a large model in the Chinese medical field. This model is based on the Ziya2-13B model, featuring 13 billion parameters. Through comprehensive pre-training, fine-tuning, and reinforcement learning with human feedback, it meets the needs of medical text processing. ChiMed-GPT outperforms other open-source models in tasks such as medical information extraction, question answering, and dialogue generation. This model holds significant importance for enhancing medical intelligence, effectively handling medical text data and responding to patient inquiries.

Dragging down the image to generate videos, DragNUWA by researchers from USTC and Microsoft is truly impressive

["Researchers from USTC and Microsoft have proposed a video generation model based on open-domain diffusion called DragNUWA, which allows for fine-grained video control through image dragging.", "DragNUWA integrates text, image, and trajectory control to achieve precise control over video content from semantic, spatial, and temporal perspectives.", "Experiments demonstrate that DragNUWA has outstanding performance in video synthesis, reliably controlling complex movements and generating stable and coherent videos."]