Apple's 'Multimodal Alchemy Furnace' Upgraded Again! MM1.5 Enhances Text Density and Multimodal Understanding

AIbase基地

Published inAI News · 4 min read · Oct 14, 2024

167

Apple Inc. has recently released a significant update for its multimodal AI model MM1, upgrading it to version MM1.5. This update represents more than just a simple version number change; it's a comprehensive enhancement that showcases the model's increased capabilities across various fields.

The core upgrade of MM1.5 lies in its innovative data processing methods. The model adopts a data-centric training approach, meticulously selecting and optimizing the training dataset. Specifically, MM1.5 incorporates high-definition OCR data and synthetic image descriptions, along with refined visual instruction tuning data. The integration of these data types has notably improved the model's performance in text recognition, image understanding, and execution of visual instructions.

In terms of model scale, MM1.5 includes multiple versions ranging from 1 billion to 300 billion parameters, including dense and mixture of experts (MoE) variants. Notably, even the smaller-scale models with 1 billion and 3 billion parameters, through carefully designed data and training strategies, achieve impressive performance levels.

The enhanced capabilities of MM1.5 are primarily reflected in the following areas: text-dense image understanding, visual referencing and localization, multi-image reasoning, video understanding, and mobile UI comprehension. These capabilities enable MM1.5 to be applied in a broader range of scenarios, such as identifying performers and instruments from concert photos, understanding chart data and answering related questions, and locating specific objects in complex scenes.

To evaluate the performance of MM1.5, researchers compared it with other advanced multimodal models. The results indicate that MM1.5-1B excels among models with 1 billion parameters, outperforming its counterparts. MM1.5-3B surpasses MiniCPM-V2.0 and is on par with InternVL2 and Phi-3-Vision. Additionally, studies show that both dense and MoE models experience significant performance improvements as their scale increases.

The success of MM1.5 not only demonstrates Apple's research and development strength in the field of artificial intelligence but also points the way forward for the future development of multimodal models. By optimizing data processing methods and model architecture, even smaller-scale models can achieve powerful performance, which is crucial for deploying high-performance AI models on resource-limited devices.

Paper link: https://arxiv.org/pdf/2409.20566

Apple's New AI System Ferret-UI 2 Refreshes UI Interaction Experience, Effortlessly Control Your Devices

Apple Inc. recently launched the next generation AI system Ferret-UI 2. This cross-platform AI assistant has made significant breakthroughs in UI element recognition, achieving a test score of 89.73, far surpassing GPT-4V's score of 77.73, demonstrating exceptional performance. The most notable feature of this system is its intelligent ability to understand user intent. Unlike traditional coordinate-based interaction methods, Ferret-UI 2 can automatically locate and execute relevant actions based on users' natural language commands.

Redefining Multimodal AI! Zhiyuan Releases the Native Multimodal World Model Emu3

Beijing Zhiyuan Artificial Intelligence Research Institute announces the launch of the native multimodal world model Emu3. This model is based on next-token prediction technology and does not rely on diffusion models or combinatorial methods to achieve understanding and generation across text, image, and video modalities. Emu3 surpasses existing well-known open-source models such as SDXL, LLaVA, and OpenSora in tasks like image generation, video generation, and visual language understanding, showcasing exceptional performance.

Apple AI Research Team Discovers Limitations of Large Model Inference, Rendering OpenAI's o1 Ineffective with Just One Sentence

In the world of artificial intelligence, the reasoning capabilities of machine learning models, particularly large language models (LLMs), have been a focal point for scientists. Recently, Apple's AI research team published a paper titled 'Understanding the Limitations of Large Language Models in Mathematical Reasoning,' shedding light on these models' limitations when addressing logical problems. In the paper, researchers demonstrate this through a simple mathematical question. They first posed a question about Oliver picking kiwis: as follows: Oliver picked 44 on Friday.

Rhymes AI Launches First Open Source Multimodal AI Model Aria, Outperforming GPT-4o Mini and Other Leading AI Models

Recently, Japan-based startup Rhymes AI unveiled their first artificial intelligence model — Aria. The company claims that Aria is the world's first open-source multimodal mixture of experts (MoE) model. This model not only possesses the capability to handle various input modalities but also claims to be on par with, or even superior to, several well-known commercial models. The design philosophy of Aria aims to provide exceptional understanding and processing capabilities across diverse input forms including text, code, images, and videos.