Recently, Meta AI open-sourced a foundational multimodal language model named SPIRIT LM, which can freely mix text and speech, opening new possibilities for multimodal tasks involving audio and text. SPIRIT LM is based on a pre-trained text language model with 7 billion parameters, which has been continuously trained on text and speech units, expanding into the speech modality. It can understand and generate text like a large text model, while also being capable of understanding and generating speech, and even mixing text and speech to create various forms of expression.

At this year's Samsung Developers Conference, Samsung officially unveiled the new Gauss 2 AI model, marking another significant advancement in its artificial intelligence endeavors. The Gauss 2 model upgrades in performance and efficiency, with applications ranging from smartphones to home appliances, encompassing a variety of smart devices. Gauss 2 has multimodal capabilities, allowing it to process images, text, and code simultaneously, enhancing the integration of AI into various device ecosystems. Samsung has developed three versions: Compact, Balanced, and

Artificial intelligence is quietly transforming the diagnostic methods in the field of mental health. A research team from Kaunas University of Technology has developed a revolutionary depression diagnosis model that employs multimodal analysis of voice and EEG data, paving the way for precise identification of mental health issues. The core of this research lies in breaking the limitations of traditional single-data diagnostics. The research team chose voice as a key data source because it can subtly reflect emotional states. Speech rate, tone, and emotional energy can all serve as potential signals of depression.

Recently, research teams from Peking University announced the release of an open-source multimodal model called LLaVA-o1, which is claimed to be the first visual language model capable of spontaneous and systematic reasoning, comparable to GPT-o1. The model excels in six challenging multimodal benchmark tests, with its 11B parameter version outperforming competitors such as Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct.

iFLYTEK recently announced the official launch of its latest development, the iFLYTEK Spark Multimodal Interaction Model. This technological breakthrough marks iFLYTEK's expansion from a single voice interaction technology to a new stage of real-time audio-visual multimodal interaction. The new model integrates voice, vision, and digital human interaction functions, allowing users to seamlessly combine all three with a single button.

In the application of artificial intelligence, achieving real-time interaction with AI has always been a significant challenge for developers and researchers. Among these challenges, integrating multimodal information (such as text, images, and audio) to form a coherent dialogue system is particularly complex. Although advanced large language models like GPT-4 have made some progress, many AI systems still struggle with achieving smoothness in real-time conversations, context awareness, and multimodal understanding, which limits their effectiveness in practical applications.

Recently, research teams from several Chinese scientific institutions have launched the super large-scale multimodal dataset named Infinity-MM, and based on this dataset, they have trained an outstanding AI model called Aquila-VL-2B. This breakthrough injects new momentum into the development of multimodal AI. The Infinity-MM dataset is astonishing in size, containing four major types of data: 10 million image descriptions, 24.4 million general visual instruction data, 6 million selected high-quality instruction data, and 3 million generated by G

The Beijing ZhiYuan Artificial Intelligence Research Institute, in collaboration with Shanghai Jiao Tong University, Renmin University of China, Peking University, and Beijing University of Posts and Telecommunications, has launched an ultra-long video understanding model named Video-XL. This model is an important demonstration of the core capabilities of multimodal large models and a key step towards General Artificial Intelligence (AGI). Compared to existing multimodal large models, Video-XL shows superior performance and efficiency in processing long videos exceeding 10 minutes.

Recently, the Meta AI team introduced LongVU, a novel spatio-temporal adaptive compression mechanism aimed at enhancing the language understanding capabilities of long videos. Traditional multimodal large language models (MLLMs) face limitations in context length when processing long videos, and LongVU was created to address this challenge. LongVU operates primarily by filtering duplicate frames and employing inter-frame token compression techniques to efficiently utilize context length, allowing it to reduce video data while preserving visual details.

Beijing Zhiyuan Artificial Intelligence Research Institute announces the launch of the native multimodal world model Emu3. This model is based on next-token prediction technology and does not rely on diffusion models or combinatorial methods to achieve understanding and generation across text, image, and video modalities. Emu3 surpasses existing well-known open-source models such as SDXL, LLaVA, and OpenSora in tasks like image generation, video generation, and visual language understanding, showcasing exceptional performance.

Apple Inc. recently launched a significant update for its multimodal AI model MM1, upgrading it to version MM1.5. This upgrade represents more than just a simple change in version number; it is a comprehensive enhancement of capabilities, showcasing the model's stronger performance across various fields. The core upgrade of MM1.5 lies in its innovative data processing methods. The model utilizes a data-centric training approach, carefully selecting and optimizing the training dataset. Specifically, MM1.5 employs high-definition OCR data and synthetic image descriptions, along with optimizations.

Recently, Japan-based startup Rhymes AI unveiled their first artificial intelligence model — Aria. The company claims that Aria is the world's first open-source multimodal mixture of experts (MoE) model. This model not only possesses the capability to handle various input modalities but also claims to be on par with, or even superior to, several well-known commercial models. The design philosophy of Aria aims to provide exceptional understanding and processing capabilities across diverse input forms including text, code, images, and videos.

Recently, Apple's AI research team launched their next-generation family of Multimodal Large Language Models (MLLMs) - MM1.5. This series of models can integrate various data types such as text and images, showcasing new capabilities of AI in understanding complex tasks. Tasks like visual question answering, image generation, and multimodal data interpretation can be better addressed with the help of these models. A major challenge for multimodal models is how to achieve effective interaction between different data types. Previous models often struggled in this aspect.