OPPO announced that its AI large model AndesGPT-2.0 performed exceptionally in the 'Chinese Large Model Benchmark Evaluation August 2024 Report' released by the authoritative third-party evaluation agency SuperCLUE, winning multiple first-place awards, marking further recognition of OPPO's technological strength in the AI field.

As AI technology continues to advance, understanding user interfaces (UI) has become a key challenge in creating intuitive and useful AI applications. Recently, researchers at Apple introduced UI-JEPA in a new paper, an architecture designed for lightweight, device-side UI understanding that maintains high performance while significantly reducing the computational requirements for UI understanding. The challenge of UI understanding lies in processing cross-modal features, including images and natural language, to capture temporal relationships within UI sequences. Despite the complexity, multimodal large language models...

Mistral AI has launched Pixtral12B, the first open-source multimodal model with 12 billion parameters, capable of processing both images and text. At 23.64GB, it's lightweight and offers easy access to model weights, reducing energy consumption and deployment barriers. It outperforms several established multimodal models in benchmark tests, driving a wave of open-source innovation in AI.....

Recently, edge computing chip and software startup SiMa.ai announced the launch of its new product—MLSoC Modalix chip, further solidifying its position in the edge AI market. SiMa.ai has just completed a $70 million funding round, with investors including industry giants like Dell Technologies Capital.

Vivo's OS product vice president Zhou Wei announced that the company will unveil its brand new Blue Heart Large Model at next month's developer conference. This new model will encompass large models in various fields including language, speech, vision, multimodal, and edge computing. Zhou emphasized in the statement that artificial intelligence (AI) will become a core technology and a key driver in the development of mobile operating systems over the next five years, playing a significant role in various modules such as audio-visual entertainment and office collaboration. Vivo plans to integrate the Blue Heart Large Model into its latest OriginOS 5 system.

At a recent Gartner IT symposium, analysts predicted that by 2027, 40% of generative artificial intelligence (GenAI) solutions will achieve multimodal integration, significantly up from 1% in 2023. The integration of multimodal GenAI can capture relationships between different data streams and expand benefits across various data and applications, supporting human multitasking in different environments. Gartner's report indicates that multimodal GenAI and open-source large language models will have a profound impact, benefiting enterprises in the next five years.

NVIDIA has collaborated with several universities to introduce NVEagle, a large visual language model capable of chatting using images. NVEagle can analyze image content and provide accurate answers, such as identifying individuals in images, like Jensen Huang. The model significantly enhances the understanding of visual information by transforming images into visual tokens and combining them with text embeddings. In addressing the challenges of high-resolution image processing, the research team has constructed models like Eagle-X5-7B and Eagle-X by exploring various visual encoders and fusion strategies.

A research team composed of Renmin University of China, Beijing University of Posts and Telecommunications, and Shanghai AI Lab has developed a technology called Ref-AVS, aimed at solving the challenge of artificial intelligence understanding the complex physical world. This technology utilizes a unique multimodal fusion method that integrates Video Object Segmentation (VOS), Video Object Reference Segmentation (Ref-VOS), and audiovisual segmentation (AVS) information, enabling AI systems to accurately identify and locate specific objects in audiovisual scenes, regardless of whether those objects produce sound. To validate the effectiveness of the technology, the research team...

On August 23, 2024, the well-known Chinese artificial intelligence company Yunzhisheng announced the launch of its latest research and development achievement – the Shanhai Multimodal Large Model in Beijing. The Shanhai Multimodal Large Model is part of Yunzhisheng's Atlas AI infrastructure, capable of receiving and processing inputs from various modalities, including text, audio, and images, and generating any combination of text, audio, and image outputs in real-time. This capability enables the Shanhai model to not only conduct efficient voice interactions but also provide a conversational experience that is close to natural human dialogue.

Google AI Studio has released a new native prompt library that integrates features such as long-text context, multimodal inputs (images, videos, audio), and structured outputs. The aim is to provide users with a rich and diverse experience across various fields including education, programming, and travel. This library includes practical features like recipe generators, math tutors, math homework generators, treasure hunt design, unit test generators, and travel suggestions, all designed to enhance user exploration and innovation through intuitive displays and comprehensive information. Users only need to log into Google AI Studio.

Microsoft has released Phi-3.5-vision, a lightweight, multimodal open source AI model designed for processing textual and visual inputs, supporting a context length of 128K. This model is suitable for resource-constrained environments and features capabilities such as image understanding, OCR, chart parsing, and multi-image summarization, showcasing excellent performance and low latency. Comprised of 4.2 billion parameters, it is trained with high-quality data to ensure performance and privacy. It includes three models: lightweight AI, expert mix, and multimodal model, all demonstrating outstanding performance in image and video processing benchmarks.

Salesforce has released an open source multimodal AI model named xGen-MM, designed to simultaneously understand and generate various data types such as text and images, significantly changing the way AI research and applications are conducted. The model performs exceptionally well in multiple benchmark tests, demonstrating strong performance compared to similar open source models, and includes pretrained models, datasets, and fine-tuning code. The largest model boasts 4 billion parameters and can handle 'interleaved data,' enabling multitasking, such as answering questions about multiple images at once. The diverse options available in the model reflect the potential of AI.

Recently, Moore Threads and Shiyin Intelligence announced a strategic partnership to jointly promote the application of domestically produced full-function GPUs in industry large model solutions. The two parties have completed the adaptation work of the Moore Threads Kuaguo Computing Cluster with Shiyin Intelligence's 'A Leaf Light Boat' multimodal large model, integrating their respective advantageous resources in the field of artificial intelligence, and are committed to providing a more intelligent and efficient service experience.

Multimodal generative models are leading a new trend in artificial intelligence, focusing on the integration of visual and textual data to create multifunctional AI systems that perform various tasks, from image generation to understanding and reasoning across different data types. A key challenge is to enhance the capabilities of autoregressive (AR) models so they can generate high-detail images based on textual descriptions. While diffusion models excel in generating high-quality images, AR models lag behind in image quality, resolution flexibility, and multitasking capabilities. Researchers from Shanghai AI Lab and The Chinese University of Hong Kong have introduced Lum

Tencent's YouTu Lab and other institutions have released the first open source multimodal large language model VITA, aimed at bridging the gap in processing Chinese dialects. Based on the Mixtral8×7B model, VITA expands the Chinese vocabulary and undergoes bilingual instruction fine-tuning, mastering both English and Chinese. Key features include: 1. **Multimodal Understanding**: VITA can handle video, images, text, and audio, which is unprecedented among open source models. 2. **Natural Interaction**: No specific wake words are required, allowing for instant response while maintaining polite and non-intrusive communication.

The latest version 2.6 of WallFacer’s MiniCPM-V series has rapidly climbed to the Top 3 on GitHub and HuggingFace trends, surpassing ten thousand stars. Since its release in February, it has accumulated over a million downloads, becoming a benchmark for on-device model capabilities. MiniCPM-V2.6 achieves performance enhancements for on-device multimodal models with 8 billion parameters, including real-time video understanding, multi-image joint understanding, and multi-image in-context learning, with a quantized backend memory of only 6GB and an inference speed of up to 18 tokens.

The Sinan OpenCompass team at the Shanghai Artificial Intelligence Laboratory has collaborated with the Modao ModelScope to launch the Compass Multi-Modal Arena, a new section of a large model evaluation platform focusing on multi-modal large models. Users can upload images and input questions to have two anonymous multi-modal large models generate answers, which can then be subjectively evaluated based on the quality of the generated content, allowing users to select the better-performing model. The platform offers an easy-to-use interface and a unique question bank.

"MedTrinity-25M" is a large-scale multimodal medical dataset released by the UCSC-VLAA team, containing 25 million medical images along with detailed annotations. This dataset features multi-granularity annotations, supports the training of multimodal large medical models, and involves a complex construction process that includes data processing, metadata extraction, region of interest localization, and the collection of medical knowledge. Detailed descriptions are generated using large-scale language models to enhance data usability. The dataset was officially released on July 21, 2024, accompanied by a pre-trained model.

Domestic large models have shown strong capabilities in the multimodal field of artificial intelligence. Tencent's Hunyuan-vision and Shanghai AI Lab's InternVL2-40B are leading their respective closed-source and open-source fields, surpassing internationally renowned models. Hunyuan-vision, as the multimodal version of Tencent's Hunyuan large model, is favored by developers for its API calls and is offered for free in Tencent's Yuanbao APP, showcasing practicality and ease of use. Yuanbao APP ranks first in China for multimodal capability assessments through a series of tests.

The Tencent Hunyuan Large Model excelled in the August rankings of the SuperCLUE-V evaluation benchmark for Chinese multimodal large models, securing the top position among domestic large models and placing in the 'Exemplary Leaders' quadrant. Multimodal understanding requires the model to accurately identify image elements, comprehend their relationships, and generate natural language descriptions, testing the model's precision in image recognition and its understanding of the complex real world.