Recently, research teams from Peking University announced the release of an open-source multimodal model called LLaVA-o1, which is claimed to be the first visual language model capable of spontaneous and systematic reasoning, comparable to GPT-o1. The model excels in six challenging multimodal benchmark tests, with its 11B parameter version outperforming competitors such as Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct.

Recently, an open source multimodal AI model named Molmo has drawn widespread attention in the industry. This AI system, based on Qwen2-72B and leveraging OpenAI's CLIP as the visual processing engine, is challenging the dominance of traditional commercial models with its outstanding performance and innovative features. Molmo's standout characteristic is its efficient performance. Despite its relatively small size, it can compete with competitors that are ten times larger in processing capability. This 'small but exquisite' design philosophy not only enhances the model's

The latest version 2.6 of WallFacer’s MiniCPM-V series has rapidly climbed to the Top 3 on GitHub and HuggingFace trends, surpassing ten thousand stars. Since its release in February, it has accumulated over a million downloads, becoming a benchmark for on-device model capabilities. MiniCPM-V2.6 achieves performance enhancements for on-device multimodal models with 8 billion parameters, including real-time video understanding, multi-image joint understanding, and multi-image in-context learning, with a quantized backend memory of only 6GB and an inference speed of up to 18 tokens.

Google has released its latest AI masterpiece, Gemini 1.5 Pro, offering an experimental version 0801 through Google AI Studio and the Gemini API. This model leads the LMSYS leaderboard with an ELO score of 1300, surpassing OpenAI's GPT-4o and Anthropic's Claude-3.5 Sonnet. Gemini 1.5 Pro excels in multilingual tasks, mathematics, coding, and visual tasks, featuring a context window of 2 million tokens.

Shusheng · Puyu Lingbi (InternLM-XComposer) Version 2.5 was developed by the Shanghai Artificial Intelligence Laboratory, focusing on long context input and output capabilities, operating smoothly within a length of 96K, and trained with 24K interleaved image-text data. Key upgrades include: high-resolution image understanding, fine-grained video understanding, and multi-turn multi-image dialogue. In application, it can create web pages and write high-quality text-image articles. Evaluations show it surpasses state-of-the-art open-source models across 16 benchmark tests and performs at par with key tasks compared to GPT-4V and Gem.

Meta is about to make a big move! They are set to launch an open-source language model called Llama3405B, which is not only their largest model to date but also the largest open-source language model in history. This behemoth, with an astonishing 405 billion parameters, can effortlessly navigate between images and text, completely revolutionizing the old ways that could only handle text.Key Highlights: Meta will release Llama3405B on July 23rd, a multimodal model with 405 billion parameters. Dec

The French independent non-profit AI research lab Kyutai has launched a voice assistant called Moshi, which is a revolutionary real-time local multimodal foundational model. This innovative model imitates and surpasses some of the functionalities demonstrated by OpenAI's GPT-4o released in May in certain aspects.Product Entry: https://top.aibase.com/tool/moshi-chat Moshi is designed to understand and express emotions, capable of conversing in different accents, including French. It can simultane

In the world of AI, we have just welcomed a remarkable new member—Cambrian-1, a large multimodal language model (MLLM) developed by industry giants LeCun and Xie Saineng. The emergence of this model represents not only a technological leap but also a profound reflection on the research of multimodal learning.

Recently, Meta quietly released six research achievements that bring new applications and technological breakthroughs to the field of AI. These include a multimodal model, a text-to-music generation model, audio watermarking technology, datasets, and several other projects. Let's take a closer look at these research achievements.

Sketchpad enables language models to draw using lines, boxes, and markers, which is closer to human sketching and facilitates reasoning. Additionally, Sketchpad can utilize specialized visual models during the drawing process, such as using object detection models to create bounding boxes and segmentation models to draw masks, further enhancing visual perception and reasoning capabilities.

Microsoft has open-sourced the multimodal model LLaVA-1.5, inheriting the LLaVA architecture and introducing new features. Researchers have tested it in visual question answering, natural language processing, image generation, and other areas, showing that LLaVA-1.5 has reached the highest level among open-source models.

The launch of GPT-4 features multimodal capabilities surpassing its predecessors, leading AI technology development. Baidu introduces Wenxin Yiyan, initiating the domestic multimodal competition with a user base exceeding 100 million. AI Sun Yanzi emerges, triggering controversies over AI music synthesis and copyright. Maoyake Camera launches, sweeping social media and igniting a trend in AI photography products. The state releases 'Interim Measures for the Management of Generative Artificial Intelligence Services' to regulate the AIGC industry.

Shanghai AI Lab, together with Ruijin Hospital affiliated to Shanghai Jiao Tong University School of Medicine, has released the medical multimodal foundational model group 'PuYi 2.0'. PuYi 2.0 adds multi-domain models and incremental language parameters, covering various data modalities such as medical images, medical texts, and bioinformatics. New open-source datasets include the medical image segmentation dataset SA-Med2D-20M and the pathology dataset SNOW. PuYi 2.0 incorporates an evaluation module to provide reference implementations for medical model capabilities, delivering a one-stop open-source solution for large medical models.

Gemini is the latest generation AI model launched by Google, featuring multimodal capabilities. Gemini comes in three sizes: Ultra, Pro, and Nano, suitable for different tasks and devices. Gemini demonstrates exceptional performance, surpassing other models in multiple benchmark tests. It has multimodal reasoning and encoding capabilities, capable of processing text, images, audio, and more. Gemini is expected to be available for developers and enterprise customers early next year.

The University of California, Santa Cruz has developed the MiniGPT-5 model, which achieves effective alignment between text and images. MiniGPT-5 utilizes a multimodal encoder and Generative Vokens technology to generate coherent, high-quality text and images. The model outperforms comparative baselines in tests across multiple datasets, demonstrating strong adaptability.

["Zhejiang University alumni collaborated with Microsoft Research to launch the multimodal model LLaVA, challenging GPT-4V.","LLaVA has demonstrated excellent performance on 11 testing datasets, garnering over 6k stars.","The model exhibits comprehensive capabilities, exceeding 85% of the level of GPT-4V.","The open-source code, model, and training data for LLaVA are now available for use."]

The CogVLM-17B, developed through a collaboration between Tsinghua University and Zhipu AI, is a domestic multimodal model with outstanding performance. CogVLM-17B can not only recognize objects in images but also distinguish between fully visible and partially visible objects. The model employs a unique deep fusion method, achieving deep alignment of image features and text features through four key components. CogVLM-17B outperforms Google's models in various fields and is aptly referred to as the '14-sided warrior', showcasing its multimodal capabilities.