Recently, Meta AI open-sourced a foundational multimodal language model named SPIRIT LM, which can freely mix text and speech, opening new possibilities for multimodal tasks involving audio and text. SPIRIT LM is based on a pre-trained text language model with 7 billion parameters, which has been continuously trained on text and speech units, expanding into the speech modality. It can understand and generate text like a large text model, while also being capable of understanding and generating speech, and even mixing text and speech to create various forms of expression.

The Yi-VL multimodal language model has been launched, including two versions: Yi-VL-34B and Yi-VL-6B. The Yi-VL model demonstrates exceptional capabilities in visual-text understanding and dialogue generation. It has achieved leading performance on English and Chinese datasets. Yi-VL-34B surpasses other multimodal large models with an accuracy rate of 41.6%. The Yi-VL model is based on the LLaVA architecture, featuring strong language understanding and generation abilities.

NExT-GPT is an open source multimodal language model developed by the National University of Singapore, capable of processing text, images, videos, and audio, providing robust support for multimedia AI applications. It features a three-layer architecture, including linear projection, Vicuna LLM core, and modality-specific transformation layers, with intermediate layer training conducted using MosIT technology. The open-source contribution enables researchers and developers to create applications that integrate multimodal inputs, with potential applications spanning a wide range of fields. What sets NExT-GPT apart is its ability to generate modalities based on user requests.

Researchers from Apple and Columbia University have developed the Ferret multimodal language model, aimed at achieving advanced image understanding and description. The Ferret model possesses strong global understanding capabilities, able to handle free text and referenced regions, outperforming traditional models. The researchers created the GRIT dataset, which includes 1.1 million samples, to guide the model in reference and localization tasks. Ferret-Bench evaluations show that Ferret outperforms the best MLLM models by an average of 20.4% while reducing object-related errors.