ManagerTower

An open source implementation of CLIP.

Macaw-LLM: Multi-Modal Language Modeling with Image, Video, Audio, and Text Integration

[ACL 2024 ?] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.

The implementation of "Prismer: A Vision-Language Model with Multi-Task Experts".

[ECCV 2024 Oral] DriveLM: Driving with Graph Visual Question Answering

A general representation model across vision, audio, language modalities. Paper: ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

Pix2Seq codebase: multi-tasks with generative modeling (autoregressive and diffusion)

A novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings.

?? LLaVA++: Extending LLaVA with Phi-3 and LLaMA-3 (LLaVA LLaMA-3, LLaVA Phi-3)

[CVPR 2024] Alpha-CLIP: A CLIP Model Focusing on Wherever You Want