Popular GitHub Repositories for Foundation Models

Making large AI models cheaper, faster and more accessible

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

DeepSeek-VL: Towards Real-World Vision-Language Understanding

Code and models for NExT-GPT: Any-to-Any Multimodal Large Language Model

? TabPFN: Foundation Model for Tabular Data ?

? Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.

[CVPR2024 Highlight][VideoChatGPT] ChatGPT with video understanding! And many more supported LMs such as miniGPT4, StableLM, and MOSS.

SuperCLUE: 中文通用大模型综合性基准 | A Benchmark for Foundation Models in Chinese

Prompt Learning for Vision-Language Models (IJCV'22, CVPR'22)

An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.

[CVPR 2025] Official PyTorch Implementation of MambaVision: A Hybrid Mamba-Transformer Vision Backbone

Must-read Papers on Knowledge Editing for Large Language Models.

A general representation model across vision, audio, language modalities. Paper: ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

A curated list of foundation models for vision and language tasks

[CVPR 2024 ?] Grounding Large Multimodal Model (GLaMM), the first-of-its-kind model capable of generating natural language responses that are seamlessly integrated with object segmentation masks.

[ICLR 2024] Official PyTorch implementation of FasterViT: Fast Vision Transformers with Hierarchical Attention

[ECCV 2024 Best Paper Candidate] PointLLM: Empowering Large Language Models to Understand Point Clouds

[MICCAI 2019 Young Scientist Award] [MEDIA 2020 Best Paper Award] Models Genesis

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models