Popular GitHub Repositories for Multimodal Learning

Reading list for research topics in multimodal machine learning

An open-source framework for training large multimodal models.

Prompt Learning for Vision-Language Models (IJCV'22, CVPR'22)

[CVPR'24] UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition

A Comparative Framework for Multimodal Recommender Systems

An official implementation for "CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval"

A collection of resources on applications of multi-modal learning in medical imaging.

Multimodal model for text and tabular data with HuggingFace transformers as building block for text data

[NeurIPS 2021] Multiscale Benchmarks for Multimodal Representation Learning

Official Pytorch implementation of "OmniNet: A unified architecture for multi-modal multi-task learning" | Authors: Subhojeet Pramanik, Priyanka Agrawal, Aman Hussain

Multi-modality pre-training

An open source implementation of "Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning", an all-new multi modal AI that uses just a decoder to generate both text and images

[NeurIPS 2023] This repository includes the official implementation of our paper "An Inverse Scaling Law for CLIP Training"

[ECCV'22] Official repository of paper titled "Class-agnostic Object Detection with Multi-modal Transformer".

My implementation of "Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution"

From scratch implementation of a vision language model in pure PyTorch

Multimodal Prompting with Missing Modalities for Visual Recognition, CVPR'23

[CVPR 2022] Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning

[NeurIPS 2024] This repo contains evaluation code for the paper "Are We on the Right Way for Evaluating Large Vision-Language Models"

A collection of resources on multimodal knowledge graph, including datasets, papers and contests.