lmms-finetune

A unified codebase for fine-tuning large multimodal models.

CommonProductProgrammingFine-tuningMultimodal

lmms-finetune is a unified codebase designed to simplify the fine-tuning process for large multimodal models (LMMs). It provides a structured framework that allows users to seamlessly integrate and fine-tune cutting-edge LMMs, supporting full fine-tuning as well as strategies like LoRA. The codebase is lightweight and straightforward, making it easy to understand and modify, and it supports various models including LLaVA-1.5, Phi-3-Vision, Qwen-VL-Chat, LLaVA-NeXT-Interleave, and LLaVA-NeXT-Video.

Provides a unified structural framework for fine-tuning
simplifying integration and optimization processes
Supports various fine-tuning strategies
including full fine-tuning
LoRA
and Q-LoRA
Maintains codebase simplicity for ease of understanding and modification
Supports multiple types of LMMs
including single-image
multi-image/interleaved image

This tool is aimed at researchers and developers
particularly those who need to fine-tune large multimodal models for specific tasks or datasets. lmms-finetune provides a simple
flexible
and easily extensible platform
allowing users to focus on model fine-tuning and experimentation without getting bogged down by low-level implementation details.

Researchers use lmms-finetune to fine-tune LLaVA-1.5 to enhance performance on specific video content analysis tasks.
Developers utilize this codebase to fine-tune the Phi-3-Vision model for new image recognition tasks.
Educational institutions adopt lmms-finetune for teaching
helping students understand the fine-tuning process and applications of large multimodal models.

Clone the repository to your local environment: `git clone https://github.com/zjysteven/lmms-finetune.git`
Set up and activate the conda environment: `conda create -n lmms-finetune python=3.10 -y` then `conda activate lmms-finetune`
Install dependencies: `python -m pip install -r requirements.txt`
Install additional libraries as needed
such as Flash Attention: `python -m pip install --no-cache-dir --no-build-isolation flash-attn`

Visit

lmms-finetune Visit Over Time

Monthly Visits

494758773

Bounce Rate

37.69%

Page per Visit

5.7

Visit Duration

00:06:29

lmms-finetune Visit Trend

lmms-finetune Visit Geography

lmms-finetune Traffic Sources

lmms-finetune Alternatives

InternVL 2.5 — Open-source multimodal large language model series

Productivity•multimodal•large language model

InternVL2_5-4B — A multimodal large language model that integrates visual and language understanding.

Image•Multimodal•Large Language Model

InternVL2_5-2B — A multimodal large language model that supports deep interaction between images and text.

Image•Multimodal•Large Language Model

InternVL2_5-1B — A large multimodal language model that supports image and text understanding.

Image•Multimodal•Large Language Model

MAmmoTH-VL — A Large-Scale Multimodal Reasoning and Instruction Tuning Platform

Others•Multimodal•Reasoning

InternVL2_5-8B — A multimodal large language model supporting interaction understanding between images and text.

Image•Multimodal•Large Language Model

InternVL2_5-26B — A large multimodal language model that integrates visual and linguistic understanding.

Image•Multimodal•Large Language Model

InternVL2_5-78B — Advanced multimodal large language model series

Image•Multimodal•Large Language Model

Llama-3.3-70B-Instruct — A large multilingual language model with 70 billion parameters.

Programming•Large Language Model•Multilingual

102

Pixtral-Large-Instruct-2411 — A 124B-parameter multimodal large language model.

Productivity•Multimodal•Large Language Model

102

Ferret-UI-Llama8b — A multimodal large language model based on Llama-3-8B, focused on UI tasks.

Programming•Multimodal•Large Language Model

186

NVLM 1.0 — Cutting-edge multimodal large language model

Productivity•Multimodal•Large Language Model

Llama-3.2-11B-Vision — A multimodal large language model that supports image and text processing.

Productivity•Multimodal•Image Processing

684

mPLUG-Owl3 — A multimodal large language model that understands long image sequences.

Image•Multimodal•Image Understanding

144

Data-Juicer — A one-stop data processing system that provides high-quality data for large language models.

Productivity•Machine Learning•Data Science

258

Llama3 — Large language model supporting various parameter sizes.

Programming•Large language model•Natural language processing

Meta-Llama-3.1-70B — A large multilingual text generation model with 70 billion parameters.

Productivity•Large Language Model•Multilingual

480

Llama-3[8B] Meditron V1.0 — An 8 billion parameter large language model (LLM) based on biomedical data

Productivity•Biomedical•Large Language Model

600

MA-LMM — MA-LMM is a large-scale multimodal model for long-term video understanding.

Video•Video Understanding•Multimodal

684

mPLUG-DocOwl — A modular multimodal large language model for document understanding

Productivity•Document Understanding•Multimodal

234

MiniGemini — A multimodal large language model capable of understanding and generating images

Programming•Multimodal•Visual Language Model

2406

M2UGen — Multimodal Music Understanding and Generation System

Music•Music Generation•Music Understanding

6114

ChatHi — Multi-functional AI Intelligent Assistant Platform

chatting•Intelligent Assistant•Large Language Model

Infini-Megrez — Multimodal understanding model for edge applications, enabling intelligent edge solutions through hardware-software collaboration.

Productivity•Artificial Intelligence•Deep Learning

POINTS-Yi-1.5-9B-Chat — Latest advancements in visual language models, integrating new technologies from WeChat AI.

Productivity•Visual Language Model•WeChat AI

POINTS-Qwen-2-5-7B-Chat — Latest advancements in visual language models

Productivity•Visual Language Models•Multimodal

Best AI Websites & Tools

lmms-finetune

lmms-finetune Visit Over Time

lmms-finetune Visit Trend

lmms-finetune Visit Geography

lmms-finetune Traffic Sources

lmms-finetune Alternatives

InternVL 2.5 — Open-source multimodal large language model series

InternVL2_5-4B — A multimodal large language model that integrates visual and language understanding.

InternVL2_5-2B — A multimodal large language model that supports deep interaction between images and text.

InternVL2_5-1B — A large multimodal language model that supports image and text understanding.

MAmmoTH-VL — A Large-Scale Multimodal Reasoning and Instruction Tuning Platform

InternVL2_5-8B — A multimodal large language model supporting interaction understanding between images and text.

InternVL2_5-26B — A large multimodal language model that integrates visual and linguistic understanding.

InternVL2_5-78B — Advanced multimodal large language model series

Llama-3.3-70B-Instruct — A large multilingual language model with 70 billion parameters.

Pixtral-Large-Instruct-2411 — A 124B-parameter multimodal large language model.

Ferret-UI-Llama8b — A multimodal large language model based on Llama-3-8B, focused on UI tasks.

NVLM 1.0 — Cutting-edge multimodal large language model

Llama-3.2-11B-Vision — A multimodal large language model that supports image and text processing.

mPLUG-Owl3 — A multimodal large language model that understands long image sequences.

Data-Juicer — A one-stop data processing system that provides high-quality data for large language models.

Llama3 — Large language model supporting various parameter sizes.

Meta-Llama-3.1-70B — A large multilingual text generation model with 70 billion parameters.

AMchat — Intelligent assistant for solving higher-level mathematics problems

OmAgent — A multimodal intelligent agent framework for tackling complex tasks

Higgs-Llama-3-70B — A large language model optimized for role-playing

Llama-3[8B] Meditron V1.0 — An 8 billion parameter large language model (LLM) based on biomedical data

MA-LMM — MA-LMM is a large-scale multimodal model for long-term video understanding.

mPLUG-DocOwl — A modular multimodal large language model for document understanding

MiniGemini — A multimodal large language model capable of understanding and generating images

M2UGen — Multimodal Music Understanding and Generation System

ChatHi — Multi-functional AI Intelligent Assistant Platform

Infini-Megrez — Multimodal understanding model for edge applications, enabling intelligent edge solutions through hardware-software collaboration.

POINTS-Yi-1.5-9B-Chat — Latest advancements in visual language models, integrating new technologies from WeChat AI.

POINTS-Qwen-2-5-7B-Chat — Latest advancements in visual language models