lmms-finetune

A unified codebase for fine-tuning large multimodal models.

CommonProductProgrammingFine-tuningMultimodal

lmms-finetune is a unified codebase designed to simplify the fine-tuning process for large multimodal models (LMMs). It provides a structured framework that allows users to seamlessly integrate and fine-tune cutting-edge LMMs, supporting full fine-tuning as well as strategies like LoRA. The codebase is lightweight and straightforward, making it easy to understand and modify, and it supports various models including LLaVA-1.5, Phi-3-Vision, Qwen-VL-Chat, LLaVA-NeXT-Interleave, and LLaVA-NeXT-Video.

Provides a unified structural framework for fine-tuning
simplifying integration and optimization processes
Supports various fine-tuning strategies
including full fine-tuning
LoRA
and Q-LoRA
Maintains codebase simplicity for ease of understanding and modification
Supports multiple types of LMMs
including single-image
multi-image/interleaved image

This tool is aimed at researchers and developers
particularly those who need to fine-tune large multimodal models for specific tasks or datasets. lmms-finetune provides a simple
flexible
and easily extensible platform
allowing users to focus on model fine-tuning and experimentation without getting bogged down by low-level implementation details.

Researchers use lmms-finetune to fine-tune LLaVA-1.5 to enhance performance on specific video content analysis tasks.
Developers utilize this codebase to fine-tune the Phi-3-Vision model for new image recognition tasks.
Educational institutions adopt lmms-finetune for teaching
helping students understand the fine-tuning process and applications of large multimodal models.

Clone the repository to your local environment: `git clone https://github.com/zjysteven/lmms-finetune.git`
Set up and activate the conda environment: `conda create -n lmms-finetune python=3.10 -y` then `conda activate lmms-finetune`
Install dependencies: `python -m pip install -r requirements.txt`
Install additional libraries as needed
such as Flash Attention: `python -m pip install --no-cache-dir --no-build-isolation flash-attn`

Visit

lmms-finetune Visit Over Time

Monthly Visits

503747431

Bounce Rate

37.31%

Page per Visit

5.7

Visit Duration

00:06:44

lmms-finetune Visit Trend

lmms-finetune Visit Geography

lmms-finetune Traffic Sources

lmms-finetune Alternatives

mPLUG-Owl3 — A multimodal large language model that understands long image sequences.

Image•Multimodal•Image Understanding

Data-Juicer — A one-stop data processing system that provides high-quality data for large language models.

Productivity•Machine Learning•Data Science

174

Llama3 — Large language model supporting various parameter sizes.

Programming•Large language model•Natural language processing

Meta-Llama-3.1-70B — A large multilingual text generation model with 70 billion parameters.

Productivity•Large Language Model•Multilingual

360

Llama-3[8B] Meditron V1.0 — An 8 billion parameter large language model (LLM) based on biomedical data

Productivity•Biomedical•Large Language Model

594

MA-LMM — MA-LMM is a large-scale multimodal model for long-term video understanding.

Video•Video Understanding•Multimodal

666

mPLUG-DocOwl — A modular multimodal large language model for document understanding

Productivity•Document Understanding•Multimodal

216

MiniGemini — A multimodal large language model capable of understanding and generating images

Programming•Multimodal•Visual Language Model

2322

M2UGen — Multimodal Music Understanding and Generation System

Music•Music Generation•Music Understanding

5808

Pixtral-12B-2409 — A multimodal model with 12 billion parameters, integrating a visual encoder for image and text processing.

Productivity•Multimodal•Image Processing

Pixtral 12B — The first multimodal Mistral model, supporting hybrid task processing for images and text.

Productivity•Multimodal•AI Model

Zhipu AI Large Model Open Platform — Integrate large models with just a few lines of code.

ChineseSelection•AI Models•Large Models

XVERSE-MoE-A36B — A large multilingual language model that supports text generation across various fields.

Programming•Multilingual•Large Language Model

LLaMA-Omni — A low-latency, high-quality end-to-end speech interaction model

chatting•Speech Interaction•End-to-End Model

102

pixtral-12b-240910 — A multimodal large language model that supports understanding of both images and text.

Image•Multimodal•Image Processing

ell — A lightweight programming library for language models, treating prompts as functions.

InternationalSelection•Language Model•Programming Library

West Lake AI Model — A multimodal model with high emotional and intellectual intelligence

ChineseSelection•Artificial Intelligence•Multimodal

120

Reflection Llama-3.1 70B — The world's leading open-source large language model

Productivity•Large Language Model•Natural Language Processing

264

OLMoE-1B-7B — An efficient open-source large language model.

Productivity•Large Language Model•Open Source

Mini-Omni — An open-source multimodal large language model that supports real-time voice input and streaming audio output.

Productivity•Multimodal•Speech Recognition

288

C4AI CommandR 08-2024 — A high-performance generation model with 3.5 billion parameters

Productivity•Large Language Model•Multilingual Support

SlowFast-LLaVA — A large language model for video understanding and reasoning that does not require training.

Productivity•Video Question Answering•Multimodal Learning

IMYAI Smart Assistant — A versatile smart assistant that meets diverse application needs.

ChineseSelection•Intelligent•Multimodal

138

Show-o — A unified transformer for multimodal understanding and generation.

Image•Artificial Intelligence•Multimodal

162

Phi-3.5-vision — An advanced multimodal model that supports image and text understanding.

Programming•Multimodal•Image Understanding

132

Phi-3.5-mini-instruct — A lightweight, multilingual advanced text generation model

Programming•Text Generation•Multilingual

Best AI Websites & Tools