model-parallelism

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

A GPipe implementation in PyTorch

Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.

LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training

Slicing a PyTorch Tensor Into Parallel Shards

Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.

A curated list of awesome projects and papers for distributed training or inference

Serving Inside Pytorch

Decentralized LLMs fine-tuning and inference with offloading