Popular GitHub Repositories for Model Parallelism

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

A GPipe implementation in PyTorch

LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training

Slicing a PyTorch Tensor Into Parallel Shards

Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.

Distributed training (multi-node) of a Transformer model

Distributed training of DNNs ? C++/MPI Proxies (GPT-2, GPT-3, CosmoFlow, DLRM)

WIP. Veloce is a low-code Ray-based parallelization library that makes machine learning computation novel, efficient, and heterogeneous.

Deep Learning at Scale Training Event at NERSC

Fast and easy distributed model training examples.

Performance Estimates for Transformer AI Models in Science

Official implementation of DynPartition: Automatic Optimal Pipeline Parallelism of Dynamic Neural Networks over Heterogeneous GPU Systems for Inference Tasks

Serving distributed deep learning models with model parallel swapping.

Model parallelism for NN architectures with skip connections (eg. ResNets, UNets)

A decentralized and distributed framework for training DNNs

The project is focused on parallelising pre-processing, measuring and machine learning in the cloud, as well as the evaluation and analysis of the cloud performance.

Torch Automatic Distributed Neural Network (TorchAD-NN) training library. Built on top of TorchMPI, this module automatically parallelizes neural network training.

distributed tensorflow (model parallelism) example repository

An MPI-based distributed model parallelism technique for MLP