Popular GitHub Repositories for Data Parallelism

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training

Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.

Ternary Gradients to Reduce Communication in Distributed Deep Learning (TensorFlow)

Orkhon: ML Inference Framework and Server Runtime

Distributed training (multi-node) of a Transformer model

Multi-GPU training for Keras

Deep Learning at Scale Training Event at NERSC

WIP. Veloce is a low-code Ray-based parallelization library that makes machine learning computation novel, efficient, and heterogeneous.

This repository provides hands-on labs on PyTorch-based Distributed Training and SageMaker Distributed Training. It is written to make it easy for beginners to get started, and guides you through step-by-step modifications to the code based on the most basic BERT use cases.

Fast and easy distributed model training examples.

Understanding the effects of data parallelism and sparsity on neural network training

A decentralized and distributed framework for training DNNs

The project is focused on parallelising pre-processing, measuring and machine learning in the cloud, as well as the evaluation and analysis of the cloud performance.

Torch Automatic Distributed Neural Network (TorchAD-NN) training library. Built on top of TorchMPI, this module automatically parallelizes neural network training.

Scaling Unet in Tensorflow

Official Repository for the paper: Distributing Deep Learning Hyperparameter Tuning for 3D Medical Image Segmentation

Towards Rehearsal-based Continual Learning at Scale: distributed CL using Horovod + PyTorch on up to 128 GPUs