Flex-MoE

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

An open source implementation of CLIP.

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

Mixture-of-Experts for Large Vision-Language Models

Macaw-LLM: Multi-Modal Language Modeling with Image, Video, Audio, and Text Integration

The implementation of "Prismer: A Vision-Language Model with Multi-Task Experts".

?? LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)

Tutel MoE: An Optimized Mixture-of-Experts Implementation

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)