Popular GitHub Repositories for Inference

A high-throughput and memory-efficient inference and serving engine for LLMs

Making large AI models cheaper, faster and more accessible

Port of OpenAI's Whisper model in C/C++

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

ncnn is a high-performance neural network inference framework optimized for the mobile platform

Machine Learning Engineering Open Book

SGLang is a fast serving framework for large language models and vision language models.

NVIDIA? TensorRT? is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

Example ? Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using ? Amazon SageMaker.

?1MB lightweight face detection model (1MB轻量级人脸检测模型)

Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

An easy to use PyTorch to TensorRT converter

TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts. TNN has been deployed in multiple Apps from Tencent, such as Mobile QQ, Weishi, Pitu, etc. Contributions are welcome to work in collaborative with us and make TNN a better framework.

On-device Speech Recognition for Apple Silicon

Pre-trained Deep Learning models and demos (high quality and extremely fast)

cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，支持sso登录，大数据平台对接，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU，边缘计算，标注平台，自动化标注，大模型微调，vllm大模型推理，llmops，私有知识库，AI模型应用商店，支持模型一键开发/推理/微调，支持国产cpu/gpu/npu芯片，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/spark/ray/volcano分布式，deepseek训练推理

LightSeq: A High Performance Library for Sequence Processing and Generation

Sparsity-aware deep learning inference runtime for CPUs

Python Library for learning (Structure and Parameter), inference (Probabilistic and Causal), and simulations in Bayesian Networks.