Popular GitHub Repositories for Inference Engine

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

校招、秋招、春招、实习好项目！带你从零实现一个高性能的深度学习推理库，支持大模型 llama2 、Unet、Yolov5、Resnet等模型的推理。Implement a high-performance deep learning inference library step by step

OneDiff: An out-of-the-box acceleration library for diffusion models.

Large-scale LLM inference engine

A highly optimized LLM inference acceleration engine for Llama and its variants.

Adlik: Toolkit for Accelerating Deep Learning Inference

The Qualcomm? AI Hub Models are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm? devices.

A Machine Learning System for Data Enrichment.

Julia package for automated Bayesian inference on a factor graph with reactive message passing

校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

docs for search system and ai infra

The Qualcomm? AI Hub apps are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm? devices.

PyTorch library for cost-effective, fast and easy serving of MoE models.

A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).

Lightweight Inference server for OpenVINO

Ai edge toolbox，专门面向边端设备尤其是嵌入式RTOS平台，AI模型部署工具链，包括模型推理引擎和模型压缩工具

A quick view of high-performance convolution neural networks (CNNs) inference engines on mobile devices.

TinyTensor is a tool for running already trained NN (Neural Network) models to be able to use them for inference of various tasks such as image classification, semantic segmentation, etc.

A robust and efficient TinyML inference engine.