Popular GitHub Repositories for Llm Inference

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

Official inference library for Mistral models

High-speed Large Language Model Serving for Local Deployment

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.

?A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, Flash-Attention, Paged-Attention, Parallelism, etc. ??

Sparsity-aware deep learning inference runtime for CPUs

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

FlashInfer: Kernel Library for LLM Serving

Code examples and resources for DBRX, a large language model developed by Databricks

MiniMax-01 is a simple implementation of the MiniMax algorithm, a widely used strategy for decision-making in two-player turn-based games like Tic-Tac-Toe. The algorithm aims to minimize the maximum possible loss for the player, making it a popular choice for developing AI opponents in various game scenarios.

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Intelligent (edge and LLM) proxy for agents. Move faster by letting Arch handle the pesky heavy lifting in building agentic apps. Fast ?? task clarification, agent routing, seamless integration of prompts with tools for common tasks, and unified access and observability of LLMs.

? Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms?

Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.