Popular GitHub Repositories for Inference Server

The goal of RamaLama is to make working with AI boring.

The simplest way to serve AI/ML models in production

A REST API for Caffe using Docker and Go

Work with LLMs on a local environment using containers

Serving AI/ML models in the open standard formats PMML and ONNX with both HTTP (REST API) and gRPC endpoints

ONNX Runtime Server: The ONNX Runtime Server is a server that provides TCP and HTTP/HTTPS REST APIs for ONNX inference.

Orkhon: ML Inference Framework and Server Runtime

Friendli: the fastest serving engine for generative AI

Advanced inference pipeline using NVIDIA Triton Inference Server for CRAFT Text detection (Pytorch), included converter from Pytorch -> ONNX -> TensorRT, Inference pipelines (TensorRT, Triton server - multi-format). Supported model format for Triton inference: TensorRT engine, Torchscript, ONNX

Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)

create your own llm inference server from scratch

Modelz is a developer-first platform for prototyping and deploying machine learning models.

An example of using Redis + RedisAI for a microservice that predicts consumer loan probabilities using Redis as a feature and model store and RedisAI as an inference server.

Serving distributed deep learning models with model parallel swapping.

Effortlessly Deploy and Serve Large Language Models in the Cloud as an API Endpoint for Inference

Run your own production inference code with Sagemaker