Popular GitHub Repositories for Speculative Decoding

? Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms?

Large-scale LLM inference engine

scalable and robust tree-based speculative decoding algorithm

Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024

[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

LLM Inference on consumer devices

From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence Generation

PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation

minimal C implementation of speculative decoding based on llama2.c

Official Implementation of "GRIFFIN: Effective Token Alignment for Faster Speculative Decoding"

Implementation of Speculative Sampling in "Accelerating Large Language Model Decoding with Speculative Sampling"

SpS-SpecDec: a fast Python lib that boosts autoregressive LM inference with speculative decoding. Inspired by DeepMind, it guesses multiple tokens using a small draft model, verifies with a big one. Get 2-2.5x speedups, no quality drop!

(Re)-implementation of "Prompt Lookup Decoding" by Apoorv Saxena, with extended ideas from LLMA Decoding.

The LLM Defense Framework enhances large language model security through post-processing defenses and statistical guarantees based on one-class SVM. It combines advanced sampling methods with adaptive policy updates and comprehensive evaluation metrics, providing researchers and practitioners with tools to build more secure AI systems.