ReDrafter

Innovative technology for accelerating LLM inference on NVIDIA GPUs

CommonProductProductivityNVIDIA GPULLM inference

ReDrafter is a novel predictive decoding method that significantly enhances the inference speed of large language models (LLMs) on NVIDIA GPUs by combining RNN draft models with dynamic tree attention mechanisms. This technology accelerates token generation for LLMs, reducing the latency experienced by users while decreasing GPU usage and energy consumption. Developed by the Apple Machine Learning Research Team in collaboration with NVIDIA, ReDrafter is integrated into the NVIDIA TensorRT-LLM inference acceleration framework, providing machine learning developers using NVIDIA GPUs with faster token generation capabilities.

- Predictive decoding: Accelerates LLM token generation using RNN draft models and dynamic tree attention mechanisms.
- Performance enhancement: Achieves up to a 3.5 times speed improvement in token generation per step on open-source models.
- TensorRT-LLM integration: Collaborated with NVIDIA to incorporate ReDrafter into the TensorRT-LLM framework
enhancing compatibility with complex models and decoding methods.
- Latency reduction: Significantly lowers user latency when using LLMs by improving inference efficiency.
- Cost reduction: Lowers computational costs by reducing GPU usage and energy consumption.
- Open-source model support: ReDrafter supports a variety of open-source LLMs
increasing the technology's accessibility and range of applications.
- Easy deployment: ML developers can easily apply ReDrafter to production LLM applications to enjoy the benefits of acceleration.

\The target audience is machine learning developers
particularly those utilizing NVIDIA GPUs for LLM inference. ReDrafter enhances inference speed and reduces latency
allowing these developers to deploy and optimize their LLM applications more rapidly
thereby improving user experience and lowering operational costs.\

Use ReDrafter to accelerate the inference process of production models with billions of parameters.
Deploy ReDrafter on NVIDIA GPUs for a 2.7 times increase in tokens generated per second.
Integrate ReDrafter into TensorRT-LLM to optimize LLM inference performance.

1. Install and configure the NVIDIA TensorRT-LLM environment.
2. Obtain the open-source code for ReDrafter from GitHub.
3. Follow the documentation to integrate ReDrafter into the TensorRT-LLM framework.
4. Prepare or select an open-source LLM model for testing.
5. Use ReDrafter to accelerate LLM inference.

Visit