vLLM
Fast and Easy-to-Use LLM Inference and Serving Platform
InternationalSelectionProgrammingLLMInference
vLLM is a fast, easy-to-use, and efficient library for large language model (LLM) inference and service provision. By leveraging the latest service throughput technologies, efficient memory management, continuous batch processing requests, CUDA/HIP graph fast model execution, quantization techniques, and optimized CUDA kernels, it provides high-performance inference services. vLLM seamlessly integrates with popular HuggingFace models, supports various decoding algorithms including parallel sampling and beam search, supports tensor parallelism for distributed inference, supports streaming output, and is compatible with OpenAI API servers. Moreover, vLLM supports both NVIDIA and AMD GPUs, as well as experimental prefix caching and multi-lora support.
vLLM Visit Over Time
Monthly Visits
184380
Bounce Rate
49.64%
Page per Visit
2.9
Visit Duration
00:04:10