vLLM
vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs. It exposes an OpenAI-compatible HTTP API (/v1) so existing OpenAI clients can serve local models.
vLLM
In Development
This script is currently in active development and may be unstable or incomplete. Use in production environments is not recommended.
This script is currently in active development and may be unstable or incomplete. Use in production environments is not recommended.
Installation
Default install:
1
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVED/main/ct/vllm.sh)"
Configuration
Config file:
1
/opt/vllm/vllm.env
Notes
Select NVIDIA GPU passthrough at container creation. vLLM installs the CUDA build by default and requires an NVIDIA GPU for usable performance; CPU-only inference is impractically slow.
The model to serve is set via VLLM_MODEL in /opt/vllm/vllm.env (default Qwen/Qwen2.5-0.5B-Instruct). Restart the service after changing it: systemctl restart vllm
The configured model is downloaded from HuggingFace on first service start, which needs internet access and disk space. For gated/private models set HF_TOKEN in /opt/vllm/vllm.env. Resource needs scale heavily with the chosen model.