vLLM
vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs. It exposes an OpenAI-compatible HTTP API (/v1) so existing OpenAI clients can serve local models.
vLLM
In Development
This script is currently in active development and may be unstable or incomplete. Use in production environments is not recommended.
This script is currently in active development and may be unstable or incomplete. Use in production environments is not recommended.
Installation
Default install:
1
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVED/main/ct/vllm.sh)"
Notes
Select NVIDIA GPU passthrough at container creation. vLLM installs the CUDA build by default and requires an NVIDIA GPU for usable performance; CPU-only inference is impractically slow.
Model, port, dtype, VRAM share and HuggingFace token can be chosen up front in the script generator; they are written to /opt/vllm/vllm.env. Restart the service after changing the file: systemctl restart vllm
Flags not covered by a field go into VLLM_SERVE_ARGS in /opt/vllm/vllm.env, which is appended to 'vllm serve' verbatim.
The configured model is downloaded from HuggingFace on first service start, which needs internet access and disk space. For gated/private models set HF_TOKEN in /opt/vllm/vllm.env. Resource needs scale heavily with the chosen model.