vLLM
vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs. It exposes an OpenAI-compatible HTTP API (/v1) so existing OpenAI clients can serve local models.
vLLM
In Development
This script is currently in active development and may be unstable or incomplete. Use in production environments is not recommended.
This script is currently in active development and may be unstable or incomplete. Use in production environments is not recommended.
Installation
Default install:
1
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/DevScripts/main/ct/vllm.sh)"
Notes
Without an API key the API accepts requests from anyone. Set VLLM_API_KEY in /opt/vllm/vllm.env (clients send it as a bearer token) and run systemctl restart vllm.
Pass a GPU through when creating the container. CPU-only inference is impractically slow.
The model downloads from HuggingFace on first start. Change it, HF_TOKEN (gated models) or extra flags (VLLM_SERVE_ARGS) in /opt/vllm/vllm.env, then run systemctl restart vllm.