llama.cpp
llama.cpp runs GGUF language models directly in C/C++ with no Python runtime. Its llama-server offers a built-in web chat and an OpenAI-compatible API, so it can back tools like AnythingLLM, Open WebUI or Claude Code.
llama.cpp
In Development
This script is currently in active development and may be unstable or incomplete. Use in production environments is not recommended.
This script is currently in active development and may be unstable or incomplete. Use in production environments is not recommended.
Installation
Default install:
1
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/DevScripts/main/ct/llama-cpp.sh)"
Notes
The web UI and the API have no authentication unless you set an API key at install time or LLAMA_ARG_API_KEY in /opt/llama-cpp.env.
The first start downloads gemma-4-E2B-it Q4_0 (about 3.5 GB). Set LLAMA_ARG_HF_REPO to another GGUF repo, or LLAMA_ARG_MODEL to a local file, then run systemctl restart llama-cpp.
Pick the build with var_backend: Vulkan works on any GPU, or use CUDA, ROCm, SYCL or OpenVINO. NVIDIA on Vulkan needs the driver's Vulkan libraries in the container.