llama.cpp
llama.cpp runs GGUF language models directly in C/C++ with no Python runtime. This installs llama-server, which serves a built-in web chat UI plus an OpenAI-compatible API on the same port, so it can back tools like AnythingLLM, Open WebUI or Claude Code. Lighter and closer to the metal than Ollama.
llama.cpp
In Development
This script is currently in active development and may be unstable or incomplete. Use in production environments is not recommended.
This script is currently in active development and may be unstable or incomplete. Use in production environments is not recommended.
Installation
Default install:
1
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVED/main/ct/llama-cpp.sh)"
Configuration
Config file:
1
/opt/llama-cpp.env
Notes
On first start llama-server downloads the model set in LLAMA_ARG_HF_REPO (default: ggml-org/gemma-3-1b-it-GGUF, roughly 800 MB) into /opt/llama-cpp_data/models, so the container is usable straight away. Swap it for any GGUF repo, or point LLAMA_ARG_MODEL at a local file, then restart.
The web UI and the OpenAI-compatible API share port 8080: /v1/chat/completions, /v1/models and friends. There is no authentication - set LLAMA_ARG_API_KEY in /opt/llama-cpp.env or keep it off untrusted networks.
This is the CPU build (LLAMA_ARG_N_GPU_LAYERS=0). For GPU offload pass a card through and install the matching build - the release also ships vulkan, rocm and sycl variants. Adjust the asset name in the script accordingly.
RAM is the limiting factor: the model must fit alongside the KV cache. 8 GB comfortably runs models up to roughly 7B at Q4. Raise LLAMA_ARG_CTX_SIZE only if you have headroom.