llama.cpp review: local LLM inference engine that changed everything

I remember the moment I first ran a 7B parameter model on my laptop and watched tokens stream out at 40 per second. That moment existed because of llama.cpp. What started as a weekend project by Georgi Gerganov has become the de facto standard for local LLM inference — powering everything from hobbyist chatbots to production APIs. If you've ever wanted to run Llama, Qwen, Mistral, or any GGUF model without a cloud GPU, this is the tool that made it possible.

What is llama.cpp?

llama.cpp is an MIT-licensed C/C++ inference engine designed to run LLMs and VLMs (vision-language models) on consumer hardware. It's built on top of the ggml tensor library, which provides the low-level compute primitives. Unlike Python-based frameworks that rely on heavy dependencies like PyTorch, llama.cpp is plain C/C++ with zero required third-party libraries. You build a binary, download a GGUF model, and you're running.

The project supports a stunning range of backends: ARM NEON, Metal, CUDA, HIP, Vulkan, SYCL, OpenCL, and even WebGPU. Its CPU implementation is heavily optimized with AVX, AVX2, AVX512, and RISC-V vector extensions. The headline feature is quantized inference — you can load a 70B model in 4-bit precision and run it on a machine with 16GB of RAM. No other project has come close to matching this breadth of hardware support.

Why it caught my eye

I got into AI tooling during the peak of cloud-API hype. Every service required an API key, a credit card, and an internet connection. Models were black boxes living in someone else's data center. The problem that llama.cpp solves is ownership: it puts open-weight models directly on your hardware, with no telemetry, no rate limits, and no monthly bill. For privacy-conscious users, offline environments, and anyone tired of paying token fees, this is a genuine paradigm shift.

The second problem it solves is portability. Python-based inference stacks are heavy — a typical PyTorch environment is gigabytes of dependencies. llama.cpp compiles to a single binary that you can copy to a Raspberry Pi, a Chromebook, or a cloud VM. For builders who ship embedded AI or want fast startup times, this is a game-changer.

How it works

At the core is the GGUF file format — a single-file format that bundles model weights, tokenizer, and metadata. Quantization maps FP16 weights into lower-precision integers (4-bit, 5-bit, 8-bit) to shrink memory and accelerate inference with minimal quality loss. The engine then loads the GGUF into a compute graph through ggml, which schedules operations across the available backends.

Key concepts:

• Backend scheduling: llama.cpp detects your hardware and routes tensor operations to CPU, GPU, or both. For models larger than VRAM, CPU+GPU hybrid inference lets you offload layers until memory runs out.
• Quantization formats: choose from q4_0, q5_1, q8_0, and newer IQ (integer quantization) variants that balance perplexity vs. speed.
• llama-server: a built-in HTTP server that exposes an OpenAI-compatible API, so you can point existing tools at a local endpoint.
• GBNF grammars: constrain output to valid JSON, code, or custom formats — essential for production use.

Everything is designed around zero-copy memory management and minimal allocation overhead. The result is blazing-fast token generation, especially on Apple Silicon through Metal and on NVIDIA GPUs via custom CUDA kernels.

Quick start

The easiest route is to use pre-built binaries from the releases page or via Docker. If you prefer building from source, you need a C++ compiler and CMake:

# Build from source

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_CUBLAS=ON  # add CUDA support if you have an NVIDIA GPU

cmake --build build --config Release -j

Then download and run a small model directly from Hugging Face:

# Install and run from a single command (binary from releases)

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Start an OpenAI-compatible server

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The -hf flag fetches the GGUF file from the Hugging Face Hub and caches it locally. After that, the model runs completely offline. The server exposes /v1/chat/completions, so you can use it with any OpenAI SDK.

Real-world example

Let's build a small semantic search script using the server API. Start llama serve with an embedding model (or a regular LLM with an embedding endpoint). Here's a Python snippet that sends a prompt to a local llama.cpp server:

import json
import urllib.request

def chat(prompt):
    data = json.dumps({
        "messages": [{"role": "user", "content": prompt}]
    }).encode()
    req = urllib.request.Request(
        "http://localhost:8080/v1/chat/completions",
        data=data, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req) as resp:
        return json.load(resp)["choices"][0]["message"]["content"]

print(chat("Explain what ggml is in one sentence."))

If you need structured output, pass a GBNF grammar that forces JSON:

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF --grammar-file grammar/json.gbnf -p "Give me a weather report for London"

That grammar ensures the output parses as valid JSON, which is invaluable when you're chaining LLM calls into pipelines. The server also supports multimodality with vision models via /v1/chat/completions and image input.

Pros and cons

Pros:

• Runs on virtually any hardware: Raspberry Pi, MacBook, gaming PC, or data center GPU.
• Single-file models (GGUF) make distribution trivial.
• Fast token generation thanks to aggressive quantization and SIMD optimizations.
• Truly open source — MIT license, no restrictions on commercial use.
• Built-in server with OpenAI-compatible API simplifies integration.

Cons:

• Setup can be intimidating for non-developers — you need to handle model files, CLI flags, and hardware quirks.
• Quality drops with aggressive quantization; you have to benchmark for your use case.
• The project moves fast, and nightly builds sometimes break. Production users must pin releases.
• No official GUI (third-party UIs exist but they're extra moving parts).

Alternatives

• Ollama — a user-friendly wrapper around llama.cpp-like runtime with a simpler CLI and model management. Great for prompters, less flexible for engineers.
• LM Studio — a polished desktop app that packages llama.cpp under the hood. Best if you want a point-and-click local LLM experience.
• llama.cpp — the engine itself. If you're embedding inference into an app or want lower-level control, nothing beats it.

My verdict — should you use it?

llama.cpp is the go-to choice for embedded LLM inference, server deployments, and any environment where Python is overkill. If you're building a product around local AI or need to ship inference on custom hardware, use it. If you just want to chat with a model on your laptop, Ollama or LM Studio will be smoother. But know that behind those friendly tools, it's llama.cpp doing the heavy lifting — and learning it directly gives you a superpower.