Local LLM Memory: Weights Are Only the Starting Point
Estimate local LLM memory from model weights, KV cache, context length, and runtime overhead. Use transparent calculations before testing your actual hardware.
Local LLM memory planning starts with model weights, but it cannot stop there. A model file that fits on disk or even in RAM may still exceed the available memory once the runtime allocates its context cache and working buffers. Plan for the workload you intend to run, not just the download size.
The useful first question is concrete: can this model, at this precision, handle this context length and number of simultaneous requests on this machine? Parameter count alone cannot answer it.
Calculate a lower bound for weights
A rough weight estimate is parameter count multiplied by bits per parameter, divided by eight. For a hypothetical dense model with 8 billion parameters at exactly 4 bits per parameter, that gives 4 billion bytes, or about 3.73 GiB.
That is a simplified payload calculation, not a promise about a real quantized file or runtime allocation. Quantization formats may add scales and metadata, and some tensors may use a different precision. The actual artifact and implementation determine the real storage requirement.
The same arithmetic makes an important limit clear: 70 billion parameters at an idealized 4 bits require 35 billion bytes, roughly 32.6 GiB, for weights alone. That does not fit entirely in 16 GiB of memory. Offloading changes where data resides; it does not make the data disappear.
Hugging Face Accelerate's memory estimator provides tooling to estimate model memory from architecture information. Treat an estimate as a planning input and verify actual inference allocation in your chosen runtime.
Add the KV cache
Many autoregressive transformer implementations cache attention keys and values so generation can reuse prior computation. The cache grows with the retained sequence length and the number of active sequences, subject to the model's attention design and runtime strategy.
Transformers' cache documentation describes dynamic, static, offloaded, and quantized cache options, with different memory and performance tradeoffs. Weight quantization and cache quantization are separate choices; lowering weight precision does not automatically lower cache precision.
For a conventional full-attention decoder with uniform layers, a simplified cache estimate is:
KV bytes = 2 × layers × KV heads × head dimension
× retained tokens × bytes per element × sequences
The factor of two accounts for keys and values. This formula is an illustrative approximation. Architectures with sliding-window attention, hybrid layers, compressed state, or other cache layouts need a model-specific calculation.
Suppose the hypothetical model has 32 layers, 8 KV heads, head dimension 128, a retained sequence of 8,192 tokens, and 2-byte cache elements. For one sequence, the formula gives 1,073,741,824 bytes: 1 GiB. Four independent sequences at that length would require about 4 GiB under those assumptions.
This example shows why a comfortable single-user test can fail under concurrent load. It is not a measured memory figure for a named model.
Leave room for runtime work
The runtime also needs buffers, activations, allocator overhead, and potentially temporary copies during loading or conversion. The operating system and other applications need memory too. On a unified-memory machine, CPU and GPU allocations compete within the same physical pool.
Avoid planning to consume every advertised byte. Measure the usable memory with the normal desktop or service workload running. If the operating system starts paging heavily, a configuration can technically continue while delivering an unacceptable experience.
Distinguish host RAM from device memory on systems with discrete GPUs. A model may use both, and the transfer path between them affects performance. “Fits in system RAM” is not equivalent to “runs entirely on the GPU.”
Loading peaks can differ from steady-state usage. A conversion step or duplicate tensor allocation may require extra capacity before the first token appears. Test startup as well as an already warmed process.
Set context and concurrency intentionally
Do not allocate the largest context length simply because the model supports it. Choose a working limit based on actual tasks and validate quality at that limit. A short classification service and a repository assistant have different memory needs.
Test the longest permitted prompt plus expected output, because generated tokens also extend the sequence. Include simultaneous requests if the product permits them. Record whether the runtime reserves cache capacity upfront or grows it dynamically.
If memory is tight, possible experiments include a smaller model, lower weight precision, shorter context, fewer active sequences, cache quantization, or offloading. Each can affect quality or latency. Change one variable at a time so the result remains interpretable.
The cache documentation specifically warns that memory-saving strategies can carry performance costs. Benchmark your workload rather than assuming a smaller allocation is automatically faster.
Turn the estimate into a measurement plan
Record the exact model artifact and checksum, runtime version, backend, precision settings, context limit, and concurrency. Measure startup peak, steady-state memory, time to first token, generation speed, and task quality.
Use representative prompts, including long inputs and cases that require reliable structured output. A configuration that fits but fails the task is not a successful deployment. Likewise, an answer-quality win may not justify a response time users cannot tolerate.
The context budgeting guide helps choose the workload whose memory requirements you are estimating. The LLM tracing guide covers observing that workload without turning sensitive prompts into routine logs.
A transparent estimate gives you a sensible starting configuration. Actual measurements tell you whether that configuration belongs on your hardware. Keep the two clearly labeled so a useful approximation does not become an unsupported hardware promise.