Colibri Weight Streaming for GLM-5.2: Running on Slow Computers

Running a large language model like GLM-5.2 on a slow computer is now possible with weight streaming. The open-source project Colibri achieves this by streaming weights from disk instead of loading the entire model into RAM or VRAM. The Hacker News community reacted with both fascination and skepticism, producing over 600 points and 140 comments. The central question: at 0.05 tokens per second, is it usable?

Colibri Weight Streaming: The Idea

Colibri uses memory-mapped files (mmap) to load only the weights required for the current computation. This allows models like GLM-5.2—a bilingual Chinese-English model—to run on hardware with very little memory, such as a Raspberry Pi or an old laptop with a spinning hard drive. The GitHub repository reports token rates as low as 0.05 to 0.1 tok/s on low-end machines.

HN Reaction: Skepticism and Fascination

The HN discussion reveals a split between those impressed by the engineering and those questioning practicality. One commenter noted: "I have seen locally hosted LLM that are as slow as 1 tok/s still be very useful if you give it a project to do something overnight." Others pushed back on sub-0.1 tok/s rates, calling them unusable for interactive use. Several participants shared similar projects, including thinfer, a weight-streaming engine for image/video generation with LRU caching.

Practicality of 0.05 Tok/s for Local LLM Deployment

At 0.05 tok/s, a chatbot is unusable—one word every 20 seconds. For interactive applications, this approach fails. However, for batch processing—such as summarizing a long document overnight—it can be viable. The bottleneck is disk I/O, but as one commenter observed, "moving mmap'd bytes in and out of VRAM is way cheap compared to compute." The technique exploits the fact that loading weights from an NVMe SSD is surprisingly fast relative to matrix multiplications.

Sophisticated implementations can use madvise with MADV_WILLNEED to prefetch upcoming pages. The operating system's virtual memory system already handles on-demand paging when you mmap the entire model file, making Colibri's approach a practical foundation for more advanced memory management.

3 Strategies for Running Large Models on Memory-Constrained Hardware

If you're building tools that need to run LLMs on edge devices or low-power hardware, consider these strategies:

1. Weight streaming: For very small memory budgets, streaming from disk works but only for batch workloads. Use mmap to let the kernel handle page-level loading.
2. Quantization: Always quantize the model to the lowest bitwidth you can tolerate (e.g., 4-bit). Colibri likely works with quantized models, but verify your inference engine supports them.
3. Prefetching and double-buffering: Pipeline weight loading and computation to hide I/O latency. The following pseudocode shows the double-buffering pattern:

import threading
from queue import Queue

def load_weights(file, offset, size):
    with open(file, 'rb') as f:
        f.seek(offset)
        return f.read(size)

def inference(model_file, input_embeds):
    weight_queue = Queue(maxsize=2)
    weight_queue.put(load_weights(model_file, 0, first_layer_size))
    for layer_idx in range(num_layers):
        if layer_idx < num_layers - 1:
            threading.Thread(target=lambda: weight_queue.put(
                load_weights(model_file, layer_offset[layer_idx+1], layer_size)
            )).start()
        current_weights = weight_queue.get()
        input_embeds = linear(input_embeds, current_weights)
    return input_embeds

Combined with mmap and memory pressure management, this pattern can make sub-1 tok/s operation feasible for non-interactive tasks.

Verdict: When to Use Weight Streaming

Colibri is a technical demo that proves a point: even huge models can run with minimal RAM if you accept snail-paced speeds. For most applications, llama.cpp with GGUF quantized models already strikes a good balance on edge hardware with 4-8GB RAM. Weight streaming shines only when memory is extremely constrained (e.g., <1GB) and latency is not critical. If you need interactive response times, focus on quantization, pruning, and distillation instead.

The technique itself—specifically the use of mmap to avoid duplicating memory—is a valuable insight for builders. But the future of truly memory-constrained inference isn't here yet. Colibri shows the path, but most developers can safely wait for more mature implementations.