Running a 2.78T-Parameter LLM on a Single CPU in 8GB RAM
Discover how a 2.78-trillion-parameter Kimi K3 model runs on a single CPU with just 8.24 GB of RAM, using portable C99 and clever memory streaming.
The AI world is obsessed with scale, but what if you could run a 2.78-trillion-parameter model on a machine you already own? That's exactly what the kimi-k3-in-c project achieves: a 2.78T-parameter Kimi K3 model running inference on a single CPU, using only 8.24 GB of RAM, with no GPU, no BLAS, and no framework. The entire engine is a 176 KB C99 binary that streams the model from disk, making it possible to run state-of-the-art AI on hardware that's anything but state-of-the-art.
This isn't a toy or a heavily quantized approximation. The output is byte-identical whether you run it on a laptop with 8 GB of RAM or a workstation with 224 GB. The only difference is speed: 32.69 seconds per token on the low end, 19.21 seconds per token on the high end. The project achieves this through a combination of clever engineering, deep understanding of the model architecture, and a willingness to challenge conventional assumptions about what's necessary for LLM inference.
The Problem: A Model That Doesn't Fit
Kimi K3 is a mixture-of-experts (MoE) model with 2.78 trillion parameters, shipped as a 1.56 TB checkpoint. No consumer machine can hold that in memory. The naive approach—loading everything into RAM—would require 5.56 TB at bfloat16 precision. That's not just impractical; it's impossible for anyone without a datacenter.
The key insight is that MoE models don't need all their parameters active at once. For any given token, only 16 of the 896 experts per layer fire. That's about 104 billion active parameters out of 2.78 trillion—just 3.7%. The rest can stay on disk, as long as they're reachable when needed.
The Four Reductions
The project achieves its memory footprint through four key reductions:
- MXFP4 experts: The routed experts ship in a 4-bit floating-point format, taking only 0.53125 bytes per weight instead of 2 bytes for bfloat16. This alone reduces the expert weights from 5.45 TB to 1.447 TB.
- KDA attention: Kimi Delta Attention uses a fixed-size recurrent state that doesn't grow with context length. This means 69 of the 93 layers have a constant memory footprint regardless of how much text you feed them.
- MLA attention: Multi-head Latent Attention caches a single 576-dimensional latent per position instead of 96 separate key/value heads, reducing the KV cache by 53x.
- Streaming the trunk: The dense layers (the "trunk") are streamed from disk layer by layer, using a pinned prefix and a single ring buffer. This turns the memory requirement from a fixed floor into a dial you can adjust.
How It Works: The Architecture
The engine is written in portable C99, with no external dependencies beyond libm and OpenMP. The codebase is remarkably small: six C files compiled into a single 176 KB binary. This is possible because the project avoids any framework overhead and implements every kernel from scratch.
Reading the Checkpoint
The 1.56 TB checkpoint consists of 96 safetensors files. The engine reads the JSON headers to build an index of all 497,220 tensors, then reads only the bytes it needs on demand. This is done with O_DIRECT reads, bypassing the page cache entirely, which the project found to be faster than buffered reads on their test hardware.
The Expert Cache
Routed experts are loaded into an LRU cache with a configurable size. However, the project discovered a surprising result: the expert cache is almost useless at small sizes. Due to the model's Quantile Balancing training, expert usage is flattened across the pool, so there's no hot subset for the cache to exploit. The cache only starts to help above about 36 GB of arena.
The Trunk Streaming
The dense trunk (108.81 GB) is read layer by layer. The engine pins as many layers as fit in the budget, and streams the rest through a single ring buffer. Because the engine walks layers in a fixed order, a pinned prefix achieves a deterministic hit rate of N/93, which is far better than an LRU cache would achieve on a cyclic scan.
Validation: Proving It's Correct
The project doesn't just claim to work; it proves it through a rigorous validation ladder:
- Weightless tests: A 13-layer oracle model with the same tensor graph as the real one, checked against a PyTorch reference. All three execution paths (teacher forcing, greedy decode, incremental decode) produce identical token IDs.
- Layer conformance: All 93 layers of the full model are verified against a PyTorch reference, with a worst-case error of 0.00x the rounding budget.
- Logit parity: The C engine's logits match the torch reference elementwise over the full 163,840-vocabulary output, with a maximum difference of 7.87e-6.
- Memory ladder: Twelve different memory budgets, from 8 GB to 224 GB, all produce byte-identical token IDs.
Performance: What to Expect
On a single CPU (AMD EPYC 7763, 124 cores), the engine achieves:
- 8 GB RAM: 32.69 s/token
- 32 GB RAM: 31.44 s/token
- 64 GB RAM: 28.60 s/token
- 128 GB RAM: 29.40 s/token (note the noise)
- 224 GB RAM: 19.21 s/token
The performance is heavily I/O-bound. Between 41% and 61% of wall-clock time is spent waiting on disk. This means the storage device matters more than the CPU. A fast NVMe drive is essential for good performance.
Getting Started
To run this yourself, you'll need:
- Linux x86-64 with AVX2 and FMA
- At least 8 GB of RAM
- ~1.7 TB of free disk space
- GCC ≥ 9 or Clang ≥ 10
- Python 3.9+ (for download and packing tools)
Clone the repo, build with make -j, and run make test to verify the engine works before downloading the 1.56 TB checkpoint. Then download the model, pack the trunk, and run:
./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
--tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental
Conclusion
The kimi-k3-in-c project is a masterclass in systems engineering. It demonstrates that with the right approach, even the largest AI models can be made accessible to ordinary hardware. The key takeaways:
- Memory is a dial, not a floor: By streaming the trunk, you can trade speed for memory usage.
- Understand your model: The architecture choices (KDA, MLA, MXFP4) are exploited to their fullest.
- Measure everything: The project's rigorous validation and measurement methodology ensures that every claim is backed by data.
This is not just a technical curiosity; it's a blueprint for making AI more accessible. If you've ever wanted to run a trillion-parameter model on your own machine, this project shows it's possible—and it's open source.