Running a 2.78T-Parameter LLM on One CPU in 8GB RAM: Inside the Kimi K3 C Engine
Discover how a 176KB C99 binary runs a 2.78-trillion-parameter Kimi K3 model on a single CPU with just 8.24GB RAM, using streaming, MXFP4, and clever memory management.
Imagine running a 2.78-trillion-parameter language model on your laptop. Not a quantized, distilled, or otherwise compromised version—the full, original Kimi K3, with 1.56TB of weights, producing byte-identical output to what you'd get from a datacenter GPU cluster. That's exactly what the kimi-k3-in-c project achieves, and it's a masterclass in systems engineering.
This isn't a toy demo. It's a fully functional inference engine written in portable C99, with no BLAS, no deep learning framework, and no GPU support. The entire engine compiles to a 176KB binary. The secret? A combination of model architecture exploitation, aggressive memory streaming, and a deep understanding of where every byte lives.
The Problem: A Model That Doesn't Fit
Kimi K3 has 2.78 trillion parameters, which at bfloat16 precision would require 5.56TB of memory. The released checkpoint is 1.56TB, still far beyond any consumer machine. But Kimi K3 is a Mixture-of-Experts (MoE) model: for each token, only 16 out of 896 experts per layer are active. That's about 104 billion active parameters—just 3.7% of the total.
The key insight: the other 96.3% of parameters don't need to be in RAM. They just need to be reachable. By streaming the inactive experts from disk on demand, the memory requirement drops dramatically.
The Four Reductions
The project achieves a 675x memory reduction from the theoretical bfloat16 baseline through four key steps:
MXFP4 Quantization: The routed experts are already stored in MXFP4 format—a microscaling 4-bit float format. Each weight is a 4-bit nibble, with a shared 8-bit exponent per 32 weights. This cuts expert storage from 5.45TB to 1.447TB.
Sparsity: Only 16 of 896 experts fire per token, so the 1.447TB of experts never need to be fully resident. They're streamed from disk as needed.
KDA Attention: 69 of 93 layers use Kimi Delta Attention, a recurrent attention mechanism with a fixed-size state that doesn't grow with context length. This eliminates the KV cache explosion for most layers.
MLA Compression: The remaining 24 layers use Multi-head Latent Attention, which caches a single 576-dimensional latent per position instead of per-head key/value pairs. This cuts KV cache size by 53x.
After these reductions, the always-resident set is just 113.49GB (the dense trunk, embeddings, and output head). But that's still too much for a laptop. The final reduction: stream the trunk itself.
Streaming the Trunk: A Dial, Not a Floor
The trunk (93 dense layers) is 108.81GB. Every layer is used on every token, so there's no sparsity to exploit. The solution is to pin as many layers in RAM as your budget allows, and stream the rest from disk through a single ring buffer.
The engine walks layers 0-92 in order on every token. This is a cyclic scan, which is the worst-case scenario for LRU caching. So instead of a cache, it uses a pinned prefix: the first N layers are permanently resident, and the rest cycle through a ring slot. This gives a deterministic hit rate of N/93.
For example, with a --preset laptop (3GB trunk budget), only 10 layers are pinned, and the rest stream from disk. The result: 8.24GB peak RSS, but 26.5 seconds per token. With --preset server (110GB trunk budget), 90 layers are pinned, and the speed jumps to 5.6 seconds per token.
The Expert Cache: A Lesson in Measurement
The engine also includes an LRU cache for the routed experts. But here's the surprising part: the cache is almost useless at small sizes. The project's measurements show that increasing the cache from 28 slots to 1,344 slots (a 48x increase) changes the bytes read per token by exactly zero.
Why? Because Kimi K3 uses a technique called Quantile Balancing, which flattens expert usage across the pool. With no hot subset of experts, an LRU cache retains nothing useful. The cache only starts to help when it's large enough to hold a significant fraction of the working set (around 36GB of arena).
This leads to a counterintuitive optimization: give memory to the trunk first, not the expert cache. At a fixed 128GB budget, allocating 110GB to the trunk and 13GB to the cache is 1.69x faster than the reverse split, even though the latter has a higher cache hit rate.
Bit-Exact Reproducibility
One of the most impressive aspects is the commitment to bit-exact reproducibility. The engine ensures that the scalar, OpenMP, and AVX2 code paths all produce identical results. This is achieved by:
- Using
-ffp-contract=offto prevent FMA contraction, which would change rounding. - Accumulating in double precision with a fixed summation order.
- Widening bf16 to fp32 via a simple shift (lossless).
This means the output is byte-identical whether you run on an 8GB laptop or a 224GB workstation. The only difference is speed.
Validation: Proving It Works
The project includes a rigorous validation suite:
- Weightless tests: A 13-layer oracle model with the same tensor graph as the full model, checked against a PyTorch reference. All gates pass exactly.
- Full checkpoint conformance: All 93 layers verified against PyTorch, with a worst-case error of 0.00x the allowed tolerance.
- Logit parity: The C engine's logits match the PyTorch reference elementwise, with a max difference of 7.87e-6.
- Memory ladder: 12 different memory budgets from 8GB to 224GB, all producing identical token IDs.
The Bottom Line
This project is a testament to what's possible with careful systems engineering. It proves that a trillion-parameter model doesn't require a datacenter—just a clever understanding of where bytes live and how to move them efficiently.
Whether you're interested in MoE architectures, memory-efficient inference, or just want to see some impressive C code, kimi-k3-in-c is worth a deep dive. The README alone is a treasure trove of technical detail, and the code is clean and well-documented.
If you're inspired to try it yourself, you'll need about 1.7TB of free disk space and a lot of patience for the download. But the payoff is running one of the largest open models on hardware you probably already own.