Run Kimi K3 Locally on CPU: K3Flight's 55GB Runtime with a 929GB Model
K3Flight runs the 2.8T-parameter Kimi K3 model on CPU with only ~55GB RAM by streaming weights from storage via cPilot Runtime. A single-file Linux inference server.
Kimi K3 is a massive 2.8-trillion-parameter Mixture-of-Experts model. Its full Q2_K checkpoint weighs in at 929GB. Running that locally seems impossible—until you realize the model doesn't need to fit in memory. K3Flight, a new open-source project, proves it: a single-file Linux inference server that runs Kimi K3 on CPU with a measured runtime RAM footprint of just ~55GB.
The secret isn't compression or distillation. It's a systems engineering approach that treats storage, memory, and CPU as one coordinated execution path. Here's how it works and how you can try it yourself.
The Problem: 929GB of Weights vs. 64GB of RAM
Most local LLM inference assumes the entire model must be resident in RAM. For a 929GB checkpoint, that means a machine with at least 1TB of memory—far beyond what most developers have. But Kimi K3 is a Mixture-of-Experts (MoE) model. According to its public specs, each token activates only 16 of 896 experts. The total parameter count describes the model's capacity, not the working set needed for any single inference step.
K3Flight exploits this sparsity. Instead of loading everything, the cPilot Runtime stages only the weights and state needed for the current execution path. The full checkpoint stays on storage, and data is streamed to CPU as needed.
The Numbers: What the Reference Run Shows
The maintainers ran the full Kimi-K3-GGUF Q2_K checkpoint on a Linux x86-64 machine with these results:
- Total parameters: 2.8T
- Runtime memory: ~55GB
- Backend: CPU-only
- Prefill: ~1 token/s
- Decode: ~0.8 token/s
These are preliminary numbers from a single reference run, not a guarantee. The exact CPU and SSD model aren't disclosed. Storage bandwidth, CPU capability, context length, and runtime version can significantly affect performance. But the point isn't speed—it's that a complete 2.8T model can execute locally without keeping 929GB resident.
How K3Flight Works
K3Flight doesn't shrink the model. The 929GB checkpoint remains on disk. Instead, cPilot Runtime manages a much smaller live working set during inference. It does this by:
- Treating storage, host memory, and CPU as one coordinated execution path
- Staging weights and runtime state as the model needs them
- Orchestrating data movement and compute to keep the active path moving
- Keeping the complete Q2_K checkpoint available without full residency
This turns a model-size problem into a systems problem. The result is a server that can run on a machine with 64GB of RAM and a fast NVMe SSD.
Getting Started: Preview Quick Start
K3Flight is currently in v0.1.0-preview. The first Linux binary is being packaged, and the maintainers promise a reproducible release soon. Here's how to get ready.
1. Download the Binary
Once the release is live, grab the Linux x86-64 archive from GitHub Releases:
curl -L -O https://github.com/onetoken-oss/K3Flight/releases/download/v0.1.0-preview/cpilot-server-v0.1.0-preview-linux-x86_64.tar.gz
tar -xzf cpilot-server-v0.1.0-preview-linux-x86_64.tar.gz
sha256sum -c cpilot-server-v0.1.0-preview-linux-x86_64.sha256
chmod +x cpilot-server-v0.1.0-preview-linux-x86_64
Make sure to download the .tar.gz release asset, not the source code archive.
2. Download the Model
You'll need the Kimi-K3-GGUF Q2_K checkpoint from Hugging Face. It's 929GB, so plan your storage accordingly. Follow the instructions in MODEL.md for the exact repository and revision.
3. Start the Server
Set your model directory and launch the server:
export MODEL_DIR="$HOME/models/Kimi-K3-Q2_K"
./cpilot-server-v0.1.0-preview-linux-x86_64 \
--model "$MODEL_DIR/Kimi-K3-Q2_K-00001-of-00094.gguf" \
--host 127.0.0.1 \
--port 8080
Keep the loopback binding unless you understand the security implications of exposing an inference server to a network.
4. Send a Request
Open http://127.0.0.1:8080 in your browser for the web UI, or use the OpenAI-compatible API:
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "kimi-k3-q2_k",
"stream": true,
"messages": [
{"role": "user", "content": "Hello!"}
]
}'
The server logs will report prefill and decode throughput.
Runtime Options
Run ./cpilot-server-v0.1.0-preview-linux-x86_64 --help to see all options. Key ones include:
-m, --model: Path to the GGUF shard (required)--host: Default127.0.0.1--port: Default8080-t: CPU threads (default 12)-c: Context size (default 512)-b: Logical batch size (default 64)-ub: Physical batch size (default 64)--cpilot-memory: Memory tuning level (0-?)
Requirements
The preview targets a narrow, verifiable configuration:
- Linux on x86-64
- 64GB+ system memory recommended
- At least 1TB free local storage
- Local NVMe SSD strongly recommended
- No GPU required
macOS support is planned next.
FAQ: Common Questions
Did you compress 929GB into 55GB? No. The model files remain ~929GB on storage. ~55GB is the observed runtime memory footprint.
Is this a smaller or distilled model? No. It's the complete Kimi-K3-GGUF Q2_K checkpoint. Q2_K is a quantized representation, so it's not numerically identical to the original, but it's the full model.
Why can a 2.8T model run this way? Kimi K3 activates only 16 of 896 experts per token. cPilot manages the live working set instead of requiring full residency.
Is ~55GB the minimum RAM? No. It's a preliminary measured result. 64GB is recommended for the preview.
Why is generation slow? The CPU path trades residency for data movement. It's designed to prove local execution, not to compete with datacenter latency.
Known Limitations
- Linux x86-64 only; macOS not yet released
- CPU-only, optimized for feasibility rather than cloud latency
- Performance varies with storage and CPU
- No production SLA; don't expose to untrusted networks
Help Build the Hardware Map
The maintainers are looking for honest results, including slow runs and failures. If you try it, share your run with details like CPU, memory, SSD model, and throughput. Controlled negative results are welcome—they help define the real hardware boundary.
Final Thoughts
K3Flight is a remarkable proof of concept. It challenges the assumption that huge models require huge memory. By leveraging MoE sparsity and smart systems engineering, it makes a 2.8T model runnable on a single machine. It's not fast, but it's local, and that opens up possibilities for edge AI, privacy, and experimentation.
If you're interested in local LLM inference, this is worth watching. Star the repo, try the preview when it drops, and contribute your measurements. The future of local AI might not require a datacenter after all.