Run 2.78T Kimi K3 on a 64GB MacBook: WASTE's NVMe Streaming Engine
Discover WASTE, a dependency-free C inference engine that streams expert weights from NVMe to run the full 2.78T-parameter Kimi K3 on consumer hardware at 0.6 tok/s.
Imagine running a 2.78-trillion-parameter model on a laptop. Not a distilled version, not a quantized approximation, but the full, original Kimi K3 — the same weights that power one of the largest open models in existence. That's exactly what WASTE (Weight-Aware Streaming Tensor Engine) achieves, and it does so with a clever combination of mixture-of-experts (MoE) architecture, aggressive quantization, and a deep understanding of how modern NVMe storage can act as a slow but vast extension of RAM.
WASTE is an embeddable inference engine written in C with zero third-party runtime dependencies. It keeps the model's shared trunk in memory, streams only the activated expert weights directly from disk, and uses the remaining RAM as a bounded cache. The result? The complete 2.78T-parameter Kimi K3 runs on a 64 GB MacBook Pro at about 0.6 tokens per second. That's slow by cloud standards, but it's a monumental step for local AI — and the project's ultimate goal is to have Kimi K3 improve itself, running entirely on your desk.
Why This Matters: The Token Waste Problem
The project's name isn't just a clever acronym. Every token you generate via a cloud API is paid for twice: once in your invoice, and once in the electricity of a datacenter running a model that could — barely, awkwardly, but genuinely — fit on hardware you already own. WASTE aims to end that waste by proving that frontier-scale models can run locally, even if slowly. It's a philosophical stance as much as a technical one.
How WASTE Works: Streaming Experts from NVMe
Kimi K3 is a mixture-of-experts model with 2.78 trillion parameters, but only about 4% of those are active for any given token. WASTE exploits this sparsity in a radical way:
- Resident Trunk: The shared (non-expert) layers stay in RAM. For K3, that's about 27.28 GB.
- Streamed Experts: The container format is arranged so that each expert requires exactly one aligned read from disk. When the router selects an expert, WASTE reads it directly from NVMe.
- Bounded Expert Cache: Unused RAM becomes a cache for recently used experts, avoiding repeated disk reads.
- Lookahead Router: A predictive router anticipates which experts the next layer will need and starts reading them early. The real router still makes the final decision, so this only changes timing, not results.
- Aggressive Quantization: Experts use 3-bit residual vector quantization, while more sensitive shared weights stay at 4 or 8 bits.
This design reduces the memory footprint dramatically. The full K3 container is 982 GB, but WASTE only needs 29.06 GB of RAM to open it. The rest of your memory (up to a configurable budget) is used for the expert cache.
Performance Numbers: What to Expect
On a 64 GB MacBook Pro with an M5 Pro and internal SSD, here's what WASTE delivers:
| Model | Container Size | Minimum RAM | Decode Speed |
|---|---|---|---|
| Kimi K3 2.78T | 982 GB | 29.06 GB | 0.45–0.62 tok/s |
| Kimi-Linear 48B | 19 GB | 1.28 GB | 10.65 tok/s |
For K3, 64 GB is the practical minimum. A 32 GB machine can technically open the model but will page heavily, making it unusable. The default memory budget on the test machine is 46.25 GB, including a 17.56 GB expert cache.
The Cache Size Trade-off
WASTE's performance is highly sensitive to the expert cache size. The team measured four configurations in a single process:
| Expert Cache | Hit Rate | Decode Speed |
|---|---|---|
| 3.32 GB | 29.1% | 0.56–0.58 tok/s |
| 17.32 GB | 36.2% | 0.63 tok/s |
| 23.32 GB | 38.4% | 0.07–0.09 tok/s |
| 29.32 GB | 41.3% | 0.07–0.08 tok/s |
The last two rows are a critical lesson: giving the process more memory doesn't always make it faster. When the cache exceeds what the OS can comfortably fit in RAM, cache hits become page faults, and throughput collapses eightfold. The engine is within its budget, but the machine isn't.
Storage Is the Real Bottleneck
A cold K3 token reads about 17 GB of experts. The internal SSD sustains 12.78 GB/s, but a tested USB enclosure managed only 0.94 GB/s — a 13x difference. Always put the container on internal NVMe storage.
Getting Started: From Zero to Running K3
Build the Engine
git clone https://github.com/sqliteai/waste
cd waste
make
make check
make builds the waste CLI and libwaste.a. make check runs a model-free test suite that creates a small synthetic model, so no weights are downloaded.
Get the Converted Container (Fastest Route)
The easiest way to get K3 is to download the pre-converted container via BitTorrent. This skips the 1.42 TB source download and the 4.7-hour conversion process. The torrent's piece hashes verify the container as it arrives.
aria2c --dir ~/models --seed-time=60 \
'magnet:?xt=urn:btih:abe7123a60b2b1171c1c4dcaa381b93c46806afe&dn=k3.waste&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337%2Fannounce'
Point --dir at internal NVMe storage — a container on an external disk will be too slow to use.
Convert from Published Weights (If You Prefer)
If you'd rather not trust a third-party copy, or you already have the original weights, you can convert them yourself. The process is resumable:
# Check required download space
tools/fetch_weights.sh --dest /Volumes/staging/k3 --dry-run
# Download the original weights
tools/fetch_weights.sh --dest /Volumes/staging/k3
# Convert them (output on internal SSD)
uv run --with torch --with safetensors python tools/convert.py \
--src /Volumes/staging/k3 \
--out ~/models/k3.waste \
--jobs 3
Conversion takes about 4.7 hours with three workers on the test machine. You'll need 1.42 TB of temporary staging storage, which can be external and freed afterward.
Run It
./waste plan ~/models/k3.waste
./waste run ~/models/k3.waste "The capital of France is" -n 32
./waste chat ~/models/k3.waste
Don't set --budget unless you have a reason. By default, WASTE chooses a safe memory budget and refuses to start below the model's floor. Inside a container, it sizes against the cgroup limit rather than the host's RAM.
Multimodal and Serving
Kimi K3 is multimodal, and WASTE supports images:
./waste run ~/models/k3.waste "Describe this image" --image photo.jpg
./waste run ~/models/k3.waste "Compare these images" --image before.png --image after.png
In interactive mode, /image FILE attaches an image to the next message. An 896×896 image uses 256 prompt positions, and each position costs about 2.8 seconds of language model time.
WASTE also includes an optional OpenAI-compatible server:
make libwaste.dylib # use libwaste.so on Linux
python3 -m serve ~/models/k3.waste --port 8000
curl localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"k3","messages":[{"role":"user","content":"Why is the sky blue?"}]}'
It supports streaming, tools, structured output, thinking controls, and images.
The Library and Validation
WASTE is also an embeddable C library. The CLI and server both use the public API in src/waste.h. The inference path depends only on libc and pthreads — no BLAS, Python, CUDA, or other external dependencies.
Validation is rigorous. All layers are checked against a PyTorch reference; final logits agree within 3.6e-06, and the vision tower agrees with its oracle within 2.3e-06. The model-free suite builds a synthetic container, and real-model checks verify conversion round trips and server prompt rendering.
Project Status and Philosophy
The format and API are not frozen. K3 is the main target and best-tested model. The CPU path is currently the fastest measured implementation, but CUDA, Metal, and other hardware-specific optimizations remain to be explored. The project keeps a docs/LEARNED.md file for failed ideas and negative results — a rare and valuable practice.
Measurements are treated as experimental results, not marketing numbers. Each is tied to the hardware, container, configuration, and commit on which it was obtained. Unstable measurements are reported as ranges, and results later found to be wrong remain recorded as such.
Final Thoughts
WASTE is a bold experiment that challenges the assumption that frontier models require datacenter-scale infrastructure. By streaming weights from NVMe and using MoE sparsity to its advantage, it brings a 2.78T-parameter model to a laptop. It's slow, but it works — and it's open source under Apache 2.0.
If you're a developer interested in pushing the boundaries of local AI, WASTE is worth a look. Start with Kimi-Linear (19 GB container, 10.7 tok/s) to get a feel for the engine, then consider the full K3 experience. The project welcomes contributors, especially for new hardware backends and performance experiments — even negative results are valued.
Check out the GitHub repository for the full documentation, including the container format, backend comparisons, and research directions.