Running a 28.9M Parameter LLM on an $8 ESP32-S3 Microcontroller
Learn how Per-Layer Embeddings from Google's Gemma models enable a 28.9M parameter language model to run entirely on an $8 ESP32-S3 chip, generating text at 9.5 tokens per second.
The Problem: LLMs Don't Fit on Microcontrollers
Microcontrollers have very little fast memory. The ESP32-S3, a popular chip costing about $8, gives you only 512KB of SRAM. Normally, the entire model must be loaded into this fast memory to run inference, which severely limits model size. The previous state-of-the-art on a similar chip was a mere 260,000 parameters.
The Breakthrough: Per-Layer Embeddings
The key insight comes from Google's Gemma 3n and Gemma 4 models: Per-Layer Embeddings (PLE). Most of a language model's parameters live in the embedding table — a large lookup table that maps tokens to vectors. The model reads from this table but doesn't compute on it. So you can leave that 25-million-row table in slow flash memory and pull only the few rows (about 450 bytes) each token needs. The small part that does the actual computation stays in fast SRAM.
This means the large model costs almost nothing to run — you never load most of it. It just sits in flash and gets sampled a little at a time.
The Numbers
| Metric | Value |
|---|---|
| Parameters | 28.9M stored (25M in flash lookup table) |
| Chip | ESP32-S3 (~$8) with 512KB SRAM, 8MB PSRAM, 16MB flash |
| Speed | ~9.5 tok/s end-to-end (9.7 tok/s pure compute) |
| Connectivity | None — everything runs on-device |
| Model size | 14.9MB at 4-bit |
Memory Layout
The trick works because of a careful memory hierarchy:
- SRAM (fast, tiny): The "thinking" core — used on every token
- PSRAM (medium): The output head and working memory
- FLASH (huge, slow): The 25M-parameter table — about 6 rows read per token (~450 bytes)
What It Does (and Doesn't)
The model was trained on TinyStories (Ronen Eldan and Yuanzhi Li, Microsoft Research), so it writes short, simple stories and mostly keeps them coherent. It will not answer questions, follow instructions, write code, or know facts. That limit comes from the small reasoning part of the model — the memory trick doesn't change what a 28.9M parameter model can say.
Running It Yourself
The full firmware, wiring diagrams, and flashing steps are in firmware/esp32_llm/README.md. Training, ablation, and quantization code is in src/ and experiments/. The complete method, ablations, and on-chip measurements are documented in RESULTS.md.
Credit
- TinyStories: The dataset — short synthetic stories simple enough for small models to learn coherent writing (arXiv:2305.07759)
- Per-Layer Embeddings: Google's design from Gemma models, enabling large models on small chips
- llama2.c: Andrej Karpathy's work that inspired the idea that you can train a tiny language model and run it in plain C
The Messy History
The repo intentionally preserves the messy history, including a bug in parameter accounting that inflated early numbers, and the corrected result. The commit history and RESULTS.md show where the numbers moved and why.
This project demonstrates that with the right architecture, even a $8 microcontroller can run a surprisingly capable language model — opening up possibilities for truly edge-based AI applications.