LocalGPT: Run a Powerful AI Assistant Locally in Rust with Minimal Deps
Introduction
LocalGPT is the first‑class open‑source AI assistant that lives entirely on your device. Written in Rust, it is bundled in a single binary, requires no JavaScript runtime or Docker image, and stores all memory and configuration in XDG‑compliant directories. The project supports Anthropic Claude, OpenAI, xAI (Grok), Ollama, GLM, and even local servers that expose an OpenAI‑compatible API. With optional Telegram integration, a web UI, and a powerful CLI, LocalGPT aims to be the Swiss‑army knife for anyone who wants a privacy‑preserving, low‑latency chatbot.
Why LocalGPT Matters
- Zero‑Cloud Dependency – All conversations, embeddings, and tools run locally.
- Persistent Markdown Knowledge Store – Store anything, from meeting notes to code snippets, and search them by keyword or semantic similarity.
- Autonomous Heartbeat – The daemon mode lets you schedule background tasks that never block your day.
- Built‑in Sandbox – Kernel‑level sandboxing (Landlock on Linux, Seatbelt on macOS, seccomp filters everywhere) protects your system while the assistant runs user‑supplied commands.
- Multiple Interfaces – Chat via the terminal, a responsive web UI, a desktop app, or even a Telegram bot.
Quick Start (Linux/macOS)
- Install the CLI –
cargo install localgpt - Initialize the configuration –
localgpt config init - Start chatting –
localgpt chat
You’ll see a prompt that looks like:
$ localgpt chat
> What’s the best way to set up a Rust project?
LocalGPT: ...
The installer pulls the latest localgpt binary from crates.io, creates a clean workspace under ~/.local/share/localgpt, and writes a minimal config.toml.
Deep Dive: Configuration and Memory
LocalGPT’s configuration lives in ~/.local/share/localgpt/config.toml:
[agent]
default_model = "claude-cli/opus"
[providers.anthropic]
api_key = "${ANTHROPIC_API_KEY}"
default_model to openai/<your‑model> and set base_url. For example, to use LM Studio:
[providers.openai]
api_key = "not-needed"
base_url = "http://127.0.0.1:1234/v1"
Workspace Memory
A Markdown‑based knowledge bank lives under <workspace>/knowledge/. Each file is automatically indexed with SQLite FTS5 for fast keyword look‑ups and sqlite-vec for semantic similarity matching. The top‑level files MEMORY.md, HEARTBEAT.md, and SOUL.md provide long‑term memory, scheduled tasks, and personality guidance respectively.
Advanced Features
Autonomy & Heartbeat
Running localgpt daemon start launches a background server featuring:
- Heartbeat – Periodic “ticks” that can trigger scheduled actions.
- HTTP API – Exposes /api/chat, /api/memory/search, etc. for integration.
- Web UI – Accessible at http://localhost:31327 or via the bundled desktop GUI.
Security Sandbox
All shell commands run through a kernel‑enforced sandbox. On Linux you get Landlock + seccomp; on macOS it’s Seatbelt. The sandbox also imposes rlimits: 120s CPU, 1 MB stdout, 64 processes, and a 50 MB file quota. The default policy blocks writes to sensitive locations like ~/.ssh and ~/.aws.
Prompt Injection Mitigations
LocalGPT strips token markers, uses regex checks for injection phrases, and wraps external content in XML tags. You can further harden the assistant by adding a LocalGPT.md policy file signed with your device key.
Community & Extensibility
- CLI Plugins – Contribute new tools by adding Rust crates under
crates/. - Telegram Bot – Pair the daemon with a Telegram bot for mobile access.
- World Generation Sub‑Project –
localgpt-genlets you instruct the assistant to create 3‑D scenes using the Bevy engine.
The project welcomes pull requests, especially for new LLM integrations, memory formats, or sandbox improvements.
Use Cases
| Scenario | How LocalGPT Helps |
|---|---|
| Personal Knowledge Base | Store meeting notes, research docs, and code snippets in Markdown and fetch them instantly. |
| Low‑Latency Customer Support Bot | Deploy on the same server handling internal tickets without external API calls. |
| Privacy‑Sensitive Workflows | All data, including embeddings, never leaves the device, satisfying strict compliance requirements. |
Getting Involved
- Start from the Repo –
git clone https://github.com/localgpt-app/localgpt - Build from Source –
cargo build --release --workspace - Join the Discord – Share your creations and ask for help.
LocalGPT is MIT‑licensed with an Apache‑2.0 license for contributions, so you’re free to use, modify, and ship your own iterations.
Conclusion
LocalGPT proves that powerful, privacy‑friendly AI assistants can be built with a minimal, all‑Rust stack. Whether you’re a developer, a researcher, or a privacy‑conscious user, installing a local GPT gives you instant, low‑latency access to advanced LLMs without sacrificing control. Give it a try today, and join a growing community that’s redefining how we interact with AI on the edge.