The Problem

Every mainstream AI assistant sends your conversation to a remote server. Someone else stores it, processes it, and may use it to train future models. For a lot of use cases that's fine. For working with sensitive documents, personal notes, or anything you'd rather keep to yourself, it's a fundamental problem with no opt-out.

I wanted a local AI that was genuinely private by design, not by policy. Something that worked without an internet connection, ran on hardware I already owned, and that I could pull apart and modify.

How It Actually Works

Sage runs llama.cpp directly as a server process, exposing an OpenAI-compatible API on the local network. FastAPI sits in front of it handling the document pipeline, scheduled tasks, and the REST endpoints the UI talks to. Document retrieval uses SQLite FTS5 for BM25 keyword search with a vector similarity fallback using numpy memory-mapped float16 embeddings. The entire frontend is a single HTML file that talks to the FastAPI endpoints directly. Nothing is abstracted away.

Build Challenges

RAM was the constant constraint. Running a language model, an embedding model, a FastAPI server, a vector index, and a document pipeline on 4GB leaves very little room for error. Every component had to be as lean as possible. Open WebUI was evaluated and rejected because its RAM footprint consumed too much of what the model needed. The frontend ended up as a single HTML file partly because every megabyte of running process costs something the model could have used.

SD card reliability was a genuine concern. llama.cpp logging continuously to an SD card accelerates wear significantly. Disabling logs and mounting key paths with noatime helped. Moving to a quality A2-rated card made the biggest practical difference.

Model selection at the 1B scale is a real quality ceiling. The model will confidently say wrong things. RAG helps when the answer is in a document, but it doesn't make the model smarter, it just gives it better material to work with. Getting the retrieval pipeline fast enough to not make the overall experience feel broken on constrained hardware was more work than expected.