Project / 04
Run local LLMs on your Mac. A zero-overhead, 100% native macOS app for Apple Silicon local AI — built on Apple MLX, with a local OpenAI-compatible API server. Since version 1.1 the model can also act on what you hand it, always behind an explicit approval.
Four decisions the whole product rests on.
MLX Swift links in as a package and runs in the same process as the app: no subprocess to supervise, no bridging layer, one memory model instead of two. It also hands over the GPU buffer cache — which is the only reason eviction is possible at all. Shipping someone else’s inference binary would have meant updating it on their schedule and giving that control up.
Dropping the tensors does not return the memory: MLX recycles buffers, and only an explicit clearCache() hands them back. So the engine caps the reserve at 32 MB at launch, counts down an idle timer, then cancels the run, releases the container, clears the cache and resets the peak. Five minutes of silence and a loaded model costs nothing.
An NWListener and a request parser, not a framework — no dependency to audit in an app whose whole claim is that nothing leaves the Mac. Unknown fields are ignored rather than rejected, so a client written against OpenAI keeps working. Structured output refuses to stream: a shape can only be enforced once the whole reply exists, and bytes already sent cannot be taken back.
Tool calls return proposals, not results. A file edit becomes a diff with just enough context around it that a one-line fix in a long file stays a short preview; calendar entries and shortcut runs become permission cards. The model is never asked to confirm first — the card is the confirmation, and asking twice only teaches people to click through. There is no autonomous mode to enable, because there is no switch.
Three things that took longer than they look.
Qwen3 and its kind wrap their chain of thought in <think> tags. The transcript folds that away rather than stripping it — it is still what the model produced, so it stays in the text.
Tool support cannot be read off a model name or a Hub tag. The template on disk is the ground truth, and it is parsed to decide whether this model can be offered tools at all.
Pro-only tools answer on the free tier with an explanation instead of a result, and the system prompt tells the model not to work around the gate — otherwise it tries another tool to get the same effect.
The parts worth explaining.
MLX Swift links into the app and runs in-process on unified memory: no subprocess to supervise, no bridging layer, one memory model. It also exposes the GPU buffer cache, which is what the idle eviction depends on. A bundled inference binary would have meant shipping and updating someone else’s release, and giving up that control.
MLX recycles buffers, so releasing tensors returns nothing to the system. The engine caps the recycled reserve at 32 MB at start-up and, when the idle countdown runs out, cancels any generation in flight, drops the model container, calls clearCache() and resets the peak counter. The next prompt pays for one model load.
From its chat template, never from its name. There is no reliable tag for it on the Hub, and the template is the ground truth: one that never reads the tool list drops it silently — the model is offered nothing, gets asked for a file, and invents one. So the template is read from disk and judged.
Because the surface that matters is small: one endpoint shape, SSE framing, and tolerant parsing. A server framework would have added a dependency and its transitive tree to an app that promises nothing leaves the machine.
By testing the seams, not the output. The test suite covers the parts with one right answer — template capability detection, tool-call parsing, the diff builder, the generation queue, structured-output validation, request handling, purchase gating. The model’s prose is never asserted on; that would only test the sampler.
Measured on device, not extrapolated.
Sustained decode throughput on an Apple M4 with 10 GPU cores and 16 GB of unified memory, running Qwen2.5-0.5B-Instruct in 4-bit, measured end to end through the MLX Swift graph.
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2.5-0.5b",
"messages": [{"role": "user", "content": "Explain unified memory."}],
"stream": true
}'