Guides · 2026-10-13

oMLX setup and tuning on Apple Silicon

Install oMLX, serve a Qwen3.6 model over OpenAI and Anthropic APIs, and tune the KV cache for long agent sessions.

Researched and drafted with AI assistance from the cited sources. Benchmarks are single-machine third-party results: treat them as indicative, not guaranteed.

Why oMLX

Coding agents resend long system prompts and keep shifting their context. On most servers that invalidates the KV cache and forces a full prefill, which is slow on a Mac. oMLX exists to fix this. It is an Apple Silicon-only MLX server with a tiered RAM + SSD KV cache whose blocks persist across restarts, plus continuous batching and multi-model serving (LLMs, VLM/OCR, embeddings, rerankers). The project says time-to-first-token on long agent contexts drops from 30–90 s to under 5 s. That is the project’s own claim, so measure it on your machine.

Facts at a glance (Oct 2026): Apache-2.0, ~22.7k GitHub stars, created Feb 2026, latest noted release v0.6.4 (Aug 29 2026). It is fast-moving, with ~609 open issues, so expect some config friction.

Requirements

• Apple Silicon Mac
• macOS 15 or newer

Install

Either download the DMG (it includes a menu-bar app) from omlx.ai or the GitHub releases page, or use Homebrew:

brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx

Serve and connect

oMLX listens on port 8000 and exposes:

• OpenAI-compatible: http://localhost:8000/v1
• Anthropic-compatible: http://localhost:8000/v1/messages
• Admin UI: http://localhost:8000/admin

Load a model from the admin UI. For coding agents, Qwen3.6-35B-A3B (~20 GB at 4-bit) is the common default if you have 32 GB+ of RAM. See choosing a Qwen3.6 quant.

Quick smoke test (use the model ID shown in the admin UI):

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Say hi"}]}'

Tuning checklist

1. Cap context to what you use. A larger context means more KV memory. Agents rarely need the full 262k native window.
2. Keep the SSD cache tier on. It’s the whole point for agents: repeat prefixes hit cache instead of recomputing.
3. Try TurboQuant KV (experimental). The 3.5-bit KV-cache setting reportedly cuts KV memory by roughly 4x, which leaves room for longer contexts. It’s experimental, so compare output quality.
4. Prefer MLX-format quants over GGUF on Apple Silicon.
5. Watch memory pressure and swap in Activity Monitor. If swap grows, shrink the context or the model. Swapping causes a sharp performance cliff.

Setting names change between releases, so check the current README for exact options.

Skip the trial and error: sourced config packs →