jundot / omlx

oMLX 
 LLM inference, optimized for your Mac Continuous batching and tiered KV caching, managed directly from your menu bar. 
 junkim.dot@gmail.com · https://omlx.ai/me 
 Install ·
 Quickstart ·
 Features ·
 Models ·
 CLI Configuration ·
 Benchmarks ·
 oMLX.ai 
 English ·
 中文 ·
 한국어 ·
 日本語 
---
> *Every LLM server I tried made me choose between convenience and control. I wanted to pin everyday models in memory, auto-swap heavier ones on demand, set context limits - and manage it all from a menu bar.*
>
> *oMLX persists KV cache across a hot in-memory tier and cold SSD tier - even when context changes mid-conversation, all past context stays cached and reusable across requests, making local LLMs practical for real coding work with tools like Claude Code. That's why I built it.*
## Install
### macOS App
Download the `.dmg` from Releases, drag to Applications, done. The app includes in-app auto-update, so future upgrades are just one click. The macOS app also installs a lightweight `~/.omlx/bin/omlx` CLI shim so terminal commands and Apple Shortcuts can control the app-managed server.
### Homebrew
 [code] 
### From Source
 [code] 
Requires macOS 15.0+ (Sequoia), Python 3.11–3.13, and Apple Silicon (M1/M2/M3/M4/M5).
> **Note on native custom kernels:** GLM-5.2, MiniMax M3, and Qwen3.5 need them.
> Without them those families silently fall back to much slower generic paths --
> for GLM-5.2 the fused DSA prefill is roughly 30x faster with the kernels
> (measured 845 vs ~29 tok/s on an M3 Ultra), and the fallback also uses more
> memory (#2137). `make install` builds them and checks that each one loads. That
> needs full Xcode with the Metal toolchain (`xcodebuild -downloadComponent
> MetalToolchain`); Command Line Tools alone do not provide it (`xcrun: error:
> unable to find utility "metal"`). Without Xcode, use the official DMG, which
> ships the kernels precompiled, or `make install-no-kernels`. A plain
> `pip install -e .` also skips them. `make kernels` rebuilds only the kernels
> from scratch. Homebrew builds them with `--with-custom-kernel`, which also needs
> full Xcode. To verify any install:
>
> [code] 
## Quickstart
### macOS App
Launch oMLX from your Applications folder. The Welcome screen guides you through three steps - model directory, server start, and first model download. That's it. To connect OpenClaw, OpenCode, Codex, Hermes Agent, Copilot, or DeepSeek Harness, see Integrations.
### CLI
 [code] 
The server discovers LLMs, VLMs, embedding models, and rerankers from subdirectories automatically. Any OpenAI-compatible client can connect to `http://localhost:8000/v1`. A built-in chat UI is also available at `http://localhost:8000/admin/chat`.
### Homebrew Service
If you installed via Homebrew, you can run oMLX as a managed background service:
 [code] 
The service runs `omlx serve` with zero-config defaults (`~/.omlx/models`, port 8000). `omlx start`, `omlx stop`, and `omlx restart` are the portable lifecycle commands; Homebrew installs