mostlygeek / llama-swap

# llama-swap
Run multiple generative AI models on your machine and hot-swap between them on demand. llama-swap works with any OpenAI and Anthropic API compatible server and is used by thousands of people to power their local AI workflows.
Built in Go for performance and simplicity, llama-swap has zero dependencies and is incredibly easy to set up. Get started in minutes - just one binary and one configuration file.
## Features:
- ✅ Easy to deploy and configure: one binary, one configuration file. no external dependencies
- ✅ On-demand model switching for many local AI servers (llama.cpp + forks, vllm, stable-diffusion.cpp, audio.cpp, ComfyUI, etc.)
 - future proof, upgrade your inference servers at any time.
- ✅ OpenAI API supported endpoints:
 - `v1/completions`
 - `v1/chat/completions`
 - `v1/responses`
 - `v1/embeddings`
 - `v1/models` - list available models
 - `v1/audio/speech` (#36)
 - `v1/audio/transcriptions` (docs)
 - `v1/audio/voices`
 - `v1/images/generations`
 - `v1/images/edits`
- ✅ Anthropic API supported endpoints:
 - `v1/messages`
 - `v1/messages/count_tokens`
- ✅ llama-server (llama.cpp) supported endpoints
 - `v1/rerank`, `v1/reranking`, `/rerank`
 - `/infill` - for code infilling
 - `/completion` - for completion endpoint
 - `/models` - list available models. same behavior as `v1/models`
 - `/props` - requires `?model={model_id}` query parameter to be provided. The autoload parameter is not supported and will be ignored.
- ✅ SDAPI via stable-diffusion.cpp's server
 - `/sdapi/v1/txt2img`
 - `/sdapi/v1/img2img`
 - `/sdapi/v1/loras` - requires `model` in request body to fetch the correct loras
- ✅ audio.cpp supported extra endpoints
 - `/audioapi/v1/tasks/run`
- ✅ `/audiocpp/` - audio.cpp UI endpoint, served by the `audiocpp_auto` model (#1221)
- ✅ `/comfyui/` - ComfyUI custom endpoint (#1001) for more reliable swapping
- ✅ llama-swap API
 - `/ui` - web UI
 - `/upstream/:model_id` - direct access to upstream server (demo) 
 - `/running` - list currently running models (#61)
 - `POST /api/models/unload` - manually unload all running models (#58)
 - `POST /api/models/unload/:model_id` - unload a specific model
 - `GET /api/profiles` - list configured profiles and the active selection
 - `PUT /api/profiles/active` - activate a profile or select none
 - `/logs` - remote log monitoring
 - `GET /logs` returns buffered plain text logs.
 - If `Accept: text/html` is sent, `/logs` redirects to `/ui/`.
 - `GET /logs/stream` keeps the connection open for live log streaming.
 - Stream endpoints send buffered history first by default; add `?no-history` to stream only new lines.
 - `GET /logs/stream/proxy` streams proxy logs only.
 - `GET /logs/stream/upstream` streams upstream process logs only.
 - `GET /logs/stream/http` streams the HTTP access log only.
 - `GET /logs/stream/{model_id}` streams logs for one model (including IDs with slashes, like `author/model`).
 - `/health` - just returns "OK"
 - `/metrics` - system and GPU metrics for prome