T-Crypt / berth

Quick start  ·  How it works  ·  Install  ·  Gateway  ·  Harbor  ·  Roadmap 
# Berth
Berth makes the programs on your PC take turns with the GPU's memory, so they stop crashing each
other, and tells you who is holding it.
Berth decides who gets your GPU. Chat models, image pipelines, voice engines, games and GPU apps
lease the card instead of grabbing it, so two of them never load at once and crash each other
with an out-of-memory error. One Go binary, no cloud account, nothing else to install.
 [code] 
## How it works
Every engine asks berthd for the card: through `berth run`, the gateway, ComfyUI's API or a game
claim. When another engine holds it, Berth:
1. queues the request by priority: `hold`, `foreground`, `interactive`, `batch`;
2. lets the holder finish its reply or image job, then unloads it;
3. waits until the driver shows the memory free;
4. grants the lease and starts the next engine, and the request that waited streams.
The numbered markers on the timeline are those four steps. A process that holds VRAM without a lease
shows up as a stowaway: Berth reports its PID and memory and leaves it running.
> **Status: alpha (v0.2).** Runs daily on an RTX 4090 workstation beside llama-swap, NInfer,
> Strata, ComfyUI and games. Capabilities lists what works and has tests; the
> Roadmap lists the phases and what is still missing.
### Berth and llama-swap
Use both. llama-swap swaps the models in its own config, and does it well. Berth governs the
whole card: llama-swap's models, ComfyUI, Ollama, vLLM, games, and any process that holds VRAM,
including the ones neither tool started. With Berth beside it, llama-swap's next model waits
for ComfyUI or the engine before it to give the memory back, and when a crashed process still
holds part of the card, Berth names its PID and how much it holds, so you aren't hunting
through `nvidia-smi` while a load fails. `berth wire llama-swap` adds the lease to every model
in your config in one pass. An upstream integration with llama-swap is planned, so the wrapper
line goes away.
## What Berth does
**Shares one card between everything**
- **Engines take turns.** You start each one through `berth run`. When another needs the
 card, Berth waits for the holder's current reply to end, unloads it (its own unload, then
 SIGTERM, then SIGKILL) and lets the next one in only after the driver shows the memory free.
- **Priorities.** `foreground`, `interactive`, `batch`, and `hold` above them all. A lower
 request waits and gets a reason naming the holder, never an OOM crash.
- **No ping-pong.** When two clients ask for different engines at once, a ranking picks one
 and the other waits, so the card stays put.
- **ComfyUI at a job boundary.** Berth waits for the running image job to finish before the handoff.
- **Survives restarts.** Upgrade or restart the daemon and your running engines stay up and
 re-attach. The gateway's engines survive it too: it starts each one as its own `berth run`
 process (in its own systemd scop