PrismML-Eng / Bonsai-demo

# Bonsai Demo
Backend and model format compatibility: BACKEND-SUPPORT.md.
 Website  | 
 GitHub  | 
 Discord 
 Models: 
 Bonsai 2 27B GGUF ·
 Bonsai 2 27B MLX 
---
Run **Bonsai 2 2B, 4B, 9B, and 27B** locally on Mac (Metal), Linux/Windows (CUDA, Vulkan, ROCm), or CPU.
For the earlier **Bonsai 1-bit and Ternary-Bonsai** models (27B, 8B, 4B, 1.7B), see
Bonsai1_README.md. For troubleshooting, see FAQ.md.
## 🌱 Bonsai 2: 2B, 4B, 9B, and 27B
**Bonsai 2 now includes 2B, 4B, and 9B models alongside 27B.** Pick a smaller model
for a lighter local setup; 27B remains the demo's default. GGUF and MLX checkpoints
are available for each size.
The following benchmark and memory highlights describe **27B**; see each smaller
model's card for its specifications:
- **98.2% of FP16 intelligence retained** at roughly a ninth of the size, with the reasoning core
 intact: math within half a point of full precision, coding level with the baseline.
- **Vision:** send photos, screenshots and PDFs and ask about them, on both llama.cpp and MLX
 (see VISION.md).
- **Agentic tool calling:** native OpenAI-style `tool_calls` with full round-trips, plus MCP servers
 in both demo UIs (see TOOLS.md).
- **Thinking:** a reasoning model; pick the reasoning effort per chat in the UI or budget it per request.
- **262K-token context**, kept practical on-device by the hybrid-attention backbone.
- **1.75 bits per weight** in the `PTQ1_0` packing. A second packing, `PQ2_0`, trades 1.3 GB for
 faster prompt processing and is what this demo downloads by default.
 See MODEL-FORMATS.md.
**Official `Q2_0` works with current upstream llama.cpp containing Bonsai 2 support.**
`PQ2_0` and `PTQ1_0` still require the PrismML fork.
This demo downloads `PQ2_0` and the matching fork binaries by default.
All three packings contain the same model weights, with different memory/speed trade-offs.
Quick Start below gets you there in two commands: `./setup.sh` downloads Bonsai 2 27B, then
`./scripts/start_llama_server.sh` gives you chat, vision and tools at http://localhost:8080.
### Choose a smaller model
 [code] 
On Windows, set `$env:BONSAI_MODEL = "4B"` before setup and launch. For Apple Silicon,
use `./scripts/run_mlx.sh` or `./scripts/start_mlx_server.sh` with the same model selection.
See small-model sampling and API examples: the 2B/4B/9B profiles
use min-p 0 and a 64-token presence-penalty window. The 27B settings below are separate.
## Best Practices (Bonsai 2 27B)
These match the model card. The start scripts already apply the thinking-mode sampling, so you only need these when calling the model from your own client.
### Generation Parameters
| | Thinking mode (default) | Instruct / non-thinking |
|---|---|---|
| `temperature` | 1.0 | 0.7 |
| `top_p` | 0.95 | 0.80 |
| `top_k` | 20 | 20 |
| `min_p` | 0.05 | 0.0 |
| `presence_penalty` | 0.0 | 1.5 |
| `repetition_penalty` | 1.0 | 1.0 |
`min_p=0.05` drops tokens far less likely than the top choice; in our tests it scored at least as well as `min_p=0.0`