jesdga95 / basalt

Basalt is a local inference engine for the **Qwen3.8-Flash-Next** mixture-of-experts model.
It runs on Linux and Windows with an NVIDIA Blackwell GPU (sm_120); see platforms for limitations.
Experts that do not fit in VRAM live in system RAM, and optionally on a second GPU used as a peer expert tier.
A large MoE model runs on one workstation, behind an OpenAI- and Anthropic-compatible API.
## Requirements
- Linux x86-64 with AVX2 (AVX-512 is used when present), and an NVIDIA Blackwell GPU (compute capability 12.0) with CUDA 13.
- CMake 3.24 or newer, a C++20 compiler, and Python 3.10 or newer.
- Enough system RAM to hold the experts that do not fit in VRAM.
## Quick start
 [code] 
Put your settings in `basalt_config.json`, with `model` pointing at the downloaded file:
 [code] 
 [code] 
The server listens on 127.0.0.1 by default.
## Models
Ready-to-run packs, with the quality of each measured against Q8_0, are on
Hugging Face:
`Qwen3.8-FN-NVFP4`, `-UD-Q4_K_XL`, `-UD-Q5_K_XL`, `-UD-Q6_K_XL`, `-Q8_0`, `-GSQ-RCO-IQ3_S`, `-GSQ-RCO-IQ3_XXS` and `-GSQ-RCO-Q2_0` (each `.basalt`).
A model is one file: weights, tokenizer, MTP drafter, expert profile and vision projector.
## Docs
- Configuration: every setting.
- Running: parallel requests, prompt chunks, memory floors, image input.
- The .basalt model file: the layout, and building a pack yourself (including NVFP4).
- Platforms: Linux and Windows requirements, the build steps and the Windows limitations.
- Building and testing, Precision and quality, Contributing.
## Credits
- Strata: the project Basalt forks (MIT).
- Qwen: the model. The packs are under the Qwen Community License 1.0, not Basalt's MIT license.
- unsloth: the UD and Q8_0 GGUFs the packs are built from.
- IST Austria DAS Lab: the GSQ-RCO quants; please cite GSQ and RCO.
- NVIDIA: the NVFP4 experts.
- ggml and llama.cpp: quantization formats and kernels extracted into `src/` (MIT, see THIRD_PARTY_LICENSES).
## License
MIT, see LICENSE.