zyhector / token-rush
git clone https://github.com/zyhector/token-rush.git
Single-stream LLM inference engine for Qwen3.8-27B on one RTX 5090 — int4 quantization + speculative decoding + custom Triton/CUDA kernels; 229–379 tok/s vs llama.cpp/SGLang/vLLM at batch size 1.
+0
stars, 24h · #78,612
+~0
stars, 7 days · #78,962
+~0
stars, 30 days · #77,819
25
★ total
5
forks · 1 open issues
Star history (as recorded here)
LanguagePython
License—
Created2026-09-04 (2mo ago)
Last push2mo ago
OwnerUser
GitHub topics
cudagpu-inferenceinference-engineint4llm-inferencellm-servingperformance-optimizationpytorchquantizationqwenrtx-5090speculative-decodingtriton
Links
README
Loading README…