syv-ai / HyperQwen

HyperQwen 
 Large Qwen models, served fast on the GPUs people actually own. 
 One request against stock vLLM, replayed in real time from recorded token arrivals, then 64 at once.
 1080p60 MP4 · how it is made 
**Today:** Qwen3.8-27B on a single 24 GB
consumer GPU with vLLM — 150k token context and an OpenAI-compatible API with key
auth, in two ready-made modes. On an RTX 3090 at 250 W: 127 tok/s single-stream
in single-user mode, ~1,035 tok/s aggregate decode at 64 concurrent in batch mode,
and 381 tok/s when the answer quotes its own prompt.
**Where it is going:** more Qwen models, more classes of card. What this repo
really is, under the serving setup, is a patch series against a pinned vLLM plus
a model-preparation pipeline — requantized heads, a calibrated draft vocabulary,
speculation that drafts out of the prompt. Most of that is not specific to one
checkpoint or one card; it is just that one checkpoint and one card are what has
been measured to death. See Roadmap for
what is and is not portable yet, and
Field tests wanted if you have half an hour on a GPU we
have never touched.
## Quick start
 [code] 
First start pulls the image (9.5 GB) and requantizes the model (~20 GB, once,
into `./models`), then serves on `:18020`. One GPU runs one mode at a time.
- **Before exposing it** — compose publishes the port on `127.0.0.1` only (`BIND` in `.env`, default `127.0.0.1`), so another machine cannot reach it until you set a key and `BIND=0.0.0.0`; outside a container, no key means the server binds `127.0.0.1` only:
 `echo "VLLM_API_KEY=$(openssl rand -hex 24)" >> .env && echo BIND=0.0.0.0 >> .env`
- **Docker Desktop on WSL2** — keep `VLLM_WSL2_ENABLE_PIN_MEMORY=1` in `.env`, or
 the V2 runner aborts with `RuntimeError: UVA is not available`
- **No compose, or no Docker at all** —
 docs/docker.md ·
 docs/install.md
## Which setup do I want?
Three questions, and the answers are the whole of your `.env`.
 [code] 
Starting from the `.env` you copied in Quick start, which already ships **B**:
| | change in `.env` | start with | what you get |
|---|---|---|---|
| **A** | nothing | `--profile batch` | ~1,035 tok/s aggregate at 64 concurrent |
| **B** | nothing | `--profile single` | 127 tok/s single stream, 64k context |
| **C** | add `DFLASH_TOKENS=15` | `--profile single` | 381 tok/s while quoting, at 4 slots and 56k |
| **D** | `SPEC=mtp` and `CTX=long` | `--profile single` | 150k context, ~95-100 tok/s |
| **E** | add `CTX=huge` | `--profile single` | 240k context, 67 tok/s mixed and 164 while quoting |
Torn between B and C: B is the safe default, and C only pays off when the output
really does repeat the input — it trades request slots and context for it. D drops
back to MTP speculation on purpose, because DFlash2 past 64k is worth it only for
reproduction and loses to `SPEC=mtp CTX=long` about 2:1 on everything else; at E
the KVarN cache buys the context instead, so DFlash2 stays on. A needs no edit at
all — batch ignores `SPEC` and says so on start