ArtificialAnalysis / aa-agentperf-local

agentperf-local 
 Benchmark how fast your machine serves an AI agent. 
 An open-source tool from Artificial Analysis .
`agentperf-local` measures how fast your machine serves an AI agent. It replays
recorded agent conversations against an OpenAI-compatible model server and
reports throughput and latency. Each request carries the full conversation so
far, as a real agent's would. It measures speed without measuring output quality.
It can benchmark a server you already run, or it can download a pinned model,
start the server, and benchmark it for you.
## Quick start
You need `uv`. It fetches Python 3.12 if you do
not have it. Docker and Rust are optional. macOS, Linux, and Windows are
supported. On Windows, managed runs need an NVIDIA GPU and llama.cpp; vLLM and
SGLang run only on Linux.
You also need either a model server that you already run with an
OpenAI-compatible API, such as llama.cpp, LM Studio, vLLM, SGLang, or Ollama,
or a serving framework that the tool can start. The bundled recipes use
llama.cpp (`llama-server` on your `PATH`), vLLM or SGLang on NVIDIA GPUs, or
Splash on Apple Silicon, at the exact version each recipe pins. The tool does
not install frameworks.
`agentperf-local deployment-options --profile-id ` shows which frameworks
can serve a recipe on this machine.
 [code] 
To try it without installing, run `uvx agentperf-local`. `pipx install
agentperf-local` also works.
To work from a source checkout instead:
 [code] 
The examples below use `uv run agentperf-local` from a checkout. With an
installed tool, drop the `uv run`.
The TUI walks you through choosing a model, setting up, and running. Arrow keys
move, Enter continues, Escape goes back, `?` opens help, and `q` quits. To
prefill settings with flags, use `agentperf-local tui`. See
TEXTUAL_TUI.md.
To run without the TUI, pick a catalog profile and a framework you have
installed:
 [code] 
`managed-run` downloads the pinned model into the Hugging Face cache, checks
every file's SHA-256, starts the server on localhost, runs the replay, and
stops the server. Every recipe it can run is a YAML file in
`recipes/`, by model then hardware.
The `qwen38-27b-splash-dflash` recipe serves on Splash,
Inco AI's Metal engine for Apple Silicon (M3 or newer, macOS 26.4 or later).
Install it with `brew install incoai/tap/splash`. The recipe pins Splash 1.2.1
and the `incoai/Qwen3.8-27B-Splash` package by commit. `managed-run` downloads
and verifies the package itself, then starts Splash's server on that snapshot,
so Splash never resolves a newer revision. Splash 1.2.1 honors `ignore_eos` on
requests with tools, so this recipe runs the `exact` policy. Its KV cache is
always eight-bit:
 [code] 
## The default run
The default replay is `agentperf-default-v1`: eight recorded agent tasks with
168 model turns. It uses the `exact` output policy, which makes every turn
generate its recorded number of tokens. This is the comparable run.
`--replay aa-mini-v1` selects a six-turn synthetic replay. It is a qu