jwuthri-cloud / Tracely-ai

# Tracely
### Production failures become regression tests.
**Trace-native CI/CD for AI agents.** Tracely grades every agent trace as it lands, clusters the
failures into issues, freezes the bad runs into hermetic replayable cases, blocks the pull request
that would ship them again — and tells you the moment any of it happens.
 production trace → failure detection → regression test → CI gate → alert 
**Website** · **Docs** · **Product guide** · **Agent skill** · **Guided tour** · **2-min demo** · **Design dossier**
**Self-host the whole stack in one click** — API, worker, UI, Postgres, ClickHouse, Redis and MinIO:
 One workspace: what ran, what broke, what is already pinned as a test. 
---
## Why another agent-observability tool?
Because observability stops at the dashboard. You can see that your agent broke — then what?
Every eval tool asks you to **hand-author a dataset**: sit down, invent questions, write ideal
answers, keep them current as the product changes. That dataset is a guess about what might break.
Production already handed you the real thing: a trace of the exact run that failed, with the exact
input, the exact tool calls, the exact model responses.
> **The recorded run _is_ the test.** Tracely freezes that trace into a hermetic regression case and
> replays it on every PR. Everything else — quality scores, failure clusters, suggested fixes, CI
> verdicts, trends, alerts — is **derived from the trace**. There are no hand-authored datasets.
| | Dataset-first tools | Tracely |
|---|---|---|
| Where tests come from | You write them | Promoted from real failing traces |
| Fidelity to production | A guess | The exact failing run, byte for byte |
| Cost to replay in CI | Live model calls | **$0** — recorded tool/LLM fixtures |
| What happens on regression | A dashboard number moves | **The PR is blocked** |
| How you find out | You go and look | **It comes to you** — Slack, email, your own webhook |
---
## The loop
Five steps, five pages in the app.
### 1 · Observe — every run, hierarchically
Traces arrive over plain OTLP. Agent semantics (`agent.id`, `conversation.id`, `turn`, `step`) are
promoted to first-class indexed columns, so runs group into conversation threads instead of a flat
span soup. The waterfall shows agent → thinking → skill → generation → hand-off, with the failing
span in red and its I/O beside it.
Evaluators are **columns on the trace table**, not a separate tab — each one grades at conversation,
run or span level and writes its verdict into the grid. Scores stream in live over SSE as judges
finish, so you watch a run get graded in place. Tokens, cost, latency, metadata and the rolling
per-turn summary are columns too.
 Two extra ways to read one conversation — a step-by-step replay, and a pixel-art office 
**Replay** walks the conversation event by event on a timeline. **Fleet** renders the same
conversation as a room: each agent is a character, hand-offs are walks across the floor, tools are
objects on the wall — whic