nyacg / pagedb

# pagedb
**A 100 GB database in R2, served by one Durable Object, with live queries in every tab.**
pagedb keeps an immutable base in R2 as 4 MiB parts of 64 KiB B+tree pages. One Durable Object
serves it and only ever holds what queries touch. Every list, board column and thread in the demo
is a **live query**: when anyone writes, the object works out which rows of which open views changed
and pushes exactly those.
It's our take on r2db, rebuilt from its public behaviour (no code copied),
with benchmarks against it. The engine is Rust: WebAssembly inside the Durable Object, and native
for the dataset builder. The Durable Object shell, sync protocol and UI are TypeScript.
Four apps run on the same engine, each one database in R2 and one object serving it:
| App | Data | What's live |
|---|---|---|
| Issue tracker | 27.1M generated issues, 96 GB | lists, board, threads, search (titles, tags, comments) |
| Hacker News | every story and comment since 2006 (~49M items) | new items stream in from the HN API; front page, threads, comments, votes |
| r/place 2022 | all ~160M pixel placements | each pixel's history, each user's pixels, a timelapse, painting |
| Taxi sheet | every NYC yellow-cab trip of 2015 (~146M) | sort, jump to any row, group counts, shared cell edits |
- How it works: https://pagedb.do-pool-lab.workers.dev/how
- The same object in a container: https://pagedb-node.do-pool-lab.workers.dev/api/…
## Results (deployed, measured)
| | pagedb | r2db demo |
|---|---|---|
| Issues on screen after opening the page (both 100 GB, same laptop, median of 8 loads) | **0.25 s** | 1.09 s |
| Every demo query from cold (100 GB) | **1 fault round**: list, board column, count, filter, detail, feed | rounds = dependency depth |
| Warm queries from a laptop (100 GB) | 35–49 ms | — |
| Count of a project's issues | 3.6 µs of engine time (counted branch pages) | 3 ms (published) |
| Your own edit | instant (optimistic UI) | after the server round trip |
| Object waking from sleep (1 GB base) | ~0 ms of I/O: manifest, interior pages and cache are in its SQLite | — |
| Fan-out, 1,000 sockets, 5 writes/s (1 GB base, laptop ↔ Cloudflare) | p50 255 ms, p99 684 ms; **container: 128 / 296 ms** | — |
| Builder | 96 GB, 27.1 M issues, 1.47 M pages in 28 min on one Cloudflare vCPU | offline |
Details: `bench/results/*.md`.
## How it works (short)
1. **Tiers.** A page is looked up in the engine's RAM (32 MiB), then the object's SQLite (synchronous,
 survives sleep), then the Cloudflare cache in the object's city (10–30 ms; parts are immutable),
 and finally R2 (100–370 ms). A page id is its position in a part, so its R2 location is arithmetic.
2. **Faults, not awaits.** The engine never blocks. A query that needs non-resident pages records
 *all* of them and returns `Fault([ids])`. The shell fetches them as coalesced 1 MiB chunks (at most 6
 at once, a DO limit) and calls again.
3. **One round per query.**
 - Every tree's interior pages and the small tables sit in a pi