BBBBchan / DeltaStreamer

### Towards Human-Like Agentic Streaming Video Understanding
**Boyuan Sun** 1 · **Deshui Miao** 2 · **Wenzhao Gao** 2 · **Shaoyong Jia** 2 · **Shaohui Jiao** 2 
**Chi Liu** 3 · **Yan Shu** 3 · **Derek Li** 3 · **Bryan Dai** 3 · **Shengsheng Qian** 4 · **Qibin Hou** 1 
 1 VCIP, Nankai University 
 2 ByteDance Inc.   3 IQuest Research 
 4 Institute of Automation, Chinese Academy of Sciences
**Overview · Getting Started · Evaluation · Deploy Interactive Demo ·Results · Citation**
> **Application released:** AI同担 (AI-Tongdan) is a livestream companion built on DeltaStreamer, with continuous observation, contextual Q&A, and danmaku suggestions. Watch the AI-Tongdan video on Bilibili →
## Overview
**DeltaStreamer** is an agentic framework for streaming video understanding inspired by how humans observe, remember, and investigate. Video frames arrive over time, while a later question may depend on a detail that was not retained during observation. DeltaStreamer connects **change-aware memory** with **query-driven planning and targeted visual replay**, allowing the system to investigate beyond its stored descriptions.
> Observe the stream, remember what happened, and revisit the visual evidence when memory falls short.
### Highlights
| Component | What it does |
| --- | --- |
| **Explicit visual deltas** | Compare consecutive frames and retain their changes alongside scene descriptions, making event transitions explicit. |
| **Structured textual memory** | Organize observations, episode summaries and pinned facts in a hierarchical DAG, with entity indexing and pointers to source evidence. |
| **Query-driven planning and replay** | Identify the evidence a question needs, guide subsequent retrieval and tool use, and revisit historical frames when visual details are missing. |
The release contains the inference implementation, preparation scripts and scoring tools for **OVO-Bench** and **StreamingBench**. It requires no additional training to run the agent with a supported local VLM checkpoint.
### Qualitative Examples
The first example uses recorded changes to recover an action sequence. In the second, the planner locates an earlier scene and requests historical visual evidence to read letters missing from textual memory.
## Getting Started
### 1. Installation
Run the following commands from the repository root, in a separate Python environment. The source environment uses **Python 3.10+, PyTorch 2.10.0, vLLM 0.19.1 and transformers 5.5.4**, with NVIDIA GPUs. FFmpeg is required for media preparation and audio extraction.
 [code] 
`setup_runtime.py` downloads official **ms-swift 4.2.0** and applies the inference compatibility file in `compat/` to an isolated `.runtime/` directory. This preserves the current per-response tool-call generation limit without changing installed packages. Use this runtime when launching the agent.
### 2. Model and Tool Checkpoints
Prepare the following checkpoints separately:
| Checkpoint | Use | How to provide it |
| --- | --- |