uclanlp / OpenVLHarness

OpenVLHarness
 Real-World Visual Agentic Tool‑Use with Multimodal Memory 
Turn any MLLM into an evidence-based visual reasoner that thinks and codes in visual space.
## ✨ Highlights
- **Training-free.** Works with the backbone's existing weights through native
 tool calling; nothing is fine-tuned.
- **Nine capabilities.** Visual grounding, OCR, depth, camera trajectory, zoom,
 text search, image search, webpage visits and a Python coding agent, each behind a
 task-level interface that returns evidence the model can read.
- **Multimodal memory.** Images, boxes, masks, depth maps and web references stay
 addressable for the whole question, so later tools and code can build on them.
- **Consistent gains.** Across 22 benchmarks in four domains, seven backbones gain
 **9.7 to 12.6 points** on average over the same models without tools.
- **Any registered model.** Presets for seven of the paper's eight backbones, plus
 every Qwen3-VL Instruct size and OpenAI's GPT-5 to GPT-6.1 models in the
 model registry.
## 🗂️ Table of Contents
- How it works
- Results
- Installation
- Quick start
- Evaluation
- Configure and extend
- Repository layout
- Documentation and tests
- Acknowledgements
- License
- Citation
## 🔍 How it works
The model chooses tools, inspects their evidence, and continues until it can answer.
Three components let it combine evidence across steps:
- **Capability layer:** nine task-level interfaces turn tool outputs into readable
 observations, including annotated images, summaries, and compact numerical data.
- **Multimodal memory:** images, structured data, and web references remain
 addressable throughout a question. Each new question starts with fresh memory.
- **Python Coding Agent:** the backbone writes code in a separate context to
 calculate over stored boxes, masks, depth maps, and other tool results.
For example, an agent can locate objects, estimate depth, and compute their spatial
relationship; or crop an unfamiliar object, search for visual matches, and verify
the answer on a webpage.
| Capability | What it provides | Default backend |
| --- | --- | --- |
| Visual Grounding | Locate objects by text; return boxes, masks, and an annotated image | SAM 3 |
| OCR | Read text in an image | PaddleOCR-VL |
| Depth Estimation | Estimate per-pixel depth; retain the numerical array and visualization | Depth Anything 3 |
| Camera Trajectory | Estimate camera poses across ordered images | Depth Anything 3 |
| Zoom-in | Crop a region for closer inspection | Built-in image processing |
| Text Search | Retrieve and summarize web, image, news, and other search results | Serper + Qwen |
| Image Search | Find visual matches with reverse image search | Serper Lens + Qwen |
| Webpage Visit | Extract evidence relevant to a goal from webpages | Serper scrape + Qwen |
| Python Coding Agent | Generate and execute computations over stored tool results | Orchestrator backbone + local Python |
Every preset uses **Qwen3-VL-8B-Instruct for multimodal search s