LINs-lab / UltraText_Bench

UltraText Bench 
 A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation 
Deyuan Liu † · Yihao Hu † · Jingxuan Zhang † · Xingying Li † · Jun Xie † 
Jiacheng Liu · Jungang Li · Yu Huang · Xuanyi Liu · Yue Ding · Zecheng Wang 
Lei Zhao · Mingda Wang · Zhenglin Cheng · Peng Sun · Tao Lin * 
Westlake University · Ant Group · Zhejiang University · Shanghai Innovation Institute 
HKUST · CASIA · Wechat AI · MBZUAI · CityU · Peking University 
 † Equal contribution   * Corresponding author
 Project page ·
 Dataset ·
 Leaderboard ·
 Quick start 
**UltraText Bench tests whether image generators can reproduce dense English and Chinese text across an entire scene.** A convincing title is only part of the task: body text, prices, labels, and small supporting regions must also preserve the requested content and placement.
The benchmark contains **432 prompts**, **24 scene categories**, **three difficulty levels**, and **2,926 annotated text regions**. English and Chinese each contain 216 prompts. Every generation prompt supplies all target strings; the structured region reference is used by the evaluator.
 L3 scene coverage: one selected GPT Image 2 output per category at API quality Low, reproduced from Figure 1. These examples illustrate the tasks; they are not human correctness labels. 
## What is included
- The complete English and Chinese prompt records, including target strings, region positions, relative sizes, roles, carriers, and importance.
- Generation runners for Qwen-Image and Z-Image, plus an explicitly configured GPT Image gateway adapter.
- The Q-Judger evaluation client, complete-reference rubric, response validation, score aggregation, and coverage reporting.
- Dataset audit snapshots, regression tests, and the current paper.
This repository accompanies UltraText Bench (arXiv:2610.09823).
## Benchmark
The 24 categories span signs and labels, documents and print, commercial materials, digital interfaces, structured data, and creative scenes. Each category has three prompts at each difficulty level in each language.
- **L1 (Hard):** four to seven regions per prompt.
- **L2 (Very Hard):** five to ten regions per prompt.
- **L3 (Extreme):** seven to twelve regions per prompt; English examples contain up to 5,230 target characters.
Difficulty levels group different prompts and workloads. They are not controlled edits of the same scene. Character counts include spaces and punctuation; a region may contain multiple lines. See dataset statistics for the exact distributions.
Q-Judger returns six integer ratings from 0 to 100. They form four reported dimensions:
 [code] 
Only successful, schema-valid ratings enter quality means. Missing images and evaluation failures remain in coverage counts with null scores. The primary mean first averages successful images within each prompt, then averages represented prompts equally. The bilingual result gives equal weight to the English and Chinese prompt-macro means.
## Leade