fyzhang1 / ExpActivator

# Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents
Fuyao Zhang, Xuan Wang, Zherui Li, Jiaming Zhang, Longtao Huang, Wei Yang Bryan Lim
Nanyang Technological University, Alibaba Group
Paper · PDF
huggingface
## Demo
Watch on the homepage · Download the 12-second MP4
Two illustrative workflows show screen-specific preference activation and a timely weather suggestion followed by user acceptance. App interfaces and actions are simulated. See Demo production notes.
## Overview
Personal GUI agents usually append retrieved user history to the policy context and assume it stays useful at every step of a task. ExpActivator treats history as something to **activate**, not just retrieve:
- **Step-level activation.** Each new screen is matched to historical states in the latent space of the frozen GUI backbone, and the matching historical action is given to the policy as a reference it may adopt, adapt, or reject.
- **Task-level activation.** A recurring intent is proposed proactively only when time and scenario support it, and the method abstains otherwise.
It is training-free: no model or prediction parameter is fitted, and the backbone stays frozen. It is evaluated on the two tasks of PersonalAlign: personalized execution (775 episodes, 7,915 actions) and proactive suggestion (215 positive and 100 negative states), with four GUI backbones (Qwen3-VL-8B, MAI-UI-8B, Qwen3.5-9B, GUI-Owl-1.5-8B).
## Install
 [code] 
## Data
| Path | Content |
|---|---|
| `data/personal_execution/` | execution benchmark samples |
| `data/proactive_suggestion/` | proactive-suggestion samples (positive and negative states) |
| `samples/total.csv` | full interaction log, split chronologically per user |
| `samples/user_profile.csv` | user profiles |
Historical screenshots are not included. Download the Fingertip20k history from PersonalAlign and point `HISTORY_ROOT` at it.
History memory is always the chronological first 80% per user of `samples/total.csv`. Do not use `samples/train_set.csv` as memory because it overlaps the execution test episodes; `tests/test_history_split.py` checks this.
## Run
 [code] 
Compile history and calibrate on history only (no test answers are opened):
 [code] 
Personalized execution on all four backbones (calibration, visual cache, evidence construction, policy inference):
 [code] 
Proactive suggestion:
 [code] 
Ablations (history size, refresh, proactive sensitivity, state conditioning) are documented in the protocol files under `expactivator/`; see expactivator/README.md for the full command list.
## Tests
 [code] 
## Layout
 [code] 
## Citation
 [code] 
## Acknowledgement
Thanks for PersonalAlign benchmark