Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps
The agent trains on live iOS apps rather than memorized static trajectories. It uses LLM-judge rewards, PPO with step-level advantages, and GiGPPO value estimation.
The claim
Online RL trains a planning-and-grounding agent on live iOS apps instead of static traces.
The problem
Frontier vision-language models navigate well but cost too much.
Smaller models trained offline memorize static trajectories.
They execute brittlely and cannot recover from errors.
How it works
The agent splits into a planner and a UI grounding model.
An LLM judge gives step rewards and a terminal reward.
PPO uses step-level advantages from that dense reward signal.
GiGPPO estimates values from visually similar states across trajectories.
Infrastructure runs hundreds of concurrent virtual iOS devices.
Figures
What changes
Agents beat SFT and terminal-only RL baselines and converge faster.
An 8B agent gains nearly sevenfold out-of-distribution on Android.
On real iOS apps it ties Gemini-3.1-Pro.
Limits
Rewards come from an LLM judge, not human labels.
Training needs an infrastructure of hundreds of virtual devices.
Source · arXiv