Video2GUI
Research code and the WildGUI dataset for a pipeline that extracts grounded GUI interaction trajectories from unlabeled internet videos to pretrain generalized graphical interface agents.
Video2GUI is the code release for an ICML 2026 paper on synthesizing large-scale interaction trajectories for GUI agent pretraining. It is a fully automated three-stage pipeline: a fine-tuned Qwen2.5-7B classifier filters over 500 million candidate videos down to about 4.2 million high-quality tutorial videos, each video is split into short segments and processed by Gemini under a sliding-window strategy to produce instruction-trajectory pairs, and the resulting data is assembled into the WildGUI dataset of 12 million interaction trajectories spanning more than 1,500 applications and websites.
Pretraining Qwen2.5-VL and MiMo-VL on WildGUI is reported to improve GUI grounding and action benchmarks by five to twenty percent over baselines, matching or exceeding state-of-the-art results. The dataset, with its annotations and screenshots, is released on Hugging Face.