AndroidWorld
A reproducible benchmark and environment, running on a live Android emulator, for building and evaluating autonomous computer-control agents across 116 hand-crafted tasks in 20 real apps with millions of randomly generated task variations.
AndroidWorld is an environment for building and benchmarking autonomous agents that control a computer, running against a live Android emulator rather than a simulated one. It contains 116 hand-crafted tasks across 20 real-world apps, dynamically instantiated with randomly generated parameters to produce millions of unique task variations, each with a durable, code-checked reward signal for reliable evaluation rather than relying on an LLM judge.
Beyond its built-in Android tasks, it also supports the MiniWoB++ web-based benchmark from Liu et al., giving it both mobile and web task suites under one evaluation harness. It has a lightweight footprint of about 2 GB of memory and 8 GB of disk, experimental Docker support for a consistent setup, and an extensible design for adding new tasks.
Setup requires the Android Studio emulator with a configured Android Virtual Device, Python 3.11 or newer, and API keys for the model providers being evaluated as agents.