OSWorld
A benchmark, presented at NeurIPS 2024, for evaluating multimodal AI agents on open-ended tasks inside real virtualized computer environments rather than sandboxed simulations.
OSWorld is a benchmark for evaluating multimodal agents on open-ended tasks inside real computer environments, rather than in artificial or heavily sandboxed simulations. It supports running its virtual machine environments on VMware or VirtualBox on a desktop, laptop or bare-metal machine, or through Docker for hosting on virtualized platforms such as AWS or Azure.
Its OSWorld-Verified update fixed community-reported issues, added AWS support that can cut evaluation time to under an hour through parallelization, and made benchmark signals more reliable; new model results using the updated benchmark are published on the project's website, and users are asked to compare results against that newer baseline rather than the original numbers when running the latest version. A separate desktop-env package can be installed without the benchmark's own task suite for environment-only use.
It is written in Python and released under the Apache 2.0 license.