SimWorld
An Unreal Engine 5 simulation platform for developing and evaluating LLM and vision-language model agents in realistic physical and social city environments, from a lightweight starter map to fully custom worlds.
SimWorld is a simulation platform for developing and evaluating large language and vision-language model agents in complex physical and social environments. It is built on Unreal Engine 5 and offers realistic, open-ended world simulation with physics and language-driven procedural generation, alongside a rich interface for multimodal perception and natural-language actions.
It supports three levels of use: a lightweight base package with two city scenes and one empty map for core agent interaction and quick testing; more than 100 additional pre-built maps for richer scenarios; and full customization with a user's own Unreal Engine environments, assets and agent models. Its architecture separates an Unreal Engine backend that provides the environment and physics, an environment layer with gym-like APIs and traffic simulation, and an agent layer where LLM and vision-language model agents reason over multimodal observations and history to navigate, plan and cooperate.
It is presented as a NeurIPS 2025 spotlight paper, with Python code for setup and use.