LLMServingSim
A cycle-level simulator for LLM serving infrastructure that models heterogeneous accelerators, disaggregated memory tiers, and multi-instance parallelism to study serving performance without real hardware.
LLMServingSim is a cycle-level simulator for LLM serving infrastructure. It pairs a Python frontend that mirrors vLLM's continuous-batching scheduler with the ASTRA-Sim C++ analytical network backend, and drives both from per-hardware latency data captured by a vLLM-based layerwise profiler.
The simulator models heterogeneous accelerators, disaggregated memory tiers such as CPU, CXL, and PIM, mixture-of-experts routing, and multi-instance parallelism across tensor, pipeline, expert, and data dimensions end to end. It includes a profile library of hardware such as the RTX 4090, validated to within 1% of a real vLLM run on time-to-first-token, time-per-output-token, and latency.
It ships as a container with example workloads and a validation suite of recorded scenarios, and is described in papers at ISPASS 2026, CAL 2025, and IISWC 2024.