FlashRT
FlashRT is a high-performance real-time inference engine for small-batch, latency-sensitive AI workloads such as robot vision-language-action control, built around hand-tuned kernels rather than a compiled execution graph.
FlashRT combines a latency-first static execution pipeline, system-level quantization and calibration, cross-layer fusion plans, captured execution graphs over stable buffers, and hand-written or explicitly adapted kernels, without an ONNX export or engine compilation step, so a new NVIDIA target needs only a routine submodule update rather than a per-driver rebuild. Its composition pattern is hardware-agnostic, currently covering NVIDIA Jetson AGX Thor and Orin through the A100 and RTX 40 and 50 series, with AMD GPU and NPU support in progress.
Its flagship use is production vision-language-action control for robotics models including Pi0, Pi0.5, GROOT N1.6, GROOT N1.7 and Pi0-FAST, validated on the LIBERO benchmark, cutting per-decision latency and raising the policy decision rate several times over compared to the models' own reference implementations. The same kernels also serve a world-model research pipeline, a text-to-speech model, video-policy generation, and single-stream large language model inference for long-context serving.
FlashRT is aimed at a workload shape, small-batch, real-time inference, rather than one model class, distinguishing it from compile-then-serve engines such as TensorRT or high-batch serving engines such as vLLM and SGLang. It is written in C++.