TensorRT-LLM
An SDK that provides a Python API for defining large language models and running them efficiently on NVIDIA GPUs, with specialized kernels, an optimized runtime and components for building custom inference servers.
TensorRT-LLM is an SDK for optimizing inference of large language models and visual generation models on NVIDIA GPUs. It provides a Python API for defining models, specialized kernels for common LLM operations, an optimized runtime, and a Python framework that can be customized and extended.
It also includes components for building Python and C++ runtimes that orchestrate inference execution, letting a team assemble their own serving stack around the optimized kernels rather than relying on a single fixed server. Ongoing engineering work covers areas such as expert-parallel serving of mixture-of-experts models, speculative and disaggregated decoding, guided decoding, quantization and sparse attention, aimed at higher throughput and lower latency on current NVIDIA hardware such as Blackwell GPUs.
The project also ships agentic coding skills covering TensorRT-LLM usage and benchmarking, for use with AI coding agents.