llmtop
A real-time terminal dashboard, styled like htop, for monitoring an LLM inference cluster running vLLM, SGLang, LMCache, NVIDIA NIM, or NVIDIA Dynamo, auto-discovering pods in Kubernetes with no port-forwarding required.
llmtop is a terminal-based monitoring tool for LLM inference clusters, showing real-time key-value cache usage, queue depth, time-to-first-token and inter-token latency, and token throughput across every worker, alongside a GPU resource view with utilization, VRAM, temperature, and power drawn through a DCGM exporter. It also offers a model-grouped view that aggregates statistics by model with drill-down detail.
In a Kubernetes cluster it auto-discovers inference pods through the API server proxy with no port-forwarding needed, or it can be pointed directly at specific endpoints or a configuration file, and it supports a one-shot snapshot mode with JSON output for scripting. It auto-detects and labels metrics from vLLM, SGLang, LMCache, NVIDIA NIM, NVIDIA Dynamo, TGI, TensorRT-LLM, Triton, llama.cpp, LiteLLM, and Ollama, filtering and labeling Dynamo's separate prefill and decode workers automatically.