Triton Inference Server
An open-source inference-serving system that deploys models from multiple deep-learning and machine-learning frameworks across cloud, data center, edge and embedded devices.
Triton Inference Server is an open-source inference-serving system that streamlines AI inferencing. It deploys models trained in frameworks such as TensorRT, PyTorch, ONNX, OpenVINO, Python and RAPIDS FIL, and serves them across cloud, data center, edge and embedded devices, on NVIDIA GPUs, x86 and ARM CPUs, or AWS Inferentia.
It supports concurrent execution of multiple models, dynamic and sequence batching with implicit state management for stateful models, and model pipelines built through ensembling or business-logic scripting. Custom backends can be added through a backend API, including backends written directly in Python. Inference is exposed over HTTP/REST and gRPC using the community KServe protocol, and a C API and a Java API let Triton be linked directly into an application for edge and other in-process use cases. It also reports metrics such as GPU utilization, throughput and latency.
It is part of NVIDIA AI Enterprise, and is written mainly in Python and released under the BSD-3-Clause license.