# Auto-Tune vLLM

A distributed hyperparameter-optimization framework for tuning a vLLM language-model serving deployment, using Ray for distributed execution and Optuna for the search.

Auto-Tune vLLM runs optimization studies that search vLLM serving configurations for the best trade-off between throughput and latency, executing trials locally or across a Ray cluster with configurable concurrency, and storing trials, metrics and logs centrally in PostgreSQL. Benchmarking during each trial uses built-in GuideLLM support, with a plugin system for custom benchmark providers, and study and parameter configuration is written in YAML.

The command-line interface can start an optimization study, stream live logs for a specific trial, and resume an interrupted study. A known limitation is that the tool does not check whether a requested concurrent-trial count is actually achievable given the cluster's available GPUs, so excess trials are silently queued by Ray rather than triggering a warning; a separate command can check available cluster resources beforehand.

It requires Python 3.10 or newer, an NVIDIA GPU with CUDA support, and a PostgreSQL database, and is released under the Apache-2.0 license.

Website: https://github.com/openshift-psap/auto-tuning-vllm