Keys-Setup — Autonomous Self-Improving Local Inference Stack
A self-hosted inference stack for a cluster of NVIDIA DGX Spark machines that routes tasks across local and cloud models, then LoRA-trains the local models on its own routing history so the local share of work grows over time.
This project wires a four-node NVIDIA DGX Spark cluster, connected over a high-speed network fabric, into a mixture-of-agents router that keeps most AI work on local hardware and escalates only the hardest tasks, such as deep audits or frontier research, to a rate-limited cloud model. A served router and aggregator model classifies each incoming task and routes it to the cheapest model capable of handling it, while logging every task and its verdict.
Those routing logs feed a LoRA self-training loop that mines the transcripts for supervision and continuously retrains the local models, so the stack is intended to rely on the cloud less over time rather than at a fixed rate. The repository also documents a single-node recipe that collapses the same routing and training loop onto one DGX Spark machine using an all-Gemma-4 model lineup.