Haru
A GPU hardware-abstraction layer for LLM inference fleets that gives hot failover between an active and standby domain at the cost of zero idle GPUs.
Haru is a GPU hardware abstraction layer for LLM inference fleets: an independent layer with its own state store that handles GPU provisioning, runtime supervision, regional failover and traffic placement, so a product's control plane can stay focused on users, catalogs and metadata and talk to Haru through a small HTTP API instead of coupling directly to one GPU provider.
Its initial architecture keeps an active domain serving OpenAI-compatible inference traffic while a standby domain keeps the same model runtimes resident but puts vLLM into level-1 sleep mode, offloading weights to CPU RAM and freeing standby VRAM for preemptible LoRA training. On failure, Haru stops training, verifies the GPUs released that VRAM, wakes vLLM from sleep, probes every model, and flips routing with a single database compare-and-swap; a promotion that fails before that last step never touches routing.
It is written in TypeScript.