# ggml-rocket

A drop-in ggml backend that offloads llama.cpp and whisper.cpp prefill computation to Rockchip RK3588 and RK3576 NPUs, leaving sequential decode on the CPU where the NPU has no advantage.

ggml-rocket is a runtime-loadable backend that makes a Rockchip NPU appear to ggml-based programs such as llama.cpp and whisper.cpp as an ordinary compute device, the same way ggml's own BLAS backend does, by pointing an environment variable at the compiled library. It runs the Whisper speech-recognition encoder end to end on the NPU and offloads the prefill stage of several language models, including Gemma, Qwen, Llama, Phi, and Ministral, to the NPU as well.

Decode stays on the CPU because it works on too few rows at a time for the NPU's per-row cost to pay off, which covers both single-row decoding in llama.cpp and the small batches used by whisper.cpp's beam search. It depends on a separate userspace driver library and documents per-model benchmarks and recommended configuration by workload.

Website: https://github.com/gregordinary/ggml-rocket