jitLLM
A large-language-model inference and serving engine written entirely in Java, using the TornadoVM compiler to just-in-time compile transformer kernels to CUDA, OpenCL or Apple Metal at runtime, so the same build runs on any GPU vendor.
jitLLM lets a developer write and ship ordinary Java while TornadoVM compiles the performance-critical transformer kernels to the GPU backend actually present on the machine, without JNI glue code, a second toolchain, or a native rebuild per GPU vendor. One jar file is reported to run the same model on NVIDIA, Intel, AMD and Apple Silicon hardware, with the backend auto-detected from the installed TornadoVM SDK and switchable with a flag.
It integrates with the existing Java AI stack through an official LangChain4j provider and a Quarkus inference engine, and can serve models behind an OpenAI-compatible HTTP API with streaming, alongside llama-bench-style benchmarking that reports results in several output formats. Supported model families, all in GGUF format, include Llama 3, Mistral, Qwen 2.5 and Qwen 3, Phi-3, IBM Granite, and DeepSeek-R1-Distill. Further work in progress includes on-device greedy sampling and static batched decoding across multiple sequences.