Llama3.java
A single-file, pure Java implementation of Llama 3, 3.1 and 3.2 inference, built as a successor to llama2.java and used to test JVM compiler optimizations.
Llama3.java implements Llama 3, 3.1 and 3.2 inference in a single Java file, with no external dependencies, as the successor to an earlier llama2.java project based on Andrej Karpathy's llama2.c. Beyond its educational value, the project is also used to test and tune compiler optimizations on the JVM, particularly for the Graal compiler.
It includes a GGUF format parser, a Llama 3-compatible tokenizer, grouped-query attention inference with ad-hoc RoPE scaling for Llama 3.1 and tied word embeddings for Llama 3.2, fast matrix-vector multiplication using Java's Vector API, and support for F16, BF16, F32 weights alongside several quantization formats. It runs through jbang, as a compiled jar built with a bundled Makefile, or with GraalVM Native Image, and includes an interactive chat and instruct mode.