Apple Silicon LLM Benchmark
Reproducible on-device LLM benchmarks for Apple Silicon, comparing Apple Core AI, MLX, LiteRT-LM and Core ML on the same models and harness, with every result tied to its quantization and capture session.
This project runs the same models through Apple Core AI, MLX, LiteRT-LM and Core ML on Apple Silicon devices such as iPhone 17 Pro and an M4 Max Mac Studio, and publishes decode tokens-per-second, prefill, time-to-first-token, memory use and GSM8K accuracy for each combination. A fixed harness runs the same prompt and token budget for every runtime under test, logging results as per-run JSONL files, and the leaderboard explicitly refuses to print cross-runtime ratios, since device state moves between capture sessions and a fair comparison only holds within one session.
Coverage includes hybrid Mamba-2 models such as Nemotron-3 Nano, Granite-4.0-H and Falcon-H1 alongside dense models like Qwen 3. The project documents Apple's Core AI runtime in iOS 27 and macOS 27, which exports PyTorch models into GPU- or Neural-Engine-executable bundles as a successor to Core ML.
Every published number states its quantization and which capture session it came from, and results and raw data are committed to the repository.