Awesome On-Device AI Systems
A curated list of inference engines, vendor SDKs, and research papers for running machine learning models, including large language models, efficiently on mobile and edge devices.
This is a curated list bridging systems research and practical deployment for on-device AI, focused on optimizing models such as LLMs and vision transformers for resource-constrained mobile and edge hardware. It lists general inference engines such as LiteRT, ExecuTorch, ONNX Runtime, MNN, and NCNN; vendor-specific SDKs for Qualcomm, Apple, NVIDIA, Intel, and MediaTek hardware; and task-specific tools such as llama.cpp, MLC LLM, TensorRT-LLM, and MLX LM for running language models specifically.
The research-papers section covers LLM inference on mobile system-on-chips, mobile processor characterization and optimization, compiler-based ML optimization, attention acceleration, quantization and sparsity, application-centric on-device AI systems, multi-model and heterogeneous runtime scheduling, on-device training and model adaptation, and profiling tools. It is organized as a reference for readers moving between academic papers and the engineering frameworks that implement similar ideas.