picoLLM Inference Engine
A cross-platform SDK for running compressed large language models fully on-device, using a quantization algorithm that learns an optimal per-weight bit allocation instead of a fixed scheme.
picoLLM Inference Engine is a cross-platform SDK for running compressed large language models, with all inference happening locally rather than over a network. It runs on Linux, macOS, Windows, Raspberry Pi, Android, iOS and in the browser, on both CPU and GPU, and is free to use with open-weight models.
Its picoLLM Compression algorithm learns an optimal bit-allocation strategy across and within a model's weights for a given task-specific cost function, rather than using a fixed bit-allocation scheme like existing quantization techniques. The vendor reports that this recovers 91%, 99% and 100% of the MMLU score degradation that GPTQ shows at 2-, 3- and 4-bit settings respectively, on Llama-3-8B.
It ships SDKs for Python, .NET, Node.js, Android, iOS, Web and C, and supports open-weight models including several Gemma, Llama-2, Llama-3 and DeepSeek-OCR variants, downloadable from the Picovoice Console.