Inference

An open local runtime that combines native inference, speculative decoding and agentic caching so useful local AI agents run faster automatically, and can also guard an agent a user already runs elsewhere.

• Agent frameworks
• Inference

A Swift and MLX implementation of the GLiNER2 information-extraction model, running named entity recognition, text classification, structured JSON extraction and relation extraction entirely on-device on Apple Silicon.

• Inference
• Libraries

An Android app for running large language models entirely on-device, so a downloaded model can be used to chat fully offline with no API key, server, or data leaving the phone.

• Inference
• Mobile

A browser-based control panel for downloading and running large language models with vLLM or llama.cpp, rating each setting against the user's own hardware so a model launches without hours of trial and error.

• Inference

A toolkit that converts a dense Gemma-3-270m model into a mixture-of-experts model, then trains, inspects and serves it on CPU through llama.cpp.

• Models
• Inference

A full-screen terminal chat interface for OpenAI-compatible AI APIs, with character cards, personas, presets and early support for the Model Context Protocol.

• Inference

A native macOS app and inference engine that runs large mixture-of-experts language models on Apple Silicon by streaming only the needed experts from SSD instead of holding a whole model in memory.

• Inference

A minimal, hand-written LLM inference engine for the MiniCPM5-1B model, built in Rust to study how transformer inference works, with a terminal UI that visualizes generation as it happens.

• Inference
• Tutorials

A pure-Rust inference engine for transformer models on Apple Silicon, paired with a native macOS app for training, quantizing and chatting with local models.

• Inference

An LLM inference engine built from scratch in C++ and CUDA for a single GPU model, the NVIDIA RTX 5090, serving models over both OpenAI and Anthropic-style APIs.

• Inference

A community-contributed benchmark database of language-model speed and code-quality results measured on Apple Silicon Macs, organized by chip generation and configuration so someone can look up expected performance before choosing a model.

• Inference
• AI evals

A free, browser-based calculator, in Korean and English, that estimates which local AI models a given GPU can run, or which GPUs a chosen model needs, along with quantization and speed estimates, with no install, sign-up or server-side processing.

• Inference

A drop-in key-value cache compressor for Hugging Face language models that quantizes the cache to 4 bits, cutting its VRAM footprint and restoring inference speed once long-context generation would otherwise start thrashing.

• Inference

A pure C, single-binary inference engine that runs BitNet ternary models and dense GGUF language models on ordinary CPUs, reported several times faster than Microsoft's own bitnet.cpp and a vendor's own llama.cpp fork on the same hardware.

• Inference

A Rust inference engine that runs a 28.9-million-parameter language model on an ESP32-S3 microcontroller, ported from a C reference implementation and benchmarked to run faster than it.

• Inference

A reproducible vLLM Docker image built specifically for the NVIDIA DGX Spark's GB10 chip, with every component in the stack pinned by commit or digest so the same version file always rebuilds the same image.

• Inference
• Containers

A header-only C++ wrapper around GGML that simplifies loading GGUF or safetensors models, building computation graphs, and running them on CPU or GPU.

• Inference
• Libraries

A small language-model inference engine for consumer GPUs that implements continuous batching, prefix caching and per-request KV-cache control, invoked directly from Python in a few lines of code.

• Inference

A drop-in vLLM plugin implementing Google's TurboQuant key-value cache quantization method, compressing the KV cache with asymmetric key/value bit-widths and validated across eight language models.

• Inference

Native Windows port of the NInfer C++/CUDA inference engine, built for a single RTX 5090 GPU with no WSL2 or Docker, serving text, image and video prompts through a local CLI or OpenAI/Anthropic-compatible HTTP APIs.

• Inference

A reproducible deployment bundle for running a distilled Gemma-based language model on a Xilinx Kria KV260 FPGA board, including the Verilog hardware sources, bitstream, tokenizer and integrity checks needed for edge inference.

• Inference

A KV-cache compression library for large language models on Apple Silicon that uses a polar-coordinate quantization scheme to shrink memory use by up to five times while preserving accuracy.

• Inference
• Libraries

A minimal Compose Multiplatform sample showing how to call on-device generative AI from shared Kotlin code, using Apple's Foundation Models framework on iOS and Google's ML Kit Prompt API on Android.

• Inference
• Mobile

Safe Rust bindings to llama.cpp that track the upstream project closely, exposing a high-level API alongside raw bindgen bindings, with example programs for chat, embeddings and an OpenAI-compatible server.

• Inference
• Libraries