Inference

A command-line tool, OpenAI-compatible server, and chat REPL that expose the language model already built into Apple Intelligence on Apple Silicon Macs.

• Inference
• CLI tools

A native Apple Silicon MLX runtime for Laya's typed decision models, answering short structured questions in single-digit milliseconds with no text generation.

• Inference

A kernel library and generator for LLM inference that provides unified, high-performance attention, GEMM, and mixture-of-experts kernels across GPU backends.

• Inference

Google's open-source orchestration layer for running large language models on edge devices, built for cross-platform, hardware-accelerated on-device inference.

• Inference

A KV-cache-centric, disaggregated serving architecture for large language models, developed as the platform behind Moonshot AI's Kimi service.

• Inference

A command-line tool that detects a machine's GPU, CPU, and RAM, then ranks which local language models actually run well on it using live benchmark data.

• Inference

A research toolkit that compresses LLM prompts and key-value caches by removing non-essential tokens, reporting up to 20x compression with little quality loss.

• Inference
• Libraries

A custom Swift and Metal runtime that runs the 26-billion-parameter Gemma 4 26B-A4B model in about 2 GB of RAM on any Apple Silicon Mac, including 8 GB models.

• Inference

A self-hosted deployment system that installs and wires together a private AI server, combining local model inference, a chat web UI, automation, and search.

• Inference
• Hosting

An inference framework built on vLLM that extends serving to omni-modality models covering text, image, audio, and video generation and understanding.

• Inference

A Python package for generating text with and fine-tuning large language models on Apple silicon using Apple's MLX framework.

• Models
• Inference

A pure C/C++ implementation, based on ggml, for running diffusion image, image-editing and video generation models such as Stable Diffusion, Flux, Wan and Qwen Image on the CPU or GPU with no external dependencies.

• Inference
• Generative media

A fast, flexible LLM inference engine in Rust, with an agentic runtime, OpenAI- and Anthropic-compatible serving, and Rust and Python SDKs.

• Inference

A native Android chat client that lets a user switch between multiple LLM providers, with Material You design, a workspace agent environment and MCP support.

• Inference
• Mobile

A toolkit for compressing, deploying and serving large language and vision-language models, with a TurboMind inference engine and PyTorch-based engine support.

• Inference

An open-source, datacenter-scale inference orchestration layer that coordinates SGLang, TensorRT-LLM or vLLM across multiple GPUs and nodes with disaggregated serving and KV-aware routing.

• Inference

A library for building a semantic cache of LLM responses, so repeated or similar queries are served from cache instead of the model, cutting API cost and latency.

• Inference

An open-source browser extension that adds a sidebar and web UI for interacting with locally running AI models from any webpage, including chatting about the page's own content.

• Inference

An on-device generative AI inference runtime for Qualcomm Snapdragon devices, running GGUF or pre-compiled models on the Hexagon NPU, Adreno GPU or CPU.

• Inference

A free, open-source mobile app that runs language models entirely on-device, so chats, voice and tool use never leave the phone.

• Inference
• Mobile

A PyTorch library that enables accessible large language models through 8-bit and 4-bit quantization, cutting memory use for inference and training.

• Models
• Inference

A portable C99 inference engine that runs the 2.78-trillion-parameter Kimi K3 model on a single CPU with no GPU, BLAS library or ML framework.

• Inference

A Python library for building online model-serving systems, turning inference scripts into REST APIs with Docker packaging and serving optimizations.

• Inference

A cross-platform desktop app for organizing multiple AI chat sessions side by side, with prompt management, session isolation and a CLI controllable from coding agents.

• Inference