Inference

Microsoft's end-to-end local AI SDK for shipping on-device AI features, with native SDKs for C#, JavaScript, Python, and Rust, automatic hardware acceleration, and an OpenAI-compatible API.

• Inference

A reference implementation for deploying vLLM at cluster scale on Kubernetes, adding request routing, KV-cache offloading, and observability without changing application code.

• Inference
• Containers

A storage engine, written in Rust on io_uring, for AI infrastructure that keeps GPUs fed by offloading and serving KV cache, vector indexes and model checkpoints with high-throughput, low-latency I/O.

• Inference
• Databases

A collection of example iOS and macOS applications and command-line tools built with MLX Swift, showing how to train and run language models, vision-language models and other machine learning workloads on Apple hardware.

• Inference
• Tutorials

A library for fine-tuning, evaluating, and running personalized, private open-source large language models locally or in a private cloud.

• Models
• Inference

A library that unifies training, inference, and evaluation of diffusion language models, aiming to make that pipeline transparent and reproducible.

• Models
• Inference

A free, open-source Android app for chatting with LLMs, running GGUF models locally via llama.cpp or connecting to remote providers such as Anthropic, DeepSeek, Ollama, and OpenAI.

• Inference

A speculative-inference server with custom GPU kernels tuned for running large language models faster on consumer and heterogeneous hardware.

• Inference

A community-maintained hardware plugin that runs the vLLM inference engine on Huawei Ascend NPUs.

• Inference

A ComfyUI plugin for Nunchaku, an inference engine that runs 4-bit quantized diffusion models such as Qwen-Image and FLUX with much lower VRAM use.

• Inference
• Generative media

A PyTorch-native library for quantizing and optimizing models across the training-to-serving lifecycle, covering low-bit formats, sparsity, and hardware-aware kernels.

• Inference
• Libraries

A high-performance inference framework for large language models that targets efficiency, flexibility, and support for multiple hardware vendors, from single GPUs to large clusters.

• Inference

An inference engine that links multiple home computers into a cluster over Ethernet to run large language models faster by splitting the work with tensor parallelism.

• Inference

A local demo application for running the Bonsai 2 27B language model, a heavily compressed reasoning model with vision and tool-calling support, on a laptop or single GPU.

• Models
• Inference

A cross-platform on-device AI app for phone and Mac that runs GGUF language models, vision, Whisper speech-to-text and Stable Diffusion image generation entirely offline, with tool calling.

• Inference
• Speech
• Generative media

A native macOS app for chatting with a local large language model that can answer using files, folders and websites on the Mac, entirely offline, with an optional bring-your-own-key mode.

• RAG
• Inference

A self-hosted, open-source inference engine that serves every model an AI agent needs, such as search, document parsing, structured output and safety, from one OpenAI-compatible cluster.

• Inference

A small local server that runs open-weight AI models on an Apple Silicon Mac's own chip and answers in Anthropic's API format, so Claude Code can work with no cloud model and no API key.

• Coding agents
• Inference

A distributed inference system that pools GPU and memory across machines into one OpenAI-compatible API, routing or splitting models across peers as needed.

• Inference

A Hugging Face library that optimizes and accelerates Transformers, Diffusers, TIMM and Sentence Transformers models for training and inference on specific hardware.

• Inference
• Libraries

FastDeploy is a PaddlePaddle-based inference and deployment toolkit for large language models and vision-language models, with an OpenAI-compatible API service and support for several accelerator families.

• Inference

A library for quantizing and compressing large language models for deployment with vLLM, covering weight, activation, KV-cache and attention quantization with Hugging Face integration.

• Inference
• Libraries

A multi-LoRA inference server that serves thousands of fine-tuned language models from a single GPU by dynamically loading LoRA adapters onto one shared base model.

• Inference

An open-source OpenAI- and Anthropic-compatible LLM inference server and Mac app for Apple Silicon, built on MLX and focused on reliable tool calling for coding agents.

• Coding agents
• Inference