Inference

An independent research branch of llama.cpp implementing DualDeadline, a component-staged expert-prefetching technique for offloaded mixture-of-experts inference, letting a GPU claim only router-selected expert tensor slices while routing and outputs stay exact.

• Inference
• Papers

A research runtime, written in Rust, that runs mixture-of-experts language models entirely on an Apple Silicon CPU, with hand-written NEON and SME kernels and no GPU code path, benchmarked against llama.cpp with corrected, re-measured comparisons.

• Inference

A narrow, self-contained inference engine for two specific Qwen models, Qwen3.6-35B-A3B and Qwen3.8-27B, with Vulkan and Metal graph executors tuned for the AMD BC-250 and Apple Silicon, plus a built-in coding agent and HTTP server.

• Coding agents
• Inference

A self-hosted LLM inference engine written in Rust that runs GGUF models with an OpenAI- and Ollama-compatible API, part of a wider open-source platform for building and distributing EU-based, verticalized language models.

• Inference

A native macOS chat client for on-device AI, built on Apple Intelligence through the apfel engine, with speech input and output, image analysis through Apple Vision, and local conversation history, requiring no API keys or network access.

• Inference

A Python library that compiles neural-network graphs directly into programs for Apple's Neural Engine, bypassing Core ML so that training, not just inference, can run on the chip.

• Inference
• Libraries

A Kubernetes-native checkpoint and restore system for NVIDIA GPU workloads. It saves a fully initialized pod's process and GPU memory, then restores that state on another node so new replicas skip cold start.

• Inference
• Containers

A shape-specialized inference setup that runs the Qwen3.8 Flash Next model across two AMD Radeon AI PRO R9700 GPUs, combining a pinned vLLM fork, GGUF loading and speculative decoding.

• Inference

A native iOS and iPadOS app that runs GGUF language and vision models fully on-device through llama.cpp, or connects to any OpenAI-compatible cloud API, with keyboard shortcuts for quick replies.

• Inference

A single web page that runs several AI models entirely inside the browser using hand-written WebGPU kernels, offering chat, image generation, document question-answering and OCR.

• Inference
• Generative media

A community-maintained hardware plugin that adds an Intel Gaudi accelerator backend to vLLM, so large language models can be served on Gaudi hardware through vLLM's existing serving stack.

• Inference

A framework for running neural network models in the browser, React Native and Electron on the user's own GPU, compiling its kernels to WebGPU shaders at runtime instead of a WASM binary.

• Inference
• Libraries

A 70 megabyte Docker image that runs Ollama on CPU only, built to be far smaller and faster to deploy than the official four-gigabyte Ollama image on systems without a GPU.

• Inference
• Containers

A zero-dependency C runtime for running massive open-weight Mixture-of-Experts language models on consumer hardware by streaming expert weights across SSD, RAM and VRAM as one memory hierarchy.

• Inference

An open-source command-line tool that unifies Ollama, vLLM and ComfyUI behind one endpoint on NVIDIA hardware, for running a private local inference stack without per-token cloud costs.

• Inference
• Hosting

An iPhone and iPad app that runs Alibaba's Qwen3.5 models locally on the device using Apple's MLX framework, with optional tool use, image input and web search.

• Inference

A single local executable that runs and manages AI models across every GPU on a machine and turns them into a citation-backed search engine over a user's own files, notes, code and the web.

• RAG
• Inference

A dependency-free Python package for reading, writing and dequantizing GGUF model files, with a bundled graphical and command-line interface for chatting with a loaded model.

• Inference
• CLI tools

A GGUF model inference engine written entirely in Rust, with quantized kernels for CPU, Apple Metal and CUDA, and an OpenAI-compatible server benchmarked directly against llama.cpp.

• Inference

A framework that lowers the cost of processing a dataset with large language models by deciding per record whether a cheap or an expensive model is used, guaranteeing a target accuracy.

• Inference
• Data engineering

A native C and Metal inference engine that runs large mixture-of-experts language models, such as a 284-billion-parameter DeepSeek model, on a single Apple Silicon Mac by streaming quantized experts from disk instead of holding everything in memory.

• Inference

Rust bindings for Meta's ExecuTorch library, letting PyTorch models exported in Python be loaded and run on mobile, embedded and edge devices from Rust code.

• Inference
• Mobile

A native macOS and iOS app that runs open-weight language models far larger than the device's memory by streaming each layer's weights from an external SSD within a memory budget the user sets, entirely on-device.

• Inference

A portable local AI environment that runs offline from a USB drive or hard disk, providing text chat, image generation and text-to-speech without installing anything on the host machine.

• Inference
• Generative media