Inference

An optimized SGLang-derived inference runtime tuned for running Qwen3.8 models on a single NVIDIA RTX PRO 6000 Blackwell GPU, with prebuilt launch configurations, a long context window, and disk-backed caching.

• Inference

A self-hosted AI workspace that chats with more than 300 models across nine providers using your own API keys, with injectable agent skills and self-drafting agents, and no ENZO account or subscription.

• Inference
• LLM gateways

A native Rust inference server for running open models on NVIDIA GPUs and Apple Silicon, with OpenAI- and Anthropic-compatible APIs and a bundled Studio for comparing models.

• Inference

A vLLM plugin, built on the FlagOS multi-chip backend, that lets the same vLLM commands run model inference across many different AI chip vendors without changing vLLM's interfaces.

• Inference

A mobile app for Android and iOS that runs language models directly on the device and turns the phone into a local inference server, with optional cloud model support.

• Inference
• Mobile

A single self-compiling binary that runs an AI assistant at home, with chat, persistent memory, tool use, and end-to-end encrypted remote access, on top of a self-built llama.cpp.

• Inference

A CPU-first LLM inference server built around ternary weight models, running an 8-billion-parameter model in about 2.2 GB of memory on a free-tier ARM cloud instance.

• Inference

A local-first AI runtime and studio for macOS and Windows that runs open text, vision, image, and voice models entirely on-device behind one OpenAI-compatible API.

• Inference
• Generative media

A browser-based calculator that finds which hardware limit actually constrains an on-premises LLM deployment, then prices and compares sizing options against it.

• Inference

A key-value cache compression library for MLX on Apple Silicon that quantizes cached keys and values to as low as 3 bits, using mixed precision to keep decode quality unchanged.

• Inference

A fork of llama.cpp tuned specifically for NVIDIA Ampere consumer GPUs such as the RTX 3090, adding a custom KV cache format and speculative decoding for large context windows.

• Inference

A Swift SDK that downloads, caches, and runs Apple's Core AI models for chat, vision, and speech in an iPhone or Mac app, without a Python conversion step or bundled weights.

• Inference
• Mobile

A native desktop chat application, built with Rust, Tauri, and Candle, for running large language models locally with no cloud connection or account.

• Inference

A research system, presented at OSDI 2025, for running large language model inference on wafer-scale AI accelerators instead of GPU clusters.

• Inference
• Papers

The official implementation of a speculative-decoding method that pairs a large language model with an adaptively trained diffusion drafter model to speed up inference.

• Inference
• Papers

A Swift and Metal inference runtime for Apple Silicon that runs large mixture-of-experts language models by streaming only the active experts from disk instead of holding the whole model in memory.

• Inference

A tool that turns scanned PDFs and images into searchable documents by reading them with a locally run vision language model instead of traditional OCR.

• RAG
• Inference

A serving runtime for robotics vision-language-action models that runs custom fused kernels on NVIDIA Jetson Thor and desktop GPUs for faster real-time prediction.

• Inference

A Dart implementation of llama.cpp that lets Flutter and Dart applications run local language models, used by the Maid mobile AI app.

• Inference
• Mobile

An SDK and WebGPU runtime, built in Rust and C++, for running GGUF language models locally in the browser, on desktop, or through a cloud gateway, behind one API.

• Inference
• Libraries

A training-free, wavefront-parallel decoding framework for block-wise diffusion language models letting a downstream block start from an informative draft instead of waiting on its predecessor, reporting higher throughput at similar accuracy on unmodified LLaDA-2 weights.

• Inference

A research framework that treats speculative decoding as an online learning problem, continuously updating the draft model from feedback produced during verification to keep speedups high as a workload shifts.

• Inference

An unofficial project that runs Japan's Digital Agency GENAI government AI entirely locally, replacing cloud dependencies with SAML authentication through Keycloak, an OpenAI-compatible backend, RAG through Qdrant, and local transcription and image generation.

• RAG
• Inference

A high-performance Rust proxy that queues requests from multiple users in front of one or more Ollama or LM Studio instances, dispatching them in parallel with a least-connections, round-robin, model-aware scheduler and a real-time terminal dashboard.

• Inference
• LLM gateways