Inference

An on-device inference engine for running LLMs locally and efficiently on any device, with bindings for Godot, Kotlin, Swift, React Native, Flutter and Python.

• Inference
• Libraries

A JavaScript SDK that runs large language models, speech recognition, text-to-speech and audio separation directly inside the browser using WebGPU, with no server required.

• Inference
• Speech

An open-source hardware guide, with parts lists, CAD and assembly photos, for building your own local machine to run large language models instead of renting inference from a cloud provider.

• Inference
• Tutorials

A Core ML and Apple Neural Engine port of the Laya typed-decision model, which answers structured choice, ordinal-score and yes/no questions in a few milliseconds without generating text tokens.

• Models
• Inference

A VS Code extension that adds local, LLM-assisted code completion, chat and an agentic coding assistant backed by llama.cpp.

• Inference
• Editors

An inference and quantization library for running large language models locally on consumer GPUs, with its own EXL3 quantization format and flexible parallel inference.

• Inference

An SLO-aware benchmarking platform that simulates realistic workloads against OpenAI-compatible and vLLM-native servers to measure LLM deployment latency, throughput and capacity limits.

• Inference
• AI evals

A command-line tool for downloading and running open-source language models such as GLM, DeepSeek and Qwen locally, or on hosted infrastructure when a job needs more capacity.

• Inference

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for Chinese AI accelerators and donated to the OpenAtom Foundation.

• Inference

A vLLM-style inference server for Apple Silicon that adds continuous batching, paged and prefix KV caching, and both OpenAI and Anthropic-compatible APIs in one process.

• Inference

An Intel post-training quantization toolkit that compresses large language and vision-language models to two to four bits with minimal accuracy loss, using sign-gradient descent.

• Models
• Inference

A local desktop and mobile app that runs open-weight LLMs offline and exposes them through a drop-in, OpenAI-compatible local API for agents, CLIs and IDE plugins.

• Inference

A beginner-friendly GNOME desktop client for chatting with local Ollama models or third-party providers such as Gemini and ChatGPT, with image and document recognition.

• Inference

A native Zig inference server for Apple Silicon that serves MLX and GGUF models behind OpenAI- and Anthropic-compatible APIs, alongside image, video, speech and 3D generation.

• Inference
• Generative media

An unofficial, community desktop client for chatting with Claude and other LLM providers, built with Tauri, Rust and React, with local chat history, MCP tools and multi-agent orchestration.

• Agent frameworks
• Inference

A patched vLLM serving setup and requantization pipeline that runs a large Qwen model on a single consumer GPU, with an OpenAI-compatible API in single-user or batch modes.

• Inference

An on-device AI inference library for React Native, built on Meta's ExecuTorch runtime, that runs machine learning models on a phone with no network calls.

• Inference
• Mobile

An open-source, continuously updated benchmark research platform that tracks real-world inference performance of popular open-source serving frameworks such as vLLM, SGLang and TensorRT-LLM across current GPU hardware and models.

• Inference
• AI evals

A community-maintained vLLM hardware plugin that runs high-performance LLM inference on Apple Silicon Macs using MLX as the compute backend.

• Inference

A KV-cache compression method for LLM inference that quantizes cached keys to 3 bits and values to 2 bits using Triton kernels, integrated with vLLM to cut memory use and raise usable context length.

• Inference

A local-first control panel for running and managing self-hosted LLM backends such as vLLM, SGLang, llama.cpp and ExLlamaV3, with GPU monitoring, an OpenAI-compatible chat proxy, and agent sessions against local or remote controllers.

• Inference

A local-first, self-hostable AI chat workspace that brings multi-model chat, an agent mode for multi-step tool workflows, and deep research with reviewable plans into one app, keeping conversation history in the browser by default.

• Agent frameworks
• Inference

A cross-platform client app for Ollama that connects to an existing Ollama server over its API, without hosting a model server on the device itself.

• Inference

A high-performance, cross-language inference engine that runs AI models directly inside an app with low latency, full data privacy and no per-inference API cost, using unified memory on Apple devices.

• Inference