Inference

A unified manager and router for llama.cpp, MLX and vLLM model instances, with a web dashboard, a built-in model downloader, and OpenAI- and Anthropic-compatible routing across multiple running models.

• Inference

A Swift library for running text-embedding models locally using Apple's MLTensor, supporting BERT, ModernBERT, NomicBERT, RoBERTa, XLM-RoBERTa and Qwen3 embedding architectures loaded directly from Hugging Face.

• RAG
• Inference

A local-first native desktop AI workspace bundling multi-provider chat, a coding agent with diff-based file edits, and other agent-driven workspaces such as an ERP builder and a 3D model generator, with no backend or telemetry.

• Coding agents
• Inference

A ready-to-run Docker Compose bundle of local AI services, including Ollama, a chat UI, an LLM gateway, embeddings, speech-to-text, text-to-speech and an MCP gateway.

• Inference
• Containers

A collection of pinned build recipes, benchmark records and failure notes for running LLM inference on Intel Arc Pro B60/B70 GPUs with vLLM and llama.cpp.

• Inference

A toolkit for running MLX models from Node.js, covering language and multimodal inference, an HTTP server, model conversion, and GRPO/SFT training, with a Rust and C++ native layer and no Python required.

• Inference
• Libraries

A from-scratch inference engine that separates the prefill and decode phases of LLM generation onto dedicated GPUs, avoiding the interference between the two phases that limits throughput when they share hardware.

• Inference

A lightweight, multi-platform chat interface for local and cloud LLMs that can run fully offline and optionally encrypts stored chat history and files.

• Inference

A terminal-native manager for local large language models that bundles a TUI, CLI, background daemon and OpenAI-compatible proxy around llama.cpp, with support for several other local inference backends.

• Inference
• Terminals

A lossless GPU-memory compression codec for AI model weights and KV caches, decoding compressed bf16 numbers on the GPU during matrix multiplication and attention so models fit in less memory.

• Inference

A community-maintained hardware plugin that lets the vLLM inference engine run on MetaX GPUs, using MetaX's CUDA-like MACA backend for near-native CUDA performance.

• Inference

A self-hosted, Kubernetes-based control plane, built on Crossplane, that provisions and schedules AI inference across any GPU infrastructure and exposes each deployed model behind one OpenAI-compatible endpoint.

• Inference
• Hosting

A plugin for the NoneBot chatbot framework that adds DeepSeek-powered conversation and question answering to a bot, with multi-turn context and Markdown-to-image rendering.

• Inference

A curated list of inference engines, vendor SDKs, and research papers for running machine learning models, including large language models, efficiently on mobile and edge devices.

• Inference
• Curated lists

A local LLM server, wrapping llama.cpp, built as a drop-in replacement for Ollama that reuses a shared prompt's KV cache across concurrent requests instead of reprocessing it each time.

• Inference

A self-hosted, ChatGPT-style web interface for vLLM and Ollama, built with Next.js, that runs entirely in the browser with chats stored in local storage.

• Inference

A macOS menu bar app that exposes Apple's on-device Foundation Models through OpenAI- and Ollama-compatible local API endpoints, so nothing leaves the machine.

• Inference
• Desktop utilities

A native macOS app for running large language models locally on Intel Macs with AMD GPUs, using a patched llama.cpp build to fix corrupted output and low speed on that hardware.

• Inference

A Discord bot that connects Ollama-hosted large language models to a Discord server, letting users chat with a locally hosted model and manage it from Discord.

• Inference

A Swift package for running LLMs such as Gemma 4, Qwen3.5 and Liquid AI's LFM2.5 on iPhone using CoreML, targeting Apple's Neural Engine so the GPU stays free for the rest of the app.

• Inference
• Mobile

A Kubernetes operator for self-hosted LLM inference across a mixed fleet of NVIDIA, AMD and Apple Silicon hardware, turning model deployment into a two-line YAML definition with GPU scheduling, caching and OpenAI-compatible endpoints.

• Inference
• Containers

An Android workbench for studying on-device AI in real scenarios such as online TV, combining llama.cpp, whisper.cpp and a customized FFmpeg to run offline LLMs and real-time transcription while watching or recording video.

• Inference
• Mobile

A Kotlin Multiplatform library for running large language models, speech-to-text and image generation fully on-device across Android, iOS, desktop and WASM, built on llama.cpp, whisper.cpp and stable-diffusion.cpp.

• Inference
• Libraries

A one-command, pinned serving stack that runs the Qwen3.8 model family on a single NVIDIA DGX Spark, with several switchable, benchmarked speed and quantization configurations.

• Inference