Inference

The first open-source implementation of NVIDIA's KVTC algorithm, compressing LLM key-value caches 6 to 9x using PCA decorrelation, adaptive quantization and entropy coding, to fit longer context in the same GPU memory.

• Inference
• Libraries

A C++ and GGUF reimplementation of the Breeze-TTS-2 bilingual English and Mandarin text-to-speech model, supporting voice design, cloning, direction, and experimental voice conversion with a Vulkan backend.

• Inference
• Speech

A dependency-free C11 implementation of Meta's Segment Anything Model 3 for image and video segmentation, with a Metal GPU backend and a multithreaded CPU backend.

• Inference

A technical guide documenting how to run local LLM and image-generation inference on an AMD RX 580 GPU through Vulkan, after AMD dropped ROCm support for that hardware generation.

• Inference
• Tutorials

A local AI load balancer that turns a set of Mac and other devices running Ollama into one cluster, auto-discovering nodes over mDNS and routing language-model, image, speech and embedding requests to the best-fit device.

• Inference
• LLM gateways

A complete neural network framework written in pure C, with no Python or CUDA toolkit dependency, that both runs and trains models, including loading and running GGUF files directly.

• Models
• Inference

A benchmark suite for local large language models on x86 hardware, measuring time-to-first-token, throughput, total duration and memory use across llama.cpp, LM Studio and Ollama backends.

• Inference
• AI evals

A Swift and Metal runtime that runs a 120-billion-parameter mixture-of-experts language model on a 16 GB Apple Silicon Mac by streaming only the experts a token actually needs from disk.

• Inference

A research inference engine that proves a full transformer forward pass can run through a single algebraic operation, exponentiation combined with a logarithm and subtraction, in place of ordinary multiplication.

• Inference
• Papers

An experimental Capacitor plugin that runs large language models entirely on-device using Apple Intelligence on iOS and Gemini Nano on Android, for text and image generation with no network requests or API keys.

• Inference
• Mobile

A GGUF language model inference engine written from scratch in Zig for Apple Silicon, with its own quantization kernels, Metal GPU shaders and parser rather than wrapping an existing runtime.

• Inference

Docker Compose configurations for serving llama.cpp GGUF language models on AMD Strix Halo hardware, covering stock Vulkan and custom ROCm builds with measured prefill, decode and quality metrics.

• Inference
• Containers

A self-hosted inference stack for a cluster of NVIDIA DGX Spark machines that routes tasks across local and cloud models, then LoRA-trains the local models on its own routing history so the local share of work grows over time.

• Models
• Inference

A native desktop interface for discovering, downloading, configuring and serving local large language models through llama.cpp, without hand-editing command-line arguments.

• Inference

Splits one large language model across a Mac and a separate Windows or Linux PC over an encrypted, framework-neutral network protocol, producing output identical to running the whole model on one machine.

• Inference

A Rust-based local AI runtime and orchestrator that synchronizes text, vision and tool execution in one memory space, with hybrid HTTP and native FFI access for building agentic workflows on edge hardware.

• Agent frameworks
• Inference

A desktop control plane for discovering, converting, running, chatting with and benchmarking open-weight language and image models entirely on your own machine.

• Inference

A collection of deployment recipes and practical guides for running the Ling model family locally with SGLang, vLLM, llama.cpp, Ollama and MLX across consumer and workstation hardware.

• Inference
• Tutorials

A compression library for embedding indexes and LLM KV caches that optimizes for the metric a downstream consumer actually uses, such as retrieval recall or generation quality, instead of reconstruction error alone.

• Inference
• Databases

A library of 43 KV-cache compression methods for MLX models on Apple Silicon, letting a locally run model use up to 16 times less cache memory with a three-line API change.

• Inference

A unified KV-cache compression toolkit for LLM inference offering twelve Python-native compression methods behind one API, with guarded add-on composition, multi-GPU deployment planning and launch-command generation for llama.cpp and vLLM.

• Inference

A scenario-based benchmark for local LLM inference on Apple Silicon that measures effective end-to-end throughput, including the prefill phase, across real workloads such as agent conversations and document classification.

• Inference
• AI evals

A free, open-source local LLM server that turns an idle Mac or GPU into a public, OpenAI-compatible inference node reachable over HTTPS, sharing spare compute with friends, apps or AI agents.

• Inference

A VL-JEPA-inspired pipeline that compresses images, text, conversations and RAG documents locally through Ollama before sending only the compact result to a cloud LLM API, cutting token costs by around 80 percent.

• Inference