Inference

A minimal, zero-dependency LLM inference engine in pure C11 for GGUF models, with NEON and AVX2 SIMD CPU backends, mixture-of-experts routing, TurboQuant KV-cache compression, and a WebAssembly build.

• Inference

A patched, native-Windows build of vLLM adding three Windows-specific fixes, a Qwen3 reasoning parser and tool-calling backports, published as prebuilt wheels for both Ampere/Ada and Blackwell RTX GPUs.

• Inference

Experimental Rust codecs for vector quantization, including PolarQuant, TurboQuant and QJL, producing derived compressed representations for semantic-search candidate generation and KV-cache compression experiments.

• Inference
• Databases

A memory-aware GGUF inference runtime in Rust that runs quantized language models across CPU, NVIDIA CUDA, Apple Metal and Android without expanding weights into oversized resident copies.

• Inference

An all-in-one local inference platform that computes each model's launch flags to fit a user's VRAM, hot-swaps between chat, image and voice models, and serves them behind one OpenAI- and Anthropic-compatible API.

• Inference

A now-concluded research project that profiled a self-built NVFP4 LLM inference engine against llama.cpp and vLLM, found the custom engine was not the leverage point, and folded its findings into upstream contributions.

• Inference

A free textbook on building an LLM inference stack from scratch in Rust, covering the driver interface, attention kernels, a paged KV cache, a scheduler, multi-GPU collectives and a quantisation pipeline.

• Inference
• Books

A dependency-free WebGPU runtime for running 1-bit, binary-weight large language models directly in the browser, with GPU-resident decoding, streaming output and cross-turn KV-cache reuse.

• Inference

An SGLang Docker image, with several out-of-tree patches, for running LLM inference on AMD's consumer Strix Halo APU (Ryzen AI Max+ 395 / Radeon 8060S), a platform AMD's own official images do not target.

• Inference
• Containers

A command-line tool for setting up and running local AI models on AMD GPUs, shipped as a single prebuilt binary with a full-screen TUI dashboard for GPU telemetry, model serving and chat.

• Inference
• CLI tools

A generative-AI-powered application framework for large-scale data management and AI model serving, connecting compute services across distributed locations through virtual workspaces and remote procedure calls.

• Inference
• Backend

A peer-to-peer network of ordinary machines that runs large language models across the network so a prompt is served without a central operator, encrypted so it never leaves the sender's device in clear text.

• Inference

A local, offline prompt-engineering studio that turns rough ideas into structured prompts using self-hosted Gemma models through Ollama, with no cloud API, account or network access required after setup.

• Inference

A distributed, peer-to-peer network for LLM inference where every peer both serves and consumes completions, metered by a BitTorrent-style upload-to-download ratio instead of money, brokered by a coordinator that never sees the request content.

• Inference

A terminal UI around llama.cpp for running, managing and benchmarking local GGUF models, with a setup wizard tuned for iGPU-class hardware and a launcher for the pi coding agent against the local server.

• Inference

A browser-based and command-line calculator that estimates VRAM and RAM requirements for a GGUF model hosted on Hugging Face, without downloading the model.

• Inference

A free, open-source macOS app for browsing, downloading and chatting with open-source AI models entirely on-device using MLX Swift, with no cloud calls, API keys or subscriptions.

• Inference

A Swift package that runs Google's Gemma 4 model on iPhone through the LiteRT-LM runtime on the Metal GPU, with text, image and audio input, and a second mode that plugs the same runtime into Apple's Foundation Models API as a custom backend.

• Inference

A collection of 182 machine-learning model families spanning language, vision, speech and audio, ported as standalone Rust crates on top of the rlx multi-backend compiler and runtime, targeting CPU, Metal, MLX, CUDA and wgpu.

• Inference

A Qwen-specialized fork of MLX adding disk-backed KV-cache checkpointing for hybrid attention models, letting a 122-billion-parameter model stay responsive across long conversations on Apple Silicon.

• Inference

An offline, Windows-only local LLM coding assistant that bundles llama.cpp, Python and Git into one folder, indexing a project with a local RAG search and code graph to answer questions and propose file edits without cloud access.

• Coding agents
• Inference

This project packages Ollama as a serverless worker for the RunPod.io platform, downloading a named model automatically on startup and serving both native Ollama and OpenAI-compatible streaming requests.

• Inference
• Hosting

gmlx is a local inference platform for running K-quant and IQ-quant GGUF models on Apple Silicon, serving them over OpenAI- and Anthropic-compatible APIs, with terminal or browser chat, voice input, a local RAG stack and LoRA fine-tuning.

• RAG
• Inference

Ghostlink is a self-hosted, open-source distributed inference fabric that discovers other machines on a LAN and splits a language model's weights across their CPU, GPU and NPU resources, serving the result behind an OpenAI-compatible API with no manual configuration.

• Inference