Inference

An AI-native distributed data plane in Rust that unifies high-performance RPC, KV cache sharing, an elastic message queue and S3-compatible file acceleration behind three standardized interfaces, aimed at AI training and inference bottlenecks.

• Inference
• Data engineering

A FastAPI and Vue local web UI wrapper around the DeepSeek-OCR model, letting a user upload a PDF or image, watch inference progress, and inspect the extracted text with bounding-box overlays, including math formulas and tables.

• Inference

Deployment scripts for running vLLM across one to many NVIDIA DGX Spark systems, connected directly by cable or through a switched fabric, with tensor parallelism that scales automatically with the cluster size and more than 40 tested model presets.

• Inference

A plug-and-play token-compression framework for video LLMs that dynamically allocates a frame token budget, cutting a model such as LLaVA-OV's generation time by over 70 percent while keeping most accuracy, unifying several published baselines.

• Inference
• Papers

A diffusion language model that decodes many tokens in parallel by revising its own earlier predictions, aiming to preserve output quality at high decoding parallelism.

• Models
• Inference

A set of runtime patches applied to vLLM in memory, without forking or rebuilding it, that speed up inference for the Qwen3.6 and Gemma 4 model families on consumer NVIDIA GPUs with 24GB or more of memory.

• Inference

A weight-streaming layer for Apple's MLX framework that runs language models larger than a Mac's RAM by loading weights from SSD as each layer executes, without quantizing them.

• Inference

A lightweight native C++ inference runtime for NVIDIA's Nemotron speech model family, covering streaming speech recognition, speaker diarization, speech translation and text-to-speech, powered by ggml with broad hardware support.

• Inference
• Speech

A public, symptom-first registry of LLM serving-path bugs that silently produce wrong benchmark numbers, covering chat templates, tool parsers, quantization kernels and eval harnesses, each entry with a reproduction status and the check that catches it.

• Inference
• AI evals

A macOS app that runs open-source large language models locally through llama.cpp, offering both an OpenAI-compatible API for developers and a built-in chat UI for non-technical users.

• Inference

A cross-platform quantization engine for PyTorch models that reduces memory use and increases inference and training speed on GPUs, NPUs, CPUs and ARM devices.

• Models
• Inference

An open handbook on capacity planning for LLM inference: given a fixed amount of GPU compute, how to calculate the request throughput a model can sustain and its P95 latency.

• Inference
• Books

An alpha-stage desktop toolkit that manages local llama.cpp server instances with a rich chat UI, voice, retrieval-augmented generation, sub-agents and an OpenAI-compatible proxy.

• Inference

An OpenAI-compatible inference proxy that offloads mixture-of-experts model layers so 100B-plus parameter models such as DeepSeek, Qwen and Mixtral can run on consumer GPUs with as little as 8 to 11 GB of VRAM.

• Inference

A tree-based speculative decoding method for Apple Silicon that extends the DFlash draft model with a draft tree, verifying several candidate continuations in one pass.

• Inference

An agent harness that compiles a model's whole forward pass into a single, provably correct, self-tuning CUDA megakernel that beats cuBLAS at batch-1 LLM decode on several GPUs.

• Coding agents
• Inference

A heterogeneous prefill/decode inference bridge for DeepSeek-V4-Flash that runs prefill on NVIDIA GPUs with vLLM and decode on Apple Silicon with oMLX, connected over ordinary networking.

• Inference

A discrete-event simulator for modern LLM serving systems, used to study parallelism, runtime optimizations and disaggregated serving without deploying to real GPU clusters.

• Inference

Hand-written CUDA kernels that run NVFP4 and FP8 quantized weights for the Qwen3.8 model family on older Tesla V100 GPUs, serving them through an OpenAI-compatible endpoint.

• Inference

An Android app for running large language models locally on-device, with document Q&A, vision input, tool use and background model downloads.

• Inference
• Mobile

A Python command-line tool that downloads models from the Ollama library without needing Ollama installed.

• Inference
• CLI tools

An Android app that runs large language models on-device using Google's LiteRT-LM runtime, with GPU acceleration, cloud-provider integration, persistent memory and offline voice interaction.

• Inference
• Mobile

A text-first command-line tool for managing and running MLX models on Apple Silicon, covering listing, cloning, converting, quantizing and serving, with curated support for select vision and audio models.

• Inference
• CLI tools

A reference implementation of a ternary-weight tiny language model with a bit-exact Python-to-C99 inference engine, sized to run inside a microcontroller's RAM budget.

• Models
• Inference