Inference

A serving system that lets multiple SGLang model instances share GPU memory by separating KV-cache capacity from FFN weight execution, so each model's resource needs no longer have to be reserved together.

• Inference

A hardware-agnostic LLM inference engine written in Rust that streams model weights layer by layer from NVMe storage, so a single consumer GPU can run large models that do not fit in its VRAM.

• Inference

A GPU hardware-abstraction layer for LLM inference fleets that gives hot failover between an active and standby domain at the cost of zero idle GPUs.

• Inference
• Hosting

A full-stack web chat application that lets users switch between five language models, ask questions grounded in uploaded documents through retrieval-augmented generation, search the web in real time, and analyze images, with streaming responses.

• RAG
• Inference

A Python package that compresses KV-cache and vector payloads for existing agent runtimes and retrieval pipelines, adding reranking and benchmarking without requiring a new framework.

• RAG
• Inference

Field notes recording how to cross-compile and deploy llama.cpp GPU inference on an NVIDIA DRIVE Thor automotive domain controller, including a troubleshooting guide and measured throughput at each optimization step.

• Inference
• Tutorials

A dependency-free Rust runtime for Apple's MLX framework that loads Hugging Face model checkpoints straight onto Apple Silicon GPUs for fast local inference, shipped as both a native Rust crate and a Node.js package.

• Inference

An experimental project for running small transformer checkpoints such as SmolLM 135M and Qwen2.5 0.5B directly on the first-generation AMD XDNA NPU in Ryzen AI processors, using MLIR-AIE kernels, a patched Ollama build and a native Colibri backend.

• Inference

A beginner-friendly guide and automated setup script for running the 350-million-parameter IBM Granite 4.0 language model fully offline on a Raspberry Pi 5 through Ollama, using a GGUF quantization tuned for the Pi 5's CPU.

• Inference
• Tutorials

A local LLM inference proxy built on llama.cpp and llama-swap that lets you load and hot-swap multiple GGUF models and serve them through an OpenAI-compatible HTTP API, with an outbound-connection design for secure remote access to home or office GPUs still on the roadmap.

• Inference
• Networking

A command-line tool that transcribes and summarizes local audio or video files or YouTube content entirely on Apple Silicon using MLX, picking Voxtral or Whisper for transcription and a Qwen3 model for summarization based on the Mac's available memory.

• Inference
• Speech

An out-of-tree vLLM plugin that adds Mobilint NPU support, so Mobilint-optimized language and vision-language models can be served through ordinary vLLM commands.

• Inference

An evidence-first research project benchmarking exact-attention latency and memory on consumer NVIDIA GPUs, working toward a hybrid attention operator.

• Inference

A React Native example app demonstrating NobodyWho, a library for running LLMs locally and efficiently on a phone, including chat, tool calling, vision, RAG, speech-to-text and text-to-speech.

• Inference
• Tutorials

A portable, self-contained package that bundles Ollama, Hollama and Caddy into one folder, giving a no-install web chat interface for running local LLMs on Windows or macOS.

• Inference

A command-line local AI model manager for macOS and Linux that stores models once in the Hugging Face cache and exposes them to tools such as LM Studio, llama.cpp, Jan and ComfyUI through symlinks, avoiding duplicate downloads.

• Inference
• CLI tools

A .NET library for on-demand local AI inference that ships with no bundled models, downloading and caching embeddings, generation, vision, OCR, translation, transcription, speech-synthesis and image-generation models the first time they are used.

• Inference
• Libraries

A byte-identical Rust port of kimi-k3-in-c that runs CPU-only inference for the 2.78-trillion-parameter Kimi K3 mixture-of-experts model, with no GPU, BLAS or PyTorch, in an 8 GB memory floor.

• Inference

A vLLM plugin that replaces a language model's dense classification head with a two-stage clustered retrieval pipeline, cutting scored vocabulary tokens roughly tenfold for faster, training-free inference.

• Inference

A Triton-based operator library providing optimized vLLM-specific kernels such as MoE routing and fused expert operators, used alongside the general FlagGems operator library for faster LLM inference.

• Inference

An MLX port of training-free inference speedups for Dream and DiffusionGemma diffusion language models on Apple Silicon, with dual-cache decoding and benchmark scripts.

• Inference

A local web chat interface that connects LM Studio and other OpenAI-compatible model providers to an in-app tool runtime and the Supertonic text-to-speech engine, with hybrid memory search across conversations.

• Inference
• Speech

A mixture-of-experts model compression toolkit that prunes, merges and quantizes large MoE language models into smaller specialist models for local inference, implementing REAP expert pruning with several scoring strategies and a self-contained evaluation harness.

• Models
• Inference

A fork of vLLM tuned to run mixture-of-experts models several times larger than the available memory on a single NVIDIA GB10 machine (DGX Spark, ASUS Ascent GX10), using 2-bit expert planes with hand-written SASS kernels, an FP4 quality-recovery tier and speculative decoding.

• Inference