Inference

A self-hosted platform for running an agentic software development lifecycle against local LLMs, using rootless Podman or Docker containers with HashiCorp Vault and mandatory GPG-signed commits.

• Coding agents
• Inference

A CUDA kernel implementation that runs the Qwen3.6-27B language model on four Tesla V100 GPUs at up to 366 tokens per second, using hand-written NVFP4 kernels and speculative decoding.

• Inference

A network-faithful simulator that predicts the serving performance of large LLM deployments before you buy or reserve hardware, modeling the fabric at packet level rather than as a simple bandwidth estimate.

• Inference

A vector compression toolkit for AI workloads that uses TurboQuant codecs to shrink embeddings and KV-cache tensors roughly tenfold, with typed encode and decode contracts and measured byte accounting.

• RAG
• Inference

A reproducible serving configuration, vLLM patches, and measured benchmarks for running the Qwen3.8-27B model on one or two RTX 5090 GPUs at 262K context for concurrent coding agents.

• Inference

A local inference server for Apple Silicon that hot-swaps MLX models on demand, loading one text, vision, embedding, TTS, STT, or decision model at a time behind an OpenAI-compatible API.

• Inference

A tool for running Hugging Face language models on Apple Silicon that are too large to fit in RAM, by streaming their weights layer by layer from disk, including large mixture-of-experts models.

• Inference

A CPU-first, eager-execution tensor and autograd library written in pure Zig, with compile-time-checked named tensor axes and no C/C++ or Python dependency.

• Inference
• Libraries

A Rust-native LLM serving engine, distributed as a single binary, that serves local models over an OpenAI-compatible API with no Python runtime needed.

• Inference

A single-binary local inference server for coding agents, written in Rust, that keeps a request's KV cache resident across turns so later turns in a long conversation start as fast as the first.

• Inference

Turns an Android phone into a private, local inference server, running Google's Gemma 4 models through LiteRT and exposing them over an OpenAI-compatible API on the local network.

• Inference

A VS Code extension that installs a curated open-source AI development stack on HP Z devices and automatically discovers them on the local network.

• Inference
• Editors

An in-browser model manager for local, on-device AI that handles download progress, cache detection, and deduplication of concurrent model loads, with React hooks and a worker bridge, working out of the box with Transformers.js.

• Inference
• Libraries

An open-source native iOS, iPadOS, and macOS chat client, built with SwiftUI, that connects directly to a user's own LiteLLM, Ollama, or OpenAI-compatible server with no OpenClient-hosted proxy or subscription in between.

• Inference

A real-time terminal dashboard, styled like htop, for monitoring an LLM inference cluster running vLLM, SGLang, LMCache, NVIDIA NIM, or NVIDIA Dynamo, auto-discovering pods in Kubernetes with no port-forwarding required.

• Inference
• Monitoring

A verification-first layer for local LLMs that adds hard test execution, symbolic math checking, and an agentic repair loop on top of a small open-weight model, aiming to match a much larger teacher model's accuracy on checkable tasks.

• Inference
• AI evals

A hands-on, first-principles course explaining why LLM inference is memory-bound and how techniques such as batching, KV-cache management, quantization, and parallelism address it, with GPU labs building up to vLLM and SGLang.

• Inference
• Tutorials

A curated list of tools, frameworks, and guides for running AI models on a user's own hardware without a cloud API, spanning inference engines, quantization, fine-tuning, retrieval, orchestration, and monitoring.

• Inference
• Curated lists

A Python library and CLI that runs mixture-of-experts language models larger than a consumer GPU's memory by offloading unused experts to CPU, reaching real-time-usable throughput on an 8 GB laptop GPU.

• Inference

A workspace for running and benchmarking local LLMs on Apple Silicon with MLX, built to measure which local model can handle routine coding operations well enough to replace paid cloud requests in an internal coding tool.

• Inference
• AI evals

A lightweight Tauri desktop control panel for running llama.cpp locally, letting a user discover GGUF models, configure and preview the launch command for llama-server, and save per-model presets.

• Inference

A drop-in ggml backend that offloads llama.cpp and whisper.cpp prefill computation to Rockchip RK3588 and RK3576 NPUs, leaving sequential decode on the CPU where the NPU has no advantage.

• Inference

A C# library that runs local large language models through the same IChatClient interface used for Azure OpenAI or Ollama, powered by ONNX Runtime GenAI, with Blazor components for building local-model chat interfaces.

• Inference

A set of patches and benchmarks that make NVFP4-quantized model inference run correctly and at full speed on Blackwell-generation GeForce and RTX PRO 6000 consumer GPUs, whose architecture differs from datacenter Blackwell.

• Inference