Inference

A measured serving recipe and benchmark comparing two 4-bit quantized checkpoints of a 27-billion-parameter Qwen model against an FP8 baseline on a single NVIDIA DGX Spark, showing where quantization helps and where it stops mattering.

• Inference
• AI evals

An out-of-tree vLLM plugin that restores GGUF quantization support after it was removed from vLLM's own codebase, letting vLLM serve GGUF-quantized text and vision-language models.

• Inference

A quantization codec and kernel pack, plus an auto-tuning launcher, for running large language models efficiently on older NVIDIA Pascal and Volta cards that lack DP4A support.

• Inference

A collection of Ubuntu snap packages that install a locally optimized runtime and model weights for a chosen open language model in one command, tuned automatically for the host hardware.

• Inference

A native C/C++ inference engine, built on ggml, for running diffusion transformer image and video generation models locally on resource-constrained and edge devices.

• Inference
• Generative media

A private desktop app for chatting with, and building with, open models that run entirely on the user's own machine, combining chat, a coding agent, image and music generation, and document Q&A.

• Coding agents
• Inference

A community-maintained vLLM plugin that serves large language models on AWS Trainium and Inferentia through a Neuron backend, using the same vLLM APIs and configuration as other backends.

• Inference

A pinned Docker Compose build that serves a 27-billion-parameter Qwen model, quantized to 4-bit AWQ, on AMD's Strix Halo integrated GPU through vLLM with speculative decoding and tool calling.

• Inference
• Containers

An OpenAI-compatible chat server that puts any Hugging Face model behind a normal chat API while also showing a live view into what the model is doing internally as it generates a reply.

• Inference
• Debugging

A complete system-on-chip project implementing a quantized convolutional neural network in custom hardware on a twenty-five-dollar FPGA board, classifying blood-cell images faster than an interpreted Python loop on a laptop CPU.

• Inference

A two-board embedded AI demo that runs a 28.9-million-parameter language model directly on an ESP32-S3, with a separate RP2040 acting as a serial OLED display node for the generated text.

• Inference

A zero-dependency Java library and command-line tool that runs GPU-accelerated large-language-model inference on Apple Silicon from a single executable JAR, by binding a Metal-enabled llama.cpp build through Java's foreign-function interface.

• Inference

The official Ollama extension for Visual Studio Code, which adds locally running or Ollama-hosted cloud models to the VS Code Chat model picker.

• Inference
• Editors

Open-source infrastructure for running self-hosted language models in production, deploying vLLM, SGLang or a custom backend across AWS, GCP, Kubernetes or RunPod behind one stable OpenAI-compatible endpoint with guarded, evidence-based releases.

• Inference
• Hosting

A free desktop app that runs open AI chat models locally, and lets multiple computers running SwarmLLM team up over the internet to run models too large for one machine.

• Inference

A community-maintained vLLM hardware plugin that integrates the IBM Spyre AI accelerator into vLLM, following vLLM's own plugin system so IBM's Artificial Intelligence Unit hardware can be used in existing vLLM inference workflows.

• Inference

An experimental C++/CUDA and PyTorch inference stack for running large language models on consumer multi-GPU systems, with model-specific kernels, low-bit quantization, tensor and expert parallelism, and benchmarks tuned for older hardware such as the RTX 2080 Ti.

• Inference

A single-command terminal benchmark that discovers the language models already installed in a local runner, such as Ollama, LM Studio, llama.cpp or vLLM, and measures their speed, memory use and answer quality together on a live leaderboard.

• Inference
• AI evals

A plug-and-play Docker image that runs vLLM with DFlash block-diffusion speculative decoding on NVIDIA Blackwell hardware such as the DGX Spark, optionally combined with TurboQuant KV-cache compression, to speed up decoding on Blackwell-exclusive NVFP4 tensor cores.

• Inference
• Containers

An open-source Python package that produces typed, calibrated decisions from any open-weights language model in a single forward pass, reading the model's own next-token probabilities rather than generating text, as an open alternative to TypeSafe's Jev.

• Inference
• Libraries

A fork of llama.cpp that adds TurboQuant, custom low-bit quantization formats with CUDA kernels for smaller memory use, and TriAttention, GPU-accelerated KV-cache pruning that scores cached tokens by importance to keep long-context inference inside a fixed memory budget.

• Inference

A fully local, multi-model AI agent framework that routes tasks such as planning, coding, reasoning and vision to specialized models running under Ollama, then synthesizes their answers through a Mixture-of-Agents aggregator, tuned for consumer GPUs such as the RTX 4080 or 5080.

• Agent frameworks
• Inference

A local chat application and HTTP gateway for running large language models entirely on a personal machine, with a model catalog for verified downloads and on-demand vision support for images and PDFs.

• Inference

A unified vLLM serving configuration, distributed as a Docker Compose stack, for running large language models on NVIDIA DGX Spark GB10 hardware, supporting either a single-node setup or two nodes joined over a 200 Gbps RDMA link.

• Inference
• Containers