Inference

A ComfyUI custom node that adds one-click virtual VRAM and multi-GPU distribution for UNet, CLIP and VAE loaders, offloading model layers to other devices to free VRAM for larger generations.

• Inference
• Generative media

A from-scratch reimplementation of the vLLM inference engine with self-contained paged and flash attention, built to benchmark attention mechanisms during prefill and decode.

• Inference
• Tutorials

jevos is an open-source, self-hosted alternative to a hosted yes/no decision service called Jev: it answers a yes/no question about a piece of text on a laptop CPU in 50 to 220 milliseconds and speaks the same wire format so existing client code keeps working.

• Inference

Ollama-rs is a Rust client library for the Ollama API, covering text completion including streaming, chat, embeddings, function calling and model management, built by following Ollama's own API documentation.

• Inference
• Libraries

A GPU document parser that combines OCR, layout analysis, table and formula recognition into one CUDA and TensorRT pipeline, converting pages to Markdown at up to hundreds of images per second.

• RAG
• Inference

YOLOs-CPP is a C++ inference library that gives every YOLO model version from v5 through v12 and YOLO26 one unified API for detection, segmentation, pose estimation, oriented bounding boxes, classification and depth estimation, built on ONNX Runtime and OpenCV.

• Models
• Inference

A high-performance operator library for large language models, implemented in the Triton language, that registers with PyTorch's ATen backend to accelerate training and inference across many hardware platforms.

• Models
• Inference

A course and full source code for building a smaller version of vLLM from scratch in C++ and CUDA, covering model loading, CUDA kernels, KV caching, batching and PagedAttention.

• Inference
• Tutorials

A self-hosted web UI for more than 30 kinds of generative AI, including image, audio, video and chat models, able to run offline on modest hardware without a dedicated GPU.

• Inference
• Generative media

A desktop MCP client and local AI playground that lets a user chat with cross-vendor LLMs through the Model Context Protocol, with no required account and full local control.

• MCP servers
• Inference

A Chrome extension that runs an on-device AI assistant entirely in the browser with WebGPU and Transformers.js, letting it manage tabs and answer questions about the current page.

• Inference
• Browser automation

A vLLM engineering fork focused on making modern large language models run fast on older NVIDIA Tesla V100 (SM70/Volta) GPUs, rather than merely able to start on them.

• Inference

A Python library of key-value cache compression methods for large language models, aimed at reducing the memory cost of serving long-context transformer models.

• Inference

The Apple MLX inference engine bundled with LM Studio, supporting text and vision model inference and speculative decoding on Apple Silicon.

• Inference

An open-source AI chatbot workspace for individuals and teams, combining multiple LLM providers, MCP tools, visual workflows, custom agents, and a realtime voice assistant.

• Agent frameworks
• Inference

A Python framework of post-training and training-time compression algorithms, such as quantization and pruning, for optimizing neural network inference in OpenVINO with minimal accuracy loss.

• Models
• Inference

A C++ engine for deploying video analytics and on-device vision-language models to edge hardware such as Sophon, Rockchip and x86, with orchestration, alarms and benchmarking.

• Inference

A modern web interface for chatting with local LLMs served by Ollama, with local chat history, Markdown rendering, dark mode, and a hosted version plus a Docker Compose setup.

• Inference

An extensible LLM quantization toolkit supporting GPTQ, AWQ, GGUF, FP8, and other formats across NVIDIA, AMD, Huawei Ascend, and Intel hardware, integrated with Transformers, vLLM, and SGLang.

• Models
• Inference

A PyTorch-native inference engine built on Diffusers that adds hybrid cache acceleration and parallelism, quantization, and CPU offload for diffusion transformer (DiT) image and video models.

• Inference
• Generative media

A small collection of DeepSeek-related utilities, including a WeChat chatbot built on the DeepSeek API, a local file-upload workaround, one-click DeepSeek model deployment scripts, and a web-search plugin for local LLMs.

• Inference

Alibaba's high-performance inference engine for large language models, used internally across Alibaba's businesses, with CUDA kernel optimizations, quantization, LoRA serving, and multi-GPU tensor parallelism.

• Inference

A distributed inference engine that lets you build your own AI cluster across ordinary machines to serve large language models, with pipeline-parallel sharding and GPU or Mac backends.

• Inference

A .NET client library for the Ollama API, recommended by Microsoft and used by Semantic Kernel, .NET Aspire and Microsoft.Extensions.AI, covering chat, embeddings, model management and tool calling.

• Inference
• Libraries