Inference

This is a PowerToys Run plugin that lets a Windows user query a local Ollama model directly from the PowerToys Run launcher, choosing which pulled model to ask on the fly.

• Inference
• Desktop utilities

ComfyUI-Muse embeds a persistent, multi-session local LLM chat panel directly inside a ComfyUI node, so a user can brainstorm image or video prompts against Ollama, LM Studio or a directly loaded GGUF model without leaving their generation workflow.

• Inference
• Generative media

autotune is a middleware layer that computes the exact KV cache a local LLM request needs, pins the system prompt in memory and manages context windows automatically for Ollama, LM Studio and Apple Silicon MLX, without any code or workflow changes.

• Inference

FerryAI runs ONNX, GGUF and RubixML models directly inside a PHP 8.3+ process through FFI, giving PHP applications chat, streaming, embeddings, RAG and vector search without a Python sidecar, HTTP microservice or Docker container.

• Inference

Crow is a local AI agent rather than a chat box: it gives a model around thirty tools, MCP access, memory, skills, goals, subagents, vision and image generation and editing, running on one GPU through its own engine or llama.cpp, or against any OpenRouter model.

• Agent frameworks
• Inference

PromptChain is a pip-installable library and MCP server for explicitly loading, unloading, pinning and swapping local language models across LM Studio and Ollama, plus a small Streamlit app that chains a drafting model into a coding model with a human review step in between.

• MCP servers
• Inference

MELLM routes a query to a small, domain-specific expert model instead of a single large general model, keeping only one specialist loaded in VRAM at a time so consumer GPUs can get expert-level answers.

• Inference
• LLM gateways

A demo project that packages a self-hosted LLM chat interface, running Qwen2-1.5B-Instruct or another Hugging Face model behind vLLM with a Streamlit front end, into a single Docker container deployable locally or on an AWS GPU instance.

• Inference
• Tutorials

Tightwad pools CUDA and ROCm GPUs across separate machines and speeds up local LLM inference with application-layer speculative decoding, sitting as a drop-in proxy in front of an existing model endpoint.

• Inference

A Rust AI inference engine for embedding local large language models in mobile and desktop apps, loading GGUF checkpoints for on-device chat with bindings for Python, Flutter, React Native, Swift and Kotlin.

• Inference

A terminal visualization tool for local LLM inference that labels a streaming response as exact token provenance or plain SSE chunk mode, and colors verified speculative-decoding steps rather than arbitrary transport chunks.

• Inference
• Debugging

A single-binary local LLM inference platform, inspired by Ollama, that downloads and runs models from the CSGHub platform with a web UI, an OpenAI-compatible API, resumable downloads and one-click coding-agent setup.

• Inference

An optimized local serving engine for the Kimi-Linear-48B model on a single consumer GPU, using INT4 quantization and fused decode kernels, with a protocol bridge that lets Claude Code, Codex, Cline, Aider and opencode connect to it unchanged.

• Inference

A step-by-step guide, with an automated installer script, for running the DeepSeek R1 model offline on a rooted Android phone using Termux, Arch Linux and Ollama.

• Inference
• Tutorials

A packaging of a durable local inference server for the Qwen3.8 27B model on a single consumer NVIDIA GPU, built to resume a coding agent's session from saved state after the inference server restarts.

• Coding agents
• Inference

A local LLM runtime and orchestration layer, positioned as a drop-in Ollama replacement, that pulls models from Hugging Face, serves them, and offers a built-in dashboard, pipelines and structured output.

• Inference

A recomputation-free key-value caching library for large language models that stitches independently precomputed document caches together at serving time with cheap positional realignment instead of recomputing tokens.

• Inference
• Libraries

A research fork that runs a 78.62 GiB DeepSeek V4 Flash GGUF model on a Linux laptop with only 7.7 GiB of RAM and no GPU, using memory-mapped NVMe demand paging instead of loading the full model into memory.

• Inference

A local multi-model inference engine, built on llama.cpp and stable-diffusion.cpp, that keeps several models resident in memory at once and switches between them without reloading weights.

• Inference
• Generative media

A memory-first C++20 inference runtime for quantized GGUF language models on CPU, trading throughput for a large cut in peak memory use compared with llama.cpp.

• Inference

A local LLM chat, image-diffusion and on-device training app for the Xbox Series S and X, built as an architecture exercise under a console's UWP sandbox and 10 GB RAM ceiling.

• Inference
• Generative media

A local context and memory engine that gives a small local LLM a billion-token pool on disk, pulling back only the slice it needs while it reasons instead of enlarging its context window.

• Agent memory
• Inference

A POWER8-specific research prototype that replaces some full-matrix attention computation with a hardware AES instruction used as a fast prefilter, so only a subset of attention paths get the full computation.

• Models
• Inference

A dual-engine benchmarking pipeline for GGUF and safetensors language models on NVIDIA GPUs, measuring both speed and answer quality with a live dashboard and exportable benchmark cards.

• Inference
• AI evals