Inference

A tool that predicts whether a given GGUF language model will run on your hardware and how fast, then builds a quantized file customized to your machine and prints the exact llama.cpp command to run it.

• Models
• Inference

A cross-platform Electron desktop app that coordinates several computers on a local network to run distributed LLM inference together, using llama.cpp's RPC backend.

• Inference

Ruby bindings for llama.cpp that run open-source language models such as Gemma, Qwen, Llama and GPT-OSS locally, directly inside a Ruby application, plus a standalone interactive CLI chat.

• Inference
• Libraries

A collection of measured benchmark results for the Intel Arc Pro B70 GPU covering LLM inference speed and power draw, video generation, and dual-card scaling, from real single- and dual-card runs.

• Inference

An Electron module for prototyping local-first apps that chat with local large language models, exposing an API modeled on Chromium's window.AI but supporting any GGUF model.

• Inference
• Desktop

A llama.cpp fork that adds sliding-window attention, multi-token prediction and a 4-bit KV cache to a Qwen hybrid model, keeping decode speed high at long context without losing long-range recall.

• Inference

A Rust and WebAssembly runtime that runs Cactus Compute's Needle tool-calling models in a browser tab, Cloudflare Workers, Node.js, Python, C FFI or embedded no_std targets, with no server or API key.

• Inference

A from-scratch C inference engine that runs a 45-million-parameter tool-calling language model entirely on a five-dollar ESP32-S3 microcontroller, with no Linux, Python or network connection.

• Inference

A local automation loop that discovers, benchmarks and safely upgrades a machine's own LLM inference setup, keeping a stable OpenAI-compatible API address and rolling back automatically if a candidate optimization fails quality checks.

• Inference

A private AI chat app for iPhone and iPad that runs entirely on-device using Apple's MLX framework, supporting models such as Llama, Qwen, Gemma and a distilled DeepSeek reasoning model with no internet connection required.

• Inference

A distributed micro-language-model system that splits a 56 million parameter model across three ESP32-S3 microcontrollers connected over ESP-NOW, with a browser interface for typing prompts and streaming generated text.

• Models
• Inference

An adaptive local-inference provider for the Hermes Agent chat bot that inventories a machine's compute and memory, recommends the best model it can safely run, and exposes it through one OpenAI-compatible endpoint.

• Inference

A Helm chart that deploys a complete LLM operations stack on Kubernetes in one command, covering model serving, a unified API gateway, observability, autoscaling and fine-tuning.

• Inference
• Containers
• Monitoring

A distributed implementation of llama.cpp that spreads a 30 to 70 billion parameter model's layers across ordinary home devices, such as laptops, desktops, phones and tablets, keeping memory pressure low without requiring a GPU on every device.

• Inference

An API layer in front of the Ollama API that adds tools from multiple Model Context Protocol servers so every Ollama chat request can call them, while every other Ollama endpoint is proxied through unchanged.

• MCP servers
• Inference

A CLI for planning whether a large language model will fit and run well on a given machine's unified memory or VRAM, before downloading the weights, with dry-run deployment scripts and benchmarking.

• Inference
• CLI tools

A lightweight local launcher and browser control panel for llama.cpp on Windows, macOS and Linux, covering installation, launching, chat, benchmarking and preset management.

• Inference

A Vercel AI SDK provider for Ollama that implements the official Ollama API with minimal dependencies, working in Node.js, browsers and edge environments.

• Agent frameworks
• Inference

A CLI that provisions and operates a complete self-hosted LLM application stack, including models, vector databases, embeddings, caching and observability, on one private Docker network from a single command.

• Inference
• Hosting

A self-hosted, mobile-first multi-model AI chat built on the official GitHub Copilot SDK, letting a user switch between Copilot's models such as Claude and GPT mid-conversation and run autonomous coding agents from any device.

• Coding agents
• Inference

A web dashboard for managing several Ollama instances at once, letting a developer pull, update, filter and delete models across local and remote endpoints from one screen.

• Inference

A native macOS app that puts one question to several large language models at once, has them critique each other's answers anonymously, and shows where they agree or disagree before the user decides.

• Inference

A modified build of vLLM that adds Continuum-style request scheduling to improve inference throughput, meant as a base for tuning key-value cache pinning logic on different workloads.

• Inference

A macOS app that runs large language models such as DeepSeek V4 and Qwen3.8 on Apple Silicon by reading model experts from SSD, with a built-in chat interface and an OpenAI-compatible API.

• Inference