Inference

An open-source Python package that answers typed questions about application state or documents using a local natural-language-inference model, returning structured choice, score, yes/no or evidence-relation values instead of free text.

• Inference
• Libraries

A local inference server that puts an OpenAI- and Anthropic-compatible API in front of open-weight language models running on hardware you control, from a laptop to a GPU cluster.

• Inference

An unofficial Claude chat client for 2007-era Nokia Series 40 phones, built as a Java ME app plus a small Go relay server, with web search, reading mode and saved chats.

• Inference

A no-backend browser client for a local Ollaya decision server: build typed questions, get calibrated probability answers, and manage the server's local models.

• Inference

A Swift package that answers multiple-choice, graded and yes/no questions about a piece of text using an on-device language model on Apple Core AI, returning a probability for each possible answer, inspired by TypeSafe's Jev.

• Inference
• Libraries

An independent implementation of TypeSafe's Jev wire format that adds typed, probability-scored multiple-choice, graded and yes/no questions on top of any OpenAI-compatible client, reading logprobs where a provider exposes them.

• Inference
• Libraries

A small vision-language model, fine-tuned with LoRA on a 500-million-parameter backbone, that answers typed multiple-choice, ordinal-score and yes/no questions about an image with calibrated probabilities, in about 400 milliseconds on a laptop.

• Models
• Inference

A free, open-source desktop app built with Tauri and Rust that brings ChatGPT, Claude, Gemini, Grok, Perplexity, Qwen, DeepSeek and Z.ai into one window, so a written prompt can be sent to a chosen provider automatically.

• Inference
• Desktop

Turns a DGX Spark GPU cluster into a self-running team of role-specialized AI agents built on the Hermes runtime, dispatching coding work across nodes, reviewing it with a dedicated agent, and restarting anything that crashes.

• Agent frameworks
• Inference

Measured benchmarks, build recipes and more than a hundred ready-to-run GGUF models for running large language models locally on AMD's Ryzen AI Max+ 395 (Strix Halo) hardware with 4-bit ROCmFP4 quantization.

• Models
• Inference

A disaggregated LLM inference framework that reallocates prefill and decode roles across a fixed GPU fleet as demand changes, while model weights stay loaded.

• Inference

A native C++ inference engine that transcribes audio into MIDI or note events, running on CPU, Apple Metal or NVIDIA CUDA through ggml.

• Inference

An inference-aware Layer 4/7 load balancer for LLM serving fleets, forked from the loxilb cloud-native load balancer, adding routing that understands vLLM and SGLang KV caches.

• Inference

A local-first control plane for running and operating models, inference runtimes, agents, tools and retrieval from one privacy-first workspace, across local and cloud environments.

• Inference

An unofficial fork of the FreeToken local LLM inference engine that benchmarks a user's own GPU with a given model and hands back the fastest settings as a ready-to-use launch profile.

• Inference

A secure, peer-to-peer generative AI platform that matches people who want to run LLM inference with volunteers donating their spare compute.

• Inference

Unofficial .NET bindings for Google's LiteRT-LM, running on-device LLM inference such as Gemma from any .NET application on Windows, Linux, Android, macOS or MAUI, without a server or the cloud.

• Inference

A modular desktop AI workspace for running, chatting with, fine-tuning and quantizing local models, and orchestrating agents and MCP tools, across GPUs, CPUs and edge devices.

• Inference

A self-contained C++ inference engine for the Qwen and Gemma model families on ggml, built around auditable answers that cite which parts of the input they used and which they never read.

• Inference

A single-binary LLM inference engine written in Rust for Apple Silicon and NVIDIA CUDA, built from scratch with native CUDA and Metal kernels, a native tokenizer and its own model format.

• Inference

A memory-compression system for long-context transformer inference that combines finite scalar quantization with dynamic latent-space mapping to shrink key-value cache tensors, with an open orchestration layer and a separately licensed core compression binary.

• Inference

A C++ and Metal extension for MLX on Apple Silicon that adds native K-quant quantization, alongside a toolchain to quantize, load, run, LoRA-train and fuse K-quant checkpoints on a stock MLX installation.

• Inference

A community-built serving stack that runs NVIDIA's GLM-5.3-Flash NVFP4 checkpoint across two DGX Spark-class GB10 systems through a pinned vLLM build, aimed at commercially usable, pinned and reversible deployment rather than peak throughput.

• Inference

A standalone C++ inference engine, built on the GGML tensor library, that runs ESMC protein language models on CPU with no Python, PyTorch or LibTorch at runtime.

• Inference