Inference

A command-line tool that launches coding agents such as Claude Code, Codex or OpenCode against a model running locally or with a hosted provider, packaging models as standard OCI images so they move like containers.

• Coding agents
• Inference

A free, open-source, self-hostable alternative to ChatGPT, built with Ruby on Rails, that lets a user bring their own API keys for OpenAI, Anthropic, Google or other providers and switch between them mid-conversation.

• Inference

A Kubernetes operator for enterprise LLM serving that automates model management, GPU scheduling and deployment across runtimes such as SGLang, vLLM, TensorRT-LLM and Triton.

• Inference
• Containers

A single Zig binary for running local large-language-model inference on AMD GPUs and Apple Silicon, loading GGUF files behind a command line, a browser chat interface, a model manager and an OpenAI-compatible API.

• Inference

This project packages vLLM for serving large language models on AMD Ryzen AI Max "Strix Halo" hardware as an Ubuntu-based, Toolbx-compatible Docker or Podman container tracking stable ROCm and vLLM releases.

• Inference
• Containers

vLLM Playground is a web interface for running, managing and chatting with vLLM inference servers, supporting GPU and CPU backends including Apple Silicon and OpenShift or Kubernetes deployment.

• Inference

A platform for hosting, deploying, building and fine-tuning open-source large language models, packaging inference and fine-tuning behind an OpenAI-compatible API and OCI-based model distribution.

• Inference
• Hosting

This project patches vLLM with hand-written SASS kernels so very large Mixture-of-Experts language models can run on consumer and workstation Blackwell GPUs that cannot otherwise fit their official checkpoints.

• Inference

A self-hosted AI agent operating system that keeps an agent's memory, conversations and files on hardware the user owns, running offline by default across a cluster of consumer devices.

• Agent frameworks
• Inference

OSCAR is a quantization method that compresses the key/value cache used during LLM inference down to 2 bits, cutting KV-cache memory by about 8x compared with BF16 while speeding up decoding and throughput.

• Inference

An inference-native agent harness for what it calls Loop Engineering: recursive agent loops built on vLLM that account for serving, routing, context windows and prefix-cache reuse while a task runs.

• Agent frameworks
• Inference

A from-scratch LLM stack that distills a large teacher model into a small student and serves it efficiently, with custom Triton and CUDA kernels, sharded training and paged-KV serving.

• Models
• Inference

A dependency-free C and C++ implementation of OpenAI's CLIP model for image and text embeddings, built on GGML with 4-bit, 5-bit and 8-bit quantization support.

• Inference

onnx2tflite converts an ONNX model to Keras or TensorFlow Lite, automatically aligning PyTorch's channel order to TensorFlow's, and supports FP16 and INT8 quantization for deployment.

• Inference

A high-performance on-device neural network inference framework from Samsung, with a runtime for CPUs, GPUs, DSPs, and NPUs and a compiler toolchain that converts models trained in frameworks such as TensorFlow or PyTorch into a unified form.

• Inference

ollama-helm is a community Helm chart that deploys Ollama on Kubernetes, with configuration for CPU-only or GPU-accelerated clusters and for pulling or creating models automatically at startup.

• Inference
• Containers

BigMoeOnEdge runs Mixture-of-Experts language models larger than a device's RAM, on a phone or PC, CPU-only, by streaming just the experts each token needs directly from flash storage, with output identical to running the full model in memory.

• Inference

MiniSearch is a self-hosted, private web search engine with an AI assistant that can run entirely inside the browser, requiring no API key, inference server or third party to see a user's queries.

• Inference

LLM Hub is an open-source mobile app for Android and iOS that runs chat, image, video and music generation entirely on the device, with CPU, GPU or NPU acceleration and a mode that builds a small app live from a spoken idea.

• Inference
• Generative media

FlashRT is a high-performance real-time inference engine for small-batch, latency-sensitive AI workloads such as robot vision-language-action control, built around hand-tuned kernels rather than a compiled execution graph.

• Inference

A fork of vLLM, the high-throughput LLM inference and serving library, adding an experimental TurboQuant key-value cache path tuned for specific CUDA workstation GPUs.

• Inference

SINQ is a calibration-free quantization method that shrinks large language models to lower precision while keeping most of their accuracy, quantizing a 14B model in about 21 seconds and a 236B model in about 5 minutes.

• Inference

llms.py is a lightweight command-line tool, API and web chat interface for using multiple large language models entirely offline, keeping all data in the browser's own storage rather than any server.

• Inference

ChatLLM Web is a browser-based, private local model studio that runs chat and agent workflows entirely in the browser using WebGPU and WebLLM, with no backend, account or telemetry.

• Inference