Inference

A browser-based chat interface for open-source language models that runs entirely on-device using WebGPU, including local retrieval-augmented question answering over uploaded files.

• RAG
• Inference

A tool that turns Discord into a collaborative front end for any OpenAI-compatible LLM, using reply chains as conversation history so no database is required.

• Inference

A Kubernetes API for deploying a group of pods as one replication unit, aimed at multi-host AI and ML inference workloads that shard a large model across multiple nodes.

• Inference
• Containers

A library for training speculative-decoding draft models that deploy directly into LLM inference engines such as vLLM, speeding up inference without changing output quality.

• Models
• Inference

A self-hosted inference server for LLMs, vision-language models and image generation on Apple Silicon, exposing OpenAI, Anthropic and Ollama compatible HTTP APIs with no third-party API keys required.

• Inference

A Swift library for running large language models locally on macOS, iOS, watchOS, tvOS and visionOS, built around llama.cpp with automatic chat-template handling from a model's GGUF metadata.

• Inference
• Libraries

A language server that connects code editors to LLMs for code completion, handling prompt construction, context-window tokenization and multi-backend model access so editor extensions can stay lightweight.

• Inference
• Editors

A local inference engine for Apple silicon that runs coding agents and OpenAI or Anthropic compatible applications on one Mac, combining speculative decoding with Metal kernels and automatic memory planning.

• Inference

A proxy written in Go that lets a local Ollama model, or DeepSeek and Mistral, act as a GitHub Copilot style code completion backend for editors such as Neovim, VS Code and Zed.

• Inference
• Editors

A serving framework for audio and multimodal AI models that runs text-to-speech, speech recognition, and omni chat models behind an OpenAI-compatible API, built to handle multi-stage generation pipelines.

• Inference
• Speech

Optimized, open-weight Whisper speech-to-text models with streaming inference engines for NVIDIA GPUs and on-device Apple Silicon, aimed at low-latency real-time transcription.

• Inference
• Speech

An open-source inference server that serves machine learning models over REST and gRPC, compliant with the KFServing V2 Dataplane spec.

• Inference

A local-first, cross-platform desktop AI workbench unifying multi-provider chat, ACP coding agents, a knowledge base, MCP tools and an API gateway, keeping application data and files under the user's own control.

• Coding agents
• Inference

A high-performance model-serving framework, with a Rust web layer and a pure-Python interface, that adds dynamic batching and pipelined CPU/GPU stages to turn a trained machine-learning model into an efficient online service.

• Inference

A production-grade C++ inference server, built on OpenVINO and optimized for Intel hardware, that serves generative AI, agentic and classic deep-learning models over OpenAI-compatible and KServe network APIs.

• Inference

A pure C++ implementation, based on ggml, for running real-time multimodal chat with retrieval-augmented generation on CPU or GPU across models from under 1B to more than 300B parameters.

• Inference

A lightweight Python CLI that estimates the inference memory a Hugging Face model needs, reading only its Safetensors or GGUF metadata over HTTP range requests rather than downloading the model.

• Inference
• CLI tools

A Swift package that is an API-compatible drop-in replacement for Apple's Foundation Models framework, adding support for Core ML, MLX, llama.cpp and several hosted model APIs behind the same interface.

• Inference
• Libraries

A signed macOS menu-bar app that launches, supervises and monitors a local DeepSeek V4 or V4.1 inference server, downloading model weights and gating launches by a Mac's available RAM.

• Inference
• Desktop utilities

A cross-platform desktop app, written in Rust and React, that iterates over combinations of local Ollama models, prompts and inference parameters and lets a user visually compare and A/B test the results.

• Inference
• AI evals

A 10-week, 30-minutes-a-day roadmap that has an engineer build, instrument and optimize one OpenAI-compatible LLM inference service, covering vLLM, SGLang, quantization, speculative decoding and cost-aware routing.

• Inference
• Tutorials

A multi-provider command-line interface for OpenAI, Azure, Perplexity, LLaMA and other large language models, with streaming, interactive chat, prompt files, image and audio input, MCP tool calls and an experimental agent mode.

• Inference
• CLI tools

A native macOS app that wraps a self-hosted vMLX inference server, serving language, vision and image-generation models over an OpenAI-compatible API on Apple Silicon with no Python setup.

• Inference

A hardware-focused fork of vLLM tuned for dual RTX 2080 Ti and other Turing (SM75) GPUs, adding launcher profiles and validated inference routes for running 27B to 35B class models locally.

• Inference