Inference

An unofficial, community-built Linux packaging of OpenAI's official ChatGPT desktop app, repackaging the signed macOS-derived Linux payload into deb, RPM, pacman, AppImage and Nix builds.

• Inference

A privacy-focused, self-hosted AI workspace combining chat, a multi-agent team feature with human review, a visual workflow builder and Telegram integration, running on a user's own infrastructure and model choice.

• Inference

A high-performance inference engine for 4-bit diffusion models, based on the SVDQuant quantization method, supporting FLUX and Qwen-Image models with ComfyUI integration.

• Inference
• Generative media

A production inference stack that compiles a single model codebase to run on NVIDIA, AMD, Intel, TPU, or Trainium hardware without rewriting it for each accelerator.

• Inference

A lightweight, Python-based inference and serving framework for large language models, built to be easy to scale and extend by drawing on techniques from FasterTransformer, TGI, vLLM and FlashAttention.

• Inference

An OpenAI API-compatible inference proxy that applies more than 20 published inference-time techniques, such as mixture-of-agents and Monte Carlo tree search, to improve LLM accuracy on reasoning tasks without retraining.

• Inference

A collection of optimized, pre-trained deep-learning models and demo applications for the OpenVINO Toolkit, intended to speed up development of high-performance inference applications instead of training models from scratch.

• Models
• Inference

A Kubernetes-native, distributed inference-serving stack, backed by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, that adds routing, KV-cache management and autoscaling on top of model servers such as vLLM and SGLang.

• Inference
• Containers

A C++ and Python library for efficient inference with Transformer models on CPU and GPU, using weight quantization, layer fusion and batch reordering to cut memory use and increase speed.

• Inference

An open-source, self-hostable AI chat client that can be deployed on premises across desktop, web and mobile, letting a team pick and switch model providers instead of depending on one vendor.

• Inference

PyTorch's open-source stack for running AI models locally on phones, wearables, laptops, browsers, embedded systems and microcontrollers.

• Inference

A toolkit for deploying and serving open-source text-embedding and sequence-classification models, supporting popular families such as FlagEmbedding, GTE and E5.

• RAG
• Inference

An open-source inference engine that profiles a user's own hardware, recommends the best open models for it, and tunes them for that exact machine, running on Apple Silicon, NVIDIA, AMD or CPU-only setups.

• Inference

A cross-platform desktop AI assistant and MCP client that connects to major model providers, supports a local knowledge base, and lets a user install tools built as MCP servers.

• MCP servers
• Inference

A package for running inference and fine-tuning Vision Language Models, including omni models with audio and video support, on Apple Silicon Macs using MLX.

• Models
• Inference

An inference library for Recursive Language Models, a task-agnostic paradigm that lets a language model programmatically examine, decompose and recursively call itself over near-infinite-length contexts.

• Models
• Inference

A model-swapping proxy that runs multiple local AI models and hot-swaps between them on demand, working with any OpenAI or Anthropic API compatible server.

• Inference

An open-source GPU cluster manager that configures and orchestrates inference engines such as vLLM and SGLang, and lets a team launch SSH-accessible GPU instances on demand.

• Inference
• Hosting

A local AI server that runs large language, speech and image-generation models on a user's own GPU or NPU, exposing OpenAI, Anthropic and Ollama-compatible APIs at no cost.

• Inference

A single-binary, OpenAI-API-compatible local inference server for GGUF models, built entirely in Rust on the project's own WebGPU transformer engine, Airframe.

• Inference

A standardized, distributed inference platform for deploying both generative and predictive AI models at scale on Kubernetes, incubating under the CNCF.

• Inference
• Containers

An open-source, Ollama-compatible macOS, iOS, and visionOS app for chatting with privately hosted language models such as Llama 2, Mistral, or Vicuna.

• Inference

A hybrid edge-cloud inference engine for running language, vision-language, and speech models directly on mobile devices and wearables.

• Inference
• Mobile

An open-source, local-first desktop client for AI agents that brings chat, tool use, Skills, and Agent Client Protocol support into one Electron application.

• Inference