Inference

An Android app that runs Gemma, Qwen and LLaMA language models entirely on the phone for offline, private chat, including voice input, image input with OCR and camera-based prompting.

• Inference

A lightweight, modular, Kubernetes-native platform for deploying and serving large language models, separating model lifecycle management from inference traffic routing behind custom resource definitions.

• Inference
• Containers

A community-built, vLLM-alike inference engine written entirely in C++, with continuous batching, paged key-value cache and RadixAttention, aiming for the same throughput as vLLM with a much smaller install.

• Inference

vLLM Router is a high-performance request router for large-scale vLLM deployments, providing cache-aware and other load-balancing algorithms plus prefill and decode disaggregation and Kubernetes-native service discovery.

• Inference
• Networking

The TPU inference backend for vLLM, unifying JAX and PyTorch model execution under a single lowering path so PyTorch models can run performantly on TPU hardware without extra code changes.

• Inference

A local LLM inference server for Apple Silicon and NVIDIA GPUs that exposes an OpenAI-compatible chat endpoint, with custom Metal and CUDA kernels tuned per model family for exact, faster decoding.

• Inference

An open-source AI desktop application for working with local language models, files, tools and agents, with local inference as its core and optional connections to cloud models and services.

• Coding agents
• Inference

A collection of official and community sample apps, model recipes and utilities for LiteRT, Google's on-device machine learning framework formerly known as TensorFlow Lite, and LiteRT-LM, its layer for running language models on-device.

• Inference
• Tutorials

A browser-based, no-install graphical interface for chatting with Ollama and other local or OpenAI-compatible language model servers, with chats stored only in the browser.

• Inference

A browser-based system that pools the GPUs of several devices over WebRTC to run one large open language model together, splitting its layers across laptops, desktops and phones.

• Inference

A cross-platform toolkit for running language models, speech recognition and text-to-speech natively inside apps and games, on Flutter, Swift, Kotlin, Unity or Rust, entirely offline.

• Inference
• Speech

An Android app that turns a phone into an OpenAI-compatible LLM server, running models locally through Google's LiteRT-LM runtime on the phone's GPU or CPU and serving them over the local network with no cloud or subscription.

• Inference

A Runpod Serverless worker template that deploys an OpenAI-compatible large language model endpoint powered by vLLM, either from a pre-built Docker image pointed at a model or by building an image with the model baked in.

• Inference
• Hosting

vLLM Kunlun is a community-maintained vLLM hardware plugin that runs large language model inference on Baidu's Kunlun XPU accelerators, supporting more than 20 mainstream and multimodal models.

• Inference

An encrypted, privacy-first on-device AI app for Android that runs chat, vision, retrieval-augmented generation, voice and a sandboxed plugin runtime against local GGUF models, with no Google Play services, telemetry or cloud dependency.

• RAG
• Inference

A Rust CLI and model server for high-performance LLM, VLM, embedding, reranking and audio inference on Apple Silicon and NVIDIA CUDA, executing MLX SafeTensors checkpoints natively with no Python in the request path.

• Inference

A cross-platform desktop app, powered by KoboldCpp, for running large language models and generating images locally on a person's own hardware, with no telemetry, accounts or external servers.

• Inference
• Generative media

APEX is a mixed-precision quantization method for Mixture-of-Experts language models that assigns bit-width per tensor type and layer, producing GGUF quantizations for llama.cpp that match or beat full Q8_0 quality at a fraction of the size.

• Inference

A dual-architecture research framework that unifies the exact output of autoregressive large language models with the parallel, high-speed token generation of diffusion models, for strictly lossless faster inference.

• Models
• Inference

A native .NET inference engine for GGUF models that runs autoregressive and text-diffusion large language models, image generation and editing, and video generation, with a console app, browser chat UI and OpenAI/Ollama-compatible HTTP APIs.

• Inference
• Generative media

An open-source vLLM plugin from Snowflake that combines advanced parallelism, speculative decoding and model optimizations to raise inference throughput and lower latency for large language models and embeddings.

• Inference

A native vLLM attention backend that quantizes the KV cache to fit three to five times more context in the same memory, without a calibration step and without changing the served model.

• Inference

A native desktop application for chatting with multiple AI providers, such as OpenAI, Claude, Gemini, Grok or a local Ollama model, searching local files and the web, running scripts, and delegating tasks to agent CLIs from one app.

• Coding agents
• Inference

A Java library that wraps the Ollama server's API, giving Java applications text generation, multi-turn chat, tool and MCP tool calling, embeddings and model management against local or remote Ollama models.

• Inference
• Libraries