Inference

A high-performance neural network inference framework optimized for mobile, embedded and desktop deployment, with no third-party runtime dependencies and both CPU and Vulkan GPU backends.

• Inference

A mobile app for running open-source generative AI models fully on-device, for exploring, trying and evaluating on-device LLMs and other generative AI features offline.

• Models
• Inference

A Mozilla.ai project that packages a large language model and its runtime into a single executable file, so it runs locally on most operating systems and CPU architectures with no installation.

• Inference

A library that reduces the memory needed to run very large language models, letting models such as a 70-billion-parameter Llama or larger mixture-of-experts models run on a single small-VRAM GPU.

• Inference

A high-performance serving framework for large language models and multimodal models, aimed at low-latency, high-throughput inference across NVIDIA and AMD GPUs as well as TPUs.

• Inference

A cross-platform framework for building customizable on-device machine learning pipelines for vision, audio and text, deployable to mobile, web, desktop and edge devices.

• Models
• Inference

A self-hostable personal AI application that chats with any local or hosted language model, answers questions from the web or a user's own documents, and can run scheduled research automations.

• RAG
• Inference

A pure C inference engine that runs very large mixture-of-experts language models on consumer hardware by treating disk, RAM and VRAM as one memory tier.

• Inference

The official inference framework for 1-bit large language models such as BitNet b1.58, with optimized kernels for fast, lossless inference on CPU and GPU.

• Inference

A desktop, mobile and web client for chatting with ChatGPT, Claude, Gemini and other large language models, including local models run through Ollama, from one interface.

• Inference

A deep learning optimization library for distributed training and inference of large models, providing memory- and compute-efficient parallelism techniques such as ZeRO and 3D parallelism.

• Models
• Inference

Jan is an open-source, offline-capable alternative to ChatGPT that downloads and runs LLMs locally, keeping full control over data and model choice.

• Inference

LibreChat is a self-hostable, enhanced ChatGPT-style web client that connects to many model providers and adds agents, MCP tools, custom skills, sandboxed code execution and a shared agent marketplace.

• Inference

exo connects a user's own devices into a single AI cluster, letting a model larger than any one device's memory run by splitting it across machines, with day-0 RDMA-over-Thunderbolt support for lower latency.

• Inference

TextGen is a desktop application for running local LLMs with text, vision and tool-calling support, offering a chat UI plus an OpenAI and Anthropic-compatible API entirely offline.

• Inference

LocalAI is an open-source AI engine that runs language, vision, voice, image and video models on any hardware with no GPU required, behind drop-in OpenAI, Anthropic and ElevenLabs-compatible APIs.

• Inference

Cherry Studio is a desktop client for Windows, Mac and Linux that connects to many LLM providers and ships with more than three hundred preconfigured AI assistants alongside custom assistant creation.

• Inference

PrivateGPT is an open-source API layer that turns a locally running, OpenAI-compatible model server into a full private-AI application backend, modeled on the Claude API's shape.

• Agent frameworks
• Inference

AnythingLLM is a self-hosted, multi-user application for chatting with a user's own documents and building custom AI agents, connecting to almost any local or cloud LLM, embedder and vector database.

• RAG
• Inference

Unsloth is a desktop application and toolkit for running and fine-tuning large language and diffusion models locally, supporting formats such as GGUF and MLX across NVIDIA, AMD and Intel GPUs as well as CPUs.

• Models
• Inference

NextChat is a self-hostable AI chat client that connects to Claude, GPT-4, Gemini and DeepSeek, deploys with one click to Vercel, and also runs as a compact native app for Linux, Windows and macOS.

• Inference

A fast, memory-efficient library for serving and running inference on large language models, supporting hundreds of Hugging Face model architectures across many hardware backends.

• Inference

A C/C++ library and set of tools for running large language model and vision-language model inference with minimal setup across a wide range of hardware, from CPUs to GPUs.

• Inference

A self-hosted web interface for AI models that works offline with Ollama and OpenAI-compatible APIs, with user roles, plugins, MCP tools, agents, memory and notes.

• Inference