Inference

A ChatGPT-like conversational interface powered by RWKV, a fully recurrent (RNN) language model that matches transformer quality while running faster and using less memory.

• Models
• Inference

A library for serving language, speech-recognition and multimodal models with a single command, exposing them through a unified, OpenAI-compatible inference API on the cloud, on-premises or on a laptop.

• Inference

A CPU/GPU inference engine for large language models that exploits activation locality to run fast local inference on consumer-grade hardware.

• Inference

A set of eight SDKs sharing one C++ core that run LLMs, vision, speech and image-generation models locally on phones, browsers, desktops and servers, routing each call to the best engine the device has.

• Inference

An open-source toolkit for optimizing and deploying deep-learning models across CPUs, GPUs and AI accelerators, supporting models from PyTorch, TensorFlow, ONNX and other frameworks.

• Inference

An open-source inference-serving system that deploys models from multiple deep-learning and machine-learning frameworks across cloud, data center, edge and embedded devices.

• Inference

A key-value cache management layer for scalable LLM inference that turns transient KV cache into reusable, shareable state across serving engines, cutting time-to-first-token and improving throughput.

• Inference

A tool for self-hosting open-source large language models as OpenAI-compatible API endpoints with a single command, including a built-in chat UI and deployment paths for Docker, Kubernetes and BentoCloud.

• Inference

A command-line tool and Python library for running prompts against OpenAI, Anthropic Claude, Gemini and dozens of other language models, including local models, with SQLite logging and plugin support.

• Inference
• CLI tools

A large-language-model development kit built on the PaddlePaddle framework, supporting efficient training across hardware types, lossless compression and high-performance inference.

• Models
• Inference

The open-source components of NVIDIA's TensorRT SDK for high-performance deep-learning inference, including plugins, an ONNX parser, sample applications and import-workflow guides.

• Inference

An edge-native mixture-of-experts serving engine that runs frontier-scale open-weight language models on consumer GPUs, with CPU-GPU co-execution, semantic caching and Anthropic/OpenAI-compatible APIs.

• Inference

An SDK that provides a Python API for defining large language models and running them efficiently on NVIDIA GPUs, with specialized kernels, an optimized runtime and components for building custom inference servers.

• Inference

A web GUI for ChatGPT and many other large language models, adding chat history management, file-based question answering, agent tools, web search and GPT fine-tuning on top of a range of hosted and locally deployed models.

• Inference

A lightweight, from-scratch reimplementation of vLLM in about 1,200 lines of Python, offering comparable offline inference throughput with prefix caching, tensor parallelism and CUDA graphs.

• Inference

A large model and multimodal model fine-tuning and deployment framework from the ModelScope community, supporting training, inference, evaluation, quantization and deployment for more than 600 text and 400 multimodal models.

• Models
• Inference

A Rust tensor library and deep learning framework that unifies training and inference in one codebase, JIT-compiling tensor operations with automatic kernel fusion across CUDA, ROCm, Metal, Vulkan and WebGPU.

• Models
• Inference

A lightweight deep learning inference and training framework, used in production across more than 30 Alibaba apps, with companion projects for running large language and stable diffusion models locally on phones, PCs and IoT devices.

• Models
• Inference

A JavaScript library that runs Hugging Face Transformers models directly in the browser with no server, using an API similar to the Python Transformers library, for natural language, vision, audio and multimodal tasks.

• Inference
• Libraries

An in-browser large language model inference engine that runs open source models directly in a web browser using WebGPU acceleration, with an API compatible with OpenAI's.

• Inference

A framework for running and fine-tuning large language models through CPU-GPU heterogeneous computing, letting large mixture-of-experts models do inference and LoRA fine-tuning on consumer hardware with CPU expert offload.

• Models
• Inference

An LLM inference server for Apple Silicon Macs, managed from the menu bar, with continuous batching and a tiered in-memory and SSD key-value cache.

• Inference
• Desktop utilities

A small, native local inference engine for running a curated set of large language models, including DeepSeek V4 and GLM, on consumer hardware such as Macs, NVIDIA GPUs and Strix Halo systems.

• Inference

A machine learning compiler and deployment engine that compiles large language models to run across GPUs and platforms including AMD, NVIDIA, Apple and Intel GPUs, web browsers, iOS and Android.

• Inference