Inference

A fully offline private AI chat app for Android that runs any GGUF model on-device through llama.cpp with no network permission at all, plus optional Vulkan GPU acceleration and per-device optimized CPU kernels.

• Inference
• Mobile

A Kubernetes operator that runs Ollama-served large language models on a cluster, letting a user deploy and manage multiple models the way they would manage any other Kubernetes workload.

• Inference
• Containers

A Flutter mobile app for iOS and Android that provides one chat interface for on-device LLMs and for local or cloud LLM servers such as LM Studio, Ollama and OpenRouter, with MCP tool support and a built-in model manager.

• Inference
• Mobile

A website and command-line tool for finding which large language models will run well on your own hardware, backed by community-submitted benchmark results.

• Inference

A curated list of articles, models, libraries, and tools built around Apple's MLX machine learning framework for Apple Silicon.

• Inference
• Curated lists

A command-line tool written in Go that generates direct download links for Ollama models and installs locally downloaded model files and manifests into Ollama.

• Inference
• CLI tools

A Raycast extension that connects to a local or remote Ollama server for chat, quick writing-assistance commands, image understanding and model management, directly from Raycast's launcher.

• Inference
• Desktop utilities

An open-source framework, no longer maintained, for production deployment of large language, embedding, reranking, speech, and image and video generation models behind a single OpenAI-compatible API port.

• Inference

An experimental vLLM plugin that adds Attention-FFN Disaggregation, splitting attention and feed-forward computation across separate GPU or Ascend NPU workers without modifying vLLM itself.

• Inference

A tool that networks computers a user already owns into a single OpenAI-compatible endpoint, routing each inference or training request to whichever machine runs the right model, with no account or relay required.

• Inference
• LLM gateways

A public-preview toolkit that builds a Hugging Face model into a versioned TensorRT inference bundle in two commands, with C++ task APIs for text generation, transcription, image and video generation, segmentation, embedding and forecasting.

• Inference

A set of ROS 2 packages that bring llama.cpp and llava.cpp into robotics projects, letting a robot run GGUF language and vision-language models, including grammar-constrained output and real-time LoRA adapter changes.

• Inference

A ComfyUI custom node that adds support for loading GGUF-format quantized diffusion models directly in ComfyUI, covering image, video and audio models with a range of quantization levels and no extra Python dependencies.

• Inference
• Generative media

A lightweight TypeScript wrapper around Chrome's built-in on-device AI APIs for prompting, translation, language detection and summarization, trading some flexibility for a simpler, type-safe interface.

• Inference
• Libraries

A fork of vLLM focused on running recent frontier models on older NVIDIA GPUs such as the A6000, 3090, and A100, distributed as prebuilt Docker images.

• Inference

A native Swift and Metal backend plugin for vLLM on Apple Silicon that replaces the Python inference hot path, serving an OpenAI-compatible API with faster short-context decoding than the existing Python and MLX engine.

• Inference

A local-LLM runner that auto-tunes launch flags to the user's GPU, serves models behind a web UI and an OpenAI or Anthropic-compatible API, and can run any llama.cpp-compatible engine, including community forks, instead of one fixed build.

• Inference

A launcher for llama.cpp and ik_llama.cpp that reads a GGUF model's tensor layout against the VRAM, RAM and per-GPU bandwidth actually available, then places large mixture-of-experts models across a mismatched multi-GPU setup without manual flag tuning.

• Inference
• CLI tools

A large-language-model inference and serving engine written entirely in Java, using the TornadoVM compiler to just-in-time compile transformer kernels to CUDA, OpenCL or Apple Metal at runtime, so the same build runs on any GPU vendor.

• Inference

An inference toolkit for tuning and deploying deep-learning models on NVIDIA GPUs, exposing a single Python API over backends such as TensorRT, Torch-TensorRT, TorchAO, Torch Inductor and ONNX Runtime.

• Inference

A local-first desktop AI chat client, built with Tauri and Next.js, that connects to cloud providers such as OpenAI or Anthropic or to local models through Ollama or LM Studio, with document parsing and a local retrieval-augmented knowledge base.

• RAG
• Inference

An open-source, bring-your-own-key chat client for iOS, Android and the web that talks directly to fifteen model providers or any compatible local or remote endpoint, without routing requests through a paid intermediary.

• Inference

A Swift package that adds large-language-model capabilities to apps built on the Spezi framework, covering local on-device inference, OpenAI-hosted models, and inference offloaded to nearby Fog nodes.

• Inference
• Libraries

A small Transformer language model and pure C inference engine for personal experimentation, running pretrained and fine-tuned models from a browser down to microcontrollers, single-board computers and routers.

• Models
• Inference