Inference

A performance-tuned build of llama.cpp for the AMD Strix Halo APU, bundling Flash Attention and MoE-prefill fixes with a current Mesa driver to speed up long-context, quantized-KV inference.

• Inference

A KV-cache compression method for LLM inference that improves on an existing method by exploiting a structural property shared across models, described in a NeurIPS 2026 submission.

• Inference

A CUTLASS-based quantized BLAS library for low-bit deep learning inference and training on NVIDIA Blackwell GPUs, supporting MXFP4, MXFP8 and NVFP4 microscaling formats.

• Inference
• Libraries

A multi-agent harness that autonomously tunes LLM and diffusion inference serving configurations, frameworks and GPU kernels for AMD Instinct GPUs, then stores proven results in a reusable recipe knowledge base.

• Agent frameworks
• Inference

A C++ inference engine built on llama.cpp for Vision-Language-Action robot policies, running open VLA models such as SmolVLA, GR00T and BitVLA as self-contained GGUF files without Python.

• Inference

An experimental storage-aware mixture-of-experts inference stack that prefetches predicted experts before decoding starts, aimed at keeping SSD-offloaded MoE models from stalling generation.

• Inference

A native AI and agent client for iOS and Apple Watch that connects to OpenAI, Anthropic Claude and Google Gemini, or runs local GGUF models offline through llama.cpp, with local RAG and MCP tools.

• Inference
• Mobile

A post-training quantization toolkit for LiteRT and LiteRT-LM models, letting developers shrink vision and LLM models for deployment on mobile CPUs, GPUs and NPUs.

• Inference
• Mobile

A native macOS app for benchmarking and comparing local large language models across any OpenAI-compatible backend, with real-time hardware telemetry alongside inference performance history.

• Inference
• AI evals

A C# client for the Ollama API, auto-generated from its OpenAPI specification, covering chat, streaming completions, embeddings, model management and tool calling for .NET applications.

• Inference
• Libraries

A self-hostable Streamlit chat application that talks to local Ollama models (or optionally OpenAI), with document RAG over PDFs, image chat and Whisper-based voice input.

• RAG
• Inference

A PHP client library for the Ollama API, covering chat and text completion (including streaming), model management and embeddings, for PHP 8.1 and above.

• Inference
• Libraries

A systems framework that coordinates GPU and NPU clusters for agentic reinforcement-learning post-training, routing requests across heterogeneous vLLM workers and adjusting resource allocation as workload changes.

• Models
• Inference

A Rust-native local inference application for chat, file work, and coding with GGUF models on your own hardware, shipping as a single binary with a desktop app, browser UI, terminal, and OpenAI-compatible API, and no Python or Node.js at runtime.

• Inference
• Desktop

An on-device neural network training and inference framework in C++ that runs 30-billion-parameter mixture-of-experts LLMs on a mobile phone by streaming only the active experts from flash storage instead of keeping the whole model in memory.

• Inference
• Mobile

A Swift package for running local large language models on iOS, macOS, and Linux, supporting GGUF and MLX model formats and Apple's FoundationModels framework behind one streaming API, with experimental multimodal input and tool calling.

• Inference
• Mobile

A Rust and CUDA inference engine that streams a giant Mixture-of-Experts model's routed experts from NVMe storage per token, letting models far larger than available VRAM run on consumer GPUs by keeping only routing and attention resident in memory.

• Inference

A Kubernetes-based platform for deploying and routing GPU-accelerated inference, streaming, and batch workloads at scale, handling function lifecycle, invocation routing, GPU cluster integration, and observability.

• Inference
• Hosting

A speculative-decoding method for faster LLM inference that drafts a whole token tree in one parallel pass while preserving causal conditioning between branches, then verifies the entire tree against the target model in a single forward pass.

• Inference

A benchmarking tool for local large-model inference that measures prefill and decode speed, offering a browser-only version and a Python backend version that removes the browser's low connection-concurrency limit for higher-concurrency testing.

• Inference
• AI evals

A portable Windows launcher that serves Qwen3.6-27B locally through an OpenAI-compatible API with no WSL, Docker, or telemetry, offering a text-based interface to pick between preset speed, context-length, and GPU configurations.

• Inference

An adaptive mixed-precision quantization format and runtime for Apple Silicon MLX, positioned as a GGUF equivalent that keeps large models quantized in GPU memory while running at full Metal speed.

• Models
• Inference

A production LLM inference engine written in pure Python and ctypes with its own kernel compiler emitting CUDA, ROCm, and Metal directly, aiming for fast cold starts and low memory use without PyTorch or Triton.

• Inference

A memory-optimized ROCm build of Qwen 3.8 27B for AMD Strix Halo APUs, combining 4-bit block quantization, speculative decoding, and a KV-cache scheme to raise inference throughput on unified-memory hardware.

• Models
• Inference