Inference

A near-lossless 5-bit transformer compression method with verifiable, reproducible reconstruction: a CLI generates text against a hosted compressed model, browses a model catalog, and verifies a downloaded pack's integrity against a published SHA-256 manifest.

• Models
• Inference

Research code for a Blackwell-GPU LLM inference method that combines a sparse FP4 main computation path with a dense FP4 residual path to recover accuracy lost to sparsification and quantization, with a PyTorch-only simulation mode for accuracy testing without the CUDA kernel.

• Inference
• Papers

A zero-dependency, single-file inference engine for Qwen 3.5 language models written in pure Java, supporting model sizes from 0.8B to 35B (including a mixture-of-experts variant), several GGUF quantizations, and GraalVM native-image compilation.

• Inference

A command-line wrapper around llama.cpp's llama-server that gives it an Ollama-like workflow: starting and stopping a background server, downloading and inspecting GGUF models interactively, and automatically applying a model's recommended sampling parameters.

• Inference
• CLI tools

A native macOS app for running open large language models entirely offline through Apple's MLX framework, with a llama.cpp/GGUF fallback engine, vision and audio input, image generation, thinking-mode visibility, and speculative decoding for faster output.

• Inference
• Desktop

A decoding method and reference implementation that speeds up LLM inference by computing geometric upper bounds on token logits to skip most of the output vocabulary at each decoding step, without retraining the model.

• Inference

A ported 4-bit NVFP4 key-value cache for serving GLM-5.2 across four DGX Spark machines with vLLM, trading some decode speed for a much larger context pool.

• Inference

A single native binary, written from scratch in C, that runs local GGUF language models on CPU, CUDA or Metal, serves them over OpenAI- and Anthropic-compatible APIs, and can train LoRA adapters directly through the quantized weights it serves.

• Inference

A single-model inference server for Qwen3.6-35B-A3B on Apple Silicon, with an OpenAI-compatible HTTP API, a custom Metal decode engine built outside MLX's op-graph, and an optional bit-for-bit lossless decoding mode.

• Inference

A .NET library and CLI that runs, fine-tunes and serves large language models in-process, adding local chat, RAG, agents and voice to .NET applications without Python, a GPU or a separate model server.

• Inference
• Libraries

A SwiftUI showcase chat app demonstrating Apple's on-device Foundation Models on iOS 26 and iOS 27, providing fast, private AI chat with no app server and no data leaving the device.

• Inference
• Tutorials

A benchmarking tool that sweeps llama.cpp settings on your own GPU or CPU to find the fastest configuration for a given model, and compares the result against a shared database of other people's hardware and settings.

• Inference
• Testing

An Expo module that wraps Apple's on-device Foundation Models on iOS and Google's Gemini Nano through ML Kit on Android behind one on-device LLM inference API for React Native apps.

• Inference
• Mobile

A WinUI 3 sample chat app from Microsoft that runs Windows' on-device Aion Instruct Preview language model on Copilot+ PCs, streaming tokens with first-token latency and tokens-per-second shown under each reply.

• Inference
• Tutorials

A V language binding for llama.cpp, letting V programs run quantized large language models such as DeepSeek or Qwen locally on CPU hardware.

• Inference
• Libraries

A Django-based platform that lets life science researchers at a Swedish research institute serve machine learning models and host data science apps such as Shiny, Gradio, Streamlit, and Dash on a shared Kubernetes cluster.

• Inference
• Hosting

A cross-platform Flutter chat app that connects to hundreds of AI models through OpenRouter and to remote MCP servers over Streamable HTTP, running an agentic tool-use loop against them.

• MCP servers
• Inference

An experimental Android app for testing Google's on-device Gemini Nano model through the ML Kit Prompt API, built with Kotlin and Jetpack Compose.

• Inference
• Mobile

A pure-Rust inference engine that runs GGUF language models directly on the GPU through WebGPU compute shaders, working across NVIDIA, AMD, Intel, integrated, and Apple Silicon GPUs without a C++ toolchain or llama.cpp.

• Inference

A self-hosted AI appliance for NVIDIA GPU boxes that installs an orchestrator per machine, serves models through OpenAI- and Anthropic-compatible APIs, and automatically clusters multiple nodes for cross-node tensor-parallel inference and fine-tuning.

• Inference

A 27B coding and tool-use language model family for agentic repository work, released both as a full-precision PEFT adapter on a Qwen3.6 base and as a standalone, packed-weight GGUF edition for native local inference.

• Models
• Inference

A deployment guide and benchmark for running the Qwen3.6-35B-A3B mixture-of-experts model with multi-token-prediction speculative decoding on a single NVIDIA L4 24GB GPU using the official llama.cpp Docker image.

• Inference
• Tutorials

A native macOS menu bar and desktop app that monitors Inco AI's Splash local inference engine, showing live token throughput, memory use, and installed models, and offering one-click model installation and server control.

• Inference
• Desktop utilities

A C# library that implements neural network inference and training as GPU compute kernels for .NET, running the same code on WebGPU, CUDA, OpenCL, WebGL, WebAssembly, and CPU, including inside a browser via Blazor WebAssembly.

• Inference
• Libraries