Inference

A local LLM chat UI for the web and Chrome, connecting to Ollama, LM Studio, AUTOMATIC1111 or Chrome's on-device Gemini Nano with no external API calls, so chats and image generations stay fully private.

• Inference

A minimal C++/GGML inference engine for OpenAI's privacy-filter PII/NER model family, detecting personal-data entity spans with exact byte offsets much faster and with far less memory than Hugging Face Transformers.

• Inference

A Rust, UEFI-resident LLM runtime that boots a PC or Raspberry Pi 5 directly into a local chat prompt from a USB stick or SD card, with no operating system underneath.

• Inference

A command-line tool that reviews GGUF model files and estimates their memory usage and maximum tokens per second, reading only the file's metadata so nothing has to be downloaded first.

• Inference
• CLI tools

A cross-platform SDK for running compressed large language models fully on-device, using a quantization algorithm that learns an optimal per-weight bit allocation instead of a fixed scheme.

• Inference

A machine learning platform for Apple Silicon, written in Rust, covering local LLM inference, LoRA/QLoRA fine-tuning, quantization and serving through a desktop GUI, a terminal TUI and Metal/Neural Engine acceleration.

• Models
• Inference

A Rust and CUDA LLM inference engine tuned specifically for NVIDIA's Blackwell-generation GPUs (RTX PRO 6000, RTX 5090, B200), with an OpenAI-compatible server and hardware-specific low-bit quantization support.

• Inference

A pure-Rust, CPU-only, memory-safe OCR engine running a small zoo of hand-ported vision-language models, including a DeepSeek-OCR-derived model, a chart-data extractor and a sheet-music transcriber, with no Python, GPU or general ML framework required.

• Inference

An MLX port of Moonshot AI's 2.78-trillion-parameter Kimi-K3 multimodal mixture-of-experts model for Apple Silicon, with a streaming weight converter, expert pruning and per-language expert-overlap analysis, though no single Mac can currently run it.

• Models
• Inference

A native Swift CLI and local server that turns an Apple Silicon Mac into a private, OpenAI-compatible AI server for Hugging Face MLX models or Apple's on-device Foundation Model.

• Inference

An MLX extension for Apple Silicon that adds INT8/INT4 activation quantization and an optimized decode attention kernel to speed up LLM prefill and autoregressive decode on M5 Macs.

• Inference

The official Triton Inference Server backend for running large language models on vLLM's engine, handling inflight batching and paged attention for served requests.

• Inference

A local runtime, modeled on Ollama, for pulling and serving small decision models that turn a piece of text plus typed questions into calibrated probabilities in milliseconds, through a TypeSafe-compatible API.

• Inference

An OpenAI-compatible API server that runs MLX text, multimodal, image and Whisper models locally on Apple Silicon, exposed through the familiar OpenAI SDK endpoints.

• Inference

A C library and command-line tool for inspecting and manipulating GGUF model files, showing metadata and tensors, comparing two models for similarity, and inspecting raw tensor weights.

• Inference
• CLI tools

A plugin for Simon Willison's LLM command-line tool that adds access to models running on a local or remote Ollama server, including chat, image attachments, tool calling, and embeddings.

• Inference

This project provides scripts and documentation for running multi-node inference of Qwen3.8-Flash-Next in NVFP4 across two NVIDIA DGX Spark nodes using vLLM with tensor parallelism, expert parallelism and multi-token prediction.

• Inference

A terminal UI that finds locally installed LLM inference backends and model files, then lets a user launch and monitor local model servers without writing config files or CLI flags.

• Inference

A native Swift engine that streams a 125-billion-parameter mixture-of-experts model from SSD and caches its busiest experts in memory, so it can run on Apple Silicon Macs that cannot hold the whole model in RAM.

• Inference

A zero-dependency C implementation for running local LLM inference with lossless KV-cache compression, letting full documents fit in context on consumer hardware instead of relying on chunked retrieval.

• Inference

ds4-on-spark packages a Blackwell-CUDA performance fork of the ds4 engine to serve DeepSeek-V4-Flash entirely on-device on an NVIDIA DGX Spark, with large prefill and decode speedups over the upstream engine.

• Inference

A cycle-level simulator for LLM serving infrastructure that models heterogeneous accelerators, disaggregated memory tiers, and multi-instance parallelism to study serving performance without real hardware.

• Inference

A Docker Compose setup that lets Intel Arc GPU owners run Ollama, Stable Diffusion and Whisper speech recognition together behind Open WebUI, using a native SYCL or Vulkan llama.cpp backend for Ollama.

• Inference
• Containers

The user-facing web frontend for the RuoYi-AI platform, providing AI conversations, agent interactions and knowledge-base question answering, built with Vue 3 and Ant Design Vue.

• Inference