Inference

A command-line tool that tells you which local LLMs fit an Apple Silicon Mac, how fast they will run, and how to launch them, in one command.

• Inference

A small, growing collection of filter, pipeline and tool extensions for Open WebUI that fill everyday usability gaps in local and cloud LLM chat workflows.

• Inference

A collection of self-contained example applications that show how to deploy AI models at scale with Lightning AI's LitServe model-serving engine.

• Inference
• Tutorials

An unmanaged, zero-allocation native AI inference runtime for Java 22 and newer that consolidates vision, speech and LLM pipelines behind one native memory boundary.

• Inference

Microsoft's on-device AI inference solution for running generative AI models locally through a CLI, SDK or REST API; this repository holds tutorials and examples for it.

• Inference

A batch-inference engine for large mixture-of-experts language models that minimizes the total time to finish a batch of requests rather than single-request latency.

• Inference

An engineering log and Python test scripts that verify Apple Neural Engine primitives such as gemm, softmax, attention and transformer blocks on Linux, comparing ANE output against NumPy on real M1 and M1 Max Asahi Linux hardware.

• Inference

An LLM inference compiler and runtime that compiles a Hugging Face checkpoint into a static packet stream and executes it with a persistent, resident GPU interpreter, avoiding per-operation CPU dispatch, with an OpenAI-compatible HTTP server.

• Inference

A local-first C++ runtime for CPU-only LLM inference with Apple Silicon acceleration, exposed through a unified command-line interface.

• Inference

A benchmarking tool that shows exactly where a local LLM ran, CPU or GPU and on which layers, alongside prefill, decode, memory and power measurements.

• Inference
• Debugging

An open-source SystemVerilog NPU project building a bare-metal transformer accelerator for experimental Gemma-class LLM inference on an AMD/Xilinx FPGA board.

• Inference

A CPU-first reference platform showing the systems around LLM inference: an OpenAI-compatible gateway, a deterministic eval CLI, an experiment registry and a RAG toolkit.

• Inference
• AI evals

Official code for K-Forcing, which distills an autoregressive language model into a push-forward model that decodes several tokens jointly in one forward pass instead of one token at a time, for faster batch inference.

• Models
• Inference

A Flutter starter app demonstrating NobodyWho, a library for running LLMs locally and efficiently on any device, covering chat, tool calling, multimodal vision and hearing, embeddings and RAG, speech-to-text and text-to-speech.

• Inference
• Mobile

An object-storage-native key-value cache for LLM inference that lets any engine pointed at the same S3 bucket reuse prefilled context across restarts and machines.

• Inference
• Databases

An LLM inference engine for AMD RDNA4 GPUs written in Rust using Vulkan compute shaders, supporting GGUF models and native FP8 matrix operations.

• Inference

A local inference daemon for running LLMs on Apple Silicon that exposes an OpenAI- and Ollama-compatible API, built for fast model loading and low latency.

• Inference

A Rust research prototype for a DNA-inspired, self-evolving control layer around LLM inference, handling routing, memory, reflection, and device adaptation around an existing inference backend rather than being a production inference engine itself.

• Inference

An unofficial command-line client for the Perplexity API, supporting interactive chat, one-off web-search queries, shell completion, and MCP integration for use from other AI tools.

• Inference
• CLI tools

A local, CPU-only agentic AI runtime for Linux that combines chat, model-selected tool use, file workflows, and static artifact analysis in one terminal client, built on a vendored llama.cpp backend.

• Coding agents
• Inference

A gateway-first local model runtime and CLI that pulls, runs, and serves GGUF or Ollama models behind an OpenAI-compatible endpoint, with a dashboard, token telemetry, and an optional GPU capacity marketplace.

• Inference
• LLM gateways

A Python GUI for generating llama.cpp server launch commands, with a machine-learning-driven optimizer that searches thread, batch, and KV-cache settings for the best measured performance on the user's hardware.

• Inference

A pure-Go, no-cgo local LLM inference engine distributed as a single static binary, running open-weight models such as Gemma, Qwen, and Llama from safetensors or GGUF with schema-constrained structured output.

• Inference

A deterministic planning layer that turns discrete llama.cpp GGUF quantization presets into a near-continuous model-size control, spending a fixed byte budget on the most useful tensor-level precision upgrades.

• Inference