Inference

yzma is a Go library that embeds llama.cpp directly into Go applications for local inference of language and vision-language models, with hardware acceleration and no CGo dependency.

• Inference

hipfire is an AMD GPU-native inference engine, written in Rust, for running large language models locally with an Ollama-style command-line interface and an OpenAI-compatible API.

• Inference

QVAC is an open-source local AI SDK for building applications that run LLMs, speech, vision, and image or video generation entirely on-device, across Linux, macOS, Windows, Android and iOS, with no cloud or API keys required.

• Inference

Swiftlet is a Swift and Metal runtime that runs large Qwen Mixture-of-Experts language models, including 35B and 80B parameter variants, on ordinary Apple devices such as iPhones, by streaming routed expert weights from storage instead of holding the whole model in memory.

• Inference

A standalone C++ inference engine for real-time, multi-person 3D body pose reconstruction from a single camera, exporting motion-capture BVH files or MPEG avatar containers with no Python dependency at runtime.

• Inference

A vector post-training quantization method that compresses large language models to under two bits per parameter while keeping them usable, with pre-quantized community models available for models as large as Llama 3.1 405B.

• Models
• Inference

mlx-dspark runs DeepSeek's DSpark and z-lab's DFlash speculative-decoding drafters natively on Apple Silicon via MLX, giving lossless decoding speedups of up to about 4x for supported consumer-size models.

• Inference

Talos worker is a downloadable GPU client that pairs with a Talos network account and serves open-model inference jobs from that account's local Ollama installation, reporting uptime toward a share of usage revenue.

• Inference

Timber is an ahead-of-time compiler that turns trained classical ML models, such as XGBoost, LightGBM, scikit-learn, CatBoost and ONNX models, into self-contained C99 inference code with no runtime dependencies, and serves them over an Ollama-compatible HTTP API.

• Inference

The Hailo Model Zoo provides pre-trained deep learning models plus a full environment for building, quantizing and evaluating them for deployment on Hailo AI accelerator devices.

• Models
• Inference

Atlas is an open-source LLM inference engine written in pure Rust and CUDA that serves an OpenAI-compatible API from a single small binary, with hand-tuned kernels, quantization and speculative decoding.

• Inference

halogen-flash-server is a specialized inference server that runs the Qwen3.8-Flash-Next model on AMD Strix Halo (gfx1151) GPUs, with kernels tuned for that one GPU and model family instead of a general-purpose runtime.

• Inference

A Rust inference and serving platform for local large language models with an OpenAI-compatible API, offering native flash attention, aggressive key-value cache compression and multi-GPU support on both CUDA and Metal.

• Inference

A toolkit for compressing large generative AI models, including language models, vision-language models and video-generation models, using quantization, pruning and token-reduction algorithms to cut size while preserving performance.

• Models
• Inference

A local inference server for Apple Silicon that serves MLX models behind both OpenAI-compatible and Anthropic-compatible API endpoints, so existing SDK-based clients can point at it as a drop-in local replacement.

• Inference

A native Swift inference server that serves MLX language models on Apple Silicon through a strict OpenAI-compatible API, without a Python runtime, and can stream oversized mixture-of-experts models directly from disk.

• Inference

A tool that converts an existing open large language model into a calibrated decision model, returning a typed answer and a real probability for a choice, a yes/no or a score, without any additional training.

• Models
• Inference

An implementation of DFlash block-diffusion speculative decoding for the MLX framework on Apple Silicon, generating multiple tokens per pass while guaranteeing output identical to standard autoregressive decoding.

• Inference

A C++ library and tutorial for running GPU inference of CNN models with NVIDIA TensorRT, wrapping engine building, caching, and execution behind a small API with no exceptions.

• Inference

A C++ and CUDA toolkit that trains and serves large language models on NVIDIA GPUs, combining pretraining, LoRA or QLoRA fine-tuning, GRPO or DPO optimization and HTTP-based serving in one system.

• Models
• Inference

A single-file, pure Java implementation of Llama 3, 3.1 and 3.2 inference, built as a successor to llama2.java and used to test JVM compiler optimizations.

• Inference

A Go SDK and model server for hardware-accelerated local inference, wrapping llama.cpp, whisper.cpp and stable-diffusion.cpp behind high-level Go APIs and OpenAI- and Anthropic-compatible endpoints.

• Inference

A single native binary that runs the full, unpruned 2.8-trillion-parameter Kimi K3 model on consumer hardware, using small draft models only to speed up generation, never to reduce quality.

• Inference

A terminal application that connects local Ollama models to one or more MCP servers, supporting agent mode, multi-server setups, model switching and human-in-the-loop tool approval.

• MCP servers
• Inference