Inference

A self-hosted home AI inference platform for AMD Strix Halo hardware that turns a Linux box into an OpenAI-compatible endpoint for chat, embeddings, reranking, transcription, speech and image generation.

• Inference

A post-training quantization method for large language models that learns per-coordinate grid assignments and per-group scales through a Gumbel-Softmax relaxation, targeting 1 to 3 bits per parameter.

• Inference

A fully offline chatbot that runs entirely on an ESP32-S3 Cardputer microcontroller, with an 8-million-parameter model quantized to 4 bits and baked into the firmware.

• Inference

A KV-cache management framework built on nano-vllm that compresses the key-value cache during LLM inference to improve memory use and throughput in high-concurrency, long-context serving.

• Inference

A Godot 4.5 plugin that runs a local LLM inside a game through a bundled llama.cpp backend, giving an in-game agent node both a text reply and an action a creature or NPC can act on.

• Inference
• Libraries

A Swift library for constraining large-language-model output to a schema, regular expression or custom grammar during generation on Apple's MLX framework, built on the XGrammar constrained-decoding library.

• Inference

A set of vLLM plugins that serve encoder, pooler and structured-prediction models such as ColBERT, GLiNER and ColPali for production inference, without forking vLLM.

• Inference

A practical, hands-on guide to running large language models and coding agents locally on Apple Silicon, covering inference engine choice, memory sizing, model selection and setting up agent tool calling.

• Inference
• Tutorials

A local model server for AMD Strix Halo and Radeon GPUs that keeps command-line compatibility with an existing tool while adding day-zero support for newer large mixture-of-experts models and hardware-tuned quantization.

• Inference

An open-source implementation of Google's TurboQuant method for compressing a language model's key-value cache, with a running summary of which of its variants the community has since found actually work well in practice.

• Inference

A local-first workbench for Apple Silicon that brings agent sessions, model comparison, fine-tuning, benchmarking and runtime monitoring for local MLX and remote language models into one interface.

• Models
• Inference

A neural-network quantization and inference toolkit that measures how sensitive each part of a model is, assigns it just enough precision to fit a memory budget, then serves the packed result with CUDA and Metal kernels.

• Models
• Inference

A native C++ and CUDA inference runtime for running mixture-of-experts language models on Blackwell GPUs, serving an OpenAI-compatible endpoint from a small binary with no Python dependency.

• Inference

A collection of pre-built Python wheels for llama-cpp-python covering different operating systems, Python versions and CUDA architectures, so users do not have to compile the CUDA extension themselves.

• Inference

A GPU compiler and benchmarking stack that optimizes a large language model for inference through kernel fusion, autotuning and scheduling, then deploys and tracks the results across GPU types.

• Inference

A deployment tool for vision-language-action robot policies that verifies a model exports correctly, proves it matches the reference implementation, then promotes it to hardware profiles such as Jetson, RTX or Apple Silicon.

• Models
• Inference

A serving build and set of patches that run a large mixture-of-experts language model across two consumer GPUs and system memory, offloading inactive experts to RAM to fit a 256,000-token context.

• Inference

An MLX implementation of Google's TurboQuant method that compresses both model weights and the key-value cache of large language models on Apple Silicon, letting bigger models fit in less memory.

• Models
• Inference

A dependency-free C inference engine that runs Qwen3-TTS text-to-speech models on the CPU using only a BLAS library, with support for nine voices and ten languages.

• Inference
• Speech

A TypeScript library that adds on-device text generation, speech-to-text and vision to Expo and React Native apps, using each platform's native AI models with no API keys or cloud calls.

• Inference
• Mobile

A Dart SDK that runs AI models directly on iPhones and Macs, covering chat, vision, speech-to-text, speech synthesis, image generation and retrieval, entirely offline.

• Inference
• Mobile

A local app that rents a GPU in your own Google Colab account and serves a large Qwen3.8 model on it, showing the cost before it starts and exposing the model to Codex or any OpenAI-compatible client.

• Inference
• Hosting

A collection of AI assistant skills that teach a coding agent how to install, configure, serve and troubleshoot vLLM-Omni, a framework for omni-modality model inference across text, image, video and audio.

• Agent skills
• Inference

A monorepo for Maple, a confidential AI research and chat application with a desktop Agent Mode, its client SDKs, a local OpenAI-compatible proxy, and the OpenSecret backend behind it.

• Inference
• LLM gateways