Inference

A local AI orchestration platform for running large language models offline, with a FastAPI backend and a SvelteKit interface for chat, document work and task automation.

• Inference

A multi-framework neural-network compiler that lowers PyTorch, JAX, Flax and Keras models into portable GGUF binaries and standalone, human-readable C++ code with CPU and CUDA GPU execution.

• Models
• Inference

A privacy-first browser extension that adds a sidepanel chat workspace for local and remote language models, including Ollama, LM Studio and llama.cpp, with local retrieval-augmented generation and browser-context tools.

• RAG
• Inference

A distributed hyperparameter-optimization framework for tuning a vLLM language-model serving deployment, using Ray for distributed execution and Optuna for the search.

• Inference

A reproduction kit for serving GLM-5.3-Flash, a 320-billion-parameter mixture-of-experts model, in production across two NVIDIA DGX Spark machines with a 1,000,000-token context window and documented fixes for prefix-cache misses.

• Inference

A ComfyUI extension that quantizes diffusion models to low-bit formats such as W4A4, W4A8 and W8A8, loads pre-quantized checkpoints and quantization-aware LoRAs, and tunes inference for low-VRAM GPUs.

• Inference
• Generative media

A layer-streaming inference tool that runs large diffusion image, image-editing and video-generation models on GPUs with as little as 4 GB of VRAM, without quantizing the model weights.

• Inference
• Generative media

A local, privacy-first inference server that separates fact storage from language composition, retrieving facts from plain markdown files with a small keyword index instead of relying on a large model's memorized weights.

• RAG
• Inference

A lightweight Android library for running language, vision, speech and image-generation models entirely on-device, built on llama.cpp and related C++ inference engines.

• Inference
• Mobile

A SwiftUI workbench for discovering, converting, running and benchmarking machine-learning models with Apple's CoreAI framework, covering language, vision, audio and diffusion model recipes.

• Models
• Inference

A dependency-free, pure CUDA and C++ reimplementation of NVIDIA's DVLT transformer that reconstructs a 3D point cloud and camera poses from a handful of images in a single forward pass, packaged as one small binary.

• Inference
• Generative media

A self-hosted, microservices-based LLM API platform with an OpenAI-compatible chat endpoint, multi-step tool orchestration, MCP tool integration, and OAuth authentication, deployable through Docker Compose or Kubernetes.

• Inference
• Backend

A fast, dependency-free inference engine for Google's Gemma 4 language models, written in a single pure-Java file with a GGUF parser and a command-line chat interface.

• Inference

A model-training and code-synthesis toolchain for Analog Devices' MAX78000 and MAX78002 ultra-low-power machine-learning accelerator chips, converting a trained PyTorch model into C code for the device.

• Models
• Inference

A pure-Rust, CPU-only library and command-line tool for running LiquidAI's LFM2.5-8B-A1B language model locally, without a GPU.

• Inference

A CUDA technique for large language models that batches many independent structured-output decisions into one parallel pass over a shared prompt, instead of generating each field as separate constrained JSON.

• Inference

A from-scratch pure C inference engine that runs GLM-5.2, a 744-billion-parameter mixture-of-experts model, on a single machine with about 16 GB of RAM by quantizing to int4 and streaming inactive experts from disk.

• Inference

A GPU-accelerated LLM inference engine written entirely in Delphi that runs the full transformer forward pass for GGUF models on the GPU through Vulkan compute shaders, with no Python or CUDA toolkit.

• Inference

An operating layer for a pair of NVIDIA DGX Spark machines that lets several large models share the hardware as switchable lanes, plus on-demand model loading and QLoRA fine-tuning on the idle node.

• Models
• Inference

A command-line tool that inspects live LLM inference processes and breaks down GPU memory use into weights, KV cache, workspace and activations, then projects how much memory different quantization strategies would save.

• Inference
• Debugging

A self-hosted chat application for local and self-hosted language models, positioned as a simpler alternative to Open WebUI, with bundled speech and web-search services so it needs no cloud provider API keys to get started.

• Inference

A native macOS app, CLI and always-on API for running local LLMs on Apple Silicon through MLX, with the inference engine itself implemented in Swift rather than wrapping a Python core.

• Inference

An inference engine for distributed LLM serving that supports dense, mixture-of-experts, multimodal and hybrid-attention models with continuous batching, paged attention and multiple forms of parallelism.

• Inference

A vLLM component that provides custom SYCL kernels for Intel GPUs, covering normalization, attention, quantization and mixture-of-experts operations, to accelerate large-language-model inference on Intel hardware.

• Inference