TensorFold
A local LLM inference server for Apple Silicon and NVIDIA GPUs that exposes an OpenAI-compatible chat endpoint, with custom Metal and CUDA kernels tuned per model family for exact, faster decoding.
TensorFold downloads a named Hugging Face model, loads it with kernels written for that model family, and serves it at an OpenAI-compatible chat completions endpoint, so any OpenAI client, SDK or curl command can use it as a drop-in local backend. On an NVIDIA DGX Spark it reports decoding 1.6 to 3 times faster than vLLM using speculative drafts.
On Apple Silicon, supported model families verify at load time that their drafted tokens reproduce plain one-row decoding on the user's exact MLX and GPU combination, and only draft when that check passes, falling back to slower undrafted decoding otherwise so output stays identical either way. It requires a Mac with Apple Silicon and Python 3.11 or newer, and installs directly from its GitHub repository with pip.