ZSE
A production LLM inference engine written in pure Python and ctypes with its own kernel compiler emitting CUDA, ROCm, and Metal directly, aiming for fast cold starts and low memory use without PyTorch or Triton.
ZSE is an LLM inference engine that owns its full stack instead of depending on PyTorch, Triton, bitsandbytes, or the transformers library. It is written in pure Python and ctypes, with a kernel compiler that emits CUDA, ROCm, and Metal code directly, which the project reports lets models load in seconds rather than minutes and serve using a fraction of the memory other engines need.
It installs as a single pip package with no transitive machine-learning dependencies and serves models through its own command-line launcher. The README publishes cold-start, VRAM, and single-sequence throughput comparisons against a popular quantized inference engine across several NVIDIA and AMD GPUs, reporting large cold-start and memory advantages alongside mixed results on raw decode throughput depending on the GPU.
It runs across NVIDIA, AMD, and Apple Silicon hardware.