ggmlc
A multi-framework neural-network compiler that lowers PyTorch, JAX, Flax and Keras models into portable GGUF binaries and standalone, human-readable C++ code with CPU and CUDA GPU execution.
ggmlc ingests models directly from PyTorch's torch.export or JAX and Flax's jaxpr, translates them into a strongly typed intermediate representation, and optimizes them automatically, removing the need to hand-write C++ inference code for each new model architecture. It serializes graphs, dynamic shapes and quantized weights into standard GGUF v3 containers rather than a proprietary format, runs on CPU or NVIDIA GPUs with zero-copy memory transfers, and can also emit self-contained C++ header files, a native entry point and a CMake build file for embedding directly into a native application.
The project validates its output with automated differential numerical testing against PyTorch and JAX reference runs on both CPU and GPU, and provides a Python binding with zero-copy NumPy buffer evaluation and multi-threaded execution, plus a hardware-accelerated persistent key-value cache for agent-serving workloads.