Lynn Engine
A now-concluded research project that profiled a self-built NVFP4 LLM inference engine against llama.cpp and vLLM, found the custom engine was not the leverage point, and folded its findings into upstream contributions.
Lynn Engine was a from-scratch inference engine using custom CUDA and Triton kernels, W4A8 and NVFP4 quantization, mixture-of-experts support and speculative decoding, built to explore whether a self-written engine could beat established local inference tools. After profiling, the project concluded that for single-request decoding, llama.cpp already wins on bandwidth and kernel-launch efficiency, while NVFP4's real benefit shows up only under concurrent, batched serving, where the bottleneck is the paged-attention and continuous-batching machinery that vLLM and SGLang already provide.
The project has therefore closed its own engine line, contributing its measured NVFP4 conversion, backend and benchmark work back to llama.cpp and several open pull requests to vLLM, and redirected its main effort to a distillation task orchestrator that fine-tunes a smaller mixture-of-experts model to imitate a larger model's step-by-step reasoning. The repository keeps a full, dated log of the intermediate experiments that led to this conclusion.