llamAmpere
A fork of llama.cpp tuned specifically for NVIDIA Ampere consumer GPUs such as the RTX 3090, adding a custom KV cache format and speculative decoding for large context windows.
llamAmpere is a fork of llama.cpp optimized for NVIDIA Ampere-generation consumer GPUs, particularly the RTX 3090 and 3090 Ti. It adds a custom TurboQuant key-value cache format, multi-token-prediction speculative decoding using the target model's own draft head with a large draft-vocabulary shortlist, and custom SM86 and Qwen-specific kernels.
The fork reports running a 27-billion-parameter Qwen model at up to 245,000 tokens of context on a single RTX 3090, with measured throughput at several context depths compared against stock llama.cpp and tuned vLLM. Later releases add support for additional weight formats, including an EXL3 trellis format and ternary models at 1.75 to 2.125 bits per weight, alongside an upstream catch-up to a recent llama.cpp commit.
It is built from source in C++ and released under the MIT license, as a fork whose base README is otherwise unchanged from upstream llama.cpp.