SharQ
Research code for a Blackwell-GPU LLM inference method that combines a sparse FP4 main computation path with a dense FP4 residual path to recover accuracy lost to sparsification and quantization, with a PyTorch-only simulation mode for accuracy testing without the CUDA kernel.
SharQ is inference code accompanying the paper 'Bridging Activation Sparsity and FP4 Quantization for LLM Inference,' targeting NVIDIA Blackwell GPUs. Its core idea is to capture the important activations with a sparse FP4 main computation path, then use a separate dense FP4 residual path to recover the accuracy that sparsification and quantization would otherwise lose.
The repository provides three runnable modes: NVFP4, a dense FP4 baseline for comparison; SHARQ, the fused sparse-residual FP4 kernel path that needs a real Blackwell (sm_120a or sm_100a) GPU; and SHARQ_SIM, a pure PyTorch fake-quantized, fake-sparse simulation that reproduces the method's accuracy numbers without building the CUDA extension, useful as an accuracy-only reference. Building the real kernel path requires cloning the CUTLASS submodule and compiling with CMake against a specific CUDA architecture flag.
Evaluation scripts measure perplexity on datasets such as WikiText-2, and a demo directory includes a simple local chat script for trying the quantized model interactively.