PowerInfer
A CPU/GPU inference engine for large language models that exploits activation locality to run fast local inference on consumer-grade hardware.
PowerInfer is a CPU/GPU inference engine for large language models that uses activation locality to serve models faster on a single device, including consumer-grade GPUs. On a single RTX 4090, it reports an 11x speedup over llama.cpp when running Falcon(ReLU)-40B in FP16, with both engines using the same hardware and fully utilizing available VRAM.
Related releases from the same project include PowerInfer-2, an inference framework for smartphones; TurboSparse, a set of sparsified Mistral and Mixtral models for faster inference; ProSparse Llama 2 models with about 90% sparsity; and Bamboo LLM. It supports Windows GPU inference and AMD devices through ROCm.
It is written in C++ and released under the MIT license.