LightLLM
A lightweight, Python-based inference and serving framework for large language models, built to be easy to scale and extend by drawing on techniques from FasterTransformer, TGI, vLLM and FlashAttention.
LightLLM is a Python-based inference and serving framework for large language models, designed for a lightweight codebase, easy scalability and high throughput. It draws on techniques from other well-known open-source projects, including FasterTransformer, TGI, vLLM and FlashAttention, and supports prefix KV-cache transfer between data-parallel ranks.
Its pure-Python design and token-level KV-cache management make it a practical basis for research, and several external projects, including LoongServe, SGLang, Aphrodite and OmniKV, use or build on parts of LightLLM. It is released under the Apache-2.0 license, with published papers on its constrained-decoding and request-scheduling techniques.