ChatLLM.cpp
A pure C++ implementation, based on ggml, for running real-time multimodal chat with retrieval-augmented generation on CPU or GPU across models from under 1B to more than 300B parameters.
ChatLLM.cpp is a pure C++ implementation of inference for a large number of language and multimodal models, from less than 1B to more than 300B parameters, built on ggml, aimed at real-time chat on a user's own computer using either CPU or GPU. It uses object-oriented design to share code across the similarities between different Transformer-based model families.
Features include memory-efficient inference with int4/int8 quantization, an optimized KV cache and parallel computing; streaming generation with a typewriter effect; effectively unlimited continued chatting through Restart and Shift context-extension methods; retrieval-augmented generation; LoRA support; and bindings for Python, JavaScript, C and Nim, plus a web demo.
The project regularly adds newly released models, and also supports vision, audio and speech models such as Fuyu, Qwen2-Audio and Orpheus-TTS alongside text-only language models.