mergekit
A toolkit for merging pre-trained large language models using an out-of-core approach that can run entirely on CPU or with as little as 8 GB of VRAM.
mergekit is a toolkit for merging pre-trained language models, using an out-of-core approach to perform elaborate merges in resource-constrained situations; merges can run entirely on CPU or be accelerated with as little as 8 GB of VRAM. Merging combines the strengths of different models without the overhead of running an ensemble or the need for additional training, keeping the same inference cost as a single model.
It supports models including Llama, Mistral, GPT-NeoX and StableLM, with many merge methods, lazy tensor loading for low memory use, piecewise assembly of models from layers ("frankenmerging"), mixture-of-experts merging, LoRA extraction, evolutionary merge methods, multi-stage merging for complex workflows, and merging of raw PyTorch models.
It installs from source in editable mode, is released under the LGPL-3.0 license, and documents its merge configuration format, including parameter, tokenizer and chat-template specification.