DataFlex
A dynamic training framework, built on LLaMA-Factory, that schedules training data during large-language-model fine-tuning through data selection, mixture, and reweighting.
DataFlex is a data-centric training framework built on top of LLaMA-Factory that dynamically schedules which training samples a large language model sees during optimization, rather than treating the training set as fixed. It integrates several difficult-to-reproduce data-selection, data-mixture, and data-reweighting methods from the research literature into one reproducible framework.
Data selection dynamically chooses samples according to a strategy, such as favoring harder examples, using gradient-based, loss-based, or distribution-based methods; data mixture adjusts the ratio of data drawn from different sources during training; and data reweighting adjusts each sample's training weight. The project supports gradient computation under DeepSpeed ZeRO-3 for training and analyzing larger models, and its accompanying paper was accepted to the NeurIPS 2026 Evaluation and Benchmark track.
DataFlex is written in Python and released under the Apache License 2.0.