OpenAutoNLU Pipeline
An open-source pipeline for training natural language understanding models for text classification and named entity recognition, which automatically chooses between few-shot and full fine-tuning methods based on how much labeled data is available.
OpenAutoNLU pipeline provides high-level training pipelines for text classification and token classification that load a user's training data, optionally run data-quality diagnostics, and then automatically select a training method based on dataset size: AncSetFit for two to five examples per class, SetFit for six to eighty, and full fine-tuning for larger datasets. Configuration such as batch size, out-of-distribution detection method and data augmentation can be overridden, and trained models can be exported to ONNX for deployment.
It also supports optional LLM-based data augmentation and synthetic test generation, out-of-distribution detection, and data-quality diagnostics such as dataset cartography. A Streamlit application and CPU and GPU Docker images are included for interactive use.
It is a Python package installable from PyPI, requires Python 3.12, and was built by MWS AI.