Kedro
A Python framework for production-ready data engineering and data science pipelines, applying software engineering practices to make them reproducible, maintainable and modular.
Kedro is an open-source Python framework, hosted by the LF AI & Data Foundation, for building data engineering and data science pipelines using standard software engineering practices so they stay reproducible, maintainable and modular.
It provides a standard, modifiable project template based on Cookiecutter Data Science, a data catalog of lightweight connectors for loading and saving data across local, network, cloud and HDFS storage with file-based versioning, and automatic dependency resolution between pipeline functions with visualization through Kedro-Viz.
It supports test-driven development with pytest, documentation with Sphinx, and linting with ruff, and can be deployed to a single machine or distributed across platforms such as Argo, Prefect, Kubeflow, AWS Batch and Databricks.