AirFlow-ML-Data-Integration
An example Apache Airflow project that orchestrates a daily ETL pipeline fetching weather data from a free API, cleaning and transforming it, and storing it in PostgreSQL, ready for downstream machine learning tasks.
This project demonstrates orchestrating a daily ETL pipeline with Apache Airflow: fetching weather data from the free OpenWeather API, cleaning and transforming it, and storing the result in a PostgreSQL database, as a reproducible, extensible example for training machine learning models on weather data later.
A quickstart clones the repository, creates a Python virtual environment, installs dependencies, and runs a test suite that verifies the Airflow DAG parses correctly without needing an API key or a database. The full stack, including Airflow and PostgreSQL, can also be run with Docker Compose after adding a free OpenWeather API key to a local .env file, which starts Airflow with the weather_data_pipeline DAG and both connections already registered.
The README also documents manual setup instructions for Airflow, the database, the project folder layout, an IDE and the Airflow-to-database and Airflow-to-API connections, for readers who prefer not to use Docker.