End-to-end ELT data pipeline built using Python, PostgreSQL, Apache Airflow, and Docker. The project extracts data from the YouTube API, orchestrates workflows with Airflow DAGs, and ensures data ...

OpenWeatherMap API → extract.py (raw JSON per city → data_lake/raw/{type}/{city}/{Y}/{M}/{D}/{ts}.json) → staging_transform.py (raw JSON → data_lake/staging ...