BEGINNER · 8 MODULES · 64 LESSONS
Airflow and Declarative Orchestration
Get your data pipeline running on its own with Apache Airflow: scheduled daily, with retries, waiting for data to arrive, and ready to reprocess past dates.
- For you if
- You have a Python pipeline that someone still runs by hand or with a cron job.
- By the end
- You'll know how to write DAGs, schedule them by logical date, set retries and sensors, run backfills, and see why a task failed.
- You need
- A tested Python pipeline package (built in the previous guide) and a laptop. No Docker or cloud.

Curriculum
01Why You Need An Orchestrator
- Module overview: from a command you run by hand to an orchestrator that decides for you
- What a chained script cannot do for you
- What an orchestrator actually is
- The orchestration landscape in 2026
- Installing Airflow locally with uv
- airflow standalone and the web UI
- Your first DAG: hello_kiosko, with the TaskFlow API
- Project: hello_kiosko's first real run
02Dags And The Taskflow Api
- Module overview: from a trivial graph to a DAG that wraps real code
- Anatomy of a DAG: nodes, edges, and why no cycles at all
- The TaskFlow API in depth: @dag and @task
- Classic operators vs. TaskFlow: a quick map
- XComs and why they aren't a data bus
- Wrapping Kiosko's extract step as a task
- Wrapping Kiosko's validate step as a task
- Project: Kiosko's first real two-task DAG
03Scheduling And The Logical Date
- Module overview: the DAG stops needing someone to trigger it by hand
- The schedule parameter: cron, presets, and timetables
- start_date, catchup, and what a run represents
- logical_date versus real execution time
- Templating with Jinja: {{ ds }}
- Parameterizing Kiosko's partition by logical_date
- Running a DAG run for a specific date
- Project: Kiosko's scheduled daily DAG
04Task Level Retries And Failure Handling
- Module overview: when the supervisor, not the worker, decides to retry
- What can fail at the task level
- retries, retry_delay, and exponential backoff
- Code-level retries vs. orchestrator-level retries
- Retry policies and selective exceptions
- Simulating a transient failure deterministically
- Reading a TaskInstance's history and its logs
- Project: Kiosko's resilient load task
05Sensors And Waiting For Upstream Data
- Module introduction: when the pipeline learns to wait
- Why a pipeline needs to wait
- Sensors: poke mode and reschedule mode
- Deferrable operators and the triggerer
- Building a sensor for Kiosko's daily file
- Timeouts: what happens when a sensor gives up
- Gating extraction on file arrival
- Project: Kiosko's sensor-gated DAG
06Backfill With Airflow
- Module introduction: reprocessing a week with no hand-written loop
- Manual backfill vs orchestrator-managed backfill
- The overwrite-partition pattern, seen at DAG level
- Running a backfill with the Airflow CLI
- `--reprocess-behavior` and what happens when a run already exists
- Proving idempotency across a week of runs
- When backfill goes wrong
- Project: Kiosko's week, reprocessed with the Airflow CLI
07Observability Alerting And Integrations
- Module introduction: eyes on the system you already built
- The DAG UI: the Grid and Graph views
- Reading task logs like an operator
- Failure callbacks and the alerting pattern
- SLAs, deadlines, and what they really mean on Airflow 3.4.0
- dbt as a task inside a DAG
- The DAG as a contract: idempotency, revisited
- Project: Kiosko's observable DAG, end to end
08Project Kioskos Orchestrated Pipeline
- Module introduction: the capstone, where seven pieces become one
- The brief: Kiosko needs a real orchestrator
- Assembling the complete DAG
- Scheduling it for production
- Running the complete week with a backfill
- Verifying idempotency end to end
- What Kiosko still needs: the ecosystem map
- Project: Kiosko's first production DAG
Reviews
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!