BEGINNER · 8 MODULES · 64 LESSONS

Airflow and Declarative Orchestration

Get your data pipeline running on its own with Apache Airflow: scheduled daily, with retries, waiting for data to arrive, and ready to reprocess past dates.

For you if
You have a Python pipeline that someone still runs by hand or with a cron job.
By the end
You'll know how to write DAGs, schedule them by logical date, set retries and sensors, run backfills, and see why a task failed.
You need
A tested Python pipeline package (built in the previous guide) and a laptop. No Docker or cloud.
  1. 01Why You Need An Orchestrator8 LESSONS
    1. Module overview: from a command you run by hand to an orchestrator that decides for you
    2. What a chained script cannot do for you
    3. What an orchestrator actually is
    4. The orchestration landscape in 2026
    5. Installing Airflow locally with uv
    6. airflow standalone and the web UI
    7. Your first DAG: hello_kiosko, with the TaskFlow API
    8. Project: hello_kiosko's first real run
  2. 02Dags And The Taskflow Api8 LESSONS
    1. Module overview: from a trivial graph to a DAG that wraps real code
    2. Anatomy of a DAG: nodes, edges, and why no cycles at all
    3. The TaskFlow API in depth: @dag and @task
    4. Classic operators vs. TaskFlow: a quick map
    5. XComs and why they aren't a data bus
    6. Wrapping Kiosko's extract step as a task
    7. Wrapping Kiosko's validate step as a task
    8. Project: Kiosko's first real two-task DAG
  3. 03Scheduling And The Logical Date8 LESSONS
    1. Module overview: the DAG stops needing someone to trigger it by hand
    2. The schedule parameter: cron, presets, and timetables
    3. start_date, catchup, and what a run represents
    4. logical_date versus real execution time
    5. Templating with Jinja: {{ ds }}
    6. Parameterizing Kiosko's partition by logical_date
    7. Running a DAG run for a specific date
    8. Project: Kiosko's scheduled daily DAG
  4. 04Task Level Retries And Failure Handling8 LESSONS
    1. Module overview: when the supervisor, not the worker, decides to retry
    2. What can fail at the task level
    3. retries, retry_delay, and exponential backoff
    4. Code-level retries vs. orchestrator-level retries
    5. Retry policies and selective exceptions
    6. Simulating a transient failure deterministically
    7. Reading a TaskInstance's history and its logs
    8. Project: Kiosko's resilient load task
  5. 05Sensors And Waiting For Upstream Data8 LESSONS
    1. Module introduction: when the pipeline learns to wait
    2. Why a pipeline needs to wait
    3. Sensors: poke mode and reschedule mode
    4. Deferrable operators and the triggerer
    5. Building a sensor for Kiosko's daily file
    6. Timeouts: what happens when a sensor gives up
    7. Gating extraction on file arrival
    8. Project: Kiosko's sensor-gated DAG
  6. 06Backfill With Airflow8 LESSONS
    1. Module introduction: reprocessing a week with no hand-written loop
    2. Manual backfill vs orchestrator-managed backfill
    3. The overwrite-partition pattern, seen at DAG level
    4. Running a backfill with the Airflow CLI
    5. `--reprocess-behavior` and what happens when a run already exists
    6. Proving idempotency across a week of runs
    7. When backfill goes wrong
    8. Project: Kiosko's week, reprocessed with the Airflow CLI
  7. 07Observability Alerting And Integrations8 LESSONS
    1. Module introduction: eyes on the system you already built
    2. The DAG UI: the Grid and Graph views
    3. Reading task logs like an operator
    4. Failure callbacks and the alerting pattern
    5. SLAs, deadlines, and what they really mean on Airflow 3.4.0
    6. dbt as a task inside a DAG
    7. The DAG as a contract: idempotency, revisited
    8. Project: Kiosko's observable DAG, end to end
  8. 08Project Kioskos Orchestrated Pipeline8 LESSONS
    1. Module introduction: the capstone, where seven pieces become one
    2. The brief: Kiosko needs a real orchestrator
    3. Assembling the complete DAG
    4. Scheduling it for production
    5. Running the complete week with a backfill
    6. Verifying idempotency end to end
    7. What Kiosko still needs: the ecosystem map
    8. Project: Kiosko's first production DAG
Reviews

What students say

These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.

No approved reviews yet.

Be the first to share your experience!

SEC. 06 · THE STUDIO

Why this studio exists

For decades, building software belonged to technical profiles with years of training, and getting a digital product into the real world took companies months or years. No-code and AI have narrowed that gap, but the knowledge is still held by a few.

Tech Benders exists to open that knowledge up, so business teams and non-technical teams can put their ideas into production with the right technical decisions behind them. AI does what you ask of it. Without a technical base, what it builds may not scale with your company’s ecosystem, and its costs can run away over time.

The team knows the pains of large organizations and the speed at which they have to move. It brings solution architecture, data engineering, product built end to end from back end to front end, and daily work with AI. Software that solves what you need in days, not months, with the cost of AI kept in check.

Founders

The studioRead about Tech Benders →

Write to ushi@techbenders.io

Made with Wislab