INTERMEDIATE · 8 MODULES · 64 LESSONS
Streaming with Kafka and Flink
Recognize when streaming is worth it and use Kafka and Flink to solve what batch can't, like knowing within minutes that a store stopped reporting.
- For you if
- You already build batch pipelines and someone is asking whether the business needs real time.
- By the end
- You'll know how to work with topics and partitions, windows with watermarks, incremental state, and Postgres change capture with Debezium.
- You need
- Intermediate Python and a working Docker setup. The dbt and lakehouse guides help.

Curriculum
01When Latency Is The Product
- Module introduction: when latency IS the product
- The weak flag: streaming that recomputes batch
- What streaming costs your company, and what going without it costs
- When latency genuinely IS the product
- Installing Kafka in KRaft mode with Docker
- Installing PyFlink locally
- Proving it: the same 35.3% funnel in Kafka and Flink
- Project: Kiosko's first Kafka and Flink "hello world"
02Kafka Topics Partitions And Offsets
- Module introduction: Kafka's real anatomy
- A topic is an ordered, append-only log
- Partitions, and why order is only guaranteed within one
- Offsets: the bookmark of a consumer
- Choosing a Python client: `confluent-kafka` vs. `kafka-python`
- Creating Kiosko's `kiosko.orders` topic
- Producing and consuming your first message
- Project: Kiosko's first Kafka topic
03Producing And Consuming Kioskos Order Stream
- Module introduction: from fixed file to real stream
- From fixed files to a message stream
- Keying messages by `store_id`
- Building a deterministic, finite producer
- Building a consumer that verifies the count
- Replaying the canonical week through Kafka
- Verifying the same `106.15` total again
- Project: Kiosko's order stream in Kafka
04Event Time Watermarks And Deterministic Windows
- Module introduction: this guide's heart
- Processing time vs. event time
- Why the wall clock is banned in this guide
- Watermarks: how Flink knows it's safe to close a window
- Bounded out-of-orderness in PyFlink
- Tumbling windows on `order_ts`
- Reproducing Kiosko's daily totals with a windowed job
- Project: Kiosko's first deterministic window
05Flink Aggregations Windows Vs Running State
- Module introduction: two ways to aggregate, one real advantage
- Revenue per window, per store
- Keyed state: the other way to aggregate
- Running units sold per product, without recomputing everything
- Why incremental beats recomputing the entire history
- Verifying both aggregates against the known totals
- What this gives you that batch doesn't
- Project: Kiosko's live sales and running inventory signal
06Detecting Kioskos Broken Day Live
- Module introduction: this guide's flagship case
- Session windows: a window that closes on silence
- Designing a "heartbeat" check for a store
- Kiosko's August 10: when S02 goes quiet
- Building the silence detector in PyFlink
- The alert: store gone silent, and when it fires
- Minutes vs. tomorrow: comparing against the daily DAG
- Project: Kiosko's live operational alert
07Cdc The Live Bridge To The Warehouse
- Module introduction: the thread another guide left open on purpose
- What CDC solves that a periodic `SELECT` can't
- Debezium and the write-ahead log
- Setting up Postgres, Kafka Connect, and Debezium with Docker
- Capturing `P002`'s price change as a real event
- Reading the change events' `before`/`after` payload
- Applying the change with PyIceberg's `table.upsert()`
- Project: Kiosko's first live CDC bridge
08Project Kioskos Streaming Pipeline
- Module introduction: the full assembly
- The brief: Kiosko needs to know in minutes, not tomorrow
- Assembling the full pipeline: from producer to alert
- Running the canonical week, and matching every batch guide
- Running the broken day, and watching the alert fire
- Running the CDC bridge end to end
- What Kiosko still needs
- Project: Kiosko's first streaming platform
Reviews
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!