INTERMEDIATE · 8 MODULES · 64 LESSONS

Streaming with Kafka and Flink

Recognize when streaming is worth it and use Kafka and Flink to solve what batch can't, like knowing within minutes that a store stopped reporting.

For you if
You already build batch pipelines and someone is asking whether the business needs real time.
By the end
You'll know how to work with topics and partitions, windows with watermarks, incremental state, and Postgres change capture with Debezium.
You need
Intermediate Python and a working Docker setup. The dbt and lakehouse guides help.
  1. 01When Latency Is The Product8 LESSONS
    1. Module introduction: when latency IS the product
    2. The weak flag: streaming that recomputes batch
    3. What streaming costs your company, and what going without it costs
    4. When latency genuinely IS the product
    5. Installing Kafka in KRaft mode with Docker
    6. Installing PyFlink locally
    7. Proving it: the same 35.3% funnel in Kafka and Flink
    8. Project: Kiosko's first Kafka and Flink "hello world"
  2. 02Kafka Topics Partitions And Offsets8 LESSONS
    1. Module introduction: Kafka's real anatomy
    2. A topic is an ordered, append-only log
    3. Partitions, and why order is only guaranteed within one
    4. Offsets: the bookmark of a consumer
    5. Choosing a Python client: `confluent-kafka` vs. `kafka-python`
    6. Creating Kiosko's `kiosko.orders` topic
    7. Producing and consuming your first message
    8. Project: Kiosko's first Kafka topic
  3. 03Producing And Consuming Kioskos Order Stream8 LESSONS
    1. Module introduction: from fixed file to real stream
    2. From fixed files to a message stream
    3. Keying messages by `store_id`
    4. Building a deterministic, finite producer
    5. Building a consumer that verifies the count
    6. Replaying the canonical week through Kafka
    7. Verifying the same `106.15` total again
    8. Project: Kiosko's order stream in Kafka
  4. 04Event Time Watermarks And Deterministic Windows8 LESSONS
    1. Module introduction: this guide's heart
    2. Processing time vs. event time
    3. Why the wall clock is banned in this guide
    4. Watermarks: how Flink knows it's safe to close a window
    5. Bounded out-of-orderness in PyFlink
    6. Tumbling windows on `order_ts`
    7. Reproducing Kiosko's daily totals with a windowed job
    8. Project: Kiosko's first deterministic window
  5. 05Flink Aggregations Windows Vs Running State8 LESSONS
    1. Module introduction: two ways to aggregate, one real advantage
    2. Revenue per window, per store
    3. Keyed state: the other way to aggregate
    4. Running units sold per product, without recomputing everything
    5. Why incremental beats recomputing the entire history
    6. Verifying both aggregates against the known totals
    7. What this gives you that batch doesn't
    8. Project: Kiosko's live sales and running inventory signal
  6. 06Detecting Kioskos Broken Day Live8 LESSONS
    1. Module introduction: this guide's flagship case
    2. Session windows: a window that closes on silence
    3. Designing a "heartbeat" check for a store
    4. Kiosko's August 10: when S02 goes quiet
    5. Building the silence detector in PyFlink
    6. The alert: store gone silent, and when it fires
    7. Minutes vs. tomorrow: comparing against the daily DAG
    8. Project: Kiosko's live operational alert
  7. 07Cdc The Live Bridge To The Warehouse8 LESSONS
    1. Module introduction: the thread another guide left open on purpose
    2. What CDC solves that a periodic `SELECT` can't
    3. Debezium and the write-ahead log
    4. Setting up Postgres, Kafka Connect, and Debezium with Docker
    5. Capturing `P002`'s price change as a real event
    6. Reading the change events' `before`/`after` payload
    7. Applying the change with PyIceberg's `table.upsert()`
    8. Project: Kiosko's first live CDC bridge
  8. 08Project Kioskos Streaming Pipeline8 LESSONS
    1. Module introduction: the full assembly
    2. The brief: Kiosko needs to know in minutes, not tomorrow
    3. Assembling the full pipeline: from producer to alert
    4. Running the canonical week, and matching every batch guide
    5. Running the broken day, and watching the alert fire
    6. Running the CDC bridge end to end
    7. What Kiosko still needs
    8. Project: Kiosko's first streaming platform
Reviews

What students say

These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.

No approved reviews yet.

Be the first to share your experience!

SEC. 06 · THE STUDIO

Why this studio exists

For decades, building software belonged to technical profiles with years of training, and getting a digital product into the real world took companies months or years. No-code and AI have narrowed that gap, but the knowledge is still held by a few.

Tech Benders exists to open that knowledge up, so business teams and non-technical teams can put their ideas into production with the right technical decisions behind them. AI does what you ask of it. Without a technical base, what it builds may not scale with your company’s ecosystem, and its costs can run away over time.

The team knows the pains of large organizations and the speed at which they have to move. It brings solution architecture, data engineering, product built end to end from back end to front end, and daily work with AI. Software that solves what you need in days, not months, with the cost of AI kept in check.

Founders

The studioRead about Tech Benders →

Write to ushi@techbenders.io

Made with Wislab