INTERMEDIATE · 8 MODULES · 64 LESSONS

Python for Data Engineering

Turn a data script into a Python package an engineering team would accept for review: installable, tested, with useful logs, retries, and fast transformations.

For you if
Your pipelines work, but they have no tests, clear logs, or failure handling.
By the end
You'll know how to package with uv, test with pytest, log in JSON, and transform data with DuckDB and Polars instead of pandas.
You need
Intermediate Python and a pipeline written as a script (from the previous guide). Just uv; everything runs locally.
  1. 01Packaging A Project With Uv8 LESSONS
    1. Guide overview: from a script you run to a package you install
    2. Why a script is not the same thing as a package
    3. uv and the modern Python toolchain
    4. pyproject.toml and the src/ layout
    5. Turning Kiosko's script into a package
    6. Imports, modules, and the __init__ file
    7. Running the package with uv run
    8. Project: Kiosko's installable pipeline
  2. 02Testing Pipelines With Pytest8 LESSONS
    1. Module overview: from "it works on my machine" to "a machine confirms it"
    2. Why data pipelines need tests
    3. Writing your first pytest test
    4. Fixtures and Kiosko's fixed test data
    5. Testing the extract and transform steps
    6. Parametrize and testing the quality gate
    7. Arrange-Act-Assert and organizing a growing suite
    8. Project: Kiosko's test suite
  3. 03Structured Logging For Pipelines8 LESSONS
    1. Module overview: from print() to a record a machine can read
    2. Why print() doesn't scale
    3. Python's logging module: levels and handlers
    4. JSON logs, and why a machine can read them
    5. Adding context with structlog
    6. Logging a pipeline run, from start to finish
    7. What to log and what to never log
    8. Project: Kiosko's structured record of a complete run
  4. 04Error Handling And Retries8 LESSONS
    1. Module overview: what to retry, what to quarantine
    2. Errors that should retry, and errors that never should
    3. Custom exceptions for a pipeline
    4. Retrying with tenacity
    5. Backoff, jitter, and when to give up
    6. Quarantine instead of crash, revisited
    7. Testing retry logic deterministically
    8. Project: Kiosko's resilient load step
  5. 05Duckdb As The Daily Engine8 LESSONS
    1. Module overview: DuckDB as the daily engine
    2. DuckDB beyond the preview
    3. Querying Parquet and CSV directly with SQL
    4. The relational API and persistence
    5. Replacing Python joins with SQL
    6. DuckDB and the Polars and Arrow ecosystem
    7. Testing SQL transformations with pytest
    8. Mini-project: Kiosko's DuckDB transform layer
  6. 06Polars Lazy And Eager8 LESSONS
    1. Module overview: Polars, eager and lazy
    2. Why Polars replaced pandas for pipelines
    3. Eager DataFrames: a first look
    4. LazyFrames and the query plan
    5. Expressions, group_by, and joins
    6. Scanning Parquet and out-of-core data
    7. Choosing DuckDB or Polars
    8. Mini-project: Kiosko's Polars transform layer
  7. 07Shipping The Package8 LESSONS
    1. Module overview: the package that ships
    2. A command-line entry point for the pipeline
    3. Configuration without hardcoded values
    4. Wiring logging, retries, and engines together
    5. An integration test for the whole pipeline
    6. Proving idempotency with a test, not a manual rerun
    7. What a code reviewer would still ask for
    8. Mini-project: Kiosko's shippable package
  8. 08Project Kioskos Production Pipeline8 LESSONS
    1. Module overview: Kiosko's capstone
    2. The brief: Kiosko needs a pipeline a team can own
    3. Assembling the complete package, end to end
    4. Running the complete test suite
    5. Running the pipeline with structured logs
    6. DuckDB vs Polars vs the original script
    7. What Kiosko still needs
    8. Mini-project: Kiosko's first production package
Reviews

What students say

These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.

No approved reviews yet.

Be the first to share your experience!

SEC. 06 · THE STUDIO

Why this studio exists

For decades, building software belonged to technical profiles with years of training, and getting a digital product into the real world took companies months or years. No-code and AI have narrowed that gap, but the knowledge is still held by a few.

Tech Benders exists to open that knowledge up, so business teams and non-technical teams can put their ideas into production with the right technical decisions behind them. AI does what you ask of it. Without a technical base, what it builds may not scale with your company’s ecosystem, and its costs can run away over time.

The team knows the pains of large organizations and the speed at which they have to move. It brings solution architecture, data engineering, product built end to end from back end to front end, and daily work with AI. Software that solves what you need in days, not months, with the cost of AI kept in check.

Founders

The studioRead about Tech Benders →

Write to ushi@techbenders.io

Made with Wislab