INTERMEDIATE · 8 MODULES · 64 LESSONS
Python for Data Engineering
Turn a data script into a Python package an engineering team would accept for review: installable, tested, with useful logs, retries, and fast transformations.
- For you if
- Your pipelines work, but they have no tests, clear logs, or failure handling.
- By the end
- You'll know how to package with uv, test with pytest, log in JSON, and transform data with DuckDB and Polars instead of pandas.
- You need
- Intermediate Python and a pipeline written as a script (from the previous guide). Just uv; everything runs locally.

Curriculum
01Packaging A Project With Uv
- Guide overview: from a script you run to a package you install
- Why a script is not the same thing as a package
- uv and the modern Python toolchain
- pyproject.toml and the src/ layout
- Turning Kiosko's script into a package
- Imports, modules, and the __init__ file
- Running the package with uv run
- Project: Kiosko's installable pipeline
02Testing Pipelines With Pytest
- Module overview: from "it works on my machine" to "a machine confirms it"
- Why data pipelines need tests
- Writing your first pytest test
- Fixtures and Kiosko's fixed test data
- Testing the extract and transform steps
- Parametrize and testing the quality gate
- Arrange-Act-Assert and organizing a growing suite
- Project: Kiosko's test suite
03Structured Logging For Pipelines
- Module overview: from print() to a record a machine can read
- Why print() doesn't scale
- Python's logging module: levels and handlers
- JSON logs, and why a machine can read them
- Adding context with structlog
- Logging a pipeline run, from start to finish
- What to log and what to never log
- Project: Kiosko's structured record of a complete run
04Error Handling And Retries
- Module overview: what to retry, what to quarantine
- Errors that should retry, and errors that never should
- Custom exceptions for a pipeline
- Retrying with tenacity
- Backoff, jitter, and when to give up
- Quarantine instead of crash, revisited
- Testing retry logic deterministically
- Project: Kiosko's resilient load step
05Duckdb As The Daily Engine
- Module overview: DuckDB as the daily engine
- DuckDB beyond the preview
- Querying Parquet and CSV directly with SQL
- The relational API and persistence
- Replacing Python joins with SQL
- DuckDB and the Polars and Arrow ecosystem
- Testing SQL transformations with pytest
- Mini-project: Kiosko's DuckDB transform layer
06Polars Lazy And Eager
07Shipping The Package
- Module overview: the package that ships
- A command-line entry point for the pipeline
- Configuration without hardcoded values
- Wiring logging, retries, and engines together
- An integration test for the whole pipeline
- Proving idempotency with a test, not a manual rerun
- What a code reviewer would still ask for
- Mini-project: Kiosko's shippable package
08Project Kioskos Production Pipeline
- Module overview: Kiosko's capstone
- The brief: Kiosko needs a pipeline a team can own
- Assembling the complete package, end to end
- Running the complete test suite
- Running the pipeline with structured logs
- DuckDB vs Polars vs the original script
- What Kiosko still needs
- Mini-project: Kiosko's first production package
Reviews
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!