INTERMEDIATE · 8 MODULES · 64 LESSONS
Lakehouse and Iceberg
Turn your Parquet files into Apache Iceberg tables with transactions, queryable history, schema changes without rewriting data, and native upserts.
- For you if
- You already write Parquet or run a warehouse and track change history by hand.
- By the end
- You'll know how to use time travel, evolve schema and partitions, keep a table healthy, and decide when a lakehouse makes sense.
- You need
- Intermediate Python, Parquet, and basic SQL. Spark with Java 17 only for the MERGE module.

Curriculum
01From File Format To Table Format
- Module overview: from a Parquet file to a real Iceberg table
- Four times Parquet alone wasn't enough
- File format vs. table format
- Installing PyIceberg and a local catalog
- Creating your first namespace and table
- Loading Kiosko's fact_orders into Iceberg
- Verifying the same 106.15 total
- Project: Kiosko's first Iceberg table
02Anatomy Of An Iceberg Table
- Module overview: the anatomy of an Iceberg table
- The catalog: a pointer to the current metadata
- The metadata file: schema, partitioning, snapshots
- Manifest lists and manifest files
- The data files: the Parquet you already know
- Inspecting Kiosko's table on disk
- Reading snapshots and history with PyIceberg
- Project: Kiosko's table anatomy, mapped
03Snapshots And Time Travel
- Module overview: snapshots and time travel
- Every write is a new snapshot
- Changing P002 with a plain overwrite
- Capturing the snapshot-id, never hardcoding it
- Time travel: AS OF a snapshot-id
- Recovering P002's history with zero extra columns
- What time travel does not replace
- Project: Kiosko's time-traveled dim_product
04Schema Evolution Without Rewriting
- Module overview: schema evolution without rewriting
- Why overwrite-partition was never atomic
- Adding a column without rewriting data
- Renaming and dropping columns safely
- Populating country in dim_store
- Reading old snapshots after a schema change
- What "ACID" actually guarantees in this exact context
- Project: Kiosko's evolved dim_store
05Hidden Partitioning And Partition Evolution
- Module overview: hidden partitioning and partition evolution
- How Spark partitioned Kiosko by folder
- Hidden partitioning: the same query, without knowing the layout
- Partition transforms: `IdentityTransform`, `BucketTransform`, `DayTransform`
- Partitioning `fact_orders_at_scale`
- Evolving the `PartitionSpec` without rewriting
- Reading old and new partition schemes, together
- Project: Kiosko's `fact_orders_at_scale`, partitioned
06Merge Into And Native Upserts
- Module overview: `MERGE INTO` and native upserts
- Three ways Kiosko already solved this
- Setting up Spark with the Iceberg runtime
- `MERGE INTO` in Spark SQL
- Running P002's change as a MERGE
- PyIceberg's upsert: the Python-native alternative
- Choosing between MERGE in SQL and upsert in Python
- Project: Kiosko's native upsert
07Catalogs Maintenance And Delta Lake By Contrast
08Project Kioskos Lakehouse
- Module overview: the capstone, Kiosko's complete lakehouse
- The brief: Kiosko needs a lakehouse, not seven demos
- Rebuilding the star schema as Iceberg tables
- Reproducing P002's history with time travel
- Evolving and partitioning at scale
- Merging updates the native way
- What Kiosko still needs
- Project: Kiosko's first lakehouse
Reviews
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!