INTERMEDIATE · 8 MODULES · 64 LESSONS
Spark and Distributed Processing
Find out when your data stops fitting on one machine and process it with Spark on your laptop, reading the execution plan to see what things cost.
- For you if
- You're fluent in Python and SQL on a single node and wonder if you really need Spark.
- By the end
- You'll know how to read the cost of a shuffle or join, when to cache, and when Spark isn't needed at all.
- You need
- Intermediate Python with DataFrames, SQL with window functions, and Java 17 installed. No cluster or cloud.

Curriculum
01Why Distribute The Single Node Ceiling
- Module introduction: why distribute and the single-node ceiling
- The single-node ceiling: what DuckDB and Polars already solved
- A real cost criterion for when to distribute
- Installing PySpark and Java locally
- Your first SparkSession
- Reading Kiosko's orders with Spark
- Verifying the same forty rows arrive
- Project: Kiosko's first Spark session
02The Spark Execution Model
- Module introduction: the Spark execution model
- The driver and the executors
- RDDs: the original abstraction, seen once and left behind
- The DataFrame API as the modern interface
- Lazy evaluation: transformations vs actions
- Building a transformation chain without running it
- Triggering execution with an action
- Project: Kiosko's first transformation chain
03Rebuilding Fact Orders With The Dataframe Api
- Module introduction: rebuilding `fact_orders` with the DataFrame API
- Reading `orders`, `dim_store`, and `dim_product` with an explicit schema
- Joining facts and dimensions with the DataFrame API
- Computing revenue, the same way, four engines later
- Grouping by store and by product
- Verifying the same 106.15 total
- Writing `fact_orders` as Parquet
- Project: Kiosko's `fact_orders` in Spark
04Partitions And The Cost Of Shuffle
- Module introduction: partitions and the cost of shuffle
- What a partition actually is
- Why `groupBy`, `join`, and `distinct` trigger a shuffle
- Generating `kiosko_orders_at_scale`, deterministically
- Reading the shuffle in `.explain()` and the Spark UI
- `repartition()` vs `coalesce()`
- `spark.sql.shuffle.partitions` and Adaptive Query Execution
- Project: Kiosko at scale, partitioned
05Joins And Window Functions At Scale
- Module introduction: joins and window functions at scale
- Broadcast join vs shuffle join
- Reading a broadcast join in `.explain()`
- Forcing and reading a `SortMergeJoin`
- Window functions: `partitionBy` and `orderBy`
- A running revenue total per store
- Ranking top products per store per day
- Project: Kiosko's scaled joins and rankings
06Catalyst Explain And Caching
07Parquet At Scale And Python Udfs
- Module introduction: Parquet at scale and Python UDFs
- Parquet at scale: partitioned writes
- Predicate and column pushdown
- Why plain Python UDFs are slow: `cloudpickle` and the executor
- `pandas_udf` and Arrow-based vectorization
- Rewriting `margin_category` as a `pandas_udf`
- Naming Structured Streaming, without building it
- Project: Kiosko's partitioned Parquet and a vectorized UDF
08Project Kioskos Distributed Pipeline
- Module introduction: Kiosko's distributed capstone
- The brief: Kiosko at scale
- Assembling the distributed pipeline, end to end
- Verifying correctness against the original `106.15`
- Choosing partitioning, caching, and `JOIN` strategy, with criteria
- The decision tree: does Kiosko actually need Spark?
- What Kiosko still needs
- Project: Kiosko's first distributed pipeline
Reviews
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!