INTERMEDIATE · 8 MODULES · 64 LESSONS

Spark and Distributed Processing

Find out when your data stops fitting on one machine and process it with Spark on your laptop, reading the execution plan to see what things cost.

For you if
You're fluent in Python and SQL on a single node and wonder if you really need Spark.
By the end
You'll know how to read the cost of a shuffle or join, when to cache, and when Spark isn't needed at all.
You need
Intermediate Python with DataFrames, SQL with window functions, and Java 17 installed. No cluster or cloud.
  1. 01Why Distribute The Single Node Ceiling8 LESSONS
    1. Module introduction: why distribute and the single-node ceiling
    2. The single-node ceiling: what DuckDB and Polars already solved
    3. A real cost criterion for when to distribute
    4. Installing PySpark and Java locally
    5. Your first SparkSession
    6. Reading Kiosko's orders with Spark
    7. Verifying the same forty rows arrive
    8. Project: Kiosko's first Spark session
  2. 02The Spark Execution Model8 LESSONS
    1. Module introduction: the Spark execution model
    2. The driver and the executors
    3. RDDs: the original abstraction, seen once and left behind
    4. The DataFrame API as the modern interface
    5. Lazy evaluation: transformations vs actions
    6. Building a transformation chain without running it
    7. Triggering execution with an action
    8. Project: Kiosko's first transformation chain
  3. 03Rebuilding Fact Orders With The Dataframe Api8 LESSONS
    1. Module introduction: rebuilding `fact_orders` with the DataFrame API
    2. Reading `orders`, `dim_store`, and `dim_product` with an explicit schema
    3. Joining facts and dimensions with the DataFrame API
    4. Computing revenue, the same way, four engines later
    5. Grouping by store and by product
    6. Verifying the same 106.15 total
    7. Writing `fact_orders` as Parquet
    8. Project: Kiosko's `fact_orders` in Spark
  4. 04Partitions And The Cost Of Shuffle8 LESSONS
    1. Module introduction: partitions and the cost of shuffle
    2. What a partition actually is
    3. Why `groupBy`, `join`, and `distinct` trigger a shuffle
    4. Generating `kiosko_orders_at_scale`, deterministically
    5. Reading the shuffle in `.explain()` and the Spark UI
    6. `repartition()` vs `coalesce()`
    7. `spark.sql.shuffle.partitions` and Adaptive Query Execution
    8. Project: Kiosko at scale, partitioned
  5. 05Joins And Window Functions At Scale8 LESSONS
    1. Module introduction: joins and window functions at scale
    2. Broadcast join vs shuffle join
    3. Reading a broadcast join in `.explain()`
    4. Forcing and reading a `SortMergeJoin`
    5. Window functions: `partitionBy` and `orderBy`
    6. A running revenue total per store
    7. Ranking top products per store per day
    8. Project: Kiosko's scaled joins and rankings
  6. 06Catalyst Explain And Caching8 LESSONS
    1. Module introduction: Catalyst, `.explain()`, and caching
    2. The Catalyst optimizer's phases
    3. Reading `.explain()` in its different modes
    4. Adaptive Query Execution
    5. When caching actually helps
    6. `persist()` and storage levels
    7. When caching only costs memory
    8. Project: Kiosko's optimized query plan
  7. 07Parquet At Scale And Python Udfs8 LESSONS
    1. Module introduction: Parquet at scale and Python UDFs
    2. Parquet at scale: partitioned writes
    3. Predicate and column pushdown
    4. Why plain Python UDFs are slow: `cloudpickle` and the executor
    5. `pandas_udf` and Arrow-based vectorization
    6. Rewriting `margin_category` as a `pandas_udf`
    7. Naming Structured Streaming, without building it
    8. Project: Kiosko's partitioned Parquet and a vectorized UDF
  8. 08Project Kioskos Distributed Pipeline8 LESSONS
    1. Module introduction: Kiosko's distributed capstone
    2. The brief: Kiosko at scale
    3. Assembling the distributed pipeline, end to end
    4. Verifying correctness against the original `106.15`
    5. Choosing partitioning, caching, and `JOIN` strategy, with criteria
    6. The decision tree: does Kiosko actually need Spark?
    7. What Kiosko still needs
    8. Project: Kiosko's first distributed pipeline
Reviews

What students say

These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.

No approved reviews yet.

Be the first to share your experience!

SEC. 06 · THE STUDIO

Why this studio exists

For decades, building software belonged to technical profiles with years of training, and getting a digital product into the real world took companies months or years. No-code and AI have narrowed that gap, but the knowledge is still held by a few.

Tech Benders exists to open that knowledge up, so business teams and non-technical teams can put their ideas into production with the right technical decisions behind them. AI does what you ask of it. Without a technical base, what it builds may not scale with your company’s ecosystem, and its costs can run away over time.

The team knows the pains of large organizations and the speed at which they have to move. It brings solution architecture, data engineering, product built end to end from back end to front end, and daily work with AI. Software that solves what you need in days, not months, with the cost of AI kept in check.

Founders

The studioRead about Tech Benders →

Write to ushi@techbenders.io

Made with Wislab