From MapReduce to Spark: Execution and Programming Models

Tue, 15 Jun 2021 11:00:00 +0900

MapReduce gives Hadoop a simple batch execution model: map tasks, a shuffle, an optional reduce, and durable output between jobs.¹ That model is easy to reason about when a job reads input, produces output, and stops. It becomes more expensive when a pipeline parses the same records, joins or aggregates them, feeds the result into another job, and then repeats that pattern for model training or reporting.

Spark changed which costs engineers had to manage. It can keep a dependent computation inside one application instead of forcing every intermediate result through replicated storage, record lineage so lost partitions can be recomputed, hold reused data resident under an explicit storage policy, reuse executor Java Virtual Machines (JVMs) across many tasks, and apply relational optimization when work is expressed as SQL or DataFrames. Each mechanism trades an old cost for a new tuning problem. For many iterative, multi-stage, and structured workloads, Spark is faster; the size of that advantage, and the exceptions to it, depend on the plan and hardware.

Distributed Systems on Boyang Yue

From MapReduce to Spark: Execution and Programming Models