Table of Contents
Why Does MapReduce Partitioning Send Same Keys to One Reducer?
Partitioning in Hadoop MapReduce is vital as it directs all values for the same key to a single reducer via hash functions, enabling correct aggregation and preventing data skew in distributed processing.
Question
Why is partitioning critical in MapReduce execution?
A. It controls the number of mappers
B. It ensures that the same key always goes to the same reducer
C. It determines the block size for HDFS
D. It merges outputs from reducers into one file
Answer
B. It ensures that the same key always goes to the same reducer
Explanation
Partitioning is critical in MapReduce execution because the partitioner determines which reducer receives specific intermediate key-value pairs from mappers, using a hash function on the key (default: key.hashCode() % numReducers) to guarantee all values for the same key are routed consistently to one reducer for correct grouping and aggregation. Without proper partitioning, values for identical keys could scatter across multiple reducers, breaking the semantic guarantee that reducers process complete sets of values per key, leading to incorrect results in operations like word count or joins. Custom partitioners extend this for advanced control (e.g., range partitioning), but the core role remains ensuring key locality across the shuffle phase, unrelated to mapper counts, HDFS blocks, or output merging.