A Dataflow pipeline counts device events per device ID over hourly windows. A few popular devices produce enormous volumes, and the workers handling those keys run out of memory because the pipeline gathers every value into a list before counting. Which change addresses this?
Show answer and explanation
Answer: D
A GroupByKey followed by iteration forces each key's complete set of values into memory on one worker, which is what runs out. A Combine with a CombineFn stores only a small accumulator per key, and workers merge partial accumulators before producing the final count, so memory stays small regardless of event volume. For extremely hot keys, hot-key fanout splits the combining work across intermediate keys before the final merge.
Why the other options are wrong
- A. Persistent disks hold shuffle and scratch data, not the in-memory value lists of a hot key, so bigger disks do not prevent the memory exhaustion.
- B. Shuffle capacity changes how fast intermediate data moves between stages, but a single hot key still gathers all of its values onto one worker.
- C. Fewer workers leave hot keys with the same load on fewer machines, which increases memory pressure per worker and makes the failures more likely.