A data science group runs one Spark job roughly twice a week and must keep compute costs low. The cluster should consume Dataproc capacity only while the job runs. Which setup fits best?
Show answer and explanation
Answer: D
A Dataproc workflow template with a managed cluster provisions an ephemeral cluster for the run, executes the job, and deletes the cluster when it finishes. Billing covers only the period the cluster exists. For intermittent Spark workloads, job-scoped ephemeral clusters are far cheaper than keeping any persistent cluster running between runs.
Why the other options are wrong
- A. An always-on cluster bills continuously, even on the days when no job runs. Sizing it for peak load makes idle cost even higher than a persistent cluster that scales down.
- B. A scaled-down persistent cluster still incurs charges for its master and remaining workers around the clock. It is cheaper than peak sizing but still pays for idle time that an ephemeral cluster avoids.
- C. Cloud Functions cannot run distributed Spark jobs. Its execution time and memory limits make it unsuitable for large Spark transformations regardless of cost.