A data pipeline must wait for a new file to land in S3, start a Glue job, retry that job on failure, and send a notification when it completes. Which service coordinates these steps most suitably?
Show answer and explanation
Answer: D
AWS Step Functions coordinates steps across services with a state machine. It can check for objects, start a Glue job and wait for it to finish, apply retry and catch rules, and then publish a notification, while keeping an execution history for troubleshooting. The other options either route events without sequencing or retries, run compute jobs, or require writing and operating custom decider code.
Why the other options are wrong
- A. SWF can coordinate such flows, but it needs a custom decider program and is the older model. Step Functions provides the same coordination without that code.
- B. Batch schedules compute jobs on managed resources. It does not sequence a file check, a Glue run and a notification as one visible workflow.
- C. EventBridge routes events to targets, often in parallel. It has no built-in ordering of waits, retries and branching across the steps.