An Airflow DAG must hold until a nightly export lands in a Cloud Storage prefix. Several such waits run at the same time, and the team wants idle waits to stop occupying executor slots. What should the sensor setup use?
Show answer and explanation
Answer: C
A reschedule-mode sensor checks the condition, then releases its worker slot and is rescheduled for the next check. Idle waits therefore do not block capacity that other tasks need. Default poke mode keeps the slot occupied for the whole wait, and a shorter interval only makes it check more often while still holding the slot. A sleep loop blocks the worker entirely, and retrying a failing task is a misuse that ends in a failed run rather than a wait.
Why the other options are wrong
- A. Poke mode holds the worker slot for the entire wait, so many concurrent waits can starve other tasks. A shorter interval changes how often the check runs, not how long the slot is held.
- B. A sleep loop keeps the worker occupied for the whole wait, which is the exact problem the team wants to avoid. It also gives no built-in timeout or reschedule behavior.
- D. Repeated retries of a failing task do not wait for an object to appear. Each failure uses up the retry budget, and the task ends in failure once the retries run out.