A nightly job loads sales records for business date D. Late records for date D keep arriving over the following days and must land in the correct date partition. Reruns after a failure must never double count. Which approach should the team use?
Show answer and explanation
Answer: D
The batch job should recompute and replace every date partition that late records touch, and deduplicate on a business key. Rerunning the same input then produces the same result, so retries never double count, and late rows land in the partition for their business date. Continuous streaming adds complexity that a daily cadence does not need and still requires partition logic. Appending without checks duplicates rows on retries, and global windows with no partition key never align records to business dates.
Why the other options are wrong
- A. A streaming job on 24-hour windows adds operational complexity for a daily use case. Unpartitioned targets also make late records hard to place and expensive to scan.
- B. Appending each file without checks duplicates rows whenever a load is retried. Late records would also remain in whatever partition the append happened to use.
- C. Global windows with no partition key never align records to their business dates. Late data would still need reconciliation, and the approach ignores the daily batch cadence.