A payments platform streams card-transaction events through Pub/Sub into BigQuery. Duplicate records would corrupt account balances, so each event must land exactly once. Which pipeline design best meets this requirement?
Show answer and explanation
Answer: A
Pub/Sub delivers messages at least once, so duplicates must be handled downstream. Dataflow's exactly-once streaming mode deduplicates and checkpoints processing, and writing through the BigQuery Storage Write API with committed writes gives atomic, offset-tracked appends. Together they provide the strongest end-to-end guarantee among the listed options. Adding a unique transaction identifier as a key further protects against duplicates introduced by upstream retries.
Why the other options are wrong
- B. Push subscriptions redeliver on timeout or error, and streaming inserts without insert IDs are not guaranteed to deduplicate. Retries can therefore produce duplicate rows.
- C. Best-effort checks in a stateless function can race between concurrent invocations and miss duplicates. That is not exactly-once, which balance accuracy requires.
- D. A nightly reload leaves data stale for a day and does not prevent duplicates from arriving during the load. It is batch processing, not an exactly-once streaming design.