A ride-sharing platform streams trip events into BigQuery. Events with missing driver IDs, wrong timestamp types, or fares outside a plausible range must never reach the reporting tables. Which pipeline design handles this?
Show answer and explanation
Answer: C
Validating inside the streaming pipeline, before the write, keeps invalid trip events out of the reporting tables. Failed records can be routed to a dead-letter destination such as a Cloud Storage bucket or a quarantine table, where engineers can inspect and reprocess them. Loading first and deleting later exposes reports to bad data and adds work. Turning checks off trades correctness for speed, and quarterly validation finds problems long after reports have used the bad data.
Why the other options are wrong
- A. Deleting rows after loading means bad data is visible to reports in the meantime, and the deletions are costly to audit. The approach is reactive rather than preventive.
- B. Skipping checks lowers latency, but the reporting tables would then contain malformed or implausible fares, undermining every downstream metric.
- D. A quarterly job would leave bad events in the reporting tables for months. Validation has to happen on arrival to protect the reports, not on a distant schedule.