A retailer's MySQL order database must feed BigQuery with inserts, updates, and deletes within a few minutes. The team wants no custom code that tracks binary log positions. Which managed service should the data engineer choose?
Show answer and explanation
Answer: A
Datastream is a serverless change data capture service. It reads the source database's change logs, such as MySQL binary logs, and continuously replicates inserts, updates, and deletes to BigQuery or Cloud Storage with low latency. Because it is managed, the team does not write or maintain log-position tracking. Periodic Sqoop extracts add latency and can miss intermediate changes. Storage Transfer Service copies files between storage systems, and an application-driven Pub/Sub feed depends on code changes in every writer.
Why the other options are wrong
- B. Sqoop-based Dataproc jobs run periodic extracts, so changes between runs are delayed or missed. The team would also have to build and operate the incremental logic itself.
- C. Storage Transfer Service moves files between storage systems on a schedule. It does not read database change logs or capture row-level inserts, updates, and deletes.
- D. A feed published from application code captures only the changes the code remembers to emit. It misses direct database writes and requires changes in every writer.