Ingesting and processing data
Practise only this domain. 110 questions with explanations, 20 at a time.
Start this domainSample questions
- A retailer's MySQL order database must feed BigQuery with inserts, updates, and deletes within a few minutes. The team wants no custom code that tracks binary log positions. Which managed service should the data engineer choose?Medium
- A payments service publishes ledger events for thousands of accounts and needs each account's events handled in sequence. What guarantee do Pub/Sub ordering keys provide?Medium
- A nightly job loads sales records for business date D. Late records for date D keep arriving over the following days and must land in the correct date partition. Reruns after a failure must never double count. Which approach should the team use?Hard
- A streaming job uses event-time windows, and events sometimes arrive several minutes late. What does the watermark tell the pipeline?Hard
- A subscription uses the default message retention period. A consumer team was away for eight days and never acknowledged the messages that arrived during that time. What happens to those unacknowledged messages?Medium
- A team receives JSON responses from a vendor API. Each response contains nested objects and arrays of line items, and the team needs one row per line item in BigQuery at high volume. Which approach is best?Medium
- A Dataflow pipeline counts device events per device ID over hourly windows. A few popular devices produce enormous volumes, and the workers handling those keys run out of memory because the pipeline gathers every value into a list before counting. Which change addresses this?Hard
- A logistics firm receives shipment events in one record type and carrier updates in another. The downstream sink accepts a single common record type, and the pipeline must combine both sources into one collection. Which steps achieve this?Easy
- A logistics platform's order-events consumer must be offline for a two-day database migration. Events published during the outage still need processing afterward, and the producers cannot be changed. What should the team do?Medium
- A Dataflow streaming job enriches events with location data from a third-party lookup API that enforces a request quota. Which design keeps the job within that quota while keeping throughput high?Hard
- A data platform team needs a workflow that runs a validation step, branches to a repair path or a publish path depending on the result, and offers a graphical view of each run's history. Which service fits?Easy
- Before an engineering group writes any code for a clickstream pipeline, it documents where events come from, where results go, which transformations apply, how encryption is handled, and how jobs are scheduled. Which phase of pipeline work is this?Easy
- A logistics company drops CSV manifests into a storage bucket at unpredictable times during the day. Each new manifest should start a load into the analytics warehouse within seconds, without any idle polling. What should the engineer set up?Easy