PDE question patterns

Dataflow, BigQuery, Pub/Sub, Workflows or Serverless for Apache Spark rather than clusters you run.

Managed services remove patching, sizing and scaling work.

Run the jobs on Dataproc, ideally ephemeral clusters reading Cloud Storage.

Dataproc runs open-source frameworks unchanged.

Ephemeral Dataproc with Spot secondary workers, FlexRS or batch-priority queries.

You pay only while the work runs, and fault-tolerant jobs tolerate preemption.

Use customer-managed keys in Cloud KMS on the datasets and buckets.

Disabling the key makes the data unreadable.

Use an external key manager through Cloud EKM.

The key material stays outside Google.

Put BigQuery and Storage inside a service perimeter.

IAM alone does not stop exfiltration by an authorized user.

Run DLP discovery and de-identify with masking, tokenization or date shifting.

DLP detects infoTypes and offers reversible and one-way transforms.

Attach a policy tag with a masking rule and grant Masked Reader.

Users get masked values instead of errors.

Create row-level access policies or an authorized view using SESSION_USER().

Filters rows per principal without copies.

Create a view in another dataset and authorize it on the source dataset.

The view reads with its own access.

Grant bigquery.jobUser in the running project and dataViewer on the data.

Running a job and reading data are separate permissions.

Use DMS continuous replication then cut over.

It replicates changes until you promote.

Use Datastream CDC to BigQuery.

Serverless, log-based change capture.

Configure a transfer on a schedule.

Managed connectors; no code.

Ship data offline with Transfer Appliance.

Network transfer would take weeks.

Schedule STS jobs, with agents for on-premises.

Incremental, managed, verified.

Use a Pub/Sub BigQuery subscription.

No pipeline needed.

Event-time windows with allowed lateness and a late trigger.

Watermarks decide completeness; lateness keeps windows open.

Use session windows with a gap duration.

Windows close on inactivity per key.

Sliding window size 10 min, period 1 min.

Overlapping windows.

Dedupe with ids, use Storage Write API exactly-once, idempotent MERGE.

Pub/Sub is at-least-once.

Drain the Dataflow job.

Cancel drops buffered data.

Configure dead-letter with max delivery attempts.

Isolates failing messages.

Use hot-key fanout or key salting.

Spreads skewed aggregations.

Build the pipeline in Data Fusion.

Drag-and-drop on Dataproc.

Define SQLX with assertions.

Version-controlled ELT in BigQuery.

Write Airflow DAGs.

Handles dependencies and backfills.

Use Bigtable with a well-designed row key.

Wide-column, high throughput.

Promote a field, salt, or reverse; avoid timestamp-first keys.

Sequential keys hotspot a tablet.

Use multi-region Spanner.

External consistency, 99.999%.

Move to AlloyDB with columnar engine.

Faster Postgres-compatible.

Enable Autoclass on the bucket.

Moves objects by access automatically.

Set a retention policy and lock it.

Locking makes it irreversible.

Partition on the date column and require partition filter.

Prunes scanned bytes.

Cluster on up to four columns.

Block pruning cuts bytes.

Consolidate into one partitioned table.

Sharding hurts metadata and speed.

Create materialized views and reserve BI Engine.

Precomputed and in-memory.

Use STRUCT and ARRAY.

Avoids join cost.

Avoid SELECT *; filter on partitions; LIMIT does not help.

Billing follows columns scanned.

Publish an exchange listing; subscribers get linked datasets.

Live read-only reference.

Use Analytics Hub clean rooms with analysis rules.

Rules block raw access.

Use ML.GENERATE_EMBEDDING or AI functions.

Calls Vertex AI from SQL.

Embed, create a vector index, run VECTOR_SEARCH.

Nearest-neighbour retrieval for RAG.

Editions with baseline slots and assignments.

Dedicated slots isolate it.

Run it as a batch job.

Waits for idle capacity.

Create per job and delete, or use scheduled deletion.

No idle cost.

Alert on oldest_unacked_message_age.

Shows stuck messages.

Check total_bytes_billed and billing export; set max bytes billed.

Finds the costly queries.

Query FOR SYSTEM_TIME AS OF or restore a snapshot.

7-day window by default.

HA plus cross-region read replica to promote.

HA covers zones only.