Ingesting and processing the data

Planning the pipeline

• Sources and sinks: files (Cloud Storage), databases (CDC through Datastream), events (Pub/Sub, Kafka), SaaS and logs. Sinks are usually BigQuery, Cloud Storage, Bigtable, Spanner or another topic.
• Latency tiers: batch (hours), micro-batch (minutes, scheduled loads and queries), streaming (seconds). Do not build streaming when the business reads a report once a day; load jobs are free and cheaper than streaming inserts.
• Transformation logic: SQL (BigQuery, Dataform) where possible; Beam when you need streaming windows, custom code or non-SQL steps; Spark for existing code and ML libraries.
• Networking: run workers in a VPC subnet in the data’s region; turn off external IPs and enable Private Google Access; use firewall rules for worker-to-worker traffic (Dataflow needs TCP ports 12345 and 12346 between workers; use a network tag); Cloud NAT for needed internet access; Shared VPC for host/service project setups. Cross-region traffic costs money and adds latency.
• Encryption: data is encrypted in transit and at rest by default; use CMEK on Dataflow, Pub/Sub, buckets and BigQuery where policy demands it.

Pub/Sub

Global, serverless messaging for decoupled ingestion. Topics receive, subscriptions deliver.

Key behaviours:

• At-least-once delivery by default: consumers must tolerate duplicates. Exactly-once delivery is an option on pull subscriptions (regional) and removes redelivery after a successful ack; it does not stop a publisher sending duplicates.
• Ack deadline defaults to 10 seconds (range 10 to 600); extend it for slow work. Unacked messages are redelivered. Retry policy can use exponential backoff.
• Message ordering: set an ordering key and enable ordering on the subscription; ordering is per key, within a region. Hot keys limit throughput.
• Dead-letter topic after N failed delivery attempts (5 to 100) keeps poison messages from blocking.
• Retention: subscriptions keep unacked messages 7 days by default (up to 31 days); a topic can retain messages up to 31 days; seek to a timestamp or a snapshot to replay. Acked messages are kept only if retention is enabled.
• Filters on a subscription drop non-matching messages cheaply (attributes only). Schemas (Avro, Protocol Buffers) validate messages on publish.
• Single message transforms (SMTs) run JavaScript user-defined functions or AI inference on messages as they are published, for light cleaning and enrichment without a pipeline.
• Limits: message up to 10 MB; publish batching raises throughput; publisher flow control protects clients. Lite is retired; for Kafka-style semantics at scale use Managed Service for Apache Kafka.
• Choose Pub/Sub vs Kafka: Pub/Sub is serverless, auto-scaling, global; Kafka when you must keep Kafka APIs, consumer groups, long-term compacted logs or a migration of existing Kafka clients.

Dataflow and Apache Beam

Dataflow is the managed runner for Apache Beam (Java, Python, Go). One programming model for batch and streaming. Autoscaling, dynamic work rebalancing, no cluster to manage.

Beam model

• PCollection (data), PTransform (step), Pipeline. Core transforms: ParDo/Map, GroupByKey, Combine, Flatten, Partition, side inputs (small lookup data broadcast to workers), side outputs (tagged outputs, used for dead letters).
• Bounded (batch) and unbounded (streaming) sources. The same code processes both.

Windowing, triggers and late data

• Late data question? The answer combines event-time windows, allowed lateness and a late-data trigger, not a bigger machine. Reading from Pub/Sub, make sure the timestamp attribute is the event time, otherwise the publish time is used.
• Exactly-once processing: Dataflow guarantees each element’s effect is applied exactly once inside the pipeline (for streaming, streaming mode: exactly-once, the default). Sinks must be idempotent or transactional: BigQuery Storage Write API gives exactly-once with committed streams; legacy insertAll is best effort with insertId deduplication for about a minute. At-least-once streaming mode is cheaper and lower latency if duplicates are acceptable.
• Deduplication: Pub/Sub message id (withIdAttribute) to remove duplicates published by the sender.

Operating Dataflow

• Templates: classic (staged JSON) and Flex Templates (container image in Artifact Registry; recommended; allow parameter flexibility). Google provides templates: Pub/Sub to BigQuery, Pub/Sub to Cloud Storage, Cloud Storage text to BigQuery, Datastream to BigQuery, JDBC to BigQuery, Bigtable and Spanner exports, Data Masking with DLP. Use them to avoid writing code.
• Streaming Engine moves state and shuffle off workers to the service (less worker memory, faster scaling). Dataflow Shuffle does the same for batch. FlexRS schedules batch jobs flexibly (delay up to 6 hours) on a mix of Spot and standard VMs for lower cost. Dataflow Prime adds vertical autoscaling and right fitting.
• Autoscaling: horizontal by backlog and CPU; set maxNumWorkers. Streaming jobs scale on backlog.
• Update a running streaming job (replace in place with compatible changes, --update and a transform name mapping) vs drain (finish in-flight data, close windows, then stop) vs cancel (stop immediately, data lost). Snapshots save streaming state for rollback or migration.
• Troubleshooting: hot keys (skewed GroupByKey; fix with Combine.withHotKeyFanout or key salting), slow sinks (backpressure), system lag and data freshness metrics, worker out-of-memory (bigger machine or Streaming Engine), quota limits, permission errors (worker service account missing dataflow.worker), and regional mismatches.
• Dataflow ML and enrichment: RunInference runs models (including Vertex AI and Hugging Face) inside the pipeline, and the enrichment transform looks up reference data from Bigtable or BigQuery per element.

Dataproc and the Hadoop ecosystem

Managed Spark, Hadoop, Hive, Flink and Presto/Trino on Compute Engine.

• Ephemeral (job-based) clusters: create, run, delete (Composer or workflow templates). Cheapest and recommended; store data in Cloud Storage via the GCS connector, not HDFS. Persistent clusters only for shared interactive use or constant load.
• Secondary workers: preemptible or Spot VMs add cheap compute (no HDFS storage); keep primary workers on regular VMs. Autoscaling policies add and remove workers from YARN metrics. Enhanced flexibility mode helps avoid shuffle loss when scaling.
• Serverless for Apache Spark (Dataproc Serverless): submit batches or interactive sessions with no cluster to size; per-second billing; good for sporadic Spark jobs.
• Initialization actions, custom images, optional components (Jupyter, Zeppelin), Component Gateway for web UIs, Dataproc Metastore (managed Hive metastore shared by clusters), workflow templates (DAG of jobs on a managed cluster), Dataproc on GKE, and BigQuery connector for Spark to read and write BigQuery (through the Storage Read/Write API; Spark BigQuery connector).
• Migrating Hadoop: lift Spark/Hive jobs unchanged to Dataproc, then replace HDFS with Cloud Storage, then consider moving SQL workloads to BigQuery. Use BigLake or external tables so Spark and BigQuery share files.
• When Dataproc over Dataflow: existing Spark/Hadoop code and libraries, special open-source frameworks, ML notebooks. When Dataflow: new pipelines, streaming, minimal operations.

Cloud Data Fusion

Visual, code-free ETL on CDAP; deploys pipelines that run on ephemeral Dataproc clusters (or other compute profiles). Editions: Developer, Basic, Enterprise (Enterprise adds streaming, triggers, and more). Has Wrangler for interactive cleansing with directives, plugins and many connectors (JDBC, SAP, Salesforce, Kafka, BigQuery), lineage, and replication for CDC. Choose it when the team wants no-code, a rich connector library or reusable drag-and-drop pipelines; choose Dataflow templates when code-free but Google-provided; choose Dataform for SQL.

Batch ingestion and BigQuery ingestion

• Streaming data lands in a streaming buffer; very recent rows may not be available to certain operations (table copy, DML conflicts in old API). Use partitioned tables with _PARTITIONTIME or a timestamp column; late events land in their event-time partition.
• Schema: Avro/Parquet carry schema; set schema_update_options (ALLOW_FIELD_ADDITION, ALLOW_FIELD_RELAXATION) for evolving sources; use --autodetect carefully; use JSON column type for semi-structured payloads.
• Cleansing: handle nulls, types, deduplication (ROW_NUMBER() OVER (PARTITION BY id ORDER BY ts DESC) = 1 or QUALIFY), standardise formats, quarantine bad rows (max_bad_records, dead-letter tables).

Transformations by mode

• Batch: Dataflow batch, Dataproc/Spark, BigQuery SQL and Dataform, Data Fusion. ELT (load raw, transform in BigQuery) is the default modern pattern.
• Streaming: Pub/Sub plus Dataflow plus BigQuery/Bigtable. Windowed aggregates, joins with side inputs, stateful processing, sessionisation. Streaming SQL in BigQuery (continuous queries) can transform and export rows as they arrive without Beam; Dataflow SQL is deprecated in favour of Beam SQL and BigQuery.
• Processing logic: stateful DoFns and timers, joins (CoGroupByKey, or side input for small data), enrichment, aggregation, deduplication, schema validation, error routing.
• AI data enrichment: call models during ingestion: BigQuery ML remote models and AI.GENERATE/ML.GENERATE_TEXT for classification, extraction and summarisation from SQL; ML.GENERATE_EMBEDDING to embed text or images; Cloud Natural Language, Document AI, Vision, Speech-to-Text for unstructured inputs; Dataflow RunInference for real-time scoring; Pub/Sub AI-inference SMT for simple cases. Keep calls batched and cached for cost and quota reasons.

Acquisition and new sources

• Connectors: DTS connectors, Datastream sources, Data Fusion plugins, Dataflow I/O (JDBC, Kafka, Bigtable, Spanner, Pub/Sub, Elasticsearch), Dataproc connectors, Application Integration and Apigee for APIs, Cloud Run functions to call SaaS REST APIs on a schedule, Storage Transfer Service for objects and Managed Service for Apache Kafka for Kafka sources.
• For an unfamiliar source, prefer managed connectors over custom code. If none exists, a Cloud Run job or function triggered by Cloud Scheduler plus Pub/Sub or Cloud Storage as a landing area keeps the rest of the pipeline unchanged.
• Change data capture: Datastream for database logs; Spanner change streams feed Dataflow; Bigtable change streams likewise; BigQuery change history (APPENDS/CHANGES functions).

Orchestration, automation and CI/CD

• Composer: environments run on GKE (managed); DAGs live in a Cloud Storage bucket; schedulers, workers, triggerers, DAG processor; environment size; private IP option; Composer service account needs roles for the services DAGs call. Use deferrable operators and sensors in reschedule mode to avoid holding worker slots.
• CI/CD for data: code in Git (Cloud Source Repositories is closed to new users; use GitHub, GitLab or Developer Connect), Cloud Build triggers run tests and build Flex Template images into Artifact Registry, then deploy Dataflow jobs or sync DAGs to the Composer bucket (gcloud composer environments storage dags import). Terraform (or Infrastructure Manager) for infrastructure; Dataform has built-in Git and release configurations. Test: unit tests with Beam TestPipeline/TestStream, DAG import tests, dry runs for SQL, integration tests in a non-prod project, then promote by parameter. Cloud Deploy is for app deployments, rarely needed here.