PDE study notes
- Designing data processing systems PDE notes on security and compliance (IAM, CMEK, DLP, residency), reliability and fidelity, ACID and availability choices, portability, governance and catalogs, and designing migrations with Datastream, DMS, DTS and Transfer Appliance. 22%
- Ingesting and processing the data PDE notes on planning pipelines, Pub/Sub, Dataflow and Apache Beam (windows, triggers, late data, exactly-once), Dataproc and Spark, Data Fusion, BigQuery ingestion, Kafka, AI enrichment, Composer, Workflows and CI/CD. 25%
- Storing the data PDE notes on choosing between BigQuery, BigLake, Cloud Storage, Bigtable, Spanner, Cloud SQL, AlloyDB, Firestore and Memorystore, BigQuery data modelling, partitioning and clustering, data lakes, Dataplex and federated governance. 20%
- Preparing and using data for analysis PDE notes on BigQuery for BI (BI Engine, materialized views, query tuning), connecting Looker and other tools, security with row access, policy tags, masking and DLP, preparing data for ML and RAG, BigQuery ML, and sharing with Analytics Hub. 15%
- Maintaining and automating data workloads PDE notes on cost and resource optimisation, Dataproc persistent vs ephemeral clusters, Composer DAG design, BigQuery editions, reservations and slots, interactive vs batch, monitoring, troubleshooting quotas and billing, and fault tolerance and recovery. 18%