Data management

Access control with IAM

IAM answers “who can do what on which resource”. A principal (user, group, service account) gets a role (a set of permissions) on a resource through an allow policy. Policies inherit down the hierarchy: organisation, folder, project, resource. A role granted high up applies to everything below it.

Role types

A permission looks like bigquery.tables.getData; roles bundle permissions; you never grant permissions directly.

Least privilege

Grant the smallest role at the lowest level that lets the job work. Use groups instead of individuals, service accounts for workloads (not user keys), and time-limited or conditional access where possible. Avoid giving Editor or Owner to analysts.

Common predefined roles

Typical combo: an analyst needs BigQuery Job User on the project (to run queries) plus BigQuery Data Viewer on the dataset (to read), not Editor on the project.

Finer BigQuery controls

• Dataset-level, table-level and view-level IAM.
• Authorized views let users query a view without access to the underlying tables.
• Row-level security (row access policies) and column-level security (policy tags from Data Catalog and Dataplex taxonomies) restrict rows or columns by principal.
• Data masking with policy tags hides or transforms values for some users.

Cloud Storage access control

Cues: “consistent permissions for the whole bucket” means uniform access; “a client without a Google account needs to download for one hour” means signed URL; “make sure nothing here is ever public” means public access prevention. Turning on uniform access disables ACLs (reversible for 90 days).

Analytics Hub

Analytics Hub (part of BigQuery sharing) is for sharing data sets across organisations or teams without copying. A publisher creates a data exchange and listings that point at shared datasets; subscribers get a linked dataset (read-only reference) in their own project. Subscribers pay for their own queries; the publisher keeps one copy and control.

Use it when: sharing BigQuery data with partners, customers or other business units; avoiding copies and stale exports; providing curated or commercial datasets; or controlling who can subscribe (private exchange). Don’t use it for sharing a file (use Cloud Storage and signed URLs) or for sharing with a user who only needs read access inside your own project (use IAM).

Cloud Storage classes and lifecycle

Storage classes

• All classes have millisecond first-byte latency and the same durability (11 nines). The colder the class, the lower the storage price and the higher the retrieval and operation cost.
• Deleting or changing class before the minimum duration still incurs the remaining charge.
• Autoclass moves objects between classes automatically based on access, good when patterns are unknown.
• Retention needs: retention policy (objects can’t be deleted or replaced until age is reached; locking it is permanent), object holds, and bucket lock.

Lifecycle rules

A lifecycle configuration holds rules, each with conditions (age, created before, number of newer versions, storage class, is live, custom time) and an action:

• Delete the object.
• SetStorageClass to a colder class (for example Standard to Nearline at 30 days, to Coldline at 90, to Archive at 365).
• AbortIncompleteMultipartUpload.

Lifecycle rules run asynchronously (changes can take up to a day to apply). With Object Versioning, non-current versions can be deleted after N days to cut cost.

BigQuery storage lifecycle

• Table expiration and dataset default table expiration; partition expiration deletes old partitions automatically (for example keep 90 days).
• Time travel (default 7 days, configurable 2 to 7) lets you query or restore past states; after that, a fail-safe period of 7 days is for Google recovery only.
• Long-term storage pricing: a table or partition not modified for 90 days drops to roughly half the active storage price automatically.
• Table snapshots and clones preserve point-in-time copies cheaply.

Archiving

High availability and disaster recovery

Key terms: RPO (how much data you can lose) and RTO (how long you can be down). Lower targets cost more.

Cloud Storage

• Durability is the same for all locations; availability differs by location type.
• Regional: survives a zone failure. Dual-region and multi-region: survive a regional failure with automatic geo-redundancy, with no application change.
• Turbo replication (dual-region) targets replication of new objects within 15 minutes (default replication is about an hour, with an RPO commitment of 15 minutes for turbo).
• Object Versioning and soft delete protect against accidental deletion or overwrite. Backup and recovery of bucket data is also done with Storage Transfer Service copies to another bucket.

Cloud SQL

Restoring a backup creates a new state on an instance; point-in-time recovery needs backups and logging enabled. HA is not a backup: it doesn’t protect from bad deletes, backups do. Replicas don’t protect against logical errors either, because they copy them.

When to use replication

Use replication when you need lower RPO and RTO, local reads in other regions, or to survive regional outage. Use backups to recover from corruption, deletion or ransomware and for compliance. Most designs use both. Primary and secondary locations should be in different regions (dual-region or cross-region replica) for regional DR, not only different zones.

Other services (awareness)

BigQuery stores data redundantly across zones in its location and supports dataset replication across regions for DR. Spanner multi-region configurations provide cross-region availability.

Security and compliance

Encryption at rest

All Google Cloud data is encrypted at rest by default, with no action needed. You choose who manages the keys.

Cloud KMS

Cloud KMS stores and manages keys in key rings at a location. A key has versions; rotation creates a new primary version automatically on a schedule while old versions decrypt old data. IAM roles separate duties: Cloud KMS Admin (manage keys), Cloud KMS CryptoKey Encrypter/Decrypter (use keys). The service agent of the data service (BigQuery, Cloud Storage) needs the encrypter/decrypter role to use your CMEK. Keys are regional or global, and the key location must be compatible with the resource (BigQuery dataset and key in the same location). Cloud KMS keeps key use in audit logs.

Envelope encryption: data is encrypted with a data encryption key (DEK), and the DEK is encrypted with a key encryption key (KEK) in KMS. This is how CMEK works without sending data to KMS.

Encryption in transit

• Data moving between you and Google, and between Google services, is protected with TLS; inside Google’s network, it is encrypted at the network layer as well. Use HTTPS endpoints; Cloud SQL supports enforcing SSL connections.
• Encryption at rest protects stored data on disks; in transit protects data moving across networks. Neither replaces access control.

Privacy and compliance tools

• Sensitive Data Protection (Cloud DLP) discovers, classifies, masks, tokenises and de-identifies personal data (names, card numbers, emails) in Cloud Storage, BigQuery and text.
• Data residency: pick the location type to keep data in a region or area (for example EU) when regulations like GDPR require it.
• Audit logs record admin activity (on by default) and data access (must be enabled for most services; BigQuery data access logs are on).
• VPC Service Controls put a perimeter around services to reduce data exfiltration.
• Apply least privilege, classify data, mask or tokenise personal data, set retention, and keep audit evidence.