Ensuring successful operation

Compute Engine operations

Connecting and inventory

gcloud compute ssh web-1 --zone=us-central1-a                   # SSH (OS Login or metadata keys)
gcloud compute ssh web-1 --zone=us-central1-a --tunnel-through-iap
gcloud compute connect-to-serial-port web-1 --zone=us-central1-a
gcloud compute instances list
gcloud compute instances list --filter="status=RUNNING AND zone:us-central1" --format="table(name,zone,machineType.basename(),networkInterfaces[0].networkIP)"
gcloud compute instances describe web-1 --zone=us-central1-a
gcloud compute instances stop|start|reset|suspend|resume|delete web-1 --zone=us-central1-a
gcloud compute instances set-machine-type web-1 --zone=us-central1-a --machine-type=e2-standard-4   # VM must be stopped

• --filter, --format (json, yaml, table, value, csv) and --sort-by work on every gcloud list command; know them.
• Cloud Asset Inventory gives organization-wide inventory and history: gcloud asset search-all-resources --scope=organizations/ORG_ID --asset-types=compute.googleapis.com/Instance.
• Changing machine type, adding or removing GPUs and editing the service account need the VM stopped. Metadata, labels and tags change live.
• A stopped VM stops compute charges but you still pay for disks and reserved static IPs. A suspended VM keeps memory state, and costs storage.
• Live migration moves a running VM to other hardware during host maintenance with no reboot (not for GPUs or Spot).

Snapshots

A snapshot is an incremental backup of a persistent disk (the first is full, later ones store changes). It is global, can be used to create a disk in any zone or region, and may be stored in a multi-regional or regional location.

gcloud compute disks snapshot data-1 --zone=us-central1-a --snapshot-names=data-1-snap
gcloud compute snapshots list
gcloud compute snapshots describe data-1-snap
gcloud compute snapshots delete data-1-snap
gcloud compute disks create data-restored --source-snapshot=data-1-snap --zone=europe-west1-b
# Scheduled snapshots with a resource policy
gcloud compute resource-policies create snapshot-schedule daily-snap \
    --region=us-central1 --max-retention-days=14 --daily-schedule --start-time=03:00
gcloud compute disks add-resource-policies data-1 --resource-policies=daily-snap --zone=us-central1-a

• Deleting a snapshot never breaks others: Google keeps the data other snapshots depend on.
• Use --guest-flush or quiesce databases for application-consistent snapshots; snapshots are crash-consistent by default.
• Moving a VM or disk to another zone or region: snapshot (or image) then create the resource in the new location. gcloud compute instances move also exists for zone moves.
• Archive snapshots are cheaper for long-term retention; machine images capture a whole VM (config plus all disks), unlike a snapshot of one disk.

Images

An image is a boot-disk template used to create new boot disks and instance templates; it is global.

gcloud compute images create web-image-v1 --source-disk=web-1 --source-disk-zone=us-central1-a
gcloud compute images create web-image-v2 --source-snapshot=web-snap
gcloud compute images list --no-standard-images
gcloud compute images describe-from-family debian-12 --project=debian-cloud
gcloud compute images delete web-image-v1
gcloud compute images create imported-img --source-uri=gs://bucket/disk.tar.gz   # from a Cloud Storage file

• Image families point at the latest non-deprecated image; use the family name in templates so new versions are picked up. Deprecate old images with gcloud compute images deprecate.
• Prefer stopping the VM (or at least a snapshot) before creating an image from its disk for consistency.
• Share images with IAM (roles/compute.imageUser). Org policy compute.trustedImageProjects restricts which projects can supply images.

GKE operations

Inventory and Kubernetes resources

gcloud container clusters list
gcloud container clusters describe my-cluster --region=us-central1
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods --all-namespaces            # -A
kubectl get svc,deploy,sts,ds,hpa -n prod
kubectl describe pod web-abc -n prod         # events: scheduling, image pull, probe failures
kubectl logs web-abc -c app --previous -f
kubectl exec -it web-abc -- sh
kubectl top nodes; kubectl top pods
kubectl scale deployment web --replicas=5
kubectl set image deployment/web web=REGISTRY/web:2.0
kubectl rollout status|history|undo deployment/web
kubectl delete -f manifest.yaml
kubectl port-forward svc/web 8080:80
kubectl get events --sort-by=.metadata.creationTimestamp

Pod stuck in Pending: no node has the resources, or a taint/affinity or unbound volume blocks it (check kubectl describe pod). ImagePullBackOff: wrong image name or no permission. CrashLoopBackOff: container crashing; use kubectl logs --previous.

Access to Artifact Registry

GKE pulls images from Artifact Registry with the node service account (Standard) or the Autopilot node identity. Repositories in the same project with the default Compute Engine service account usually work with no extra setup on current defaults, but least-privilege setups need:

gcloud artifacts repositories create repo --repository-format=docker --location=us-central1
gcloud auth configure-docker us-central1-docker.pkg.dev
docker tag web us-central1-docker.pkg.dev/my-proj/repo/web:1.0
docker push us-central1-docker.pkg.dev/my-proj/repo/web:1.0
gcloud artifacts repositories add-iam-policy-binding repo --location=us-central1 \
    --member="serviceAccount:NODE_SA@my-proj.iam.gserviceaccount.com" --role="roles/artifactregistry.reader"

• Give the cluster’s node service account roles/artifactregistry.reader on the repository (cross-project repository: grant on that project).
• Legacy access scopes on nodes also matter on older clusters: the node needs read access to storage/Artifact Registry (cloud-platform scope with IAM roles is the clean option; --scopes=storage-ro for older setups).
• Use Workload Identity Federation for GKE so Pods use their own identities for other Google APIs.
• Container Registry (gcr.io) has been shut down; Artifact Registry replaced it.

Node pools

A node pool is a group of nodes with the same configuration (machine type, disk, labels, taints, image type) inside a Standard cluster. Autopilot has no node pools to manage.

gcloud container node-pools create highmem-pool --cluster=my-cluster --region=us-central1 \
    --machine-type=n2-highmem-8 --num-nodes=1 --node-labels=pool=highmem \
    --node-taints=dedicated=highmem:NoSchedule --spot
gcloud container node-pools list --cluster=my-cluster --region=us-central1
gcloud container clusters resize my-cluster --node-pool=default-pool --num-nodes=5 --region=us-central1
gcloud container node-pools update default-pool --cluster=my-cluster --region=us-central1 \
    --enable-autoscaling --min-nodes=1 --max-nodes=10
gcloud container clusters upgrade my-cluster --node-pool=default-pool --region=us-central1
gcloud container node-pools delete old-pool --cluster=my-cluster --region=us-central1

• Machine type of an existing pool can’t change: create a new pool, cordon and drain the old one (kubectl cordon|drain NODE), then delete it.
• Surge upgrades (max-surge, max-unavailable) and blue/green node pool upgrades control disruption. Auto-repair and auto-upgrade are on by default for current clusters.
• Use node selectors, taints and tolerations, or affinity to steer Pods to pools (for example GPU or Spot pools).

Autoscaling

Don’t use HPA and VPA on the same metric (CPU) for one Deployment. Pods must set resource requests or CPU-based HPA cannot work. HPA adds Pods, cluster autoscaler then adds nodes if they don’t fit. Autopilot handles node scaling for you.

Cloud Run operations

Every deployment creates an immutable revision. A service routes traffic across revisions.

gcloud run deploy api --image=REGISTRY/api:2 --region=us-central1 --no-traffic --tag=canary
gcloud run services update-traffic api --region=us-central1 --to-revisions=api-00002-xyz=10,api-00001-abc=90
gcloud run services update-traffic api --region=us-central1 --to-latest
gcloud run services update-traffic api --region=us-central1 --to-tags=canary=5
gcloud run revisions list --service=api --region=us-central1
gcloud run services describe api --region=us-central1
gcloud run services update api --region=us-central1 --min-instances=1 --max-instances=50 --concurrency=80 --cpu=2 --memory=1Gi
gcloud run services logs read api --region=us-central1

• Deploying new versions: gcloud run deploy again; by default 100% of traffic goes to the new revision. Use --no-traffic to deploy without serving, then gradually migrate or roll back by sending 100% to the earlier revision (--to-revisions=REV=100).
• Traffic splitting splits by percentage across revisions (for canary and blue/green). A revision tag gives a dedicated URL (https://canary---api-xyz.a.run.app) to test a revision before traffic.
• Scaling parameters: min instances (keeps warm instances to avoid cold starts, costs money), max instances (limits scale and protects downstream resources), concurrency (requests per instance, up to 1000), CPU allocation (request-based vs always allocated), and the request timeout.
• Cloud Run scales to zero when min-instances is 0.
• Control who calls a service with roles/run.invoker (--allow-unauthenticated grants it to allUsers).

Storage and databases

Cloud Storage objects and security

gcloud storage ls gs://bucket
gcloud storage cp file.txt gs://bucket/dir/
gcloud storage cp -r gs://src gs://dst
gcloud storage rsync -r ./local gs://bucket/dir
gcloud storage rm gs://bucket/dir/** 
gcloud storage buckets update gs://bucket --versioning
gcloud storage buckets add-iam-policy-binding gs://bucket --member=group:data@example.com --role=roles/storage.objectViewer
gcloud storage objects update gs://bucket/o --storage-class=COLDLINE
gsutil -m cp -r dir gs://bucket     # older tool, still seen on the exam

• Access control: uniform bucket-level access (IAM only, recommended) vs fine-grained (IAM plus object ACLs). Public access prevention blocks allUsers. Roles: storage.objectViewer, objectCreator, objectUser, objectAdmin, storage.admin.
• Signed URLs give time-limited access to one object without a Google account (gcloud storage sign-url --duration=1h with a service account key or impersonation). Signed policy documents control browser uploads.
• Object versioning keeps older versions on overwrite or delete. Soft delete (default 7-day retention) restores deleted objects. Retention policy and object holds prevent deletion until a date (a locked retention policy is permanent). Bucket Lock is that lock.
• Encryption: Google-managed by default; CMEK (Cloud KMS keys) or customer-supplied keys on request.

Lifecycle management

Lifecycle rules act on objects automatically based on conditions (age, creation date, storage class, number of newer versions, isLive, name prefix) with actions SetStorageClass or Delete (and AbortIncompleteMultipartUpload).

cat > lifecycle.json <<'EOF'
{"rule":[
 {"action":{"type":"SetStorageClass","storageClass":"NEARLINE"},"condition":{"age":30,"matchesStorageClass":["STANDARD"]}},
 {"action":{"type":"SetStorageClass","storageClass":"COLDLINE"},"condition":{"age":90}},
 {"action":{"type":"Delete"},"condition":{"age":365}}]}
EOF
gcloud storage buckets update gs://bucket --lifecycle-file=lifecycle.json

Classes only go down in coldness by lifecycle (Standard to Nearline to Coldline to Archive); lifecycle can’t move colder data back to Standard (use Autoclass or a rewrite for that). Rules run asynchronously, not at an exact moment.

Querying data

BigQuery cost control: it charges by bytes scanned (on-demand) or slot capacity, so select only needed columns (no SELECT *), partition and cluster tables, use --maximum_bytes_billed and --dry_run. LIMIT does not reduce bytes scanned.

Estimating storage costs

• Use the Google Cloud Pricing Calculator to estimate (storage class and location, operations, network egress, retrieval fees for colder classes, early deletion charges, BigQuery storage vs query).
• BigQuery long-term storage drops the price automatically for tables not edited for 90 days.
• Watch for network egress and operations charges, which surprise people more than storage per GB.

Backup and restore

Reviewing job status

bq ls -j -a --max_results=10          # list jobs
bq show -j JOB_ID
gcloud dataflow jobs list --region=us-central1
gcloud dataflow jobs describe JOB_ID --region=us-central1
gcloud dataflow jobs cancel|drain JOB_ID --region=us-central1

For Dataflow, drain finishes in-flight data and then stops a streaming job; cancel stops immediately. Job failures show in the console job graph and in Cloud Logging. In BigQuery, INFORMATION_SCHEMA.JOBS lists jobs with errors and bytes billed.

Networking operations

# Add a subnet
gcloud compute networks subnets create new-sub --network=prod-vpc --region=europe-west1 --range=10.20.0.0/24
# Expand a subnet (primary range can only get bigger, never smaller, and can't overlap)
gcloud compute networks subnets expand-ip-range web-subnet --region=us-central1 --prefix-length=22
# Reserve addresses
gcloud compute addresses create web-ip --region=us-central1                   # regional external
gcloud compute addresses create global-ip --global                            # global external (for global LB)
gcloud compute addresses create internal-ip --region=us-central1 --subnet=web-subnet --addresses=10.10.0.50
gcloud compute instances add-access-config web-1 --zone=... --address=34.1.2.3
gcloud compute addresses list

• Subnet expansion is non-disruptive and irreversible; you cannot shrink or change a subnet’s range to a non-superset.
• Static IPs: ephemeral addresses change when the VM is stopped; reserved static addresses persist. An unused reserved external IP costs money (in-use is cheaper). Internal static IPs come from subnet ranges. Global addresses are for global load balancers, regional for regional resources and VMs.
• Cloud DNS: managed authoritative DNS. Public zones answer the internet; private zones answer only inside chosen VPCs. Supports DNSSEC, forwarding zones (send queries to on-premises DNS), peering zones, response policies and routing policies (weighted, geo, failover).

gcloud dns managed-zones create example-zone --dns-name="example.com." --description="public" --visibility=public
gcloud dns managed-zones create corp --dns-name="corp.internal." --visibility=private --networks=prod-vpc --description=x
gcloud dns record-sets create www.example.com. --zone=example-zone --type=A --ttl=300 --rrdatas=34.1.2.3
gcloud dns record-sets list --zone=example-zone

• Cloud NAT: managed outbound source NAT for VMs and GKE nodes without external IPs; configured on a Cloud Router per region and VPC. Provides outbound-only connectivity (no inbound connections initiated from the internet). Not needed for Google APIs when Private Google Access is on.

gcloud compute routers create nat-router --network=prod-vpc --region=us-central1
gcloud compute routers nats create nat-cfg --router=nat-router --region=us-central1 \
    --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges

Monitoring and logging

Cloud Monitoring

• Metrics (system, agent, custom, log-based, Prometheus) are time series. Dashboards chart them. Alerting policies have conditions (threshold, absence, rate of change, forecast, MQL/PromQL), a duration, notification channels and optional documentation. Uptime checks probe HTTP(S), TCP or ping endpoints from multiple regions and can alert.
• Alert examples: CPU above 80% for 5 minutes; no data (absence) from a VM; an uptime check failing from two locations; a log-based metric exceeding a count.
• Incidents open when conditions are met and close when they stop; snooze silences notifications.

gcloud alpha monitoring policies create --policy-from-file=policy.json
gcloud monitoring channels list
gcloud beta monitoring channels create --display-name=ops --type=email --channel-labels=email_address=ops@example.com

Custom metrics

• Write your own time series using the Cloud Monitoring API or OpenTelemetry; the type is custom.googleapis.com/... (or workload.googleapis.com/... from the Ops Agent). Pick the metric kind (gauge, delta, cumulative) and value type.
• Log-based metrics: turn log entries matching a filter into counter or distribution metrics, then alert or chart them. System-defined ones exist; user-defined ones are created with gcloud logging metrics create errors --description=x --log-filter='severity>=ERROR'.
• Ops Agent can also collect third-party application metrics (nginx, MySQL, Redis and others) via receivers.

Ops Agent and Managed Service for Prometheus

The Ops Agent is the single agent (replaces the legacy Monitoring and Logging agents) that collects logs and metrics from inside a VM, including memory, disk usage and process metrics, which are not available by default from the hypervisor. Install and manage it:

curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
sudo bash add-google-cloud-ops-agent-repo.sh --also-install
sudo systemctl status google-cloud-ops-agent*
# Fleet-wide with an agent policy (via VM Manager / OS Config)
gcloud compute instances ops-agents policies create ops-pol --agent-rules="type=ops-agent,version=latest,package-state=installed,enable-autoupgrade=true" --os-types=short-name=debian,version=12 --zones=us-central1-a

Config file: /etc/google-cloud-ops-agent/config.yaml. The VM’s service account needs roles/logging.logWriter and roles/monitoring.metricWriter. GKE has its own built-in collection (Autopilot enables it) and does not use the Ops Agent.

Google Cloud Managed Service for Prometheus is a Prometheus-compatible, globally scalable metric store in Monarch. On GKE enable managed collection (gcloud container clusters update C --enable-managed-prometheus) and define PodMonitoring resources to scrape; query with PromQL in Cloud Monitoring or Grafana. Choose it when teams already use Prometheus and want no server to run.

Cloud Logging

gcloud logging read 'resource.type="gce_instance" AND severity>=ERROR' --limit=20 --freshness=1d --format=json
gcloud logging sinks create bq-sink bigquery.googleapis.com/projects/P/datasets/logs \
    --log-filter='resource.type="gce_instance"' --include-children --organization=ORG_ID
gcloud logging sinks create arch storage.googleapis.com/my-archive-bucket --log-filter='severity>=WARNING'
gcloud logging buckets create sec-logs --location=global --retention-days=365 --enable-analytics
gcloud logging sinks list
gcloud logging write my-log "hello" --severity=INFO

• After creating a sink, grant its writer identity (printed in the output, a service account) permission on the destination (BigQuery Data Editor, Storage Object Creator, Pub/Sub Publisher). Missing this is the classic “sink exports nothing” fault.
• Aggregated sinks (--include-children at folder or organization) centralise logs from many projects.
• Exclusion filters drop noisy logs to save cost. Excluded logs can’t be recovered.
• Logs Explorer uses the Logging query language: resource.type="k8s_container" AND jsonPayload.status>=500. Expand a log entry to see fields, trace, labels and request details; correlate by trace ID and use Log Analytics for SQL aggregation.
• Retention beyond 30 days: change bucket retention or export a sink to Cloud Storage or BigQuery.

Audit logs

Configure data access logs in the IAM audit log configuration (gcloud projects get-iam-policy then add an auditConfigs block: logType: DATA_READ/DATA_WRITE/ADMIN_READ, with exemptedMembers if needed). Audit logs answer “who did what, where, when”.

Diagnostics and status

• Error Reporting groups and counts application exceptions. Cloud Trace shows latency of requests across services. Cloud Profiler shows continuous CPU and memory hot spots in code. (Cloud Debugger is retired.) Use Logs Explorer with trace correlation to follow a request, and Cloud Monitoring for resource metrics. For connectivity issues use Network Intelligence Center (Connectivity Tests, Network Topology, Performance Dashboard, Firewall Insights). Recommender / Active Assist gives idle resource and right-sizing suggestions.
• Google Cloud Status Dashboard (status.cloud.google.com) lists global service incidents; Personalized Service Health in the console shows incidents that affect your projects and can alert through Monitoring. Check these first when many resources fail at once with no change on your side.