Building and implementing CI/CD pipelines

The Google Cloud delivery stack

Third-party tooling works alongside: Jenkins (with the Google Cloud plugins, agents on GKE or Compute Engine), Argo CD and Flux (pull-based GitOps on GKE; Config Sync is Google’s equivalent), Packer for VM images, kpt for packaging and mutating Kubernetes config, Git as the source of truth. Pick Google-native tools when the question stresses managed, least overhead; pick Jenkins or Argo when the scenario says the team already uses it, but secure it equally.

Cloud Build

A serverless CI service. A build runs a series of steps, each a container, on a worker, sharing the /workspace directory.

Configuration

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build','-t','us-docker.pkg.dev/$PROJECT_ID/app/web:$SHORT_SHA','.']
- name: 'us-docker.pkg.dev/$PROJECT_ID/app/web:$SHORT_SHA'
  entrypoint: 'npm'
  args: ['test']
images: ['us-docker.pkg.dev/$PROJECT_ID/app/web:$SHORT_SHA']
timeout: 1200s

• Steps run in order unless you use waitFor (a list of step ids, with ['-'] meaning start immediately) to run steps in parallel.
• images pushes built images (and records them in the build results and provenance); artifacts uploads files to Cloud Storage; options sets machine type, logging mode, disk size, the worker pool and requestedVerifyOption.
• Default substitutions: $PROJECT_ID, $BUILD_ID, $COMMIT_SHA, $SHORT_SHA, $REPO_NAME, $BRANCH_NAME, $TAG_NAME, $REVISION_ID. User-defined substitutions must start with an underscore (_ENV) and can be set on the trigger.
• Tag images with the commit SHA, not only latest, so every build maps to a commit and promotion refers to an exact digest.
• Default timeout is 60 minutes, adjustable up to 24 hours.
• Caching: Docker layer caching with --cache-from (pull the previous image), Kaniko with --cache=true (stores layers in the registry), or caching language dependencies in Cloud Storage.
• Machine types: default is a small machine; larger ones (E2_HIGHCPU_8, E2_HIGHCPU_32) speed builds.
• Builders: Google’s community and official builder images (docker, gcloud, git, kubectl, maven, npm, go) or any public/private image as a step. Buildpacks (gcr.io/k8s-skaffold/pack) build container images with no Dockerfile, and gcloud run deploy --source uses them.

Triggers

Trigger options: included/ignored files filters (so a docs-only change doesn’t trigger a build), a build config in the repo (cloudbuild.yaml), a Dockerfile, or buildpacks, a service account, and require approval (a build waits until someone with roles/cloudbuild.builds.approver approves; good for production deploy triggers).

If a trigger never fires, check: branch regex or tag pattern, file filters, repository connection/authorisation, the trigger being disabled, and event type.

Identity and permissions

• Builds run as a service account. Google’s guidance is to use a user-specified, dedicated service account per trigger or pipeline with only the roles needed (such as artifactregistry.writer, logging.logWriter, clouddeploy.releaser). The legacy Cloud Build service account is over-privileged by default. New projects use the Compute Engine default service account if you don’t choose one, which is also too broad.
• Deploying to Cloud Run from a build needs the build service account to hold run.admin (or developer) and also iam.serviceAccountUser on the runtime service account.
• Builds from other projects: grant cross-project roles to the build service account.
• Cloud Build logs go to Cloud Logging and/or Cloud Storage. With a user-specified service account you must pick a logging option (CLOUD_LOGGING_ONLY is common) or provide a bucket.

Private pools and networking

• Default builds run on a public, shared pool: Google-managed, internet access, no VPC.
• Private pools are dedicated workers in a Google-managed VPC peered to your VPC. Use them to reach private resources (an on-premises Git server, a private GKE control plane, Cloud SQL private IP), to disable public IPs, to use VPC Service Controls, and to pick larger machines. Peering is non-transitive, so reaching on-premises through your VPC needs Cloud VPN with custom routes advertised.
• Cloud Build repositories (2nd gen) connect to GitHub, GitLab and Bitbucket (including self-hosted GitHub Enterprise / GitLab through a host connection), with the token stored in Secret Manager. The older “1st gen” GitHub app triggers are legacy.

Secrets in builds

Use Secret Manager through availableSecrets:

availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_ID/secrets/api-key/versions/latest
    env: API_KEY
steps:
- name: gcr.io/cloud-builders/curl
  entrypoint: bash
  args: ['-c','curl -H "Authorization: $$API_KEY" ...']
  secretEnv: ['API_KEY']

$$ escapes so the shell, not Cloud Build, expands it. Build service account needs secretmanager.secretAccessor. Don’t pass secrets in substitutions or Docker --build-arg (they’re visible in logs and image history). Use docker build --secret for build-time secrets.

Security features

• Build provenance: for container builds Cloud Build generates signed SLSA provenance (build level 3) describing source, builder, and the resulting image digest, and stores it as an attestation in Artifact Analysis. Verify before deploying.
• Vulnerability scanning by Artifact Analysis when images are pushed.
• Approvals, VPC-SC, private pools, least-privilege service accounts, and branch protection in Git protect the pipeline itself.

Artifact Registry

The single place to store container images, language packages (Maven, npm, Python, Go), OS packages (Apt, Yum), Helm charts (OCI), and generic files. Container Registry is shut down; gcr.io hostnames are served by Artifact Registry repositories (use gcr.io domain support or migrate paths to REGION-docker.pkg.dev).

• Repository modes: standard (you push), remote (a pull-through cache of Docker Hub, Maven Central, PyPI, npm and others; speeds builds and shields you from upstream outages), virtual (one endpoint merging several standard and remote repos with priorities; fixes dependency confusion by listing internal repos first).
• Locations: regional or multi-regional. Keep repos near the workloads that pull from them to cut latency and egress.
• IAM at repository level: artifactregistry.reader (pull), writer (push), repoAdmin. Cloud Run and GKE node service accounts need reader. Use separate repos per environment, with promotion by copying or by retagging an immutable digest.
• Cleanup policies delete or keep versions by age, count or tag prefix (dry-run first). Immutable tags stop overwriting a tag.
• Vulnerability scanning: Artifact Analysis (Container Scanning API) scans images pushed to Artifact Registry automatically and keeps rescanning for new CVEs; on-demand scanning (gcloud artifacts docker images scan) scans a local or remote image earlier in the pipeline. Findings are occurrences that can gate promotion. It also covers language package vulnerabilities and stores attestations, SBOMs and provenance as metadata.
• Audit: Artifact Registry writes Cloud Audit Logs; Admin Activity always, Data Access optional (who pulled what).

Cloud Deploy

A managed continuous delivery service that promotes a release through ordered targets (dev, staging, prod) with approvals, rollbacks and verification. Supports GKE, GKE Enterprise/multi-cluster, Cloud Run, and custom targets.

Resources

How a release is rendered and deployed

• It uses Skaffold (skaffold.yaml) with Kustomize, Helm or raw manifests and profiles per target, so the same artifact gets different config per environment. Skaffold can render and deploy; Cloud Deploy handles render at release creation and deploy per rollout.
• The release pins image digests, so what was tested in staging is exactly what goes to production.
• Deploy parameters (on pipeline, target or release) supply per-environment values.
• Hooks: pre-deploy and post-deploy hooks (for example a database migration job).
• Verify: after deploying, run a container-based verification (smoke or integration test) defined in Skaffold; failure marks the rollout failed.

Deployment strategies in Cloud Deploy

Blue/green is done through traffic splitting of a candidate revision before full cutover (on Cloud Run: deploy with --no-traffic and a tag, test, then shift), or by running two Deployments and flipping a Service selector or load balancer on GKE.

Approvals, promotion and rollback

• Approval: a target with requireApproval pauses rollouts until a user with clouddeploy.approver approves. Notifications go through Pub/Sub (clouddeploy-approvals, clouddeploy-operations, clouddeploy-resources topics) to chat or email. Use separation of duties: the person who created the release shouldn’t approve it.
• Promote manually (gcloud deploy releases promote) or with an automation.
• Rollback: roll the target back to a previous successful release; a rollback is a new rollout using the previous release. Automations can roll back on failure. Release abandon stops further promotion of a bad release.
• IAM roles: clouddeploy.releaser (create releases), approver, operator (promote, roll back), admin; the execution service account needs rights on the target (container.developer, run.developer) and iam.serviceAccountUser for the workload service account.
• Audit and tracking: Cloud Deploy shows a release history; each action is in Cloud Audit Logs; the delivery metrics page reports DORA metrics (deployment frequency, lead time, change failure rate, time to restore).

Pipeline design

CI/CD for applications

A common flow: developer opens PR → trigger builds, unit tests, lint and a vulnerability scan → merge to main builds the image, pushes it to Artifact Registry tagged by SHA → CI creates a Cloud Deploy release (gcloud deploy releases create) → dev automatic → staging automatic with verification and load test → production requires approval and uses a canary → monitoring alerts decide rollback.

• Trunk-based development and short-lived branches; every merge is releasable.
• Build once, deploy many: the same image digest goes through every environment, with config injected per environment. Rebuilding per environment breaks traceability.
• Test pyramid: fast unit tests on every commit; integration tests against ephemeral dependencies; end-to-end and load tests in staging; security tests (SAST, dependency, container scanning) as gates. Fail early.
• Pipeline triggers: Git events, Pub/Sub events from Artifact Registry (image pushed) or Cloud Deploy (rollout succeeded), schedules for nightly tasks, manual for sensitive operations.
• Hybrid and multi-cloud: GKE Enterprise (fleet) lets Cloud Deploy and Config Sync target clusters on-premises or in other clouds; private pools or agents reach private networks.

CI/CD for infrastructure

Terraform in Git, plan on pull request (posted as a comment), manual or automatic apply on merge using a dedicated service account in Cloud Build or Infrastructure Manager, with policy checks and drift detection. Separate state per environment; protect state with Cloud Storage versioning and restricted IAM. Promote module versions the same way as application versions.

CI/CD for ML workloads

• Vertex AI Pipelines (Kubeflow Pipelines or TFX; part of the Gemini Enterprise Agent Platform, formerly Vertex AI) orchestrate data prep, training, evaluation and deployment as containerised steps.
• CI tests code and pipeline components and builds the pipeline; CD deploys the pipeline and, after evaluation gates (metric thresholds against a baseline), registers the model in the Model Registry and deploys it to an endpoint, using traffic splitting between model versions as a canary.
• Continuous training is triggered by schedule, new data or drift detection through Model Monitoring. Treat data, code, model and pipeline as versioned artifacts. Success metrics come from model telemetry (prediction latency, error rate, skew and drift, quality scores).

Deployment strategies

Implementation details:

• Cloud Run: each deploy creates a revision. Split traffic by percentage (gcloud run services update-traffic --to-revisions=rev1=90,rev2=10), use --no-traffic plus a tag to get a dedicated test URL, and roll back by sending 100% to the earlier revision (instant). Gradual rollouts can be automated through Cloud Deploy.
• GKE Deployment: RollingUpdate with maxSurge (extra Pods) and maxUnavailable; readiness probes keep traffic off unready Pods; kubectl rollout undo rolls back. For canary: two Deployments with a shared Service label, or Gateway API weighted routes, or Cloud Service Mesh traffic splitting.
• Managed instance groups: rolling update (maxSurge, maxUnavailable, minimal action replace or restart, proactive vs opportunistic update type) and canary update (apply a second instance template to a target size or percent). Roll back by setting the previous template. Add autohealing with a health check.
• App Engine: traffic splitting by IP, cookie or random between versions.
• Feature flags: OpenFeature SDKs, Firebase Remote Config, LaunchDarkly. Pair with progressive delivery: ship dark, enable for staff, then 1%, 10%, 100%.

Success metrics for any rollout are defined first and read from telemetry: error rate, latency percentiles, saturation, business metrics (checkout rate), and for ML, prediction quality. Automate the decision where possible: Cloud Deploy automations or verification jobs, and monitoring alerts wired to rollback.

Troubleshooting deployment issues

Pipeline configuration and secrets

Cloud KMS

• Hierarchy: key ring → key → key version. Keys are regional or multi-regional. Rotation creates a new primary version (automatic schedule); old versions stay for decrypt. Protection levels: software, HSM (Cloud HSM), external (Cloud EKM). Cloud KMS Autokey provisions CMEK keys for you.
• Uses in pipelines: CMEK on Artifact Registry, Cloud Storage, Secret Manager, Cloud Build logs; asymmetric signing keys for Binary Authorization attestors.
• Separate duties: key admins don’t get cryptoKeyEncrypterDecrypter.

Secret Manager

• Stores secrets as versions (immutable); latest alias or pin a number. Disable or destroy old versions. Rotation schedule publishes a Pub/Sub message that your function handles.
• Regional secrets keep data in a chosen region; automatic or user-managed replication for global secrets. CMEK supported. IAM: secretmanager.secretAccessor (read value), secretVersionManager, admin. Audit logging shows access.
• Cloud Run can expose a secret as an environment variable or a mounted file; mounting a volume with latest picks up new versions without redeploying (env vars read at start). GKE: Secret Manager add-on (CSI driver) mounts secrets as files; or External Secrets Operator.
• Parameter Manager stores non-secret and sensitive configuration (JSON, YAML) with versions and a render option that resolves Secret Manager references, so config is separate from code and secrets.
• Certificate Manager issues and renews Google-managed TLS certificates (including wildcard through DNS authorization) and attaches them to load balancers; certificate maps for many domains.

Build-time versus runtime secrets

Never bake secrets into images, commit them, or store them in manifests; reference them by name. Rotate them without rebuilding where you can (volume mounts, runtime fetch).

Workload Identity Federation and short-lived credentials

• Workload Identity Federation lets workloads outside Google Cloud (GitHub Actions, GitLab, Jenkins, AWS, Azure, on-premises with OIDC or SAML) get Google credentials by exchanging their own token. No service account keys. Pieces: a workload identity pool, a provider, and an attribute mapping and condition (for instance restrict to one repository and branch). Then either grant roles to the federated principal directly or let it impersonate a service account.
• Workload Identity Federation for GKE maps a Kubernetes service account to IAM, so Pods get tokens without keys.
• Short-lived credentials: OAuth access tokens last 1 hour by default (configurable through generateAccessToken lifetime up to 12 hours with an organization policy allowing it). Use impersonation for pipelines and humans. Prefer them over keys, which don’t expire. Block key creation with iam.disableServiceAccountKeyCreation; if keys exist, rotate and monitor.

Securing the deployment pipeline

Software supply chain

SLSA (Supply-chain Levels for Software Artifacts) is a framework of build-integrity levels. The build track runs Level 1 (provenance exists), Level 2 (hosted build and signed provenance) and Level 3 (hardened, isolated, non-forgeable provenance). Cloud Build provides Level 3 provenance for container builds.

Controls across the chain:

Binary Authorization

A deploy-time policy that allows only trusted images on GKE, Cloud Run, and GKE attached clusters (it checks the image digest).

• Policy rules: default rule plus per-cluster (or per-service-account/namespace/Cloud Run) rules. Evaluation modes: ALWAYS_ALLOW, ALWAYS_DENY, REQUIRE_ATTESTATION (needs attestations from one or more attestors).
• Enforcement: ENFORCED_BLOCK_AND_AUDIT_LOG blocks and logs; DRYRUN_AUDIT_LOG_ONLY logs violations without blocking (use to trial a policy).
• Attestor: an entity with public keys (from Cloud KMS asymmetric keys, or PGP) whose signature on the image digest is an attestation, stored as a Container Analysis note and occurrence. A pipeline step signs after tests and scans pass, so “passed QA” or “built by Cloud Build” becomes machine-checkable. Cloud Build can create a built-by-Cloud-Build attestation. Separate attestors for different gates (vulnerability scan, QA, security) can be all required.
• Exempt images (allowlist patterns) for system images. Break-glass: annotate the Pod to bypass, with the bypass audit-logged and alerted on.
• Continuous validation re-evaluates running Pods against policy over time and logs violations.
• Also needs the Binary Authorization API enabled and the cluster flag (GKE) turned on.
• To enforce IAM by environment: developers can push images but only the pipeline’s service account can deploy to prod; production projects grant few humans anything. Separate projects per environment, a dedicated deploy service account per environment, and approvals on prod targets.

Managing multiple environments (guide 1.4)

• Environment layout: separate projects (and GKE clusters or namespaces) for dev, staging and production, under folders carrying environment-specific IAM and org policy. Keep staging production-like.
• Configuration and policy: base config plus overlays (Kustomize or Helm values) per environment; Config Sync applies from Git; Policy Controller (OPA Gatekeeper) enforces constraints and Policy Controller’s constraint templates can check Terraform plans as well; Organization Policy covers cloud resources.
• Ephemeral environments: create a short-lived environment per pull request or test run (a Cloud Run service with a tagged revision and --no-traffic, a namespace in a shared cluster, a Terraform workspace or Infrastructure Manager deployment), run tests, tear down by TTL or on PR close. Saves cost and removes environment contention. Use Terraform or Config Connector so create and destroy are scripted; label resources with owner and expiry.
• GKE fleets: group clusters (any location, including on-premises and other clouds with GKE Enterprise) to manage them as a unit: team scopes and namespaces across clusters, shared Config Sync and Policy Controller config, multi-cluster Services and Ingress/Gateway, fleet-level observability and rollout sequencing (upgrade a canary fleet first).
• Serverless environments: Cloud Run services per environment with separate service accounts, revisions for gradual rollout, min-instances in prod only, and IaC for config.
• Safe patching and upgrades:
  • GKE release channels: Rapid, Regular, Stable, Extended (Extended keeps a version supported longer, extra cost after the standard period). Control plane and nodes upgrade automatically within a channel. Maintenance windows and exclusions (no minor upgrades / no upgrades for a period) schedule them. Surge upgrades (maxSurge, maxUnavailable) or blue/green node pool upgrades (keep the old pool until validated, quick rollback) reduce risk. Use Pod Disruption Budgets, multiple replicas, and test in a lower environment or a canary cluster first. Use rollout sequencing to upgrade dev before prod.
  • Compute Engine: VM Manager OS patch management applies patches by schedule and by label, with pre and post patch scripts, rollout in zone waves, and a compliance dashboard. Or bake new images (Packer, Image Builder) and replace instances through a MIG rolling update; immutable is safest.
  • Dependencies and base images: rebuild on a schedule so patched bases are picked up; scanning surfaces stale ones.

Secure cloud development environments (guide 1.5)

Bootstrapping: bake a custom image (base image plus language toolchains, the Google Cloud SDK, linters, company certificates, extensions), publish it in Artifact Registry, point the workstation configuration at it, and rebuild on a schedule. Use startup scripts for per-user setup.

AI assistance: Gemini Code Assist (completion, chat, code generation and review in IDEs and Git hosts, with an agent mode), Gemini Cloud Assist (understand and troubleshoot Google Cloud resources, run investigations across logs, metrics and configuration, in the console), Gemini CLI and Antigravity (agentic coding tools), and Model Context Protocol (MCP) servers, which give AI agents standardised access to tools and data. Google provides MCP servers for Google Cloud services; you can host your own (for instance on Cloud Run) and configure them in the IDE or agent. Secure them like any API: least-privilege service accounts, authentication on the endpoint, allow-lists of tools, audit logs, and human approval for destructive actions. For code-executing agents, isolate execution (for example GKE Agent Sandbox, which runs untrusted agent code with strong isolation in GKE).