Applying SRE principles and bootstrapping the organization

SLIs, SLOs and SLAs

Rules that get tested:

• The SLO is stricter than the SLA, so you notice and fix problems before you owe anyone money.
• An SLO is never 100%. It’s unaffordable, users can’t tell the difference past a point (their ISP and device fail more often), and it leaves no room to ship changes.
• Measure what users experience, as close to the user as possible (load balancer metrics, not CPU). CPU is a cause, not a symptom of user pain.
• Few SLIs are better than many. Pick the ones that matter to critical user journeys.
• Choose the SLI type by the kind of service:

• Percentiles beat averages for latency: an average hides the slow tail. Use a threshold (“95% under 300 ms”) rather than a mean.
• Valid events exclude things that aren’t the service’s fault, such as 4xx client errors from malformed requests. Excluding genuine failures inflates the SLI.

The nines

Allowed downtime for an availability SLO, over a 30-day month (and a year):

Opportunity cost: each extra nine costs roughly an order of magnitude more (redundancy, multi-region, slower release, more people). It also shrinks the budget to near zero, which stops change. If a dependency offers only 99.9%, your service can’t honestly promise 99.99% without mitigation. For serial dependencies, availabilities multiply: three 99.9% components in a row give about 99.7%.

Error budgets

Error budget = 1 − SLO. A 99.9% SLO over 30 days allows 0.1% bad events, which is 43 minutes if the SLI is time-based. For request-based SLIs it’s 0.1% of requests, so 1,000 failed requests in 1 million.

Budgets resolve the tension between developers (want to ship) and SREs (want stability). Both sides agree to the SLO, and the budget decides:

• Budget remaining → ship features, take risks, run experiments and chaos tests.
• Budget nearly spent or exhausted → slow or freeze feature releases, work on reliability, only ship fixes and security patches, until the window recovers.

An error budget policy is written down in advance and approved by product and engineering. It says what happens at defined thresholds (for example: alert owners at 50% spent, freeze releases at 100%, escalate to leadership if a freeze is disputed). The key exam point: the policy, not the opinion of the loudest person, decides, and the decision belongs to the agreed owners, not the on-call alone.

Budget spend should be shared sensibly: not one risky release consuming everything. Track what burned the budget (bad rollouts, dependency outages, capacity) and fix the biggest source first.

Burn rate and alerting on SLOs

Burn rate is how fast you consume budget relative to the “even” rate. Burn rate 1 uses exactly the whole budget by the end of the window. Burn rate 10 on a 30-day window would exhaust it in 3 days.

Common recommended settings for a 30-day SLO:

The short window makes the alert stop firing soon after recovery. The long window stops brief blips from paging. Cloud Monitoring creates SLO burn-rate alerting policies for you: you give a lookback duration and a threshold.

SLOs in Google Cloud

• Cloud Monitoring SLO monitoring works on a service. Services are detected automatically (Cloud Service Mesh / Istio, Cloud Run, App Engine, GKE services and workloads) or defined as custom services.
• SLI types: request-based (good requests / total requests, e.g. availability and latency from built-in metrics) and windows-based (fraction of time windows, usually 1 minute, in which a criterion holds, e.g. “latency p99 under 500 ms in the window”).
• Compliance period: calendar (week or month, resets at the boundary) or rolling (last N days). Rolling suits user-facing services because there’s no “reset cliff” that encourages risky changes just after a boundary.
• Cloud Service Mesh provides SLO definitions out of the box for services in the mesh. Cloud Run services show SLOs in the console.
• Create the SLO in the console, the API or Terraform, so it’s code-reviewed. Alert on burn rate, not on raw availability.
• SLO charts show error budget remaining and burn rate; those are the numbers to watch in reviews.

Toil and the SRE way of working

Toil is work that is manual, repetitive, automatable, tactical (interrupt-driven), has no enduring value, and grows linearly with the service. Examples: manually resizing disks, hand-approving routine quota requests, restarting a leaky process. Not toil: designing automation, a postmortem, the on-call response itself (when it needs judgment).

• Google’s guideline: keep toil under 50% of an SRE’s time; the rest is engineering work that reduces future toil.
• Fix toil by automating, removing the cause (bug, missing self-healing), or pushing the work back to the owning team. If toil keeps consuming the budget of time, hand the service back to developers.
• Blameless culture, postmortems, gradual change (small, frequent, reversible), and simplicity are the other recurring themes.
• Feedback loops: the cycle of SLO review, incident data, postmortem action items and the backlog. Reliability work must be tracked and prioritised like feature work, with owners and dates. Also the DORA delivery metrics: deployment frequency, lead time for changes, change failure rate, failed deployment recovery time (and rework rate in the newer five-metric set). Use them to see whether speed and stability are improving together.

Managing the service lifecycle

Services pass through planning, deployment (launch), maintenance and retirement.

Design reviews and early SRE involvement are cheaper than fixing reliability after launch.

Capacity planning

Capacity is about having enough resources for forecast demand plus headroom for failure and spikes, without paying for idle ones.

• Forecast organic growth and planned events (launches, sales). Add headroom for N+1 or N+2 (surviving the loss of a zone), and for the time capacity takes to arrive.
• Load test to learn per-instance limits before relying on autoscaling.
• Quotas and limits are a first-class risk. Per-project quotas (CPUs per region, IPs, API rates) stop scale-ups. Check them in Quotas & System Limits, set alerts at a percentage of quota usage (quota metrics in Cloud Monitoring), and request increases early since some need review. Autoscaling that hits a quota silently fails to scale: a typical troubleshooting question. Use quota adjusters and quota preferences (Cloud Quotas API) to manage them as code.
• Reservations ensure Compute Engine capacity exists in a zone when you need it: they’re resources that hold VMs for you and are billed while held. Reservations can be shared across projects. Future reservations request capacity for a date. Reservations guarantee capacity; committed use discounts (CUDs) give a price discount; the two are separate and are often combined.
• Dynamic Workload Scheduler queues requests for scarce accelerators (GPUs, TPUs). Flex-start mode runs a job when capacity becomes available (suited to batch, training), and Calendar mode reserves a block for a fixed period. Use it for AI/ML batch work rather than holding expensive idle reservations.
• Spread across zones (regional MIGs, regional GKE clusters) for resilience, and across regions where the SLO demands it.

Autoscaling

Also: Kubernetes Pod Disruption Budgets keep a minimum available during voluntary disruption (upgrades, scale-down). Cloud Run, App Engine and Cloud Functions scale to zero. Autoscaling doesn’t fix a failing dependency or a quota; it can make a downstream overload worse, so combine it with limits, backoff and load shedding.

Mitigating incident impact (SRE view)

Covered in depth in the incident management notes. The principle: stop user pain first, find root cause later. Drain or redirect traffic away from the bad component, add capacity, or roll back. Rollback is usually faster and safer than a forward fix.

Foundations: resource hierarchy (guide 1.1)

Hierarchy

Organization → folders → projects → resources. The organization node comes from Cloud Identity or Google Workspace. IAM policies and organization policies are inherited downward and are additive for IAM (a child can’t remove a parent’s grant, except with deny policies).

• The project is the unit of resource ownership, quota, billing link, API enablement and a common isolation boundary.
• Folders group projects (by environment, business unit, or team) so policy applies once. Folders can nest (up to 10 levels).
• Application-centric design: a project per application per environment (shop-dev, shop-prod), grouped into folders by team or business unit. Environment separation (dev, staging, prod in separate projects) limits blast radius and allows different IAM.
• Use labels and tags (tags are resource-manager tags that IAM and policies can key off, labels drive billing reports).
• Bootstrap: verify the domain, set up Cloud Identity, create admin groups (grant roles to groups, never individuals), link billing, create the organization policies and folders, and set up foundation projects (logging, monitoring, networking, security). The Google Cloud setup checklist and enterprise foundations blueprint (Terraform) automate this.

IAM and organization policy

• IAM answers “who can do what”; organization policy answers “what is allowed at all”, whatever the IAM role. Constraints to know:
  • constraints/gcp.resourceLocations limits where resources can be created (data residency).
  • constraints/iam.disableServiceAccountKeyCreation: no downloadable keys.
  • constraints/iam.allowedPolicyMemberDomains (domain restricted sharing).
  • constraints/compute.vmExternalIpAccess, compute.requireOsLogin, compute.skipDefaultNetworkCreation.
  • Custom constraints exist for resource-specific rules.
• Prefer predefined roles, then custom roles; avoid basic roles (Owner, Editor, Viewer) in production. Use IAM Recommender to remove excess permissions, Policy Troubleshooter to debug access, Policy Simulator to test changes.
• IAM Conditions (time, resource name, tags), deny policies, Privileged Access Manager for just-in-time elevation.

Service accounts

• Types: user-managed (you create), default (Compute Engine and App Engine defaults, which carry the broad Editor role in old projects and should be avoided), and service agents (Google-managed, in Google-owned projects).
• Best practices: one dedicated service account per workload, least privilege, attach it to the resource (VM, Cloud Run service, GKE via Workload Identity Federation for GKE) so no key is needed. Avoid service account keys; they leak. If code outside Google Cloud needs access use Workload Identity Federation. Use impersonation for humans and pipelines: roles/iam.serviceAccountTokenCreator lets a principal mint short-lived tokens for the account; roles/iam.serviceAccountUser lets a principal attach the account to a resource (a classic privilege-escalation path, so grant it narrowly, on the specific service account rather than the project).
• Keep separate service accounts for build, deploy and runtime, and by environment.
• Monitor with the service account insights and last authentication info, and disable unused accounts before deleting.

Shared networking

Multi-project monitoring and logging

• Metrics scope: one scoping project can view metrics from many monitored projects (up to 375) so one dashboard and alert policy sees the whole application. Alerting policies and dashboards live in the scoping project. Put the scoping project in a dedicated “monitoring” project, not a production one.
• Aggregated log sinks at the folder or organization (--include-children) route logs from every project to a central bucket, BigQuery dataset or Pub/Sub topic. Use for audit and security retention.

Data residency

Regions and multi-regions of services, organization policy resource locations, Assured Workloads (compliance regimes with controls, e.g. EU sovereign), CMEK/Cloud KMS with key location control, log bucket location (choose region when creating the bucket; the _Default bucket location can be set by organization-level log settings), and Access Transparency / Access Approval. Data in log buckets, backups and BigQuery datasets all have locations to check.

Foundations: managing infrastructure (guide 1.2)

Google-recommended practices: everything in version control, changes by pull request with review and a plan shown, validated by policy checks (for example terraform validate, Terraform plan checked against organization policy with policy validation tooling) before applying, applied by a pipeline with a dedicated service account and not from laptops, separate state per environment, and drift detection. Don’t hand-edit resources that Terraform owns.