Managing incidents and troubleshooting

Incident response process

An incident is any event that threatens an SLO enough that a coordinated response is needed. Declare early; an incident that turns out small costs little, one declared late costs a lot. Define severity levels in advance (for example SEV1 a full outage or data loss, SEV2 a major feature down or error budget burning fast, SEV3 a minor degradation) because severity decides who is paged and how often you update.

Lifecycle

1. Detect: an alert (SLO burn, uptime check, log-based alert), a customer report, or an engineer notices. Good detection is symptom-based.
2. Triage: acknowledge the page, assess impact (who, how many, how bad) and assign a severity.
3. Declare and assemble the response; open the incident channel or video bridge and a live incident document.
4. Mitigate to stop user impact (roll back, drain, add capacity, disable a feature). Don’t wait for the cause.
5. Resolve: confirm the SLI has recovered and holds, close the incident.
6. Learn: blameless postmortem, action items tracked to completion.

Roles (Google’s incident command model)

Principles: clear command (everyone knows who is in charge), one operations lead (no two people changing production at once), separate roles in a small team if possible but never skip the IC, and explicit handoffs for long incidents (a tired IC at hour eight is a risk). The IC role summary is “coordinate, communicate, control”.

Communication

• A single channel (chat room or bridge) and a living document with timeline, current hypotheses, actions and owners.
• Regular updates at an agreed cadence (every 30 minutes for a SEV1) even if the news is “no change”. Say what is known, impact, what’s being done and when the next update is.
• Use internal and external wording; customers get plain impact statements and a status page entry, not speculation.
• Google Cloud customers use Personalized Service Health (events affecting their projects, with alerts, export to Cloud Logging, and notification channels) and the public status dashboard to tell whether the problem is theirs or Google’s, then open a Cloud Customer Care case with the right priority (P1 critical down to P4) and attach evidence.
• Cloud Monitoring incidents show the alert, its documentation, links to logs and charts; acknowledge them so teammates don’t duplicate effort.

Postmortems

A blameless postmortem is a written record of an incident that focuses on systems and processes, not people. The aim is learning. Blame makes people hide information.

Typical contents: summary, impact (duration, users, SLO and error budget consumed), timeline with detection and mitigation times, root cause and contributing factors (ask “why” repeatedly), what went well and badly, where luck helped, and action items with an owner, priority and due date (prevent, detect, mitigate faster). Review it in a meeting and share it widely. Write one for any user-visible outage above a severity threshold, any data loss, and any incident where an on-call intervention was needed or the process failed.

Also: error budget accounting (the incident consumed X% of budget, which feeds the budget policy), trends across postmortems, game days / Wheel of Misfortune exercises to practise, and disaster recovery tests. Track time to detect, time to mitigate and time to resolve to measure and improve response.

Mitigating incident impact on users (guide 3.3)

Mitigation reduces or stops the impact without necessarily understanding the cause. The options in rough order of speed and safety:

Guidance:

• Mitigate before you diagnose. A rollback that works tells you where the cause is.
• Prefer reversible, low-risk actions; avoid new code changes (a forward fix) under pressure unless rollback is impossible, for example when a data migration already ran.
• Check what changed: deployments, config and flag changes, scaling events, traffic pattern, dependency incidents, certificate or quota expiry. Cloud Audit Logs and the Cloud Deploy history list recent changes.
• Capacity addition has traps: autoscaling limited by quota or max instances, cold start time, a downstream (database, API) that gets crushed by the extra load, health checks marking healthy new instances unhealthy under load.
• Prove recovery by watching the SLI, not the absence of alerts.
• Prepare ahead: health checks and autohealing, multi-zone and multi-region layout, tested rollback procedure, runbooks, circuit breakers and retries with exponential backoff and jitter (to avoid retry storms), and graceful degradation.

Troubleshooting (guide 4.4)

Approach: restate the symptom, check what changed, check the golden signals (latency, traffic, errors, saturation), use logs, metrics and traces to narrow down by component, form a hypothesis, test it, mitigate. Use Gemini Cloud Assist investigations for a fast first pass over logs, metrics and recent changes, then confirm.

Infrastructure

CI/CD pipeline

Cloud Build: read the build log (Cloud Logging, step-by-step), check the failing step’s exit code, the service account permissions, timeout, substitutions, private network reachability, quota on concurrent builds or machine types, secret access. Cloud Deploy: the rollout and job run logs; render errors (Skaffold), execution service account permissions, target connectivity, verification failures, approvals waiting, automations suspended; pipeline stuck because a previous rollout is still pending. Trigger not firing: filters and regex, repository connection. Supply chain: image denied by Binary Authorization (check the audit log for the reason: missing attestation, wrong digest vs tag, new image not signed) or vulnerability policy.

Application

• Error Reporting groups exceptions, shows first and last seen, and notifies on new errors. It makes a regression after a release obvious (a new error group).
• Logs Explorer filtered by trace, severity and revision; compare before and after a release.
• Metrics: error ratio by version or revision, latency percentile, saturation (CPU, memory, threads, connection pool), queue depth.
• Traces: find the slow or failing span across services (see the monitoring notes). Profiler finds hot code. Cloud Debugger and Snapshot Debugger were shut down, so rely on structured logs, traces and profiles, and reproduce in a lower environment.
• Common causes: bad release (roll back), config or flag, dependency failure or timeout (retry storm), resource leak, thundering herd after a restart, expired certificate or credential, quota (API rate limit).

Observability issues (the monitoring system itself)

Performance and latency

1. Where is the latency: client, network, load balancer, service, dependency, database? The load balancer logs and metrics (backend_latencies, total_latencies) separate edge from backend time.
2. Trace waterfall to find the slow span; Profiler for CPU and memory hot spots; Query Insights for slow SQL.
3. Check saturation: CPU throttling (Pod limits), memory pressure and garbage collection, thread or connection pool exhaustion, disk IOPS, network egress limits, Cloud NAT port exhaustion.
4. Check cold starts (Cloud Run min instances, startup CPU boost, smaller images), autoscaler lag (cool-down, target too high), cross-region calls, and missing caching or CDN.
5. Fixes: scale or right-size, cache (Memorystore, Cloud CDN), batch or parallelise calls, add indexes, connection pooling, move the dependency closer, and use timeouts, retries with backoff and circuit breakers so one slow dependency doesn’t cascade.