Ensuring solution and operations reliability

Operational excellence pillar

The Well-Architected Framework’s operational excellence pillar asks you to run workloads efficiently and keep improving. Its main recommendations:

• Ensure operational readiness and performance through clear ownership, runbooks, capacity planning, on-call, and SLOs that define “good enough”.
• Manage incidents and problems: detect fast, mitigate, communicate, learn by postmortem, fix causes.
• Manage and optimise cloud resources: monitor utilisation and cost, right-size, review regularly.
• Automate and manage change: infrastructure as code, CI/CD, automated tests and rollbacks, small frequent changes, controlled change management.
• Continuously improve and innovate: review incidents and metrics, adopt new capabilities, reduce toil (manual, repetitive, automatable work that grows with the service).

Typical “operational excellence” answers are automate, make observable, define ownership, test recovery, and learn from failures. Wrong answers rely on heroics, manual steps and one person who knows how it works.

Observability

Cloud Monitoring

• Metrics from Google Cloud services automatically; custom metrics through the API or OpenTelemetry; Managed Service for Prometheus collects Prometheus metrics from GKE and elsewhere (Altostrat and EHR already run Prometheus or other open source tools, which can be kept and unified). Ops Agent collects logs and metrics from Compute Engine VMs (memory and disk are not visible without it).
• Metrics scope: one project can view metrics of many projects. Use a dedicated monitoring project for central visibility across an organization (EHR’s requirement for centralised visibility).
• Dashboards, uptime checks (public or private endpoints from several regions; use for black-box availability), synthetic monitors, SLO monitoring (define an SLO and see budget burn), and Service Monitoring for Cloud Run, GKE and App Engine services.
• Metric types: gauge, delta, cumulative; MQL and PromQL for queries. Use log-based metrics to turn log patterns into metrics and alerts.

Cloud Logging

• Logs arrive in log buckets through sinks; _Required (400 days, audit logs, not changeable) and _Default (30 days, configurable). Create user-defined buckets for custom retention, regional placement for data residency, and CMEK.
• Exclusion filters drop noisy low-value logs to save cost; log sinks export to Cloud Storage, BigQuery, Pub/Sub or another bucket. Aggregated sinks at organization or folder level give centralised logging.
• Log Analytics runs SQL over log buckets upgraded for analytics; link to BigQuery.
• Error Reporting groups and counts exceptions. Logs-based alerts notify on a matching log.
• Consistent logging, retention and alerting across multiple environments (EHR) means: structured JSON logs, an agreed format, a central logging project with sinks, defined retention per class of log, and uniform alert policies defined as code.

Tracing, profiling, benchmarking

Profiling finds code-level cost; benchmarking compares options in a controlled setup; load testing checks behaviour at scale. Do not confuse them.

Alerting strategies

• Alert on symptoms that affect users (error rate, latency, SLO burn), not on every cause (CPU 80%). Cause-based alerts create noise and are ignored: EHR’s email alerts “are often ignored”, the classic anti-pattern.
• Alert policies have conditions (threshold, absence, rate of change, forecast), duration windows, and notification channels (email, Slack, PagerDuty, SMS, webhooks, Pub/Sub). Route by severity: page for urgent user impact, ticket for non-urgent.
• Multi-window, multi-burn-rate SLO alerts page quickly on a fast burn (for example 14.4x over 1 hour) and ticket on a slow burn (for example 1x over 3 days).
• Include documentation (runbook link) in the alert; use snooze and maintenance windows; group related alerts into incidents to cut noise.
• Alerts must be actionable, have an owner and be reviewed regularly; delete alerts nobody acts on.

Reliability engineering

SLIs, SLOs, SLAs and error budgets

Use the budget to decide: while budget remains, ship features and take risk; when it is spent, freeze risky releases and invest in reliability. The SLO is chosen with the product owner on what users need; 100% is the wrong target (too costly, and users cannot tell). EHR’s 99.9% is an SLO-style target for customer-facing systems. SLIs for common systems: availability and latency for request/response, freshness and correctness for data pipelines, durability for storage, throughput for batch.

Reliability practices

• Redundancy across zones and regions, health checks and autohealing, graceful degradation (serve stale or reduced content), load shedding, timeouts, retries with backoff and jitter, circuit breakers, and capacity headroom.
• Avoid cascading failure: bound queues, limit retries, isolate noisy neighbours, use Pub/Sub to buffer, set autoscaler maxima so a bug cannot create unbounded cost.
• Toil reduction: automate recurring manual tasks, since SRE guidance caps toil at about half of an engineer’s time.
• On-call with clear rotation, escalation and training, and incident command roles (incident commander, communications, operations) for major events.

Deployment and release management

• Release strategies: rolling, blue/green, canary, with automatic rollback on SLO or error signal; feature flags; dark launches; regional staged rollout (one region, then more); release windows away from peaks (retail freeze before holidays).
• Tooling: Cloud Deploy for delivery pipelines with approvals and rollback, Cloud Run revisions and traffic splitting, GKE rollouts, MIG rolling and canary updates, Config Sync for GitOps with auditable history, Terraform with reviewed plans.
• Version control for everything, small changes, automated pre-release verification, post-deploy verification (synthetic checks, SLO watch), and a tested rollback path. Database changes use backward-compatible steps.
• Change management: classify changes (standard, normal, emergency), peer review, record in a change log (Cloud Audit Logs also record), communicate to stakeholders.
• Version and dependency management: GKE release channels and maintenance windows for upgrades, base image refresh, patching through VM Manager, deprecation tracking.

Supporting deployed solutions

• Support model: tiers (L1, L2, L3), runbooks, escalation to Google Cloud Support (plans: Standard, Enhanced, Premium; Customer Care; Technical Account Manager for enterprise), the Personalized Service Health dashboard and status alerts for Google-side events, and Cloud Support cases with priority P1 to P4.
• Incident handling: detect, declare, assign roles, mitigate first, communicate regularly, resolve, then blameless postmortem (timeline, impact, root causes and contributing factors, what worked, action items with owners and dates, shared widely).
• Capacity and quota management: monitor quota usage and alert at thresholds, request increases ahead of launches, understand regional resource availability and reservations for scarce GPUs.
• Maintenance: GKE and Cloud SQL maintenance windows, live migration, patch schedules.
• Gemini Cloud Assist investigations and recommendations speed diagnosis. Active Assist (Recommender) shows reliability, security and cost suggestions.
• Knowledge: runbooks, architecture docs, dashboards and alerts as code so new staff can operate the system.

Quality control measures

• Define quality gates: unit and integration test coverage, static analysis, vulnerability scans, policy checks, performance thresholds, SLO checks after deploy. Block release on failure.
• Code review, pair programming, coding standards, dependency pinning, and definition of done including observability and runbooks.
• Production verification: canary analysis, synthetic monitoring, error budget tracking, defect escape rate, change failure rate and mean time to restore (DORA).
• Data quality: validation in pipelines (Dataplex data quality, Dataflow validation), schema checks, reconciliation counts after migration.
• AI quality: evaluation sets and metrics, grounding checks, human-in-the-loop review (Cymbal), monitoring for drift, safety filters (Model Armor), audit and explanation records (Altostrat).
• Architecture review against the Well-Architected Framework and the Architecture Center reference designs; periodic operational readiness reviews before launch.

Ensuring reliability in production

Chaos engineering

Deliberately inject failure to learn how the system behaves: terminate instances, kill pods, add latency, block a dependency, fail a zone, revoke a credential. Rules: start with a hypothesis about steady state, begin in non-production or with a small blast radius, have an abort switch, run during working hours with the team present, and fix what you find. Tools include Chaos Mesh, LitmusChaos, Gremlin, and manual game days; in GKE, delete pods or drain nodes to test PodDisruptionBudgets; in Compute Engine, gcloud compute instances simulate-maintenance-event and zone-failure drills for MIGs.

Penetration testing

You may test your own Google Cloud resources without notifying Google, as long as you follow the Acceptable Use Policy and do not attack other customers or Google infrastructure (no denial-of-service tests against shared infrastructure). Use qualified testers, written scope and rules of engagement, test staging copies where possible, and feed findings into the backlog. Web Security Scanner and Security Command Center find common web vulnerabilities and misconfigurations automatically; regulations such as PCI DSS require periodic pen tests.

Load testing

Test at expected, peak and above-peak load against a production-like environment: confirm autoscaling reacts in time, find bottlenecks (database connections, quotas, cold starts, hot keys), measure p95 and p99 latency (not just average), verify graceful degradation. Warm caches realistically, use realistic data distribution, run long enough to catch leaks, and compare results against SLOs. Coordinate quota increases and (for sudden spikes) pre-warm, because some resources (load balancers, minimum instances) need advance preparation.

Related practices

• Disaster recovery drills proving RTO and RPO; backup restore tests.
• Canary and shadow traffic replay for risky changes.
• Capacity planning from trends, with headroom.
• Postmortem follow-through: action items tracked and closed.

Case study cues