Reliability and business continuity

Scaling compute

EC2 Auto Scaling

• An Auto Scaling group (ASG) keeps between min and max instances, starting at desired, spread over the AZs of its subnets. Instances come from a launch template (AMI, type, security groups, role, user data). Launch configurations are legacy.
• Scaling policies
  • Target tracking: hold a metric near a target (average CPU 50%, ALBRequestCountPerTarget). Simplest, usually the right answer.
  • Step scaling: add or remove by size of alarm breach.
  • Simple scaling: one adjustment then a cooldown. Older.
  • Scheduled: known peaks at set times.
  • Predictive scaling: forecasts from history (daily and weekly cycles) and scales ahead.
• Cooldown (default 300 seconds) and instance warm-up stop the group reacting before new capacity has any effect.
• Health checks: EC2 status checks by default; turn on ELB health checks so an instance that fails the load balancer is replaced. The health check grace period gives new instances time to boot before checks count.
• Lifecycle hooks pause an instance in Pending:Wait or Terminating:Wait (up to 48 hours) so you can run bootstrap or drain scripts through EventBridge, SNS or Lambda.
• Termination policy default: balance AZs first, then the instance with the oldest launch template or configuration, then closest to the next billing hour. Scale-in protection protects named instances.
• Instance refresh rolls out a new AMI or template across the group, with a minimum healthy percentage. Warm pools keep pre-initialized stopped instances for faster scale-out.
• Mixed instances policy combines On-Demand and Spot across several instance types; attribute-based instance type selection picks types by vCPU and memory.
• An ASG that “keeps terminating instances” is usually failing a health check (wrong port or path, app slow to start, security group), or the launch is failing (AMI gone, key or KMS permission, subnet out of IPs, instance limit). Check the activity history.
• Rebalancing: if an AZ was impaired, the group launches in other AZs, then rebalances.

Other compute scaling

Queue-based scaling: scale workers on SQS backlog per instance (queue depth divided by instances), not on queue depth alone.

Caching

• Lazy loading (cache-aside) caches on a miss and can serve stale data; write-through updates the cache on every write and avoids staleness but caches data that may never be read. Use TTLs with both.
• Put session state in ElastiCache or DynamoDB so any instance can serve any user and the web tier can scale and be replaced freely. ALB sticky sessions are a weaker alternative.
• ElastiCache needs the app to be changed to use the cache; CloudFront doesn’t.
• Watch CacheHitRate, Evictions (memory pressure, scale up or out) and CurrConnections. Serverless ElastiCache scales automatically.

Scaling databases

DynamoDB throttling: hot partition keys, a provisioned limit set too low, or sudden spikes beyond what auto scaling reacts to; fixes are better key design, on-demand mode, raising capacity, or DAX for reads. Auto scaling reacts in minutes, so on-demand suits unpredictable spikes.

Load balancers and health checks

• Health checks: protocol, path, port, interval, timeout, healthy and unhealthy thresholds, and (ALB) a success matcher such as 200-299. An unhealthy target stops receiving traffic; if all targets are unhealthy, ALB sends to all of them (fail-open).
• Troubleshooting unhealthy targets: the target’s security group must allow the load balancer’s security group (or health check source) on the health check port; the path must return the matcher code (a redirect or auth page is a failure); the application must listen on the right port; and a slow start or too-tight timeout can fail a slow app.
• Cross-zone load balancing: always on for ALB (no charge for inter-AZ data); off by default for NLB and GWLB (inter-AZ data charges when enabled).
• Connection draining / deregistration delay (default 300 seconds) lets in-flight requests finish when a target is removed or unhealthy.
• ALB errors: 502 bad gateway (target closed the connection or sent a malformed response), 503 no healthy targets, 504 gateway timeout (target didn’t answer within idle timeout, default 60 seconds), 460/463 client-related. Turn on access logs to S3 to diagnose.
• Target types and weighted target groups support blue/green and canary at the ALB. Automatic target weights and zonal shift (ARC) steer traffic away from a failing AZ.

Route 53 health checks and failover

• Health checks monitor an endpoint (HTTP, HTTPS, TCP, with optional string match), a calculated check (combines other checks), or a CloudWatch alarm (the way to check things inside a private VPC, since checkers are on the public internet). Checkers sit in multiple Regions; an endpoint is healthy when enough checkers report success (default: more than 18% of them).
• Attach a health check to a record for failover (active-passive), weighted, latency or multivalue routing. Alias records to AWS resources can use Evaluate target health without a separate check.
• Troubleshoot: firewall or security group not allowing the Route 53 health checker IP ranges, wrong path or port, a string match that doesn’t appear in the first 5,120 bytes, or a private endpoint being checked directly.
• Application Recovery Controller (ARC) adds routing controls and readiness checks for controlled cross-Region failover; use it when failover must be explicit and safe from a single operator switch.

Fault tolerance

• Multi-AZ: place resources across at least two AZs. RDS Multi-AZ instance keeps a synchronous standby (not readable) and fails over automatically in about one to two minutes by flipping the DNS name; Multi-AZ DB cluster (MySQL, PostgreSQL) has two readable standbys and faster failover; Aurora stores six copies across three AZs and fails over to a replica in about 30 seconds or less.
• Test failover with reboot with failover. Applications must reconnect through the endpoint name, not an IP, and not cache DNS for long.
• ElastiCache Multi-AZ with automatic failover for Redis/Valkey. EFS and S3 are multi-AZ by design (EFS One Zone and S3 One Zone-IA are not). EBS is single-AZ: snapshot it.
• NAT gateway is zonal: use one per AZ and route each private subnet to its own. Elastic IP remapping gives manual failover for a single instance.
• Static stability: remove dependencies on a control plane during an event. Pre-provision capacity instead of relying on scaling during a failure.
• EC2 auto recovery (simplified, on by default for supported instances) restarts an impaired instance on new hardware. Placement and dedicated hosts change the failure domain, not the AZ.
• AWS Fault Injection Service runs chaos experiments (stop instances, throttle APIs, inject latency, fail over) to prove the design.
• Service quotas are a reliability issue: monitor them with Trusted Advisor and CloudWatch usage metrics, and request increases before a launch.

Backup

• AWS Backup is the central, policy-driven service. Backup plans hold rules (schedule, backup window, lifecycle to cold storage, retention, copy to another Region or account) and assign resources by tag or ARN to a vault.
• Supports EC2 (AMIs), EBS, RDS, Aurora, DynamoDB, EFS, FSx, S3, DocumentDB, Neptune, Storage Gateway, VMware, and more. Continuous backup (point-in-time recovery) works for S3, RDS and Aurora, and is a plan option.
• Backup Vault Lock makes a vault WORM (write once, read many): compliance mode can’t be undone after the grace period. Logically air-gapped vaults and cross-account copy protect against a compromised account. Backup Audit Manager reports on compliance. Organization-wide backup policies apply plans to all accounts.
• Backups are incremental after the first. Encryption uses KMS; copies keep or change the key.
• Individual services’ own mechanisms: EBS snapshots (incremental, stored in S3, copy cross-Region, Data Lifecycle Manager automates creation and retention; the Recycle Bin recovers deleted snapshots and AMIs), RDS automated backups (daily snapshot plus transaction logs, retention 0 to 35 days) and manual snapshots (kept until deleted), DynamoDB on-demand backups and PITR (35 days), S3 replication and versioning.
• Take consistent backups of databases on EC2 by quiescing or using application-aware snapshots (Systems Manager VSS for Windows).

Restoring

• RDS point-in-time restore (PITR) restores to any second within the retention window (up to the latest restorable time, about the last five minutes). It always creates a new instance with a new endpoint; you cut applications over. Restoring from a snapshot is the same: new instance.
• Aurora backtrack (MySQL-compatible) rewinds a cluster in place without a restore, for quick undo. Aurora clones copy fast using copy-on-write for test or recovery.
• DynamoDB PITR restores to a new table (35 days, per second); on-demand backups have no expiry. Restores don’t copy auto scaling settings, TTL or streams automatically in all cases: re-apply them.
• EBS: create a volume from a snapshot (lazy-loaded; use fast snapshot restore if first-read speed matters). EC2: launch from an AMI.
• EFS, FSx, S3: restore through AWS Backup; S3 can restore to the same or a new bucket.
• RTO vs RPO: RTO is how long you can be down; RPO is how much data you can lose. Backup frequency sets RPO; restore speed and automation set RTO. A read replica promotion is faster than a snapshot restore; cross-Region replicas give a low RPO for regional loss. Cheaper = slower recovery.

Versioning

• S3 versioning keeps every version and uses delete markers; permanently deleting needs the version ID. Once enabled it can only be suspended. MFA delete requires MFA to delete versions or change versioning (root only). Pair with lifecycle rules for noncurrent version expiration, since old versions cost money. Versioning is required for replication (CRR and SRR) and Object Lock (retention and legal hold, governance and compliance modes).
• FSx keeps point-in-time copies: Shadow Copies on FSx for Windows (users restore “previous versions”), snapshots on ONTAP and OpenZFS, and automatic daily backups for all types.
• AMI and snapshot versioning: tag by date, use Image Builder versions, and launch template versions.
• DynamoDB has no versioning; use PITR or an optimistic locking version attribute.

Disaster recovery strategies

• Tools: cross-Region copy of snapshots and AWS Backup; RDS and Aurora cross-Region read replicas; Aurora Global Database (storage-level replication, typically under one second, promote a secondary in about a minute); DynamoDB global tables; S3 CRR; Route 53 failover or latency routing with health checks; Elastic Disaster Recovery (DRS) for continuous block-level replication of servers into a low-cost staging area with launch in minutes.
• Pick the cheapest strategy that meets the stated RTO and RPO. “Minutes of data loss, hour of downtime” is pilot light or warm standby; “no downtime, no data loss” is active/active.
• Practise and automate: runbooks in Systems Manager Automation, regular restore tests, documented failback.