Continuous improvement for existing solutions

Operational excellence

• Monitoring and logging strategy:
  • CloudWatch metrics (standard 1 or 5 minute, detailed 1 minute, high-resolution custom down to 1 second), alarms (static, anomaly detection, metric math, composite alarms to cut noise), dashboards, Logs with Logs Insights queries, metric filters, subscription filters, Contributor Insights, Synthetics canaries, RUM, Application Signals for SLOs on services, Container Insights and Lambda Insights.
  • The CloudWatch agent is needed for memory, disk and custom OS metrics; EC2 doesn’t publish them by default.
  • X-Ray traces requests across services; use it to find which downstream call is slow.
  • CloudTrail answers “who did what”; Config answers “what does the resource look like and how did it change”; VPC Flow Logs show network flows; ALB access logs show requests.
  • Centralize: cross-account and cross-Region CloudWatch observability with a monitoring account, Logs subscription to Kinesis or Firehose, then S3 and Athena or OpenSearch for retention and search. Set log retention (default never expires) to control cost.
• Alerting and automatic remediation:
  • Alarm to SNS for humans; alarm or event to EventBridge to Lambda or Systems Manager Automation for fixes.
  • Config rules with automatic remediation (SSM Automation documents) for compliance drift, for example closing a security group or enabling bucket encryption.
  • EC2 auto recovery alarms, Auto Scaling health replacement, ECS service restarts, Lambda destinations and dead-letter queues, Step Functions retries and catch.
  • AWS Health events through EventBridge for planned maintenance and incidents; Organizational Health view for multi-account.
• Deployment process improvement: replace manual or SSH deployments with CodePipeline and CodeDeploy; replace in-place with blue/green or canary; add automated tests, alarms during deployment and automatic rollback; use immutable AMIs from EC2 Image Builder; use CloudFormation (with change sets and drift detection) rather than console changes; use StackSets for many accounts.
• Prioritizing automation: automate the repetitive, error-prone, frequent and high-risk work first (patching, provisioning, deployments, backups, scaling, incident response), and the things that block recovery. Tools: Systems Manager (Automation, Run Command, State Manager, Patch Manager, maintenance windows), Lambda, EventBridge Scheduler, Step Functions.
• Configuration management automation: Systems Manager State Manager keeps instances in the desired state, Parameter Store and AppConfig hold configuration (AppConfig deploys gradually with validators and rollback), Inventory collects software, OpsWorks (Chef/Puppet) is end of life, Compliance reports on patch and association state. Quick Setup configures SSM across the organization.
• Failure scenario exercises: AWS Fault Injection Service (stop instances, throttle APIs, inject network latency, fail over RDS) with stop conditions on alarms; game days; Route 53 ARC and DR drills; chaos tests in pre-production first. Capture findings and update runbooks.
• Disaster recovery planning as an ongoing process: re-evaluate RTO/RPO when the workload changes, test restores, document, automate failover steps.
• Well-Architected Tool and the six pillars (operational excellence, security, reliability, performance efficiency, cost optimization, sustainability) frame reviews and improvement plans.

Security improvement

• Data retention, sensitivity and regulation: classify data (Macie discovers PII in S3), apply retention with S3 lifecycle, Object Lock (governance or compliance mode, legal hold), Glacier Vault Lock, AWS Backup Vault Lock; encryption with KMS; residency with SCP Region restrictions; Artifact for compliance reports; Audit Manager for evidence; CloudTrail log validation.
• Automated monitoring and remediation:
  • Config managed rules and conformance packs detect (public S3 buckets, unencrypted volumes, open SSH, unused IAM credentials), with remediation actions through SSM Automation.
  • Security Hub aggregates findings and automation rules or EventBridge trigger remediation.
  • GuardDuty findings through EventBridge: isolate an instance by swapping its security group, revoke a role’s sessions, snapshot for forensics.
  • Inspector continuous scanning for EC2, ECR and Lambda; rank by exploitability and reachability, then patch or rebuild the image.
  • Detective to investigate; Macie for data exposure.
• Secrets: move credentials out of code and environment variables into Secrets Manager (automatic rotation, cross-Region replication, resource policies, KMS) or Parameter Store SecureString (no automatic rotation, free standard tier). Use IAM database authentication or IAM roles where possible, Lambda extension or SDK caching to avoid per-request fetches. Find leaked keys with the IAM credential report, Access Analyzer unused access and Config.
• Auditing for least privilege: Access Analyzer unused access findings and policy generation from CloudTrail, last accessed information in IAM, remove wildcard actions, replace IAM users with roles and Identity Center, require MFA, SCPs for organization guardrails, permission boundaries for delegated administrators, review cross-account trust policies, IAM Access Analyzer custom policy checks in pipelines.
• Security at every layer (defence in depth): edge (CloudFront, WAF, Shield), network (private subnets, security groups, NACLs, Network Firewall, endpoints), compute (patching, Inspector, IMDSv2 required, Session Manager instead of SSH), application (input validation, Cognito, API authorizers), data (KMS, TLS, Macie, backups), identity (least privilege, MFA), detection (GuardDuty, Security Hub, CloudTrail).
• Traceability: organization CloudTrail with data events where required (S3 objects, Lambda, DynamoDB), CloudTrail Lake queries, Config timeline, VPC Flow Logs, DNS query logs, S3 server access logs or CloudTrail data events, ALB logs, X-Ray for requests. Protect logs with a separate account, Object Lock and log file integrity validation.
• Prioritizing automated responses: stop active threats first (credential exposure, public data, internet-exposed management ports), then known critical vulnerabilities with exploits, then hygiene.
• Patch and update process: Patch Manager with baselines and maintenance windows; patch policies through Quick Setup across the organization; ECS and EKS: rebuild images and redeploy rather than patching running containers; Lambda managed runtimes patched by AWS (you must move off deprecated runtimes); managed databases through maintenance windows; EC2 Image Builder pipelines with test and distribution to accounts; Inspector for what is unpatched.
• Backup process: AWS Backup plans by tag, cross-Region and cross-account copies, Vault Lock, restore testing, immutable (logically air-gapped) vaults against ransomware, KMS keys that don’t die with a compromised account, monitored by Backup Audit Manager.
• Remediation techniques: quarantine security group, detach IAM policy or add a deny, rotate secrets, revoke role sessions (add an inline policy denying tokens issued before a time), restore from clean backup, replace instances from a known good AMI, preserve evidence first (snapshot, memory capture).

Performance improvement

• Business requirements to measurable metrics: turn “fast” into latency percentiles (p50, p95, p99), throughput, error rate, availability. Define SLIs, SLOs and SLAs; use CloudWatch Application Signals and alarms against the SLOs. Use p99, not averages, for tail problems.
• Find bottlenecks (measure first):
  • Application latency: X-Ray service map, ALB target response time, Application Signals.
  • Database: Performance Insights (top SQL, wait events), slow query log, FreeableMemory, ReadIOPS, connection counts, replica lag.
  • Compute: CloudWatch CPU, memory (agent), network, EBS queue length, Compute Optimizer.
  • Storage: EBS VolumeQueueLength, burst balance, IOPS/throughput limits, EFS PercentIOLimit, S3 request rates and 503 SlowDown.
  • Network: instance bandwidth limits, NAT gateway limits (ErrorPortAllocation), cross-AZ hops, DNS.
• Typical fixes:

• High-performing architectures: Auto Scaling with target tracking, EC2 Fleet and Spot Fleet with diversified instance types, placement groups, Elastic Fabric Adapter, Nitro instances, Graviton.
• Global services: CloudFront (caching, signed URLs and cookies, origin shield, field-level encryption, Lambda@Edge, CloudFront Functions, origin groups for failover), Global Accelerator (static anycast IPs, health-based routing, non-HTTP), Route 53 latency routing, Local Zones, Wavelength, Outposts.
• Test remediation before rollout: load test in a staging environment (Distributed Load Testing on AWS), canary release, A/B with weighted routing, compare metrics.
• Adopt new technology and managed services when they remove a bottleneck or operational work: Aurora Serverless v2, Graviton, gp3, S3 Express One Zone, EKS Auto Mode, Lambda response streaming, Bedrock for AI features.
• Rightsize from observed utilization: Compute Optimizer needs at least 30 hours of metrics, looks back 14 days by default, and can use up to three months with enhanced infrastructure metrics; downsize overprovisioned, upsize saturated, change family where recommended.

Reliability improvement

• Understand growth: CloudWatch trends and Cost Explorer usage, forecast service quota consumption, schedule or predictive scaling before known peaks, Trusted Advisor service limits and Service Quotas alarms.
• Find weak points: single-AZ resources (single instance, single NAT gateway, single-AZ RDS), unreplicated data, hard-coded endpoints, synchronous dependencies, no health checks, no backups, shared fate across Regions, manual failover. Tools: Well-Architected reliability review, Trusted Advisor fault tolerance checks, Resilience Hub (assess an application against RTO/RPO targets and get recommendations), Fault Injection Service.
• Remove single points of failure:

• Data replication and self-healing: Aurora replicas and Global Database, DynamoDB global tables and PITR, S3 versioning and replication, EBS snapshots through Data Lifecycle Manager, EFS and FSx backups and replication, ElastiCache Multi-AZ. Self-healing: ELB health checks with Auto Scaling replacement, ECS and EKS restarting tasks and pods, Lambda retries and DLQs, SQS redrive.
• Scaling methodologies: horizontal over vertical; load balancers with health checks; target tracking; queue-based scaling; schedule for known events; predictive scaling for cyclical traffic; Aurora Auto Scaling for replicas; DynamoDB auto scaling or on-demand; set a minimum capacity that survives losing an AZ (N+1 across AZs).
• Quotas and limits: watch usage (Service Quotas CloudWatch metrics), request increases early, design around hard limits (API Gateway timeout 29 seconds, Lambda 15 minutes, SQS message size 1 MiB for standard queues (extended client for S3 payloads), SNS 256 KB messages, DynamoDB item 400 KB, S3 object 5 TB, Kinesis shard limits, Step Functions history limits). Spread across accounts or Regions when quotas bind.
• Backups and DR methods as in the earlier domains: pilot light, warm standby, DRS, AWS Backup.

Cost improvement

• Cost-conscious choices: Spot for fault-tolerant work (diversify instance types and AZs, capacity-optimized allocation, handle the two-minute notice), Graviton, rightsized and scheduled non-prod, scaling policies that scale in, serverless for low or spiky use, S3 Intelligent-Tiering and lifecycle, delete unused EBS volumes, snapshots, Elastic IPs and idle load balancers, gp3 instead of gp2.
• Find unused and underused resources: Trusted Advisor (low utilization EC2, idle load balancers, unassociated Elastic IPs, underused EBS, idle RDS), Compute Optimizer, Cost Optimization Hub, Cost Explorer rightsizing recommendations, S3 Storage Lens (incomplete multipart uploads, cold buckets), Config inventory, CloudWatch utilization metrics. Delete first, then rightsize, then commit.
• Pricing model adoption: analyze steady-state usage in Cost Explorer and use Savings Plans or RI recommendations; cover the baseline, not the peak. Check coverage and utilization reports. Prefer Compute Savings Plans when instance types or Regions will change, EC2 Instance Savings Plans or Standard RIs when stable, Convertible RIs when you want exchange. Marketplace sale of unused Standard RIs.
• Networking and data transfer: find big-ticket data transfer lines in the Cost and Usage Report. Typical fixes: S3 and DynamoDB gateway endpoints to bypass NAT charges, one NAT gateway per AZ while keeping same-AZ traffic local, CloudFront in front of S3 or ALB, keep services in the same AZ and Region when allowed, PrivateLink and TGW costs weighed against peering, compress responses, Direct Connect for sustained high volume, S3 Transfer Acceleration only when needed, replication only for data that needs it.
• Alerting and reporting: Budgets (cost, usage, RI/SP coverage, with notifications and actions), billing alarms in CloudWatch on EstimatedCharges (us-east-1, must enable billing alerts) built from expected usage patterns, Cost Anomaly Detection with monitors by service, account, tag or cost category, Cost and Usage Reports to S3 queried by Athena (hourly or daily, resource IDs, tags, amortized costs), Data Exports, QuickSight dashboards.
• Granular CUR investigation: enable resource IDs and split cost allocation data (for ECS and EKS), query with Athena by line_item_usage_type, product, tag and account to find drivers such as DataTransfer-Regional-Bytes, NatGateway-Bytes, TimedStorage.
• Tagging for cost allocation: standard keys (CostCenter, Project, Environment, Owner), enforced with tag policies and SCPs, activated as cost allocation tags in Billing; remember untagged resources and tags apply from activation onward; use cost categories to group.