Monitoring, logging, analysis, remediation and performance optimization

CloudWatch metrics

• A metric is a time-ordered set of data points, identified by namespace, name and dimensions (name/value pairs such as InstanceId). Each unique dimension combination is a separate metric.
• Standard EC2 metrics arrive every 5 minutes. Detailed monitoring gives 1-minute data for a charge. Custom metrics are standard resolution (1 minute) or high resolution (down to 1 second).
• Metrics are kept for 15 months, rolled up: 1-second data for 3 hours, 1-minute for 15 days, 5-minute for 63 days, 1-hour for 455 days.
• What CloudWatch sees by default for EC2: CPU, network, disk I/O (operations and bytes), status checks. It cannot see memory, disk space used, swap or processes — the hypervisor doesn’t know. You need the CloudWatch agent for those.
• Metric math combines metrics with expressions (error rate = errors / requests). Metric streams push metrics continuously to Firehose for third parties or S3. Metrics Insights queries metrics with SQL-like syntax across many resources.
• Anomaly detection builds a model of expected values (with daily and weekly seasonality) and alarms on a band instead of a fixed threshold.
• Publish custom metrics with PutMetricData, or with embedded metric format (EMF) — structured log lines that CloudWatch Logs turns into metrics automatically. EMF is the usual answer for Lambda and containers.

CloudWatch agent

The unified agent collects system-level metrics (memory, disk, swap, per-process via procstat) and log files from EC2 and on-premises servers, and from Amazon ECS and EKS.

Troubleshooting an agent that reports nothing: check the role permissions, the agent is running (amazon-cloudwatch-agent-ctl -a status), the config file path, network path to CloudWatch (VPC endpoints for monitoring and logs in private subnets), and the agent log at /opt/aws/amazon-cloudwatch-agent/logs/.

Container Insights collects CPU, memory, network and storage metrics per cluster, service and pod, and has an enhanced observability mode for EKS. Application Signals gives service-level objectives and application maps for instrumented applications. Lambda Insights is an extension layer for memory, init and cold start data. Amazon Managed Service for Prometheus is a managed Prometheus-compatible store for container and Kubernetes metrics (queried with PromQL), usually paired with Amazon Managed Grafana. Pick them when the question mentions Prometheus, Grafana or open-source tooling.

Monitoring AI workloads: Amazon Bedrock publishes invocation metrics (latency, tokens, throttles) to CloudWatch, and model invocation logging can send prompts and responses to CloudWatch Logs or S3. Treat it like any other service: metrics, alarms, logs, CloudTrail for API calls.

CloudWatch alarms

An alarm watches one metric (or a math expression) over a number of periods and sits in one of three states: OK, ALARM, INSUFFICIENT_DATA.

• Period: length of each evaluation (10, 30 or multiples of 60 seconds). Evaluation periods and datapoints to alarm (M out of N) control how many breaches are needed.
• Missing data treatment: missing (default; state is held), notBreaching, breaching, ignore. A server that stops sending data usually should be treated as breaching for a heartbeat metric and notBreaching for a sparse one.
• Actions: SNS topic, Auto Scaling policy, EC2 actions (stop, terminate, reboot, recover), Systems Manager OpsItem or incident, and Lambda only via SNS or EventBridge. Alarm state changes are also published to EventBridge, which can invoke almost anything.
• EC2 auto-recover: alarm on StatusCheckFailed_System with the recover action. The instance keeps its ID, private IP, Elastic IP and EBS volumes. Instance-level failures (StatusCheckFailed_Instance) need reboot.
• Composite alarms combine other alarms with AND, OR, NOT. Use them to cut noise: page only when CPU is high AND latency is high. A composite alarm can take actions; the child alarms can have none.
• Alarm actions need permission: the SNS topic policy must allow cloudwatch.amazonaws.com to publish, and for an encrypted topic the KMS key policy must allow it too (a frequent cause of “alarm fires, no email”).
• Alarms in a cross-account setup: alarms can watch metrics from another account through cross-account observability, or you create the alarm in the account that owns the metric.

Dashboards

• A dashboard is a customizable page of widgets (graphs, numbers, text, alarm status, logs queries). It’s global, not Regional, and a single dashboard can mix metrics from multiple Regions.
• Cross-account, cross-Region dashboards work through CloudWatch cross-account observability: designate a monitoring account, and source accounts link to it with an OAM (Observability Access Manager) sink and link. You can link an entire organization. Metrics, logs, traces and Application Insights data can be shared.
• Dashboards can be shared publicly, with specific email addresses (through SSO), or with an account via a link, without giving AWS console access. Dashboards can also be created from code (CloudFormation AWS::CloudWatch::Dashboard, JSON body).
• Automatic dashboards exist per service in the CloudWatch console, so you don’t have to build a basic one.

Notifications with SNS

• SNS topics fan out to subscribers: email, SMS, HTTP(S), Lambda, SQS, Firehose, mobile push. Standard topics are best-effort ordered; FIFO topics (FIFO SQS subscribers only) preserve order.
• Many services publish to SNS: CloudWatch alarms, S3 event notifications, Auto Scaling lifecycle and scaling notifications, RDS event subscriptions, CloudFormation stack events, AWS Backup events, AWS Config.
• The topic access policy must allow the publishing service principal (and the source ARN condition). This is the first thing to check when notifications never arrive.
• Message filtering with subscription filter policies sends only matching messages to a subscriber. Failed deliveries can go to a dead-letter queue.
• AWS User Notifications is a central place to configure notification channels (email, chat through Amazon Q Developer in chat applications, mobile push) for events from many services, instead of building SNS topics by hand. AWS Chatbot is now part of Amazon Q Developer in chat applications.

CloudWatch Logs

• Hierarchy: log group (retention, encryption, access) → log stream (one source) → events. Retention defaults to never expire; set it to control cost.
• Metric filters turn matching log text into a metric (count ERROR, extract a latency value with a JSON or space-delimited pattern), which you can alarm on. They don’t back-fill; only events after creation count.
• Subscription filters stream matching events in near real time to Lambda, Kinesis Data Streams, Firehose or OpenSearch. A log group allows a limited number of subscription filters (two). Cross-account delivery uses a destination.
• Logs Insights is an interactive query language (fields, filter, stats, sort, parse, limit). It queries multiple log groups, and can be saved or added to dashboards. Live Tail shows events as they arrive.
• Export to S3 (CreateExportTask) is for batch, can take up to 12 hours to be complete and isn’t real time. For continuous delivery use a subscription filter to Firehose.
• Log classes: Standard and Infrequent Access (cheaper, fewer features: no metric filters or subscriptions). Data protection policies detect and mask sensitive data such as card numbers in logs. Contributor Insights finds top talkers in logs (top IPs or users).
• Encrypt a log group with a KMS key by associating it; the key policy must allow the CloudWatch Logs service principal.

CloudTrail

• Records management events (control-plane API calls; on by default for the last 90 days in Event history), data events (S3 object-level, Lambda invoke, DynamoDB item activity — off by default, extra charge) and Insights events (unusual API rate or error rate).
• Create a trail to deliver beyond 90 days to S3 (and optionally CloudWatch Logs). A trail can be organization-wide and multi-Region. Log file integrity validation gives signed digest files that prove logs weren’t altered.
• CloudTrail Lake is a managed event data store you query with SQL, with longer retention, including events from other sources.
• Typical uses: who deleted the resource, who changed the security group. For near-real-time reaction, API calls also appear on EventBridge (management events on the default bus), so a rule can respond to StopLogging or AuthorizeSecurityGroupIngress without waiting for S3 delivery.
• Send a trail to CloudWatch Logs, add a metric filter (for example unauthorized API calls, root login, console sign-in without MFA) and an alarm. This is the classic detective control.

EventBridge

• Event bus receives events. The default bus gets AWS service events; custom buses carry your application events; partner buses carry SaaS events. A rule has an event pattern (or a schedule) and up to five targets (Lambda, SNS, SQS, Step Functions, SSM Automation or Run Command, ECS tasks, API destinations, another bus, and more).
• Route: rules filter by pattern (prefix, numeric, exists, anything-but). Enrich: input transformers reshape the event before delivery, and EventBridge Pipes connects a source (SQS, DynamoDB stream, Kinesis) to a target with optional filtering and enrichment (a Lambda or API call). Deliver: targets with retry policy and dead-letter queue.
• EventBridge Scheduler runs one-off or recurring schedules at scale with time zones, a replacement for cron-style rules.
• Archive and replay store events and replay them later. Cross-account and cross-Region delivery sends events to a bus in another account, whose resource policy must allow it.
• Troubleshooting a rule that doesn’t fire: wrong bus (a custom event sent to the default bus, or the reverse); pattern mismatch (case-sensitive, arrays, wrong detail structure); the rule is disabled; the target lacks permission (the resource policy for Lambda, or the role EventBridge assumes for Step Functions and Run Command); events from a service in a different Region; check the FailedInvocations and TriggeredRules metrics and the target’s DLQ. The sandbox in the console tests a pattern against a sample event.
• API calls reach EventBridge as “AWS API Call via CloudTrail” events only for management events, and the service must be in the same Region as the rule (global services such as IAM appear in us-east-1).

Automated remediation

Newer tooling: AWS DevOps Agent is an autonomous operations agent that investigates incidents across your telemetry and tools and suggests mitigations; Kiro is an AI-assisted development environment that can help write automation and infrastructure code. Both are mentioned in the guide as aids; the exam still tests the fundamentals above. Cost anomalies are caught by AWS Cost Anomaly Detection, and operational issues by AWS Health.

Systems Manager Automation

• A runbook (Automation document, YAML or JSON) has steps such as aws:executeAwsApi, aws:runCommand, aws:invokeLambdaFunction, aws:changeInstanceState, aws:waitForAwsResourceProperty, aws:approve (human approval), aws:branch, aws:executeScript (inline Python or PowerShell).
• Predefined runbooks (AWS-RestartEC2Instance, AWS-StopEC2Instance, AWS-CreateImage, AWS-UpdateLinuxAmi, AWSSupport-TroubleshootSSH, AWS-AttachEBSVolume) cover common jobs. Write custom runbooks when none fits.
• Runs use the caller’s permissions or an assume role (AutomationAssumeRole), which must trust ssm.amazonaws.com. Execution can be rate-controlled over many targets (concurrency and error thresholds) and multi-account, multi-Region.
• Start by console, CLI, SDK, EventBridge, Config remediation, Maintenance Window or State Manager.
• Compare: Run Command runs a command document on instances now; Automation orchestrates multi-step workflows across AWS APIs; State Manager keeps a desired state; Patch Manager patches; Maintenance Windows schedule disruptive tasks; OpsCenter collects OpsItems for investigation; Incident Manager manages on-call and escalation.
• Systems Manager reaches an instance only if the SSM Agent runs, the instance profile has AmazonSSMManagedInstanceCore, and the instance has a network path to the SSM endpoints (internet, NAT or VPC endpoints ssm, ssmmessages, ec2messages). “Instance doesn’t appear as a managed node” is almost always one of these three.

Compute performance

• Start with CloudWatch metrics: CPU utilization, network, and status checks; add memory through the agent. High CPU credit usage on burstable (T) instances: look at CPUCreditBalance; unlimited mode avoids throttling but charges for surplus credits, and a sustained load belongs on a fixed-performance family.
• AWS Compute Optimizer analyzes utilization (needs memory metrics for best results) and recommends right-sizing for EC2, Auto Scaling groups, EBS, Lambda and ECS on Fargate. Trusted Advisor flags low-utilization instances. Cost Explorer and resource tags attribute cost and ownership; tags drive Resource Groups, cost allocation and tag-based automation.
• Placement groups: cluster (one AZ, lowest latency and highest throughput, for HPC; risk is correlated failure), partition (separate racks for big distributed stores like HDFS or Cassandra; up to 7 per AZ), spread (each instance on separate hardware, max 7 per AZ per group, for a few critical instances). Launching into a full cluster group gives an insufficient capacity error; stop and start, or launch all instances at once, with one instance type.
• Enhanced networking (ENA), EFA for HPC and ML, jumbo frames within a VPC (9001 MTU), and instances with instance store for very high local I/O (data is lost on stop).
• Lambda: raise memory (it also scales CPU) to cut duration; use provisioned concurrency for cold starts; watch Throttles, Duration, ConcurrentExecutions.

EBS performance

• Moving gp2 to gp3 is the standard cost and performance win: cheaper per GiB, with IOPS and throughput set separately. Elastic Volumes changes type, size and IOPS without detaching the volume (then extend the file system; wait about six hours between modifications of a volume).
• Metrics: VolumeQueueLength (sustained high means the volume is saturated), VolumeReadOps/WriteOps, BurstBalance (gp2, st1, sc1; zero means throttled), VolumeThroughputPercentage and VolumeConsumedReadWriteOps for provisioned types. An instance can also be the limit: check the instance’s EBS bandwidth and IOPS caps (EBS-optimized).
• Restoring from a snapshot is lazy: blocks load from S3 on first access, so the first reads are slow. Fix with fast snapshot restore (FSR, per snapshot per AZ) or by initializing (reading) the volume.
• Multi-Attach attaches one io1/io2 volume to several Nitro instances in one AZ; the application must handle concurrent writes.

S3 performance and data transfer

• S3 scales to at least 3,500 write and 5,500 read requests per second per prefix, and you can use many prefixes in parallel. No random-prefix trick is needed any more.
• Multipart upload: recommended above 100 MB, required above 5 GB (single PUT max is 5 GB; an object max is 5 TB). Parts are 5 MB to 5 GB, up to 10,000 parts, and upload in parallel and retry individually. Abort incomplete multipart uploads with a lifecycle rule or you pay for orphaned parts.
• Byte-range fetches read parts in parallel or only the header.
• S3 Transfer Acceleration uses CloudFront edge locations to speed long-distance uploads to a bucket (a special bucket.s3-accelerate.amazonaws.com endpoint; bucket names can’t contain dots).
• AWS DataSync is the managed bulk-transfer service for on-premises NFS, SMB, HDFS or other clouds to S3, EFS or FSx, with scheduling, bandwidth limits and integrity checks. Snowball is for offline petabyte moves, Storage Gateway for hybrid access, Transfer Family for SFTP/FTPS into S3 or EFS.
• Lifecycle policies transition objects between classes (Standard → Standard-IA after 30 days minimum → Glacier tiers) and expire them, and clean up old noncurrent versions. Intelligent-Tiering moves objects automatically with no retrieval fee and fits unknown access patterns.
• Storage classes: Standard, Intelligent-Tiering, Standard-IA, One Zone-IA, Glacier Instant Retrieval (milliseconds), Glacier Flexible Retrieval (minutes to hours), Glacier Deep Archive (hours; 180-day minimum). Express One Zone is single-digit-millisecond storage for the hottest data.
• S3 Storage Lens and S3 Inventory give usage visibility; CloudWatch request metrics (opt-in per bucket or prefix) show 4xx and 5xx rates and first-byte latency. 503 Slow Down means too many requests to a prefix: spread, back off and retry.
• S3 Files is a newer way to mount S3 buckets as a file system (NFS-style access to objects), which fits when applications need file semantics on data that has to stay in S3. Compare it with EFS and FSx below.

Shared storage

EFS optimization: performance modes (General Purpose default; Max I/O is legacy for massive parallelism), throughput modes (Elastic automatically scales and suits spiky loads; Provisioned; Bursting, tied to size), storage classes (Standard, Infrequent Access, Archive) with lifecycle management moving files not read for 7 to 365 days and intelligent tiering moving them back on access. Mount with the EFS mount helper (amazon-efs-utils) and TLS; mount targets are per AZ, with a security group allowing TCP 2049.

RDS performance

• Metrics: CPUUtilization, FreeableMemory, DatabaseConnections, ReadIOPS/WriteIOPS, latency, ReplicaLag, FreeStorageSpace. Enhanced Monitoring gives OS-level metrics at down to 1-second granularity (an agent on the host, delivered to CloudWatch Logs).
• Performance Insights shows database load (average active sessions) broken down by wait events, SQL statement, host and user. It identifies the exact query behind CPU or lock waits. Proactive recommendations in the newer RDS performance analysis point to fixes. Slow query, error and general logs can be exported to CloudWatch Logs.
• Fixes: scale instance class up, use read replicas for read-heavy loads, move from magnetic/gp2 to gp3 or Provisioned IOPS, add an ElastiCache layer, tune parameters via a parameter group (static parameters need a reboot).
• RDS Proxy pools and shares connections, which stops connection storms (typically from Lambda) and shortens failover (it keeps connections during a Multi-AZ failover). It uses IAM authentication and Secrets Manager credentials, and must run in the same VPC.
• Storage autoscaling grows storage when free space is low. Aurora separates compute from shared storage (up to 128 TiB), supports up to 15 replicas, and Aurora Serverless v2 scales capacity in fine steps.
• DynamoDB: on-demand or provisioned with auto scaling; ThrottledRequests, hot partitions and DAX caching are the performance topics.