Monitoring and logging

CloudWatch metrics

• A metric is a time series in a namespace (AWS/EC2, AWS/Lambda, custom), identified by dimensions (for example InstanceId=i-123).
• EC2 basic monitoring is every 5 minutes; detailed monitoring every minute. Custom metrics (PutMetricData or the agent) can be high-resolution, down to 1 second.
• EC2 has no memory or disk-space metrics by default. Install the CloudWatch agent — usually installed with Systems Manager and configured from a Parameter Store parameter, so one change updates the fleet. The agent also ships OS and application log files to CloudWatch Logs.
• Metric math builds ratios such as error rate = 100 × errors ÷ requests. Percentile statistics (p50, p90, p99): alarm on p99 latency, not the average, to catch the tail.
• Embedded Metric Format (EMF): code writes structured JSON logs and CloudWatch extracts metrics from them — no API calls from Lambda.
• Dashboards can span accounts and Regions.

Alarms

• States: OK, ALARM, INSUFFICIENT_DATA. You choose period, evaluation periods, datapoints to alarm, and how missing data is treated.
• INSUFFICIENT_DATA means no datapoints: the agent isn’t installed or running, the metric name or dimensions are wrong, or cloudwatch:PutMetricData permission is missing. It isn’t a threshold problem.
• Actions: SNS, Auto Scaling policies, EC2 actions (stop, terminate, reboot, recover), Systems Manager OpsItems or incidents.
• Composite alarms combine alarms with AND, OR and NOT so you page only when it matters.
• Anomaly detection learns a band from history, including daily and weekly seasonality, and alarms when the metric leaves it.

CloudWatch Logs

• Log group (per app, with a retention period and optional KMS encryption) → log streams (per instance or container) → log events.

Centralising logs across accounts

1. In the central account, create a log destination backed by Kinesis Data Streams or Firehose, with a destination policy allowing the source accounts.
2. Each source account adds a subscription filter pointing to it (or an account-level subscription policy).
3. Firehose delivers to S3, and optionally OpenSearch for search. Query S3 with Athena.

Cheap long-term retention: a short CloudWatch retention (say 30 days), a subscription filter → Firehose → S3, and an S3 lifecycle rule to Glacier. The CloudWatch Logs Infrequent Access log class is cheaper than Standard but still costs more than Glacier.

Other log sources

Load balancer metrics: HTTPCode_ELB_5XX means the load balancer itself failed; HTTPCode_Target_5XX means your application did.

Application monitoring

• CloudWatch Synthetics canaries run scripted checks (Node or Python, Puppeteer or Selenium) on a schedule against endpoints and UI flows, catching failures before users do — with screenshots, HAR files and alarms. Blueprints: heartbeat, API, broken link checker, visual monitoring, canary recorder, GUI workflow builder (multi-step flows like checkout).
• CloudWatch RUM measures real users in the browser. Application Signals gives service dashboards and SLOs. Container Insights (ECS and EKS, no code) and Lambda Insights add per-container and per-function metrics.
• AWS X-Ray traces requests across services:
  • Trace → segments (per service) → subsegments (downstream calls); a service map shows latency and errors per component.
  • Annotations are indexed and searchable; metadata isn’t.
  • Sampling rules control cost (default: the first request each second, then 5%).
  • Enable it with Lambda active tracing, an API Gateway stage setting, the X-Ray daemon plus SDK on EC2 (daemon listens on UDP 2000), or a daemon or ADOT collector sidecar on ECS and EKS. Needs xray:PutTraceSegments.
  • The trace header X-Amzn-Trace-Id propagates context; wrap AWS SDK calls (DynamoDB, SQS) to record them as subsegments.
• OpenSearch Service for log search and dashboards; Managed Grafana and Managed Service for Prometheus for teams using those tools.

Cross-account observability

CloudWatch cross-account observability links source accounts to a monitoring account (an OAM sink), which then views metrics, logs and traces from all of them without copying data.

Audit and operational insight

• CloudTrail records API calls — who, what, when, from where.
  • Event history shows 90 days of management events for free.
  • A trail delivers to S3, and optionally CloudWatch Logs for metric filters and alarms on API activity.
  • Management events are on by default; data events (S3 object-level, Lambda invoke) must be enabled and cost more. Insights events flag unusual API call rates.
  • An organization trail logs every account to a central bucket, and member accounts can’t disable it.
  • Log file integrity validation writes hourly digest files with hashes, proving logs weren’t altered. CloudTrail Lake queries events with SQL.
• AWS Config records configuration history and evaluates rules (see incident response).
• AWS Health reports AWS-side events affecting you — scheduled maintenance, instance retirement, outages, and AWS_RISK_CREDENTIALS_EXPOSED. Events go to EventBridge (aws.health); the organizational view covers every account.
• Trusted Advisor best-practice checks (cost, security, fault tolerance, performance, service limits), including exposed access keys. Full checks need Business or Enterprise Support; events go to EventBridge in us-east-1.
• DevOps Guru uses machine learning to find operational anomalies and suggest fixes. Compute Optimizer recommends right-sizing.