Troubleshooting and optimization

Logging, monitoring and observability

• Logging records discrete events (a request, an error). Monitoring watches known metrics and alarms against thresholds. Observability is the ability to ask new questions about a system from its outputs, using logs, metrics and traces together, so you can debug failures you did not predict.
• Structured logging writes logs as JSON with consistent fields (level, requestId, userId, orderId, duration) instead of free text. It makes Logs Insights queries, metric filters and correlation simple. Include a correlation ID (the request ID or X-Ray trace ID) in every line. Use log levels (DEBUG, INFO, WARN, ERROR) and set verbosity by environment. Never log secrets or PII.
• Logging strategy: log inputs at boundaries, state changes, errors with stack traces and context, and external call timings. Set retention on log groups (the default is never expire, which costs money).

Amazon CloudWatch

Metrics

• A metric is a time series defined by namespace, name and dimensions. Standard resolution is 1 minute (EC2 basic monitoring is 5 minutes, detailed is 1 minute). High-resolution custom metrics go to 1 second.
• Built-in metrics come free from services: Lambda (Invocations, Errors, Duration, Throttles, ConcurrentExecutions, IteratorAge for streams, DeadLetterErrors), SQS (ApproximateNumberOfMessagesVisible, ApproximateAgeOfOldestMessage), DynamoDB (ConsumedReadCapacityUnits, ThrottledRequests, SystemErrors), API Gateway (Count, 4XXError, 5XXError, Latency, IntegrationLatency, CacheHitCount).
• Memory and disk metrics on EC2 are not built in. Use the CloudWatch agent.
• Custom metrics: PutMetricData (an API call with a cost and latency in your code path), or better Embedded Metric Format (EMF).

Embedded Metric Format (EMF)

EMF lets you write a specially structured JSON log line that CloudWatch automatically turns into custom metrics, with no PutMetricData calls. The log line holds an _aws block (timestamp, CloudWatchMetrics with namespace, dimensions and metric names) plus the values. In Lambda, anything printed to stdout goes to CloudWatch Logs, so EMF is asynchronous and adds no latency or API throttling. Libraries: the EMF client libraries and Powertools for AWS Lambda Metrics. The raw log line is kept, so you can still query high-cardinality detail with Logs Insights. Pick EMF for “custom metrics from Lambda with the least overhead”.

Alarms and dashboards

• An alarm watches one metric (or a metric math expression, or other alarms in a composite alarm) over a period and evaluates a threshold across EvaluationPeriods and DatapointsToAlarm. States: OK, ALARM, INSUFFICIENT_DATA. Missing data treatment decides what happens when data stops.
• Actions: notify SNS, run an Auto Scaling policy, EC2 actions (stop, terminate, reboot, recover), and create a Systems Manager OpsItem. Alarms are the standard way to notify on quota limits (a metric of usage approaching a Service Quotas limit, for example the Lambda ConcurrentExecutions metric against the limit) and on deployment completion (CodeDeploy, CodePipeline, CodeBuild state changes go through EventBridge to SNS, and CodeDeploy and CodeCommit also support notification rules).
• Anomaly detection learns a normal band. Dashboards combine metrics, logs widgets and alarms across Regions and accounts. CloudWatch Application Insights and Container Insights and Lambda Insights give curated views and extra metrics for those workloads. CloudWatch Synthetics canaries test endpoints on a schedule, and RUM measures real users in the browser.

CloudWatch Logs

• Log group (settings and retention) contains log streams (one per source, such as a Lambda environment or instance). Lambda writes /aws/lambda/<function> automatically if its role allows logs:CreateLogGroup, CreateLogStream, PutLogEvents. Missing logs from Lambda usually means missing permissions on the execution role (or the function never ran).
• Metric filters turn matching log patterns (for example the term ERROR) into a metric you can alarm on. Subscription filters stream logs in real time to Lambda, Kinesis or Firehose. Export to S3 is batch (and can take up to hours). Data protection policies mask sensitive data. The CloudWatch agent ships OS and application logs from EC2 and on-premises.
• Logs Insights is the query language for log groups:

fields @timestamp, @message, @requestId
| filter @message like /ERROR/
| stats count() as errors by bin(5m)
| sort errors desc
| limit 20

Common commands: fields, filter, stats (with count(), avg(), pct(), max()), sort, limit, parse (extract fields with a pattern or regex), display. JSON log lines are auto-discovered as fields, which is why structured logging helps. Lambda REPORT lines carry Duration, Billed Duration, Memory Size, Max Memory Used and Init Duration (only on cold starts), so a Logs Insights query over @type = "REPORT" finds over-provisioned memory, slow functions and cold-start rates. Queries can span multiple log groups, and results can be added to a dashboard.

AWS X-Ray

X-Ray gives distributed tracing: you follow one request across API Gateway, Lambda, SQS, DynamoDB and downstream HTTP calls.

• Instrumenting: add the X-Ray SDK (or AWS Distro for OpenTelemetry, now the preferred open standard) and wrap the AWS SDK client (captureAWSClient/patch_all) so downstream calls become subsegments. Create custom subsegments around your own code, and add annotations and metadata.
• Lambda: turn on Active tracing (a function setting) and the role needs xray:PutTraceSegments and xray:PutTelemetryRecords (AWSXRayDaemonWriteAccess managed policy). No daemon to run. If a function is invoked by an upstream service that already sampled the request, the trace continues (Tracing: PassThrough or Active).
• EC2, on-premises, ECS (self-managed): run the X-Ray daemon (UDP port 2000) or the OpenTelemetry collector. In ECS, run it as a sidecar container. Elastic Beanstalk can enable it with an option setting or .ebextensions. API Gateway has a per-stage tracing checkbox.
• Use X-Ray to answer “which service in the chain is slow or failing”. Use CloudWatch for aggregate health. Use CloudTrail for “who called the API”.

Root-cause analysis tools

Debugging approach: read the error code and message, find the log or trace for the request ID, check recent deployments or configuration changes (CloudTrail, CloudFormation events), reproduce locally (SAM local), then fix and add a test. For deployment failures read the service output logs: CloudFormation stack events (the first *_FAILED event is the root cause, later ones are consequences), CodeBuild logs per phase in CloudWatch, CodeDeploy deployment events and instance logs, Elastic Beanstalk eb logs and the events page, and CodePipeline execution details.

HTTP status codes

Common SDK and service exceptions

Retry 5xx and throttling. Do not retry 4xx (except 429) because they fail the same way every time.

Typical symptoms in Lambda

Optimization

Lambda performance and cost

• Memory is the main tuning lever: CPU, network and cost all scale with it. Billing is duration x memory (per ms), so more memory can be cheaper if it cuts duration more than proportionally. Find the minimum that meets the target with AWS Lambda Power Tuning (a Step Functions tool) or Compute Optimizer, and read Max Memory Used from the REPORT line.
• Move initialization (SDK clients, connection setup, config fetch) outside the handler to reuse across invocations. Lazy-load rarely used libraries. Use arm64 for better price-performance.
• Concurrency: reserved concurrency guarantees and limits. Provisioned concurrency removes cold starts for latency-sensitive paths (costs while idle, and can be scheduled with Application Auto Scaling). Protect downstream databases by limiting concurrency or buffering with SQS (a maximum concurrency setting on the SQS event source mapping caps the pollers).
• Batching (batch size and batching window) reduces invocations. Event filtering on the event source mapping skips events you don’t care about. Keep functions single-purpose and avoid recursion (a function that triggers itself through S3 or SQS can loop expensively) and Lambda-calling-Lambda synchronously (you pay for waiting twice).
• Use Lambda extensions and caching for config and secrets. Profile with CodeGuru Profiler and X-Ray.

Caching

• CloudFront caches at the edge. The cache key can include selected headers, cookies and query strings, so the cached object varies on them. Include only what changes the response (every extra header lowers the hit ratio). Use a cache policy (what forms the key and TTLs) and an origin request policy (what is forwarded to the origin without being part of the key). Cache-Control, Expires and Vary headers from the origin influence TTLs, and invalidations clear objects early (versioned file names are cheaper than invalidation). “Cache content based on request headers” maps to CloudFront cache policies, or API Gateway cache keys with headers as part of the key.
• API Gateway cache (REST): TTL default 300 s, cache keys with parameters, per-method overrides, flush on demand.
• DAX for DynamoDB reads. ElastiCache for database results and sessions. Caching strategies (lazy loading, write-through, TTL) are covered in the development notes.
• Application-level caching inside a warm Lambda environment (in memory or /tmp) works but is per environment and can go stale.

SQS and SNS optimization

• Long polling (WaitTimeSeconds up to 20) reduces empty receives and cost. Batching (SendMessageBatch, receive up to 10, DeleteMessageBatch) reduces calls and cost. Tune the visibility timeout so messages are not processed twice while still running, and use DLQs to isolate failures.
• SNS filter policies let each subscriber receive only relevant messages, so subscribers don’t pay to receive and discard. Filter on message attributes (or payload). It is cheaper and simpler than code in every consumer.
• Scale consumers on queue depth (ApproximateNumberOfMessagesVisible or backlog per instance as the metric for Auto Scaling).

DynamoDB and storage optimization

• Use Query not Scan, the right index, projection expressions, strongly consistent reads only when needed, batch operations, on-demand for unpredictable load, auto scaling for steady load, and DAX or caching for hot reads. Throttling from a hot partition is a key design problem, not a capacity problem.
• Compress large payloads, store big objects in S3, use S3 Transfer Acceleration, multipart upload and byte-range reads, Intelligent-Tiering and lifecycle rules for cost.
• Cost and capacity: use Cost Explorer, Budgets and cost allocation tags. Choose the right compute (Lambda for spiky, containers or EC2 for steady high load).

Workload health checks, liveness and readiness

• A health check tells the platform whether an instance or container can serve traffic. ALB/NLB target group health checks (path, port, interval, healthy and unhealthy thresholds, matcher codes) remove failing targets from rotation. EC2 Auto Scaling can use ELB health checks so unhealthy instances are replaced. ECS uses container health checks in the task definition plus the load balancer check. Route 53 health checks drive DNS failover. Beanstalk enhanced health reports colors from instance and request metrics.
• On Kubernetes (EKS): a liveness probe answers “is this container stuck?” and failure restarts it. A readiness probe answers “is it ready for traffic?” and failure removes the pod from Service endpoints without restarting it. A startup probe gives slow starters time. Probes can be HTTP GET, TCP or a command.
• Design the endpoint: a shallow check (process is up) for liveness, a deeper check (can reach the database and required dependencies) for readiness. Do not make liveness depend on downstream services, or one outage restarts everything. Return 200 only when healthy, and keep it fast and unauthenticated for the load balancer.
• A wrong health check path or port makes the load balancer mark healthy targets unhealthy, and the symptom is repeated task or instance replacement or 502/503 responses. A security group that blocks the health check source does the same.

Using AWS AI tools to troubleshoot and optimize

AI assistants (such as Amazon Q Developer, Kiro and Bedrock-backed tools, and the CloudWatch investigations feature) can speed up diagnosis, but you still verify the answer against evidence.

• Analyze errors: paste or point the tool at a stack trace, a CloudWatch Logs excerpt, a failed CodeBuild or CloudFormation event, or an AccessDenied message, and ask for the likely cause and a fix. Good tools also read the surrounding code and recent changes. Give it the request ID, time window, error text and the relevant code, not secrets or customer data.
• Generate troubleshooting suggestions: ranked hypotheses, the next logs or metrics to check, a Logs Insights query, an IAM policy fix, or a patch. Treat them as leads. Confirm with traces, metrics and a reproduction, and test any fix before deploying it.
• Identify optimization opportunities: performance bottleneck detection (slow queries, N+1 calls, blocking calls inside loops), resource usage (over-provisioned memory or instances, idle resources), and code efficiency (better algorithms, batching, caching, connection reuse). Pair with CloudWatch metrics, X-Ray, Lambda Insights, Compute Optimizer and Cost Explorer for the data the suggestions should rest on.
• Limits: AI can hallucinate API names, parameters and limits, and it may not know your environment. Check against documentation, and keep the human in the decision loop for production changes.

Concurrency, pooling and scaling

• Lambda: each concurrent request uses its own environment. Provisioned concurrency pre-warms environments for latency. Reserved concurrency reserves and caps. Scale limits (burst and the account quota) cause throttling. A function scaled out widely can overwhelm a database, so cap concurrency (or SQS maximum concurrency) and use RDS Proxy.
• EC2 and containers: Auto Scaling adds and removes instances or tasks using target tracking (keep CPU or requests per target near a value), step, scheduled or predictive policies. Use a cooldown or warm-up so scaling doesn’t oscillate. Scale on the metric that reflects the bottleneck (queue backlog per worker for queue consumers).
• Thread pool management: size thread or worker pools for the workload (CPU-bound near core count, I/O-bound larger), bound queues, and use async I/O. An unbounded pool or queue causes memory growth and timeouts. Set a timeout on every outbound call.
• Connection pooling: reuse database and HTTP connections instead of opening one per request. In Lambda, create the client outside the handler (one connection per environment), and use RDS Proxy to pool across many environments. Set SDK keep-alive, and size pool maximum below the database connection limit divided by the number of instances. Too many connections shows up as Too many connections errors or slow connection setup.

Detecting bottlenecks

Start from the user-facing latency and follow it down: API Gateway Latency versus IntegrationLatency (the difference is gateway overhead), then Lambda Duration and Init Duration, then X-Ray subsegments to the slow downstream call, then the data store (DynamoDB ThrottledRequests and SuccessfulRequestLatency, RDS CPU and connection counts, ReadIOPS, Performance Insights for slow queries), then the queue (ApproximateAgeOfOldestMessage). Use CloudWatch dashboards and Logs Insights to compare before and after a change, and Lambda Insights, Container Insights for resource use.