Incident response

The response lifecycle

AWS frames incident response after NIST: prepare, detect and analyse, contain, eradicate, recover, post-incident review. Use it to answer ordering questions.

Response plans and runbooks

• A plan says who does what, how to escalate and communicate; a runbook is the step-by-step procedure for a specific scenario (compromised key, exposed bucket, crypto-mining instance).
• Systems Manager Automation runbooks (SSM documents) turn those steps into executable, auditable workflows: approvals, branching, running commands, snapshotting, tagging. AWS publishes prebuilt ones, for example for isolating an instance or disabling an IAM key.
• Systems Manager OpsCenter collects OpsItems for a central place to triage, link related resources, and run runbooks against them. Findings can create OpsItems automatically.
• Incident Manager (Systems Manager) adds response plans, on-call contacts, escalation, chat channels and post-incident analysis.
• Notebooks: SageMaker AI notebooks (and Jupyter-style notebooks) can hold executable investigation playbooks for analysts who query logs with Athena.
• Step Functions orchestrates multi-step responses with human approval and error handling. Lambda does the single actions.
• AWS Security Incident Response is a managed service that helps triage and respond to events, with access to the AWS customer incident response team.

Preparing the environment

Provision access ahead of time.

• A separate security (forensics) account where responders work, with cross-account roles into workload accounts that are read-only for investigation plus a few contain/snapshot actions. Roles are pre-created and ready; no one creates IAM users during an incident.
• Break-glass access: tightly guarded emergency roles, alarmed on use, with MFA.
• Responders use temporary credentials (Identity Center or STS).

Deploy security tools ahead of time. GuardDuty, Security Hub, Detective (needs GuardDuty and a ~48 hour data build before it has history), Macie, Config, CloudTrail organization trail, Flow Logs. You cannot turn on logging retrospectively and get history.

Minimise the blast radius.

• Multiple accounts by workload and environment (an account is the strongest boundary).
• Least-privilege IAM, permission boundaries, SCPs.
• Network segmentation: separate VPCs/subnets, tight security groups.
• Short-lived credentials, no long-term access keys, secrets rotated.
• Backups in another account/Region with immutability (see data protection).
• Tag resources with owner and data classification so responders know what’s affected.

AWS Shield Advanced for DDoS readiness: protected resources (CloudFront, Route 53, ALB/CLB, NLB, Elastic IP, Global Accelerator), Shield Response Team (SRT) engagement, proactive engagement (SRT contacts you when health checks fail), cost protection against scaling charges, automatic application layer DDoS mitigation (managed WAF rules), and advanced metrics in CloudWatch. SRT access requires a Business Support+ or Enterprise Support plan and an IAM role that lets SRT touch WAF. Associate health-based detection with a Route 53 health check.

Testing and validating the plan

For ordering/choice: FIS breaks things on purpose; Resilience Hub assesses and recommends; Application Recovery Controller (ARC) shifts traffic and controls failover (routing controls, readiness checks, zonal shift/autoshift).

Automatic remediation

The standard pattern: detection source → EventBridge rule → target (Lambda, Step Functions, SSM Automation, SNS).

• Security Hub custom actions and automation rules act on findings (change severity, suppress, add a note) or trigger EventBridge for responses.
• Config rules with remediation actions call an SSM Automation runbook, automatic or manual, with retries (for example, re-enable bucket block public access, remove an open security group rule).
• GuardDuty finding → EventBridge → Lambda to quarantine an instance or deactivate an access key.
• Automated Forensics Orchestrator for Amazon EC2 is an AWS Solutions implementation using Step Functions that, on a GuardDuty finding, snapshots volumes, captures memory, isolates the instance, and runs analysis in an isolated forensic VPC.
• Application Recovery Controller for automated failover as part of recovery.
• Systems Manager Run Command/Automation operate on instances without SSH.

Design remediation to be idempotent, scoped by tags/account, and logged. Add a human approval for destructive actions.

Capturing forensic artifacts

Skill: capture and store logs as forensic artifacts. Evidence must be complete, protected and traceable (chain of custody).

• Store logs in a dedicated account and bucket with S3 Object Lock (compliance mode), versioning, and CloudTrail log file validation.
• EBS snapshots of affected volumes (encrypted; share to the forensics account with a customer managed key that account may use), taken before terminating anything.
• Memory capture on the live instance, because it’s lost on stop. Tools through Systems Manager run on the running host. Don’t stop or terminate first if memory matters.
• AMI of the instance, instance metadata, security group and network config, Flow Logs, DNS logs, application logs.
• Tag the instance with an investigation tag, and keep a record of every action taken.
• Work on copies in an isolated forensic VPC with no internet access, never the original.
• Hash artifacts to prove integrity.

Search and correlate

• CloudTrail Lake and Athena over CloudTrail, Flow Logs, ALB and WAF logs to answer who/what/when.
• CloudWatch Logs Insights for application logs; OpenSearch for large-scale correlation.
• Security Lake to query across sources in one schema.
• Key pivot fields: access key ID, role session name, source IP, user agent, instance ID, eventSource, error codes.

Validate findings and scope the impact

Not every finding is a true compromise. Validate:

1. Read the finding details: resource, actor, action, count, timestamps, Region.
2. Check whether it’s expected (a pen test, a known scanner, a trusted IP). Add suppression rules for confirmed benign patterns.
3. In Detective, open the finding’s entity: see the baseline of behaviour, new geolocations, new API calls, volume of traffic.
4. In CloudTrail, find everything the principal did before and after: created users, new keys, changed policies, data access.
5. Determine the scope: which accounts, Regions, resources and data (Macie to check what sensitive data was in an exposed bucket).

Containment, eradication and recovery

Compromised EC2 instance

1. Isolate by replacing its security groups with one that has no inbound and no outbound rules. Existing tracked connections may continue, so also use a NACL deny to cut them, or stop the traffic at the subnet.
2. Keep it running if you need memory; snapshot volumes.
3. Remove the instance’s role permissions (detach the instance profile, or attach an inline deny with a aws:TokenIssueTime condition to revoke sessions older than now).
4. Investigate from a copy; then terminate and relaunch a clean instance from a known-good AMI or IaC.
5. Protect against re-entry: patch, rotate secrets the instance could read, check for persistence (cron, new users, new IAM entities).

Do not just terminate first (evidence lost) and do not patch in place and carry on.

Compromised IAM credentials

1. Deactivate (not just delete) the access key so you can investigate, and rotate afterwards.
2. For a role: attach a revoke older sessions policy (the console button sets a deny for tokens issued before a time). Existing temporary credentials otherwise stay valid until they expire.
3. Reset the password and MFA, remove the console profile.
4. Review CloudTrail for what the credential did; delete persistence (new users, keys, roles, trust policies, Lambda functions, EventBridge rules, OAuth/federation changes).
5. Restrict with an SCP if the blast area is wide. For root user compromise, use the root recovery procedure, rotate password and MFA, and check all actions.
6. For leaked keys in public code, AWS may attach AWSCompromisedKeyQuarantineV3; still rotate and investigate.

Exposed or tampered S3 bucket

Block public access, review the bucket policy/ACLs, check versioning to restore overwritten objects, use data event logs and Macie to see what was accessed, and check Object Lock/Replication copies. Enable MFA delete and versioning afterwards.

Ransomware or destructive attack

Restore from AWS Backup logically air-gapped vaults or vault-locked backups, S3 versions or replicas in another account, and redeploy from IaC. Verify the backup is from before the compromise (use AWS Backup restore testing and malware scanning of recovery points where available).

Compromised container / EKS / Lambda

Isolate pods with a deny-all network policy, cordon the node, preserve pod and node logs, rotate secrets, and redeploy from a clean image. For Lambda, set reserved concurrency to 0 to stop it, then fix the code and role.

DDoS

Shield Advanced with SRT, WAF rate-based rules and geo rules, CloudFront in front of origins, and scale. Not a forensic case, but plan the response beforehand.

Root cause analysis with Detective

Amazon Detective builds a behaviour graph from CloudTrail, VPC Flow Logs, GuardDuty findings, EKS audit logs and more, kept up to a year. It requires GuardDuty to be on for 48 hours to start. You can:

• Investigate finding groups: related GuardDuty findings clustered into a single event (and Detective’s investigation feature produces an AI-summarised report with indicators of compromise).
• Open entity profiles (IAM user/role, IP, EC2, S3 bucket, EKS pod) and see time-based baselines and geolocation.
• Pivot from a finding to the actors and resources involved, answering “how did they get in, what did they touch”.

Delegated administrator and organization-wide enablement are supported. Detective doesn’t detect threats (GuardDuty does) and doesn’t remediate.

After the incident, record the root cause, the detection gap, and the fix (a new SCP, rule, detection, or runbook step).

Common traps

• Terminating an instance before capturing memory and disks.
• Assuming a security group change kills existing connections (tracked connections persist).
• Deleting a compromised access key before checking what it did (deactivate first).
• Forgetting that temporary credentials outlast the user’s deletion; revoke sessions.
• Sharing an encrypted snapshot without sharing the customer managed key (an AWS managed key snapshot can’t be shared at all).
• Investigating on the production instance instead of a copy in isolation.
• Believing Detective works instantly; it needs GuardDuty history.