Organizational complexity

Network connectivity between VPCs

• Transitive routing is the keyword. Peering can’t do it; TGW can. A shared-services or inspection VPC needs TGW.
• TGW supports appliance mode on a VPC attachment so that traffic in both directions stays on the same Availability Zone appliance (stateful firewalls). Without it, return traffic can take another AZ and break the session.
• Overlapping CIDRs: use PrivateLink, or NAT between them. Peering and TGW can’t route overlapping ranges.
• Centralized egress: workload VPCs send 0.0.0.0/0 to TGW, the egress VPC holds NAT gateways in each AZ. Centralized inspection: route to a VPC with Gateway Load Balancer appliances or AWS Network Firewall. Use TGW route tables to separate “spoke” and “inspection” traffic.
• Multicast is supported on TGW. IPv6 works through dual-stack VPCs; an egress-only internet gateway gives IPv6 outbound only.
• Gateway endpoints (S3 and DynamoDB) are free and added to route tables. Interface endpoints cost per hour and per GB, and use ENIs with private DNS. Gateway endpoints aren’t reachable from on premises or other Regions; for on-premises access to S3 privately use an interface endpoint for S3 (or a proxy). Endpoint policies and bucket policies with aws:SourceVpce lock access to a path.

Hybrid connectivity

Direct Connect pieces:

• Virtual interfaces (VIFs): private (to a VPC through a virtual private gateway or Direct Connect gateway), public (AWS public endpoints, like S3), transit (to a TGW through a Direct Connect gateway).
• Direct Connect gateway is global: one DX connection reaches VPCs and TGWs in many Regions and accounts. It does not allow VPC-to-VPC traffic across it.
• Resilience tiers: single connection (none), two connections in one location (device resilience), two locations (location resilience, the “maximum resilience” model uses separate devices at two locations). Cheapest backup for DX is a VPN over the internet. A second DX is the answer for consistent performance.
• DX link aggregation groups (LAG) bundle up to four same-speed ports. Jumbo frames are supported on private and transit VIFs.
• Lead time for a new DX is weeks. If the requirement is “connectivity within days”, answer VPN first and DX later.
• BGP: prefer longest prefix match, then local preference and AS path. For symmetric routing, advertise more specific prefixes over the preferred link.
• Prefix limits: a Site-to-Site VPN and a Direct Connect private VIF each accept a limited number of prefixes advertised from on premises (about 100), and a Direct Connect gateway attached to a TGW has its own allowed-prefix cap. When a limit is hit, summarize routes.

Hybrid DNS

• Route 53 Resolver is the VPC DNS (the .2 address). Inbound endpoints let on-premises resolvers forward queries to a VPC for private hosted zone names. Outbound endpoints with forwarding rules send queries for on-premises domains to on-premises DNS. Each endpoint is ENIs in at least two AZs.
• Share forwarding rules across accounts with RAM; associate them with each VPC. Centralize endpoints in a shared-services VPC.
• Private hosted zones associate with VPCs; for cross-account use the CLI/API authorization (CreateVPCAssociationAuthorization) or Resolver rules/ Route 53 Profiles, which bundle zones, rules and DNS Firewall rule groups and share them through RAM.
• Route 53 Resolver DNS Firewall blocks or allows queries by domain list. Resolver query logs go to S3, CloudWatch Logs or Firehose.
• Enable enableDnsHostnames and enableDnsSupport for private hosted zones and interface endpoint private DNS.

Traffic monitoring and troubleshooting

• VPC Flow Logs capture accepted and rejected IP traffic metadata at VPC, subnet or ENI level. Deliver to S3 (cheapest, Athena queries), CloudWatch Logs or Firehose. They record connections, not packet contents. Not captured: DNS to the VPC resolver, DHCP, instance metadata, Windows license traffic.
• Traffic Mirroring copies actual packets from ENIs to a monitoring appliance or NLB (deep packet inspection).
• Reachability Analyzer checks configuration-based paths between two resources and names the blocking component. Network Access Analyzer finds unintended access paths against a scope you define. Network Manager and Transit Gateway Network Manager show global topology and route analysis.
• Remember that security groups are stateful and NACLs are stateless, so a NACL must also allow return traffic on ephemeral ports (1024-65535). Rejected flow log records usually point at one of the two.

Selecting Regions and Availability Zones

Choose by latency to users, data residency or compliance, service availability (not every service or instance type is in every Region), price, and the number of AZs the design needs. Cross-AZ data transfer is billed; cross-Region more. Use Local Zones or Wavelength for single-digit-millisecond needs near a city or 5G network, and Outposts for AWS infrastructure on premises. Services like CloudFront, Route 53 and IAM are global or edge-based.

Cross-account access and identity

• Preferred cross-account pattern: IAM role in the target account with a trust policy naming the source account or principal, assumed through STS. Credentials are temporary. For a resource-based policy (S3, SQS, KMS, Lambda, Secrets Manager), the principal can use its own identity without assuming a role, but the identity policy also has to allow the action when the accounts differ.
• External ID in the trust policy protects against the confused deputy problem when a third party assumes a role in your account.
• Permission evaluation: explicit Deny wins; SCPs and permission boundaries and session policies only restrict; they never grant. Cross-account needs both sides to allow.
• IAM Identity Center (successor to AWS SSO) is the answer for workforce access to many accounts: permission sets are rolled out as roles in each account, with an external IdP (Microsoft Entra ID, Okta, Ping) through SAML 2.0 and SCIM provisioning, or the built-in directory or Active Directory (through AWS Managed Microsoft AD or AD Connector). Use it instead of IAM users and per-account SAML providers. Delegate administration to a member account.
• Customer identities (mobile and web app users): Amazon Cognito user pools and identity pools, not Identity Center.
• Federation without Identity Center: SAML 2.0 identity providers in IAM, or OIDC providers (for example GitHub Actions or EKS pods using IRSA or EKS Pod Identity). Never put long-lived access keys in workloads.
• IAM Access Analyzer: external access findings (resources shared outside the zone of trust, with the organization as the zone), unused access findings (unused roles, keys, permissions), policy validation and policy generation from CloudTrail for least privilege, and custom policy checks in CI.
• Resource Access Manager (RAM) shares subnets, TGW, Route 53 Resolver rules, License Manager configs, Aurora clusters and more without copying. Enable sharing with the organization once; recipients in the organization accept automatically.

Encryption and certificates

• KMS: AWS managed keys (rotate yearly, can’t change policy), customer managed keys (key policy under your control, optional rotation, usable cross-account by key policy plus IAM in the caller), multi-Region keys (same key ID and material in several Regions, for global tables, cross-Region copies and client-side encryption), imported key material, custom key stores (CloudHAM or external key store when keys must live outside AWS). Grants allow delegated temporary use.
• Cross-account encrypted snapshots/AMIs need the customer managed key shared with the target account; AWS managed keys can’t be shared.
• Envelope encryption: data keys encrypt data, KMS keys encrypt data keys. The KMS Encrypt call is limited to 4 KB; use data keys beyond that.
• ACM issues public and private (via AWS Private CA) certificates, auto-renews if DNS-validated. Public ACM certs attach to ALB, NLB, CloudFront, API Gateway but can’t be exported to EC2. CloudFront needs the cert in us-east-1. For certificates on EC2 or on-premises use ACM exportable certs or Private CA.
• In transit: TLS at ALB/NLB/CloudFront, end-to-end re-encryption to targets, S3 aws:SecureTransport deny, RDS rds.force_ssl, VPN/MACsec on Direct Connect.
• Secrets: Secrets Manager rotates on a schedule with Lambda; Parameter Store SecureString is cheaper without built-in rotation.

Central security events and auditing

• CloudTrail organization trail: one trail logs all accounts and Regions, writes to a log archive bucket owned by a dedicated account (S3 Object Lock, log file validation, KMS). Member accounts can’t remove an organization trail. CloudTrail Lake gives SQL queries and event data stores.
• GuardDuty, Security Hub, Inspector, Macie, Detective, IAM Access Analyzer: designate a delegated administrator account and enable auto-enable for new accounts. Findings aggregate into the administrator account; Security Hub supports cross-Region aggregation with an aggregator Region and central configuration policies.
• AWS Config with an aggregator (organization-wide) and conformance packs and organization rules; Audit Manager for evidence; Security Lake to normalize security data in OCSF into S3.
• Findings flow through EventBridge to SNS, Lambda or Systems Manager Automation. Cross-account event delivery: send events to the event bus in a central account (resource policy allows the organization).
• Network protection: AWS Firewall Manager applies WAF, Shield Advanced, security group, Network Firewall and DNS Firewall policies to every account in the organization, including new ones.

Reliability and disaster recovery

Start with the numbers: RTO is how long recovery may take, RPO is how much data loss is acceptable.

• AWS Elastic Disaster Recovery (DRS): agent-based, continuous block-level replication of servers (physical, virtual, cloud) into a low-cost staging area in a target Region; recovery in minutes, RPO in seconds. It’s the modern pilot-light answer for servers; point-in-time recovery and non-disruptive drills are built in.
• Data replication: Aurora Global Database (typical lag under a second, managed failover, RTO about a minute); RDS cross-Region read replicas (promote manually; MySQL, MariaDB, PostgreSQL, Oracle); DynamoDB global tables (multi-writer, last writer wins); S3 Cross-Region Replication (versioning required, S3 Replication Time Control gives a 15-minute SLA for 99.99% of objects); EBS snapshot copy; AWS Backup cross-Region and cross-account copy; EFS replication; FSx backups and replication.
• AWS Backup: central policies by tag or resource type, backup plans, vaults, Vault Lock (WORM compliance mode), cross-account copy with organization backup policies, restore testing, audit manager framework. It answers “centrally manage backups across accounts”.
• Automatic recovery: Auto Scaling health replacement, EC2 auto recovery alarms, Multi-AZ RDS (synchronous standby, DNS failover in 60-120 seconds), Aurora replicas (failover in about 30 seconds), ELB health checks, Route 53 health checks with failover routing.
• Scale up vs scale out: scale out (more, smaller instances behind a load balancer) gives availability and elasticity; scale up is for single-threaded, licensing-bound or stateful workloads. Databases scale up the writer and out the readers.
• DR testing: scheduled drills, game days, AWS Fault Injection Service experiments, DRS launches in drill mode, Application Recovery Controller readiness checks and routing controls.

Multi-account design

• AWS Organizations: management account, organizational units (OUs), member accounts. Features: consolidated billing and volume discounts, SCPs, tag policies, backup policies, AI opt-out policies, delegated administrators.
• SCPs set the maximum permissions for accounts in an OU. They don’t apply to the management account, don’t grant, and don’t affect service-linked roles. Use deny lists (default FullAWSAccess with explicit denies) rather than allow lists unless strict. Common SCPs: deny leaving the organization, deny disabling CloudTrail/GuardDuty/Config, restrict Regions with aws:RequestedRegion (exempt global services), deny root user actions, require tags or encryption.
• Resource control policies (RCPs) limit permissions on resources across the organization (for example, deny S3 access from outside the organization), where SCPs limit principals.
• Declarative policies enforce service baseline configuration (such as blocking public AMI sharing or IMDSv2 defaults) at the organization level.
• Control Tower builds a landing zone: management account, Log Archive and Audit accounts, IAM Identity Center, preventive controls (SCPs), detective controls (Config rules) and proactive controls (CloudFormation hooks). Account Factory provisions accounts through Service Catalog; Account Factory for Terraform (AFT) does it through a pipeline; Customizations for Control Tower adds extra resources. Use Control Tower when you want governed, opinionated multi-account setup; use Organizations directly when you need custom structure. Existing accounts are enrolled into Control Tower.
• Typical OU design: Security (log archive, security tooling), Infrastructure (network, shared services), Workloads (Prod, Non-prod/Dev), Sandbox, Suspended, Policy staging. Separate accounts per environment and per workload for blast radius and billing.
• Central logging: CloudTrail, Config, VPC Flow Logs, ALB and S3 access logs to the log archive account; CloudWatch Logs cross-account through subscription filters to a destination (Kinesis Data Streams or Firehose) in the central account, or CloudWatch cross-account observability (monitoring account with Observability Access Manager sinks).
• Central notifications: EventBridge cross-account bus, SNS cross-account topic policies, AWS User Notifications, Health events through Organizational Health view.
• Governance model: SCPs (guardrails), Control Tower controls, Config conformance packs, tag and backup policies, Service Catalog portfolios shared with the organization, CloudFormation StackSets (service-managed, auto-deploy).

Cost visibility and optimization

• Consolidated billing pools usage for tiers, Reserved Instance and Savings Plans sharing across accounts (can be turned off per account).
• Tools: Cost Explorer (history, forecast, RI/SP recommendations), Budgets (cost, usage, RI/SP coverage and utilization alerts and actions), Cost and Usage Report / Data Exports (hourly, resource-level, to S3 for Athena), Cost Anomaly Detection, Trusted Advisor (cost, performance, security, fault tolerance checks), Pricing Calculator (estimates), Compute Optimizer (EC2, EBS, Lambda, ECS-on-Fargate recommendations from CloudWatch metrics), S3 Storage Lens (organization-wide storage metrics and recommendations), Cost Optimization Hub.
• Tagging strategy: define keys with tag policies, activate as cost allocation tags in Billing (they only apply going forward), AWS-generated tags such as aws:createdBy, and enforce with SCPs (deny create without a tag), Config rules or service control. Cost categories group spend into business units without perfect tags. Untagged shared costs go to cost categories split charges.
• Purchasing options: