Data security and governance

Authentication

IAM basics for data services

• Users (long-term credentials, avoid), groups (collections of users), roles (temporary credentials assumed by a service, user or another account). Services such as Glue, Lambda, EMR, Redshift and Step Functions each need a role with a trust policy naming the service principal (for example glue.amazonaws.com) and permissions policies for what it may do.
• Managed vs customer-managed policies: AWS managed policies are maintained by AWS and broad (AWSGlueServiceRole); customer managed policies are yours, versioned and tightly scoped; inline policies are embedded in one identity. Use a customer managed policy when the AWS managed one gives too much or too little.
• Role setup per service: Lambda execution role; Glue job role (needs S3, catalog and iam:PassRole for whoever creates the job); EMR has a service role, an EC2 instance profile and optionally an autoscaling role; Redshift has IAM roles associated with the cluster for COPY, UNLOAD and Spectrum; API Gateway uses IAM authorisation, Cognito or Lambda authorizers; CloudFormation uses a service role; the CLI and SDKs use roles through profiles and IAM Identity Center.
• Authentication methods: password-based (database users, Secrets Manager rotation), certificate-based (mutual TLS for MSK, IoT), and role-based (IAM roles, IAM authentication for RDS tokens, Redshift GetClusterCredentials/IAM, MSK IAM auth). Prefer role-based.
• Managed vs unmanaged services: with managed services (Glue, Athena, Redshift Serverless) AWS runs the infrastructure and patches it, and you handle identity, data and configuration. With unmanaged setups (self-installed Kafka or Hadoop on EC2) you also handle patching, OS security and networking. This is shared responsibility, and a frequent “which is lower overhead” cue.

Network controls

• VPC security groups are stateful, allow-only, attached to ENIs. Glue connections and Redshift clusters need security group rules; Glue needs a self-referencing rule (all traffic from the group itself) so its workers can talk to each other. Update groups to allow only needed ports (5439 Redshift, 3306 MySQL, 5432 PostgreSQL).
• Subnets: put databases in private subnets, reach the internet through a NAT gateway only when required.
• VPC endpoints: gateway endpoints (S3, DynamoDB; free, route table entries) and interface endpoints (PrivateLink, ENIs; for Glue, Kinesis, Secrets Manager, KMS, Athena, STS and others). They keep traffic off the public internet and let policies restrict access (aws:sourceVpce).
• S3 Access Points: named endpoints with their own policy (and optionally VPC-only), so each team or application gets a tailored path to a shared bucket instead of one huge bucket policy. Multi-Region access points route to the closest bucket.
• PrivateLink exposes services privately (including to other accounts and VPCs).
• Edge protection: AWS WAF for web requests, Shield for DDoS, CloudFront for delivery, Route 53 for DNS. API Gateway can use WAF and throttling.

Secrets and credentials

Glue connections, Lambda environment variables and EMR bootstrap scripts should fetch from Secrets Manager rather than hardcoding. Rotating a secret requires the consumer to read it at runtime, not copy it into config.

Authorization

• Methods: role-based (RBAC: access follows job role), policy-based (identity and resource policies), tag-based/attribute-based (ABAC: allow when the principal’s tag matches the resource’s tag, scaling without editing policies), and Lake Formation (database-style grants).
• Policy evaluation: an explicit deny always wins; otherwise access needs an allow; for same-account access an identity or resource policy allow is enough, but cross-account access needs both sides to allow. Permissions boundaries and SCPs cap the maximum; they don’t grant.
• Least privilege: grant specific actions on specific ARNs, use conditions (aws:SourceIp, aws:SourceVpce, aws:PrincipalOrgID, s3:prefix, aws:RequestedRegion), and review with IAM Access Analyzer (generates policies from CloudTrail, finds external access).
• Resource policies: S3 bucket policies, KMS key policies, Secrets Manager policies, Glue Data Catalog resource policies, SQS/SNS policies, Lambda permissions. A bucket policy can enforce HTTPS (aws:SecureTransport) and deny unencrypted uploads.
• S3 Block Public Access (account and bucket level) and ACLs disabled (bucket owner enforced) are baseline.

AWS Lake Formation

Centralised, fine-grained governance for the data lake, on top of the Glue Data Catalog and S3.

• Registers S3 locations; Lake Formation vends temporary credentials to engines, so users don’t need direct S3 permission.
• Permissions model: grant SELECT, INSERT, ALTER, DESCRIBE and others on databases, tables, columns, with row filters and cell-level security through data filters. This is the answer to “analysts may see only some columns/rows” without copying data.
• LF-Tags (tag-based access control): tag databases, tables and columns (classification=pii) and grant on tags. Scales better than naming resources (named resource method).
• Hybrid access mode and IAM allowed principals: a table can be governed by both IAM and Lake Formation. In the default Lake Formation setting, IAM-only access to the catalog and S3 still works until you tighten it, so a common trap is Lake Formation grants that appear to do nothing because IAM already allows access (or, the reverse, access denied because S3 location isn’t registered).
• Works with Athena, Redshift Spectrum, EMR, Glue, QuickSight and SageMaker. Each query engine’s role must be permitted by Lake Formation.
• Cross-account sharing uses resource links and AWS RAM (or direct grants) so a consumer account sees shared databases and tables.
• Governed tables have been superseded by open formats like Iceberg.

Database-level permissions in Redshift

• Users, groups and roles inside the database: CREATE ROLE, GRANT SELECT ON schema.table TO ROLE analyst, GRANT ROLE. Assign IAM roles for external access (COPY, Spectrum).
• Fine-grained: column-level grants, row-level security (RLS policies), dynamic data masking (policies that mask values for certain roles) and role-based access control with system roles.
• IAM integration: federated login with IAM Identity Center / SAML and GetClusterCredentials; the Data API uses IAM or Secrets Manager.
• Redshift datashares use their own grants (below).

Encryption and masking

At rest

KMS essentials

• Key types: AWS owned, AWS managed (aws/s3, can’t change policy, can’t share cross-account), customer managed (full control, rotation, policies). Envelope encryption: KMS encrypts a data key that encrypts the data.
• Key policy is the primary control (a key with no policy allowing IAM isn’t reachable through IAM alone). Grants give temporary delegated use to services.
• Rotation: automatic yearly for customer managed keys (configurable period), manual for imported key material; the key ID stays the same.
• Multi-Region keys share material across Regions (for DR and cross-Region copies).
• Cross-account encryption: use a customer managed key, add the other account in the key policy, and let that account’s role also have IAM kms:Decrypt/GenerateDataKey. Snapshots and S3 replication to another account need the destination key access. Data encrypted with the AWS managed key can’t be read by another account.
• Throttling on KMS appears as ThrottlingException for heavy S3 or Glue workloads, which Bucket Keys help avoid.
• CloudHSM when dedicated hardware modules are required.

In transit

• Use TLS/HTTPS everywhere: enforce with aws:SecureTransport in bucket policies, require_ssl parameter in Redshift (require_SSL), TLS between EMR nodes via security configuration, MSK TLS in-transit encryption, RDS rds.force_ssl, Kinesis (HTTPS endpoints), JDBC/ODBC SSL options. Private paths: VPN, Direct Connect (with a VPN on top for encryption, as DX alone is not encrypted), VPC endpoints.

Masking, anonymisation and salting

• Masking hides part of a value (show last 4 digits). Anonymisation irreversibly removes identity. Pseudonymisation replaces identifiers with tokens that can be mapped back under control. Hashing one way; add a salt (random per-value or secret value) so common values (phone numbers, ids) can’t be recovered with rainbow tables or by guessing. Key salting means mixing a secret into a hash or into a partition key (also used to avoid hot keys).
• Tools: DataBrew (mask, hash, substitute, shuffle, encrypt columns), Redshift dynamic data masking, Lake Formation column/cell filtering, Glue ETL custom transforms, the Glue sensitive data detection transform (detect and redact/hash PII in-job), Athena SQL (regexp_replace, to_hex(sha256(...))).
• Apply by compliance need: GDPR/CCPA (right to erasure, minimisation), HIPAA, PCI. Mask in non-production copies.

Audit logging

• CloudTrail records API activity: management events (control plane), data events (S3 object-level, DynamoDB items, Lambda invokes; opt-in, extra cost) and Insights. The event history shows 90 days of management events; create a trail (multi-Region, organization trail) writing to S3 for retention, optionally to CloudWatch Logs, with log file integrity validation (digest files) and KMS encryption.
• CloudTrail Lake: a managed event data store with SQL queries across events (and from other sources via integrations), retention up to years, no S3/Athena setup; right for “query centralised audit logs with minimal setup”.
• CloudWatch Logs for application logs: log groups, retention, KMS, Insights, subscription filters; centralise logs from many accounts into one account using cross-account subscriptions or CloudWatch cross-account observability.
• Service-specific logs: S3 server access logs and CloudTrail data events, VPC Flow Logs, ALB and CloudFront logs, Redshift audit logging (connection, user and user-activity logs to S3 or CloudWatch Logs, enable enable_user_activity_logging for activity), Glue/EMR logs, Athena query history, Lake Formation access in CloudTrail.
• Analysing: Athena (partitioned CloudTrail tables), CloudWatch Logs Insights, OpenSearch dashboards, EMR for massive log volumes (Spark across billions of lines).
• Centralise: an organization trail to a dedicated log-archive account with a locked-down bucket (S3 Object Lock, MFA delete, deny-delete policy).
• Prepare for audit: retain per regulation, keep logs immutable, tag resources, and provide read-only audit roles. AWS Audit Manager and Config give evidence collection.

Privacy and governance

• PII: names, addresses, IDs, card numbers, health data. Amazon Macie scans S3 with managed data identifiers (and custom ones) and produces findings (to EventBridge, Security Hub). Combine Macie to find PII, then Lake Formation (LF-tags pii=true) to restrict, mask or filter columns, and DataBrew or Glue detection to transform it.
• Data sovereignty: data stays in Regions the law allows. Prevent copying out with:
  • SCP denying actions when aws:RequestedRegion is not in the approved list,
  • S3 replication rules limited to approved destinations, no cross-Region backup copy (AWS Backup copy settings, snapshot sharing),
  • Control Tower Region deny guardrail,
  • IAM conditions restricting Regions.
• Redshift data sharing: producers share live schemas, tables and views through datashares to other clusters, workgroups or accounts with no copy; the consumer queries the producer’s current data. Cross-account needs authorisation by the producer admin and association by the consumer. Combine with Lake Formation for governance. AWS Data Exchange is for subscribing to third-party datasets.
• Lake Formation cross-account sharing and Glue Data Catalog sharing give governed access across accounts. Amazon DataZone / SageMaker Catalog provides a business data catalog with publish and subscribe approval workflows and lineage.
• AWS Config: records resource configuration and changes, evaluates rules (S3 encryption on, public access blocked, RDS encrypted), supports remediation (Systems Manager Automation), and conformance packs. Use to detect drift and prove compliance. Config shows what changed and when; CloudTrail shows who called the API.
• Other: Security Hub aggregates findings; GuardDuty detects threats (including S3 data events); Audit Manager; AWS Artifact for compliance reports; tagging policies.
• Backups and replication: ensure DR copies respect sovereignty (see above), and encrypt copies with a key in the destination Region.