Network security, operations and troubleshooting

Cloud Armor

Cloud Armor is Google’s WAF and DDoS service. It filters at Google’s edge, before traffic reaches your backends.

Policy types

Cloud Armor has the Standard (pay per use) and Enterprise (Paygo or annual) tiers. Enterprise adds Adaptive Protection auto-deploy, threat intelligence, DDoS response support and cost protection.

Rules

• Priority: lowest number is evaluated first; evaluation stops at the first match. The default rule has priority 2147483647 and allows by default (change it to deny for an allowlist).
• Match: basic (source IP ranges) or advanced with a CEL expression using attributes such as origin.ip, origin.region_code, request.headers, request.path, request.query.
• Actions: allow, deny (403, 404 or 502), throttle, rate_based_ban, redirect (to reCAPTCHA challenge or external URL). Preview mode logs what would match without enforcing, so test new rules in preview first.
• Preconfigured WAF rules from the OWASP Core Rule Set cover SQL injection (sqli), cross-site scripting (xss), local and remote file inclusion (lfi, rfi), remote code execution (rce), scanner detection, protocol attack, PHP, Java, Node.js, session fixation and method enforcement. Rules are used with evaluatePreconfiguredWaf('sqli-v33-stable', {'sensitivity': 1}). Sensitivity levels run 0 to 4: higher catches more but increases false positives, and you can exclude individual rule IDs or request fields to tune.
• Rate limiting: a throttle rule limits requests per key (IP, X-Forwarded-For, region, header, cookie, HTTP path) over an interval, returning 429 above the limit; a rate-based ban blocks the key for a period after the threshold is crossed.
• Adaptive Protection: machine learning builds a baseline of normal layer 7 traffic, detects volumetric application layer DDoS attacks, and suggests a rule (with an expression for the attack signature); Enterprise can auto-deploy the rule. Turn it on per policy.
• Bot management: integrates reCAPTCHA Enterprise. Use redirect actions for challenges and allow rules that verify reCAPTCHA tokens (session or action tokens) to separate humans from automation.
• Google Threat Intelligence: named IP address lists (for example known malicious IPs, Tor exit nodes, public cloud ranges, and search crawlers) usable in rules, for example evaluateThreatIntelligence('iplist-tor-exit-nodes'). Enterprise feature.
• Header and custom response actions, custom error pages and geo-based rules (origin.region_code == 'XX') are also supported.
• Logging: Cloud Armor decisions appear in the load balancer request logs (enforcedSecurityPolicy, previewSecurityPolicy); enable logging on the backend service. Monitor with Cloud Monitoring (allowed, denied, throttled requests).
• Only one security policy of each type can be attached to a backend service, but one policy can attach to many services.
• Cloud Armor works behind a Google external LB; it does not apply to internal LBs (internal LB WAF needs other controls).

VPC firewall rules and Cloud NGFW

Cloud NGFW is the current firewall family: distributed, stateful, and enforced at each VM’s virtual NIC, not at a central choke point.

Policy types and evaluation order

Order of evaluation: organization policy, then folder policy, then VPC firewall rules and global network policy (order configurable, VPC rules first by default), then regional network policy, and finally the implied rules. Within each, the lowest priority number wins; the first matching allow or deny ends evaluation, unless the action is goto_next, which delegates to the next level. The implied rules at the end (priority 65535, not editable): deny all ingress and allow all egress. Hierarchical decisions that deny or allow cannot be overridden lower; goto_next leaves the decision to project-level rules. To see what applies to an instance use the effective firewall rules view (gcloud compute instances network-interfaces get-effective-firewalls).

Rule components

• Direction (ingress or egress), action (allow, deny, goto_next, and apply_security_profile_group for Enterprise layer 7 inspection), priority (0 to 65535 for VPC rules; up to 2147483647 for policy rules), protocols and ports, enforcement (enabled or disabled), logging.
• Match criteria: source or destination IP ranges (IPv4 and IPv6), FQDNs, geolocations, Google threat intelligence lists, and address groups (reusable sets of ranges) in policy rules (Standard and higher). Legacy VPC rules support IP ranges and target selectors only.
• Targets: all instances in the network, network tags, service accounts, or secure tags (policies). The same options act as sources for ingress rules (service account and tags).

Network tags vs service accounts vs secure tags (micro-segmentation)

Metadata-only information, like GCE labels, cannot be used in firewall rules.

Firewall strategy

• Use hierarchical policies at org or folder for company-wide must-haves (allow IAP TCP range 35.235.240.0/20 for SSH, block known bad egress), and network policies for application teams. Use goto_next where teams should decide.
• Layers: edge (Cloud Armor), network (NGFW), host (OS firewall), identity (IAP). Third-party NGFW appliances integrate through NVA routing (below) or Cloud NGFW Enterprise with a partner.
• Allow health check probes and proxy-only subnets explicitly: probe ranges 130.211.0.0/22 and 35.191.0.0/16, plus the proxy-only subnet for Envoy-based LBs. In GKE, allow these ranges to nodes and Pods; for micro-segmentation on GKE nodes, use secure tags or service accounts on the node pool, and Kubernetes network policies (Dataplane V2) for pod-level rules (firewall rules cannot see Pod-to-Pod traffic inside a node).
• IAP TCP forwarding (range 35.235.240.0/20) replaces public SSH and bastions.

Cloud NGFW tiers

Enterprise layer 7 setup: create firewall endpoints (zonal, organization-level resource), security profiles (threat prevention) and a security profile group, a TLS inspection policy (with a Certificate Authority Service CA pool) if decrypting, associate endpoints with the VPC, then write a firewall policy rule with action apply_security_profile_group. Matched traffic is redirected to the endpoint for inspection and returned. Use it to meet requirements to inspect encrypted traffic or detect intrusions without managing appliances.

Migrating from VPC firewall rules to policies

Use the migration tool (gcloud compute firewall-rules migrate, with a source and a target policy) to convert existing VPC rules to a global network firewall policy while keeping behavior; check the result in a policy before enforcing, associate the policy to the VPC, and then remove or disable the old rules. Where the old rules used network tags the migrated policy can use secure tags (requires re-tagging VMs).

Logging and insights

• Firewall rules logging is enabled per rule; it records the connections matched (allowed or denied) including connection, rule and instance details, sampled and metadata optional. Logs go to Cloud Logging. Denied traffic from the implied deny is not logged unless you add an explicit low-priority deny rule with logging enabled.
• Firewall Insights (in Network Intelligence Center, requires firewall rules logging) reports shadowed rules (hidden by a higher priority rule), overly permissive rules, unused rules (no hits over a period) and deny hits, so you can clean up.

Cloud NAT (public NAT)

• A regional, software-defined service (no proxy VMs, not in the data path) that lets VMs and GKE nodes without external IPs initiate outbound connections to the internet. It also provides outbound NAT for connection responses only; inbound-initiated connections are not possible.
• Configured on a Cloud Router in the region. A VM with an external IP on the NIC does not use NAT for that IP.
• NAT IP addresses: automatic (Google allocates and scales) or manual (you reserve static external IPs). Use manual when the destination must allow-list your egress IPs, and a bigger set for scale. Each NAT IP provides about 64,512 ports (1024 to 65535).
• Subnet ranges: choose all primary and secondary ranges, or specific ones (GKE Pod ranges need to be included for private clusters).
• Port allocation:
  • Static port allocation: a fixed number per VM (default 64 minimum ports per VM). Number of VMs a NAT IP can serve = 64,512 / ports per VM.
  • Dynamic port allocation (DPA): each VM starts at the minimum (32 by default) and grows up to a maximum (up to 65,536) as it opens more connections to different destinations. It requires endpoint-independent mapping disabled. Use DPA when a few VMs need many ports and many VMs need few.
  • Endpoint-independent mapping reuses the same NAT IP and port for a source regardless of destination, supporting NAT-sensitive apps (but uses ports less efficiently).
• Ports are consumed per unique destination (IP, port, protocol), so many connections to the same destination are the main cause of exhaustion. Symptoms: dropped packets with reason OUT_OF_RESOURCES, “allocation failed” logs. Fixes: add IPs, raise min ports, enable DPA, lower TCP time-wait or timeouts, or reduce connections (keepalive).
• Timeouts (adjustable): TCP established 1200 s, TCP transitory 30 s, TCP time-wait 120 s, UDP 30 s, ICMP 30 s.
• NAT rules pick a different NAT IP set based on destination (for example a partner needing a specific egress IP).
• Logging: errors only, translations only, or all. Monitoring metrics include allocated ports, port usage, dropped packets, and sent/received bytes.
• Cloud NAT covers the internet; Private NAT (see VPC notes) handles private-to-private translation with NCC.
• Private Google Access traffic to Google APIs does not use Cloud NAT.

Secure Web Proxy (SWP)

• A managed, cloud-native explicit proxy (Envoy based, regional, in a proxy-only subnet) for egress web traffic (HTTP and HTTPS). Workloads set the proxy address as their forward proxy, or you route traffic to it.
• Policies: rules written with an application matcher (CEL: URLs, hostnames, methods) and a session matcher (source identity: secure tags, service accounts, IP ranges), with allow or deny actions and priorities. Deny by default; allow only approved sites.
• TLS inspection: decrypt and inspect traffic for URL-path level rules using a CA pool (Certificate Authority Service) and a policy; clients must trust the CA.
• Use SWP when the requirement is URL or domain-level egress control with identity-aware rules and no appliances to run. Cloud NAT and NGFW (FQDN rules) operate at lower layers. SWP is for web protocols only; other protocols use NGFW or NAT.

Network virtual appliances and Packet Mirroring

In-line NVAs

• Multi-NIC VMs (up to 8 NICs, one per VPC) run NGFW or router images. Enable IP forwarding on the instance. Traffic reaches them through custom static routes with an internal passthrough NLB as next hop (HA) or policy-based routes (to match specific traffic). Each NIC sits in a different VPC (spoke, hub, untrust).
• Health checks plus an LB with symmetric hashing keep flows consistent; protect with MIG autohealing across zones; remember firewall rules still apply around the appliance; and set the MTU consistently.
• Out-of-band Network Security Integration (partner IDS/IPS): uses mirroring deployments, deployment groups and endpoint groups, where a firewall policy rule with a mirror action sends a copy of traffic to the partner’s collector without sitting in path (no latency or failure impact). In-band integration (intercept) inserts the partner’s appliances in path.

Packet Mirroring

• Clones traffic from mirrored sources (instances, subnets or network tags) to a collector destination: an internal passthrough Network LB with the --is-mirroring-collector flag, backed by collector instances (IDS, packet capture, analytics).
• Same region requirement: the collector LB and the mirrored sources must be in the same region; the collector can be in a peered or Shared VPC network when set up in that network.
• Filters limit by CIDR, protocol and direction (ingress, egress or both). Mirroring captures full packet payload including headers, so it uses VM egress bandwidth and may incur cost. Mirrored traffic is not encrypted at the collector.
• Use for intrusion detection, forensic capture and application performance troubleshooting; use VPC Flow Logs for metadata at lower cost.

VPC Service Controls (VPC-SC)

• A perimeter around projects (and VPC networks) restricts which Google APIs and services are accessible from outside and prevents data exfiltration (copy to a bucket in another project, public bucket, or stolen credentials used from the internet).
• A perimeter protects a list of restricted services (for example storage.googleapis.com, bigquery.googleapis.com). Calls crossing the perimeter boundary are denied unless allowed by an access level (Access Context Manager: IP ranges, device policy, identity) or an ingress or egress rule (identity, source/target resources and methods).
• Dry-run mode records violations in audit logs without blocking. Always test in dry-run first.
• VPC accessible services limits which APIs are reachable from inside the perimeter’s VPC networks.
• Private access to Google APIs for protected services uses restricted.googleapis.com (199.36.153.4/30) or a PSC endpoint with the vpc-sc bundle. Configure DNS to resolve *.googleapis.com to that VIP in both the VPC and on-premises (via forwarding), and routes to it.
• Perimeter bridges (legacy) and ingress/egress rules connect perimeters. Logs: Audit logs with a vpcServiceControlsUniqueId for violations, filter by protoPayload.metadata.@type of VPC-SC audit.
• VPC-SC is not a firewall: it works at the API layer, and it does not replace IAM; both must allow the request.

Logging, monitoring and troubleshooting

Logs to know

Metrics to know

Cloud Monitoring gives metrics for: VPN tunnel status, bytes, dropped packets; VLAN attachment and Interconnect link utilization, operational status, optical power; Cloud Router BGP session up, prefixes learned and advertised; load balancer request count, latency, backend latency, 4xx/5xx errors, backend health; Cloud Armor allowed, denied, throttled; Cloud NAT allocated and used ports, dropped packets, new connections. Create alerting policies for tunnel down, BGP down, high NAT port usage and utilization above 80%.

Network Intelligence Center (NIC)

Typical troubleshooting playbooks

• Cannot connect to a VM: Connectivity Test; check effective firewall rules (implied deny ingress), route (is there a route to the destination, correct tags?), OS firewall, IP forwarding, service listening port, health of the LB path, then firewall logs and VPC Flow Logs to see if packets arrive. Use Packet Mirroring for packet-level evidence.
• Intermittent or slow: Performance Dashboard, MTU mismatches, Cloud NAT port exhaustion, VPN throughput caps (about 3 Gbps per tunnel), Interconnect utilization, LB backend latency metrics.
• LB returns 502 or all backends unhealthy: firewall blocks health check or proxy-only range, wrong health check port or path, backend timeouts, capacity at max; check LB logs statusDetails (for example failed_to_connect_to_backend, backend_timeout, client_disconnected_before_any_response).
• Drain traffic for maintenance on an Application LB: set capacity scaler to 0 or use traffic splitting, rely on connection draining, then monitor backend request metrics (see load balancing notes).
• VPN or Interconnect down: tunnel and BGP status, logs, physical, as in the hybrid notes.
• Unexpected blocked API call: VPC-SC violation logs; run the perimeter in dry-run to find the rule.