Implementing observability

Collecting telemetry

Where telemetry comes from

Ops Agent

The Ops Agent is the single agent for Compute Engine VMs (Linux and Windows) and replaces the legacy Logging and Monitoring agents. It combines Fluent Bit (logs) and the OpenTelemetry Collector (metrics).

• Collects syslog and common system metrics by default, and third-party application integrations (nginx, Apache, MySQL, PostgreSQL, Redis, Kafka, JVM and more) when you configure a receiver.
• Configuration in /etc/google-cloud-ops-agent/config.yaml: receivers (where data comes from), processors (parse, modify, exclude), pipelines (wire them), restarted after changes.
• Install per VM, in bulk with VM Manager agent policies (an OS Config policy installing and keeping the agent current), or in images. Many “no memory metric” or “no logs from the VM” scenarios come down to no Ops Agent, or the VM’s service account lacking roles/logging.logWriter and roles/monitoring.metricWriter (or access scopes on old VMs).
• Needs outbound access to Google APIs (Private Google Access works).

OpenTelemetry

OpenTelemetry (OTel) is the vendor-neutral standard for instrumentation: SDKs and auto-instrumentation in the app, and the OpenTelemetry Collector as a pipeline (receive, process, export). Benefits: no lock-in, one instrumentation for logs, metrics and traces, same code on-premises and in any cloud. Google Cloud accepts OTel data through exporters (googlecloud exporter), the Telemetry API (OTLP endpoint), and the Ops Agent’s OTel based metrics. Run the Collector as a sidecar, DaemonSet, or central gateway. Agent frameworks such as the Agent Development Kit and agents running on the Gemini Enterprise Agent Platform emit OTel traces, logs and metrics (model calls, tool calls, token counts, latency) to Cloud Trace, Logging and Monitoring; GKE and Cloud Run workloads write to stdout and send OTel data the same way.

Managed Service for Prometheus

Google Cloud Managed Service for Prometheus is a Prometheus-compatible global store backed by Monarch (the same system as Cloud Monitoring).

• Managed collection on GKE: scraping is configured with PodMonitoring and ClusterPodMonitoring custom resources; no Prometheus server to run. Self-deployed collection: your own Prometheus with a drop-in binary that writes to the service. OTel Collector can also send Prometheus metrics.
• Query with PromQL in Metrics Explorer, dashboards, alerting and Grafana (via the Prometheus UI frontend or data source). Data retention is 24 months, and queries can span projects and clusters (global view).
• Metrics appear under prometheus.googleapis.com/ and are billed per sample ingested, so cardinality (the number of label combinations) drives cost.
• Works for GKE, other Kubernetes, on-premises and other clouds.

Cloud Service Mesh and platform signals

Cloud Service Mesh (managed Istio) emits golden-signal metrics (request count, latency, errors) per service without code changes, access logs and traces, and populates SLO services. Cloud Audit Logs and VPC Flow Logs are platform and network telemetry (below).

Log sources and kinds

• Cloud Audit Logs: Admin Activity (configuration changes; always on, free, cannot disable), Data Access (reads of data and metadata; off by default except BigQuery; can be high volume and costly), System Event (Google-initiated changes; always on) and Policy Denied (denied by security policy).
• VPC Flow Logs: sampled network flows per subnet; configure the sampling rate, aggregation interval and metadata; costs rise with traffic, so lower the sample rate or filter. Firewall Rules Logging is separate and per rule.
• Application and platform logs: write structured JSON to stdout so the agent parses it into jsonPayload; special fields (severity, message, logging.googleapis.com/trace, spanId, labels, httpRequest) populate the log entry.
• Third-party and on-premises logs: Ops Agent or OTel Collector; or push through the Logging API and client libraries.

Optimising log volume and cost

Cloud Logging charges for ingestion (per GiB, after a free monthly allotment per project) and for retention beyond the default. Levers, in order of effect:

1. Exclusion filters on a sink (including _Default): drop matching entries so they are never stored. A sampling exclusion keeps only a percentage (e.g. exclude 90% of severity<WARNING entries). An excluded entry is not stored in that sink’s destination, but it is still counted by log-based metrics, because those are evaluated when the entry is received, before exclusion filters apply.
2. Source-side filtering and sampling in the Ops Agent or OTel Collector (do not generate or send debug logs); lower application log level in production.
3. Route high-volume, low-value logs (VPC flow logs, data access) to Cloud Storage (cheap) instead of a Logging bucket, or turn on only what you need.
4. Shorter retention on buckets for data you don’t need online.
5. Avoid duplicates (the same log in multiple projects or sinks).
6. Watch the Logs Volume and logging.googleapis.com/billing/bytes_ingested metrics, and alert on spikes (a runaway debug log).

Synthetic monitoring and uptime checks

Use them to detect outages when there’s no traffic (black-box monitoring), to measure from the user’s side, and as an availability SLI source. White-box monitoring (internal metrics) tells you why; black-box tells you whether users are affected.

Custom and log-based metrics

• Custom metrics: your code or the Ops Agent writes time series under custom.googleapis.com/, workload.googleapis.com/ (OTel and agent), external.googleapis.com/ or prometheus.googleapis.com/. Choose a metric kind (gauge, delta, cumulative), value type, and a few labels. High-cardinality labels (user IDs, request IDs) inflate cost and slow queries; keep label values bounded.
• Log-based metrics turn logs into metrics without code changes:
  • Counter metrics count matching entries (errors per service); distribution metrics record a numeric value extracted from the log (latency).
  • System-defined (predefined) vs user-defined. Labels come from field extraction (regex or fields).
  • They count entries from creation on (no backfill), are scoped to a project (or a bucket), and can then feed charts and alerts. Use them when the signal exists only in logs (a specific error string, a business event).
  • Best practice: prefer a real metric when you can instrument; use log-based alerts (below) for rare events.

Managing and analyzing logs

Logs Explorer and the Logging query language

Query examples:

resource.type="k8s_container" resource.labels.namespace_name="shop"
severity>=ERROR
jsonPayload.message:"timeout"
timestamp>="2026-11-09T10:00:00Z"

• Operators: =, !=, >, <, >=, <=, : (has / substring), =~ and !~ (regex). Boolean AND, OR, NOT; space means AND. Field paths such as jsonPayload.x, httpRequest.status, labels.k. Use logName, resource.type and a time range to make queries fast.
• Correlate by trace: trace="projects/PROJECT/traces/TRACE_ID". Logs Explorer has histogram, field filters, log fields, saved queries and links to the trace and to related resources. Log Analytics adds a SQL interface over a log bucket upgraded for analytics, and an optional linked BigQuery dataset so you can join logs with other data without exporting.
• Gemini Cloud Assist and AI-assisted Logs Explorer can turn a question in plain language into a query, summarise log clusters and explain errors, and investigations correlate logs, metrics and changes. Always check the generated query and treat the explanation as a hypothesis.

Log router, buckets and retention

Every entry enters the Log Router, which evaluates sinks. Each sink has a filter, optional exclusion filters and a destination.

Destinations for sinks:

Key facts:

• Each sink has a writer identity (service account); grant it the right role on the destination (bigquery.dataEditor, storage.objectCreator, pubsub.publisher, logging.bucketWriter) or nothing arrives. Failure to export is a classic troubleshooting item.
• Aggregated sinks at a folder or organization with --include-children collect logs from many projects. Use intercepting aggregated sinks to block a pattern organization-wide.
• Sinks route only logs received after creation; old logs don’t backfill (you can copy logs from a bucket to Cloud Storage manually).
• Export for retention to meet compliance (e.g. keep audit logs 7 years in a locked Cloud Storage bucket with bucket lock / retention policy), for analytics (BigQuery) and for third parties (Pub/Sub).
• Log buckets can be locked (retention can’t be shortened).

Handling sensitive data

• Don’t log PII or PHI in the first place: filter at source, mask in the application, and avoid secrets in logs.
• Log processors and transformations: use Ops Agent or OTel Collector processors to modify or drop fields before ingestion; Sensitive Data Protection (formerly Cloud DLP) can inspect logs, find infoTypes (names, card numbers, national IDs) and de-identify (redact, mask, tokenize) them in a pipeline (Pub/Sub → Dataflow with the DLP API → destination) or as a scheduled scan of stored log tables. Sensitive Data Protection also profiles BigQuery data and can attach findings to logs.
• Limit access: log views restrict who sees what; field-level access control; IAM roles logging.viewer (non-sensitive), logging.privateLogViewer (Data Access logs), logging.admin. Use CMEK on buckets and separate projects for sensitive logs.
• Use exclusion filters to drop entries that shouldn’t be stored.

Managing metrics, dashboards and alerts

Metrics Explorer and PromQL

Metrics Explorer charts any metric with filters, group-by, aggregation (aligners, reducers, alignment period) and PromQL or the older MQL editors (PromQL is the recommended and supported query language; MQL is legacy). Key ideas: monitored resource (gce_instance, k8s_container, cloud_run_revision), metric type, labels, kind (gauge, delta, cumulative) and aligner (rate for cumulative counters, percentile for distributions). Charts of percentiles from distribution metrics show tails.

Data retention in Cloud Monitoring: metrics are kept up to 24 months, with fine-grained points for a shorter period. A metrics scope gathers many projects.

Dashboards

• Predefined dashboards for services (GKE, Cloud Run, Cloud SQL, VMs, load balancing, Prometheus) and custom dashboards of charts, scorecards, text, logs panels, alert and SLO widgets, and incident lists.
• Share with IAM (monitoring.dashboardEditor/viewer), export as JSON, manage as Terraform or via the Dashboards API so they are version-controlled and consistent across environments. Use filters and template variables (dashboard-level filters by project, cluster, namespace) to reuse a dashboard across services.
• Playbooks: link runbooks and playbooks into alert documentation and dashboards (text widgets, links) so the person at 3 a.m. sees what to do. Put the four golden signals (latency, traffic, errors, saturation) and SLO burn on the top of service dashboards.
• Lay out for the user journey, not for every metric; fewer, better charts.

Alerting policies

An alerting policy = conditions + notification channels + documentation (+ severity, labels, snooze).

Design rules:

• Alert on symptoms (user-visible: error ratio, latency, SLO burn), not causes (CPU, restarts). Page only when a human must act now; everything else is a ticket or a dashboard.
• Use a duration and sensible alignment period; ratio conditions (errors / requests) rather than raw counts; group by to avoid one alert per instance when per-service is enough.
• Documentation field (Markdown with variables such as ${resource.label.project_id}) should say what it means, impact, first checks and a runbook link.
• Incidents auto-close when the condition clears (configurable, default 7 days max for no data), and can be acknowledged from the console or app.
• Snooze stops notifications temporarily during planned maintenance (it doesn’t stop evaluation; incidents aren’t created).
• Cost control: alerting is charged per condition and per time series evaluated, and notification channels have no charge. Avoid conditions that match thousands of high-cardinality series; aggregate in the query; remove unused policies; alert on SLOs instead of dozens of low-level alerts.

Notification channels and third-party tools

Email, SMS, Slack, Google Chat, mobile app, Pub/Sub, webhooks (including basic auth and OAuth), PagerDuty, and other integrations (ServiceNow, Jira, Discord, Rootly or similar incident tools via webhook or Pub/Sub). Prefer a webhook or Pub/Sub for tools without a native channel. Create channels in the scoping project; the channel is a resource with its own IAM. For on-call, use an escalation tool (PagerDuty) so unacknowledged pages escalate, rather than email alone. Cloud Run functions subscribed to a Pub/Sub channel can reshape the payload (add context, route by severity, auto-remediate).

AI-assisted metrics interpretation

Gemini Cloud Assist explains what a chart or alert means, suggests related metrics and probable causes, and helps write PromQL. Use it to speed triage and for onboarding; it doesn’t replace validating against data and runbooks.

Capturing and analyzing distributed traces

A trace is one request’s journey, made of spans (a unit of work with start, duration, attributes, parent). Cloud Trace stores and displays them.

• Instrumentation: OpenTelemetry SDKs and auto-instrumentation (Java, Python, Go, Node.js…) with the Google Cloud trace exporter, or the Telemetry API (OTLP). Context propagation uses the W3C traceparent header (and X-Cloud-Trace-Context). All services in the path must propagate it or the trace breaks into pieces. Cloud Run, App Engine and Cloud Service Mesh can create spans for incoming requests and the mesh adds them without code changes.
• Sampling: capturing every request is costly; set a sampling rate (probabilistic or rate-limited; head-based or tail-based in the Collector so errors and slow requests are kept). Lower it for cost, raise it temporarily when investigating.
• Trace Explorer: filter by service, latency, status, attributes; see a waterfall (spans in time order). Read it for where the time goes: a long span with no children (slow code or one slow dependency), many serial small spans (N+1 calls, missing batching), gaps between spans (queueing, cold start, network), parallelisable sequential calls, retries.
• Latency distribution and analysis reports compare trace populations (today vs last week).
• Correlate with logs: write logging.googleapis.com/trace (projects/PROJECT/traces/TRACE_ID) and spanId in structured logs so Logs Explorer shows “logs for this trace” and the trace shows its logs. Cloud Run and GKE do this automatically when the app writes the standard fields or uses OTel log correlation. Exemplars on distribution metrics link a slow data point to a trace.
• Cloud Profiler shows CPU, heap, contention in code (continuous, low overhead, flame graphs), Error Reporting groups exceptions. Both complement traces: the trace tells you which span, the profiler which function.
• AI-assisted trace analysis (Gemini Cloud Assist) summarises a trace, spots anomalous spans and explains root causes.
• Traces are retained for 30 days.