Instrumentation strategy

Azure Monitor overview

Azure Monitor collects, analyses and acts on telemetry from Azure, on-premises and other clouds.

Key components:

• Log Analytics workspace: the log store and query engine. One workspace can be shared by many resources. Use resource-context or workspace-context RBAC, plus table-level access and retention per table. Tables can use Analytics (full query), Basic (cheap, limited query, short interactive retention) or Auxiliary plans for high-volume data.
• Diagnostic settings: per-resource setting that sends resource logs and platform metrics to a Log Analytics workspace, a storage account (archive), an event hub (stream to SIEM) or a partner. Activity log (control-plane operations, subscription level) needs a diagnostic setting to be kept beyond 90 days.
• Data collection rules (DCRs) and the Azure Monitor Agent (AMA): AMA replaces the Log Analytics agent (MMA) and the diagnostics extension; a DCR defines what to collect from which machines and where to send it, with transformations to filter and reshape data. Machines are associated with DCRs (data collection rule associations). On non-Azure machines install AMA through Azure Arc.
• Insights: curated monitoring experiences (Application Insights, VM insights, Container insights, Storage insights, Network insights).
• Workbooks, dashboards, Grafana (Azure Managed Grafana) and Power BI for visualisation.
• Alerts, action groups, autoscale for response.

Integrating Azure Monitor with DevOps tools

• Azure Pipelines: the Query Azure Monitor alerts check/gate blocks or fails a deployment if alerts are firing on the target, enabling safe deployment with automatic rollback. Release annotations in Application Insights mark deployments on charts (the Release Annotation task or az monitor app-insights / REST).
• Send pipeline telemetry to Log Analytics: use Azure DevOps auditing streams (to Azure Monitor logs, Event Hubs, Splunk) and the Log Analytics HTTP data collector or Logs Ingestion API from a pipeline step. GitHub audit log streaming goes to Azure Event Hubs, Azure Blob Storage, Splunk and Datadog.
• Action groups invoke Azure Functions, Logic Apps, webhooks, Automation runbooks, ITSM, Event Hub, email/SMS/push/voice. A webhook to Azure Pipelines or GitHub repository_dispatch can start a remediation pipeline.
• Azure Boards / GitHub Issues: an alert can create a work item through a Logic App or webhook, so incidents get tracked. In Application Insights, work item integration creates Azure DevOps work items from a telemetry item (Work Item Integration setting on the resource).
• Teams / Slack / PagerDuty: through action groups (Logic Apps, webhooks) or the Azure Monitor connector.
• Microsoft Sentinel reads Azure DevOps and GitHub audit logs for security analytics.

Application Insights

Application Insights is the application performance management (APM) feature of Azure Monitor. All new resources are workspace-based: the data is stored in a Log Analytics workspace (classic standalone resources were retired).

What it collects

Enabling it

• Autoinstrumentation (codeless): toggle on App Service, Functions, VMs, AKS (some languages), Container Apps. No code changes. Limited customisation.
• Azure Monitor OpenTelemetry Distro: the recommended SDK path for .NET, Java, Node.js and Python (UseAzureMonitor(), and the Java agent). Set the connection string (APPLICATIONINSIGHTS_CONNECTION_STRING), not the old instrumentation key alone.
• Classic Application Insights SDKs are being superseded by OpenTelemetry.
• JavaScript SDK (client-side) for browser telemetry.
• Sampling reduces volume and cost: adaptive (default in the .NET SDK), fixed rate and ingestion sampling. Telemetry stays correlated within a sampled operation.

Features to know

Distributed tracing

A distributed trace follows one request across services. Each unit of work is a span, linked by an operation ID (trace ID) and parent IDs. Application Insights correlates requests, dependencies and exceptions across services using the W3C Trace Context standard (traceparent header), and the Application Map, transaction diagnostics (end-to-end transaction view, showing a Gantt of calls and timings) and Logs (requests, dependencies, exceptions tables joined on operation_Id) let you inspect it.

Requirements for good tracing: every service instrumented with the same OpenTelemetry/Application Insights setup, context propagated across HTTP and messaging, and a consistent cloud role name (cloud_RoleName) so services show as distinct nodes on the map. In KQL, an operation ID joins across tables:

union requests, dependencies, exceptions, traces
| where operation_Id == "<id>"
| project timestamp, itemType, name, duration, success, resultCode
| order by timestamp asc

VM insights, Container insights, Storage and Network monitoring

Also relevant: Network Watcher tools (IP flow verify, next hop, packet capture, connection troubleshoot). Azure Monitor Application Insights for app-level data. Azure Monitor managed Prometheus and Managed Grafana for Kubernetes dashboards.

Decision cues:

• “Memory and process-level data inside a VM and dependencies between servers” -> VM insights.
• “AKS pod restarts, node CPU, container logs” -> Container insights.
• “Throttled blob requests or latency on a storage account” -> Storage insights with diagnostic logs.
• “Traffic between subnets blocked or slow, topology” -> Network insights, Connection Monitor, flow logs.
• “Exceptions, slow requests, dependency failures in the app” -> Application Insights.

Monitoring in GitHub

Insights and charts

• Repository Insights: Pulse (activity summary), Contributors, Community, Traffic (views, clones, referrers, popular content), Commits, Code frequency, Dependency graph, Network, Forks.
• Organisation insights and security overview: security alerts by repo, coverage of GHAS features, and risk.
• Projects insights: create charts (bar, column, line, stacked area) from project data, with filters and grouping by fields or iteration, such as items per status over iterations (burn-up style), and pin them in a project. Enable via the project’s Insights tab, New chart, set X-axis, Y-axis, group by and filter.
• Actions usage metrics and performance metrics (organisation level): jobs, minutes, run durations, failure rate, queue times, per workflow or runner. Billing and usage reports show minutes and storage.
• Copilot metrics for Copilot usage where applicable.

Alerts for GitHub Actions events

• Notifications: each user can choose to be notified on Actions workflow runs failures (settings > Notifications > Actions: failed workflows only or all, email or web).
• In a workflow: add a final job with if: failure() (or always()) that posts to Teams or Slack (webhook, slackapi/slack-github-action) or opens an issue; use needs so it sees upstream results. A workflow_run workflow can react to completion of another workflow (types: [completed], check github.event.workflow_run.conclusion).
• Webhooks and the GitHub app for Teams: subscribe a channel to workflow and deployment events (@github subscribe owner/repo workflows:{event:"push"...}).
• Status checks and badges surface health in the repository.
• Log streaming: audit logs to a SIEM; runner metrics through Azure Monitor on self-hosted runners with AMA.

Alerts for Azure Pipelines

• Notification subscriptions (project settings > Notifications): pre-defined for build completion and failure; add custom subscriptions with filters for build failed, release deployment failed, approval pending, and so on, to email, a team or group.
• Service hooks send build completed, release deployment completed and run state changed events to Teams, Slack, a webhook, Service Bus or an Azure Function.
• Azure Pipelines app for Teams/Slack subscribe a channel with filters (@azure pipelines subscribe <url>, then adjust the subscription filters for failed runs only).
• Pipeline-run status badges on the README, and dashboard widgets (Build history, Deployment status).
• Alert-on-pipeline-metrics: export pipeline data to Log Analytics (via audit streams, REST, or Power BI/Analytics) and create log alerts.

Alerts in Azure Monitor

Alert components:

• Alert rule: scope (resource), condition (signal and logic), severity (Sev 0 critical to Sev 4 verbose), evaluation frequency and window, and action groups.
• Action group: reusable set of notifications (email, SMS, push, voice) and actions (webhook, Logic App, Azure Function, Automation runbook, ITSM, Event Hub, Secure webhook) – shared across many rules.
• Alert processing rules (formerly action rules): suppress alerts during planned maintenance or add action groups to alerts in a scope, without editing every rule.
• Alert states: New, Acknowledged, Closed. Metric alerts are stateful (fire once, resolve automatically). Log alerts can be stateful too.
• Multiple resources: a single metric rule can cover many resources of the same type in a subscription or resource group in one region, and dimensions split by instance.
• Good alerting: alert on symptoms (user-facing latency and errors, SLO burn) more than causes, avoid noise, route by severity, document runbooks.

Analysing metrics

Infrastructure performance indicators

Tools: Metrics explorer (split by dimension, apply filters, use aggregations avg/min/max/sum/count, pin to dashboards), VM insights Performance tab, Workbooks, Advisor recommendations, and autoscale driven by the same metrics.

Application metrics

• Application performance: server response time (use percentiles, p95/p99, not just average), request rate, failed requests, dependency duration and failure rate, exception rate, availability, and Apdex-style satisfaction.
• Usage: unique users, sessions, page views, funnels (where users drop off), retention, user flows, events, cohorts. Use them to confirm whether a release delivered value (a feedback loop for DevOps).
• Golden signals: latency, traffic, errors, saturation.
• Compare before and after a deployment using release annotations, and cross-check with a canary group.
• Log-based metrics (computed from logs) vs pre-aggregated metrics (cheap, fast, sent by the SDK) in Application Insights.

Basic KQL

Kusto Query Language queries Log Analytics and Application Insights. It reads left to right with pipes: each operator transforms the table from the previous step.

String operators: == (case-sensitive equal), =~ (case-insensitive equal), contains, has (whole term, faster), startswith, in, matches regex. Time filter first for performance.

Examples:

// Failed requests per 5 minutes in the last day
requests
| where timestamp > ago(1d) and success == false
| summarize failures = count() by bin(timestamp, 5m)
| render timechart

// Slowest operations, 95th percentile
requests
| where timestamp > ago(6h)
| summarize p95 = percentile(duration, 95), n = count() by name
| top 10 by p95 desc

// Top exceptions
exceptions
| where timestamp > ago(24h)
| summarize count() by type, outerMessage
| order by count_ desc

// VM CPU above 80% (Log Analytics, Azure Monitor Agent)
Perf
| where ObjectName == "Processor" and CounterName == "% Processor Time"
| summarize avg(CounterValue) by Computer, bin(TimeGenerated, 5m)
| where avg_CounterValue > 80

// Heartbeat: machines not reporting
Heartbeat
| summarize LastBeat = max(TimeGenerated) by Computer
| where LastBeat < ago(15m)

// Container insights: restarts
KubePodInventory
| where TimeGenerated > ago(1h)
| summarize restarts = max(ContainerRestartCount) by Name, Namespace
| where restarts > 0

// Azure activity log: who deleted resources
AzureActivity
| where OperationNameValue endswith "DELETE" and ActivityStatusValue == "Success"
| project TimeGenerated, Caller, ResourceGroup, OperationNameValue

Application Insights tables use timestamp and names like requests, dependencies, exceptions, traces, pageViews, customEvents, availabilityResults, customMetrics; the workspace-based versions show up in Log Analytics as AppRequests, AppDependencies, AppExceptions, AppTraces, AppPageViews, and use TimeGenerated. Don’t mix the two naming sets. The classic names work when you query from inside the Application Insights resource’s Logs blade.

Know: summarize needs an aggregate; project after summarize selects its output columns; count() makes count_ when ordering and countif() counts conditionally; sort by is descending by default; has is faster than contains; case-sensitivity of ==.

Site reliability basics that come up

• SLI (what you measure), SLO (the target, such as 99.9% of requests under 300 ms), SLA (contract with consequences) and error budget (1 minus SLO). When the budget is spent, slow releases and prioritise reliability.
• Alert on SLO burn rate rather than a single threshold.
• Blameless post-mortems and tracking MTTR, MTTD feed continuous improvement (ties back to the metrics in the first domain).
• Azure Service Health and Resource Health tell you whether Azure itself has a problem. Service Health alerts are activity log alerts.
• Azure Advisor gives reliability, performance, cost and security recommendations.