Azure Monitor is Microsoft's cloud monitoring service for gathering, visualizing, and analyzing telemetry data from applications, infrastructure, and networks. The company recently added a data collection capability in preview with the edge pipeline, which enables the collection and routing of telemetry data before it's sent to the cloud.
The Azure Monitor edge pipeline is like an Extract, Transform, Load (ETL) process and offers improved data collection methods. According to the company, it simplifies data collection from multiple sources by utilizing a unified ingestion pipeline and a standardized configuration approach, which is more efficient and scalable. This is particularly beneficial for Azure cloud-based monitoring.
The pipeline is a containerized solution deployed on an Arc-enabled Kubernetes cluster. It utilizes OpenTelemetry Collector and a pipeline configuration file to manage incoming client data flows. Subsequently, it forwards them to the cloud while using a local cache when required.
The Open Telemetry support for Azure Monitor is based on the company's acknowledgment, as Matthew McCleary, senior product manager, Azure Monitor at Microsoft, wrote in an earlier Azure Monitor blog post:
As OpenTelemetry gained momentum and became more mature, customers asked for OpenTelemetry compatibility in downstream agents and service endpoints. As a result, we have significantly increased OpenTelemetry-related investments across the Azure Monitor platform to expand our support for OpenTelemetry on Azure.
The pipeline configuration file, a vital component of the edge pipeline, establishes data flow and cache settings. The Data Collection Rule (DCR), another crucial element, defines the data schema, transformation criteria, and destination for cloud pipeline transmission. Each data flow definition in the pipeline configuration specifies the DCR and its associated stream for processing within the cloud pipeline, ensuring a seamless and efficient operation.
Overview of Azure Monitor Pipeline (Source: Microsoft Learn)
To leverage the pipeline, users must deploy the Azure Monitor pipeline on a single Arc-enabled Kubernetes cluster in their environment. This enables resource telemetry emission for ingestion into Azure Monitor for observability. Subsequently, they access the Extension section once the prerequisites are met to create an instance of the Azure Monitor pipeline extension or use the Azure portal's search bar to create a pipeline instance directly. Finally, they can use the Dataflow tab to manage dataflows for the pipeline instance, configuring resource telemetry emission accordingly.
Create an Azure Monitor Pipeline (Source: Microsoft Learn)
Xema Pathak, a senior product manager at Microsoft, writes:
With the Azure Monitor pipeline at edge, organizations can tap into the same highly scalable platform with a standardized configuration and reliability. Whether dealing with petabytes of data or seeking consistent observability experience across Azure, edge, and multi-cloud, this solution empowers organizations to reliably collect telemetry and drive operational excellence.
Lastly, using the Azure Monitor pipeline to transmit data to Azure Monitor has additional costs; charges only apply for data ingestion based on the current pricing.