Azure Resource Metrics Integration

The Azure Monitor feature collects and aggregates logs and metrics from a variety of sources into a common data platform where it can be used for analysis, visualization, and alerting.

The Azure Resource Metrics will periodically retrieve the Azure Monitor metrics using the Azure REST APIs as MetricList. Additional Azure API calls can be used to retrieve information regarding the resources targeted by the user.

The Azure Resource Metrics collects one type of data: metrics.

Metrics are numerical values that describe some aspects of a system at a particular point in time. They are collected at regular intervals and are identified with a timestamp, a name, a value, and one or more defining labels.

The following data streams are available:

monitor - Allows users to retrieve metrics from specified resources. Added filters can apply here as the interval of retrieving these metrics, metric names, aggregation list, namespaces and metric dimensions. The monitor metrics will have a minimum timegrain of 5 minutes, so the period for monitor dataset should be 300s or multiples of 300s.

compute_vm - Collects metrics from the virtual machines, these metrics will have a timegrain every 5 minutes, so the period for compute_vm should be 300s or multiples of 300s.

compute_vm_scaleset - Collects metrics from the virtual machine scalesets, these metrics will have a timegrain every 5 minutes, so the period for compute_vm_scaleset should be 300s or multiples of 300s.

storage_account - Collects metrics from the storage accounts, these metrics will have a timegrain every 5 minutes, so the period for storage_account should be 300s or multiples of 300s.

container_instance - Collects metrics from specified container groups, these metrics will have a timegrain every 5 minutes, so the period for container_instance should be 300s or multiples of 300s.

container_registry - Collects metrics from the container registries, these metrics will have a timegrain every 5 minutes, so the period for container_registry should be 300s or multiples of 300s.

container_service - Collects metrics from the container services, these metrics will have a timegrain every 5 minutes, so the period for container_service should be 300s or multiples of 300s.

database_account - Collects relevant metrics from specified database accounts, these metrics will have a timegrain every 5 minutes, so the period for database_account should be 300s or multiples of 300s.

For each individual data stream, you can check the exported fields in the Metrics reference section.

The Elastic Agent fetches metric data from the Azure Monitor API and sends it to dedicated data streams named azure-monitor.<metricset>-default in Elasticsearch.

                       ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐

                       │  ┌─────────────────┐  │
┌─────────────────┐       │  azure-monitor  │       ┌─────────────────┐
│    Azure API    │◀───┼──│  <<metricset>>  │──┼───▶│  Elasticsearch  │
└─────────────────┘       └─────────────────┘       └─────────────────┘
                       │                       │
                        ─ Elastic Agent ─ ─ ─ ─
		

Elastic Agent needs an App Registration to access Azure on your behalf to collect data using the Azure APIs programmatically.

To use this integration you will need:

• Azure App Registration: You need to set up an Azure App Registration to allow the Agent to access the Azure APIs. See more details in the Setup section.
• Elasticsearch and Kibana: You need Elasticsearch to store and search your data and Kibana to visualize and manage it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, the Native Azure Integration, or self-manage the Elastic Stack on your hardware.

Authentication on the Azure side All the tasks executed against the Azure Monitor REST API use the Azure Resource Manager authentication model. Therefore, all requests must be authenticated with Microsoft Entra. To authenticate the client application, create a Microsoft Entra service principal and retrieve the authentication (JWT) token. For more details, check the following procedures:

• Create an Azure service principal with Azure PowerShell
• Use the portal to create a Microsoft Entra application and service principal that can access resources.

Make sure that the roles assigned to the application contain at least reading permissions to the monitor data. Check Azure built-in roles for more details.

Authentication on the Elastic side Elastic handles authentication by creating or renewing the authentication token. It is recommended to use dedicated credentials for Metricbeat only.

Costs Metric queries are charged based on the number of standard API calls. Check Azure Monitor pricing for more detailsgit.

A generic integration is fully customizable and can support any Azure service. There are no out-of-the-box dashboards for visualizing data, giving users complete control over the process. You must install the integration and customize the configuration before sending metrics to the data stream. You have the maximum flexibility to customize the configuration, custom pipelines, and mappings fully.

To start using the generic metrics integration, enable "Collect Azure Monitor metrics" and set up your custom configuration.

A specialized integration specializes in a specific Azure service and comes with a built-in configuration that provides the most appropriate mapping for each field with one or more out-of-the-box dashboards to visualize data. You cannot edit the built-in configurations. When you install the integration, you can send the metrics to the data stream, and can immediately visualize and search the data. You still have customization options like custom pipelines and mappings, but they are optional for specific needs.

Specialized integrations include the Azure Virtual Machine, Storage Account, Container Registry, and other Container-related metrics integrations.

Elastic Agent must be installed. For more details, check the Elastic Agent installation instructions. You can install only one Elastic Agent per host.

Elastic Agent is required to collect data from Azure Monitor and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.

Elastic Managed integrations allow you to collect data without having to manage Elastic Agent in your cloud. They make manual agent deployment unnecessary, so you can focus on your data instead of the agent that collects it. For more information, refer to Elastic Managed integrations.

Elastic Managed deployments are only supported in Elastic Serverless and Elastic Cloud environments. This functionality is in beta and is subject to change. Beta features are not subject to the support SLA of official GA features.

• Missing Lookback Support: The integration currently lacks a lookback window for metric collection. If the Elastic Agent stops or an Elastic Managed deployment restarts, metrics might be missed. A lookback feature is currently in development to address this.
• Batch API Issue: When Batch API is active (default is inactive), a lack of matching resources in a single integration can cause collection to halt for all integrations.
  • Fixes Scheduled: 8.19.17, 9.3.6, and 9.4.2.
  • Workarounds: Turn off the Batch API setting, or ensure each integration targets at least one Azure resource.

What is lookback? The integration stores the timestamp of the last successful collection. Upon restart, it uses this to backfill any missing data since that point.

To start collecting data with this integration, you need to:

• Register a new Azure app, by adding credentials, and assigning roles.
• Specify the integration settings in Kibana, which will determine how the integration will access the Azure APIs.

To register your app, follow these steps:

Step 1: Create the app registration

1. Sign in to the Azure Portal.
2. Search for and select Microsoft Entra ID.
3. Under Manage, select App registrations > New registration.
4. Enter a display Name for your application (for example, "elastic-agent").
5. Specify who can use the application.
6. Don't enter anything for Redirect URI. This is optional and the agent doesn't use it.
7. Select Register to complete the initial app registration.

Take note of the Application (client) ID, which you will use later when specifying the Client ID in the integration settings.

Step 2: Add credentials

Credentials allow your application to access Azure APIs and authenticate itself, requiring no interaction from a user at runtime.

This integration uses Client Secrets to prove its identity.

1. In the Azure Portal, select the application you created in the previous section.
2. Select Certificates & secrets > Client secrets > New client secret.
3. Add a description (for example, "Elastic Agent client secrets").
4. Select an expiration for the secret or specify a custom lifetime.
5. Select Add.

Take note of the content in the Value column in the Client secrets table, which you will use later when specifying a Client Secret in the integration settings. This secret value is never displayed again after you leave this page. Record the secret's value in a safe place.

Step 3: Assign role

1. In the Azure Portal, search for and select Subscriptions.
2. Select the subscription to assign the application.
3. Select Access control (IAM).
4. Select Add > Add role assignment to open the Add role assignment page.
5. In the Role tab, search and select the role Monitoring Reader.
6. Select the Next button to move to the Members tab.
7. Select Assign access to > User, group, or service principal, and select Select members. This page does not display Microsoft Entra applications in the available options by default.
8. To find your application, search by name (for example, "elastic-agent") and select it from the list.
9. Click the Select button.
10. Then click the Review + assign button.

Take note of the following values, which you will use later when specifying settings.

• Subscription ID: use the content of the "Subscription ID" you selected.
• Tenant ID: use the "Tenant ID" from the Microsoft Entra you use.

Your App Registration is now ready for the Elastic Agent.

Add the Azure Resource Metrics integration in Kibana and specify settings.

If you're new to integrations, you can find step-by-step instructions on how to set up an integration in the Getting started guide.

The settings' main section contains all the options needed to access the Azure APIs and collect the monitoring data. You will now use all the values from App registration including:

Client ID string : The unique identifier of the App Registration (sometimes referred to as Application ID).

Client Secret string : The client secret for authentication.

Subscription ID string : The unique identifier for the Azure subscription. You can provide just one subscription ID. The Agent uses this ID to access Azure APIs.

Tenant ID string : The unique identifier of the Microsoft Entra Tenant ID.

There are the following additional advanced options:

Latency string : Optional. Latency is the time it takes for the Azure service to publish the metric values to Azure Monitor. The integration uses the latency value to compensate for the delay in metric value publishing.

Default value is 0. Typical values are 30s or 1m.

  |                                                        |
  |                                                        |             Now
  |                                                        │              |
  |                                            time grain  │              |
  │                                          │◀──(PT1M)──▶ │              |
  │                                                        │              |
  ├──────────────────────────────────────────┼─────────────┼──────────────|
  │                                                        │              |
  │                       timespan           │             │              |
  |◀───────────────────────(5min)─────────────────────────▶│              |
  │                                          │             │              |
  |                        period                          │              |
  │◀───────────────────────(5min)─────────────────────────▶|              │
  │                                                        │              |
  │                                                        │              |
  |                                                        │   latency    |
  |                                                        | ◀──(1min)──▶ |
  │                                                        │              |
  │                                                        │              |
Start                                                     End             |
  │                                                        │              |
		

Resource Manager Endpoint string : Optional. By default, the integration uses the Azure public environment. To override, users can provide a specific resource manager endpoint to use a different Azure environment.

Examples:

• https://management.chinacloudapi.cn for Azure ChinaCloud
• https://management.microsoftazure.de for Azure GermanCloud
• https://management.azure.com for Azure PublicCloud
• https://management.usgovcloudapi.net for Azure USGovernmentCloud

Microsoft Entra Endpoint string : Optional. By default, the integration uses the associated Microsoft Entra Endpoint. To override, users can provide a specific active directory endpoint to use a different Azure environment.

Examples:

• https://login.chinacloudapi.cn for Azure ChinaCloud
• https://login.microsoftonline.de for Azure GermanCloud
• https://login.microsoftonline.com for Azure PublicCloud
• https://login.microsoftonline.us for Azure USGovernmentCloud

Enable Batch Api boolean : Optional, by default is set to False. Set this to True when facing scalability issues. When configured, the azure batch api will be used to fetch metrics of multiple resources in one api call. Currently supported data streams are monitor, container_registry, container_instance, container_service, compute_vm, compute_vm_scaleset, database_account and storage_account.

monitor This data stream allows users to retrieve metrics from specified resources. Added filters can apply here as the interval of retrieving these metrics, metric names, aggregation list, namespaces and metric dimensions. The monitor metrics will have a minimum timegrain of 5 minutes, so the period for monitor dataset should be 300s or multiples of 300s.

compute_vm This data stream will collect metrics from the virtual machines, these metrics will have a timegrain every 5 minutes, so the period for compute_vm should be 300s or multiples of 300s.