Documentation Grafana Cloud Observe and act Connect to data sources Data sources Prometheus Troubleshooting

Troubleshoot Prometheus data source issues

This document provides troubleshooting information for common errors you may encounter when using the Prometheus data source in Grafana.

Connection errors

The following errors occur when Grafana cannot establish or maintain a connection to Prometheus.

Failed to connect to Prometheus

Error message: “There was an error returned querying the Prometheus API”

Cause: Grafana cannot establish a network connection to the Prometheus server.

Solution:

1. Verify that the Prometheus server URL is correct in the data source configuration.
2. Check that Prometheus is running and accessible from the Grafana server.
3. Ensure the URL includes the protocol (http:// or https://).
4. Verify the port is correct (the Prometheus default port is 9090).
5. Ensure there are no firewall rules blocking the connection.
6. If Grafana and Prometheus are running in separate containers, use the container IP address or hostname instead of localhost.
7. For Grafana Cloud, ensure you have configured Private data source connect if your Prometheus instance is not publicly accessible.

Request timed out

Error message: “context deadline exceeded” or “request timeout”

Cause: The connection to Prometheus timed out before receiving a response.

Solution:

1. Check the network latency between Grafana and Prometheus.
2. Verify that Prometheus is not overloaded or experiencing performance issues.
3. Increase the Query timeout setting in the data source configuration under Interval behavior.
4. Check the Grafana server timeout configuration for server-level timeout settings.
5. Reduce the time range or complexity of your query.
6. Check if any network devices (load balancers, proxies) are timing out the connection.

Failed to parse data source URL

Error message: “Failed to parse data source URL”

Cause: The URL entered in the data source configuration is not valid.

Solution:

1. Verify the URL format is correct (for example, http://localhost:9090 or https://prometheus.example.com:9090).
2. Ensure the URL includes the protocol (http:// or https://).
3. Remove any trailing slashes or invalid characters from the URL.

Data doesn’t appear in Metrics Drilldown or Explore

Symptom: You’ve successfully tested the data source connection, but no metric data appears in Explore or Metrics Drilldown.

Cause: The wrong data source is selected, or the data source name doesn’t match expectations.

Solution:

1. Verify you’ve selected the correct data source from the data source drop-down menu.
2. When using remote_write to send metrics to Grafana Cloud, the data source name follows the convention grafanacloud-<stackname>-prom.
3. Check that metrics are being actively scraped by querying up in the Explore view.
4. If using Grafana Cloud, verify the remote_write endpoint URL and credentials are correct.

PDC connectivity errors

Error messages: host unreachable, EOF, network unreachable, connection reset by peer, dial tcp: lookup ... no such host

Symptom: Prometheus queries fail intermittently or consistently when using Private data source connect (PDC) to reach a Prometheus instance behind a private network. The data source test may pass occasionally but queries fail under load.

Cause: PDC tunnels traffic through an SSH connection between Grafana Cloud and your PDC agent. Connectivity failures are most commonly caused by DNS resolution issues, network configuration on the customer side, or the PDC agent’s default connection limits being too low for the query volume.

Solutions:

1. Verify DNS resolution from the PDC agent host. The PDC agent must be able to resolve the Prometheus hostname from the machine it runs on. Run nslookup or dig for the Prometheus URL from the agent host to confirm.

2. Check network connectivity from the agent. Ensure the PDC agent can reach the Prometheus endpoint directly (for example, curl http://prometheus-host:9090/-/healthy from the agent machine).

3. Increase parallel SSH connections. The PDC agent defaults to 1 parallel SSH connection, which can bottleneck under load from multiple alert evaluations or dashboard queries. Increase this by setting the --ssh-connections flag (or PDC_SSH_CONNECTIONS environment variable) to a higher value (for example, 4 or 8):

   sh Copy

   pdc-agent --ssh-connections=4

4. Check firewall rules. Ensure the PDC agent’s outbound SSH connection to Grafana Cloud isn’t being interrupted by firewall rules, NAT gateways, or idle connection timeouts.

5. Verify the PDC agent is running and healthy. Check agent logs for connection errors or restarts. The agent must maintain a persistent connection to Grafana Cloud.

6. Check for idle timeout issues. If the connection drops after periods of inactivity, configure TCP keepalive on the host or add a keepalive setting to the PDC agent configuration.

For general PDC setup and configuration, refer to Private data source connect (PDC) and Configure PDC.

Certificate verification failed

Error message: “x509: certificate signed by unknown authority” or “certificate verify failed”

Cause: Grafana cannot verify the TLS certificate presented by Prometheus.

Solution:

1. If using a self-signed certificate, enable Add self-signed certificate in the TLS settings and add your CA certificate.
2. Verify the certificate chain is complete and valid.
3. Ensure the certificate has not expired.
4. As a temporary workaround for testing, enable Skip TLS verify (not recommended for production).

TLS handshake error

Error message: “TLS: handshake failure” or “connection reset”

Cause: The TLS handshake between Grafana and Prometheus failed.

Solution:

1. Verify that Prometheus is configured to use TLS.
2. Check that the TLS version and cipher suites are compatible.
3. If using client certificates, ensure they are correctly configured in the TLS client authentication section.
4. Verify the server name matches the certificate’s Common Name or Subject Alternative Name.

Authentication errors

The following errors occur when there are issues with authentication credentials or permissions.

Unauthorized (401)

Error message: “401 Unauthorized” or “Authorization failed”

Cause: The authentication credentials are invalid or missing.

Solution:

1. Verify that the username and password are correct if using basic authentication.
2. Check that the authentication method selected matches your Prometheus configuration.
3. If using a reverse proxy with authentication, verify the credentials are correct.
4. For AWS SigV4 authentication, verify the IAM credentials and permissions. Alternatively, consider using the Amazon Managed Service for Prometheus data source for simplified AWS authentication.

OAuth token expiration errors (GCP and Azure)

Error messages: “ACCESS_TOKEN_EXPIRED”, “401 Unauthorized” in alerting but not in Explore

Symptom: Queries in Explore and dashboards work correctly, but alert rule evaluations fail intermittently with 401 errors. This is most common with Google Managed Prometheus (GMP) and Azure-managed Prometheus endpoints using OAuth/OIDC authentication.

Cause: The Grafana alerting backend and the interactive query path (Explore, dashboards) handle credential refreshes differently. The alerting evaluator can use a cached OAuth token beyond its expiry window due to a token staleness check issue in the Prometheus data source. This causes alerting to fail with expired credentials while interactive queries succeed because they trigger a fresh token exchange.

Solutions:

For Google Managed Prometheus (GMP):

1. Use the GCP datasource-syncer pattern: run a sidecar process that refreshes OAuth tokens and updates the Grafana data source credentials on a schedule shorter than the token lifetime (typically every 45 minutes for a 60-minute token). This ensures the stored token is always valid when the alerting backend uses it.
2. Alternatively, use the Google Cloud Monitor data source with its built-in GCP credential management rather than pointing the Prometheus data source at a GMP endpoint.
3. If running on GKE with Workload Identity, ensure the Kubernetes service account token refresh is functioning and the projected token volume is mounted correctly.

For Azure Managed Prometheus:

1. Verify the Azure AD app registration’s client secret hasn’t expired.
2. If using Managed Identity, ensure the Grafana instance’s identity has the Monitoring Data Reader role on the Azure Monitor workspace.
3. Check that Forward OAuth identity is not enabled alongside Azure AD authentication. Both use the same HTTP authorization headers and conflict with each other.

General steps:

1. Check the Grafana server logs for token refresh errors around the time alerts fail.
2. Verify the data source test (Save & test) passes. This confirms current credentials are valid but doesn’t guarantee the alerting backend has a fresh token.
3. If the issue persists, set Alert state if execution error or timeout to Keep Last State to prevent false alarms while investigating. Refer to Configure alert state for execution errors.

LBAC not restricting data on non-Mimir backends

Symptom: You’ve configured Team LBAC rules, but users can still see all metrics regardless of their team assignments.

Cause: Label-Based Access Control (LBAC) for the Prometheus data source only works when the backend is Grafana Cloud Metrics (Mimir) or Grafana Enterprise Metrics (GEM). It doesn’t work with Google Managed Prometheus, self-managed Prometheus, Thanos, or other Prometheus-compatible endpoints. The LBAC enforcement relies on Mimir-specific HTTP headers (X-Scope-OrgID and team-scoped label matchers) that other backends ignore.

Solution:

1. Verify your backend is Grafana Cloud Metrics or GEM. If you’re using a different Prometheus-compatible backend, LBAC isn’t supported.
2. For Google Managed Prometheus or other external endpoints, use data source permissions to control which teams can access the data source entirely, rather than per-label access control.
3. If you need per-label restrictions on a non-Mimir backend, consider proxying through a Mimir instance or using a separate data source per team with different credentials scoped to the appropriate data.

Azure AD or SigV4 authentication options not available

Symptom: The Azure AD or SigV4 authentication options don’t appear in the authentication drop-down when configuring the Prometheus data source.

Cause: These authentication methods require server-side feature flags that aren’t enabled by default, particularly on Grafana Cloud.

Solution:

1. Grafana Cloud: Contact Grafana Support to request the feature flag be enabled for your stack. This isn’t a self-service setting.
2. Self-managed Grafana: Add the required setting to your Grafana configuration file:
   • For Azure AD: set azure_auth_enabled = true under [auth]
   • For SigV4: set sigv4_auth_enabled = true under [auth]
3. For Amazon Managed Prometheus: Consider using the dedicated Amazon Managed Service for Prometheus data source instead, which handles AWS authentication natively without needing the SigV4 flag.

Forbidden (403)

Error message: “403 Forbidden” or “Access denied”

Cause: The authenticated user does not have permission to access the requested resource.

Solution:

1. Verify the user has read access to the Prometheus API.
2. Check Prometheus security settings and access control configuration.
3. If using a reverse proxy, verify the proxy is not blocking the request.
4. For AWS Managed Prometheus, verify the IAM policy grants the required permissions. Alternatively, consider using the Amazon Managed Service for Prometheus data source for simplified AWS authentication.

Query errors

The following errors occur when there are issues with PromQL syntax or query execution.

Query syntax error

Error message: parse error: unexpected character or bad_data: 1:X: parse error

Cause: The PromQL query contains invalid syntax.

Alternative cause: A proxy between Grafana and Prometheus requires authentication. When proxy authentication fails, the proxy redirects the request to an HTML authentication page. Grafana cannot parse the HTML response, which results in a parse error. This appears to be a query issue but is actually a proxy authentication issue.

Solution:

1. Check your query syntax for typos or invalid characters.
2. Verify that metric names and label names are valid identifiers.
3. Ensure string values in label matchers are enclosed in quotes.
4. Use the Prometheus expression browser to test your query directly.
5. Refer to the Prometheus querying documentation for syntax guidance.
6. If you have a proxy between Grafana and Prometheus, verify that proxy authentication is correctly configured. Check your proxy logs for authentication failures or redirects.

Query returns no data for a metric

Symptom: The query returns no data and the visualization is empty.

Cause: The specified metric does not exist in Prometheus, or there is no data for the selected time range.

Solution:

1. Verify the metric name is spelled correctly.
2. Check that the metric is being scraped by Prometheus.
3. Use the Prometheus API to browse available metrics at /api/v1/label/__name__/values.
4. Use the target metadata API to verify which metrics a target exposes.
5. Verify the time range includes data for the metric.

Query timeout limit exceeded

Error message: “query timed out in expression evaluation” or “query processing would load too many samples”

Cause: The query took longer than the configured timeout limit or would return too many samples.

Solution:

1. Reduce the time range of your query.
2. Add more specific label filters to limit the data scanned.
3. Increase the Query timeout setting in the data source configuration.
4. Use aggregation functions like sum(), avg(), or rate() to reduce the number of time series.
5. Increase the query.timeout or query.max-samples settings in Prometheus if you have admin access.

Too many time series

Error message: exceeded maximum resolution of 11,000 points per timeseries or maximum number of series limit exceeded

Cause: The query is returning more time series or data points than the configured limits allow.

Solution:

1. Reduce the time range of your query.
2. Add label filters to limit the number of time series returned.
3. Increase the Min interval or Resolution in the query options to reduce the number of data points.
4. Use aggregation functions to combine time series.
5. Adjust the Series limit setting in the data source configuration under Other settings.

Memory limit exceeded for high-cardinality queries

Error messages: “max-estimated-memory-consumption-per-query limit exceeded”, “query requires too much memory”, “max samples limit reached”

Symptom: Queries against high-cardinality metrics (thousands of unique label combinations) over long time ranges (days or weeks) fail with memory or sample limit errors. Short time ranges may work but expanding the range causes the failure.

Cause: Prometheus and Mimir enforce per-query memory and sample limits to protect the system from resource exhaustion. High-cardinality metrics (for example, metrics with a pod,