Troubleshooting Kube-Prometheus-Stack: Fixing the Most Common Errors

Almost every kube-prometheus-stack problem you'll hit in practice falls into one of five buckets: Grafana showing nothing, Grafana's database locking up, a component refusing connections, an install stuck on a webhook, or a ServiceMonitor that Prometheus silently ignores. Here's how to diagnose and fix each one.

How to Approach Troubleshooting Here

Before chasing any specific symptom, run kubectl get pods -n monitoring and kubectl describe pod <pod> -n monitoring on anything not Running. Most of the errors below show up clearly in either the pod's events or its logs (kubectl logs <pod> -n monitoring) — the sections below assume you've already narrowed it down to one of these specific symptoms.

Grafana Shows "No Data"

This is almost always one of three things: a datasource misconfiguration, a time range problem, or a genuinely missing metric. Check, in order:

1. In Grafana, go to Connections → Data Sources and confirm the Prometheus URL points to http://<release>-kube-prometheus-stack-prometheus.monitoring:9090 (or your actual service name).
2. Widen the dashboard's time range — a common cause is simply that Prometheus hasn't been running long enough to have data for the selected window.
3. Query the metric directly in Prometheus's own UI (port-forward to 9090) to confirm it's actually being collected before assuming Grafana is broken.

"database is locked" in Grafana

Grafana ships with SQLite by default, which does not handle concurrent writes well. This error typically appears after a pod restart under load, or when running more than one Grafana replica against a shared volume — SQLite was never designed for that.

Fix it by either giving Grafana a dedicated ReadWriteOnce persistent volume (fine for a single replica) or, if you need multiple Grafana replicas for high availability, pointing Grafana at an external MySQL or PostgreSQL database through the grafana.grafana.ini database section in your values.yaml.

Connection Refused Errors

"Connection refused" between components — Prometheus to a scrape target, Alertmanager to Prometheus, Grafana to Prometheus — almost always means the target pod isn't Ready yet, or a NetworkPolicy is blocking the path. Confirm the target service has healthy endpoints with kubectl get endpoints <service> -n monitoring; an empty endpoints list means the underlying pod's readiness probe is failing, not that the network is misconfigured.

Stuck Admission Webhook / Install Hook

A Helm install or upgrade hanging on waiting for completion of hook: job/kube-prometheus-stack-admission-create almost always means the admission webhook Job's pod can't reach the Kubernetes API server — commonly a restrictive NetworkPolicy, an image pull failure, or a Kubernetes API server the pod can't route to.

Diagnose it directly: kubectl logs job/<release>-kube-prometheus-stack-admission-create -n monitoring. If the Job is stuck rather than erroring, check kubectl describe job/<release>-kube-prometheus-stack-admission-create -n monitoring for scheduling or image-pull events.

ServiceMonitor Not Being Scraped

The Prometheus Operator only picks up ServiceMonitors matching the Prometheus custom resource's serviceMonitorSelector and serviceMonitorNamespaceSelector. By default, kube-prometheus-stack's Prometheus CR is often scoped to ServiceMonitors carrying a specific release label — a ServiceMonitor without that label is silently invisible to Prometheus, with no error anywhere.

Check the label selector on your Prometheus custom resource (kubectl get prometheus -n monitoring -o yaml) and make sure your ServiceMonitor's metadata.labels actually match it.

Alerts Firing But Never Arriving in Slack or PagerDuty

When Prometheus shows an alert as firing but nothing shows up in Slack, PagerDuty, or email, the break is almost always somewhere in Alertmanager's routing tree rather than Prometheus itself. Confirm the alert actually reached Alertmanager first — check its /#/alerts UI (port-forward to 9093) before assuming the notification channel is broken.

If the alert is visible in Alertmanager but silent downstream, the most common causes are a route block that doesn't match the alert's labels (falling through to a default receiver nobody's watching), or an incorrect webhook URL in the receiver configuration. Test the receiver in isolation with amtool config routes test before assuming the integration itself is at fault.

Prometheus Pod Repeatedly OOMKilled

Prometheus keeps its entire active time-series index in memory, so memory usage scales with the number of unique label combinations it's tracking — its cardinality — not with raw data volume. A Prometheus pod that keeps getting OOMKilled almost always means either the memory limit in values.yaml is too low for the cluster's actual metric volume, or something upstream is generating unbounded label values, like a label containing a request path, user ID, or timestamp.

Check prometheus_tsdb_head_series in Prometheus's own metrics to see the current series count, then run a topk query grouped by metric name to find the worst offenders before simply raising the memory limit — raising memory papers over unbounded cardinality rather than fixing it.

Cleaning Up a Failed Install

If an install failed partway and you need to start over, a plain helm uninstall will not remove the CRDs it created — they're intentionally left behind by Helm's design. Remove them explicitly with kubectl delete crd -l app=kube-prometheus-stack (or by name) before reinstalling, otherwise a fresh install can conflict with leftover custom resources from the failed attempt.

Frequently Asked Questions

Why does Grafana show "No Data"?

Check the datasource URL, widen the time range, and confirm the metric exists directly in Prometheus before assuming Grafana is broken.

How do I fix "database is locked"?

Add persistent storage for a single Grafana replica, or move to an external MySQL/PostgreSQL database for multiple replicas.

Why isn't my ServiceMonitor being scraped?

Its labels don't match the Prometheus custom resource's serviceMonitorSelector — this is the most common cause by far.

Conclusion

Almost every kube-prometheus-stack incident traces back to one of these five patterns. Once you recognize the symptom, the fix is usually a label, a selector, or a persistence setting — not a deep architectural problem. Keep kubectl describe and kubectl logs as your first move before searching further.