Troubleshooting Kube-Prometheus-Stack

Most kube-prometheus-stack problems fall into a handful of recurring categories: a ServiceMonitor that isn't being picked up, Grafana showing "No Data" because of a datasource or label mismatch, pods stuck Pending due to unbound persistent volume claims, or a Helm upgrade failing because of a CRD version conflict.

This category is reserved for practical, symptom-first troubleshooting guidance — diagnosing why a target shows as down in Prometheus, why alerts aren't reaching Alertmanager, and how to safely recover from a failed chart upgrade.

The guides here are organized around what you're actually seeing on screen rather than the underlying component, since that's usually how the problem presents itself first: a blank Grafana panel, a stuck Helm install, or an alert that never seems to fire. Start with the specific symptom you're hitting rather than reading top to bottom — each guide is written to stand on its own.

Frequently Asked Questions

What's the first command to run when something in kube-prometheus-stack isn't working?

kubectl get pods -n monitoring, followed by kubectl describe pod <pod> -n monitoring on anything not Running — most issues show up clearly in pod events or logs before you need to dig further.

Does a failed Helm install leave anything behind that needs manual cleanup?

Yes — CRDs are not removed by helm uninstall by Helm's design, so a failed or reset install may need kubectl delete crd run manually before reinstalling cleanly.