Kube-Prometheus-Stack Security: RBAC, Webhooks & Network Policies

A default kube-prometheus-stack install is tuned for getting monitoring running quickly, not for a locked-down security posture. Understanding what RBAC it creates, how its admission webhook actually works, and where to add network policies turns a fast install into one that's actually production-safe.

What RBAC Ships by Default

Prometheus needs to discover and scrape targets anywhere in the cluster, so the chart grants its ServiceAccount a ClusterRole with read access to pods, services, endpoints, and related objects across all namespaces — this breadth is intentional, not an oversight, since service discovery is the whole point. kube-state-metrics similarly needs broad read access to report on the state of Deployments, Jobs, and other objects cluster-wide.

Grafana and Alertmanager, by contrast, don't need cluster-wide API access at all — their RBAC footprint is much smaller, scoped mostly to reading their own ConfigMaps and Secrets within the monitoring namespace.

How the Admission Webhook Works

The Prometheus Operator uses a validating and mutating admission webhook to catch malformed PrometheusRule and other custom resources before they're accepted by the API server. Two short-lived Jobs — kube-prometheus-stack-admission-create and admission-patch — run once during install to generate the webhook's TLS certificate and register it with the API server.

Because these Jobs need to reach the Kubernetes API server directly, an overly restrictive NetworkPolicy applied too early (before the Jobs complete) is the most common cause of an install hanging — see the troubleshooting guide for diagnosing that specific failure.

Adding Network Policies

Kubernetes doesn't restrict pod-to-pod traffic by default, so the monitoring namespace is fully reachable from anywhere in the cluster unless you add policies. A reasonable production baseline:

1. Allow ingress to Prometheus only from Grafana and Alertmanager pods within the monitoring namespace.
2. Allow ingress to Grafana only from your ingress controller's namespace, not the whole cluster.
3. Allow Prometheus broad egress to any pod's scrape port — restricting this defeats service discovery, since Prometheus must be able to reach whatever it discovers.
4. Allow the admission webhook Jobs egress to the API server specifically, so future upgrades don't hit the same hang described above.

Handling Secrets Properly

The chart's default Grafana admin password is meant for a first login, not for production. Set grafana.admin.existingSecret to reference a Secret you create separately (or one managed by an external secrets operator) rather than leaving grafana.adminPassword as plaintext in values.yaml committed to Git. Apply the same discipline to Alertmanager receiver webhook URLs and API tokens — they belong in Secrets referenced by the chart, not inline in a values file.

Scoping ServiceMonitor Selection

By default, Prometheus's serviceMonitorSelector is often left unset or scoped broadly, meaning any ServiceMonitor in the cluster — created by any team with namespace access — can add itself to Prometheus's scrape configuration. In a multi-tenant cluster, scoping this selector deliberately (and using serviceMonitorNamespaceSelector to limit which namespaces are trusted) prevents an unrelated team's ServiceMonitor from silently affecting Prometheus's scrape load or cardinality.

Frequently Asked Questions

What RBAC does kube-prometheus-stack create?

Broad, cluster-wide read access for Prometheus and kube-state-metrics, needed for service discovery — this is intentional, not excessive by accident.

Why does the admission webhook need API server access?

Its setup Jobs register a TLS certificate with the API server during install; blocking that access with a NetworkPolicy causes the install to hang.

Should I add network policies?

Yes for production — scope Prometheus and Grafana ingress narrowly while leaving Prometheus's scrape egress broad, since that's core to how it works.

Conclusion

kube-prometheus-stack's broad default RBAC and open namespace networking are deliberate trade-offs for fast setup, not oversights — but they're trade-offs worth revisiting before the cluster goes to production. Scope network policies, move secrets out of values.yaml, and decide deliberately who else's ServiceMonitors Prometheus should trust.