Persistent Storage & Metric Retention in Kube-Prometheus-Stack

A default kube-prometheus-stack install runs entirely on ephemeral storage. It works fine for a demo, and it silently loses everything the moment a Prometheus or Grafana pod restarts. Persistence is one of the first things to fix before this stack monitors anything real.

Why Persistence Matters

Kubernetes pods are disposable by design — nodes get replaced, pods get rescheduled, deployments roll out new versions. Without a PersistentVolumeClaim behind it, Prometheus's local time-series database and Grafana's dashboard/user database both live on the pod's ephemeral filesystem, which is wiped clean every time the pod restarts for any reason, planned or not.

Configuring Prometheus Storage

Prometheus persistence is configured under prometheus.prometheusSpec.storageSpec:

prometheus:
  prometheusSpec:
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: "standard"
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi

This creates a dedicated PVC per Prometheus replica through the underlying StatefulSet — a proper values.yaml configuration, not an afterthought bolted on after install.

Grafana Persistence

Grafana's own database — dashboards created through the UI, users, and API keys — needs the same treatment, configured separately under grafana.persistence:

grafana:
  persistence:
    enabled: true
    size: 10Gi

Dashboards provisioned through ConfigMaps survive restarts regardless, since they're re-applied on every pod start — persistence here mainly protects manually created dashboards and Grafana's own settings.

Retention Size vs. Retention Time

Prometheus supports two independent retention limits, and it enforces whichever is hit first:

• retention — a time window, like 15d or 90d. Data older than this is deleted regardless of disk space remaining.
• retentionSize — a disk-space cap, like 45GB. Data is deleted oldest-first once this limit is reached, regardless of how recent it is.

In practice, set retentionSize to roughly 80-90% of the actual PVC size. Without it, an unexpected spike in metric cardinality can fill the volume completely before the time-based retention setting ever gets a chance to clean anything up — and a full disk stops Prometheus from writing new data entirely.

Using an Existing PVC

To bind Prometheus to a PersistentVolume that already exists — common when migrating storage or reusing a volume across reinstalls — use a selector in the volumeClaimTemplate matching labels on the existing PV, or pre-create a PVC with the exact name the StatefulSet expects (<prometheus-name>-db-prometheus-<release>-0 by default) before the chart creates its own.

Choosing the Right StorageClass

Prometheus writes constantly — every scrape, every compaction of the time-series database — which makes disk I/O performance matter more here than for most stateful workloads. A StorageClass backed by network-attached SSD-tier storage (most cloud providers' default "fast" or "premium" tier) noticeably outperforms spinning-disk-backed classes once a cluster is scraping more than a few thousand targets; on managed Kubernetes, this is usually just a matter of setting storageClassName to the provider's SSD-backed class rather than the cluster default.

Resizing later is possible if the StorageClass has allowVolumeExpansion: true set — increase the storage request in values.yaml and reapply, then expand the underlying filesystem, without needing to recreate the volume or lose existing data.

Frequently Asked Questions

Does kube-prometheus-stack persist data by default?

No — both Prometheus and Grafana are ephemeral until persistence is explicitly configured in values.yaml.

What's the difference between retention and retentionSize?

retention is time-based; retentionSize is disk-space based. Whichever limit is hit first triggers cleanup.

Can I reuse an existing PVC?

Yes, via a matching selector in volumeClaimTemplate, or by pre-creating a PVC with the exact expected name.

Conclusion

Persistent storage isn't optional for any kube-prometheus-stack deployment that matters — it's the difference between metrics that survive a routine pod restart and metrics that vanish with it. Configure both Prometheus and Grafana storage explicitly, and set retentionSize as a safety net under whatever time-based retention window you choose.