Remote Write: Sending Metrics to Mimir & VictoriaMetrics

Prometheus's remote_write feature streams every scraped sample to a second storage backend in real time, independent of local retention. For kube-prometheus-stack, that's the simplest path to long-term storage when a managed or centrally-run backend like Grafana Mimir or VictoriaMetrics already exists.

Why Remote Write Metrics at All

Local Prometheus storage inside kube-prometheus-stack is bounded by disk and typically kept to 15-30 days. Teams that need longer retention, cross-cluster querying, or centralized metrics from many clusters need a second backend — remote_write is the mechanism that gets data there without changing anything about how Prometheus scrapes or how ServiceMonitors work.

Enabling remote_write

Configure it directly in values.yaml under the Prometheus custom resource spec:

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: https://mimir.example.com/api/v1/push
        basicAuth:
          username:
            name: mimir-credentials
            key: username
          password:
            name: mimir-credentials
            key: password

Prometheus keeps writing to its own local TSDB as normal — remoteWrite is additive, not a replacement for local storage.

Sending to Grafana Mimir

Mimir accepts the standard Prometheus remote_write protocol at its /api/v1/push endpoint, so the configuration above works whether Mimir is self-hosted or run as part of Grafana Cloud — only the URL and authentication method change. Mimir's horizontally-scalable architecture makes it a common choice once a single self-hosted long-term storage backend needs to ingest from many kube-prometheus-stack clusters at once.

Sending to VictoriaMetrics

VictoriaMetrics implements the same remote_write protocol, typically at a /api/v1/write path on its vminsert component. It's a popular choice specifically for its lower resource footprint compared to Mimir or Cortex at similar ingestion volume, without requiring any different configuration shape from Prometheus's side.

remote_write vs. Thanos

These solve overlapping but distinct problems. Thanos extends Prometheus itself — the same Prometheus and Grafana you already run gain unlimited retention and cross-cluster querying through a sidecar and object storage, with no separate ingestion backend to operate. remote_write to Mimir or VictoriaMetrics instead ships data to a genuinely separate system, which makes sense when that system is centrally managed (Grafana Cloud) or already serves other teams' clusters.

As a rule of thumb: reach for Thanos when you want to keep everything within the Prometheus/Grafana ecosystem you already have; reach for remote_write when a centralized backend already exists and you just need to feed it.

Where Grafana Alloy Fits

Grafana Alloy is Grafana's newer, OpenTelemetry-based telemetry collector — the successor to Grafana Agent. It runs as a separate deployment alongside kube-prometheus-stack rather than replacing Prometheus, and can forward metrics (via remote_write), logs, and traces to Mimir, Loki, and Tempo from a single collector. Teams already standardizing on Alloy for logs and traces often route Prometheus's remote_write through it as well, rather than configuring Prometheus to talk to Mimir directly.

Frequently Asked Questions

How do I enable remote_write?

Set prometheus.prometheusSpec.remoteWrite in values.yaml with the target URL and credentials.

remote_write or Thanos — which one?

Thanos extends your existing Prometheus/Grafana; remote_write ships data to a separate backend like Mimir or VictoriaMetrics, better when one already exists centrally.

Does this work with VictoriaMetrics?

Yes, it accepts the same remote_write protocol as any other compatible backend.

Conclusion

remote_write is the lowest-friction way to get kube-prometheus-stack's metrics into a long-term or centralized backend — no sidecar, no new query layer, just a URL and credentials. Choose it over Thanos specifically when that backend already exists; choose Thanos when you'd rather extend the stack you already have.