Custom Scrape Configs & Exporters in Kube-Prometheus-Stack

ServiceMonitors and PodMonitors handle almost every scrape target a kube-prometheus-stack deployment needs. The remaining cases — a static external endpoint, an uptime check, a batch job that only runs for thirty seconds — need one of three specific tools instead.

Adding additionalScrapeConfigs

For targets a ServiceMonitor or PodMonitor can't express — a static IP, an endpoint outside Kubernetes entirely — additionalScrapeConfigs lets you paste a raw Prometheus scrape_configs block straight into values.yaml:

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      - job_name: 'external-service'
        static_configs:
          - targets: ['external.example.com:9100']

It bypasses the Operator's CRD-based model entirely, which is exactly the point — it's an escape hatch for the handful of targets that genuinely don't fit the ServiceMonitor pattern, not a replacement for it.

Blackbox Exporter for Uptime Probes

The Blackbox Exporter answers a different question than everything else in the stack: not "what metrics does this service expose," but "is this endpoint even reachable." It performs HTTP, TCP, DNS, and ICMP probes against a target and reports success, latency, and response codes as metrics.

kube-prometheus-stack doesn't bundle Blackbox by default, but the Prometheus Operator supports it natively through the Probe custom resource — deploy Blackbox Exporter separately, define a Probe pointing at the URL to check, and the Operator wires the scrape configuration automatically, the same declarative pattern as ServiceMonitors.

Pushgateway for Batch Jobs

Prometheus's pull model breaks down for short-lived batch jobs — a cron job that runs for ten seconds and exits before Prometheus's next scrape interval simply never gets scraped. Pushgateway exists to bridge that gap: the job pushes its metrics to Pushgateway before exiting, and Prometheus scrapes Pushgateway itself as a normal, always-available target.

Like Blackbox, Pushgateway isn't installed by kube-prometheus-stack automatically. Deploy it as its own lightweight service, then add a ServiceMonitor pointing at it exactly as you would for any custom application — from Prometheus's perspective, Pushgateway is just another scrape target.

Global vs. Per-Target Scrape Intervals

The chart's default global scrape interval is 30 seconds, set under prometheus.prometheusSpec.scrapeInterval. Individual ServiceMonitors can override this per-target with their own interval field — useful for a noisy, high-value metric that needs 10-second resolution without dropping every other target to the same frequency and multiplying total cardinality unnecessarily.

Dropping Noisy Labels with metricRelabelings

Not every label an exporter exposes is worth keeping — request paths, user IDs, or pod-generated suffixes can quietly explode a metric's cardinality without adding any real monitoring value. A ServiceMonitor's metricRelabelings field runs after scraping but before storage, letting you drop or rewrite specific labels before they ever hit Prometheus's time-series database:

metricRelabelings:
  - sourceLabels: [__name__]
    regex: 'http_request_duration_seconds_bucket'
    action: keep

This is distinct from relabelings, which runs before the scrape and can rewrite the target itself (its address, its job name) — metricRelabelings only ever touches the metrics already collected from a target.

Frequently Asked Questions

What is additionalScrapeConfigs?

A values.yaml field for pasting raw Prometheus scrape configs for targets a ServiceMonitor can't express.

How do I monitor external endpoints?

Deploy Blackbox Exporter and define a Probe custom resource — the Operator handles the rest.

Does kube-prometheus-stack include Pushgateway?

No — deploy it separately and monitor it with a standard ServiceMonitor.

Conclusion

ServiceMonitors and PodMonitors are the right tool for nearly everything running inside the cluster. Reach for additionalScrapeConfigs, Blackbox, or Pushgateway only for the specific gaps they were built for — static external targets, uptime checks, and short-lived batch jobs, respectively.