Alertmanager Watchdog & External Alertmanager

Every kube-prometheus-stack install ships with an alert called Watchdog that never stops firing — and that's the point. It's a dead man's switch for the alerting pipeline itself, not a real incident. This guide covers what it's for, when to disable it, and how to swap the bundled Alertmanager for one you already run.

What Is the Watchdog Alert?

Watchdog is a synthetic PrometheusRule included in the chart's default alert rules, defined with an expression that always evaluates true — usually just vector(1). It exists purely to prove the full path from Prometheus to Alertmanager to your notification channel is working end to end.

Most teams route Watchdog to a dead-man's-snitch style service — PagerDuty's "Dead Man's Snitch", Cronitor, or a similar heartbeat monitor — configured to alert them if Watchdog ever stops arriving. That absence is the actual signal: it means Prometheus, Alertmanager, or the network between them has broken silently.

Why You Probably Want to Keep It

It's tempting to disable Watchdog because it clutters the Alertmanager UI with a permanently firing alert. Resist that — without it, a broken alerting pipeline (a crashed Alertmanager pod, a misconfigured route, a dropped webhook) fails completely silently. You'd only find out when a real incident's alert also never arrived.

The better fix for UI clutter is routing Watchdog to its own dedicated receiver that nobody has to look at day to day, rather than removing the safety net entirely.

Disabling It Anyway

If you're certain you don't need it — for example, you already monitor Alertmanager's own health through a separate uptime check — disable the rule group in values.yaml:

defaultRules:
  rules:
    general: false

Using an External Alertmanager

Some organizations already run a shared, cluster-independent Alertmanager — often fronting several Prometheus instances across multiple clusters — and don't want kube-prometheus-stack deploying a second one. Two changes handle this:

1. Set alertmanager.enabled: false to stop the chart from deploying its own Alertmanager StatefulSet.
2. Point Prometheus at the external instance via prometheus.prometheusSpec.alertingEndpoints, giving it the external Alertmanager's service name, namespace, and port.

Everything else — ServiceMonitors, PrometheusRule objects, scrape configs — continues to work unchanged; only the alert delivery destination moves outside the chart.

Disabling Individual Default Rules

kube-prometheus-stack ships dozens of default alert rules grouped by component (kubernetes-apps, node-exporter, kubernetes-storage, and more). If one specific rule is too noisy rather than the whole group, defaultRules.rules.<group>: false turns off that entire group — there's no built-in per-rule toggle, so silencing a single noisy rule inside an otherwise-useful group usually means either living with it, silencing it in Alertmanager, or forking the rule via a Kustomize patch on the rendered manifests.

Customizing Notification Templates

The message format Alertmanager sends to Slack, email, or a webhook is controlled by Go templates under alertmanager.templateFiles in values.yaml, referenced from your receiver configuration. This is where you'd add cluster name, runbook links, or a custom Slack message layout rather than the default terse alert summary.

Frequently Asked Questions

What is the Watchdog alert for?

It's a synthetic, always-firing alert used as a dead man's switch to confirm the alerting pipeline is healthy — its absence is the real signal.

How do I use an Alertmanager I already run?

Set alertmanager.enabled: false and point prometheus.prometheusSpec.alertingEndpoints at your existing instance.

Can I disable a single default alert rule?

Not individually out of the box — defaultRules.rules toggles whole groups; a single rule needs a values override or manifest patch.

Conclusion

Watchdog looks like noise until the day it silently stops — that's the entire reason it exists. Combine it with a properly scoped external Alertmanager setup when you need one, and the alerting layer of kube-prometheus-stack stays observable in its own right, not just a black box that sends notifications somewhere.