Skip to main content
  • Kind: Observation
  • Version: 1
  • Fleet Manager owner: Fleet Manager squad, #fleet-manager-chat (individual claimant pending)
  • Counterpart owner: unassigned, Observability squad, #observability-chat
  • Last reviewed: 2026-09-09

​
What flows

The per-satellite health-status-agent (src/fleet_manager/agents/health_status/) runs PromQL against Mimir every 30 s with a 5 min resync as deployed (healthStatusAgent.pollInterval in the satellite chart; the Python model’s own fallback is 60 s and is not what runs). The default query is the chassis_health recording rule; an optional second query covers switch health. The agent’s only output today is Device.health_status (healthy, degraded, unknown) for compute and switch devices, written with device:update. The TelemetryHealthy condition is defined in the condition registry with source health-status but no code writes it yet. The optional NVLink-domain query is ingested as a validated snapshot only; domain reconciliation and Radar submission are deferred (src/fleet_manager/agents/health_status/main.py). Fleet Manager reads only; it never writes metrics into Mimir through this path. Fleet Manager’s own fleet_resource_* metrics reach the central stack because Fleet Manager runs on the same cluster, not through a contract.

​
What breaks if this changes

A renamed or altered recording rule flips every device at a site to unknown health, which the Radar projection reports as unavailable. A Mimir outage freezes health at its last value with no Fleet Manager alert distinguishing “stale” from “healthy”.

​
Enforcement

  • Principal: health-status-agent-<site>.
  • Mechanism: towards Fleet Manager, FM OAuth client (private-key JWT) in the hub chart satellite roster; towards Mimir, the username and password in the mimir-login 1Password item configured in the satellite chart.
  • Satellite roster client health-status-agent-<site> (device:update) in deployment/charts/hub/values.yaml.
  • Settings: src/fleet_manager/settings/agents/health_status.py (mimir_query, switch_mimir_query, domain_mimir_query).

​
Gaps

  • Most of the health signal behind chassis_health originates from BMCs, and BMC firmware is often unreliable; BMC health itself is owned by Operational Engineering and Support, so a wrong health_status can be a data-plane fault upstream of both Mimir and Fleet Manager.
  • The recording rules Fleet Manager depends on are not versioned or owned in any shared place; a rule change needs no Fleet Manager review today.
  • Ownership of Fleet Manager API alerting between FOE and Observability is unresolved.

​
Change log

  • 2026-09-09 v1: created; describes present behaviour.