> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Relationship: Mimir (Observability platform)

> Mimir recording rules Fleet Manager reads to derive device health, with NVLink-domain health ingested as a snapshot only.

* **Kind:** Observation
* **Version:** 1
* **Fleet Manager owner:** Fleet Manager squad, #fleet-manager-chat (individual claimant pending)
* **Counterpart owner:** unassigned, Observability squad, #observability-chat
* **Last reviewed:** 2026-09-09

## What flows

The per-satellite `health-status-agent` (`src/fleet_manager/agents/health_status/`) runs PromQL against Mimir
every 30 s with a 5 min resync as deployed (`healthStatusAgent.pollInterval` in the satellite chart; the Python
model's own fallback is 60 s and is not what runs). The default query is the `chassis_health` recording rule; an optional second query
covers switch health. The agent's only output today is `Device.health_status` (healthy, degraded, unknown) for
compute and switch [devices](../start-here/glossary#node), written with `device:update`. The `TelemetryHealthy` condition is defined in the
condition registry with source `health-status` but no code writes it yet. The optional [NVLink](../start-here/glossary#nvlink)-domain query is
ingested as a validated snapshot only; domain reconciliation and [Radar](../start-here/glossary#radar-api) submission are deferred
(`src/fleet_manager/agents/health_status/main.py`). Fleet Manager reads only; it never writes metrics into Mimir
through this path. Fleet Manager's own `fleet_resource_*` metrics reach the central stack because
Fleet Manager runs on the same cluster, not through a contract.

## What breaks if this changes

A renamed or altered recording rule flips every device at a site to `unknown` health, which the Radar
projection reports as unavailable. A Mimir outage freezes health at
its last value with no Fleet Manager alert distinguishing "stale" from "healthy".

## Enforcement

* **Principal:** `health-status-agent-<site>`.
* **Mechanism:** towards Fleet Manager, FM OAuth client (private-key JWT) in the hub chart satellite roster; towards
  Mimir, the username and password in the `mimir-login` [1Password](../start-here/glossary#1password) item configured in the satellite chart.
* Satellite roster client `health-status-agent-<site>` (`device:update`) in `deployment/charts/hub/values.yaml`.
* Settings: `src/fleet_manager/settings/agents/health_status.py` (`mimir_query`, `switch_mimir_query`,
  `domain_mimir_query`).

## Gaps

* Most of the health signal behind `chassis_health` originates from [BMCs](../start-here/glossary#bmc), and BMC firmware is often unreliable;
  BMC health itself is owned by Operational Engineering and Support, so a wrong `health_status` can be a
  data-plane fault upstream of both Mimir and Fleet Manager.

* The recording rules Fleet Manager depends on are not versioned or owned in any shared place; a rule change
  needs no Fleet Manager review today.

* Ownership of Fleet Manager API alerting between FOE and Observability is unresolved.

## Change log

* 2026-09-09 v1: created; describes present behaviour.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.