> ## Documentation Index > Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt > Use this file to discover all available pages before exploring further. # SEV-1 Runbook > What each Fleet Manager Sev-2 alert, and each satellite alert for a hub outage, means, how to confirm it, how to recover, and how to prove recovery — the SEV-1 HealthPlan runbook for on-call responders. This page is the runbook for Fleet Manager's paging alerts, and for the [satellite alerts](#satellite-alerts) that report a hub outage the hub cannot report itself. Each alert in [the alert tables](#sev1-metrics) links straight to its own section; every section follows the same order — symptoms, blast radius, confirm, recover, validate, roll back. **Every paging alert here is a Large-Scale Event candidate.** A Sev-2 page means Incident Command applies: name an Incident Commander, open the incident channel, and follow the [Incident Severity and Response Standard](https://app.notion.com/p/3abcaf6bfadc81c7b699fd4bef589f26). Acknowledge within 15 minutes. The [satellite alerts](#satellite-alerts) do not page; treat `FleetManagerHubApiUnreachable` firing on every satellite the same way.

Purpose

Fleet Manager is the control plane for the physical fleet: operators and automation use its API to enrol, test, hand over and repair [nodes](../start-here/glossary#node), satellite agents mirror Ironic and SLURM state into it, and connectors ship inventory to customers. Customer workloads keep running when Fleet Manager is down, so its SEV-1 surface is narrow — it is Sev-2 impact: * **Operators lose the API** — no enrolment, burn-in, handover, or repair can start. * **Device state goes stale** — Fleet Manager keeps showing the last state it saw, so decisions are made on wrong data. * **Jobs stop executing** — dispatched work sits in `pending` or never progresses. * **A contracted customer feed breaches its SLA** — the Microsoft Matrix inventory feed.

SEV-1 metrics and alerts

The six alerts in this table are evaluated by central Grafana against `infra-mimir`, labelled `severity=error` with "Sev-2" in the summary, and page the Fleet Manager PagerDuty service. The **[Fleet Manager — SEV-1 dashboard](https://observability-grafana.nscale.teleport.sh/d/fleet-sev1)** (`tsh apps login observability-grafana`) has one panel per alert with the threshold drawn as a red line. | Alert | Signal | Condition | Metric source | | - | - | - | - | | [FleetManagerApiErrorRateHigh](#fleetmanagerapierrorratehigh) | Hub API answering 5xx | > 5% of requests for 10 min | nginx ingress `fleet-manager-hub-api` | | [FleetManagerApiTrafficAbsent](#fleetmanagerapitrafficabsent) | Hub API receiving no traffic | \< 5 req/s for 15 min (baseline \~80) | nginx ingress `fleet-manager-hub-api` | | [FleetManagerOpenStackStateStale](#fleetmanageropenstackstatestale) | Ironic state not mirrored | Agent restarted ≥ 3 times in 30 min, or Ironic node reflector not listed for > 600 s, for 10 min | `kube_pod_container_status_restarts_total`, `agent_list_reflector_last_list_time_seconds` | | [FleetManagerWorkflowAgentStale](#fleetmanagerworkflowagentstale) | Jobs not picked up | Agent restarted ≥ 3 times in 30 min, or reflector not listed for > 600 s, for 10 min | `kube_pod_container_status_restarts_total`, `agent_list_reflector_last_list_time_seconds` | | [FleetManagerTemporalWorkerDown](#fleetmanagertemporalworkerdown) | Workflows cannot run | 0 available temporal-worker replicas for 10 min | kube-state-metrics | | [FleetManagerMatrixFeedStalled](#fleetmanagermatrixfeedstalled) | MSFT Matrix feed late | Oldest undelivered event > 900 s for 10 min | `fleet_manager_matrix_connector_delivery_queue_oldest_undelivered_age_seconds` | Thresholds were backtested over 14 days of production data (FLEET-2759, FLEET-2762). The API error-rate, Matrix, and workflow-agent alerts each fired for the 23 September 2026 hub API outage; the OpenStack alert fired for a 10-hour agent restart loop on sys2-sta1 on 16–17 September; the rest never fired. An agent whose reflector keeps failing gives up and restarts, and each restart resets its reflector timestamp. That is why the OpenStack and workflow alerts also fire on restarts: reflector age alone never crossed 600 s during the 23 September outage.

Satellite alerts

The hub cannot alert on its own outage: on 2 October 2026 every hub VM rebooted, taking its Prometheus and Alertmanager with it. Each production satellite with the observability stack's blackbox exporter therefore probes the hub API every 15 seconds on the URL its agents use and evaluates these rules itself; the observability chart deploys no exporter for sys2-lon14, so neither alert can fire for it. They post to **`#fleetmanager-alerts-prod`**, with copies in Cloud Native Platform Reliability Engineering's `#k8s-alerts-prod` (the critical) and `#k8s-warnings-all-envs` (the warning), and do not page. The message's `Cluster:` line names the satellite; look it up in [Sites](#sites). | Alert | Signal | Condition | Metric source | | - | - | - | - | | [FleetManagerHubApiUnreachable](#fleetmanagerhubapiunreachable) | A satellite cannot reach the hub API | \< 90% of probes of `/healthz` succeed over 5 min, for 5 min | `probe_success{job="fleet-manager-satellite-hub-probe"}` | | [FleetManagerHubProbeMissing](#fleetmanagerhubprobemissing) | A satellite stopped probing the hub | No probe result for 15 min, after reporting in the last day | `probe_success{job="fleet-manager-satellite-hub-probe"}` | On the 2 October pattern, the VIP up for about 25 seconds and down for about 25, the unreachable alert fires about 6½ minutes after the flapping starts.

Preconditions

* Teleport: `tsh login`, `tsh apps login observability-grafana` for the dashboard and metrics, and `tsh kube login --all --set-context-name '{{.KubeName}}'` so the context names below resolve (without `--set-context-name` they are prefixed `nscale.teleport.sh-`). * Fleet Manager API: `fleetctl api login --scope read-only`. * OpenStack checks: the `openstack` CLI, and 1Password signed in with access to the site's `-operators` vault (`fleetctl openstack auth` reads the credential from it). * Everything runs in namespace `fleet-manager`. Restarting or rolling back a component is a Fleet Manager engineer action — see [Escalation](#escalation).

Sites

The alert's `cluster` label is the key. Look up the names each tool wants: | `cluster` label | `kubectl` context | Overlay in `k8s-cluster-fleet-manager` | `fleetctl openstack --deployment` | | - | - | - | - | | `fleet-mgr-glo1` (hub) | `fleet-mgr-prod-glo1` | `glo1/production` | — | | `fleet-mgr-sys1-ice2` | `fleet-mgr-prod-sys1-ice2` | `sys1-ice2/production` | `sys1-ice2` | | `fleet-mgr-sys2-sta1` | `fleet-mgr-prod-sys2-sta1` | `sys2-sta1/production` | `sys2-sta1` | | `fleet-mgr-sys2-osl1` | `fleet-mgr-prod-sys2-osl1` | `sys2-osl1/production` | `sys2-osl1` | | `fleet-mgr-mgt-sys1-lon14` | `fleet-mgr-prod-mgt-sys1-lon14` | `mgt-sys1-lon14/production` | `sys1-lon14` | | `fleet-mgr-mgt-sys2-lon14`¹ | `fleet-mgr-prod-mgt-sys2-lon14` | `mgt-sys2-lon14/production` | `sys2-lon14` | ¹ sys2-lon14 does not ship metrics to central Mimir yet, so no paging alert fires for it, and the observability chart deploys no blackbox exporter there, so no satellite alert does either. Commands below write ``, `` and `` for the values in this table.

Safety checks

Before changing anything: 1. **Confirm the cluster.** `kubectl config current-context` must be the context [Sites](#sites) gives for the alert's `cluster` label. 2. **Check for a deploy in flight.** A rollout that started shortly before the alert is the most likely cause, and rolling it back is usually the fix: `kubectl -n fleet-manager get rs --sort-by=.metadata.creationTimestamp`. 3. **Check for an API alert.** The workflow agent and the Matrix connector read from the hub API, so if `FleetManagerApiErrorRateHigh`, `FleetManagerApiTrafficAbsent` or `FleetManagerHubApiUnreachable` is also firing, work that one first. The OpenStack agent reads Ironic and the Temporal worker needs Temporal — their alerts are independent of the API. 4. **Never delete Fleet Manager data or Temporal workflows** to make an alert clear.

FleetManagerApiErrorRateHigh

**Symptoms.** More than 5% of requests to `api.fleet.nscale.com` return 5xx for 10 minutes. Dashboard panel **API 5xx ratio**. The UI and `fleetctl api` fail with 500/502/503. **Blast radius.** Every operator and every automated caller: no job can be dispatched, no device updated. Satellite agents that read from the hub (impairment cordon/release, workflow agent) restart-loop until it returns. Running customer workloads are unaffected. **Confirm.** 1. Which status codes? In Grafana Explore on `infra-mimir`: `sum by (status) (rate(nginx_ingress_controller_requests{cluster="fleet-mgr-glo1",ingress="fleet-manager-hub-api",status=~"5.."}[5m]))`. `502`/`503` means the API pods are not serving; `500` means the application is erroring. 2. Pod state: `kubectl --context fleet-mgr-prod-glo1 -n fleet-manager get pods -l app.kubernetes.io/component=api`. 3. Readiness checks the database — if pods are not ready, check PostgreSQL: `kubectl --context fleet-mgr-prod-glo1 -n fleet-manager get pods -l cnpg.io/cluster=fleet-manager-postgres-cluster`. 4. Application errors: logs in `product-loki`, `{cluster="fleet-mgr-glo1",namespace="fleet-manager",container="api"} |= "ERROR"`. **Recovery procedure.** 1. **A rollout started just before the alert** → roll back the release (see [Rollback](#rollback-api)). 2. **Pods crash-looping or not ready and PostgreSQL is unhealthy** → escalate to Fleet Manager with the PostgreSQL pod state; the API cannot recover until the database does. 3. **Pods healthy but returning `500`** → capture one failing request's `trace_id` from the logs and escalate to Fleet Manager. 4. **Pods stuck with no clear cause** → a Fleet Manager engineer restarts them: `kubectl --context fleet-mgr-prod-glo1 -n fleet-manager rollout restart deployment/fleet-manager-hub-api`. **Validation.** The alert resolves; **API 5xx ratio** is back under 1%; `fleetctl api device list --format count` succeeds; workflow-agent pods stop restarting. **Rollback.** Revert the release bump commit in `k8s-cluster-fleet-manager`. It moves the `targetRevision` of every overlay at once, so revert the whole commit, not one file, or the hub and satellites end up on different releases. Let Argo sync. Schema migrations are not reversed by a rollback: if the failing release ran a migration, escalate before rolling back.

FleetManagerApiTrafficAbsent

**Symptoms.** The hub API ingress sees fewer than 5 requests per second for 15 minutes (baseline is about 80), or the metric disappears. Dashboard panel **API request rate**. **Blast radius.** Same as [FleetManagerApiErrorRateHigh](#fleetmanagerapierrorratehigh) if the API is unreachable. If traffic is merely absent, callers cannot reach the ingress at all — DNS, the load balancer, or the ingress controller — and they see timeouts rather than 5xx. **Confirm.** 1. Can you reach it? `curl -s -o /dev/null -w '%{http_code}\n' https://api.fleet.nscale.com/healthz` — expect `200`. 2. Ingress controller running: `kubectl --context fleet-mgr-prod-glo1 get pods -A -l app.kubernetes.io/name=ingress-nginx`. 3. If the metric itself is missing but `curl` succeeds, the ingress metrics pipeline is broken, not the API — escalate to the Observability Squad instead. **Recovery procedure.** 1. `curl` fails and the ingress controller is down → escalate to the platform owner of `fleet-mgr-glo1` (Cloud Native Platform Reliability Engineering) with the controller pod state. 2. `curl` fails and the controller is up → follow [FleetManagerApiErrorRateHigh](#fleetmanagerapierrorratehigh) from step 2. 3. `curl` succeeds and traffic is still near zero → the callers are down. Check the satellite agents (next sections) and the Fleet Manager UI. **Validation.** The alert resolves; **API request rate** returns to its normal band; `curl` returns `200`. **Rollback.** As for [FleetManagerApiErrorRateHigh](#rollback-api) when a Fleet Manager release caused it.

FleetManagerHubApiUnreachable

**Symptoms.** Fewer than 90% of a satellite's probes of `https://api.fleet.nscale.com/healthz` succeeded over 5 minutes, for 5 minutes. It posts to `#fleetmanager-alerts-prod` and `#k8s-alerts-prod`, and does not page. When the hub ingress is at fault, every satellite fires within a minute or two of each other, alongside `FleetManagerOpenStackAttribution*`, agent `CrashLoopBackOff` and `TargetDown` noise from the same satellites. A total outage of 90 seconds to 5 minutes fires only after it has ended, and clears within a minute: a steady `200` from the curl loop below then means a short outage that is already over. **Blast radius.** Every satellite agent that calls the hub fails or restart-loops, so device state, job progress and attribution stop flowing, as for [FleetManagerApiErrorRateHigh](#fleetmanagerapierrorratehigh). If only one satellite fires, the fault is that site's network path or its blackbox exporter, and other sites are unaffected. **Confirm.** 1. One satellite or all? If only one fires, go to recovery step 3. 2. Probe the VIP once a second: `while sleep 1; do curl -s -o /dev/null --max-time 2 -w '%{http_code} %{remote_ip}\n' https://api.fleet.nscale.com/healthz; done`. Steady `200` from `193.143.123.43` means the hub answers. Runs of `200` broken by runs of `000` (the TLS handshake is reset, curl exit 35) are the load-balancer VIP flapping, as on 13 September and 2 October 2026. A steady `502` or `503` means the ingress is up and the API pods are not. 3. Was the load-balancer pair rebuilt? proxmox-ccm owns the VIP: `kubectl --context fleet-mgr-prod-glo1 get loadbalancerallocations.proxmox.nscale.com nginx-ingress-nginx-controller-lba -o jsonpath='{range .status.vmInstances[*]}{.conditions[?(@.type=="Planned")].lastTransitionTime}{"\n"}{end}'`. A time from today means proxmox-ccm replaced the load-balancer VMs. 4. proxmox-ccm logs in `product-loki` (its pods restart, so `kubectl logs` loses them): `{cluster="fleet-mgr-glo1",namespace="proxmox-ccm"} |~ "VM disappeared|not found|resetting for re-provisioning"`. "1 of 2 VMs missing, resetting for re-provisioning" is the rebuild that leaves the old VMs running, so two of them answer for the VIP at once. **Recovery procedure.** 1. **VIP flapping or the load-balancer pair rebuilt** → escalate to Cloud Native Platform Reliability Engineering with the curl output, the LoadBalancerAllocation times and the log lines. Fleet Manager has no Proxmox access; they remove the old load-balancer VMs still answering for the VIP. 2. **VIP steady but the API answers `502`/`503`** → follow [FleetManagerApiErrorRateHigh](#fleetmanagerapierrorratehigh) from step 2. 3. **Only one satellite fires** → check its exporter: `kubectl --context -n prometheus-blackbox-exporter get pods`. Unhealthy pods go to the Observability Squad; healthy pods mean that site cannot reach the hub, which goes to the site's platform owner. **Validation.** The alert resolves on every satellite, the curl loop prints only `200`, and the satellite agents stop restarting. **Rollback.** As for [FleetManagerApiErrorRateHigh](#rollback-api) when a Fleet Manager release caused it. A flapping VIP is not a Fleet Manager change and has nothing to roll back.

FleetManagerHubProbeMissing

**Symptoms.** A satellite's hub probe reported in the last day and has not reported for 15 minutes, so [FleetManagerHubApiUnreachable](#fleetmanagerhubapiunreachable) cannot fire for it. A `TargetDown` on job `fleet-manager-satellite-hub-probe` means scraping the exporter's `/probe` endpoint fails, for example an unknown module or an exporter that does not answer. It does not mean the hub is down: a failing hub probe still scrapes cleanly and reports `probe_success 0`. The warning clears a day after the probe stopped, fixed or not. **Blast radius.** No outage. That satellite can no longer report a hub outage; the others still can. **Confirm.** The probe runs through the observability blackbox exporter: `kubectl --context -n prometheus-blackbox-exporter get service,pods`. It must have a `prometheus-blackbox-exporter` Service with port `http` and the `http_2xx` module. **Recovery procedure.** Escalate to the Observability Squad: the exporter is theirs (`charts/observability/templates/otel/agent/applications/blackbox-exporter-application.yaml` in `nscaledev/k8s-deploy-observability`). On `fleet-mgr-mgt-sys1-lon14` and `fleet-mgr-sys2-sta1` it was running without an owning Argo application as of October 2026, so it may have been removed as unused. **Validation.** In `metrics-all`, `probe_success{cluster="",job="fleet-manager-satellite-hub-probe"}` reports again and the warning resolves. **Rollback.** None — nothing in Fleet Manager changed.

FleetManagerOpenStackStateStale

**Symptoms.** A satellite's OpenStack agent is restart-looping (3 or more restarts in 30 minutes), or has not listed Ironic nodes for more than 10 minutes. Dashboard panel **OpenStack agent: Ironic reflector age** — a restart loop shows as a sawtooth that never settles under a minute. The alert's `cluster` label names the site. **Blast radius.** Every device at that site. Fleet Manager keeps its last-known `operational_status`, provision state and conditions, so quarantines, failures and allocations that happen in Ironic are not reflected. Workflows that preflight on Fleet Manager state act on stale data. Other sites are unaffected. **Confirm.** 1. Agent restarts: `kubectl --context -n fleet-manager get pods -l app.kubernetes.io/component=openstack-agent` (`RESTARTS` column), then `kubectl --context -n fleet-manager logs --previous` for why the last run exited. 2. Agent errors: `{cluster="",namespace="fleet-manager",container="openstack-agent"} |= "ERROR"` in `product-loki`. Authentication or connection errors point at Ironic/Keystone, not the agent. 3. Ironic reachable: `eval "$(fleetctl openstack auth --deployment --env prod)"`, then `openstack baremetal node list --limit 1`. **Recovery procedure.** 1. **Ironic or Keystone unreachable or rejecting credentials** → escalate to the site's OpenStack owner (Compute - OpenStack/Proxmox); the agent recovers by itself once Ironic answers. 2. **Agent pod crash-looping or not running** → escalate to Fleet Manager with the pod events. 3. **Agent running, no errors, reflector still not listing** → a Fleet Manager engineer restarts it: `kubectl --context -n fleet-manager rollout restart deployment/fleet-manager-satellite-openstack-agent`. **Validation.** The alert resolves; the pod's `RESTARTS` count stops climbing; the site's line on **Ironic reflector age** stays under a minute; a recently changed node shows its new state in `fleetctl api device list --name-exact --format json`. **Rollback.** As for [FleetManagerApiErrorRateHigh](#rollback-api) when a Fleet Manager release caused it.

FleetManagerWorkflowAgentStale

**Symptoms.** A satellite's workflow agent is restart-looping (3 or more restarts in 30 minutes), or has not listed jobs for more than 10 minutes. Dashboard panel **Workflow agent: reflector age**. If every satellite fires at once, the cause is the hub API. **Blast radius.** New jobs for that site are not submitted to Temporal, and running jobs stop reporting progress, so job and device status freeze. Work already running inside Temporal continues. **Confirm.** 1. Is the hub API healthy? If an API alert is firing, work that first — the workflow agent reads jobs from the API. 2. Agent restarts: `kubectl --context -n fleet-manager get pods -l app.kubernetes.io/component=workflow-agent` (`RESTARTS` column), then `kubectl --context -n fleet-manager logs --previous`. 3. Agent errors: `{cluster="",namespace="fleet-manager",container="workflow-agent"} |= "ERROR"`. **Recovery procedure.** 1. **Hub API down** → recover the API; the agent catches up on its own. 2. **Agent pod not running or crash-looping** → escalate to Fleet Manager with the pod events. 3. **Agent running, no errors, reflector not listing** → a Fleet Manager engineer restarts it: `kubectl --context -n fleet-manager rollout restart deployment/fleet-manager-satellite-workflow-agent`. **Validation.** The alert resolves; the pod's `RESTARTS` count stops climbing; **Workflow agent: reflector age** stays under a minute; a newly dispatched job at that site leaves `pending`. **Rollback.** As for [FleetManagerOpenStackStateStale](#fleetmanageropenstackstatestale).

FleetManagerTemporalWorkerDown

**Symptoms.** A satellite has no available `fleet-manager-satellite-temporal-worker` replica for 10 minutes. Dashboard panel **Temporal worker: available replicas** at 0. **Blast radius.** No workflow activity runs at that site: enrolment, burn-in, reimage, handover and repair all stop mid-step. Temporal keeps the workflows, so they resume when a worker returns — nothing is lost, but everything waits. **Confirm.** 1. `kubectl --context -n fleet-manager get deploy,pods -l app.kubernetes.io/component=temporal-worker`. 2. `kubectl --context -n fleet-manager describe pod ` — look for image pull errors, `OOMKilled`, or unschedulable events. 3. Temporal server itself: `kubectl --context -n temporal get pods`. **Recovery procedure.** 1. **Image pull or config error after a rollout** → roll back the satellite release (below). 2. **`OOMKilled` or unschedulable** → escalate to Fleet Manager with the pod events; capacity or limits need an engineer. 3. **Temporal server pods unhealthy** → escalate to Fleet Manager; workers cannot connect until the server recovers. **Validation.** The alert resolves; available replicas back at the deployment's desired count; a waiting workflow advances (its job's status changes in `fleetctl api job list`). **Rollback.** As for [FleetManagerOpenStackStateStale](#fleetmanageropenstackstatestale).

FleetManagerMatrixFeedStalled

**Symptoms.** The oldest undelivered Matrix delivery-queue event is older than 15 minutes, for 10 minutes. Dashboard panel **Matrix connector: oldest undelivered event**. The alert's `connector` label names the instance (`matrix-prod` for sin1, `matrix-wdc1-prod` for wdc1). **Blast radius.** Microsoft's Matrix inventory in Kusto is behind the contracted 5-minute freshness SLA — this is customer-visible for that instance. Other instances are independent. **Confirm, recover, validate, roll back.** Follow [Matrix Connector Operations](./matrix-connector-operations): its [operator health check](./matrix-connector-operations#for-operators) confirms the breach, and the [engineer error classes](./matrix-connector-operations#for-engineers) cover causes and fixes. If the hub API is also down, recover the API first — the connector reads from it. **Customer communication.** Do not contact the customer directly. Fleet Manager decides when Microsoft is told, per the [escalation order](./matrix-connector-operations#who-to-call).

Escalation

| Situation | Who | | - | - | | Any alert on this page, or unsure | Fleet Manager on-call via PagerDuty for the paging alerts (the page you received); Slack **`#fleetmanager-alerts-prod`** for all of them | | Ironic, Keystone, or OpenStack unavailable at a site | Compute - OpenStack/Proxmox | | Ingress, DNS, the load-balancer VIP, or cluster platform on `fleet-mgr-glo1` | Cloud Native Platform Reliability Engineering | | Dashboards, alert routing, metrics missing, or a satellite's blackbox exporter | Observability Squad | | Customer impact confirmed (Matrix) | Fleet Manager, who informs the customer |

References

* [Incident Severity and Response Standard](https://app.notion.com/p/3abcaf6bfadc81c7b699fd4bef589f26) * [Runbook Authoring Standards](https://app.notion.com/p/35ecaf6bfadc81528594e83b4bbcc222) * [Status Reference](./status-reference) — what each device and job status means * [Matrix Connector Operations](./matrix-connector-operations) * Alert rules: `charts/observability/templates/grafana/alerts/fleet-manager-alerts.yaml` in `nscaledev/k8s-deploy-observability`; the satellite alerts are `deployment/charts/satellite/templates/hub-probe/prometheusrule.yaml` in this repository This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.