Skip to main content
This page is the runbook for Fleet Manager’s paging alerts, and for the satellite alerts that report a hub outage the hub cannot report itself. Each alert in the alert tables links straight to its own section; every section follows the same order — symptoms, blast radius, confirm, recover, validate, roll back.
Every paging alert here is a Large-Scale Event candidate. A Sev-2 page means Incident Command applies: name an Incident Commander, open the incident channel, and follow the Incident Severity and Response Standard. Acknowledge within 15 minutes. The satellite alerts do not page; treat FleetManagerHubApiUnreachable firing on every satellite the same way.

​
Purpose

Fleet Manager is the control plane for the physical fleet: operators and automation use its API to enrol, test, hand over and repair nodes, satellite agents mirror Ironic and SLURM state into it, and connectors ship inventory to customers. Customer workloads keep running when Fleet Manager is down, so its SEV-1 surface is narrow — it is Sev-2 impact:
  • Operators lose the API — no enrolment, burn-in, handover, or repair can start.
  • Device state goes stale — Fleet Manager keeps showing the last state it saw, so decisions are made on wrong data.
  • Jobs stop executing — dispatched work sits in pending or never progresses.
  • A contracted customer feed breaches its SLA — the Microsoft Matrix inventory feed.

​
SEV-1 metrics and alerts

The six alerts in this table are evaluated by central Grafana against infra-mimir, labelled severity=error with “Sev-2” in the summary, and page the Fleet Manager PagerDuty service. The Fleet Manager — SEV-1 dashboard (tsh apps login observability-grafana) has one panel per alert with the threshold drawn as a red line. Thresholds were backtested over 14 days of production data (FLEET-2759, FLEET-2762). The API error-rate, Matrix, and workflow-agent alerts each fired for the 23 September 2026 hub API outage; the OpenStack alert fired for a 10-hour agent restart loop on sys2-sta1 on 16–17 September; the rest never fired.
An agent whose reflector keeps failing gives up and restarts, and each restart resets its reflector timestamp. That is why the OpenStack and workflow alerts also fire on restarts: reflector age alone never crossed 600 s during the 23 September outage.

​
Satellite alerts

The hub cannot alert on its own outage: on 2 October 2026 every hub VM rebooted, taking its Prometheus and Alertmanager with it. Each production satellite with the observability stack’s blackbox exporter therefore probes the hub API every 15 seconds on the URL its agents use and evaluates these rules itself; the observability chart deploys no exporter for sys2-lon14, so neither alert can fire for it. They post to #fleetmanager-alerts-prod, with copies in Cloud Native Platform Reliability Engineering’s #k8s-alerts-prod (the critical) and #k8s-warnings-all-envs (the warning), and do not page. The message’s Cluster: line names the satellite; look it up in Sites. On the 2 October pattern, the VIP up for about 25 seconds and down for about 25, the unreachable alert fires about 6½ minutes after the flapping starts.

​
Preconditions

  • Teleport: tsh login, tsh apps login observability-grafana for the dashboard and metrics, and tsh kube login --all --set-context-name '{{.KubeName}}' so the context names below resolve (without --set-context-name they are prefixed nscale.teleport.sh-).
  • Fleet Manager API: fleetctl api login --scope read-only.
  • OpenStack checks: the openstack CLI, and 1Password signed in with access to the site’s <deployment>-operators vault (fleetctl openstack auth reads the credential from it).
  • Everything runs in namespace fleet-manager. Restarting or rolling back a component is a Fleet Manager engineer action — see Escalation.

​
Sites

The alert’s cluster label is the key. Look up the names each tool wants: ¹ sys2-lon14 does not ship metrics to central Mimir yet, so no paging alert fires for it, and the observability chart deploys no blackbox exporter there, so no satellite alert does either. Commands below write <cluster>, <context> and <deployment> for the values in this table.

​
Safety checks

Before changing anything:
  1. Confirm the cluster. kubectl config current-context must be the context Sites gives for the alert’s cluster label.
  2. Check for a deploy in flight. A rollout that started shortly before the alert is the most likely cause, and rolling it back is usually the fix: kubectl -n fleet-manager get rs --sort-by=.metadata.creationTimestamp.
  3. Check for an API alert. The workflow agent and the Matrix connector read from the hub API, so if FleetManagerApiErrorRateHigh, FleetManagerApiTrafficAbsent or FleetManagerHubApiUnreachable is also firing, work that one first. The OpenStack agent reads Ironic and the Temporal worker needs Temporal — their alerts are independent of the API.
  4. Never delete Fleet Manager data or Temporal workflows to make an alert clear.

​
FleetManagerApiErrorRateHigh

Symptoms. More than 5% of requests to api.fleet.nscale.com return 5xx for 10 minutes. Dashboard panel API 5xx ratio. The UI and fleetctl api fail with 500/502/503. Blast radius. Every operator and every automated caller: no job can be dispatched, no device updated. Satellite agents that read from the hub (impairment cordon/release, workflow agent) restart-loop until it returns. Running customer workloads are unaffected. Confirm.
  1. Which status codes? In Grafana Explore on infra-mimir: sum by (status) (rate(nginx_ingress_controller_requests{cluster="fleet-mgr-glo1",ingress="fleet-manager-hub-api",status=~"5.."}[5m])). 502/503 means the API pods are not serving; 500 means the application is erroring.
  2. Pod state: kubectl --context fleet-mgr-prod-glo1 -n fleet-manager get pods -l app.kubernetes.io/component=api.
  3. Readiness checks the database — if pods are not ready, check PostgreSQL: kubectl --context fleet-mgr-prod-glo1 -n fleet-manager get pods -l cnpg.io/cluster=fleet-manager-postgres-cluster.
  4. Application errors: logs in product-loki, {cluster="fleet-mgr-glo1",namespace="fleet-manager",container="api"} |= "ERROR".
Recovery procedure.
  1. A rollout started just before the alert → roll back the release (see Rollback).
  2. Pods crash-looping or not ready and PostgreSQL is unhealthy → escalate to Fleet Manager with the PostgreSQL pod state; the API cannot recover until the database does.
  3. Pods healthy but returning 500 → capture one failing request’s trace_id from the logs and escalate to Fleet Manager.
  4. Pods stuck with no clear cause → a Fleet Manager engineer restarts them: kubectl --context fleet-mgr-prod-glo1 -n fleet-manager rollout restart deployment/fleet-manager-hub-api.
Validation. The alert resolves; API 5xx ratio is back under 1%; fleetctl api device list --format count succeeds; workflow-agent pods stop restarting. Rollback. Revert the release bump commit in k8s-cluster-fleet-manager. It moves the targetRevision of every overlay at once, so revert the whole commit, not one file, or the hub and satellites end up on different releases. Let Argo sync. Schema migrations are not reversed by a rollback: if the failing release ran a migration, escalate before rolling back.

​
FleetManagerApiTrafficAbsent

Symptoms. The hub API ingress sees fewer than 5 requests per second for 15 minutes (baseline is about 80), or the metric disappears. Dashboard panel API request rate. Blast radius. Same as FleetManagerApiErrorRateHigh if the API is unreachable. If traffic is merely absent, callers cannot reach the ingress at all — DNS, the load balancer, or the ingress controller — and they see timeouts rather than 5xx. Confirm.
  1. Can you reach it? curl -s -o /dev/null -w '%{http_code}\n' https://api.fleet.nscale.com/healthz — expect 200.
  2. Ingress controller running: kubectl --context fleet-mgr-prod-glo1 get pods -A -l app.kubernetes.io/name=ingress-nginx.
  3. If the metric itself is missing but curl succeeds, the ingress metrics pipeline is broken, not the API — escalate to the Observability Squad instead.
Recovery procedure.
  1. curl fails and the ingress controller is down → escalate to the platform owner of fleet-mgr-glo1 (Cloud Native Platform Reliability Engineering) with the controller pod state.
  2. curl fails and the controller is up → follow FleetManagerApiErrorRateHigh from step 2.
  3. curl succeeds and traffic is still near zero → the callers are down. Check the satellite agents (next sections) and the Fleet Manager UI.
Validation. The alert resolves; API request rate returns to its normal band; curl returns 200. Rollback. As for FleetManagerApiErrorRateHigh when a Fleet Manager release caused it.

​
FleetManagerHubApiUnreachable

Symptoms. Fewer than 90% of a satellite’s probes of https://api.fleet.nscale.com/healthz succeeded over 5 minutes, for 5 minutes. It posts to #fleetmanager-alerts-prod and #k8s-alerts-prod, and does not page. When the hub ingress is at fault, every satellite fires within a minute or two of each other, alongside FleetManagerOpenStackAttribution*, agent CrashLoopBackOff and TargetDown noise from the same satellites. A total outage of 90 seconds to 5 minutes fires only after it has ended, and clears within a minute: a steady 200 from the curl loop below then means a short outage that is already over. Blast radius. Every satellite agent that calls the hub fails or restart-loops, so device state, job progress and attribution stop flowing, as for FleetManagerApiErrorRateHigh. If only one satellite fires, the fault is that site’s network path or its blackbox exporter, and other sites are unaffected. Confirm.
  1. One satellite or all? If only one fires, go to recovery step 3.
  2. Probe the VIP once a second: while sleep 1; do curl -s -o /dev/null --max-time 2 -w '%{http_code} %{remote_ip}\n' https://api.fleet.nscale.com/healthz; done. Steady 200 from 193.143.123.43 means the hub answers. Runs of 200 broken by runs of 000 (the TLS handshake is reset, curl exit 35) are the load-balancer VIP flapping, as on 13 September and 2 October 2026. A steady 502 or 503 means the ingress is up and the API pods are not.
  3. Was the load-balancer pair rebuilt? proxmox-ccm owns the VIP: kubectl --context fleet-mgr-prod-glo1 get loadbalancerallocations.proxmox.nscale.com nginx-ingress-nginx-controller-lba -o jsonpath='{range .status.vmInstances[*]}{.conditions[?(@.type=="Planned")].lastTransitionTime}{"\n"}{end}'. A time from today means proxmox-ccm replaced the load-balancer VMs.
  4. proxmox-ccm logs in product-loki (its pods restart, so kubectl logs loses them): {cluster="fleet-mgr-glo1",namespace="proxmox-ccm"} |~ "VM disappeared|not found|resetting for re-provisioning". “1 of 2 VMs missing, resetting for re-provisioning” is the rebuild that leaves the old VMs running, so two of them answer for the VIP at once.
Recovery procedure.
  1. VIP flapping or the load-balancer pair rebuilt → escalate to Cloud Native Platform Reliability Engineering with the curl output, the LoadBalancerAllocation times and the log lines. Fleet Manager has no Proxmox access; they remove the old load-balancer VMs still answering for the VIP.
  2. VIP steady but the API answers 502/503 → follow FleetManagerApiErrorRateHigh from step 2.
  3. Only one satellite fires → check its exporter: kubectl --context <context> -n prometheus-blackbox-exporter get pods. Unhealthy pods go to the Observability Squad; healthy pods mean that site cannot reach the hub, which goes to the site’s platform owner.
Validation. The alert resolves on every satellite, the curl loop prints only 200, and the satellite agents stop restarting. Rollback. As for FleetManagerApiErrorRateHigh when a Fleet Manager release caused it. A flapping VIP is not a Fleet Manager change and has nothing to roll back.

​
FleetManagerHubProbeMissing

Symptoms. A satellite’s hub probe reported in the last day and has not reported for 15 minutes, so FleetManagerHubApiUnreachable cannot fire for it. A TargetDown on job fleet-manager-satellite-hub-probe means scraping the exporter’s /probe endpoint fails, for example an unknown module or an exporter that does not answer. It does not mean the hub is down: a failing hub probe still scrapes cleanly and reports probe_success 0. The warning clears a day after the probe stopped, fixed or not. Blast radius. No outage. That satellite can no longer report a hub outage; the others still can. Confirm. The probe runs through the observability blackbox exporter: kubectl --context <context> -n prometheus-blackbox-exporter get service,pods. It must have a prometheus-blackbox-exporter Service with port http and the http_2xx module. Recovery procedure. Escalate to the Observability Squad: the exporter is theirs (charts/observability/templates/otel/agent/applications/blackbox-exporter-application.yaml in nscaledev/k8s-deploy-observability). On fleet-mgr-mgt-sys1-lon14 and fleet-mgr-sys2-sta1 it was running without an owning Argo application as of October 2026, so it may have been removed as unused. Validation. In metrics-all, probe_success{cluster="<cluster>",job="fleet-manager-satellite-hub-probe"} reports again and the warning resolves. Rollback. None — nothing in Fleet Manager changed.

​
FleetManagerOpenStackStateStale

Symptoms. A satellite’s OpenStack agent is restart-looping (3 or more restarts in 30 minutes), or has not listed Ironic nodes for more than 10 minutes. Dashboard panel OpenStack agent: Ironic reflector age — a restart loop shows as a sawtooth that never settles under a minute. The alert’s cluster label names the site. Blast radius. Every device at that site. Fleet Manager keeps its last-known operational_status, provision state and conditions, so quarantines, failures and allocations that happen in Ironic are not reflected. Workflows that preflight on Fleet Manager state act on stale data. Other sites are unaffected. Confirm.
  1. Agent restarts: kubectl --context <context> -n fleet-manager get pods -l app.kubernetes.io/component=openstack-agent (RESTARTS column), then kubectl --context <context> -n fleet-manager logs <pod> --previous for why the last run exited.
  2. Agent errors: {cluster="<cluster>",namespace="fleet-manager",container="openstack-agent"} |= "ERROR" in product-loki. Authentication or connection errors point at Ironic/Keystone, not the agent.
  3. Ironic reachable: eval "$(fleetctl openstack auth --deployment <deployment> --env prod)", then openstack baremetal node list --limit 1.
Recovery procedure.
  1. Ironic or Keystone unreachable or rejecting credentials → escalate to the site’s OpenStack owner (Compute - OpenStack/Proxmox); the agent recovers by itself once Ironic answers.
  2. Agent pod crash-looping or not running → escalate to Fleet Manager with the pod events.
  3. Agent running, no errors, reflector still not listing → a Fleet Manager engineer restarts it: kubectl --context <context> -n fleet-manager rollout restart deployment/fleet-manager-satellite-openstack-agent.
Validation. The alert resolves; the pod’s RESTARTS count stops climbing; the site’s line on Ironic reflector age stays under a minute; a recently changed node shows its new state in fleetctl api device list --name-exact <node> --format json. Rollback. As for FleetManagerApiErrorRateHigh when a Fleet Manager release caused it.

​
FleetManagerWorkflowAgentStale

Symptoms. A satellite’s workflow agent is restart-looping (3 or more restarts in 30 minutes), or has not listed jobs for more than 10 minutes. Dashboard panel Workflow agent: reflector age. If every satellite fires at once, the cause is the hub API. Blast radius. New jobs for that site are not submitted to Temporal, and running jobs stop reporting progress, so job and device status freeze. Work already running inside Temporal continues. Confirm.
  1. Is the hub API healthy? If an API alert is firing, work that first — the workflow agent reads jobs from the API.
  2. Agent restarts: kubectl --context <context> -n fleet-manager get pods -l app.kubernetes.io/component=workflow-agent (RESTARTS column), then kubectl --context <context> -n fleet-manager logs <pod> --previous.
  3. Agent errors: {cluster="<cluster>",namespace="fleet-manager",container="workflow-agent"} |= "ERROR".
Recovery procedure.
  1. Hub API down → recover the API; the agent catches up on its own.
  2. Agent pod not running or crash-looping → escalate to Fleet Manager with the pod events.
  3. Agent running, no errors, reflector not listing → a Fleet Manager engineer restarts it: kubectl --context <context> -n fleet-manager rollout restart deployment/fleet-manager-satellite-workflow-agent.
Validation. The alert resolves; the pod’s RESTARTS count stops climbing; Workflow agent: reflector age stays under a minute; a newly dispatched job at that site leaves pending. Rollback. As for FleetManagerOpenStackStateStale.

​
FleetManagerTemporalWorkerDown

Symptoms. A satellite has no available fleet-manager-satellite-temporal-worker replica for 10 minutes. Dashboard panel Temporal worker: available replicas at 0. Blast radius. No workflow activity runs at that site: enrolment, burn-in, reimage, handover and repair all stop mid-step. Temporal keeps the workflows, so they resume when a worker returns — nothing is lost, but everything waits. Confirm.
  1. kubectl --context <context> -n fleet-manager get deploy,pods -l app.kubernetes.io/component=temporal-worker.
  2. kubectl --context <context> -n fleet-manager describe pod <pod> — look for image pull errors, OOMKilled, or unschedulable events.
  3. Temporal server itself: kubectl --context <context> -n temporal get pods.
Recovery procedure.
  1. Image pull or config error after a rollout → roll back the satellite release (below).
  2. OOMKilled or unschedulable → escalate to Fleet Manager with the pod events; capacity or limits need an engineer.
  3. Temporal server pods unhealthy → escalate to Fleet Manager; workers cannot connect until the server recovers.
Validation. The alert resolves; available replicas back at the deployment’s desired count; a waiting workflow advances (its job’s status changes in fleetctl api job list). Rollback. As for FleetManagerOpenStackStateStale.

​
FleetManagerMatrixFeedStalled

Symptoms. The oldest undelivered Matrix delivery-queue event is older than 15 minutes, for 10 minutes. Dashboard panel Matrix connector: oldest undelivered event. The alert’s connector label names the instance (matrix-prod for sin1, matrix-wdc1-prod for wdc1). Blast radius. Microsoft’s Matrix inventory in Kusto is behind the contracted 5-minute freshness SLA — this is customer-visible for that instance. Other instances are independent. Confirm, recover, validate, roll back. Follow Matrix Connector Operations: its operator health check confirms the breach, and the engineer error classes cover causes and fixes. If the hub API is also down, recover the API first — the connector reads from it. Customer communication. Do not contact the customer directly. Fleet Manager decides when Microsoft is told, per the escalation order.

​
Escalation

​
References