Skip to main content
  • Kind: Observation
  • Version: 2
  • Fleet Manager owner: Fleet Manager squad, #fleet-manager-chat (individual claimant pending)
  • Counterpart owner: unassigned, HPC (Infrastructure and Systems Engineering)
  • Last reviewed: 2026-09-16

​
What flows

The per-satellite slurm-agent (src/fleet_manager/agents/slurm/, ADR-0036) polls the SLURM REST API every 60 s and the Fleet Manager device list every 5 min, resyncing every 15 min, and writes the SlurmMember and SlurmSchedulable conditions. Fleet Manager workflows adopt, release, drain and resume nodes through the same API and submit burn-in jobs as sbatch. A second path runs the other way. Fleet Manager is the authority for the per-site, per-device-role SLURM provisioning configuration (SlurmBootstrapConfig, ADR-0035): the Glance image, Nova flavor, SLURM features, GRES and base64 cloud-init user-data a compute node is built from. HPC operations write it through POST and PATCH /api/v1/sites/{site_id}/slurm-bootstrap-configs/ with the slurm-bootstrap-config:create|update scopes; no agent holds those scopes, so the writer is a person or HPC tooling on a user login. The satellite Temporal worker reads it lazily at adoption time (src/fleet_manager/core/clients/slurm_bootstrap_config.py, src/fleet_manager/core/domain/slurm_adoption.py) and passes image, flavor and user-data to the Nova create or in-place rebuild that turns a device into a SLURM node; burn-in tests then run on nodes built from that configuration. A row with a null image or flavor fails adoption for that role.

​
What breaks if this changes

If the SLURM API or node naming changes, devices show validation while the scheduler cannot see them (the gap ADR-0024 was written for), burn-in dispatch fails, and drain and resume workflows stall. A stale bootstrap configuration breaks adoption for every node at that site.

​
Enforcement

  • Principal: slurm-agent-<site> and temporal-worker-<site> towards Fleet Manager; HPC user logins for the bootstrap configuration writes.
  • Mechanism: towards Fleet Manager, FM OAuth client (private-key JWT) in the hub chart satellite roster; towards SLURM, the SLURM REST JWT read from the 1Password item named by jwt_op_item (src/fleet_manager/settings/clients/slurm.py).
  • Satellite roster client slurm-agent-<site> (device:update, device-condition:create) and temporal-worker-<site> (slurm-bootstrap-config:read) in deployment/charts/hub/values.yaml; the create and update scopes are held by HPC users, not by any roster client.
  • Settings: src/fleet_manager/settings/agents/slurm.py; workflows under src/fleet_manager/temporal/.

​
Gaps

  • Burn-in test authoring and thresholds are moving to FOE; the SLURM cluster itself is operated by HPC. Who answers for a SLURM-side change that breaks Fleet Manager dispatch is not written down.

​
Change log

  • 2026-09-16 v2: added the SLURM bootstrap configuration path (HPC writes, Temporal worker reads at adoption).
  • 2026-09-09 v1: created; describes present behaviour.