> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Relationship: SLURM

> SLURM as the scheduler Fleet Manager observes and adopts nodes into.

* **Kind:** Observation
* **Version:** 2
* **Fleet Manager owner:** Fleet Manager squad, #fleet-manager-chat (individual claimant pending)
* **Counterpart owner:** unassigned, HPC (Infrastructure and Systems Engineering)
* **Last reviewed:** 2026-09-16

## What flows

The per-satellite `slurm-agent` (`src/fleet_manager/agents/slurm/`, ADR-0036) polls the [SLURM](../start-here/glossary#slurm) REST API every
60 s and the Fleet Manager [device](../start-here/glossary#node) list every 5 min, resyncing every 15 min, and writes the `SlurmMember` and
`SlurmSchedulable` conditions. Fleet Manager workflows adopt, release, drain and resume nodes through the same
API and submit [burn-in](../start-here/glossary#burn-in) jobs as sbatch.

A second path runs the other way. Fleet Manager is the authority for the per-site, per-device-role SLURM
provisioning configuration (`SlurmBootstrapConfig`, ADR-0035): the Glance image, [Nova](../start-here/glossary#nova) flavor, SLURM features, GRES
and base64 cloud-init user-data a compute node is built from. HPC operations write it through
`POST` and `PATCH /api/v1/sites/{site_id}/slurm-bootstrap-configs/` with the `slurm-bootstrap-config:create|update`
scopes; no agent holds those scopes, so the writer is a person or HPC tooling on a user login. The satellite [Temporal](../start-here/glossary#temporal)
worker reads it lazily at adoption time (`src/fleet_manager/core/clients/slurm_bootstrap_config.py`,
`src/fleet_manager/core/domain/slurm_adoption.py`) and passes image, flavor and user-data to the Nova create or
in-place rebuild that turns a device into a SLURM node; burn-in tests then run on nodes built from that
configuration. A row with a null image or flavor fails adoption for that role.

## What breaks if this changes

If the SLURM API or node naming changes, devices show `validation` while the scheduler cannot see them (the
gap ADR-0024 was written for), burn-in dispatch fails, and drain and resume workflows stall. A stale
bootstrap configuration breaks adoption for every node at that site.

## Enforcement

* **Principal:** `slurm-agent-<site>` and `temporal-worker-<site>` towards Fleet Manager; HPC user logins for the
  bootstrap configuration writes.
* **Mechanism:** towards Fleet Manager, FM OAuth client (private-key JWT) in the hub chart satellite roster; towards
  SLURM, the SLURM REST JWT read from the [1Password](../start-here/glossary#1password) item named by `jwt_op_item`
  (`src/fleet_manager/settings/clients/slurm.py`).
* Satellite roster client `slurm-agent-<site>` (`device:update`, `device-condition:create`) and `temporal-worker-<site>`
  (`slurm-bootstrap-config:read`) in `deployment/charts/hub/values.yaml`; the create and update scopes are held by
  HPC users, not by any roster client.
* Settings: `src/fleet_manager/settings/agents/slurm.py`; workflows under `src/fleet_manager/temporal/`.

## Gaps

* Burn-in test authoring and thresholds are moving to FOE; the SLURM [cluster](../start-here/glossary#cluster) itself is operated by HPC. Who
  answers for a SLURM-side change that breaks Fleet Manager dispatch is not written down.

## Change log

* 2026-09-16 v2: added the SLURM bootstrap configuration path (HPC writes, Temporal worker reads at adoption).
* 2026-09-09 v1: created; describes present behaviour.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.