Workflow State Transitions

Fleet Manager workflows do not form one device state machine by themselves. A workflow records job progress and performs operations against source systems such as NetBox, Ironic, Placement, Nova, SLURM, partner custody systems, or a BMC. Fleet Manager agents and condition derivation then turn those source-system changes into the device fields that users see. For the complete enum reference and condition rule ladder, see Device State Machines. For job and job-device status values, see Status Reference.

The Mental Model

There are three different kinds of state in play:

State	What it answers	Main writer
`Job.status`	Did this workflow run finish, fail, or keep running?	Workflow agent
`JobDeviceStatus.status`	What happened for this device inside that job?	Workflow agent or batched workflow progress handler
Device status fields	Where is the device in inventory, customer readiness, and health?	NetBox agent, Ironic/SLURM conditions, health agent, custody connector/workflows

Do not infer device readiness from job success alone. A workflow can complete successfully and still leave the device in quarantine because a separate condition is decisive, for example an Ironic maintenance flag, a missing SLURM node, a failed provision state, or a custody hold.

Locking And Concurrency

The Workflow.exclusive flag controls device-level conflict checks:

Incoming workflow	Conflict rule
Exclusive	Blocks if any selected device has an active non-read-only workflow.
Non-exclusive	Blocks only if a selected device has an active exclusive workflow.
Non-contending read-only	Skips the conflict check and never blocks other workflows.

Current non-contending workflows are report generation and firmware inventory. Burn-in and connectivity verification are non-exclusive, but they still conflict with an active exclusive workflow. A workflow can also hold a device it does not run on by naming it in a parameter (HELD_DEVICE_PARAMETERS in core/workflow/names.py). The node swap holds the spare named in replacement_host_name. While such a job is active, every contending job on the named device is refused, exclusive or not, so an enrolment of a swap’s spare waits until the swap finishes. Names match case-insensitively within the job’s site. Admission also checks and locks the named device as one of the holder’s own devices, so a node swap naming a busy spare is refused, and of two submissions touching the same device at the same moment, the second is refused. Cancellation is a job-level operation. Calling POST /api/v1/jobs/{id}/cancel/ moves the job to cancellation_requested; the workflow agent then dispatches Temporal cancellation for in-flight device workflows and later converges the job and device rows to terminal statuses. Fleet Manager only ever cancels the top-level (per-device or batch) workflow — child workflows, such as a burn-in’s per-test SingleTestWorkflow, are not tracked or cancelled directly. Parents start their children with cancellation_type=ABANDON and parent_close_policy=REQUEST_CANCEL so a cancel terminalizes the parent promptly and then cascades a graceful cancel to the child by workflow id — which reaches the child’s live run even after it has continue_as_new’d, where a run-id-pinned cancel would miss it.

Workflow Groups

Fleet Manager workflows are easier to read by operational intent. Each section below uses the same columns: when to use it, submission restrictions, parameters, and the device-state effect.

Group	Use it for	Workflows
Enrolment	Bringing devices into Fleet Manager, or rolling enrollment back before a corrected re-enrolment.	Enrol hosts, Node Unenrollment
Burn-in testing	Running validation tests and packaging test evidence.	Single-node burn-in, multi-node burn-in, Generate report
Customer release	Moving devices between Nscale test/maintenance control and customer-facing use.	Move to Test Cluster, Release to Customer Pool
In-service ops	Readiness checks, repair actions, firmware observation, and connector-driven maintenance while devices are already in the lifecycle.	Verify host connectivity, Ironic node clean, Gather firmware inventory, custody connector and impairment paths

The API stores workflow inputs in Job.parameters. Some values are operator-supplied; others, such as engagement_mode, are stamped by the workflow agent from site configuration so Temporal replay stays deterministic.

Enrolment

Workflow	When to use	Submit restrictions	Parameters	Device-state effect
Enrol hosts	Bring a NetBox device into Fleet Manager control.	Exclusive. Device inventory must be `active` or `staged`; custody-frozen statuses are blocked; allowed device classes are GPU compute, CPU baremetal, and CPU hypervisor.	`run_inspection` boolean, default `true`; `node_uuid` optional UUID used to pin the recreated Ironic node identity during hardware-swap recovery.	Configures DHCP and BMC, moves NetBox `staged -> active` after BMC configuration, creates or recovers the Ironic node, drives it through inspection and cleaning, and tags it as enrolled. A freshly enrolled node normally remains in `quarantine` until a release path makes it customer-ready.
Node Unenrollment	Roll back enrollment before re-enrolling corrected hardware or metadata, or tear down an onsite device during vendor return or retirement.	Exclusive. Inventory may be `active`, `failed`, `offline`, `rma`, `decommissioning`, or `staged`; `decommissioned` is rejected because offsite hardware cannot pass the BMC preflight. Operational status may be `unknown`, `quarantine`, or `failed`; `validation`, `available`, `in_use`, and custody-frozen statuses are blocked, so a node in the SLURM or customer pool must be released into quarantine first; enrollment-eligible device classes only.	None; parameters are `{}`.	Refuses before any teardown while Nova still holds the node — an Ironic instance binding Nova confirms, or allocations on the node’s Placement resource provider (which survive a failed teardown that cleared the Ironic binding). A device whose move to `staged` takes it out of a topology Region’s discovery (in practice NetBox `active`) then has its removal declared to topology-service through the hub (Sync Intent `unenrollment-<device>-<job id>`), and the job fails as `TOPOLOGY_SYNC_INTENT_FAILED` before any teardown when that cannot be declared. It then deletes the Ironic node and Placement resource provider, re-reads the status and declares then if the device entered discovery during the teardown, moves reusable devices to NetBox `staged` while preserving `rma`, `decommissioning`, or `decommissioned`, re-posts a declared intent to trigger the Region’s topology sync (a failure there only warns), factory-resets the BMC, and removes credentials. Preserving `decommissioned` protects an already-running job if NetBox advances while earlier steps execute. The Ironic deletion reconcile clears infrastructure fields and resets operational status to `unknown`; the NetBox agent mirrors the resulting NetBox inventory status.

Enrollment Completion Proof

A normal Enrollment with run_inspection=true captures the request-time infrastructure identity and establishes or applies Device proof only at terminal completion. It does not need a start-of-work condition transition: workflow exclusivity prevents a concurrent workflow from consuming proof while Enrollment is active, and terminal failure uses the captured identity for a non-revoking guarded failure write. Matching proof is preserved; stale metadata is cleared when the operated identity still matches. Inspection-half success means the configured inspection checks passed against either fresh or cached Ironic inventory. An already available, active, or Nova-owned node is not reinspected because that could disrupt a deployed machine; its cached inventory must still be present and pass the same hardware, link, and cable validation. Missing or invalid cached data fails Enrollment and cannot produce proof. The versioned inspection activity returns the exact resolved Ironic infrastructure_ref, and the workflow returns qualifying proof for that reference only after cleaning and every later configured stage succeeds. A diagnostic Enrollment with run_inspection=false does not validate cached inspection data or return completion proof. It still captures the resolved Ironic reference so a later failure can be finalized against the correct infrastructure identity. At a site with temporalWorker.enrollmentSteps.ironicEnrol=false, Enrollment likewise runs every other enabled stage and returns a diagnostic completion, but skips the entire Ironic stage and retains the request-time identity for failure finalization. Because no Ironic validation runs, it neither invalidates older completion proof at start nor produces new proof on success. See Device State Machines for the proof lifecycle and terminal-write rules.

Burn-In Testing

Workflow	When to use	Submit restrictions	Parameters	Device-state effect
Single-node burn-in	Validate one or more devices with per-device hardware tests.	Non-exclusive. Inventory must be `active`, operational status must be `validation`, `infrastructure_ref` must be present, and Ironic must report `active` with a current instance. Custody-frozen statuses are blocked; compute classes only; at least one device. The worker also verifies that the requested SLURM node exists and is not down before submission.	`tests[]` required list of `{slug, params}` entries; `partition` optional SLURM partition; `max_wait_time_seconds` optional positive wait limit.	Runs hardware tests through SLURM and records test executions, artifacts, and per-device job results. It does not directly change `operational_status` or `health_status`.
Multi-node burn-in (pairwise)	Validate network or fabric paths across node pairs.	Non-exclusive. Same inventory, operational, Ironic-instance, and live SLURM readiness gates as single-node burn-in; compute classes only; at least two devices; all devices must share one resource class.	`test_slug` required pairwise test slug; `test_params` optional test-specific parameter overrides; `partition` optional; `max_wait_time_seconds` optional.	Runs one pairwise network burn-in test and records per-device and per-pair progress/results. It does not directly change device readiness fields.
Multi-node burn-in (collective)	Validate collective communication across a cohort.	Non-exclusive. Same inventory, operational, Ironic-instance, and live SLURM readiness gates as single-node burn-in, but GPU compute only; at least two devices; shared resource class required.	`tests[]` required list of `{slug, params}` entries; duplicate slugs are rejected; `partition` optional; `max_wait_time_seconds` optional.	Runs collective communication tests and records per-device progress/results. It does not directly change device readiness fields.
Generate report	Package historical burn-in/test evidence for selected devices.	Read-only and non-contending. Compute classes only; custody-frozen statuses are blocked. Job creation is routed through report-run APIs, not the generic job endpoint.	Report-run APIs supply `sections` (`inventory`, `enrollment`, `health`, `burn_in`), `filters` (`start_date`, `end_date`, `test_slugs`, `result_filter`), and `generate_logs`. `test_selection_scope`, `org_slug`, and `site_slug` are derived during report-run creation.	Reads burn-in/test data and writes report artifacts. It does not change device state.

Customer Release

Do not use force_reclaim=true under normal conditions. It can take an in-use node away from a customer by deleting the attached customer Nova instance before Fleet Manager moves the device back to quarantine. It is only supported on explicitly enabled fully managed trait-based sites; elsewhere the workflow fails before doing work. Treat it as an emergency/operator-approved reclaim path only.

Workflow	When to use	Submit restrictions	Parameters	Device-state effect
Move to Test Cluster	Pull a device out of customer-facing use and back into Nscale test/maintenance control.	Exclusive. Compute classes only. A fully managed device with no current instance is admitted whenever `inventory_status=active` and an `infrastructure_ref` are present — regardless of `ironic_provision_state`: `manageable`, `error`, `clean failed`, or any other non-`available` shape is accepted, not rejected. Admission gates on instance binding, not provision-state readiness, because the hub’s copy of provision state is an eventually-consistent projection; `adopt_host` re-reads Ironic authoritatively and either recovers the node, defers a still-settling one, or fails naming the real state. With `force_reclaim=true`, an `operational_status=in_use` device with Ironic `active` and a current instance is also eligible; a force-reclaim request against a no-instance node is admitted the same way and fails loudly in the activity if the node isn’t actually `available`. For normal quarantine, `delegated` and `pending_handover` remain exempt because their custody paths do not adopt into Ironic/SLURM; they are rejected when `force_reclaim=true`, which is fully managed only.	`force_reclaim` boolean, default `false`; dangerous operator take-back switch, not a normal workflow option. `force_reclaim=true` only works for fully managed, trait-based sites where force reclaim is enabled. `engagement_mode` is stamped by the workflow agent and selects the fully managed, Metal, or external partner custody path.	For fully managed devices, removes customer scheduling access, optionally force-reclaims an allowed in-use node, then adopts the node into SLURM. The resulting Ironic/SLURM conditions usually derive `validation` when SLURM is usable, or `quarantine` when the node is not yet schedulable. For Metal delegated devices, the workflow performs reclaim and writes `operational_status=quarantine` directly. For external partner custody, the current workflow branch is a no-op for device status.
Release to Customer Pool	Return a device from Nscale test/maintenance control to customer-facing use.	Exclusive. Compute classes only; only devices with `operational_status=quarantine` or `validation` are admitted; devices in triage are blocked. Because admission reads an eventually consistent Fleet Manager projection, every Fleet Manager-owned customer handoff re-reads authoritative NetBox in the same activity and requires `inventory_status=active` before exposing the device.	`release_to_customer_pool` boolean, default `true`. When false, the workflow releases from SLURM but does not return the device to customer scheduling or custody handoff. `engagement_mode` is stamped by the workflow agent.	Releases the node from SLURM. With `release_to_customer_pool=true`, fully managed trait-based sites check NetBox immediately before adding the customer-ready trait, legacy sites check it before clearing the maintenance lessee, and Metal sites check it before writing `operational_status=delegated`. A missing or non-`active` NetBox device fails that customer handoff non-retryably; transient NetBox read failures remain retryable. External partner custody delivery and the `pending_handover` write are connector-owned; the current workflow branch dispatches no activities for that mode. Ironic/Placement reconciliation decides whether fully managed devices appear as `available`, `in_use`, or remain blocked.

In-Service Ops

Workflow or path	When to use	Submit restrictions	Parameters	Device-state effect
Verify host connectivity	Check host connectivity before testing or while diagnosing readiness.	Non-exclusive. Compute classes only; custody-frozen statuses are blocked.	None; parameters are `{}`.	Runs connectivity checks such as the BMC authentication probe. It does not change device status fields; success or failure is reflected in job statuses and failure details.
Ironic node clean	Recover an Ironic node from failed cleaning or manageable state.	Exclusive. Inventory must be `active`; operational status must be `failed` or `drained`; custody-frozen statuses are blocked; enrollment-eligible device classes only.	None; parameters are `{}`.	Clears Ironic maintenance if needed and re-runs the Ironic clean path from `clean failed` or `manageable` back to `available`. The next reconcile updates failure fields and derives operational status from the new Ironic state.
Gather firmware inventory	Read observed firmware from one rack of hosts.	Read-only and non-contending. Compute classes only; custody-frozen statuses are blocked; selected devices must be in one rack. In-band collection requires a mapped resource class and an enabled satellite worker.	`include_in_band` boolean, default `false`. Set it to `true` to add the bounded SLURM collector.	Scans BMC firmware inventory. With `include_in_band=true`, it also collects CX7, CX8, BF3, and NVMe host-sysfs evidence through SLURM. Fleet Manager evaluates collection completeness against the mapped device resource class and refuses unmapped classes before submission. Collection failures affect per-device job status, not device operational-status fields.
External partner recovery	External partner connector detects a returned device after break-fix.	Automatic connector path, not a normal operator-submitted job.	Connector-supplied.	The connector writes updated credentials, starts the workflow idempotently, then advances the device to `operational_status=quarantine` so normal lifecycle reconciliation can resume.
Custody connector	External partner custody state changes.	Automatic connector path, not a normal operator-submitted job.	Connector-supplied.	Writes `pending_handover`, `delegated`, `drained`, or `quarantine` according to partner custody state. These writes are connector-owned, not derived from Ironic.
Impairment-event handling	Impairment events created through the impairment-event API. Legacy, with no known caller; Radar does not call it.	Not operator-submitted.	API payload.	Creating an event sets an `impairment-event-id` hold annotation and `operational_status=quarantine`; the OpenStack agent’s cordon removes the customer-ready trait. Clearing the event queues release work that restores the trait and the pre-hold status. Ironic maintenance is not set.

In-band firmware collection

Deploy compatible satellite workers before the hub applies admission migrations 0202 and 0203. Confirm that no old Temporal worker remains. Before enabling temporalWorker.hardwareTests.firmwareInventoryInBandEnabled (default false) at a site:

Confirm site readiness, SLURM node access, and resource-class mappings. This release supports GB300 and B200_8NIC. Unmapped classes fail before SLURM dispatch.
Verify that the satellite worker’s credentials can PUT and GET objects in its configured Fleet Manager bucket. The worker checks storage access at startup.
Set the flag to true and confirm that the new worker starts successfully.

The flag blocks new SLURM dispatch, but it does not block Job creation. Keep include_in_band=false until these checks pass. Disabling the flag preserves status, result, and cancellation activities for collectors already submitted. Both admission migrations are irreversible. Before a worker rollback, disable the flag and deploy a forward repair that closes include_in_band and dry_run admission. Remove these properties from stored workflow schemas and new-organisation defaults; keep additionalProperties=false. Drain or cancel incompatible Jobs before starting old workers. Reversing the migrations does not close admission.

Readable Split Transition Views

These views summarize the workflow and connector paths that commonly move devices between operational statuses. The canonical operational-status derivation remains Device State Machines. Each diagram stays focused on one lifecycle, while the aggregate map and table connect the paths back to workflows and parameters. The actual fully managed derivation is condition-based and ordered: Ironic failure, maintenance, enrollment state, validation allocation, customer allocation, customer-ready traits, and SLURM membership/schedulability all contribute. Custody states are connector/workflow-owned. If a diagram and the condition details appear to disagree, trust the conditions shown on the device detail page.

Aggregate Status Map

Use this as the quick orientation view. It shows which operational statuses can connect without trying to carry every workflow parameter on the graph.

Transition	Driver	Parameters that choose this path
`unknown -> quarantine`	Enrol hosts	None; `run_inspection` and `node_uuid` do not choose the resulting operational status.
`unknown -> failed`	Ironic/condition reconciliation	Not workflow-driven.
`quarantine -> validation`	Move to Test Cluster	`engagement_mode=fully_managed`; final status still depends on SLURM/Ironic conditions.
`validation -> quarantine`	Release to Customer Pool	`release_to_customer_pool=false`.
`validation -> available`, `validation -> in_use`	Release to Customer Pool	`release_to_customer_pool=true`; final status depends on customer-ready and allocation signals.
`quarantine -> available`, `quarantine -> in_use`	Release to Customer Pool	`release_to_customer_pool=true`; final status depends on customer-ready and allocation signals.
`available <-> in_use`	Ironic/Placement/Nova reconciliation	Not workflow-driven.
`available -> quarantine`	Move to Test Cluster	`engagement_mode=fully_managed`.
`in_use -> quarantine`	Move to Test Cluster	`force_reclaim=true`, only for fully managed, trait-based sites where force reclaim is enabled.
`quarantine -> delegated`	Release to Customer Pool	`engagement_mode=metal_delegated`, `release_to_customer_pool=true`.
`delegated -> quarantine`	Move to Test Cluster or external partner recovery	`engagement_mode=metal_delegated` for Metal reclaim; external partner recovery is connector-supplied.
`available -> pending_handover`	Custody connector	External partner outbound delivery writes `pending_handover`; the current release workflow branch for this mode dispatches no activities.
`pending_handover -> delegated`, `delegated -> drained`, `drained -> delegated`, `drained -> quarantine`	Custody connector	Connector-supplied.
`available -> failed`, `in_use -> failed`, `validation -> failed`, `quarantine -> failed`	Ironic/condition reconciliation	Not workflow-driven.
`failed -> quarantine`	Ironic node clean	None.

Fully Managed Lifecycle

The in_use -> quarantine path uses force_reclaim=true. It is dangerous because it can take a node from a customer and should not be used under normal conditions. It is only supported for fully managed, trait-based sites where force reclaim is enabled.

Metal Delegated Lifecycle

External Partner Custody Lifecycle

Failure And Repair Overlay

These failure edges are condition-derived, not normal workflow requests. They appear when the infrastructure state reports a failed provision, cleaning, or maintenance outcome; Ironic node clean is the workflow that can recover the device back to quarantine.

Inventory And Health Are Separate

Workflows that affect operational_status usually do not affect inventory_status or health_status.

Axis	Workflow relationship
`inventory_status`	Mirrored primarily from NetBox. Enrollment can move NetBox `staged -> active`; unenrollment moves reusable states back to `staged`. Customer-facing release branches re-read NetBox authoritatively inside their final handoff activity and require `active`; this prevents a lifecycle change such as `active -> rma` after workflow admission from exposing the device.
`operational_status`	Derived from Ironic/Placement/SLURM conditions, except custody workflows/connectors that write custody states directly.
`health_status`	Written by the health-status agent from telemetry. Workflows do not make a device healthy by completing.
`device_conditions`	Latest observations that explain why operational status was derived. Conditions are current-state observations, not workflow history.
`in_triage`	Manual flag. Release workflows opt in to blocking triaged devices; most workflows do not.

Why A Successful Workflow May Leave A Device Blocked

Common cases:

Symptom	Usual explanation
Move to Test Cluster succeeded, but device is `quarantine` instead of `validation`.	The workflow completed its actions, but SLURM conditions report the node missing or unschedulable, or Ironic reports maintenance/failure.
Release succeeded, but device is not `available`.	The customer-ready trait or allocation signal has not appeared, the node still has a blocking condition, or the device is in a custody state.
Burn-in passed, but `health_status` is still `degraded` or `unknown`.	Burn-in results and health telemetry are separate. The health agent updates health from metrics.
Device is `delegated` or `pending_handover`, and most workflows are ineligible.	Custody-owned states freeze normal lifecycle reconciliation and are blocked by most eligibility policies. Metal delegated reclaim is workflow-owned; external partner recovery is connector-owned.
Job is terminal but a device state changed later.	Job status is workflow history. Device status is current state and can change after the workflow because agents continue reconciling external systems.

What To Check After Running A Workflow

Check Job.status for the aggregate outcome.
Check JobDeviceStatus.status, failure_code, phase, and failure_message for per-device detail.
Open the device and inspect operational_status, health_status, and operational_status_conditions.
If the device is ineligible for a next workflow, call POST /api/v1/eligibility/check/ with the target workflow and device set; the response gives structured reasons.
For burn-in workflows, use the job Results tab and the Burn-in Testing Guide rather than treating device status as test evidence.

Start Here

Operator Guide

Developer Guide

Relationships

Deployment Guide

Archived

API Reference

Workflow State Transitions

The Mental Model

Locking And Concurrency

Workflow Groups

Enrolment

Enrollment Completion Proof

Burn-In Testing

Customer Release

In-Service Ops

In-band firmware collection

Readable Split Transition Views

Aggregate Status Map

Fully Managed Lifecycle

Metal Delegated Lifecycle

External Partner Custody Lifecycle

Failure And Repair Overlay

Inventory And Health Are Separate

Why A Successful Workflow May Leave A Device Blocked

What To Check After Running A Workflow

​The Mental Model

​Locking And Concurrency

​Workflow Groups

​Enrolment

​Enrollment Completion Proof

​Burn-In Testing

​Customer Release

​In-Service Ops

​In-band firmware collection

​Readable Split Transition Views

​Aggregate Status Map

​Fully Managed Lifecycle

​Metal Delegated Lifecycle

​External Partner Custody Lifecycle

​Failure And Repair Overlay

​Inventory And Health Are Separate

​Why A Successful Workflow May Leave A Device Blocked

​What To Check After Running A Workflow

The Mental Model

Locking And Concurrency

Workflow Groups

Enrolment

Enrollment Completion Proof

Burn-In Testing

Customer Release

In-Service Ops

In-band firmware collection

Readable Split Transition Views

Aggregate Status Map

Fully Managed Lifecycle

Metal Delegated Lifecycle

External Partner Custody Lifecycle

Failure And Repair Overlay

Inventory And Health Are Separate

Why A Successful Workflow May Leave A Device Blocked

What To Check After Running A Workflow