Skip to main content
Fleet Manager tracks two job-level status concepts that operators routinely confuse, so they are defined separately here. For the full device lifecycle — inventory, operational, health, Ironic, custody, and state diagrams — see Device State Machines.
  • Job status lives on a job (a single workflow execution across one or more devices).
  • Job device status lives on the join between a job and a single device, and describes how that device’s execution went within that job.
Operational Racks derives a separate five-value cell status from this history. A raw cancelled, terminated, or superseded execution remains visible without owning the cell’s placement or status. See DC Standup Pipeline for that operator-facing logic.

​
Job Status

The outcome of a single workflow execution, summarised across every device it touched. Terminal statuses (job will not transition further): completed, failed, partially_failed, cancelled, terminated, superseded, timed_out. For NVSwitchFirmwarePolicyVerificationWorkflow, completion means the frozen plan check succeeds. The archived plan-check workflow collects no hardware evidence and makes no compliance claim. Its result contains PLAN_CHECKED and the snapshot ID. Evidence collection requires later release gates.

​
Cancelling a job

Cancellation is asynchronous, and for a burn-in it does not stop the Slurm job at the moment the job reads cancelled. POST /api/v1/jobs/{id}/cancel/ moves the job to cancellation_requested; the workflow agent then cancels the parent workflow. The parent’s wait on its child raises at once, the parent unwinds and closes cancelled (or terminated) — and only then does parent_close_policy=REQUEST_CANCEL reach the child, which scancels its Slurm job. That release finishes shortly after, so cancelled does not prove the nodes are free. Because of that ordering, do not treat a cancelled burn-in job as “devices released” — a rerun, release, or requalification submitted straight after a cancel can race a still-live allocation. To confirm the allocation is actually gone, check that the child workflow {job_id}-{test_slug}-{device_id} closed CANCELLED (or run squeue on the cluster). If the child’s cleanup gave up — a second cancel stops cleanup rather than renewing it — the allocation can leak with the job already terminal, and the node must be scancelled manually. If a job stays in cancellation_requested well beyond that, the usual cause is a burn-in that was already running when a Fleet Manager release deployed. A workflow’s cancellation behaviour is fixed when a test execution starts, so an execution that began under an older release can ignore a cancel until it finishes on its own — for burn-in that can be most of a day. Force-terminating the workflow in Temporal is the workaround, and it is the escalation path for any cancel that does not take effect. Suspect this for jobs created before the most recent release that are still running or cancellation_requested.

​
Job Device Status

The per-device execution status within a job. Open the job’s Devices tab to see one of these per device. A row is created for every selected device as soon as the job is submitted; the status is empty until Temporal begins execution for that device, at which point it transitions to running. There is no explicit pending value at the device level — an empty status means the device is still waiting to start. Terminal statuses: completed, failed, cancelled, terminated, superseded, timed_out. For failed device executions, failure_code is the stable diagnostic category and failure_message contains host-specific detail. IRONIC_INSPECTION_IPA_IMAGE_NOT_FOUND indicates that the IPA kernel or ramdisk chosen for Ironic inspection was not present; validate architecture selection, node overrides, and artifacts on every conductor before retrying a single node. Follow docs/runbooks/ipa-image-update.md for the full diagnostic and artifact-restaging procedure. IRONIC_DRIVER_MISMATCH indicates that an Ironic power cycle targeted a node whose driver is not ipmi; switch the node’s Ironic driver before retrying.

​
Node Diagnostics Summary

A Node Diagnostics job execution carries an extra field, node_diagnostics. It is null for every other workflow, and for a Node Diagnostics run that failed before it could classify the failure. When present it holds: failure_signature says why Ironic failed. It is not failure_code, which says why the Fleet Manager workflow failed. A run can carry both.

​
Collector Status

The ironic-node collector is required: if it cannot read Ironic, the job fails with NODE_DIAGNOSTICS_CLASSIFICATION_FAILED rather than completing without a signature. NODE_DIAGNOSTICS_UNEXPECTED_PROVISION_STATE means the device was not in clean failed when the run started — check the device’s current Ironic state before re-dispatching. NODE_DIAGNOSTICS_STATE_CHANGED means the node was re-cleaned while the run was collecting, so the evidence would have described a different failure; re-run the diagnostics against the current one.

​
Downloading the evidence bundle

Every file of the bundle is fetched through the job execution:
Each returns {status, url, expires_at, filename, content_type}. A status of available carries a short-lived presigned url; unavailable means the run has not written that file, or object storage could not be read. Read manifest.json first: it is written last, so its presence is what tells you the bundle is complete.

​
Job Name vs Job ID

A job has two identifiers and they are not interchangeable:
  • Job name — the human-readable name shown in the Fleet Manager UI.
  • Job ID — the unique identifier used by Temporal, Loki, and the Fleet Manager API.
Use the job name to find a job in the UI. Use the job ID when pasting into Temporal, building a LogQL query, or hitting the API directly.