> ## Documentation Index > Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt > Use this file to discover all available pages before exploring further. # Status Reference > The job-level status values Fleet Manager exposes, and what each one means. Fleet Manager tracks two job-level status concepts that operators routinely confuse, so they are defined separately here. For the full [device](../start-here/glossary#node) lifecycle — inventory, operational, health, [Ironic](../start-here/glossary#ironic), [custody](../start-here/glossary#customer-custody), and state diagrams — see [Device State Machines](../developer-guide/device-state-machines). * **Job status** lives on a *job* (a single workflow execution across one or more devices). * **Job device status** lives on the *join* between a job and a single device, and describes how that device's execution went within that job. Operational Racks derives a separate five-value cell status from this history. A raw cancelled, terminated, or superseded execution remains visible without owning the cell's placement or status. See [DC Standup Pipeline](./dc-standup) for that operator-facing logic. ## Job Status The outcome of a single workflow execution, summarised across every device it touched. | Value | Meaning | | - | - | | `pending` | Initial state — job is awaiting [Temporal](../start-here/glossary#temporal) submission. | | `running` | Corresponds to the Temporal workflow running state. | | `cancellation_requested` | Cancellation was requested by a user; the Temporal cancellation may still be in flight. | | `completed` | Job completed successfully on every device. | | `partially_failed` | At least one device succeeded and at least one device failed. | | `failed` | Job failed. | | `cancelled` | Job was cancelled. | | `terminated` | Job was terminated. | | `superseded` | Job was superseded by another job. | | `timed_out` | Job execution timed out. | Terminal statuses (job will not transition further): `completed`, `failed`, `partially_failed`, `cancelled`, `terminated`, `superseded`, `timed_out`. For `NVSwitchFirmwarePolicyVerificationWorkflow`, completion means the frozen plan check succeeds. The archived plan-check workflow collects no hardware evidence and makes no compliance claim. Its result contains `PLAN_CHECKED` and the snapshot ID. Evidence collection requires later release gates. ### Cancelling a job Cancellation is asynchronous, and for a [burn-in](../start-here/glossary#burn-in) it does **not** stop the [Slurm](../start-here/glossary#slurm) job at the moment the job reads `cancelled`. `POST /api/v1/jobs/{id}/cancel/` moves the job to `cancellation_requested`; the workflow agent then cancels the parent workflow. The parent's wait on its child raises at once, the parent unwinds and closes `cancelled` (or `terminated`) — and only *then* does `parent_close_policy=REQUEST_CANCEL` reach the child, which `scancel`s its Slurm job. That release finishes shortly after, so **`cancelled` does not prove the nodes are free**. Because of that ordering, do not treat a `cancelled` burn-in job as "devices released" — a rerun, release, or requalification submitted straight after a cancel can race a still-live allocation. To confirm the allocation is actually gone, check that the child workflow `{job_id}-{test_slug}-{device_id}` closed `CANCELLED` (or run `squeue` on the [cluster](../start-here/glossary#cluster)). If the child's cleanup gave up — a second cancel stops cleanup rather than renewing it — the allocation can leak with the job already terminal, and the node must be `scancel`led manually. If a job stays in `cancellation_requested` well beyond that, the usual cause is a burn-in that was **already running when a Fleet Manager release deployed**. A workflow's cancellation behaviour is fixed when a test execution starts, so an execution that began under an older release can ignore a cancel until it finishes on its own — for burn-in that can be most of a day. Force-terminating the workflow in Temporal is the workaround, and it is the escalation path for any cancel that does not take effect. Suspect this for jobs created before the most recent release that are still `running` or `cancellation_requested`. ## Job Device Status The per-device execution status within a job. Open the job's Devices tab to see one of these per device. A row is created for every selected device as soon as the job is submitted; the status is empty until Temporal begins execution for that device, at which point it transitions to `running`. There is no explicit `pending` value at the device level — an empty status means the device is still waiting to start. | Value | Meaning | | - | - | | `running` | Corresponds to the Temporal workflow running state for this device. | | `completed` | Device execution completed successfully. | | `failed` | Device execution failed. | | `cancelled` | Device execution was cancelled. | | `terminated` | Device execution was terminated. | | `superseded` | Device execution was superseded. | | `timed_out` | Device execution timed out. | Terminal statuses: `completed`, `failed`, `cancelled`, `terminated`, `superseded`, `timed_out`. For failed device executions, `failure_code` is the stable diagnostic category and `failure_message` contains host-specific detail. `IRONIC_INSPECTION_IPA_IMAGE_NOT_FOUND` indicates that the [IPA](../start-here/glossary#ironic-python-agent) kernel or ramdisk chosen for Ironic inspection was not present; validate architecture selection, node overrides, and artifacts on every conductor before retrying a single node. Follow `docs/runbooks/ipa-image-update.md` for the full diagnostic and artifact-restaging procedure. `IRONIC_DRIVER_MISMATCH` indicates that an Ironic power cycle targeted a node whose driver is not `ipmi`; switch the node's Ironic driver before retrying. ## Node Diagnostics Summary A Node Diagnostics job execution carries an extra field, `node_diagnostics`. It is null for every other workflow, and for a Node Diagnostics run that failed before it could classify the failure. When present it holds: | Field | Meaning | | - | - | | `failure_signature` | Why the Ironic clean failed: `clean_failed.neutron_ports`, `clean_failed.bmc_power_off`, `clean_failed.virtual_media`, or `clean_failed.other`. Always present when the summary is. | | `findings` | Every typed conclusion the run drew, including `clean_failed.recurring` when the node has failed this way before. | | `collectors` | One entry per collector the run executed, with its outcome. | | `artifact` | Where the run's evidence bundle lives in object storage. | `failure_signature` says why *Ironic* failed. It is not `failure_code`, which says why the Fleet Manager workflow failed. A run can carry both. ### Collector Status | Value | Meaning | | - | - | | `completed` | The collector read its source and wrote its evidence. Only a completed collector with no findings passed. | | `error` | The collector could not read its source. It reports no findings and its manifest entry has no object. | | `unsupported` | The source cannot answer on this device or API version. Not a fault. | The `ironic-node` collector is required: if it cannot read Ironic, the job fails with `NODE_DIAGNOSTICS_CLASSIFICATION_FAILED` rather than completing without a signature. `NODE_DIAGNOSTICS_UNEXPECTED_PROVISION_STATE` means the device was not in `clean failed` when the run started — check the device's current Ironic state before re-dispatching. `NODE_DIAGNOSTICS_STATE_CHANGED` means the node was re-cleaned while the run was collecting, so the evidence would have described a different failure; re-run the diagnostics against the current one. ### Downloading the evidence bundle Every file of the bundle is fetched through the job execution: ``` GET /api/v1/job-device-statuses/{id}/artifacts/manifest.json GET /api/v1/job-device-statuses/{id}/artifacts/evidence-ironic-node.json ``` Each returns `{status, url, expires_at, filename, content_type}`. A `status` of `available` carries a short-lived presigned `url`; `unavailable` means the run has not written that file, or object storage could not be read. Read `manifest.json` first: it is written last, so its presence is what tells you the bundle is complete. ## Job Name vs Job ID A job has two identifiers and they are not interchangeable: * **Job name** — the human-readable name shown in the Fleet Manager UI. * **Job ID** — the unique identifier used by Temporal, Loki, and the Fleet Manager API. Use the job name to find a job in the UI. Use the job ID when pasting into Temporal, building a LogQL query, or hitting the API directly. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.