> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Device State Machines

> All status axes Fleet Manager tracks on a device, with state diagrams and transition notes.

Fleet Manager does **not** use a single monolithic state machine. A **[device](../start-here/glossary#node)** carries
several **orthogonal status axes** plus **raw subsystem mirrors** that record what external
systems report. This page lists every enum state and the main transitions between them.

For a focused job-status reference (without the device axes below), see
[Status Reference](../operator-guide/status-reference). This page also documents job and
job-device statuses in full.

For the workflow-facing view of how current workflows move devices between these axes, see
[Workflow State Transitions](./workflow-state-transitions).

<Warning>
  Naming collision: ADR-0003 uses "operational status" for the [NetBox](../start-here/glossary#netbox) lifecycle enum
  (`planned`, `staged`, `active`, …). In the Fleet Manager API and UI, operational status
  means customer-pool readiness (`quarantine`, `available`, `in_use`, …). These are
  different concepts. This page uses the API field names.
</Warning>

## Device composite model

```mermaid theme={null}
flowchart TB
    subgraph Device["Device (Fleet Manager)"]
        INV["inventory_status — NetBox lifecycle"]
        OPS["operational_status — customer-pool readiness"]
        HLTH["health_status — Mimir telemetry"]
        COND["device_conditions[] — multi-source observations"]
        TRIAGE["in_triage (boolean)"]
    end

    subgraph Raw["Raw subsystem fields (mirrors, not derived)"]
        IRONIC["ironic_provision_state"]
        SLURM["slurm_states[]"]
        PLACE["placement_traits[]"]
        MAINT["ironic_maintenance + reason + fault"]
    end

    ATTR["resource_binding.resolution_status — UNI attribution"]

    subgraph Sources["Integration writers"]
        NB["NetBox agent → inventory"]
        IAG["OpenStack agent → conditions + raw Ironic/Placement"]
        SLURM_SRC["Slurm agent → SLURM conditions + slurm_states"]
        HAG["Health-status agent → health_status + hardware-health conditions"]
        CUST["Google connector → operational_status (CustodyIntent planned)"]
        IMPAIR["Impairment-event API (legacy, no known caller) → hold + quarantine"]
        RADAR_OUT["Radar ingestion agent → customer projection (read FM)"]
        WF["Workflow agent → job statuses"]
        OPS_UI["Operators → triage API"]
    end

    NB --> INV
    IAG --> COND
    IAG --> OPS
    IAG --> IRONIC
    IAG --> MAINT
    IAG --> PLACE
    IAG --> ATTR
    SLURM_SRC --> COND
    SLURM_SRC --> SLURM
    HAG --> HLTH
    CUST --> OPS
    OPS_UI --> TRIAGE
    WF --> JobStatuses["Job / JobDeviceStatus"]
    ATTR --> RADAR_OUT
```

## UNI resource attribution

`DeviceResourceBinding.resolution_status` is an independent, observable axis for
materialised attribution to UNI, [Placement](../start-here/glossary#placement), and Kubernetes resources. It does
not replace the device's operational or health status.

| State | Meaning |
| - | - |
| `pending` | Reconciliation has not completed successfully yet. |
| `unassigned` | The device is not currently assigned to a [Nova](../start-here/glossary#nova) server. |
| `resolved` | All expected attribution resources were resolved. |
| `partially_resolved` | Some attribution resources were resolved, but at least one was not. |
| `unresolved` | Reconciliation completed without resolving the attribution. |
| `stale` | The last resolved observation exceeded the 15-minute freshness window. |

Each typed resource exposes its own `status` (`resolved` or `stale`) and
freshness timestamps. A failed resource retains its last-known row as `stale`,
while fresh resources remain `resolved`; readers derive stale state from the
persisted timestamps at read time.

## Inventory status

**Field:** `Device.inventory_status` — NetBox lifecycle (ADR-0003: Host Statuses).

| Value | Label |
| - | - |
| `planned` | Planned |
| `staged` | Staged |
| `active` | Active |
| `failed` | Failed |
| `offline` | Offline |
| `rma` | RMA — vendor return approved or in progress; device remains onsite |
| `returned` | Returned |
| `decommissioning` | Decommissioning |
| `decommissioned` | Decommissioned |
| `inventory` | Inventory |
| `unknown` | Unknown |

```mermaid theme={null}
stateDiagram-v2
    direction LR

    [*] --> planned
    planned --> staged: racked
    staged --> active: first Ironic available / provisioned

    staged --> failed: provisioning failure
    active --> failed: critical failure
    active --> offline: maintenance
    failed --> staged: operator recovery
    failed --> offline: take out of service
    failed --> rma: vendor RMA raised
    offline --> active: back in service
    offline --> rma: vendor RMA raised
    offline --> returned: returned to vendor
    rma --> decommissioned: shipped offsite

    active --> decommissioning: retire
    offline --> decommissioning: retire
    decommissioning --> decommissioned: removed from Ironic
```

A cableless node swap moves its failed tray from `active`, `staged` or `planned` to `offline`, so
topology-service stops discovering it; any other status is left alone.

NetBox also tracks **provisioning progress** via tags (not a Fleet Manager column):
`Not Started` → `DHCP Configured` → `BMC Configured` → `Ironic Enrolled` → `Provisioned`.

Release admission reads Fleet Manager's eventually consistent inventory projection. Before any
Fleet Manager-owned customer handoff, the final handoff activity therefore re-reads NetBox and
requires `active`: adding the customer-ready trait, clearing a maintenance lessee, and writing
`operational_status=delegated` all refuse a missing or non-`active` device. This closes the window
where NetBox can move a device to `rma` after the release job was admitted.

## Operational status

**Field:** `Device.operational_status` — Fleet Manager's **internal** customer-pool readiness
vocabulary. FM defines these enum values and the derivation rules; they are not mirrors of a
single external system's state machine. **Derived** from [Ironic](../start-here/glossary#ironic), Placement, and [SLURM](../start-here/glossary#slurm) conditions
(ADR-0024). The OpenStack and Slurm agents report source conditions and raw subsystem mirrors; the
hub is the only writer of the Ironic/Placement/SLURM-derived status. `delegated`,
`pending_handover`, and `drained` are **frozen** while Google owns the custody projection,
and an active impairment hold freezes its quarantine — condition observations continue
without overwriting either hold.

| Value | Typical cause |
| - | - |
| `unknown` | Unrecognised Ironic provision state |
| `quarantine` | Enroll/verifying, maintenance, orphan node, SLURM demotion |
| `validation` | Maintenance-pool instance, or manageable + instance |
| `available` | Customer pool, no instance, customer-ready trait |
| `in_use` | Nova instance attached |
| `delegated` | Google custody (`GOOGLE_OWNED`) |
| `pending_handover` | Custody transition in flight; retained in the status contract, but no current production path writes it |
| `drained` | Borneo `DRAINED` (CUJ4/CUJ5) |
| `failed` | Ironic failure provision states |

```mermaid theme={null}
stateDiagram-v2
    direction TB

    [*] --> unknown: ironic UNKNOWN
    unknown --> quarantine: enrolled / verifying
    unknown --> failed: ironic failure state

    quarantine --> validation: instance in maintenance pool
    quarantine --> available: customer-ready trait, no instance
    quarantine --> in_use: customer-ready + instance

    validation --> quarantine: SLURM member/schedulable FALSE
    validation --> in_use: customer-ready + instance
    validation --> available: released, trait set

    available --> in_use: instance allocated
    in_use --> available: instance released

    available --> failed: ironic failure
    in_use --> failed: ironic failure
    quarantine --> failed: ironic failure

    pending_handover --> delegated: Borneo GOOGLE_OWNED
    delegated --> drained: Borneo DRAINED
    drained --> delegated: maintenance complete (Borneo GOOGLE_OWNED inbound)
```

`pending_handover` remains a valid frozen status for existing rows, but has no current writer.
The production Borneo poll persists the credential returned with `PARTNER_OWNED` and then skips
the post-MVP recovery path, so CUJ3 does not currently change Fleet Manager operational status.

The `quarantine → in_use` edge (without passing through `validation` or `available`) is
intentional: when a maintenance-pool node carries the customer-ready trait and still
has a Nova instance, the reconciler treats it as `in_use` — the trait is authoritative proof
of customer [handover](../start-here/glossary#handover) and outranks the maintenance-pool lessee (`reconciler.py`, maintenance-pool
branch). Same path exists via conditions (`CustomerReady=True` + `InstanceAllocated=True`).

`ImageNodeWorkflow` (FLEET-2453) installs the configured per-role SLURM image and flavour on an
overcloud node without registering it with SLURM or entering the customer pool. Before Nova
creation it moves the Ironic lease from the enrol project to the SLURM project, then creates the
server through the dedicated SLURM credential using the configured SLURM network and keypair.
The node may project as `quarantine` or `validation` while the image is installed; neither status
means customer [allocation](../start-here/glossary#allocation). A retry recovers the server from Ironic's `instance_uuid`, or from a
sole exact host-name match in the SLURM Nova project before Ironic has recorded that binding.
Multiple exact-name matches fail closed. Terminal Nova states also fail without automatically
deleting the discovered server.

On a standalone-Ironic site (`placement_enabled=false`, no Nova or Placement — North Carolina,
Sys2-LON14) `ImageNodeWorkflow` selects the Ironic-direct backend (ADR-0042) instead. Fleet
Manager mints the `instance_uuid` itself, then drives Ironic from `AVAILABLE` through `DEPLOYING`
to `ACTIVE` with the per-site configured image, checksum, and config-drive public keys — no Nova
server, no SLURM-project lease move. The resulting `InstanceAllocated` condition still derives from
the bound `instance_uuid`, so operational status projects the same as the Nova arm. A fresh
execution mints a new `instance_uuid`; a node found mid-deploy recovers its in-flight binding,
but an idle `AVAILABLE` node still carrying a prior `instance_uuid` is rejected for cleanup rather
than adopted (it may be an undeploy that reached `AVAILABLE` but has not yet cleared its identity).
`DEPLOYFAIL` fails closed (deploy-arm undeploy-recovery is a follow-up). `UnimageNodeWorkflow`
selects the same Ironic-direct backend to tear a standalone-imaged node down: it drives Ironic's
`deleted` action (`ACTIVE`/`DEPLOYFAIL` → cleaning → `AVAILABLE`), polls to observed `AVAILABLE`,
then atomically clears the node's `instance_uuid`/`instance_info` so it derives as un-allocated
again — no Nova server to delete. The disk, config-drive, and injected-key wipe is Ironic
cleaning's work on that transition, so it only happens when the site conductor runs automated
cleaning with a disk-erase step; Fleet Manager drives `deleted` and observes `AVAILABLE` but does
not itself force the erase, so a site that has cleaning disabled returns the node to the pool
without wiping it (an operational prerequisite pinned in the deployment guide, not enforced here). A node already tearing down (`DELETING`/`CLEANING`/`CLEANWAIT`/`DELETED`) is not
re-fired, and the binding clear is guarded on observed `AVAILABLE` under the node's own uuid so a
concurrent rebind cannot be clobbered. The backend is chosen from the site's current capability;
an Ironic-imaged node must be undeployed before its site gains Nova (ADR-0042 review-trigger).

At a full-OpenStack site the Nova arm runs instead: `UnimageNodeWorkflow` only deletes the Nova
server identified by Ironic's `instance_uuid`; it never
searches Nova by host name. An unbound node is already complete only when Ironic is `AVAILABLE`.
Otherwise a missing binding, or a bound UUID whose Nova server is missing, fails for explicit
operator repair. After requesting Nova deletion, the workflow waits for the server to disappear
and Ironic to return to `AVAILABLE`, then returns the lease to the enrol project. A retry accepts
that already-completed enrol-leased state. A bound `DEPLOYFAIL` is admitted for this explicit
cleanup, and the workflow tolerates that original state while Nova deletion starts. Other Ironic
failure states are rejected before Nova deletion and reported immediately if reached during
teardown.

Image admission requires authoritative `maintenance=false` and rejects every Ironic failure state,
so Image Node never silently clears maintenance or recovers a failed node as successfully imaged.
Unimage remains an explicit operator action and can remove the exact Nova UUID bound to a
SLURM-leased `DEPLOYFAIL` node. Undercloud/CPU-hypervisor routing remains out of scope and is
rejected at runtime.

### Ironic → operational mapping (simplified)

| Ironic signal | Operational status |
| - | - |
| `UNKNOWN` | `unknown` |
| Failure states (`deploy failed`, `clean failed`, `inspect failed`, `rescue failed`, `unrescue failed`, `adopt failed`, `service failed`, `error`) | `failed` |
| `maintenance=True` | `quarantine` |
| `enroll`, `verifying` | `quarantine` |
| `manageable` + no instance | `quarantine` |
| `manageable` + instance | `validation` |
| Maintenance-pool lessee + instance | `validation` or `in_use` (trait) |
| Customer pool + instance | `in_use` |
| Customer pool + no instance + trait | `available` |

## Health status

**Field:** `Device.health_status` — independent telemetry axis from Mimir. Not derived from
operational status or raw subsystem fields.

| Value | Meaning |
| - | - |
| `unknown` | No metric observed |
| `healthy` | Metric value is zero |
| `degraded` | Metric value is greater than zero |
| `critical` | Reserved; not emitted by the health-status agent yet — operators will not see this value today |

```mermaid theme={null}
stateDiagram-v2
    [*] --> unknown: no metric
    unknown --> healthy: metric = 0
    unknown --> degraded: metric > 0
    healthy --> degraded: metric > 0
    degraded --> healthy: metric = 0
```

**Customer-ready** is a composite predicate (not a stored state), implemented on
`Device.is_customer_ready` / `Device.customer_ready_q()`:

* **Inventory-only devices are never customer-ready** (`inventory_only=True` excludes them)
* **Standard pool:** `inventory_status=active`, `operational_status` in `{available, in_use}`,
  and `health_status=healthy`
* **Delegated (Metal Managed):** `inventory_status=active`, `operational_status=delegated`, and
  `health_status` in `{healthy, unknown}` — healthy delegated devices are customer-ready; only
  `degraded` / `critical` exclude them

Site / device operational and health **count aggregations** also exclude `inventory_only`
devices from totals and per-status buckets so projection-only inventory does not read as
unhealthy [fleet](../start-here/glossary#fleet).

## Device conditions

**Table:** `device_conditions` — multi-source assertions (ADR-0024). Each row is the **latest**
value for one semantic aspect. Most are refreshed observations; durable lifecycle assertions change
only explicitly. Sources upsert conditions; the hub **derives** `operational_status` (and eventually
`health_status`) only from the conditions assigned to those domains.

Conditions are **not** a lifecycle state machine — each type is tri-state:

| `status` | Meaning |
| - | - |
| `True` | Aspect satisfied (positive polarity — `True` never means "broken") |
| `False` | Aspect not satisfied; `reason` names the cause |
| `Unknown` | Source cannot vouch (or row absent — treated as unknown) |

Each row also carries two hub-maintained fields (FLEET-1547):

| Field | Values | Maintained by |
| - | - | - |
| `condition_of` | `operational_status` / `health_status` / `none` | Stamped at create time from the code-owned registry (`core/conditions/registry.py`); a fixed property of the type — sources never send it. `none` is eligibility-only and feeds neither top-level status |
| `decisive` | boolean | Reconciled by the hub derivation on every materialise; independent of [out-of-band](../start-here/glossary#oob) status overrides |

### Condition types and owning sources

The **Feeds** column is persisted on each row as `condition_of`:

| Type | Source | Feeds |
| - | - | - |
| `ProvisioningHealthy` | `ironic` | `operational_status` |
| `MaintenanceClear` | `ironic` | `operational_status` |
| `Enrolled` | `ironic` | `operational_status` |
| `InValidationPool` | `ironic` | `operational_status` |
| `InstanceAllocated` | `ironic` | `operational_status` |
| `CustomerAllocated` | `ironic` | `operational_status` |
| `CustomerReady` | `ironic` (Placement trait or Ironic node trait) | `operational_status` |
| `SlurmMember` | `slurm` | `operational_status` |
| `SlurmSchedulable` | `slurm` | `operational_status` |
| `TelemetryHealthy` | `health-status` | `health_status` |
| `SystemHealthy` | `health-status` | `health_status` |
| `ChassisHealthy` | `health-status` | `health_status` |
| `ProcessorHealthy` | `health-status` | `health_status` |
| `MemoryHealthy` | `health-status` | `health_status` |
| `StorageHealthy` | `health-status` | `health_status` |
| `DiskHealthy` | `health-status` | `health_status` |
| `PowerSupplyHealthy` | `health-status` | `health_status` |
| `GpuHealthy` | `health-status` | `health_status` |
| `FpgaHealthy` | `health-status` | `health_status` |
| `CustodyIntent` | `google-custody` | `operational_status` (durable; schema exists, derivation not shipped) |
| `FrontendPortsInCustomerVrf` | `google-custody` | `none` (durable custody latch; [VRF](../start-here/glossary#vrf) flip workflows and the Google custody path write it) |
| `NvLinkGpusInRepairPartition` | `nvlink-fabric` | `none` (durable; the [NVLink](../start-here/glossary#nvlink) move workflows write it) |
| `BackendPortsInMaintenanceVrf` | `backend-fabric` | `none` (durable; the back-end VRF flip workflows write it on real runs, and an operator per the runbook at sites without real writes) |

`CustodyIntent`, `FrontendPortsInCustomerVrf`, `NvLinkGpusInRepairPartition`, and
`BackendPortsInMaintenanceVrf` are **durable** — no TTL or
observation-age expiry; they
change explicitly only. Observation types retain their existing freshness and source-silence behavior.
`condition_of=none` conditions never participate in operational or health derivation and are never
decisive; eligibility may consume them independently.

### Enrollment identity proof

ADR-0038 stores current Enrollment proof directly on the Device:

| Field | Values | Meaning |
| - | - | - |
| `enrollment_validated_ref` | infrastructure reference or empty | Exact infrastructure identity for which qualifying Enrollment completed |
| `enrollment_proof_source` | `enrolled`, `legacy`, `none` (`unmigrated` is reserved and never written) | Provenance of that proof; `enrolled` and `legacy` require a non-empty validated ref, while `none` requires an empty ref |

The workflow-agent writes these fields only through `POST /devices/{id}/enrollment-proof`, protected by
the internal `enrollment-proof:write` scope. A qualifying typed completion establishes
`enrolled` proof for its result reference. A started non-success submits a non-revoking failure write
guarded by the identity pinned in the workflow: if the Device now carries a replacement
`infrastructure_ref`, the write safely no-ops; if matching `enrolled` or `legacy` proof already exists,
the earlier validation is preserved. Stale proof metadata is cleared when the operated identity still
matches. A device without prior proof remains explicitly at `none`. Diagnostic success
(`run_inspection=false`), legacy string results, and failures without a trustworthy pinned identity
make no proof write. The hub also clears proof atomically when a Device leaves enrollment-managed
scope. An in-scope identity change does not need an invalidation write: the eligibility
comparison requires `enrollment_validated_ref == infrastructure_ref`.

**Eligibility gate (ADR-0038).** A policy that sets `require_enrollment_proof` refuses a device
unless `enrollment_validated_ref` is non-empty and equal to `infrastructure_ref` — a pure comparison
of the two references (`enrollment_proof_source` plays no part). It fails closed: an empty validated
ref (new device, failed enrolment, scope re-entry, rejected legacy) and a mismatched ref both block,
emitting `enrollment_proof_missing`. The flag is set on **Move to Test Cluster** (Quarantine);
`cordon_only` is exempt (it removes the customer-ready trait without adopting, so it stays runnable on
an unproven customer node), and the custody-reclaim statuses (`delegated`, `pending_handover`) are
exempt exactly as they are for the condition and Ironic-shape gates. Enrollment itself never requires
proof (it produces it), and burn-in and Image Node stay proof-exempt. The gate is evaluated on the
same eligibility pass as every other policy at job creation.

**Runtime enforcement (in-activity).** Admission is only a snapshot — the submitter starts runs
asynchronously, and a supported `infrastructure_ref` change afterwards clears proof — so both
customer-facing mutations re-read the Device and re-apply the same comparison **inside** the activity
that mutates, failing non-retryably with `ENROLLMENT_PROOF_MISSING` when proof is absent (a failed
proof *read* is `ENROLLMENT_PROOF_UNAVAILABLE` instead — retryable, owned by Fleet Manager — so an
API outage never reads as an identity verdict). Move to Test Cluster re-checks inside
`adopt_host_activity`, just before the SLURM adoption, and inside `force_reclaim_host_activity` when
`force_reclaim` is set (it deletes the customer instance before the adopt, so it is bound to the proven
identity too); fully managed Release
re-checks inside the customer-return activities (`add_customer_ready_trait` /
`release_to_customer_pool_activity`), beside the NVLink partition-latch preflight. On every enforced path
the proof is read by the mutation itself, immediately before its first external write (the activity
hands the domain service a resolver, so no setup, resolution or wait sits between read and write), and
the write is bound to that identity: the mutation refuses if the Ironic node it resolved does not carry
the proven `infrastructure_ref` UUID. Release also
*must* enforce at runtime because the hub can't see the satellite-injected `engagement_mode`. Doing it in the
same activity as the mutation keeps them atomic (no queue gap). Adopt and the customer-return
activities always receive `device_id` and an explicit `enforce_enrollment_proof=True`; on Release the
NVLink partition latch has its own `enforce_partition_latch` argument, so proof enforcement never
depends on the latch check. The proof check is keyed on its enforce argument alone: an empty
`device_id` under enforcement is refused, never skipped.
`cordon_only` returns before the adopt and is exempt, as are the custody-reclaim statuses. Release keys
on `engagement_mode=fully_managed` + `release_to_customer_pool`, so its other paths stay exempt: internal
Release (`release_to_customer_pool=false`) returns to no customer, and metal / self-managed returns hand
off through their own (non-Ironic) paths.

### Front-end VRF custody latch contract

`FrontendPortsInCustomerVrf` records where a device's front-end switch ports were last **verified**
to be. It exists because a VRF flip has no downstream witness the way an enrolment has inspection:
the flip's own read-back is the only proof, and this latch is where that proof is kept so the
invariant *ports must be out of the customer VRF before the device goes to anyone* can be enforced
without re-reading the fabric.

The flip reasons were `HandoverFlipVerified` and `ReturnFlipVerified` until FLEET-2825; migration
`0211` rewrote any existing rows to the new names.

| Status | Reason | Meaning | Contracted writer |
| - | - | - | - |
| `True` | `FrontendVrfReturnVerified` | A verified front-end VRF return moved the ports into the customer VRF | `FrontendVrfReturnWorkflow` |
| `False` | `FrontendVrfIsolateVerified` | A verified front-end VRF isolate moved the ports out of the customer VRF | `FrontendVrfIsolateWorkflow` |
| `Unknown` | `CustodyEntered` | The device entered Google custody after a verified front-end VRF return; Fleet Manager no longer observes its ports | Google connector inbound path |
| `Unknown` | `HandoverBreach` | The device entered Google custody with no verified front-end VRF return on record | Google connector inbound path |

**Who reads it.** `ENROLLMENT` declares this type in
`EligibilityPolicy.disqualifying_condition_types` — the blocking complement of
`required_condition_types` — so an enrolment job is **refused** while the latch reads `True` or
`Unknown`, and admitted on `False` or absence. The refusal surfaces at job validation as
`condition_disqualifier_asserted` / `condition_disqualifier_unproven` carrying the failing
`condition_type`, and is cleared by running the front-end VRF isolate, which writes `False`. `ENROLLMENT`
is the only gated workflow: `RELEASE` also hosts the CUJ2 handover branch, whose precondition is the
opposite one, and release-to-pool and `CUSTOMER_RELEASE` are recorded as unreachable for a
latch-carrying device rather than gated (spec 0045 §6). The handover-side wall is a preflight inside
the CUJ2 PATCH and belongs to FLEET-2253.

Absence means the same thing as `False` to a reader — no verified observation places the ports in a
customer VRF. That makes absence the **permissive** state, so a device that is already `delegated`
must not be left without a row: the Google connector writes the latch whenever the row is missing,
not only on the status transition, which lazily backfills every device already in custody within one
backstop poll of deploying (reported as `HandoverBreach`, because no verified flip is on record for
them). On an actual transition into `delegated`, a latch-write failure is fail-closed: the exception
prevents the operational-status write, leaving the device in the poll's target set so the entire
transition is retried. Only missing-row repair for a device already in `delegated` is best effort: a
failed repair is logged and retried on a later poll. A device whose row already exists and whose status
is unchanged is left completely alone, so the breach is not re-logged on every poll.

The custody-entry write is **hub-ordered**: the hub stamps `observed_at` and the write is exempt from
source-observation staleness. This type has two writers whose
clocks are independent — one on the hub, one on a satellite — and the API tolerates five minutes of
future skew, so a future-dated flip observation would otherwise silently stale-reject a *later*
custody entry and leave the permissive value standing. The flip writers stay observation-ordered
(they are reporting what they read from the switch at a moment) and verify the row the hub returns:
a 200 whose body is the stored row rather than the submitted one means the write was stale-rejected,
and the activity fails rather than reporting a latch it did not record.

Both flip workflows write the latch only when **every** per-switch result reports `simulated=false`.
A simulated run verified nothing, and a simulated return that cleared the latch would remove the
only record standing in front of a device whose ports never left the customer VRF. A verified flip whose latch write fails after retries fails the job with
`SWITCH_VRF_LATCH_WRITE_FAILED` — distinct from `SWITCH_VRF_FLIP_FAILED` because the ports *did*
move; the recovery is to re-run the same flip job, which no-ops on the already-applied switch and
writes the latch.

On the inbound side the write happens on Borneo `GOOGLE_OWNED` — on the custody-entry transition, and
on any poll that finds no latch row (see above) — and never on `PARTNER_OWNED` or `DRAINED`, because a
return is proven by the front-end VRF isolate, not by Borneo's word. A `HandoverBreach` is logged at error level
as the tripwire. Two breaches are expected rather than defects: an in-place maintenance cycle
(`delegated → drained → delegated`) re-enters custody with the latch already `Unknown`, because the
last verified observation is genuinely gone; and every device already in custody when this shipped is
backfilled as a breach, because no verified flip is on record for it. The answer is to re-run the front-end VRF return before the CUJ2 PATCH, which is why
`drained` is in that workflow's eligibility allowlist. Operator procedure lives in the
`docs/runbooks/switch-vrf-flip.md` runbook.

### NVLink repair-partition latch contract

`NvLinkGpusInRepairPartition` is the durable record of the partition membership a real NVLink move
just verified. It is owned solely by `nvlink-fabric` and has no derivation role.

| Status | Reason | Meaning | Contracted writer |
| - | - | - | - |
| `Unknown` | `MoveStarted` | A real move passed custody preflight and is about to mutate NMX-M; membership is unproven until verification | `NvLinkActivities.apply_partition_move` |
| `True` | `IsolateVerified` | A verified isolate placed the tray's [GPUs](../start-here/glossary#gpu) in OFR | `NvLinkIsolateWorkflow` |
| `False` | `ReturnVerified` | A verified return placed the tray's GPUs back in Default | `NvLinkReturnWorkflow` |

The customer-facing `RELEASE` handoff reads this latch: `True` and `Unknown` refuse the handoff,
while `False` and absence are permissive. A SLURM-only Release remains available because it does not
return the tray to customers. Simulated runs write nothing. The handoff reads the durable latch
immediately before each customer-facing mutation; a queued release therefore cannot retain
authorization after an isolate writes `Unknown` or `True`. The `Unknown` write is inside the real
move activity after its second custody check, planning, and final drift check, immediately before the
first NMX-M write, so a pre-write refusal cannot leave a false latch. A
verified move whose post-write fails after retries ends with `NVLINK_LATCH_WRITE_FAILED`; re-run the
same move, which converges from live membership and records the latch. The condition is keyed to the
Device, so it survives the tray swap it describes until a verified return clears it.

### Back-end VRF latch contract

`BackendPortsInMaintenanceVrf` records whether a host's RoCE back-end ports may be in the
maintenance VRF (spec 0064). It has the NVLink latch's polarity and lifecycle, is owned solely by
`backend-fabric`, and has no derivation role.

| Status | Reason | Meaning |
| - | - | - |
| `Unknown` | `MoveStarted` | A back-end flip was about to write a switch; where the ports are is unproven |
| `True` | `IsolateVerified` | Every back-end port was verified in the maintenance VRF |
| `False` | `ReturnVerified` | Every back-end port was verified back in the customer VRF |
| absent | | Fleet Manager has never isolated this host's back end; permissive |

The back-end VRF flip workflows (spec 0064) write it on a real run, at a site that sets
`backend_vrf_flip_allowed`. Each apply records `Unknown` after its checks pass and just before it
writes a switch with a port to move; once every switch verified, isolate records `True` and return
`False`. A failed run leaves whatever the applies wrote, so `Unknown` holds until a rerun records a
definite value; a simulated run writes nothing. At sites without real writes, an operator records
hand isolations by following the release wall in `docs/runbooks/backend-vrf-flip.md`, with the
runbook's own reasons (`ManualIsolation`, `ManualReturn`).

The customer-facing `RELEASE` handoff reads it exactly as it reads the NVLink latch: `True` and
`Unknown` refuse, `False` and absence permit, and a SLURM-only Release is unaffected. Admission
refuses through the `release_to_customer_pool` gate. In flight, each hand-back activity re-reads both
latches before its customer-facing mutation: the Metal hand-back always does, and the fully managed
hand-backs do when the Release run carries `enforce_partition_latch` (set from
`release_activity_args_ready`). A back-end refusal fails with `RELEASE_BACKEND_VRF_LATCH_REFUSED`. The condition is keyed to the Device, while VRF membership
belongs to switch ports, so a replacement tray from a Node Swap can inherit isolated ports with no
row.

### When conditions are emitted

Rules below mirror `agents/openstack/conditions.py` and `agents/slurm/reconciler.py`. Absence
of a row is treated as `Unknown` at derivation time.

| Source | Condition | Emitted when |
| - | - | - |
| `ironic` | `ProvisioningHealthy` | `Unknown` if provision state is `UNKNOWN`; `False` if `provision_state.is_failure` (reason = raw state, message = `last_error` or fallback); otherwise `True` |
| `ironic` | `MaintenanceClear` | `False` if `node.maintenance`; otherwise `True` |
| `ironic` | `Enrolled` | `False` for `enroll` / `verifying`; otherwise `True` |
| `ironic` | `InValidationPool` | `True` for `manageable` or lessee in maintenance/enrolment projects; `False` for customer project or no lessee |
| `ironic` | `InstanceAllocated` | `True` if `instance_uuid`; otherwise `False` |
| `ironic` | `CustomerAllocated` | `True` if leased to a non-maintenance project; otherwise `False` |
| `ironic` | `CustomerReady` | `Unknown` when not evaluated; `True` when customer-ready trait present; `False` when trait missing. Full sites read from Placement resource providers; standalone-Ironic sites read directly from the Ironic node trait |
| `slurm` | `SlurmMember` | `Unknown` if latest SLURM fetch failed; `False` if node absent from snapshot; `True` if present |
| `slurm` | `SlurmSchedulable` | `Unknown` if fetch failed or node absent from SLURM; `False` if present but down; `True` if present and schedulable |
| `google-custody` | `FrontendPortsInCustomerVrf` | `True`/`False` after a verified, non-simulated VRF flip (handover / return); `Unknown` on Borneo custody entry. See the contract above |
| `health-status` | `TelemetryHealthy` | `True` when the current complete Mimir snapshot contains hardware telemetry for the host; otherwise `Unknown` |
| `health-status` | `SystemHealthy`, `ChassisHealthy`, `ProcessorHealthy`, `MemoryHealthy`, `StorageHealthy`, `DiskHealthy`, `PowerSupplyHealthy` | Aggregated by type from [Redfish](../start-here/glossary#redfish)-derived `redfish_health`: `True` for OK, `False` for Degraded/Critical, `Unknown` for unusable or missing telemetry |
| `health-status` | `GpuHealthy` | Aggregated across `nvidia_gpu_health` series for the host; `Unknown` when the host has no GPU series |
| `health-status` | `FpgaHealthy` | Aggregated from `nvidia_fpga_health`; `Unknown` when the host has no FPGA series |

Health conditions use source-specific reasons such as `RedfishHealthCritical`,
`RedfishTelemetryUnavailable`, `NvidiaGpuHealthDegraded`, and `NvidiaFpgaTelemetryUnavailable`; every
message opens with the exact Mimir series selector. The `source` remains `health-status`; provenance
belongs in the reason and message. Every host in scope carries all ten rows every cycle, like the
Ironic and SLURM sources, so nothing is remembered between cycles. These conditions explain `Device.health_status`; they do not drive it. `health_status` stays
computed from the `chassis_health` composite, so `degraded` may be explained by a condition that says
either Degraded or Critical.

### Operational-status derivation (ordered rules)

Hub-owned pure derivation (`core/conditions/derivation.py`) — same semantics as the legacy Ironic
reconciler, with one deliberate SLURM divergence. This is a priority-ordered classifier, not a
transition engine. First matching row wins:

| Derived `operational_status` | Condition shape (all prior rules failed) | Why this rule is ordered here |
| - | - | - |
| *(abstain — no write)* | No Ironic-owned condition in snapshot | SLURM-only or empty snapshots cannot decide Ironic-owned states; abstain preserves the current materialised value instead of flattening it to `unknown` |
| `unknown` | `ProvisioningHealthy=Unknown` | Ironic provision state `UNKNOWN` emits all Ironic conditions as `Unknown`; `ProvisioningHealthy` is the anchor for that source-wide uncertainty |
| `failed` | `ProvisioningHealthy=False` | Ironic failure states must outrank maintenance, enrolment, and allocation so `failure_reason` / `failure_details` surface the root failure |
| `quarantine` | `MaintenanceClear=False` | Operator/Ironic maintenance cordons the node before pool or allocation interpretation |
| `quarantine` | `Enrolled=False` | Enroll / verifying nodes are not usable, regardless of any lower-priority allocation-looking signals |
| `available` | `InValidationPool=True`, `InstanceAllocated≠True`, `CustomerReady=True` | Legacy #1795: the customer-ready trait is authoritative handover proof, even when the lessee still looks like the maintenance/enrolment project |
| `quarantine` | `InValidationPool=True`, `InstanceAllocated≠True`, `CustomerReady≠True` | Idle validation-pool nodes are not customer-available unless the customer-ready trait positively proves readiness |
| `in_use` | `InValidationPool=True`, `InstanceAllocated=True`, `CustomerReady=True` | Customer-ready plus an attached instance means handover is complete; this outranks the maintenance-pool lessee and SLURM validation checks |
| `quarantine` | `InValidationPool=True`, `InstanceAllocated=True`, `CustomerReady≠True`, and `SlurmMember=False` or `SlurmSchedulable=False` | ADR-0024 divergence: only a node still in validation can be demoted by explicit scheduler `False`; absent or `Unknown` SLURM never demotes |
| `validation` | `InValidationPool=True`, `InstanceAllocated=True`, `CustomerReady≠True`, no explicit SLURM `False` | Maintenance-pool instances are still validation capacity until customer-ready proves handover |
| `in_use` | `InstanceAllocated=True` (outside validation-pool branch) | Outside validation, an attached Nova instance is the strongest usage signal, including when lessee metadata is missing |
| `available` | `CustomerAllocated=True` (`CustomerReady` included in decisive set only when `True`) | Customer-pool idle nodes remain available while the customer-ready trait backfill completes (FLEET-868) |
| `quarantine` | Orphan: `InValidationPool=False`, `CustomerAllocated=False`, `InstanceAllocated=False` | A node with no validation, customer, or instance allocation is orphaned and must not be advertised as available |
| `unknown` | Partially-unknown snapshot not matched above (emitters do not produce this today) | Surface unexpected partial evidence visibly instead of guessing a customer-facing state |

**Materialisation and reconciliation:** `POST /api/v1/device-conditions/apply/` applies one
source's full condition snapshot and always verifies the materialised projection, including
freshness-only resyncs where condition values have not changed. A matching `operational_status`
skips that status-column write, while decisive flags, `entered_service_at`, and failure metadata
are still reconciled. A stale projection is repaired. The single-condition
`POST /api/v1/device-conditions/` path re-derives only after a semantic change (create, status,
reason, or message); freshness-only and stale-ignored single-condition writes skip derivation.

**External-hold freeze:** while `operational_status` is `delegated`, `pending_handover`, or
`drained`, or the device carries an active impairment-event hold, the hub does not re-materialize
operational status from conditions. Conditions and decisive flags still update; only the top-level
write is frozen. A drained device therefore remains in Ironic reconciliation scope for its bounded
maintenance window without losing the Google-owned custody projection. The custody part is the V1
guardrail ported to the hub until `CustodyIntent` derivation ships.

Use `GET /api/v1/devices/{id}/` (device retrieve) — field `operational_status_conditions` shows
which conditions were **decisive** for the current operational status, read from the persisted
`decisive` flag so it always agrees with the `/api/v1/device-conditions/` list (filterable by
`condition_of` and `decisive`) and the materialised status.

## Lifecycle event log

**Table:** `DeviceLifecycleEvent` — the append-only history of the three stored status axes
above. The status columns themselves are classifiers that hold only the current value; this
table is what answers "how did this device get here and how long has it been here".

| Column | Meaning |
| - | - |
| `kind` | Which axis changed: `operational_status`, `health_status`, `inventory_status` |
| `value` | The new value of that axis (one of that axis's enum members) |
| `created_at` | When Fleet Manager observed the change (not the upstream event time) |

**How rows are written.** `Device.save()` diffs the tracked columns against the values it loaded
and appends one row per changed column in the same transaction as the device write. Writers that
bypass `save()` (the impairment cordon/release `bulk_update`) append rows through the same
`Device.lifecycle_events_for_changes()` diff. A device gets one birth row per kind on creation.
Devices that predate the table received one baseline row per kind at migration time, so every
timeline has a "this value since at least here" starting point. There is no stored previous
value: the prior state is the previous row of the same kind. Rows are never updated or deleted
by application code.

**Reading it.** `GET /api/v1/devices/{id}/lifecycle-events/` (scope `device:read`) returns the
device's rows newest first, optionally narrowed with `?kind=`. Each row carries `ended_at`, the
`created_at` of the next row of the same kind, computed with a window function so it is correct
across pages; `ended_at` is null on the row describing the current value, whose duration is
measured to now. Triage is not tracked here: `Device.entered_triage_at` already answers "in
triage since".

## Integration points

Fleet Manager aggregates many external systems. Not all of them own a device status field — some write
conditions, some write raw mirrors, some trigger workflows, some only read FM state outward.

| System | Agent / path | What it writes on `Device` | Notes |
| - | - | - | - |
| **NetBox** | NetBox agent | `inventory_status`, identity, rack, interfaces | Inventory source of truth inbound |
| **OpenStack Ironic** | OpenStack agent | `ironic_provision_state`, maintenance fields, Ironic conditions | Hub derives `operational_status`, `entered_service_at`, and provisioning-failure metadata; resolved by `infrastructure_ref`, falling back to FM-canonical name. A node swap re-pins a node UUID onto a different tray, so when a ref match's name disagrees with the node's and another device claims that name, the name wins: the projection re-points onto that device and the row it left is retired to `Unknown` (reason `IdentityMoved`) |
| **OpenStack Placement** | OpenStack agent | `CustomerReady` condition (full sites: Placement resource-provider trait; standalone-Ironic sites: Ironic node trait) | Trait `CUSTOM_CUSTOMER_READY` |
| **OpenStack Nova** | OpenStack agent | `current_instance_uuid` (denormalized) | Instance ↔ device resolution |
| **SLURM** | Slurm agent | `slurm_states`, `slurm_state_reason`, `SlurmMember` / `SlurmSchedulable` conditions | Own satellite process (ADR-0036); resolves devices by FM-canonical name |
| **Mimir / telemetry** | Health-status agent | `health_status` (direct write from `chassis_health`), hardware-health device conditions | Conditions aggregate Redfish-derived and NVIDIA GPU/FPGA health per host and explain `health_status`, which stays computed from `chassis_health`; independent of operational status |
| **Google Borneo** | Google connector + custody workflows | The deployed poll writes `operational_status` as `delegated` or `drained` and writes `FrontendPortsInCustomerVrf` on custody entry; Temporal custody activities can write `delegated` or `quarantine` | `pending_handover` has no current writer. The inbound helper retains a `PARTNER_OWNED` → `quarantine` mapping, but the production poll persists the CUJ3 credential and returns before invoking it |
| **Site switches (NVUE)** | VRF flip workflows ([Temporal](../start-here/glossary#temporal) worker) | `FrontendPortsInCustomerVrf` condition | Written only for a verified, non-simulated flip; the mirror of the same type the connector writes on custody entry |
| **Impairments (legacy, no known caller)** | Impairment-event API | `operational_status=quarantine`, `impairment-event-id` hold annotation | Radar does not call this API. Hub derivation freezes the top-level status while the hold is active; final release restores the prior status as an abstain fallback, then immediately re-derives from current conditions |
| **Radar (projection)** | Radar ingestion agent | **Reads** FM device state | Publishes availability/health outward to Radar |
| **Temporal workflows** | Temporal worker + workflow agent | `Job.status`, `JobDeviceStatus.status`; the terminal handler owns `enrollment_validated_ref` / `enrollment_proof_source` | Typed workflow results establish Device proof through the purpose-scoped endpoint; non-success uses the replay-safe identity query for a non-revoking guarded failure write that preserves matching proof and clears stale metadata when safe |
| **Operators** | API | `in_triage`, manual inventory corrections | |

```mermaid theme={null}
flowchart LR
    subgraph External["External systems"]
        NBX[NetBox]
        OSK[OpenStack Ironic / Placement / Nova]
        SLURM[SLURM]
        MIM[Mimir]
        BOR[Google Borneo]
        RAD[Radar NATS]
    end

    subgraph FM["Fleet Manager hub"]
        API[API + DB]
        DERIVE[Condition derivation]
    end

    subgraph Sat["Satellite agents"]
        NAG[NetBox agent]
        IAG[OpenStack agent]
        SAG[Slurm agent]
        HAG[Health-status agent]
        WAG[Workflow agent]
        RAG[Radar ingestion agent]
    end

    NBX --> NAG
    NAG --> API
    OSK --> IAG
    IAG --> API
    SLURM --> SAG
    SAG --> API
    MIM --> HAG
    HAG --> API
    BOR --> API
    API --> DERIVE
    API -->|projection| RAG
    RAG --> RAD
    WAG <-->|Temporal| API
```

### Impairment events (separate from device conditions)

Impairment events are created through `POST /api/v1/operations/impairment-events/`; Radar does not
call it. FM models these as **`ImpairmentEvent`** rows — a cordon workflow, not a persisted device
status axis.

**Event status:** `accepted` → `cordoned` → `cleared` (or `partially_failed` / `failed`; release
sub-states: `release_pending`, `release_retrying`, `release_failed`)

**Per-device status within an event:** `pending` → `cordoned` → `cleared` (or `failed` / `retrying` /
release sub-states)

Effect on devices: creating an event sets an `impairment-event-id` hold annotation and
`operational_status=quarantine`, which the hold freezes; the OpenStack agent's cordon removes the
customer-ready trait. Clearing the event queues release work, which restores the trait for a pre-hold
`available` or `in_use` host, removes the hold and re-derives the status. The release does not read
the fabric isolation latches.

### Radar outbound projection

The Radar ingestion agent **reads** FM `inventory_status`, `operational_status`, and `health_status`
and maps them to Radar `ResourceAvailabilityState` / `ResourceHealthState`. This path does not write
back to FM device fields.

Tenants can carry `annotations.radar_ignore=true` to opt out of projection (with required reason).

## Triage flag

**Field:** `Device.in_triage` — orthogonal boolean. A device can be in triage regardless of
inventory, operational, or health status. Set via `POST /api/v1/devices/triage/`.

```mermaid theme={null}
stateDiagram-v2
    [*] --> not_in_triage: default
    not_in_triage --> in_triage: bulk triage API
    in_triage --> not_in_triage: clear triage
```

**Ticket pointers (caller-managed, both `blank=True` strings rendered as `null` when empty):**

* `Device.triage_ticket_url` — the ticket tracking the *current* triage cycle.
* `Device.parent_ticket_url` — the device's persistent umbrella ticket. Set once and left
  alone across triage cycles; every `triage_ticket_url` opened for the device is expected to
  nest under it, so it accumulates the device's full triage history independently of
  `triage_ticket_url`'s own lifecycle.

## Inventory-only flag

**Field:** `Device.inventory_only` — orthogonal boolean (default `False`). Marks a core
`Device` that exists solely so `DeviceInventoryProjection` / customer connectors can publish
inventory **without** Fleet Manager managing the hardware.

* **Site-driven inheritance:** `Site.inventory_only` (a model-level boolean, immutable after
  create) is the authoritative source. On device create, `Device.save()` calls
  `_inherit_inventory_only_from_site()` which forces `inventory_only=True` when the site
  carries the flag — regardless of what the writer passed. The NetBox agent sets the site
  flag from deployment config (`netboxAgent.inventoryOnlySites`); the Matrix inventory
  seeder sets it per-device.
* **Immutable** via ordinary API/model updates; promote-to-managed is an explicit audited
  operation.
* Always ineligible for workflows (eligibility `DEVICE_INVENTORY_ONLY`); rejected by
  `Allocation.validate_devices`; excluded from [burn-in](../start-here/glossary#burn-in) target selection; blocked by the
  workflow-agent submitter before Temporal start.
* Excluded from customer-ready and from per-site operational/health count aggregations.
* Operational/health stay at their defaults (`unknown`/`unknown`) by design — nothing derives
  them for these rows. Enrollment is job-gated, so a Device row alone is never enrolled.

```mermaid theme={null}
stateDiagram-v2
    [*] --> managed: default inventory_only=False
    [*] --> inventory_only: create at inventory-only site
    inventory_only --> managed: audited promote-to-managed (out of band)
```

## Job status

**Field:** `Job.status` — workflow execution aggregate across all devices in the job.
Terminal values will not transition further.

| Value | Meaning |
| - | - |
| `pending` | Awaiting Temporal submission |
| `running` | Workflow executing |
| `cancellation_requested` | User requested cancel; Temporal may still be in flight |
| `completed` | Every device succeeded |
| `partially_failed` | Mixed success and failure across devices |
| `failed` | Job failed |
| `cancelled` | Cancelled |
| `terminated` | Terminated |
| `superseded` | Superseded by another job |
| `timed_out` | Timed out |

```mermaid theme={null}
stateDiagram-v2
    [*] --> pending
    pending --> running: Temporal start
    running --> cancellation_requested: user cancel
    cancellation_requested --> cancelled
    running --> completed: all devices OK
    running --> failed: all devices failed
    running --> partially_failed: mixed outcomes
    running --> terminated
    running --> superseded
    running --> timed_out
    running --> cancelled
```

## Job device status

**Field:** `JobDeviceStatus.status` — per-device execution within a job. A row is created at
submit; the status is **empty** until Temporal starts that device (there is no `pending`
value).

| Value | Meaning |
| - | - |
| *(empty)* | Waiting to start |
| `running` | Device workflow executing |
| `completed` | Succeeded |
| `failed` | Failed |
| `cancelled` | Cancelled |
| `terminated` | Terminated |
| `superseded` | Superseded |
| `timed_out` | Timed out |

```mermaid theme={null}
stateDiagram-v2
    [*] --> empty: row created at submit
    empty --> running: Temporal starts device
    running --> completed
    running --> failed
    running --> cancelled
    running --> terminated
    running --> superseded
    running --> timed_out
```

### Failure codes

**Field:** `JobDeviceStatus.failure_code` — a stable, machine-readable reason for a failed
device workflow. It is separate from `JobDeviceStatus.status`: `status=failed` says the
execution ended unsuccessfully, while `failure_code` routes the failure to the right
triage queue and `failure_message` gives its host-specific detail.

`failure_code` is a large, fast-growing set (one value per distinct failure mode), so the
canonical list of codes, their meaning, and their triage owner is not duplicated here —
see `CODE_OWNER` in
`src/fleet_manager/core/workflow/failure_modes/taxonomy.py`, or the `FailureCodeEnum`
component in `docs/openapi.yaml` for the values exposed over the API.

For example, `IRONIC_INSPECTION_IPA_IMAGE_NOT_FOUND` means Ironic selected an [IPA](../start-here/glossary#ironic-python-agent) kernel or
ramdisk path that was absent. Operators should compare the NetBox and Ironic `cpu_arch`,
node-level deploy-image overrides, per-architecture conductor configuration, and artifact
presence on every conductor before retrying one node.
`IRONIC_DRIVER_MISMATCH` means an Ironic power cycle was requested for a node that does not
use the required `ipmi` driver; switch the node's Ironic driver before retrying.
The detailed operator procedure extends the existing
`docs/runbooks/ipa-image-update.md` runbook.

## Ironic provision state

**Field:** `Device.ironic_provision_state` — raw OpenStack Ironic mirror. Stored as a plain
string so new upstream states do not require a migration; the API enforces the closed set at
the boundary.

### All states

| Group | States |
| - | - |
| Enrollment | `enroll`, `verifying`, `manageable` |
| Inspection | `inspecting`, `inspect wait`, `inspect failed` |
| Cleaning | `cleaning`, `clean wait`, `clean failed`, `clean hold` |
| Available pool | `available` |
| Deploy cycle | `deploying`, `wait call-back`, `deploy failed`, `deploy complete`, `deploy hold`, `active` |
| Teardown | `deleting`, `deleted` |
| Adopt | `adopting`, `adopt failed` |
| Rescue | `rescue`, `rescuing`, `rescue wait`, `rescue failed`, `unrescuing`, `unrescue failed` |
| Service | `service`, `servicing`, `service wait`, `service failed`, `service hold` |
| Other | `rebuild`, `error`, `unknown` |

**Stable states** (require an explicit verb to leave): `enroll`, `manageable`, `available`,
`active`, `deploy failed`, `clean failed`, `error`, `inspect failed`, `adopt failed`,
`rescue`, `rescue failed`, `unrescue failed`, `service failed`.

**Failure states** (map to `operational_status=failed`): `deploy failed`, `clean failed`,
`inspect failed`, `rescue failed`, `unrescue failed`, `adopt failed`, `service failed`,
`error`.

```mermaid theme={null}
stateDiagram-v2
    direction TB

    [*] --> enroll
    enroll --> verifying
    verifying --> manageable
    manageable --> inspecting
    inspecting --> available: success

    manageable --> cleaning
    cleaning --> available: success

    available --> deploying
    deploying --> active: success
    active --> deleting
    deleting --> deleted
    deleted --> cleaning

    deploying --> deploy_failed: failure
    cleaning --> clean_failed: failure
    inspecting --> inspect_failed: failure

    available --> servicing
    servicing --> available

    available --> rescuing
    rescuing --> rescue
    rescue --> unrescuing
    unrescuing --> available

    available --> adopting
    adopting --> available
    available --> rebuild
    rebuild --> deploying
```

Diagram node IDs abbreviate Ironic API values where spaces are invalid in mermaid (e.g.
`deploy_failed` on the diagram = `deploy failed` in `ironic_provision_state`).

For the full canonical diagram, see the
[OpenStack Ironic state diagram](https://docs.openstack.org/ironic/latest/_images/states.svg).

## Borneo custody

Google Borneo custody states overlay operational status during delegated-device workflows.
The deployed connector poll and its inbound helper are not equivalent for `PARTNER_OWNED`:
the poll persists the returned credential and exits before calling `handle_state_change`.

| Borneo state | Deployed poll behaviour | Inbound helper mapping |
| - | - | - |
| `PARTNER_OWNED` | Persists the CUJ3 credential, then skips status recovery | `quarantine`, but dormant in the production poll |
| `PENDING_HANDOVER` | No FM write | No FM write |
| `GOOGLE_OWNED` | `delegated` and latches `FrontendPortsInCustomerVrf=Unknown` | `delegated` |
| `DRAINED` | `drained` | `drained` |

```mermaid theme={null}
stateDiagram-v2
    PARTNER_OWNED --> PENDING_HANDOVER: FM custody workflow (outbound)
    PENDING_HANDOVER --> GOOGLE_OWNED: handover complete (Borneo)
    GOOGLE_OWNED --> DRAINED: CUJ4/CUJ5 drain (Borneo)
    DRAINED --> PARTNER_OWNED: return workflow (Borneo)
```

No current production path writes Fleet Manager `pending_handover`.

## Standup pipeline (derived UI view)

For a concise day-to-day explanation, start with
[DC Standup Pipeline](../operator-guide/dc-standup). This section defines the exact derivation.

The Operational Racks matrix does **not** store pipeline state on the device. Placement is
derived per request from `operational_status` and job history.

**Columns** — the real stage a device is *at or awaiting*, not the last it completed:
`enrolment`, `quarantine`, `single_node_test`, `multi_node_test`, `release`. A fresh device is
`column=enrolment, status=not_started`; `not_started` is a status, never a column.

The effective sequence is derived per device. Policy-backed GPU compute uses the full chain.
CPU bare-metal always includes quarantine (Move to Test [Cluster](../start-here/glossary#cluster) / SLURM) even without an
acceptance policy — `enrolment → quarantine → release`; a policy adds single-node acceptance.
CPU hypervisors terminate at enrolment.
Switches have no pipeline: they are excluded from device cells, and a rack of only switches is
omitted from the response entirely.

**Placement precedence:** current customer handover state is the first decision. If
`operational_status` is `available` or `in_use`, the device is placed at its effective terminal
stage with `status=succeeded` (`release` for customer hosts, `enrolment` for hypervisors). This
override is independent of Job timestamps and outcomes: newer failed, pending, or running Jobs
remain available through drill-through context but do not move the cell. Acceptance-gate summaries
also remain visible as historical evidence. The override skips both gate holds and the prefix
clamp.

If the device is not currently `available` or `in_use`, placement uses Job history as described
below, then advances successful stages and applies the prefix clamp.

**Advance-on-success:** when a device succeeds a stage it is shown in the *next* column as
`not_started` (a completed enrolment appears under `quarantine`). Advancement out of
`single_node_test` / `multi_node_test` is **held** — the device stays `succeeded` in that
column — until its burn-in acceptance gate passes ("site conditions"). `multi_node_test`
composes across *every* configured multi-node collective gate (e.g. an nvlink-domain rack
gate and a pod gate): the device advances only once all of them pass. Any configured gate
that can't yet be evaluated (missing evidence / topology) holds too. The matrix advances the
*column* only; the next workflow still runs on an operator click.

**How Job history places a device**

For devices without the customer-ready override, cancelled, terminated, and superseded attempts
are ignored for placement but kept in Job history. The newest execution is retained for Job
drill-through; placement uses the newest execution that is still active or established a result.

| Per-device outcome | Effect on placement |
| - | - |
| Pending or running | The active attempt owns its stage |
| Completed | Clears the stage |
| Failed or timed out | Does not clear the stage and may pull the cell back |
| Cancelled, terminated, or superseded | Does not change placement; inspect it through the latest Job link |

An interruption is not a success. If the stage has no earlier successful result, it remains
incomplete. If it does have an earlier success, the interruption does not revoke it. The
prefix clamp then prevents the cell from appearing beyond the first incomplete stage.

```text theme={null}
completed → cancelled = stage remains cleared
failed → cancelled    = stage remains failed
cancelled only        = stage remains incomplete
completed → timed out = stage becomes uncleared
```

Placement is derived on every GET, so no backfill is required. Interrupted rows are never
deleted or hidden from Job history. Placement does not prove current physical state or
write-time eligibility; live device conditions and the Jobs API remain authoritative before
an operator starts work.

**Prefix-completeness clamp:** when the customer-ready override does not apply, the cell cannot
sit past a stage whose placement-relevant Job is not a successful terminal run. A later
quarantine / single-node / multi-node attempt does **not** make an unenrolled device look
enrolled. Re-enrolment moves a device back once its current operational state is no longer
`available` or `in_use`: its newer placement-relevant Enrolment Job then owns the column.
Drill-through (`latest_in_sequence_workflow` / job ids) still names the overall latest *observed*
Job; a clamped failed cell's failure detail follows the uncleared stage's placement-relevant
execution.

**Per-cell status:** `not_started`, `pending`, `running`, `succeeded`, `failed`. `failed` is a
genuine negative placement result (workflow failure or timeout). Interrupted executions
(`cancelled` / `terminated` / `superseded`) never own the cell status. Note a device
can sit in `single_node_test` / `multi_node_test` with `not_started` (advanced in, stage not
yet run) — so the burn-in results drill-through must key off the stage that actually ran
(`latest_in_sequence_workflow`), not column membership alone.

**Action suppression:** every cell with an empty `available_actions` list carries at least one
typed `action_suppression_reasons` value. These values describe read-path decisions; the Jobs
API remains authoritative for eligibility at submission time.

| Value | Meaning |
| - | - |
| `concurrent_workflow_running` | The newest placement-relevant workflow, or the workflow owning the displayed cell, is pending or running. |
| `workflow_missing` | An otherwise-eligible action target has no registered Workflow row. |
| `no_next_stage` | The device has succeeded the terminal stage in its effective pipeline. |

Reasons can coexist. A customer-ready terminal cell whose newest placement-relevant workflow is
still pending or running reports both `concurrent_workflow_running` and `no_next_stage`: current
operational state owns its green placement without hiding the active Job concurrency signal.

An in-flight cell never offers actions, so a clamped cell whose own stage Job is still pending or
running also reports `concurrent_workflow_running` — including when the newest placement-relevant
workflow is a later stage that already reached a terminal state.

## SLURM mirrors

SLURM node state is stored on `Device.slurm_states` as a token list (e.g. `IDLE`,
`ALLOCATED`, `MIXED`, `DOWN`, `DRAIN`, `MAINTENANCE`, `POWERED_DOWN`) — one base state plus
any number of flags, as slurmrestd reports them. Written by the Slurm agent; freshness
bounded by `SLURM_AGENT_RESYNC_INTERVAL` — see `slurm_state_updated_at`.

The set is **closed**: the agent normalises what slurmrestd reports against
`SlurmNodeState` before writing, and the 30 resulting tokens are published as
`DeviceSlurmStateEnum` and filterable with `?slurm_states=`. Three consequences:

* **Each token stands alone.** A compound string (`DOWN+DRAIN`) is split into its parts and
  the not-responding marker (`DOWN*`) is stripped, so a device is matched by filtering for
  either token.
* **Only wire tokens exist.** The composite words the SLURM CLIs print — `DRAINED`,
  `DRAINING`, `MAINT`, `MIX`, `NO_RESPOND`, `REBOOT`, `REBOOTING` — are `sinfo`/`scontrol`
  presentation, never stored or filterable. A node displayed as `DRAINED` is stored as
  `IDLE`+`DRAIN`, one displayed as `DRAINING` as `ALLOCATED`+`DRAIN`.
* **An unmodelled token is dropped, not stored.** A state a future SLURM release adds is
  left out of the mirror and logged as a warning naming the node and the token, rather than
  failing the sync. Expect a missing token, not a stale row or an outage.

From this snapshot the SLURM condition source derives:

| Condition | `True` when | `False` / `Unknown` example reasons |
| - | - | - |
| `SlurmMember` | Node present in SLURM cluster | `False`: `NotPresentInCluster`; `Unknown`: `SchedulerUnreachable`, `NoLongerObserved` |
| `SlurmSchedulable` | Node up and schedulable | `False`: `NodeDown`; `Unknown`: `NotPresentInCluster`, `SchedulerUnreachable`, `NoLongerObserved` |

`SchedulerUnreachable` is a scheduler outage: the last slurmrestd list failed, so membership is
unknown and the `slurm_*` mirrors are deliberately left at their last-known values.
`NoLongerObserved` is retirement — the device left the Slurm agent's observed scope *and* SLURM
stopped listing it, so nothing can refresh the rows again and the mirrors are cleared.

These conditions feed operational derivation (validation → quarantine when `False`). They do
**not** directly assign `operational_status` without passing through the hub derivation.

SLURM **job** states are tracked separately during hardware-test workflows (`PENDING`,
`RUNNING`, `COMPLETED`, `FAILED`, …) — see burn-in workflow docs.

## Intentionally out of scope

This page documents **stored or derived device/job status axes**. It does not enumerate:

* **Enrollment workflow steps** ([DHCP](../start-here/glossary#dhcp), [BMC](../start-here/glossary#bmc), [1Password](../start-here/glossary#1password) credential writes) — Temporal activities
  with NetBox tags, not FM device columns
* **Location / rack status** enums (`Location.status`, `Rack.status`) — facility inventory, not device lifecycle
* **Connector outbox / event publisher** statuses (`pending`, `in_progress`, `dead_letter`)
* **Test execution / cohort / report-run** statuses — burn-in domain, not device readiness
* **Full SLURM job-state enum** — workflow-scoped, not persisted on `Device`
* **NetBox matrix / [Kusto](../start-here/glossary#kusto) projection** — separate inventory-enrichment pipeline
* **Journey** (deferred ADR concept) — chronological workflow history, not yet a device field

## Quick reference

| Axis | Count | Stored on |
| - | - | - |
| Inventory | 11 | `Device.inventory_status` |
| Operational | 9 | `Device.operational_status` |
| Health | 4 | `Device.health_status` |
| Device condition types | 23 | `device_conditions` table |
| Condition status | 3 | per condition row (`True`/`False`/`Unknown`) |
| Condition sources | 6 | `ironic`, `slurm`, `health-status`, `google-custody`, `nvlink-fabric`, `backend-fabric` |
| Condition domains (`condition_of`) | 3 | per condition row (`operational_status`/`health_status`/`none`) |
| Enrollment completion reasons | 8 | stable ADR-0032 reason codes |
| Triage | 2 (boolean) | `Device.in_triage` |
| Lifecycle event kinds | 3 | `DeviceLifecycleEvent.kind` |
| Ironic provision | 35 | `Device.ironic_provision_state` |
| Job | 10 | `Job.status` |
| Job device | 7 (+ empty) | `JobDeviceStatus.status` |
| Borneo custody | 4 | external → operational |
| Impairment event | 8 | `ImpairmentEvent.status` |
| Impairment device | 8 | `ImpairmentEventDeviceStatus.status` |
| Standup cell | 5 | derived |
| Standup action suppression | 3 | derived |
| SLURM node | 30 | `Device.slurm_states` |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.