Device composite model
UNI resource attribution
DeviceResourceBinding.resolution_status is an independent, observable axis for
materialised attribution to UNI, Placement, and Kubernetes resources. It does
not replace the device’s operational or health status.
Each typed resource exposes its own
status (resolved or stale) and
freshness timestamps. A failed resource retains its last-known row as stale,
while fresh resources remain resolved; readers derive stale state from the
persisted timestamps at read time.
Inventory status
Field:Device.inventory_status — NetBox lifecycle (ADR-0003: Host Statuses).
A cableless node swap moves its failed tray from
active, staged or planned to offline, so
topology-service stops discovering it; any other status is left alone.
NetBox also tracks provisioning progress via tags (not a Fleet Manager column):
Not Started → DHCP Configured → BMC Configured → Ironic Enrolled → Provisioned.
Release admission reads Fleet Manager’s eventually consistent inventory projection. Before any
Fleet Manager-owned customer handoff, the final handoff activity therefore re-reads NetBox and
requires active: adding the customer-ready trait, clearing a maintenance lessee, and writing
operational_status=delegated all refuse a missing or non-active device. This closes the window
where NetBox can move a device to rma after the release job was admitted.
Operational status
Field:Device.operational_status — Fleet Manager’s internal customer-pool readiness
vocabulary. FM defines these enum values and the derivation rules; they are not mirrors of a
single external system’s state machine. Derived from Ironic, Placement, and SLURM conditions
(ADR-0024). The OpenStack and Slurm agents report source conditions and raw subsystem mirrors; the
hub is the only writer of the Ironic/Placement/SLURM-derived status. delegated,
pending_handover, and drained are frozen while Google owns the custody projection,
and an active impairment hold freezes its quarantine — condition observations continue
without overwriting either hold.
pending_handover remains a valid frozen status for existing rows, but has no current writer.
The production Borneo poll persists the credential returned with PARTNER_OWNED and then skips
the post-MVP recovery path, so CUJ3 does not currently change Fleet Manager operational status.
The quarantine → in_use edge (without passing through validation or available) is
intentional: when a maintenance-pool node carries the customer-ready trait and still
has a Nova instance, the reconciler treats it as in_use — the trait is authoritative proof
of customer handover and outranks the maintenance-pool lessee (reconciler.py, maintenance-pool
branch). Same path exists via conditions (CustomerReady=True + InstanceAllocated=True).
ImageNodeWorkflow (FLEET-2453) installs the configured per-role SLURM image and flavour on an
overcloud node without registering it with SLURM or entering the customer pool. Before Nova
creation it moves the Ironic lease from the enrol project to the SLURM project, then creates the
server through the dedicated SLURM credential using the configured SLURM network and keypair.
The node may project as quarantine or validation while the image is installed; neither status
means customer allocation. A retry recovers the server from Ironic’s instance_uuid, or from a
sole exact host-name match in the SLURM Nova project before Ironic has recorded that binding.
Multiple exact-name matches fail closed. Terminal Nova states also fail without automatically
deleting the discovered server.
On a standalone-Ironic site (placement_enabled=false, no Nova or Placement — North Carolina,
Sys2-LON14) ImageNodeWorkflow selects the Ironic-direct backend (ADR-0042) instead. Fleet
Manager mints the instance_uuid itself, then drives Ironic from AVAILABLE through DEPLOYING
to ACTIVE with the per-site configured image, checksum, and config-drive public keys — no Nova
server, no SLURM-project lease move. The resulting InstanceAllocated condition still derives from
the bound instance_uuid, so operational status projects the same as the Nova arm. A fresh
execution mints a new instance_uuid; a node found mid-deploy recovers its in-flight binding,
but an idle AVAILABLE node still carrying a prior instance_uuid is rejected for cleanup rather
than adopted (it may be an undeploy that reached AVAILABLE but has not yet cleared its identity).
DEPLOYFAIL fails closed (deploy-arm undeploy-recovery is a follow-up). UnimageNodeWorkflow
selects the same Ironic-direct backend to tear a standalone-imaged node down: it drives Ironic’s
deleted action (ACTIVE/DEPLOYFAIL → cleaning → AVAILABLE), polls to observed AVAILABLE,
then atomically clears the node’s instance_uuid/instance_info so it derives as un-allocated
again — no Nova server to delete. The disk, config-drive, and injected-key wipe is Ironic
cleaning’s work on that transition, so it only happens when the site conductor runs automated
cleaning with a disk-erase step; Fleet Manager drives deleted and observes AVAILABLE but does
not itself force the erase, so a site that has cleaning disabled returns the node to the pool
without wiping it (an operational prerequisite pinned in the deployment guide, not enforced here). A node already tearing down (DELETING/CLEANING/CLEANWAIT/DELETED) is not
re-fired, and the binding clear is guarded on observed AVAILABLE under the node’s own uuid so a
concurrent rebind cannot be clobbered. The backend is chosen from the site’s current capability;
an Ironic-imaged node must be undeployed before its site gains Nova (ADR-0042 review-trigger).
At a full-OpenStack site the Nova arm runs instead: UnimageNodeWorkflow only deletes the Nova
server identified by Ironic’s instance_uuid; it never
searches Nova by host name. An unbound node is already complete only when Ironic is AVAILABLE.
Otherwise a missing binding, or a bound UUID whose Nova server is missing, fails for explicit
operator repair. After requesting Nova deletion, the workflow waits for the server to disappear
and Ironic to return to AVAILABLE, then returns the lease to the enrol project. A retry accepts
that already-completed enrol-leased state. A bound DEPLOYFAIL is admitted for this explicit
cleanup, and the workflow tolerates that original state while Nova deletion starts. Other Ironic
failure states are rejected before Nova deletion and reported immediately if reached during
teardown.
Image admission requires authoritative maintenance=false and rejects every Ironic failure state,
so Image Node never silently clears maintenance or recovers a failed node as successfully imaged.
Unimage remains an explicit operator action and can remove the exact Nova UUID bound to a
SLURM-leased DEPLOYFAIL node. Undercloud/CPU-hypervisor routing remains out of scope and is
rejected at runtime.
Ironic → operational mapping (simplified)
Health status
Field:Device.health_status — independent telemetry axis from Mimir. Not derived from
operational status or raw subsystem fields.
Customer-ready is a composite predicate (not a stored state), implemented on
Device.is_customer_ready / Device.customer_ready_q():
- Inventory-only devices are never customer-ready (
inventory_only=Trueexcludes them) - Standard pool:
inventory_status=active,operational_statusin{available, in_use}, andhealth_status=healthy - Delegated (Metal Managed):
inventory_status=active,operational_status=delegated, andhealth_statusin{healthy, unknown}— healthy delegated devices are customer-ready; onlydegraded/criticalexclude them
inventory_only
devices from totals and per-status buckets so projection-only inventory does not read as
unhealthy fleet.
Device conditions
Table:device_conditions — multi-source assertions (ADR-0024). Each row is the latest
value for one semantic aspect. Most are refreshed observations; durable lifecycle assertions change
only explicitly. Sources upsert conditions; the hub derives operational_status (and eventually
health_status) only from the conditions assigned to those domains.
Conditions are not a lifecycle state machine — each type is tri-state:
Each row also carries two hub-maintained fields (FLEET-1547):
Condition types and owning sources
The Feeds column is persisted on each row ascondition_of:
CustodyIntent, FrontendPortsInCustomerVrf, NvLinkGpusInRepairPartition, and
BackendPortsInMaintenanceVrf are durable — no TTL or
observation-age expiry; they
change explicitly only. Observation types retain their existing freshness and source-silence behavior.
condition_of=none conditions never participate in operational or health derivation and are never
decisive; eligibility may consume them independently.
Enrollment identity proof
ADR-0038 stores current Enrollment proof directly on the Device:
The workflow-agent writes these fields only through
POST /devices/{id}/enrollment-proof, protected by
the internal enrollment-proof:write scope. A qualifying typed completion establishes
enrolled proof for its result reference. A started non-success submits a non-revoking failure write
guarded by the identity pinned in the workflow: if the Device now carries a replacement
infrastructure_ref, the write safely no-ops; if matching enrolled or legacy proof already exists,
the earlier validation is preserved. Stale proof metadata is cleared when the operated identity still
matches. A device without prior proof remains explicitly at none. Diagnostic success
(run_inspection=false), legacy string results, and failures without a trustworthy pinned identity
make no proof write. The hub also clears proof atomically when a Device leaves enrollment-managed
scope. An in-scope identity change does not need an invalidation write: the eligibility
comparison requires enrollment_validated_ref == infrastructure_ref.
Eligibility gate (ADR-0038). A policy that sets require_enrollment_proof refuses a device
unless enrollment_validated_ref is non-empty and equal to infrastructure_ref — a pure comparison
of the two references (enrollment_proof_source plays no part). It fails closed: an empty validated
ref (new device, failed enrolment, scope re-entry, rejected legacy) and a mismatched ref both block,
emitting enrollment_proof_missing. The flag is set on Move to Test Cluster (Quarantine);
cordon_only is exempt (it removes the customer-ready trait without adopting, so it stays runnable on
an unproven customer node), and the custody-reclaim statuses (delegated, pending_handover) are
exempt exactly as they are for the condition and Ironic-shape gates. Enrollment itself never requires
proof (it produces it), and burn-in and Image Node stay proof-exempt. The gate is evaluated on the
same eligibility pass as every other policy at job creation.
Runtime enforcement (in-activity). Admission is only a snapshot — the submitter starts runs
asynchronously, and a supported infrastructure_ref change afterwards clears proof — so both
customer-facing mutations re-read the Device and re-apply the same comparison inside the activity
that mutates, failing non-retryably with ENROLLMENT_PROOF_MISSING when proof is absent (a failed
proof read is ENROLLMENT_PROOF_UNAVAILABLE instead — retryable, owned by Fleet Manager — so an
API outage never reads as an identity verdict). Move to Test Cluster re-checks inside
adopt_host_activity, just before the SLURM adoption, and inside force_reclaim_host_activity when
force_reclaim is set (it deletes the customer instance before the adopt, so it is bound to the proven
identity too); fully managed Release
re-checks inside the customer-return activities (add_customer_ready_trait /
release_to_customer_pool_activity), beside the NVLink partition-latch preflight. On every enforced path
the proof is read by the mutation itself, immediately before its first external write (the activity
hands the domain service a resolver, so no setup, resolution or wait sits between read and write), and
the write is bound to that identity: the mutation refuses if the Ironic node it resolved does not carry
the proven infrastructure_ref UUID. Release also
must enforce at runtime because the hub can’t see the satellite-injected engagement_mode. Doing it in the
same activity as the mutation keeps them atomic (no queue gap). Adopt and the customer-return
activities always receive device_id and an explicit enforce_enrollment_proof=True; on Release the
NVLink partition latch has its own enforce_partition_latch argument, so proof enforcement never
depends on the latch check. The proof check is keyed on its enforce argument alone: an empty
device_id under enforcement is refused, never skipped.
cordon_only returns before the adopt and is exempt, as are the custody-reclaim statuses. Release keys
on engagement_mode=fully_managed + release_to_customer_pool, so its other paths stay exempt: internal
Release (release_to_customer_pool=false) returns to no customer, and metal / self-managed returns hand
off through their own (non-Ironic) paths.
Front-end VRF custody latch contract
FrontendPortsInCustomerVrf records where a device’s front-end switch ports were last verified
to be. It exists because a VRF flip has no downstream witness the way an enrolment has inspection:
the flip’s own read-back is the only proof, and this latch is where that proof is kept so the
invariant ports must be out of the customer VRF before the device goes to anyone can be enforced
without re-reading the fabric.
The flip reasons were HandoverFlipVerified and ReturnFlipVerified until FLEET-2825; migration
0211 rewrote any existing rows to the new names.
Who reads it.
ENROLLMENT declares this type in
EligibilityPolicy.disqualifying_condition_types — the blocking complement of
required_condition_types — so an enrolment job is refused while the latch reads True or
Unknown, and admitted on False or absence. The refusal surfaces at job validation as
condition_disqualifier_asserted / condition_disqualifier_unproven carrying the failing
condition_type, and is cleared by running the front-end VRF isolate, which writes False. ENROLLMENT
is the only gated workflow: RELEASE also hosts the CUJ2 handover branch, whose precondition is the
opposite one, and release-to-pool and CUSTOMER_RELEASE are recorded as unreachable for a
latch-carrying device rather than gated (spec 0045 §6). The handover-side wall is a preflight inside
the CUJ2 PATCH and belongs to FLEET-2253.
Absence means the same thing as False to a reader — no verified observation places the ports in a
customer VRF. That makes absence the permissive state, so a device that is already delegated
must not be left without a row: the Google connector writes the latch whenever the row is missing,
not only on the status transition, which lazily backfills every device already in custody within one
backstop poll of deploying (reported as HandoverBreach, because no verified flip is on record for
them). On an actual transition into delegated, a latch-write failure is fail-closed: the exception
prevents the operational-status write, leaving the device in the poll’s target set so the entire
transition is retried. Only missing-row repair for a device already in delegated is best effort: a
failed repair is logged and retried on a later poll. A device whose row already exists and whose status
is unchanged is left completely alone, so the breach is not re-logged on every poll.
The custody-entry write is hub-ordered: the hub stamps observed_at and the write is exempt from
source-observation staleness. This type has two writers whose
clocks are independent — one on the hub, one on a satellite — and the API tolerates five minutes of
future skew, so a future-dated flip observation would otherwise silently stale-reject a later
custody entry and leave the permissive value standing. The flip writers stay observation-ordered
(they are reporting what they read from the switch at a moment) and verify the row the hub returns:
a 200 whose body is the stored row rather than the submitted one means the write was stale-rejected,
and the activity fails rather than reporting a latch it did not record.
Both flip workflows write the latch only when every per-switch result reports simulated=false.
A simulated run verified nothing, and a simulated return that cleared the latch would remove the
only record standing in front of a device whose ports never left the customer VRF. A verified flip whose latch write fails after retries fails the job with
SWITCH_VRF_LATCH_WRITE_FAILED — distinct from SWITCH_VRF_FLIP_FAILED because the ports did
move; the recovery is to re-run the same flip job, which no-ops on the already-applied switch and
writes the latch.
On the inbound side the write happens on Borneo GOOGLE_OWNED — on the custody-entry transition, and
on any poll that finds no latch row (see above) — and never on PARTNER_OWNED or DRAINED, because a
return is proven by the front-end VRF isolate, not by Borneo’s word. A HandoverBreach is logged at error level
as the tripwire. Two breaches are expected rather than defects: an in-place maintenance cycle
(delegated → drained → delegated) re-enters custody with the latch already Unknown, because the
last verified observation is genuinely gone; and every device already in custody when this shipped is
backfilled as a breach, because no verified flip is on record for it. The answer is to re-run the front-end VRF return before the CUJ2 PATCH, which is why
drained is in that workflow’s eligibility allowlist. Operator procedure lives in the
docs/runbooks/switch-vrf-flip.md runbook.
NVLink repair-partition latch contract
NvLinkGpusInRepairPartition is the durable record of the partition membership a real NVLink move
just verified. It is owned solely by nvlink-fabric and has no derivation role.
The customer-facing
RELEASE handoff reads this latch: True and Unknown refuse the handoff,
while False and absence are permissive. A SLURM-only Release remains available because it does not
return the tray to customers. Simulated runs write nothing. The handoff reads the durable latch
immediately before each customer-facing mutation; a queued release therefore cannot retain
authorization after an isolate writes Unknown or True. The Unknown write is inside the real
move activity after its second custody check, planning, and final drift check, immediately before the
first NMX-M write, so a pre-write refusal cannot leave a false latch. A
verified move whose post-write fails after retries ends with NVLINK_LATCH_WRITE_FAILED; re-run the
same move, which converges from live membership and records the latch. The condition is keyed to the
Device, so it survives the tray swap it describes until a verified return clears it.
Back-end VRF latch contract
BackendPortsInMaintenanceVrf records whether a host’s RoCE back-end ports may be in the
maintenance VRF (spec 0064). It has the NVLink latch’s polarity and lifecycle, is owned solely by
backend-fabric, and has no derivation role.
The back-end VRF flip workflows (spec 0064) write it on a real run, at a site that sets
backend_vrf_flip_allowed. Each apply records Unknown after its checks pass and just before it
writes a switch with a port to move; once every switch verified, isolate records True and return
False. A failed run leaves whatever the applies wrote, so Unknown holds until a rerun records a
definite value; a simulated run writes nothing. At sites without real writes, an operator records
hand isolations by following the release wall in docs/runbooks/backend-vrf-flip.md, with the
runbook’s own reasons (ManualIsolation, ManualReturn).
The customer-facing RELEASE handoff reads it exactly as it reads the NVLink latch: True and
Unknown refuse, False and absence permit, and a SLURM-only Release is unaffected. Admission
refuses through the release_to_customer_pool gate. In flight, each hand-back activity re-reads both
latches before its customer-facing mutation: the Metal hand-back always does, and the fully managed
hand-backs do when the Release run carries enforce_partition_latch (set from
release_activity_args_ready). A back-end refusal fails with RELEASE_BACKEND_VRF_LATCH_REFUSED. The condition is keyed to the Device, while VRF membership
belongs to switch ports, so a replacement tray from a Node Swap can inherit isolated ports with no
row.
When conditions are emitted
Rules below mirroragents/openstack/conditions.py and agents/slurm/reconciler.py. Absence
of a row is treated as Unknown at derivation time.
Health conditions use source-specific reasons such as
RedfishHealthCritical,
RedfishTelemetryUnavailable, NvidiaGpuHealthDegraded, and NvidiaFpgaTelemetryUnavailable; every
message opens with the exact Mimir series selector. The source remains health-status; provenance
belongs in the reason and message. Every host in scope carries all ten rows every cycle, like the
Ironic and SLURM sources, so nothing is remembered between cycles. These conditions explain Device.health_status; they do not drive it. health_status stays
computed from the chassis_health composite, so degraded may be explained by a condition that says
either Degraded or Critical.
Operational-status derivation (ordered rules)
Hub-owned pure derivation (core/conditions/derivation.py) — same semantics as the legacy Ironic
reconciler, with one deliberate SLURM divergence. This is a priority-ordered classifier, not a
transition engine. First matching row wins:
Materialisation and reconciliation:
POST /api/v1/device-conditions/apply/ applies one
source’s full condition snapshot and always verifies the materialised projection, including
freshness-only resyncs where condition values have not changed. A matching operational_status
skips that status-column write, while decisive flags, entered_service_at, and failure metadata
are still reconciled. A stale projection is repaired. The single-condition
POST /api/v1/device-conditions/ path re-derives only after a semantic change (create, status,
reason, or message); freshness-only and stale-ignored single-condition writes skip derivation.
External-hold freeze: while operational_status is delegated, pending_handover, or
drained, or the device carries an active impairment-event hold, the hub does not re-materialize
operational status from conditions. Conditions and decisive flags still update; only the top-level
write is frozen. A drained device therefore remains in Ironic reconciliation scope for its bounded
maintenance window without losing the Google-owned custody projection. The custody part is the V1
guardrail ported to the hub until CustodyIntent derivation ships.
Use GET /api/v1/devices/{id}/ (device retrieve) — field operational_status_conditions shows
which conditions were decisive for the current operational status, read from the persisted
decisive flag so it always agrees with the /api/v1/device-conditions/ list (filterable by
condition_of and decisive) and the materialised status.
Lifecycle event log
Table:DeviceLifecycleEvent — the append-only history of the three stored status axes
above. The status columns themselves are classifiers that hold only the current value; this
table is what answers “how did this device get here and how long has it been here”.
How rows are written.
Device.save() diffs the tracked columns against the values it loaded
and appends one row per changed column in the same transaction as the device write. Writers that
bypass save() (the impairment cordon/release bulk_update) append rows through the same
Device.lifecycle_events_for_changes() diff. A device gets one birth row per kind on creation.
Devices that predate the table received one baseline row per kind at migration time, so every
timeline has a “this value since at least here” starting point. There is no stored previous
value: the prior state is the previous row of the same kind. Rows are never updated or deleted
by application code.
Reading it. GET /api/v1/devices/{id}/lifecycle-events/ (scope device:read) returns the
device’s rows newest first, optionally narrowed with ?kind=. Each row carries ended_at, the
created_at of the next row of the same kind, computed with a window function so it is correct
across pages; ended_at is null on the row describing the current value, whose duration is
measured to now. Triage is not tracked here: Device.entered_triage_at already answers “in
triage since”.
Integration points
Fleet Manager aggregates many external systems. Not all of them own a device status field — some write conditions, some write raw mirrors, some trigger workflows, some only read FM state outward.Impairment events (separate from device conditions)
Impairment events are created throughPOST /api/v1/operations/impairment-events/; Radar does not
call it. FM models these as ImpairmentEvent rows — a cordon workflow, not a persisted device
status axis.
Event status: accepted → cordoned → cleared (or partially_failed / failed; release
sub-states: release_pending, release_retrying, release_failed)
Per-device status within an event: pending → cordoned → cleared (or failed / retrying /
release sub-states)
Effect on devices: creating an event sets an impairment-event-id hold annotation and
operational_status=quarantine, which the hold freezes; the OpenStack agent’s cordon removes the
customer-ready trait. Clearing the event queues release work, which restores the trait for a pre-hold
available or in_use host, removes the hold and re-derives the status. The release does not read
the fabric isolation latches.
Radar outbound projection
The Radar ingestion agent reads FMinventory_status, operational_status, and health_status
and maps them to Radar ResourceAvailabilityState / ResourceHealthState. This path does not write
back to FM device fields.
Tenants can carry annotations.radar_ignore=true to opt out of projection (with required reason).
Triage flag
Field:Device.in_triage — orthogonal boolean. A device can be in triage regardless of
inventory, operational, or health status. Set via POST /api/v1/devices/triage/.
Ticket pointers (caller-managed, both blank=True strings rendered as null when empty):
Device.triage_ticket_url— the ticket tracking the current triage cycle.Device.parent_ticket_url— the device’s persistent umbrella ticket. Set once and left alone across triage cycles; everytriage_ticket_urlopened for the device is expected to nest under it, so it accumulates the device’s full triage history independently oftriage_ticket_url’s own lifecycle.
Inventory-only flag
Field:Device.inventory_only — orthogonal boolean (default False). Marks a core
Device that exists solely so DeviceInventoryProjection / customer connectors can publish
inventory without Fleet Manager managing the hardware.
- Site-driven inheritance:
Site.inventory_only(a model-level boolean, immutable after create) is the authoritative source. On device create,Device.save()calls_inherit_inventory_only_from_site()which forcesinventory_only=Truewhen the site carries the flag — regardless of what the writer passed. The NetBox agent sets the site flag from deployment config (netboxAgent.inventoryOnlySites); the Matrix inventory seeder sets it per-device. - Immutable via ordinary API/model updates; promote-to-managed is an explicit audited operation.
- Always ineligible for workflows (eligibility
DEVICE_INVENTORY_ONLY); rejected byAllocation.validate_devices; excluded from burn-in target selection; blocked by the workflow-agent submitter before Temporal start. - Excluded from customer-ready and from per-site operational/health count aggregations.
- Operational/health stay at their defaults (
unknown/unknown) by design — nothing derives them for these rows. Enrollment is job-gated, so a Device row alone is never enrolled.
Job status
Field:Job.status — workflow execution aggregate across all devices in the job.
Terminal values will not transition further.
Job device status
Field:JobDeviceStatus.status — per-device execution within a job. A row is created at
submit; the status is empty until Temporal starts that device (there is no pending
value).
Failure codes
Field:JobDeviceStatus.failure_code — a stable, machine-readable reason for a failed
device workflow. It is separate from JobDeviceStatus.status: status=failed says the
execution ended unsuccessfully, while failure_code routes the failure to the right
triage queue and failure_message gives its host-specific detail.
failure_code is a large, fast-growing set (one value per distinct failure mode), so the
canonical list of codes, their meaning, and their triage owner is not duplicated here —
see CODE_OWNER in
src/fleet_manager/core/workflow/failure_modes/taxonomy.py, or the FailureCodeEnum
component in docs/openapi.yaml for the values exposed over the API.
For example, IRONIC_INSPECTION_IPA_IMAGE_NOT_FOUND means Ironic selected an IPA kernel or
ramdisk path that was absent. Operators should compare the NetBox and Ironic cpu_arch,
node-level deploy-image overrides, per-architecture conductor configuration, and artifact
presence on every conductor before retrying one node.
IRONIC_DRIVER_MISMATCH means an Ironic power cycle was requested for a node that does not
use the required ipmi driver; switch the node’s Ironic driver before retrying.
The detailed operator procedure extends the existing
docs/runbooks/ipa-image-update.md runbook.
Ironic provision state
Field:Device.ironic_provision_state — raw OpenStack Ironic mirror. Stored as a plain
string so new upstream states do not require a migration; the API enforces the closed set at
the boundary.
All states
Stable states (require an explicit verb to leave):
enroll, manageable, available,
active, deploy failed, clean failed, error, inspect failed, adopt failed,
rescue, rescue failed, unrescue failed, service failed.
Failure states (map to operational_status=failed): deploy failed, clean failed,
inspect failed, rescue failed, unrescue failed, adopt failed, service failed,
error.
Diagram node IDs abbreviate Ironic API values where spaces are invalid in mermaid (e.g.
deploy_failed on the diagram = deploy failed in ironic_provision_state).
For the full canonical diagram, see the
OpenStack Ironic state diagram.
Borneo custody
Google Borneo custody states overlay operational status during delegated-device workflows. The deployed connector poll and its inbound helper are not equivalent forPARTNER_OWNED:
the poll persists the returned credential and exits before calling handle_state_change.
No current production path writes Fleet Manager
pending_handover.
Standup pipeline (derived UI view)
For a concise day-to-day explanation, start with DC Standup Pipeline. This section defines the exact derivation. The Operational Racks matrix does not store pipeline state on the device. Placement is derived per request fromoperational_status and job history.
Columns — the real stage a device is at or awaiting, not the last it completed:
enrolment, quarantine, single_node_test, multi_node_test, release. A fresh device is
column=enrolment, status=not_started; not_started is a status, never a column.
The effective sequence is derived per device. Policy-backed GPU compute uses the full chain.
CPU bare-metal always includes quarantine (Move to Test Cluster / SLURM) even without an
acceptance policy — enrolment → quarantine → release; a policy adds single-node acceptance.
CPU hypervisors terminate at enrolment.
Switches have no pipeline: they are excluded from device cells, and a rack of only switches is
omitted from the response entirely.
Placement precedence: current customer handover state is the first decision. If
operational_status is available or in_use, the device is placed at its effective terminal
stage with status=succeeded (release for customer hosts, enrolment for hypervisors). This
override is independent of Job timestamps and outcomes: newer failed, pending, or running Jobs
remain available through drill-through context but do not move the cell. Acceptance-gate summaries
also remain visible as historical evidence. The override skips both gate holds and the prefix
clamp.
If the device is not currently available or in_use, placement uses Job history as described
below, then advances successful stages and applies the prefix clamp.
Advance-on-success: when a device succeeds a stage it is shown in the next column as
not_started (a completed enrolment appears under quarantine). Advancement out of
single_node_test / multi_node_test is held — the device stays succeeded in that
column — until its burn-in acceptance gate passes (“site conditions”). multi_node_test
composes across every configured multi-node collective gate (e.g. an nvlink-domain rack
gate and a pod gate): the device advances only once all of them pass. Any configured gate
that can’t yet be evaluated (missing evidence / topology) holds too. The matrix advances the
column only; the next workflow still runs on an operator click.
How Job history places a device
For devices without the customer-ready override, cancelled, terminated, and superseded attempts
are ignored for placement but kept in Job history. The newest execution is retained for Job
drill-through; placement uses the newest execution that is still active or established a result.
An interruption is not a success. If the stage has no earlier successful result, it remains
incomplete. If it does have an earlier success, the interruption does not revoke it. The
prefix clamp then prevents the cell from appearing beyond the first incomplete stage.
available or in_use: its newer placement-relevant Enrolment Job then owns the column.
Drill-through (latest_in_sequence_workflow / job ids) still names the overall latest observed
Job; a clamped failed cell’s failure detail follows the uncleared stage’s placement-relevant
execution.
Per-cell status: not_started, pending, running, succeeded, failed. failed is a
genuine negative placement result (workflow failure or timeout). Interrupted executions
(cancelled / terminated / superseded) never own the cell status. Note a device
can sit in single_node_test / multi_node_test with not_started (advanced in, stage not
yet run) — so the burn-in results drill-through must key off the stage that actually ran
(latest_in_sequence_workflow), not column membership alone.
Action suppression: every cell with an empty available_actions list carries at least one
typed action_suppression_reasons value. These values describe read-path decisions; the Jobs
API remains authoritative for eligibility at submission time.
Reasons can coexist. A customer-ready terminal cell whose newest placement-relevant workflow is
still pending or running reports both
concurrent_workflow_running and no_next_stage: current
operational state owns its green placement without hiding the active Job concurrency signal.
An in-flight cell never offers actions, so a clamped cell whose own stage Job is still pending or
running also reports concurrent_workflow_running — including when the newest placement-relevant
workflow is a later stage that already reached a terminal state.
SLURM mirrors
SLURM node state is stored onDevice.slurm_states as a token list (e.g. IDLE,
ALLOCATED, MIXED, DOWN, DRAIN, MAINTENANCE, POWERED_DOWN) — one base state plus
any number of flags, as slurmrestd reports them. Written by the Slurm agent; freshness
bounded by SLURM_AGENT_RESYNC_INTERVAL — see slurm_state_updated_at.
The set is closed: the agent normalises what slurmrestd reports against
SlurmNodeState before writing, and the 30 resulting tokens are published as
DeviceSlurmStateEnum and filterable with ?slurm_states=. Three consequences:
- Each token stands alone. A compound string (
DOWN+DRAIN) is split into its parts and the not-responding marker (DOWN*) is stripped, so a device is matched by filtering for either token. - Only wire tokens exist. The composite words the SLURM CLIs print —
DRAINED,DRAINING,MAINT,MIX,NO_RESPOND,REBOOT,REBOOTING— aresinfo/scontrolpresentation, never stored or filterable. A node displayed asDRAINEDis stored asIDLE+DRAIN, one displayed asDRAININGasALLOCATED+DRAIN. - An unmodelled token is dropped, not stored. A state a future SLURM release adds is left out of the mirror and logged as a warning naming the node and the token, rather than failing the sync. Expect a missing token, not a stale row or an outage.
SchedulerUnreachable is a scheduler outage: the last slurmrestd list failed, so membership is
unknown and the slurm_* mirrors are deliberately left at their last-known values.
NoLongerObserved is retirement — the device left the Slurm agent’s observed scope and SLURM
stopped listing it, so nothing can refresh the rows again and the mirrors are cleared.
These conditions feed operational derivation (validation → quarantine when False). They do
not directly assign operational_status without passing through the hub derivation.
SLURM job states are tracked separately during hardware-test workflows (PENDING,
RUNNING, COMPLETED, FAILED, …) — see burn-in workflow docs.
Intentionally out of scope
This page documents stored or derived device/job status axes. It does not enumerate:- Enrollment workflow steps (DHCP, BMC, 1Password credential writes) — Temporal activities with NetBox tags, not FM device columns
- Location / rack status enums (
Location.status,Rack.status) — facility inventory, not device lifecycle - Connector outbox / event publisher statuses (
pending,in_progress,dead_letter) - Test execution / cohort / report-run statuses — burn-in domain, not device readiness
- Full SLURM job-state enum — workflow-scoped, not persisted on
Device - NetBox matrix / Kusto projection — separate inventory-enrichment pipeline
- Journey (deferred ADR concept) — chronological workflow history, not yet a device field