> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Enrollment Workflow

> What the enrollment workflow does to NetBox, DHCP, the BMC, and Ironic — the living domain-logic reference for reviewing enrollment changes.

> **Living reference.** This page describes what the enrollment workflow *does* to [NetBox](../start-here/glossary#netbox), [DHCP](../start-here/glossary#dhcp), the [BMC](../start-here/glossary#bmc), and [Ironic](../start-here/glossary#ironic) — for review by the people who own those systems. Any PR that changes the workflow's observable behaviour (step order, decisions, side effects, params, failure semantics) must update this page in the same PR. It documents the **latest logic only** — the path a workflow started today takes; rollout and back-compat mechanics are deliberately absent.

**Enrollment turns a racked, NetBox-registered host into a schedulable Ironic node in `available`: it configures DHCP and the BMC, then enrols, inspects, and cleans the node in Ironic — verifying NetBox's recorded intent (addressing, cabling, hardware inventory) against what the infrastructure and hardware actually report.**

* **Systems touched**: NetBox (reads + three writes: [device](../start-here/glossary#node) status `staged → active`, tags, and the `platform_class` custom field when it is empty), the DHCP server (verified SSH upload + service restart when configuration changes), the BMC ([Redfish](../start-here/glossary#redfish); full vendor-specific configuration), Ironic (node/portgroup/port creation, `manage`/`provide`/`abort` provisioning actions, power on), Fleet Manager (after terminal completion the workflow-agent establishes or submits an identity-guarded failure write for the Device's identity-scoped Enrollment proof). [Nova](../start-here/glossary#nova) is never called. The DHCP and BMC steps run *before* the Ironic ownership guard, so a rerun on a deployed host still reconverges those settings.
* **Hardware identifiers**: `resource_class` remains the operator-owned, customer-facing scheduler resource. Enrollment records the distinct, Fleet Manager-owned per-device `platform_class`. Burn-in reads it to derive GPU-count and GPU-name test parameters that a test's params leave unset from the device's hardware descriptor; a test that needs such a value on a device without a recorded `platform_class` fails. Threshold and topology selection still use `resource_class`. [Phase 0b](#0b-platform-class) defines its format, gate, and write policy.
* **Terminal proof safety**: a well-formed qualifying completion establishes proof only after Fleet Manager's projected Ironic reference matches the resolved reference. The workflow agent retries that compare-and-set for a bounded period derived from [Temporal](../start-here/glossary#temporal) close time, then fails proof closed with `ENROLLMENT_COMPLETION_PROOF_REJECTED`. Every later reference change atomically clears proof, so an `A → B → A` cycle cannot resurrect old validation. Failure, cancellation, termination, timeout, or malformed qualifying output submits an identity-guarded failure write: matching current proof is preserved, stale metadata is cleared when the operated identity still matches, and a replacement identity is untouched. Diagnostic and unclassifiable malformed completions preserve existing proof without a write.
* **Idempotent, but not non-disruptive on a deployed node.** Every step re-reads live state and no-ops when already satisfied, and a rerun auto-recovers `inspect failed` / `clean failed` (`deploy failed` / `error` abort and need a human). But BMC convergence runs before the Nova guard and can restart a Lenovo or iDRAC host whose BIOS settings drifted — do not rerun enrollment against a live workload.
* **End state** (all phases enabled, fresh unowned node): Ironic node `available` and powered on, NetBox device `active` and tagged `ironic` (configurable slug) + `ipmi_exporter`. A node found already **enrolment-satisfied** — `active`, `available`, or `manageable` with an `instance_uuid` — skips all enrolment mutations and is re-tagged; an unowned `available` node is additionally ensured powered on.
* **Verification is a hard gate**: inspection data must match NetBox (LLDP vs cables, [GPU](../start-here/glossary#gpu)/CPU/disk/memory/BIOS checks), and — where the BMC can report link state — a frontend link must come up; unknown link telemetry deliberately fails open (phase 5). Mismatches fail the enrolment as operator-actionable errors rather than letting a miscabled node into the pool.

## High-level flow

```
routing preflight ─▶ NetBox status gate ─▶ platform class‡ ─▶ DHCP config* ─▶ BMC config* ─▶ NetBox staged→active ─▶ Ironic phases* ─┐
   ┌─────────────────────────────────────────────────────────────────────────────────────────┘
   ▼
inspection half ──────────────▶ [Redfish→IPMI driver switch]† ─▶ cleaning half ──────────────▶ power on +
(ENROLL → MANAGEABLE → INSPECT)                                  (PROVIDE → CLEANING → AVAILABLE) 'ironic' tag
```

Strictly sequential. The routing preflight, the status gate and the NetBox activation always run; the three `*` groups are gated per site (`enrollment_configure_dhcp`, `enrollment_configure_bmc`, `enrollment_ironic_enrol`); `†` is the per-site `switch_to_ipmi_after_inspection` option; `‡` runs once the per-site `platform_class_activity_ready` worker-readiness gate is on (phase 0b).

Ironic provision-state walk for a fresh node: `ENROLL` → `MANAGEABLE` (manage) → inspection → `MANAGEABLE` → `CLEANING`/`CLEAN WAIT` (provide) → `AVAILABLE`. The workflow drives the transitions; Ironic does the actual inspecting and cleaning.

## Phase details

### Routing preflight (before any activity)

The workflow resolves which Ironic instance owns the device from the request's `device_class`, before the NetBox status gate and before any activity runs. Two cases fail here: the request carries no `device_class` at all, or it carries one that no Ironic instance manages (e.g. `SWITCH`).

Both fail non-retryably with failure code `IRONIC_ROUTING_UNRESOLVED`, recorded against phase `resolve_ironic_instance` and owned by INFRA\_ENG. **Operator action**: fix the device role in NetBox so it carries a managed `device_class`, or dispatch a workflow that supports this device class, then rerun. No NetBox, DHCP, BMC, or Ironic state is touched before this point, so a failure here is always safe to rerun.

This is a routing/metadata problem rather than an engineering defect, which is why it is a classified operator-actionable failure and not an unhandled error: an unclassified exception raised here would fail the Temporal workflow *task*, leaving the job RUNNING under indefinite retry with no failure code for an operator to act on.

### 0. NetBox status gate

Device must exist in NetBox with status `staged` or `active`; anything else fails non-retryably. (Job submission is separately gated by an API-side eligibility policy — Fleet Manager inventory status `active`/`staged`, compute/hypervisor device classes only, not in [customer custody](../start-here/glossary#customer-custody), and placed in a rack U, so a device NetBox has in its rack without a position is refused with `device_not_placed` until the NetBox agent syncs its U — but that reads FM's eventually-consistent projection; this NetBox check is the workflow's own authoritative read, and everything later re-validates against live Ironic/BMC state.)

### 0b. Platform class

The workflow computes the device's ADR-0028 `platform_class` — `<GPUTYPE>_<DEVICETYPE>_<FABRICTYPE>_<NN>`, the key hardware descriptors and burn-in parameter resolution are scoped by — and records it in both systems of record. It runs before any host-touching step so a NetBox data gap fails in the first minute, where a rerun after the fix is cheap.

**Worker-readiness gate.** The step runs only when the request's `platform_class_activity_ready` metadata is true. The workflow agent snapshots that from the per-site `workflowAgent.platformClassActivityReady` chart value at submission, so a flip cannot change a running execution. Keep it false while old and new Temporal workers share the task queue — an old worker polled for `record_platform_class_activity` fails it as unregistered — and turn it on per site once the rollout has completed and a dry run of `fleetctl netbox backfill-platform-class` shows the site's `gpu_type` coverage is complete. While it is false the step is skipped and no key is recorded.

Inputs, all read live from NetBox:

* **GPUTYPE** — the device type's `custom_fields.gpu_type` (e.g. `GB300`). A **GPU compute** device whose device type has no value fails; an empty value on a CPU baremetal or hypervisor device resolves the literal `NOGPU`.
* **DEVICETYPE** — the device type's manufacturer name and model, each upper-cased with every non-alphanumeric character stripped, then concatenated (`Lenovo` + `HG635N V2` → `LENOVOHG635NV2`).
* **Length** — a computed key, or a populated NetBox value being kept, longer than 128 characters is unresolvable: NetBox would accept it and Fleet Manager would refuse it, splitting the two systems.
* **FABRICTYPE** — derived from `config_context.ztp` **for GPU devices only**: no `ztp` block → `NORDMA`; a block with an `infiniband` section → `IB`; a block that is present (including an empty `{}`) but has no `infiniband` section fails, because a guessed fabric would select the wrong descriptor. The RoCE marker is not yet pinned (ADR-0028 review trigger). A `NOGPU` device never has its `ztp` block read and is always `NORDMA`; revisit if a non-GPU compute node is ever RDMA-backed.
* **NN** — always `01`. There is no collision handling.

Write policy: when NetBox `custom_fields.platform_class` is **empty**, the computed key is written with a PATCH. When it is **already populated and equal**, nothing is written. When it is **populated and different**, NetBox is **not** overwritten — the field is read-only for humans (INFENG-1035), so a divergent value is almost always one Fleet Manager wrote from earlier inputs — a warning is logged, and the NetBox value is the effective one. The Fleet Manager `Device.platform_class` is then set to the effective value only if it differs. A rerun on an unchanged host therefore performs no writes.

Failure codes: `PLATFORM_CLASS_UNRESOLVABLE` (non-retryable, INFRA\_ENG) when NetBox data cannot produce a key — **operator action**: populate `gpu_type` on the device type or fix the `ztp` block, then rerun; `PLATFORM_CLASS_RECORD_FAILED` (retryable, FLEET\_MANAGER) when NetBox or the Fleet Manager API errors while recording it — check that the `platform_class` custom field exists on this site's NetBox.

**Backfill and correction.** Hosts enrolled before this step existed are backfilled with `fleetctl netbox backfill-platform-class`, which runs the same resolver and is also the correction path, since enrolment runs once per lifecycle. A NetBox fix after a failed enrolment needs nothing extra: rerun enrolment or the backfill. A key that was written and is later found wrong is corrected by clearing `custom_fields.platform_class` in NetBox and running the backfill for that device by name; the resolver never overwrites a populated value on its own. There is no periodic re-derivation today; see ADR-0028 for the reconciliation options if that changes.

### 1. DHCP configuration

The workflow reads the device's management MAC and [out-of-band](../start-here/glossary#oob) address from NetBox, renders its DHCP entry, and uploads that file to the DHCP-owned remote configuration directory. The DHCP adapter resolves the private key and `known_hosts` data from the runtime vault's `ssh-config-dhcp` item, uses the key only for the lifetime of the client context, and requires host-key verification. Connection, login, each remote command, and each SFTP transfer have independent configurable deadlines (120 seconds each by default).

The upload is content-idempotent: a matching remote SHA-256 hash skips both transfer and service restart. Changed or unavailable remote content is uploaded, then the DHCP service is restarted and validated using the existing service-specific logic. Completed hash or remote-path preparation commands with non-zero status retain the existing best-effort behaviour; SSH authentication, host-key, connectivity, timeout, command-transport, and transfer failures stop the activity. The existing DHCP Temporal failure classification remains unchanged.

### 2. BMC configuration

Vendor is detected from the Redfish service root, then the vendor strategy runs. The fullest flows (Lenovo XCC, Dell iDRAC), in order:

1. **Password setup** — ensures the per-host BMC admin credential in [1Password](../start-here/glossary#1password) works, rotating the BMC password if needed.
2. **Cold reset** of the BMC — only when the password was rotated (some BMCs require it to apply credentials) or when `force_bmc_restart=true` is passed; waits for Redfish to come back.
3. **Settings convergence** — hostname, DHCP-on-BMC-NIC, security hardening, NTP + syslog (targets from the NetBox site), BIOS/hardware settings, boot order (the relative order of individual settings varies by vendor). On iDRAC the boot-order step is the same **boot restriction** as AMI (see below) when the provisioning NIC is known and `restrict_boot` is set — it disables every other NIC BootOption and promotes the provisioning NIC first, subsuming and replacing the label-based reorder; without a known provisioning NIC or with `restrict_boot=false` it falls back to that reorder and disables nothing. On iDRAC, BIOS and boot-order changes are staged as a job that only applies on the next boot, so those two steps wait for the Lifecycle Controller to schedule the job before requesting the host reset — resetting sooner lets the boot walk past the job, which then stays queued and rejects every later settings write. A job that never schedules **fails** the step and is deleted so a rerun can stage it again; a job some other reboot already carried into the Lifecycle Controller is left to finish without a further reset.
4. **[IPMI](../start-here/glossary#ipmi) monitoring user + IPMI-over-LAN** — creates/updates the shared `ipmi_exporter` monitoring user. Missing `ipmi_exporter` credentials in 1Password **fail** BMC configuration; only a device model with no IPMI-user endpoint skips this step. The user is for monitoring only; Ironic always authenticates with the per-host admin credential, even on the `ipmi` driver.

**AMI MegaRAC** keeps password setup + cold reset but converges a smaller set: hostname, NTP, IPMI user, virtual-media enablement, and **boot restriction** — no security, syslog, or BIOS convergence. Boot restriction disables every NIC BootOption except the single provisioning NIC (the one Ironic PXE-enables — the first frontend interface of the host's `interface_type`, resolved by the same rule so the two never disagree, matched by MAC against each BootOption's `UefiDevicePath`) and promotes that NIC to the front of the boot order, ahead of the disk. This stops the host wasting boot attempts on NICs it cannot provision over and stops it silently falling back to a stale disk image. Disk, virtual-media, and shell options are left bootable. It **fails closed**: if the provisioning NIC's BootOption cannot be positively identified, nothing is disabled. iDRAC runs this same restriction over its own Redfish path (per-option `BootOptionEnabled`, then the staged `OnReset` reorder job). The restriction is on by default and shared across vendors; the per-satellite `restrict_boot` toggle (`TEMPORAL_WORKER_ENROLLMENT_RESTRICT_BOOT`) opts a deployment out. Opting out is vendor-specific: iDRAC then runs its label-based reorder (nothing disabled), while AMI — which has no separate reorder step — skips boot configuration entirely. **Supermicro** is a smaller subset again (hostname, NTP, syslog, IPMI user — no password setup; reset only when forced). On success the device gets the NetBox `ipmi_exporter` tag **regardless of which steps its vendor ran** — the tag means "BMC configuration completed", not "monitoring user exists".

**Busy BMCs.** A configuration write answered with a 5xx is the BMC being busy, not refusing: an AMI BMC answers boot writes with 503 while its host is booting. The step retries it within its budget, and if the BMC is still busy it fails as `BMC_HTTP_SERVICE_UNAVAILABLE` with the BMC's own reason in the message; rerun once the host has booted.

### 3. NetBox activation

Moves the device `staged → active`. Deliberately **not** gated by any step toggle: it is the only path from `staged` back to `active`, and unenrolment leaves a device `staged`, so a deployment with BMC configuration disabled would otherwise never get its re-enrolled nodes back. Only a `staged` device is updated; any other current status is left untouched.

### 4. Ironic enrolment — shared prelude

Both Ironic halves (inspection, cleaning) start from the same idempotent prelude, so either can be retried or re-run standalone. How the prelude dispatches on the node it finds:

```mermaid theme={null}
flowchart TD
    resolve["resolve NetBox device + BMC creds;<br/>get-or-create Ironic node"] --> state{"node state?"}
    state -->|"active / available /<br/>manageable + instance_uuid"| satisfied["enrolment satisfied —<br/>no mutations; live-verify<br/>cached inventory when<br/>run_inspection"]
    state -->|"instance_uuid in<br/>any other state"| nova["fail: Nova-owned node,<br/>refuse all mutation"]
    state -->|"cleaning / clean wait<br/>(cleaning starter only)"| pollback["hand back to the workflow<br/>cleaning poll loop"]
    state -->|"inspect failed /<br/>clean failed"| recover["auto-recover via manage<br/>(clean failed: clear maintenance,<br/>restore it if recovery fails)"]
    state -->|"deploy failed / error"| human["fail: needs human review"]
    state -->|"enroll / manageable"| converge["converge creds, arch trait,<br/>RAID, root_device hint"]
    recover --> converge
```

The details:

* Re-resolve the NetBox device (`oob_ip` + config-context `ironic_settings` required) and the per-host BMC credentials.
* **Get-or-create the Ironic node.** On create: driver and interfaces from `ironic_settings`; `driver_info` is Redfish (address from `oob_ip`, or `redfish_address_override` for emulated nodes) or IPMI depending on `ironic_settings.driver` — both use the per-host BMC credential; `owner`/`lessee` set to the enrol project; `resource_class`, `capabilities`, `cpu_arch`, root-device hint from NetBox. `raid_interface` is mandatory. Optional `network_interface` is passed through when set — a standalone-Ironic (no-Neutron) site sets `"noop"` so Ironic does not attempt [tenant](../start-here/glossary#tenant) networking; unset lets Ironic pick its per-driver default (`flat`/`neutron`). Like the other interfaces it is **create-only** (applied when the node is first created, not reconciled onto an existing node). Operator extras in `ironic_settings.ironic_driver_info` are merged but can never override Fleet-managed keys.
* **Nova guard**: enrolment-satisfied means `active`, `available`, or `manageable` with an `instance_uuid` — such nodes skip all enrolment mutations (no reinspect, recover, reconfigure, or power-cycle). When `run_inspection` is on (the default), the inspection half still live-reads Ironic inventory (`get_node_inventory`) and runs the same verification checks against that cached data; `run_inspection=false` skips verification too. A node holding an `instance_uuid` in any *other* state fails immediately. The Ironic phases never recover, reconfigure, or power-cycle a node the scheduler may own.
* Wait for a stable provision state, then **auto-recover** `inspect failed` / `clean failed` via `manage` (for `clean failed`, Ironic's maintenance flag is cleared first and restored if the `manage` recovery fails). `deploy failed` / `error` are never auto-recovered.
* Converge node state that can drift: BMC credentials in `driver_info` are overwritten with the per-host credential (keyed off the node's *actual* driver, whatever was stored before); the configured `HW_ARCH_*` scheduler trait is ensured present (stale traits are **not** removed); `raid_interface` is synced only for nodes with a `target_raid_config`, and only in states where Ironic permits the change; and the `properties/root_device` hint is converged from NetBox. The root-device hint is only ever removed for software-RAID nodes; on anything else it is the OS-disk pin and is kept even if NetBox stops setting it (with a warning).

### 5. Inspection half

Brings the node to `MANAGEABLE` and validates the hardware. Whether the node actually reboots into inspection:

```mermaid theme={null}
flowchart TD
    run{"run_inspection?"} -->|"false"| skip["skip inspection + IPA/LLDP<br/>verification (cleaning link<br/>guard still runs)"]
    run -->|"true"| force{"force_inspection?"}
    force -->|"false"| existing{"existing inventory<br/>passes every check?"}
    force -->|"true"| fresh["fresh Ironic inspection<br/>(+ BMC frontend-link guard)"]
    existing -->|"yes"| reuse["reuse existing data —<br/>no reboot"]
    existing -->|"no"| fresh
    fresh --> verify{"checks pass<br/>on live data?"}
    verify -->|"no"| fail["fail: operator-actionable<br/>(miscable / hardware / NetBox)"]
    verify -->|"yes"| done["node MANAGEABLE,<br/>powered on"]
```

The details:

* A node in `ENROLL` first gets its **networking objects created**: one portgroup (bond mode/miimon/xmit-hash from NetBox `ironic_settings.bond`, MAC of the first frontend interface) and one port per frontend NIC that has a MAC in NetBox. A host whose `ironic_settings` declares **no bond** gets no portgroup: its ports are created standalone, and the switch link always resolves to the physical port. PXE is enabled **only on the first port**. Each port's `local_link_connection` records the NetBox switch device + port name (the cable peer — or its LAG parent when the bond is LAG); `switch_id` is a fixed placeholder MAC Ironic requires. Hosts with `ironic_settings.skip_cable_verification` keep the link for a cabled interface and are excused an uncabled one, whose port is created without `local_link_connection`; the flag skips only the LLDP check. Portgroups are **create-only**: an existing one is reused as-is. An existing port is reused too (a port whose MAC belongs to another node or portgroup fails), with one repair: a port that has no `local_link_connection` but whose interface is cabled in NetBox gets the link recorded, and for `skip_cable_verification` hosts that repair also runs on a `MANAGEABLE` node before inspection, so a node enrolled before links were kept is not stuck at inspection. Then `manage`; landing anywhere but `MANAGEABLE` is a hard failure.
* RAID target config from NetBox is (re)applied — Ironic only accepts it in `MANAGEABLE`.
* **Inspection** (default on, `run_inspection=false` to skip): existing inspection inventory is reused when it passes all checks; otherwise (or with `force_inspection=true`) a fresh Ironic inspection runs. While the node boots for inspection, the workflow watches the **BMC-reported frontend link** (5-minute budget, starting once Ironic reports the node `power on`): **any** link `UP` passes; any link state the BMC can't report (`UNKNOWN`) fails open, continuing with Ironic as the source of truth; only all-links-explicitly-`DOWN` for the full budget aborts the inspection — converting what would be an opaque \~25-minute `inspect failed` (Neutron port never `ACTIVE`) into an immediate, onsite-actionable "check switch/cabling/optics" failure. The fail-open applies to unknown link *values* only: a BMC whose Redfish link observation is unsupported or errors raises and fails the attempt rather than continuing.
* **Ironic node-lock contention is waited out, not failed**: when Ironic rejects a request with a `NodeLocked` 409 (another conductor operation holds the node's reservation), the activity attempt returns immediately instead of holding a worker slot, and the workflow re-runs the whole idempotent attempt on durable timers (10s → 60s backoff) for up to a configurable budget (`nodeLockMaxWaitSeconds`, default 15 minutes). Only a node that stays locked for the entire budget fails the enrolment, and that failure reads the node's live state so it names the lock holder and provision state.
* **Recoverable inspection failures earn whole-attempt reruns**: an unexpected provision state, inspection timeout, or power-on timeout reruns the entire attempt from the workflow — the rerun's prelude re-recovers the node — within the same total attempt budget the retry policy defines (a per-job `max_attempts_override=1` still fails fast). Each rerun enters the node-lock backoff afresh, so a failed inspection's runtime is never billed to the lock budget, and lock contention is never misreported as an inspection failure (or vice versa).
* **Verification** against NetBox `inspection_validation` config: no frontend interface may report `has_carrier` explicitly false in the [IPA](../start-here/glossary#ironic-python-agent) inventory (missing/unreported carrier data passes; this check always runs), then CPU cores/sockets/arch, NIC count/vendors/link, disk count/models, memory + DIMM consistency, BIOS version allow/block, GPU PCI device counts — and finally **LLDP vs NetBox cables**: every frontend interface's observed switch + port must match the NetBox cable records (skippable per host via `ironic_settings.skip_cable_verification`, e.g. emulated nodes). The LLDP system name must equal the NetBox switch's name, and the NetBox switch port name must equal either the LLDP port ID or the port description, because switches differ in which of the two carries it: Cumulus sends the name as the port ID and the connected host as the description. Any mismatch fails the enrolment; these failures are environmental (miscable, dead optic, stale NetBox) rather than FM bugs.
* The node is left in `MANAGEABLE`, powered on.

### 6. Optional Redfish→IPMI driver switch (per-site)

At sites where inspection must run over Redfish virtual-media but Redfish power/virtual-media is unreliable for deploy/clean (LON14 Lenovo), the site's workflow-agent config sets `switch_to_ipmi_after_inspection`: after inspection the node's driver is flipped to `ipmi` (per-host credential, IPMI-over-LAN enabled in phase 2). Fires only on an unowned node in `MANAGEABLE` whose driver is `redfish` or `idrac` (Dell's vendored Redfish hardware type) — other drivers, already-switched nodes, and Nova-owned nodes are left alone. The switch itself runs under the same durable node-lock backoff as the Ironic halves.

### 7. Cleaning half

Drives `PROVIDE → AVAILABLE` with the wait in the workflow (durable deadline), not inside an activity. The retry loop:

```mermaid theme={null}
flowchart TD
    start["start activity: prelude,<br/>clear maintenance, provide,<br/>wait power-on + link guard"] --> already{"node already available<br/>or deployed?"}
    already -->|"yes"| final["finalize: power on (skipped<br/>for deployed nodes), tag"]
    already -->|"no"| poll{"poll provision state<br/>(workflow sleep, backoff)"}
    poll -->|"cleaning / clean wait /<br/>manageable"| poll
    poll -->|"available"| final
    poll -->|"clean failed"| budget{"attempts left,<br/>deadline not hit?"}
    budget -->|"yes"| start
    budget -->|"no"| fail["fail"]
    poll -->|"any other state,<br/>or deadline hit"| fail
```

The details:

1. **Start**: shared prelude, re-apply RAID target config, **clear Ironic maintenance** (Ironic silently skips clean steps on a maintenance node — the clean would hang), then `provide`. Wait for power-on, then run the same BMC frontend-link guard as inspection (same 5-minute budget and three-state semantics); all-links-down aborts the clean and fails with the cabling diagnosis. (This guard talks Redfish to the BMC even for `ipmi`-driver nodes.) A `NodeLocked` 409 here gets the same durable lock backoff as inspection, with the lock budget clamped to the overall cleaning deadline.
2. **Poll**: the workflow sleeps between short provision-state probes (exponential backoff) until `available`, bounded by a configurable overall deadline. `clean failed` within the attempt budget re-runs the start step, whose prelude recovers the node and re-triggers the clean. Any other unexpected state fails immediately.
3. **Finalize**: power the node on (skipped for deployed nodes — no power churn) and upsert the NetBox `ironic` tag marking the host enrolled.

A node found already `available`/deployed skips straight to finalize.

## Operator parameters

| Param | Default | Effect |
| - | - | - |
| `run_inspection` | `true` | `false` skips fresh inspection and its IPA inventory + LLDP verification — **the cleaning-half BMC link guard still runs**. Diagnostic bypass only. Decision tree: phase 5 diagram. |
| `force_inspection` | `false` | Fresh inspection even when existing data passes checks. Ignored when `run_inspection=false`. Decision tree: phase 5 diagram. |
| `force_bmc_restart` | `false` | BMC cold reset even when the password didn't rotate. |
| `node_uuid` | none | Pin the Ironic UUID when the node is *created* — hardware-swap flow, so reservations keyed on `infrastructure_ref` survive a tray swap. Fails loudly if the host already has a node under a different UUID or the UUID is in use: the paired unenrol must delete the old node first. Never re-points an existing node. |

## Invariants (the never-do list)

* The Ironic phases never mutate, power-cycle, or recover a node Nova owns (`instance_uuid` set); enrolment-satisfied nodes skip all enrolment mutations — though finalisation still ensures an unowned `available` node is powered on before tagging (deployed nodes avoid power churn). This guard covers Ironic only: BMC configuration runs before it.
* Ironic never authenticates with the shared `ipmi_exporter` monitoring user — always the per-host BMC admin credential.
* Never overwrite a populated NetBox `custom_fields.platform_class`; a divergent value is kept and warned about, never corrected in place.
* Never strip the `root_device` hint from a non-software-RAID node — it is the only OS-disk pin.
* PXE on exactly one port (the first frontend interface) — enforced at port creation; existing ports are reused, and only a missing `local_link_connection` is reconciled.
* `deploy failed` / `error` always stop the workflow for human review; only `inspect failed` / `clean failed` are auto-recovered.
* Maintenance cleared to auto-recover a `clean failed` node is restored if the `manage` recovery itself fails. The restore window is exactly the `manage` call: a convergence failure after a successful `manage` (credentials, traits, RAID, root-device), or a failure after the maintenance clear that precedes a normal `provide`, leaves maintenance off.

## Open questions

* **Redfish→IPMI switch granularity**: `switch_to_ipmi_after_inspection` is a per-site workflow-agent setting today, but the constraint it works around is a property of the BMC family, so it arguably belongs per hardware type in NetBox config-context (review note, John Garbutt). A site mixing BMC families under one workflow agent would need the finer grain.
* **Factory-reset BMC remediation and maintenance mode**: when a BMC needs a factory reset as remediation, should the maintenance-mode handling live inside this workflow's recovery path or in a separate remediation workflow? Currently unresolved (FLEET-1936); enrollment today only touches maintenance as described above (clear-before-clean, clear+restore around `clean failed` recovery).

## Sources of truth

| Behaviour | Code |
| - | - |
| Orchestration, step order, cleaning poll | `src/fleet_manager/temporal/workflows/enrollment.py` |
| Submission eligibility policy (phase 0) | `src/fleet_manager/core/eligibility/policies.py` |
| Per-site phase gates | `src/fleet_manager/settings/agents/temporal_worker.py` |
| Durable node-lock backoff mechanics | `src/fleet_manager/temporal/workflows/ironic_lock_backoff.py` |
| Node-lock wait budget (`nodeLockMaxWaitSeconds`) | `src/fleet_manager/settings/clients/openstack.py` |
| Per-site workflow-agent options (`switch_to_ipmi_after_inspection`, `platform_class_activity_ready`, proof-projection grace) | `src/fleet_manager/settings/agents/workflow.py` |
| Activity boundaries + failure classification | `src/fleet_manager/temporal/activities/enrollment.py` |
| Platform-class computation, NetBox write policy, FM mirror (phase 0b) | `src/fleet_manager/core/domain/platform_class.py` |
| DHCP SSH credentials + verified client construction | `src/fleet_manager/core/datacenter/dhcp/ssh.py` |
| DHCP configuration + explicit remote destination | `src/fleet_manager/core/domain/dhcp_configuration.py`, `src/fleet_manager/core/datacenter/dhcp/server.py` |
| Generic SSH transport + typed failures | `src/fleet_manager/core/clients/ssh/` |
| Ironic domain logic (prelude, inspection, verification, cleaning) | `src/fleet_manager/core/domain/ironic_enrolment.py` |
| Typed Enrollment result / proof contracts | `src/fleet_manager/core/workflow/enrollment_results.py` |
| Ironic `infrastructure_ref` encoding | `src/fleet_manager/core/clients/openstack/ironic/infrastructure_ref.py` |
| BMC configuration | `src/fleet_manager/core/domain/bmc_provisioning.py`, vendor flows in `src/fleet_manager/core/datacenter/bmc/vendors/` |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.