Skip to main content
Living reference. This page describes what the enrollment workflow does to NetBox, DHCP, the BMC, and Ironic — for review by the people who own those systems. Any PR that changes the workflow’s observable behaviour (step order, decisions, side effects, params, failure semantics) must update this page in the same PR. It documents the latest logic only — the path a workflow started today takes; rollout and back-compat mechanics are deliberately absent.
Enrollment turns a racked, NetBox-registered host into a schedulable Ironic node in available: it configures DHCP and the BMC, then enrols, inspects, and cleans the node in Ironic — verifying NetBox’s recorded intent (addressing, cabling, hardware inventory) against what the infrastructure and hardware actually report.
  • Systems touched: NetBox (reads + three writes: device status staged → active, tags, and the platform_class custom field when it is empty), the DHCP server (verified SSH upload + service restart when configuration changes), the BMC (Redfish; full vendor-specific configuration), Ironic (node/portgroup/port creation, manage/provide/abort provisioning actions, power on), Fleet Manager (after terminal completion the workflow-agent establishes or submits an identity-guarded failure write for the Device’s identity-scoped Enrollment proof). Nova is never called. The DHCP and BMC steps run before the Ironic ownership guard, so a rerun on a deployed host still reconverges those settings.
  • Hardware identifiers: resource_class remains the operator-owned, customer-facing scheduler resource. Enrollment records the distinct, Fleet Manager-owned per-device platform_class. Burn-in reads it to derive GPU-count and GPU-name test parameters that a test’s params leave unset from the device’s hardware descriptor; a test that needs such a value on a device without a recorded platform_class fails. Threshold and topology selection still use resource_class. Phase 0b defines its format, gate, and write policy.
  • Terminal proof safety: a well-formed qualifying completion establishes proof only after Fleet Manager’s projected Ironic reference matches the resolved reference. The workflow agent retries that compare-and-set for a bounded period derived from Temporal close time, then fails proof closed with ENROLLMENT_COMPLETION_PROOF_REJECTED. Every later reference change atomically clears proof, so an A → B → A cycle cannot resurrect old validation. Failure, cancellation, termination, timeout, or malformed qualifying output submits an identity-guarded failure write: matching current proof is preserved, stale metadata is cleared when the operated identity still matches, and a replacement identity is untouched. Diagnostic and unclassifiable malformed completions preserve existing proof without a write.
  • Idempotent, but not non-disruptive on a deployed node. Every step re-reads live state and no-ops when already satisfied, and a rerun auto-recovers inspect failed / clean failed (deploy failed / error abort and need a human). But BMC convergence runs before the Nova guard and can restart a Lenovo or iDRAC host whose BIOS settings drifted — do not rerun enrollment against a live workload.
  • End state (all phases enabled, fresh unowned node): Ironic node available and powered on, NetBox device active and tagged ironic (configurable slug) + ipmi_exporter. A node found already enrolment-satisfied — active, available, or manageable with an instance_uuid — skips all enrolment mutations and is re-tagged; an unowned available node is additionally ensured powered on.
  • Verification is a hard gate: inspection data must match NetBox (LLDP vs cables, GPU/CPU/disk/memory/BIOS checks), and — where the BMC can report link state — a frontend link must come up; unknown link telemetry deliberately fails open (phase 5). Mismatches fail the enrolment as operator-actionable errors rather than letting a miscabled node into the pool.

​
High-level flow

Strictly sequential. The routing preflight, the status gate and the NetBox activation always run; the three * groups are gated per site (enrollment_configure_dhcp, enrollment_configure_bmc, enrollment_ironic_enrol); † is the per-site switch_to_ipmi_after_inspection option; ‡ runs once the per-site platform_class_activity_ready worker-readiness gate is on (phase 0b). Ironic provision-state walk for a fresh node: ENROLL → MANAGEABLE (manage) → inspection → MANAGEABLE → CLEANING/CLEAN WAIT (provide) → AVAILABLE. The workflow drives the transitions; Ironic does the actual inspecting and cleaning.

​
Phase details

​
Routing preflight (before any activity)

The workflow resolves which Ironic instance owns the device from the request’s device_class, before the NetBox status gate and before any activity runs. Two cases fail here: the request carries no device_class at all, or it carries one that no Ironic instance manages (e.g. SWITCH). Both fail non-retryably with failure code IRONIC_ROUTING_UNRESOLVED, recorded against phase resolve_ironic_instance and owned by INFRA_ENG. Operator action: fix the device role in NetBox so it carries a managed device_class, or dispatch a workflow that supports this device class, then rerun. No NetBox, DHCP, BMC, or Ironic state is touched before this point, so a failure here is always safe to rerun. This is a routing/metadata problem rather than an engineering defect, which is why it is a classified operator-actionable failure and not an unhandled error: an unclassified exception raised here would fail the Temporal workflow task, leaving the job RUNNING under indefinite retry with no failure code for an operator to act on.

​
0. NetBox status gate

Device must exist in NetBox with status staged or active; anything else fails non-retryably. (Job submission is separately gated by an API-side eligibility policy — Fleet Manager inventory status active/staged, compute/hypervisor device classes only, not in customer custody, and placed in a rack U, so a device NetBox has in its rack without a position is refused with device_not_placed until the NetBox agent syncs its U — but that reads FM’s eventually-consistent projection; this NetBox check is the workflow’s own authoritative read, and everything later re-validates against live Ironic/BMC state.)

​
0b. Platform class

The workflow computes the device’s ADR-0028 platform_class — <GPUTYPE>_<DEVICETYPE>_<FABRICTYPE>_<NN>, the key hardware descriptors and burn-in parameter resolution are scoped by — and records it in both systems of record. It runs before any host-touching step so a NetBox data gap fails in the first minute, where a rerun after the fix is cheap. Worker-readiness gate. The step runs only when the request’s platform_class_activity_ready metadata is true. The workflow agent snapshots that from the per-site workflowAgent.platformClassActivityReady chart value at submission, so a flip cannot change a running execution. Keep it false while old and new Temporal workers share the task queue — an old worker polled for record_platform_class_activity fails it as unregistered — and turn it on per site once the rollout has completed and a dry run of fleetctl netbox backfill-platform-class shows the site’s gpu_type coverage is complete. While it is false the step is skipped and no key is recorded. Inputs, all read live from NetBox:
  • GPUTYPE — the device type’s custom_fields.gpu_type (e.g. GB300). A GPU compute device whose device type has no value fails; an empty value on a CPU baremetal or hypervisor device resolves the literal NOGPU.
  • DEVICETYPE — the device type’s manufacturer name and model, each upper-cased with every non-alphanumeric character stripped, then concatenated (Lenovo + HG635N V2 → LENOVOHG635NV2).
  • Length — a computed key, or a populated NetBox value being kept, longer than 128 characters is unresolvable: NetBox would accept it and Fleet Manager would refuse it, splitting the two systems.
  • FABRICTYPE — derived from config_context.ztp for GPU devices only: no ztp block → NORDMA; a block with an infiniband section → IB; a block that is present (including an empty {}) but has no infiniband section fails, because a guessed fabric would select the wrong descriptor. The RoCE marker is not yet pinned (ADR-0028 review trigger). A NOGPU device never has its ztp block read and is always NORDMA; revisit if a non-GPU compute node is ever RDMA-backed.
  • NN — always 01. There is no collision handling.
Write policy: when NetBox custom_fields.platform_class is empty, the computed key is written with a PATCH. When it is already populated and equal, nothing is written. When it is populated and different, NetBox is not overwritten — the field is read-only for humans (INFENG-1035), so a divergent value is almost always one Fleet Manager wrote from earlier inputs — a warning is logged, and the NetBox value is the effective one. The Fleet Manager Device.platform_class is then set to the effective value only if it differs. A rerun on an unchanged host therefore performs no writes. Failure codes: PLATFORM_CLASS_UNRESOLVABLE (non-retryable, INFRA_ENG) when NetBox data cannot produce a key — operator action: populate gpu_type on the device type or fix the ztp block, then rerun; PLATFORM_CLASS_RECORD_FAILED (retryable, FLEET_MANAGER) when NetBox or the Fleet Manager API errors while recording it — check that the platform_class custom field exists on this site’s NetBox. Backfill and correction. Hosts enrolled before this step existed are backfilled with fleetctl netbox backfill-platform-class, which runs the same resolver and is also the correction path, since enrolment runs once per lifecycle. A NetBox fix after a failed enrolment needs nothing extra: rerun enrolment or the backfill. A key that was written and is later found wrong is corrected by clearing custom_fields.platform_class in NetBox and running the backfill for that device by name; the resolver never overwrites a populated value on its own. There is no periodic re-derivation today; see ADR-0028 for the reconciliation options if that changes.

​
1. DHCP configuration

The workflow reads the device’s management MAC and out-of-band address from NetBox, renders its DHCP entry, and uploads that file to the DHCP-owned remote configuration directory. The DHCP adapter resolves the private key and known_hosts data from the runtime vault’s ssh-config-dhcp item, uses the key only for the lifetime of the client context, and requires host-key verification. Connection, login, each remote command, and each SFTP transfer have independent configurable deadlines (120 seconds each by default). The upload is content-idempotent: a matching remote SHA-256 hash skips both transfer and service restart. Changed or unavailable remote content is uploaded, then the DHCP service is restarted and validated using the existing service-specific logic. Completed hash or remote-path preparation commands with non-zero status retain the existing best-effort behaviour; SSH authentication, host-key, connectivity, timeout, command-transport, and transfer failures stop the activity. The existing DHCP Temporal failure classification remains unchanged.

​
2. BMC configuration

Vendor is detected from the Redfish service root, then the vendor strategy runs. The fullest flows (Lenovo XCC, Dell iDRAC), in order:
  1. Password setup — ensures the per-host BMC admin credential in 1Password works, rotating the BMC password if needed.
  2. Cold reset of the BMC — only when the password was rotated (some BMCs require it to apply credentials) or when force_bmc_restart=true is passed; waits for Redfish to come back.
  3. Settings convergence — hostname, DHCP-on-BMC-NIC, security hardening, NTP + syslog (targets from the NetBox site), BIOS/hardware settings, boot order (the relative order of individual settings varies by vendor). On iDRAC the boot-order step is the same boot restriction as AMI (see below) when the provisioning NIC is known and restrict_boot is set — it disables every other NIC BootOption and promotes the provisioning NIC first, subsuming and replacing the label-based reorder; without a known provisioning NIC or with restrict_boot=false it falls back to that reorder and disables nothing. On iDRAC, BIOS and boot-order changes are staged as a job that only applies on the next boot, so those two steps wait for the Lifecycle Controller to schedule the job before requesting the host reset — resetting sooner lets the boot walk past the job, which then stays queued and rejects every later settings write. A job that never schedules fails the step and is deleted so a rerun can stage it again; a job some other reboot already carried into the Lifecycle Controller is left to finish without a further reset.
  4. IPMI monitoring user + IPMI-over-LAN — creates/updates the shared ipmi_exporter monitoring user. Missing ipmi_exporter credentials in 1Password fail BMC configuration; only a device model with no IPMI-user endpoint skips this step. The user is for monitoring only; Ironic always authenticates with the per-host admin credential, even on the ipmi driver.
AMI MegaRAC keeps password setup + cold reset but converges a smaller set: hostname, NTP, IPMI user, virtual-media enablement, and boot restriction — no security, syslog, or BIOS convergence. Boot restriction disables every NIC BootOption except the single provisioning NIC (the one Ironic PXE-enables — the first frontend interface of the host’s interface_type, resolved by the same rule so the two never disagree, matched by MAC against each BootOption’s UefiDevicePath) and promotes that NIC to the front of the boot order, ahead of the disk. This stops the host wasting boot attempts on NICs it cannot provision over and stops it silently falling back to a stale disk image. Disk, virtual-media, and shell options are left bootable. It fails closed: if the provisioning NIC’s BootOption cannot be positively identified, nothing is disabled. iDRAC runs this same restriction over its own Redfish path (per-option BootOptionEnabled, then the staged OnReset reorder job). The restriction is on by default and shared across vendors; the per-satellite restrict_boot toggle (TEMPORAL_WORKER_ENROLLMENT_RESTRICT_BOOT) opts a deployment out. Opting out is vendor-specific: iDRAC then runs its label-based reorder (nothing disabled), while AMI — which has no separate reorder step — skips boot configuration entirely. Supermicro is a smaller subset again (hostname, NTP, syslog, IPMI user — no password setup; reset only when forced). On success the device gets the NetBox ipmi_exporter tag regardless of which steps its vendor ran — the tag means “BMC configuration completed”, not “monitoring user exists”. Busy BMCs. A configuration write answered with a 5xx is the BMC being busy, not refusing: an AMI BMC answers boot writes with 503 while its host is booting. The step retries it within its budget, and if the BMC is still busy it fails as BMC_HTTP_SERVICE_UNAVAILABLE with the BMC’s own reason in the message; rerun once the host has booted.

​
3. NetBox activation

Moves the device staged → active. Deliberately not gated by any step toggle: it is the only path from staged back to active, and unenrolment leaves a device staged, so a deployment with BMC configuration disabled would otherwise never get its re-enrolled nodes back. Only a staged device is updated; any other current status is left untouched.

​
4. Ironic enrolment — shared prelude

Both Ironic halves (inspection, cleaning) start from the same idempotent prelude, so either can be retried or re-run standalone. How the prelude dispatches on the node it finds: The details:
  • Re-resolve the NetBox device (oob_ip + config-context ironic_settings required) and the per-host BMC credentials.
  • Get-or-create the Ironic node. On create: driver and interfaces from ironic_settings; driver_info is Redfish (address from oob_ip, or redfish_address_override for emulated nodes) or IPMI depending on ironic_settings.driver — both use the per-host BMC credential; owner/lessee set to the enrol project; resource_class, capabilities, cpu_arch, root-device hint from NetBox. raid_interface is mandatory. Optional network_interface is passed through when set — a standalone-Ironic (no-Neutron) site sets "noop" so Ironic does not attempt tenant networking; unset lets Ironic pick its per-driver default (flat/neutron). Like the other interfaces it is create-only (applied when the node is first created, not reconciled onto an existing node). Operator extras in ironic_settings.ironic_driver_info are merged but can never override Fleet-managed keys.
  • Nova guard: enrolment-satisfied means active, available, or manageable with an instance_uuid — such nodes skip all enrolment mutations (no reinspect, recover, reconfigure, or power-cycle). When run_inspection is on (the default), the inspection half still live-reads Ironic inventory (get_node_inventory) and runs the same verification checks against that cached data; run_inspection=false skips verification too. A node holding an instance_uuid in any other state fails immediately. The Ironic phases never recover, reconfigure, or power-cycle a node the scheduler may own.
  • Wait for a stable provision state, then auto-recover inspect failed / clean failed via manage (for clean failed, Ironic’s maintenance flag is cleared first and restored if the manage recovery fails). deploy failed / error are never auto-recovered.
  • Converge node state that can drift: BMC credentials in driver_info are overwritten with the per-host credential (keyed off the node’s actual driver, whatever was stored before); the configured HW_ARCH_* scheduler trait is ensured present (stale traits are not removed); raid_interface is synced only for nodes with a target_raid_config, and only in states where Ironic permits the change; and the properties/root_device hint is converged from NetBox. The root-device hint is only ever removed for software-RAID nodes; on anything else it is the OS-disk pin and is kept even if NetBox stops setting it (with a warning).

​
5. Inspection half

Brings the node to MANAGEABLE and validates the hardware. Whether the node actually reboots into inspection: The details:
  • A node in ENROLL first gets its networking objects created: one portgroup (bond mode/miimon/xmit-hash from NetBox ironic_settings.bond, MAC of the first frontend interface) and one port per frontend NIC that has a MAC in NetBox. A host whose ironic_settings declares no bond gets no portgroup: its ports are created standalone, and the switch link always resolves to the physical port. PXE is enabled only on the first port. Each port’s local_link_connection records the NetBox switch device + port name (the cable peer — or its LAG parent when the bond is LAG); switch_id is a fixed placeholder MAC Ironic requires. Hosts with ironic_settings.skip_cable_verification keep the link for a cabled interface and are excused an uncabled one, whose port is created without local_link_connection; the flag skips only the LLDP check. Portgroups are create-only: an existing one is reused as-is. An existing port is reused too (a port whose MAC belongs to another node or portgroup fails), with one repair: a port that has no local_link_connection but whose interface is cabled in NetBox gets the link recorded, and for skip_cable_verification hosts that repair also runs on a MANAGEABLE node before inspection, so a node enrolled before links were kept is not stuck at inspection. Then manage; landing anywhere but MANAGEABLE is a hard failure.
  • RAID target config from NetBox is (re)applied — Ironic only accepts it in MANAGEABLE.
  • Inspection (default on, run_inspection=false to skip): existing inspection inventory is reused when it passes all checks; otherwise (or with force_inspection=true) a fresh Ironic inspection runs. While the node boots for inspection, the workflow watches the BMC-reported frontend link (5-minute budget, starting once Ironic reports the node power on): any link UP passes; any link state the BMC can’t report (UNKNOWN) fails open, continuing with Ironic as the source of truth; only all-links-explicitly-DOWN for the full budget aborts the inspection — converting what would be an opaque ~25-minute inspect failed (Neutron port never ACTIVE) into an immediate, onsite-actionable “check switch/cabling/optics” failure. The fail-open applies to unknown link values only: a BMC whose Redfish link observation is unsupported or errors raises and fails the attempt rather than continuing.
  • Ironic node-lock contention is waited out, not failed: when Ironic rejects a request with a NodeLocked 409 (another conductor operation holds the node’s reservation), the activity attempt returns immediately instead of holding a worker slot, and the workflow re-runs the whole idempotent attempt on durable timers (10s → 60s backoff) for up to a configurable budget (nodeLockMaxWaitSeconds, default 15 minutes). Only a node that stays locked for the entire budget fails the enrolment, and that failure reads the node’s live state so it names the lock holder and provision state.
  • Recoverable inspection failures earn whole-attempt reruns: an unexpected provision state, inspection timeout, or power-on timeout reruns the entire attempt from the workflow — the rerun’s prelude re-recovers the node — within the same total attempt budget the retry policy defines (a per-job max_attempts_override=1 still fails fast). Each rerun enters the node-lock backoff afresh, so a failed inspection’s runtime is never billed to the lock budget, and lock contention is never misreported as an inspection failure (or vice versa).
  • Verification against NetBox inspection_validation config: no frontend interface may report has_carrier explicitly false in the IPA inventory (missing/unreported carrier data passes; this check always runs), then CPU cores/sockets/arch, NIC count/vendors/link, disk count/models, memory + DIMM consistency, BIOS version allow/block, GPU PCI device counts — and finally LLDP vs NetBox cables: every frontend interface’s observed switch + port must match the NetBox cable records (skippable per host via ironic_settings.skip_cable_verification, e.g. emulated nodes). The LLDP system name must equal the NetBox switch’s name, and the NetBox switch port name must equal either the LLDP port ID or the port description, because switches differ in which of the two carries it: Cumulus sends the name as the port ID and the connected host as the description. Any mismatch fails the enrolment; these failures are environmental (miscable, dead optic, stale NetBox) rather than FM bugs.
  • The node is left in MANAGEABLE, powered on.

​
6. Optional Redfish→IPMI driver switch (per-site)

At sites where inspection must run over Redfish virtual-media but Redfish power/virtual-media is unreliable for deploy/clean (LON14 Lenovo), the site’s workflow-agent config sets switch_to_ipmi_after_inspection: after inspection the node’s driver is flipped to ipmi (per-host credential, IPMI-over-LAN enabled in phase 2). Fires only on an unowned node in MANAGEABLE whose driver is redfish or idrac (Dell’s vendored Redfish hardware type) — other drivers, already-switched nodes, and Nova-owned nodes are left alone. The switch itself runs under the same durable node-lock backoff as the Ironic halves.

​
7. Cleaning half

Drives PROVIDE → AVAILABLE with the wait in the workflow (durable deadline), not inside an activity. The retry loop: The details:
  1. Start: shared prelude, re-apply RAID target config, clear Ironic maintenance (Ironic silently skips clean steps on a maintenance node — the clean would hang), then provide. Wait for power-on, then run the same BMC frontend-link guard as inspection (same 5-minute budget and three-state semantics); all-links-down aborts the clean and fails with the cabling diagnosis. (This guard talks Redfish to the BMC even for ipmi-driver nodes.) A NodeLocked 409 here gets the same durable lock backoff as inspection, with the lock budget clamped to the overall cleaning deadline.
  2. Poll: the workflow sleeps between short provision-state probes (exponential backoff) until available, bounded by a configurable overall deadline. clean failed within the attempt budget re-runs the start step, whose prelude recovers the node and re-triggers the clean. Any other unexpected state fails immediately.
  3. Finalize: power the node on (skipped for deployed nodes — no power churn) and upsert the NetBox ironic tag marking the host enrolled.
A node found already available/deployed skips straight to finalize.

​
Operator parameters

​
Invariants (the never-do list)

  • The Ironic phases never mutate, power-cycle, or recover a node Nova owns (instance_uuid set); enrolment-satisfied nodes skip all enrolment mutations — though finalisation still ensures an unowned available node is powered on before tagging (deployed nodes avoid power churn). This guard covers Ironic only: BMC configuration runs before it.
  • Ironic never authenticates with the shared ipmi_exporter monitoring user — always the per-host BMC admin credential.
  • Never overwrite a populated NetBox custom_fields.platform_class; a divergent value is kept and warned about, never corrected in place.
  • Never strip the root_device hint from a non-software-RAID node — it is the only OS-disk pin.
  • PXE on exactly one port (the first frontend interface) — enforced at port creation; existing ports are reused, and only a missing local_link_connection is reconciled.
  • deploy failed / error always stop the workflow for human review; only inspect failed / clean failed are auto-recovered.
  • Maintenance cleared to auto-recover a clean failed node is restored if the manage recovery itself fails. The restore window is exactly the manage call: a convergence failure after a successful manage (credentials, traits, RAID, root-device), or a failure after the maintenance clear that precedes a normal provide, leaves maintenance off.

​
Open questions

  • Redfish→IPMI switch granularity: switch_to_ipmi_after_inspection is a per-site workflow-agent setting today, but the constraint it works around is a property of the BMC family, so it arguably belongs per hardware type in NetBox config-context (review note, John Garbutt). A site mixing BMC families under one workflow agent would need the finer grain.
  • Factory-reset BMC remediation and maintenance mode: when a BMC needs a factory reset as remediation, should the maintenance-mode handling live inside this workflow’s recovery path or in a separate remediation workflow? Currently unresolved (FLEET-1936); enrollment today only touches maintenance as described above (clear-before-clean, clear+restore around clean failed recovery).

​
Sources of truth