> ## Documentation Index > Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt > Use this file to discover all available pages before exploring further. # GB300 Compute Tray Replacement > Operator runbook for swapping a failed GB300 compute tray with a pre-staged spare via the NodeSwap workflow. ## Overview Replace a failed GB300 compute tray with a pre-staged spare, keeping the production slot's [Ironic](../start-here/glossary#ironic) identity (so Unity/UNI reservations survive). The flow is three parts: 1. **Swap node workflow** (Fleet Manager) — run on the *failed* tray. Drains it, updates [NetBox](../start-here/glossary#netbox) to the target wiring/location, deletes both Ironic nodes and re-adds them with cross-pinned UUIDs to `manageable`, powered off. Everything except the physical move. 2. **Physical swap** (datahall) — a technician moves + re-cables the two trays to match NetBox. 3. **Enrolment** (Fleet Manager) — run the normal enrol on each node; inspection then validates the cabling matches, and each node goes on to the customer or into [SLURM](../start-here/glossary#slurm). Design and rationale: see the node-swap design spec (`docs/specs/0031-gb300-node-swap.md`). [Device](../start-here/glossary#node) **names stay with the trays**; only the Ironic node UUID moves (pinned to the topology slot). This is a **manual** flow for now — CHAD can wrap it later. `NodeSwapWorkflow` itself is **hardware-independent**: the seven steps below, the run-once rule, recovery mode and the failure-code table apply to every hardware type the swap is wired for, and this page is the shared reference for them. For a **B200** — a whole server rather than a compute tray, with no NVL72 fabric behind it — see [B200 Node Replacement](./b200-node-replacement), which covers only the site facts and spare-staging preconditions that differ. > **Moving trays during standup, before anything is contracted?** You want > [`netbox_only` mode](#day-0-records-only-swap-netbox_only), not the full flow above. It > rewrites NetBox cabling and location and stops. > > **Failed tray's [BMC](../start-here/glossary#bmc) dead, or its Ironic node deleted [out-of-band](../start-here/glossary#oob)?** That is > [`failed_tray_unreachable`](#swapping-a-tray-whose-bmc-is-dead-failed_tray_unreachable) — > still a full swap, minus the work that needed the dead half's BMC. > **Run it exactly once per physical swap.** The NetBox swap and UUID cross-pin invert > state, so re-running the swap node workflow on the same pair swaps everything back. > If it fails partway, inspect NetBox and Ironic before doing anything else. ## Preconditions * The **replacement** tray is a spare **modelled in NetBox** at the **same site** — placed in a rack position in the hospital/spare rack, or in any rack with no U (see [Box-fresh spare with no rack position](#box-fresh-spare-with-no-rack-position)) — is the **same model and role** as the failed tray, and has firmware levelled to production. It does **not** have to be enrolled: a spare straight from stock has no Ironic node, and the workflow handles that — the failed tray simply gets a freshly minted UUID instead of the spare's. (Only the *production* slot's UUID has to survive the swap; that is what reservations key on.) An already-enrolled spare must be settled `available` with no instance attached. * The replacement is a tray **Fleet Manager could enrol after the swap** — it exists in FM at the site and passes the same eligibility policy the post-swap enrolment job will apply (so not an inventory-only projection row, not decommissioned or otherwise non-enrollable, not in [customer custody](../start-here/glossary#customer-custody), and an enrollable device class). * **Both** trays can actually be re-added to `manageable`: NetBox `config_context.ironic_settings` is present with a non-empty `raid_interface`, and each tray has at least one frontend interface carrying a MAC address so Ironic ports can be registered. A spare with no Ironic node whose frontend interfaces lack MACs is the exception on a GB300 tray: the swap fills them from its BMC in step 1 (see [Box-fresh spare with no rack position](#box-fresh-spare-with-no-rack-position)). * **The spare's Ironic settings match the failed tray's.** The swap re-adds the spare from its own `ironic_settings`, so the preflight refuses a spare whose block differs from the failed tray's on `driver`, `boot_mode`, a hardware interface (`raid`, `boot`, `inspect`, `bios`, `management`, `power`, `vendor`, `network`), `resource_class`, a node property (`capabilities`, `cpu_arch`, `vendor`) or a virtual-media setting (`vmedia_transport_protocol`, `boot_iso_name_no_hyphens`), and names each difference. `ironic_driver_info` is not compared: it is free-form, so check it by hand when a spare carries its own. It compares NetBox with NetBox, never with the failed tray's live node: LON14 switches enrolled nodes to `ipmi` after inspection, so a live node there never matches its own block. Under `failed_tray_unreachable` the dead tray's NetBox block is still the reference; if it has none, the check is skipped and only the worker log says so. A mismatch usually means the spare is in another cluster: put it in the failed tray's cluster. Where either tray carries a per-device block, decide which settings the slot needs rather than hand-writing a match. * **Both trays are cabled in NetBox.** Registering an Ironic port records the switch port the interface's cable lands on, so a frontend interface with no cable fails the re-add. The swap moves each tray onto the *other* tray's cables, so the preflight checks each tray against its **counterpart's** cabling — an uncabled hospital rack is what breaks the **failed** tray's re-add, not the spare's. `ironic_settings.skip_cable_verification` exempts a tray, and it has to be set on the tray **moving into** the uncabled rack (the one that will have no cables), not on the uncabled tray itself. In practice both trays resolve the same `ironic_settings` — production scopes that block by device type and cluster, and a swap changes neither — so the flag is a property of the pair, not of one side; do not hand-write it per device (see the [B200 runbook](./b200-node-replacement)). Otherwise get the cabling into NetBox before submitting. Under `failed_tray_unreachable` only the replacement is checked — it is the only tray re-added — so an uncabled hospital rack is no obstacle to a dead-tray swap. The preflight checks all of the above because an unenrolled spare proves none of it: an inventory-only row can be fully racked with a matching model and role, and a stock spare has never had its Ironic config or its cabling exercised. Every one of them would otherwise fail at the *last* step — after the customer instance is deleted and NetBox is rewired. * **Double-check the replacement name.** It is free text, so it is the one parameter the API cannot validate for you. A name that exists in NetBox but is the wrong hardware is rejected by the preflight on the model check — but that check is what stands between a typo and a drained customer node, so read it back before submitting. While the swap is active, Fleet Manager refuses any other job on the spare it names, an enrolment included. A swap naming a spare that another active job is using, or that another active swap names, is refused at submission. * You know both **device names**: the failed tray and the replacement spare. * **Both trays' BMCs are reachable.** The Ironic node *delete* needs no BMC, but re-adding to `manageable` talks to each tray's BMC. The preflight (step 1) probes both and fails fast if either is unreachable — before anything is drained or rewired (see [Failure handling](#failure-handling)). A dead **replacement** BMC always fails here: fix it or stage a different spare. A dead **failed-tray** BMC is a supported input — see [`failed_tray_unreachable`](#swapping-a-tray-whose-bmc-is-dead-failed_tray_unreachable). A replacement with no Ironic node needs no BMC login beforehand where the site's enrolment configures BMCs and that configuration sets the BMC's password, as it does on every GB300 tray's AMI BMC: step 1 configures its BMC as enrolment would, creating the login in the operators vault, before probing it. Supermicro's configuration sets no password, so a Supermicro spare still needs its login stored first. ## Step 1 — run the swap node workflow Submit a Job for the **Swap node** workflow (`NodeSwapWorkflow`) on the **failed tray**, with two parameters: ```json theme={null} { "replacement_host_name": "", "force_reclaim": false } ``` Flip `force_reclaim` to `true` only when the failed tray is still serving a customer — see below. `false` is right for the usual case, a tray with no [Nova](../start-here/glossary#nova) instance bound. The site must have force-[reclaim](../start-here/glossary#reclaim) enabled (`WorkflowAgentConfig.force_reclaim_allowed`), and that gate is checked first — a site that has not enabled it says so rather than asking for the parameter. `force_reclaim` is the operator's confirmation that the drain may delete the Nova instance bound to the failed tray, and it is **required only when the tray actually has one**. A failed tray usually does not: the preflight reads the binding, and with nothing bound the drain is skipped outright, so the swap runs with `force_reclaim` left `false` and goes straight from the trait removal to the record updates. Set it to `true` when the tray is still serving a customer — the workflow refuses at the preflight otherwise, before it has written anything, which is why the refusal is placed ahead of the stale marking in step 2. Two edges worth knowing. If Nova binds an instance between the preflight and the cordon, a run that was allowed through on `false` refuses at that point rather than delete it. Two writes have happened by then and both are cheap: the stale marking of step 2 — which a rerun recognises rather than repeating — and the (idempotent) trait removal. So rerun with `force_reclaim: true`; recovery mode is not needed. And recovery mode always requires `true`, because it skips the preflight and so has no binding to read. (Via the FM UI: pick the failed tray → *Swap node* → set the replacement host name, and tick force-reclaim if the tray still carries a customer instance. Via the API: `POST /api/v1/jobs` with the failed tray's device, that workflow, and the parameters above.) The workflow runs these steps in order. Every step **fails fast** on error — except step 5's NetBox writes, which the workflow retries itself (see *Why the NetBox rewrite retries itself* below), and a never-enrolled spare's BMC configuration in step 1, which keeps enrolment's retry budget: 1. Preflight — verify both trays are swappable and capture the Ironic node UUIDs. The failed tray must have an Ironic node (its production UUID to preserve); the pair must be in the same site and the same model and role, the failed tray placed in a rack position and the replacement placed or in a rack with no U, in different racks when both are placed; and the replacement, *if* it has a node, must be a safe spare to tear down (no instance, settled `available`). Both trays' BMCs must also be reachable (step 7 re-adds each via its BMC). A bad request or an unreachable BMC fails here, before anything below runs. A replacement with no Ironic node has its BMC configured first, as enrolment's BMC step would (on a GB300 tray's AMI BMC: a factory login rotated and stored in the operators vault, a working stored login kept), at a site whose enrolment configures BMCs. A replacement with no Ironic node whose frontend interfaces lack MACs in NetBox then has them filled from its BMC, after its probe: the BMC must report the MAC NetBox records on the interface carrying the spare's OOB IP, and NetBox must not contradict what it reports (an interface carrying another MAC, a MAC recorded on another device). Only a GB300 tray's AMI BMC reports them; any other spare without MACs is refused there. A run refused for want of `force_reclaim` is refused before either write, so it still writes nothing. 2. Mark the [burn-in](../start-here/glossary#burn-in) runs that qualified either tray stale, under the `node_swap` operation — a swapped rack must stop reading as accepted on evidence measured over hardware that has moved, and that covers both trays' single-node runs plus every rack- and pod-level collective either took part in. Every hold is listed in the workflow log and is individually undoable, for a swap abandoned before the trays physically move. Which tests this invalidates is deployment policy, not workflow code: it is the `node_swap` entry for the trays' device type in `stale_test_operations.yaml`. **If no entry covers that device type, the swap stops here** — the first thing it does, before the tray is cordoned and before the customer's instance is deleted. Add the mapping and rerun; nothing has been changed beyond a never-enrolled spare's BMC configuration and MAC fill, which the rerun keeps. (See [Hardware without a mapping](#hardware-without-a-node_swap-mapping) below.) A rerun of the same swap recognises its own earlier hold and does not create a second. 3. Remove the customer-ready trait from the failed tray. 4. Force-reclaim the failed tray to `available` (deletes the customer instance, cleans). **This is the first irreversible step in the swap** — past here, the customer's workload is gone. Skipped when no instance is bound to it — there is nothing to reclaim, and step 6 deletes the node outright anyway. The check is re-read from Ironic here, after step 3 has cordoned the tray: the preflight reads it too, but the tray is still schedulable until the trait comes off, so a workload landing in between is reclaimed normally. 5. Move the failed tray's [allocation](../start-here/glossary#allocation) memberships onto the replacement, so the customer's allocation names the tray now in their rack, then update NetBox cabling + location to the desired post-swap state. Allocation first: it is one API transaction and can be swapped straight back, whereas the NetBox rewire has no rollback. The workflow retries the NetBox rewrite itself, against the snapshot step 1 captured — see [Why the NetBox rewrite retries itself](#why-the-netbox-rewrite-retries-itself). 6. Delete both Ironic nodes and their [Placement](../start-here/glossary#placement) resource providers. A dead tray whose node was already deleted has its provider deleted by the supplied `production_uuid`: it would otherwise sit on the very UUID the replacement is re-pinned to. The spare's teardown runs even when it had no node. This also clears each tray's SLURM node object — both trays change rack, and a surviving dynamic-node record would hand the next adoption the tray's *old* row and rack features, which passes registration and then fails the first rack-scoped collective with `BadConstraints`. 7. Re-add both to `manageable` with cross-pinned UUIDs (no inspection). An unenrolled spare leaves nothing for the failed tray to inherit, so Ironic mints it a fresh UUID. 8. Park both in Ironic maintenance and power them off. Maintenance is what makes the power-down stick — Ironic's power sync would otherwise turn a hand-powered tray straight back off — so the re-racked tray can be powered up by hand to verify. Enrolment clears the flag. When it completes, NetBox reflects the *target* state and both nodes are `manageable` and powered off. **Do not skip to enrolment yet** — the trays are not physically moved. ## Swapping a tray whose BMC is dead (`failed_tray_unreachable`) A tray is often swapped *because* it is dead — and the sequence above needs the failed tray's BMC twice (the preflight probe and the re-add) and needs it to still have an Ironic node. Set `failed_tray_unreachable: true` to assert the tray cannot be reached: ```json theme={null} { "replacement_host_name": "", "force_reclaim": true, "failed_tray_unreachable": true } ``` What it drops — all of it work the swap does for the failed tray's *own* sake, which serves only its disposable hospital identity: its BMC probe, its re-add eligibility checks, its re-add to `manageable`, and its power-off. What still runs is everything protecting the customer's slot: the cordon, the drain, the allocation move, the NetBox rewrite, the delete of **both** Ironic nodes and **both** Placement providers, and the replacement's re-add pinned to the production UUID. `force_reclaim` and the per-site gate still apply — the drain deletes the same customer instance. The drain runs whenever the failed tray's Ironic node still exists, even with no instance bound, so such a run needs `force_reclaim: true`; without it the swap is refused before it writes anything. What changes is how it finishes: it waits for the Nova instance to be *gone* rather than for the node to reach `available`, because reaching `available` means Ironic cleaning, and cleaning boots a host this one cannot boot. The node is put into maintenance, unless Ironic has already put it there, so the abandoned clean cannot hold it against the node delete. **If the Ironic node was already deleted out-of-band**, add `production_uuid` — the production slot's original Ironic UUID. Without a node there is nowhere else to read the slot's identity, and that identity is the whole point of the swap. Conversely, if the node still exists, `production_uuid` is **refused**: Ironic is authoritative there, and a stale value would be pinned onto the customer's slot. The swap is refused if a node holds that UUID, another active swap uses it, or a server still uses its Placement provider. A UUID no node holds is taken as given. The failed tray is **not powered off** — there is no BMC to do it through — and for the same reason it is not parked in Ironic maintenance either. Verify power at the PDU before pulling it. It is left with no Ironic node; a plain enrolment mints it a new identity after repair. Its SLURM node object needs no manual cleanup: the teardown deletes that for both trays, including one that has no Ironic node. The maintenance/undeploy interaction on an unbootable host has not yet been verified against real hardware. Run the sacrificial-node check before the first production dead-tray swap. ## Box-fresh spare with no rack position A box-fresh tray that goes straight into the failed tray's slot has no position NetBox can give it: the failed tray's record still holds that U, so NetBox shows the spare in the rack as *Not racked*. The swap takes such a spare as it is, in any rack at the site, the failed tray's own included. Afterwards the spare holds the failed tray's slot and the failed tray is in the spare's rack with no U. 1. **Stage the spare's records.** It needs a rack assignment (not a U), an OOB IP that the site's DHCP hands out, on the interface that carries its BMC's MAC, and NetBox status `staged` when the swap runs, because the preflight holds it to the enrolment policy and enrolment accepts only `staged` and `active`. A GB300 spare needs no BMC login and no frontend MACs: the swap creates the login in step 1, where the site's enrolment configures BMCs (a Supermicro spare still needs its login stored first), and then fills its eth0 and eth1 MACs from the BMC, once the BMC reports the MAC NetBox records for it. Any other spare needs its frontend interfaces with their real MACs. The swap configures whatever BMC answers at that OOB IP, though, so check the IP is the spare's own before you submit. 2. **Create the record as `inventory` and move it to `staged` just before the swap.** Fleet Manager syncs the record as soon as it exists. A spare that moves from `planned` to `staged` emits a standup postcard, and CHAD opens a day-zero enrolment pipeline for it, which Fleet Manager refuses while the spare has no U; `inventory` to `staged` emits no postcard at all. At LON14 and OSL1 the order matters twice over: their topology Regions discover `planned` and `staged` devices, and a discovered tray with no cables holds the Region until the swap moves the failed tray's cables onto it. Wait until Fleet Manager shows the spare as `staged` before you submit. 3. **Once the spare physically sits in the failed tray's U, submit the swap with [`failed_tray_unreachable: true`](#swapping-a-tray-whose-bmc-is-dead-failed_tray_unreachable).** A normal swap re-adds the failed tray to Ironic on the spare's cables, and the spare has none in NetBox. Dead-tray mode never re-adds the failed tray, so it ends with no Ironic node, and the failed tray is out of the rack by then, so the assertion is true as well. Without the flag the preflight refuses the swap with `NETBOX_INTERFACE_CABLE_MISSING`, and the error names the flag. [`netbox_only`](#day-0-records-only-swap-netbox_only) does not take a spare with no U. 4. **After the swap, confirm the failed tray left topology.** The swap declares the failed tray's removal and sets it `offline` itself (see [A spare with no NetBox cables](#a-spare-with-no-netbox-cables-cableless-swap)). Move it to `rma` once the vendor RMA is raised, and read the Region's sync status as that section describes, escalating anything it does not list: ```bash theme={null} fleetctl api topology-sync-status get ``` 5. **Enrol the spare**, which NetBox now shows in the failed tray's slot. Enrolment refuses a device with no U, so it becomes eligible once the NetBox agent syncs the new slot. Burn-in progress for the spare's resource class at the site pauses until topology publishes the enrolled spare: until then it is a GPU host in the slot with no NVLink-domain membership, while its rack-mates have one. ## A spare with no NetBox cables (cableless swap) A box-fresh spare often has no cables in NetBox yet. The preflight accepts one only with `failed_tray_unreachable: true` — the failed tray is not re-added, so nothing needs the spare's cables — and the rewrite then leaves the failed tray with none. topology-service reads NetBox, so a discovered tray with no uplink holds the whole Region's topology sync (`node_not_rooted`), and from topology-service 0.7.0 a published tray that disappears holds it too unless its removal was declared first. When the preflight finds the spare uncabled, the workflow therefore: 1. declares the failed tray's removal to topology-service (a Sync Intent named `node-swap-`) before its first write to Fleet Manager or the NetBox records. If that call fails, the swap stops with `TOPOLOGY_SYNC_INTENT_FAILED` before any such write (a never-enrolled spare's BMC may already be configured, which a rerun keeps); rerun once topology-service is back; 2. sets the failed tray's NetBox status to `offline` just before the NetBox rewrite, so no Region discovers it. A tray already in a status no Region discovers, such as `rma`, keeps it; 3. triggers the Region's topology sync straight after the rewrite. A rerun declares under its own job's intent. A failed declaration can still leave one behind, though: if topology-service took the intent but its answer was lost, `node-swap-` stays pending and excuses the failed tray's removal for up to six hours. Before treating that removal as undeclared, or taking the tray out of NetBox by hand, read `GET /internal/v1/regions//sync/intents/node-swap-` on topology-service, as its breaking-change runbook reads the sync status. The job's result says whether the removal was declared. A site that the hub maps to no topology Region (the topology agent's targets plus `api.extraTopologyTargets`) skips the declaration and the trigger; the tray still goes `offline`. After the swap, read the Region's sync status with `fleetctl api topology-sync-status get `: * `published` or `unchanged`: done. * `failed` with code `timeout`, or `held` with code `breaking_topology_change`: read it again after the next hourly scheduled sync. A triggered sync can time out on a large Region, and a scheduled sync that was already running when the swap declared the removal holds it as a breaking change. The next scheduled sync publishes the removal while the declaration excuses it, for six hours. * Any other code, or either of those still there after the next scheduled sync: escalate to Fleet Manager on-call with the status output. The Region keeps serving its pre-swap topology until it publishes, and a hold follows topology-service's [`breaking_topology_change` runbook](https://github.com/nscaledev/topology-service#runbook-breaking_topology_change-hold). Move the failed tray to `rma` in NetBox once the vendor RMA is raised. After repair it still sits uncabled in the spare's old slot: cable it in NetBox before setting it back to `staged` to re-enrol it, because at LON14 and OSL1 a `staged` tray with no cables holds the Region again. ## A spare with NetBox cables (cabled swap) A swap with a cabled spare moves both trays: each takes the other's cables and rack slot, so both move in the topology, and topology-service holds that move until a Sync Intent declares the pair. Straight after the NetBox rewrite the workflow declares the swap (`expect_nodes_swapped`, under the `node-swap-` intent) and triggers the Region's topology sync, which publishes the exchange. A `netbox_only` rewrite of two cabled trays does the same. topology-service pins the pair to where both trays sit in the topology it is serving when the intent arrives. The exchange is the only change the intent excuses: anything else, a tray landing somewhere else included, still holds. Once the exchange is published, swapping the trays back is a move like any other and holds until that swap is declared too. A failed declaration does not fail the swap: the records are already rewritten. The job's result says so, as does its failure message if a later step fails, and the Region holds on `breaking_topology_change`, naming both trays, until the pair is declared. Declare it and start the sync that publishes it in one call, as the workflow does, with `POST /internal/v1/regions//sync` on topology-service: ```json theme={null} {"intent": {"id": "node-swap-", "expect_nodes_swapped": [["", ""]], "ttl_seconds": 21600}} ``` Keep `ttl_seconds`: an intent declared without it expires after 30 minutes, and declaring through `/sync/intents` starts no sync, so the intent can expire before the hourly sync sees the swap. Then read the Region's sync status as for a cableless swap above. A site that the hub maps to no topology Region skips the declaration, and the job's result says that too. ## Recovery reruns (`recovery`) Recovery is now an explicit flag rather than something inferred from `production_uuid`, because a *fresh* dead-tray run legitimately carries that UUID too: ```json theme={null} { "replacement_host_name": "", "force_reclaim": true, "recovery": true, "production_uuid": "", "hospital_uuid": "" } ``` Use it after a failure that already applied the one-way records changes — the NetBox rewrite above all, which is symmetric and would swap straight back if re-applied. It skips the preflight, drain, allocation move and NetBox rewrite and goes to re-pinning and powering off. Add `failed_tray_unreachable: true` if the failed tray's BMC is why you are here — including the case where it died *between* the preflight probe and the re-add, which used to need manual handling. Recovery never declares a cabled swap to topology-service. If the failed run could not declare it either, its failure message says so, and the Region holds until you declare the pair as in [A spare with NetBox cables](#a-spare-with-netbox-cables-cabled-swap). Because it skips the preflight and the drain, a recovery run's first look at the pair would otherwise be the teardown, which refuses one node at a time — so a replacement that picked up an instance between the runs surfaced only after the failed tray's node was already deleted. One read-only check now runs first and refuses on either node, before anything is deleted; it says so in the message. Recovery does not drain, so clear the instance yourself and re-run. A node the failed run already deleted is not an error — re-adding it is what recovery is for — but its absence is not a clearance either. Deleting an Ironic node does not delete the Nova server that was on it, so the check reads the slot's surviving Placement provider: allocations still standing against it mean a server is still consuming the slot, and the rerun is refused the same way. Delete that server, or clear the stale allocation, then re-run. Only when the node and the provider are both gone does the slot count as clear. The same check verifies the UUIDs themselves. `production_uuid` and `hospital_uuid` are free text, and step 5 re-pins them onto this pair, so a UUID that belongs to a third host would overwrite that host's slot identity. Each one is resolved in Ironic and refused if it names a node outside the pair. A UUID whose node was deleted resolves to nothing, which is the normal recovery shape and passes. Which tray holds which UUID is not checked: a rerun of a recovery that already got through the re-add has them cross-pinned, and that is correct. Supplying `production_uuid` on its own, with no `recovery` and no dead-tray assertion, still means recovery: that is the form every recovery took before the flag existed. ## Day-0 records-only swap (`netbox_only`) During standup, trays get physically re-arranged before anything is contracted to a customer, and NetBox has to be told. The full flow above cannot help there — it demands a per-site force-reclaim opt-in, two reachable BMCs and an existing Ironic node on the failed tray. Hand-editing rack and position instead is what leaves the **cabling behind**, and a tray recorded in one rack but cabled to another rack's leaves breaks pod derivation for the whole site. So submit the same workflow on either tray with: ```json theme={null} { "replacement_host_name": "", "netbox_only": true } ``` In the UI that is the **Swap node in NetBox only** preset, seeded per resource class alongside the full **Swap node** one; it pre-fills `netbox_only` and leaves the replacement host to you. It runs two steps and stops: a commitment preflight, then the same NetBox cabling and location rewrite step 4 performs. **Nothing else** — no drain, no allocation move, no Ironic teardown or re-pin, no BMC contact, no power-off. `force_reclaim` is not needed and the per-site gate does not apply, because nothing here is destructive to a customer. It is **refused** if either tray is committed to a customer: in use, mid-[handover](../start-here/glossary#handover), carrying a [tenant](../start-here/glossary#tenant), or named by an allocation that has not yet ended (future-dated allocations included — an allocation is a commitment to the *slot*). After that point a records-only move would leave Ironic and the allocation naming the tray that left the customer's rack, silently, because Unity/UNI reads both and neither moved. Use the full flow instead. Also **refused**: either tray being part of another active job at the site (as its device, or as another swap's `replacement_host_name`) — the rewrite is symmetric, so two overlapping runs can revert or half-apply each other's records — and combining `netbox_only` with any recovery-family parameter (`recovery`, `production_uuid`, `hospital_uuid`, `failed_tray_unreachable`), which signals the wrong mode was picked. Two things to know before you run it: * **Exactly once per physical move.** The rewrite is symmetric, so a second *submission* puts NetBox back where it started — that is also how you undo a records move. Within a single job you are safe: the preflight captures where the pair sits *and* how it is cabled, and each retry applies only the half that has not landed — no-oping outright once both have. (The full swap works the same way; see *Why the NetBox rewrite retries itself*.) So a job that failed *after* applying does not need — and must not get — a fresh submission without checking NetBox first. * **A genuinely half-applied rewrite fails loud rather than retrying.** If an attempt leaves the records in an arrangement no attempt of ours could have produced, every retry refuses with `neither the pre- nor post-swap arrangement` instead of guessing. Repair NetBox by hand — the failed attempt logs the full pre-swap cable payloads, plus which cables it deleted and which it recreated swapped, for exactly this — and only then resubmit. The same refusal covers a pair whose rack positions moved without its cabling, whether they were hand-edited or left that way by a rewrite that applied and was then inverted by a blind rerun. * **A node already adopted into SLURM keeps its old rack's features.** `rack-*` / `row-*` features and the topology group are resolved from NetBox at adopt time only, so re-adopt the node after the move or it will fail placement with `BadConstraints`. Being racked in NetBox is still required of both trays (same site, both with a rack position, different racks, same model and role) — that contract is what makes the cable move derivable, and it is checked before anything is written. A spare with no U is refused here: this mode leaves both Ironic nodes in place, so the failed tray would stay enrolled with no slot. ## Step 2 — physical swap (datahall) The workflow's success message is the go-signal: *NetBox is updated, go make it so.* A technician physically **moves and re-cables the two trays to match NetBox**: the spare into the failed tray's production slot, the failed tray to the hospital rack. Wire each exactly as NetBox now shows. ## Step 3 — enrol each node Once the physical move matches NetBox, run the normal **Enrolment** workflow on **each** node. Each resumes from `manageable`, so enrolment runs inspection (which validates the front-end cabling against NetBox) and cleaning, ending `available`. * **Replacement** (now in production): on a clean pass, release it to the customer. * **Failed tray** (now in hospital): send it into SLURM for diagnosis / RMA triage. > A miscable surfaces here as `CABLE_LLDP_VERIFICATION_FAILED` during inspection (routes > to dc\_eng) — the swap cannot reach customer service with the wrong cabling. > Releasing straight to customer on pass, and dropping the bad node straight into SLURM, > are planned enrol-playbook options (one action). For now they are separate steps. ## Hardware without a `node_swap` mapping `stale_test_operations.yaml` defines which tests a `node_swap` invalidates **per device-type resource class**. Today it covers **GB300 only**. **A swap of hardware with no entry — B200 included — stops at step 2 and leaves both trays' records alone.** The failure is `No stale-test mapping defined for operation 'node_swap' on device type(s): …`, classified as a validation error. Nothing is cordoned, no instance is deleted, no allocation or cabling is touched: the tray keeps serving its customer and the swap can be rerun once the mapping exists. Only a never-enrolled spare's BMC may already be configured by then, as enrolment would leave it. This is deliberate. Completing a swap without invalidating the runs that qualified the trays would leave the rack reading as accepted on evidence measured over hardware that has since moved — the failure mode the stale marking exists to prevent — so the swap refuses rather than proceed silently. **To unblock a device type**, add its `node_swap` entry to `src/fleet_manager/core/burn_in/stale_test_operations.yaml` and redeploy. The list is the requalification suite for that hardware, so it is a hardware-policy decision, not a mechanical one. Use the GB300 entry as the shape, and note the manifest has no "all" sentinel by design — a new test is never swept into a mapping automatically, so even "re-run everything" is written as an explicit slug list. B200 is the one blocked platform today, tracked in FLEET-2432 (Oscar Savolainen, 2026-08-19: *"Node swap not tested on B200 yet, we'll get there"* — the swap has never been run on B200, so the block is not a regression). ## Why the NetBox rewrite retries itself Nearly everything else in this workflow fails fast, on purpose: a failed step did nothing, so you fix the cause and rerun. Step 5's NetBox cabling + location rewrite breaks that assumption twice over, which is why it is the exception. (Two other writes retry: a cableless swap's status write just before the rewrite, on the same budget, and a never-enrolled spare's BMC configuration in step 1, which keeps enrolment's retry budget because it converges from whatever state it finds.) * **It is symmetric.** Running it again does not repeat the write, it *undoes* it. So "fail fast and let the operator rerun" is not a safe default here — it is the failure mode. Your rerun is the thing that inverts the records. * **A failure usually means it applied.** The cable writes are concurrent and all awaited before the error surfaces, so a NetBox wobble mid-rewrite is normally a lost *response* over a committed *write*, not a write that never happened. Put together: the step most likely to have succeeded when it reports failure was also the step where retrying by hand does the most damage. So the machine retries it instead, against the snapshot the preflight captured — where a retry can tell "already applied" from "not yet" and converge on the right answer. An operator rerun is a *fresh* job with a *fresh* snapshot, which cannot make that distinction, and never will: across submissions the symmetry is the contract, because a second run is also how you undo a records move. This is not hypothetical. On 2026-09-02 an ICE2 swap (`gpu12` → `gpu30`) lost a rewrite to a five-second NetBox outage after it had fully applied. The job reported failure, the operator reran, the symmetric rewrite put the cabling back while the locations moved once, and the second job reported **success** over records that contradicted each other. That is the failure this exception exists to remove. ## Failure handling * **Read the failure message before you rerun anything.** A failure that touched the records says so, in capitals, at the front of the message on the job page. The `failure_code` alone cannot tell you this — it names the step that failed, not what the earlier steps had already written. **The two prefixes mean different things and take different remedies:** | Prefix | What it means | What to do | | - | - | - | | *(none)* | Nothing outside Fleet Manager changed | Fix the cause and rerun normally | | `ALLOCATION …` only | The allocation moved; NetBox was **not** rewritten | Swap the allocation back, then rerun normally. **Not** recovery mode — see below | | `NETBOX …` (with or without `ALLOCATION …`) | NetBox describes the post-swap world | Verify NetBox is *fully* applied, then continue in **recovery mode** — or revert NetBox and rerun normally | In every case with a prefix, do **not** rerun against a different spare: that layers a second swap on the first. Whatever the prefix, a never-enrolled spare's BMC may already be configured, as enrolment would leave it, and a rerun keeps it. Recovery mode skips the whole of step 4 — the allocation move **and** the NetBox rewrite — because it assumes both already landed. Entering it after an allocation-only failure leaves NetBox still describing the pre-swap world while the nodes get re-pinned and powered off, so the records end up inconsistent in a way nothing downstream will flag. The `NETBOX` prefix is what authorises recovery; the `ALLOCATION` prefix on its own does not. An `… ATTEMPTED …` prefix means the write failed *inside* its activity. For the NetBox rewrite the prefix says **most likely applied in full**, and it means it: the cable writes run concurrently and are all awaited before the failure surfaces, so a failure there is usually a lost *response* over a committed *write*. The activity log names every cable it deleted and every one it recreated swapped. Inspect the records before choosing a remedy from the table — and never read the prefix as "nothing landed, safe to rerun". * **No automatic retries, with two exceptions.** If a step fails, the Job fails fast. Fix the underlying problem and use the **retry-failed-jobs** button — do not hand-rerun a partially applied swap node workflow (see the run-once warning above). The exceptions are step 5's NetBox writes, the cabling + location rewrite and a cableless swap's status write before it, which retry themselves (see [Why the NetBox rewrite retries itself](#why-the-netbox-rewrite-retries-itself)), and a never-enrolled spare's BMC configuration, which keeps enrolment's retry budget. * **Busy, not broken.** Two failures only mean a BMC or Ironic was busy. Both stop before the records rewrite, so wait and rerun normally: * `BMC_HTTP_SERVICE_UNAVAILABLE` whose message says the host is booting (*Host System Reboot might be in progress …*): a never-enrolled spare's AMI BMC refuses boot-order writes until its host has booted. Rerun a few minutes later. * `IRONIC_NODE_LOCKED` from the dead-tray drain: the drain went to put the failed tray's node into maintenance while Ironic was still tearing down the instance it had just deleted, a minute or two until the node reaches `clean wait`. It happens only when Ironic had not already put the tray into maintenance itself. The rerun finds no instance and only sets maintenance. * **Dead BMC.** Re-add-to-`manageable` (step 7) needs each tray's BMC. The preflight (step 1) probes both up front, so a dead BMC fails the workflow *before* anything is drained or rewired — nothing to unwind, just fix the BMC and rerun (or handle the swap manually). A BMC that dies *between* the preflight and step 7 still fails at step 7; recover with recovery mode (the run-once warning applies). A never-enrolled spare whose BMC cannot be configured fails the same way, before anything is drained or rewired. One whose factory login is rejected while its vault item is missing or archived fails with credential recovery required: restore the item, or reset the BMC, and rerun. * **Uncabled tray.** The preflight rejects either tray whose **counterpart's** frontend interfaces have no cable in NetBox — the cabling it will inherit — before anything is drained or rewired. Left to step 7 this is the worst place to fail: NetBox already describes the post-swap world and both Ironic nodes are gone, so the way out is recovery mode or a hand revert — never a fresh run against a different spare, which layers a second swap on the first. * **Missing Ironic node.** The preflight (step 1) fails loud — before anything is drained or rewired — if the **failed** tray has no Ironic node, because its production UUID is what the swap exists to preserve. A **replacement** with no Ironic node is fine and is not an error; one that has a node but is in use or unsettled is rejected. * **Recovering an unenrolled-spare swap.** Recovery mode needs `production_uuid`; `hospital_uuid` stays empty when the spare had no node, exactly as the failed run logged it. Supplying `hospital_uuid` on its own is rejected. Every pre-swap rejection is classified to a precise `failure_code`, so CHAD can route it without parsing the message (the responsible team is derived from the code in the failure taxonomy): | Rejection | `failure_code` | | - | - | | Failed tray has no Ironic node | `IRONIC_INSPECTFAIL` | | Replacement still has a customer instance, or a recovery rerun found either node live before the teardown | `UNENROLLMENT_NODE_DEPLOYED_IN_NOVA` | | Replacement enrolled but not settled `AVAILABLE` | `IRONIC_NODE_UNEXPECTED_PROVISION_STATE` | | Bad pair (same device twice / missing or different-site device / unplaced failed tray, or a replacement with no rack / two placed trays in one rack / different model or role), a tray that could not be re-added to `manageable` (incomplete `config_context.ironic_settings`, no MAC-bearing frontend interface), a spare whose `ironic_settings` differ from the failed tray's on driver, boot mode, a hardware interface, resource class, node property or virtual-media setting, or a spare whose frontend MACs could not be filled from its BMC (a BMC that does not report the MAC NetBox records on the interface carrying the spare's OOB IP, a different number of ports and interfaces, an interface carrying another MAC, a MAC NetBox records elsewhere, or a BMC that cannot report them: fix NetBox, or enter the MACs by hand) | `NETBOX_DEVICE_STATUS_INVALID` | | A tray has a frontend interface with no cable in NetBox | `NETBOX_INTERFACE_CABLE_MISSING` | | Replacement contended, unknown to FM, or refused by the enrolment eligibility policy (inventory-only, non-enrollable status/class, in custody); force-reclaim not confirmed/allowed; `hospital_uuid` supplied without `production_uuid`; or a recovery slot UUID that belongs to a host outside the pair | `NODE_SWAP_VALIDATION_FAILED` | | A cableless swap could not declare the failed tray's removal: the hub has no topology-sync endpoint, topology-service is unreachable or errored, or it refused the intent | `TOPOLOGY_SYNC_INTENT_FAILED` | | A tray's BMC unreachable, or a never-enrolled spare's BMC could not be configured | `BMC_*` (e.g. `BMC_REDFISH_DISCONNECT`) | | Miscable at post-swap inspection | `CABLE_LLDP_VERIFICATION_FAILED` | ## Drill Before trusting this in anger, run a live drill on the ICE2 hospital-rack nodes with the workflow owner, support on-call, and the onsite datahall lead. Ping them once the workflow is in prod. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.