GB300 Compute Tray Replacement

​

Overview

1. Swap node workflow (Fleet Manager) — run on the failed tray. Drains it, updates NetBox to the target wiring/location, deletes both Ironic nodes and re-adds them with cross-pinned UUIDs to manageable, powered off. Everything except the physical move.
2. Physical swap (datahall) — a technician moves + re-cables the two trays to match NetBox.
3. Enrolment (Fleet Manager) — run the normal enrol on each node; inspection then validates the cabling matches, and each node goes on to the customer or into SLURM.

Moving trays during standup, before anything is contracted? You want netbox_only mode, not the full flow above. It rewrites NetBox cabling and location and stops. Failed tray’s BMC dead, or its Ironic node deleted out-of-band? That is failed_tray_unreachable — still a full swap, minus the work that needed the dead half’s BMC.

Run it exactly once per physical swap. The NetBox swap and UUID cross-pin invert state, so re-running the swap node workflow on the same pair swaps everything back. If it fails partway, inspect NetBox and Ironic before doing anything else.

​

Preconditions

• The replacement tray is a spare modelled in NetBox at the same site — placed in a rack position in the hospital/spare rack, or in any rack with no U (see Box-fresh spare with no rack position) — is the same model and role as the failed tray, and has firmware levelled to production. It does not have to be enrolled: a spare straight from stock has no Ironic node, and the workflow handles that — the failed tray simply gets a freshly minted UUID instead of the spare’s. (Only the production slot’s UUID has to survive the swap; that is what reservations key on.) An already-enrolled spare must be settled available with no instance attached.
• The replacement is a tray Fleet Manager could enrol after the swap — it exists in FM at the site and passes the same eligibility policy the post-swap enrolment job will apply (so not an inventory-only projection row, not decommissioned or otherwise non-enrollable, not in customer custody, and an enrollable device class).
• Both trays can actually be re-added to manageable: NetBox config_context.ironic_settings is present with a non-empty raid_interface, and each tray has at least one frontend interface carrying a MAC address so Ironic ports can be registered. A spare with no Ironic node whose frontend interfaces lack MACs is the exception on a GB300 tray: the swap fills them from its BMC in step 1 (see Box-fresh spare with no rack position).
• The spare’s Ironic settings match the failed tray’s. The swap re-adds the spare from its own ironic_settings, so the preflight refuses a spare whose block differs from the failed tray’s on driver, boot_mode, a hardware interface (raid, boot, inspect, bios, management, power, vendor, network), resource_class, a node property (capabilities, cpu_arch, vendor) or a virtual-media setting (vmedia_transport_protocol, boot_iso_name_no_hyphens), and names each difference. ironic_driver_info is not compared: it is free-form, so check it by hand when a spare carries its own. It compares NetBox with NetBox, never with the failed tray’s live node: LON14 switches enrolled nodes to ipmi after inspection, so a live node there never matches its own block. Under failed_tray_unreachable the dead tray’s NetBox block is still the reference; if it has none, the check is skipped and only the worker log says so. A mismatch usually means the spare is in another cluster: put it in the failed tray’s cluster. Where either tray carries a per-device block, decide which settings the slot needs rather than hand-writing a match.
• Both trays are cabled in NetBox. Registering an Ironic port records the switch port the interface’s cable lands on, so a frontend interface with no cable fails the re-add. The swap moves each tray onto the other tray’s cables, so the preflight checks each tray against its counterpart’s cabling — an uncabled hospital rack is what breaks the failed tray’s re-add, not the spare’s. ironic_settings.skip_cable_verification exempts a tray, and it has to be set on the tray moving into the uncabled rack (the one that will have no cables), not on the uncabled tray itself. In practice both trays resolve the same ironic_settings — production scopes that block by device type and cluster, and a swap changes neither — so the flag is a property of the pair, not of one side; do not hand-write it per device (see the B200 runbook). Otherwise get the cabling into NetBox before submitting. Under failed_tray_unreachable only the replacement is checked — it is the only tray re-added — so an uncabled hospital rack is no obstacle to a dead-tray swap. The preflight checks all of the above because an unenrolled spare proves none of it: an inventory-only row can be fully racked with a matching model and role, and a stock spare has never had its Ironic config or its cabling exercised. Every one of them would otherwise fail at the last step — after the customer instance is deleted and NetBox is rewired.
• Double-check the replacement name. It is free text, so it is the one parameter the API cannot validate for you. A name that exists in NetBox but is the wrong hardware is rejected by the preflight on the model check — but that check is what stands between a typo and a drained customer node, so read it back before submitting. While the swap is active, Fleet Manager refuses any other job on the spare it names, an enrolment included. A swap naming a spare that another active job is using, or that another active swap names, is refused at submission.
• You know both device names: the failed tray and the replacement spare.
• Both trays’ BMCs are reachable. The Ironic node delete needs no BMC, but re-adding to manageable talks to each tray’s BMC. The preflight (step 1) probes both and fails fast if either is unreachable — before anything is drained or rewired (see Failure handling). A dead replacement BMC always fails here: fix it or stage a different spare. A dead failed-tray BMC is a supported input — see failed_tray_unreachable. A replacement with no Ironic node needs no BMC login beforehand where the site’s enrolment configures BMCs and that configuration sets the BMC’s password, as it does on every GB300 tray’s AMI BMC: step 1 configures its BMC as enrolment would, creating the login in the operators vault, before probing it. Supermicro’s configuration sets no password, so a Supermicro spare still needs its login stored first.

​

Step 1 — run the swap node workflow

{ "replacement_host_name": "<spare-tray-device-name>", "force_reclaim": false }

1. Preflight — verify both trays are swappable and capture the Ironic node UUIDs. The failed tray must have an Ironic node (its production UUID to preserve); the pair must be in the same site and the same model and role, the failed tray placed in a rack position and the replacement placed or in a rack with no U, in different racks when both are placed; and the replacement, if it has a node, must be a safe spare to tear down (no instance, settled available). Both trays’ BMCs must also be reachable (step 7 re-adds each via its BMC). A bad request or an unreachable BMC fails here, before anything below runs. A replacement with no Ironic node has its BMC configured first, as enrolment’s BMC step would (on a GB300 tray’s AMI BMC: a factory login rotated and stored in the operators vault, a working stored login kept), at a site whose enrolment configures BMCs. A replacement with no Ironic node whose frontend interfaces lack MACs in NetBox then has them filled from its BMC, after its probe: the BMC must report the MAC NetBox records on the interface carrying the spare’s OOB IP, and NetBox must not contradict what it reports (an interface carrying another MAC, a MAC recorded on another device). Only a GB300 tray’s AMI BMC reports them; any other spare without MACs is refused there. A run refused for want of force_reclaim is refused before either write, so it still writes nothing.
2. Mark the burn-in runs that qualified either tray stale, under the node_swap operation — a swapped rack must stop reading as accepted on evidence measured over hardware that has moved, and that covers both trays’ single-node runs plus every rack- and pod-level collective either took part in. Every hold is listed in the workflow log and is individually undoable, for a swap abandoned before the trays physically move. Which tests this invalidates is deployment policy, not workflow code: it is the node_swap entry for the trays’ device type in stale_test_operations.yaml. If no entry covers that device type, the swap stops here — the first thing it does, before the tray is cordoned and before the customer’s instance is deleted. Add the mapping and rerun; nothing has been changed beyond a never-enrolled spare’s BMC configuration and MAC fill, which the rerun keeps. (See Hardware without a mapping below.) A rerun of the same swap recognises its own earlier hold and does not create a second.
3. Remove the customer-ready trait from the failed tray.
4. Force-reclaim the failed tray to available (deletes the customer instance, cleans). This is the first irreversible step in the swap — past here, the customer’s workload is gone. Skipped when no instance is bound to it — there is nothing to reclaim, and step 6 deletes the node outright anyway. The check is re-read from Ironic here, after step 3 has cordoned the tray: the preflight reads it too, but the tray is still schedulable until the trait comes off, so a workload landing in between is reclaimed normally.
5. Move the failed tray’s allocation memberships onto the replacement, so the customer’s allocation names the tray now in their rack, then update NetBox cabling + location to the desired post-swap state. Allocation first: it is one API transaction and can be swapped straight back, whereas the NetBox rewire has no rollback. The workflow retries the NetBox rewrite itself, against the snapshot step 1 captured — see Why the NetBox rewrite retries itself.
6. Delete both Ironic nodes and their Placement resource providers. A dead tray whose node was already deleted has its provider deleted by the supplied production_uuid: it would otherwise sit on the very UUID the replacement is re-pinned to. The spare’s teardown runs even when it had no node. This also clears each tray’s SLURM node object — both trays change rack, and a surviving dynamic-node record would hand the next adoption the tray’s old row and rack features, which passes registration and then fails the first rack-scoped collective with BadConstraints.
7. Re-add both to manageable with cross-pinned UUIDs (no inspection). An unenrolled spare leaves nothing for the failed tray to inherit, so Ironic mints it a fresh UUID.
8. Park both in Ironic maintenance and power them off. Maintenance is what makes the power-down stick — Ironic’s power sync would otherwise turn a hand-powered tray straight back off — so the re-racked tray can be powered up by hand to verify. Enrolment clears the flag.

​

Swapping a tray whose BMC is dead (failed_tray_unreachable)

{
  "replacement_host_name": "<spare-tray-device-name>",
  "force_reclaim": true,
  "failed_tray_unreachable": true
}

​

Box-fresh spare with no rack position

1. Stage the spare’s records. It needs a rack assignment (not a U), an OOB IP that the site’s DHCP hands out, on the interface that carries its BMC’s MAC, and NetBox status staged when the swap runs, because the preflight holds it to the enrolment policy and enrolment accepts only staged and active. A GB300 spare needs no BMC login and no frontend MACs: the swap creates the login in step 1, where the site’s enrolment configures BMCs (a Supermicro spare still needs its login stored first), and then fills its eth0 and eth1 MACs from the BMC, once the BMC reports the MAC NetBox records for it. Any other spare needs its frontend interfaces with their real MACs. The swap configures whatever BMC answers at that OOB IP, though, so check the IP is the spare’s own before you submit.
2. Create the record as inventory and move it to staged just before the swap. Fleet Manager syncs the record as soon as it exists. A spare that moves from planned to staged emits a standup postcard, and CHAD opens a day-zero enrolment pipeline for it, which Fleet Manager refuses while the spare has no U; inventory to staged emits no postcard at all. At LON14 and OSL1 the order matters twice over: their topology Regions discover planned and staged devices, and a discovered tray with no cables holds the Region until the swap moves the failed tray’s cables onto it. Wait until Fleet Manager shows the spare as staged before you submit.
3. Once the spare physically sits in the failed tray’s U, submit the swap with failed_tray_unreachable: true. A normal swap re-adds the failed tray to Ironic on the spare’s cables, and the spare has none in NetBox. Dead-tray mode never re-adds the failed tray, so it ends with no Ironic node, and the failed tray is out of the rack by then, so the assertion is true as well. Without the flag the preflight refuses the swap with NETBOX_INTERFACE_CABLE_MISSING, and the error names the flag. netbox_only does not take a spare with no U.
4. After the swap, confirm the failed tray left topology. The swap declares the failed tray’s removal and sets it offline itself (see A spare with no NetBox cables). Move it to rma once the vendor RMA is raised, and read the Region’s sync status as that section describes, escalating anything it does not list:
   fleetctl api topology-sync-status get <region>
   
5. Enrol the spare, which NetBox now shows in the failed tray’s slot. Enrolment refuses a device with no U, so it becomes eligible once the NetBox agent syncs the new slot.

​

A spare with no NetBox cables (cableless swap)

1. declares the failed tray’s removal to topology-service (a Sync Intent named node-swap-<job id>) before its first write to Fleet Manager or the NetBox records. If that call fails, the swap stops with TOPOLOGY_SYNC_INTENT_FAILED before any such write (a never-enrolled spare’s BMC may already be configured, which a rerun keeps); rerun once topology-service is back;
2. sets the failed tray’s NetBox status to offline just before the NetBox rewrite, so no Region discovers it. A tray already in a status no Region discovers, such as rma, keeps it;
3. triggers the Region’s topology sync straight after the rewrite.

• published or unchanged: done.
• failed with code timeout, or held with code breaking_topology_change: read it again after the next hourly scheduled sync. A triggered sync can time out on a large Region, and a scheduled sync that was already running when the swap declared the removal holds it as a breaking change. The next scheduled sync publishes the removal while the declaration excuses it, for six hours.
• Any other code, or either of those still there after the next scheduled sync: escalate to Fleet Manager on-call with the status output. The Region keeps serving its pre-swap topology until it publishes, and a hold follows topology-service’s breaking_topology_change runbook.

​

A spare with NetBox cables (cabled swap)

{"intent": {"id": "node-swap-<job id>", "expect_nodes_swapped": [["<failed>", "<spare>"]], "ttl_seconds": 21600}}

​

Recovery reruns (recovery)

{
  "replacement_host_name": "<spare>",
  "force_reclaim": true,
  "recovery": true,
  "production_uuid": "<from the failed run's logs>",
  "hospital_uuid": "<from the failed run's logs, if the spare was enrolled>"
}

​

Day-0 records-only swap (netbox_only)

{ "replacement_host_name": "<other-tray-device-name>", "netbox_only": true }

• Exactly once per physical move. The rewrite is symmetric, so a second submission puts NetBox back where it started — that is also how you undo a records move. Within a single job you are safe: the preflight captures where the pair sits and how it is cabled, and each retry applies only the half that has not landed — no-oping outright once both have. (The full swap works the same way; see Why the NetBox rewrite retries itself.) So a job that failed after applying does not need — and must not get — a fresh submission without checking NetBox first.
• A genuinely half-applied rewrite fails loud rather than retrying. If an attempt leaves the records in an arrangement no attempt of ours could have produced, every retry refuses with neither the pre- nor post-swap arrangement instead of guessing. Repair NetBox by hand — the failed attempt logs the full pre-swap cable payloads, plus which cables it deleted and which it recreated swapped, for exactly this — and only then resubmit. The same refusal covers a pair whose rack positions moved without its cabling, whether they were hand-edited or left that way by a rewrite that applied and was then inverted by a blind rerun.
• A node already adopted into SLURM keeps its old rack’s features. rack-* / row-* features and the topology group are resolved from NetBox at adopt time only, so re-adopt the node after the move or it will fail placement with BadConstraints.

​

Step 2 — physical swap (datahall)

​

Step 3 — enrol each node

• Replacement (now in production): on a clean pass, release it to the customer.
• Failed tray (now in hospital): send it into SLURM for diagnosis / RMA triage.

A miscable surfaces here as CABLE_LLDP_VERIFICATION_FAILED during inspection (routes to dc_eng) — the swap cannot reach customer service with the wrong cabling.

Releasing straight to customer on pass, and dropping the bad node straight into SLURM, are planned enrol-playbook options (one action). For now they are separate steps.

​

Hardware without a node_swap mapping

​

Why the NetBox rewrite retries itself

• It is symmetric. Running it again does not repeat the write, it undoes it. So “fail fast and let the operator rerun” is not a safe default here — it is the failure mode. Your rerun is the thing that inverts the records.
• A failure usually means it applied. The cable writes are concurrent and all awaited before the error surfaces, so a NetBox wobble mid-rewrite is normally a lost response over a committed write, not a write that never happened.

​

Failure handling

• Read the failure message before you rerun anything. A failure that touched the records says so, in capitals, at the front of the message on the job page. The failure_code alone cannot tell you this — it names the step that failed, not what the earlier steps had already written. The two prefixes mean different things and take different remedies:

  Prefix	What it means	What to do
  (none)	Nothing outside Fleet Manager changed	Fix the cause and rerun normally
  ALLOCATION … only	The allocation moved; NetBox was not rewritten	Swap the allocation back, then rerun normally. Not recovery mode — see below
  NETBOX … (with or without ALLOCATION …)	NetBox describes the post-swap world	Verify NetBox is fully applied, then continue in recovery mode — or revert NetBox and rerun normally

  In every case with a prefix, do not rerun against a different spare: that layers a second swap on the first. Whatever the prefix, a never-enrolled spare’s BMC may already be configured, as enrolment would leave it, and a rerun keeps it.
  Recovery mode skips the whole of step 4 — the allocation move and the NetBox rewrite — because it assumes both already landed. Entering it after an allocation-only failure leaves NetBox still describing the pre-swap world while the nodes get re-pinned and powered off, so the records end up inconsistent in a way nothing downstream will flag. The NETBOX prefix is what authorises recovery; the ALLOCATION prefix on its own does not.
  An … ATTEMPTED … prefix means the write failed inside its activity. For the NetBox rewrite the prefix says most likely applied in full, and it means it: the cable writes run concurrently and are all awaited before the failure surfaces, so a failure there is usually a lost response over a committed write. The activity log names every cable it deleted and every one it recreated swapped. Inspect the records before choosing a remedy from the table — and never read the prefix as “nothing landed, safe to rerun”.
• No automatic retries, with two exceptions. If a step fails, the Job fails fast. Fix the underlying problem and use the retry-failed-jobs button — do not hand-rerun a partially applied swap node workflow (see the run-once warning above). The exceptions are step 5’s NetBox writes, the cabling + location rewrite and a cableless swap’s status write before it, which retry themselves (see Why the NetBox rewrite retries itself), and a never-enrolled spare’s BMC configuration, which keeps enrolment’s retry budget.
• Busy, not broken. Two failures only mean a BMC or Ironic was busy. Both stop before the records rewrite, so wait and rerun normally:
  • BMC_HTTP_SERVICE_UNAVAILABLE whose message says the host is booting (Host System Reboot might be in progress …): a never-enrolled spare’s AMI BMC refuses boot-order writes until its host has booted. Rerun a few minutes later.
  • IRONIC_NODE_LOCKED from the dead-tray drain: the drain went to put the failed tray’s node into maintenance while Ironic was still tearing down the instance it had just deleted, a minute or two until the node reaches clean wait. It happens only when Ironic had not already put the tray into maintenance itself. The rerun finds no instance and only sets maintenance.
• Dead BMC. Re-add-to-manageable (step 7) needs each tray’s BMC. The preflight (step 1) probes both up front, so a dead BMC fails the workflow before anything is drained or rewired — nothing to unwind, just fix the BMC and rerun (or handle the swap manually). A BMC that dies between the preflight and step 7 still fails at step 7; recover with recovery mode (the run-once warning applies). A never-enrolled spare whose BMC cannot be configured fails the same way, before anything is drained or rewired. One whose factory login is rejected while its vault item is missing or archived fails with credential recovery required: restore the item, or reset the BMC, and rerun.
• Uncabled tray. The preflight rejects either tray whose counterpart’s frontend interfaces have no cable in NetBox — the cabling it will inherit — before anything is drained or rewired. Left to step 7 this is the worst place to fail: NetBox already describes the post-swap world and both Ironic nodes are gone, so the way out is recovery mode or a hand revert — never a fresh run against a different spare, which layers a second swap on the first.
• Missing Ironic node. The preflight (step 1) fails loud — before anything is drained or rewired — if the failed tray has no Ironic node, because its production UUID is what the swap exists to preserve. A replacement with no Ironic node is fine and is not an error; one that has a node but is in use or unsettled is rejected.
• Recovering an unenrolled-spare swap. Recovery mode needs production_uuid; hospital_uuid stays empty when the spare had no node, exactly as the failed run logged it. Supplying hospital_uuid on its own is rejected.

​

Drill