Skip to main content

​
Overview

A B200 replacement uses the same workflow as a GB300 tray swap — NodeSwapWorkflow — with no hardware-specific branch anywhere in it. A B200 is a whole 8-GPU server: one NetBox device row, one Fleet Manager Device, one Ironic node. There is no NVL72 rack fabric behind it, so a B200 swap has no NVLink-domain or IMEX dimension: nothing outside the two servers being exchanged is fenced, re-mapped or re-tested by the swap. Because the workflow is hardware-independent, the mechanics are documented once, on GB300 Compute Tray Replacement: the numbered pre-swap steps, the run-it-exactly-once warning, recovery mode, and the failure_code table. Read that page for how the swap runs. This page covers only what is different for B200:
  • which sites have B200 at all, and which one can actually be swapped today;
  • what a B200 spare must look like in NetBox before the preflight will accept it.
The workflow’s parameter descriptions, log lines and error messages say “tray” and “hospital rack” — it was built for GB300 and the vocabulary stuck. For B200 read those as “server” and “spares rack”; it is the same code path, not a different one.
B200 swaps are blocked today. The swap marks the burn-in runs that qualified both servers stale, and the test list for that comes from the node_swap entry in stale_test_operations.yaml — which currently covers GB300 only. A B200 swap therefore stops at step 2 with No stale-test mapping defined for operation 'node_swap' on device type(s): ….It stops before doing anything: the failed server is not cordoned, its instance is not deleted, and no allocation or cabling is touched. The swap is simply refused, and can be rerun once a B200 mapping exists.This is not a regression: a B200 swap has not been run end-to-end yet either way (Oscar Savolainen, 2026-08-19: “Node swap not tested on B200 yet, we’ll get there, but not blocking on this PR”). Unblocking it means adding a B200 node_swap entry to src/fleet_manager/core/burn_in/stale_test_operations.yaml — a hardware-policy call about which tests requalify a B200 — tracked in FLEET-2432. See Hardware without a node_swap mapping.

​
Where a B200 swap can run

B200 exists at exactly two sites, and they are not equivalent. Note the slug divergence at STA1: Fleet Manager calls the site stavanger while NetBox calls it sta1. Anything scoped by FM site slug needs stavanger. STA1 cannot be swapped today. There is no spare to swap in, and all 255 production B200 are in_use under live customer allocations (NVIDIA, 233; Dynamo SemiAnalysis, 22). A swap needs an unallocated spare sitting in a different rack from the node it replaces — pulling a production node out of a live allocation to act as the spare is not a swap, it is an outage for a second customer. Making STA1 swappable means racking and staging spares, not configuring anything. OSL1 is the viable target.

​
Force-reclaim must be enabled at the site

The swap drains the failed node, which can delete the customer’s Nova instance, so it requires the per-site opt-in WorkflowAgentConfig.force_reclaim_allowed (src/fleet_manager/settings/agents/workflow.py) — plus force_reclaim: true on the job itself when the failed node still has an instance bound. The flag is set per site in k8s-cluster-fleet-manager, alongside ICE2 and LON14. OSL1 has it enabled; STA1 does not. So OSL1 is blocked only on spare staging, while a run at STA1 fails naming the site gate rather than the parameter — the per-site gate is evaluated before the operator’s confirmation. Enabling it is a deliberate decision: it authorises destroying a running customer workload, and at a trait site it also unlocks quarantine’s force-reclaim take-back. NodeSwapWorkflow additionally requires quarantine_flavor = trait, because the swap’s only cordon is the customer-ready trait and that trait does not exist at a legacy_slurm site. STA1 is therefore rejected on the flavour before anything else is evaluated — one more reason it is not a target. OSL1 is trait and unaffected.

​
Spare pre-staging checklist

This is the part that actually blocks a B200 swap. gpu1001, the only B200 spare at OSL1, fails four of these gates today, and every one of them is a NetBox precondition an operator can fix before dispatching anything. Check each against the production node the spare will replace, not against an abstract standard — most gates are equality checks between the two devices. Gates 1, 2, 3 and 4 are the ones gpu1001 fails.

​
1. Role must match the production node

validate_swap_pair (src/fleet_manager/core/domain/tray_swap.py) rejects a pair whose NetBox role differs. Production B200 at OSL1 is gpu-node (role 3); gpu1001 is gpu-spare (role 139), so the pair is refused before anything else is looked at. This is not cosmetic. Role — not device type — is what resolves a device’s Fleet Manager device class, and device class selects which Ironic instance the swap reads. A same-model device under a different role can live on the other instance, where it reads as “no Ironic node” and is indistinguishable from a never-enrolled spare — so the swap would mint a second node for it pinned to the production UUID while its real node lingered elsewhere. Clear it by re-roling the spare to gpu-node — see There is no spare role.

​
2. The spare must be a Fleet Manager device

The preflight refuses a replacement that is not an FM device at the site (“Pick a managed spare”, src/fleet_manager/temporal/activities/tray_replacement.py), because the swap ends by re-adding both nodes and the operator then has to enrol the spare. The NetBox agent’s ingest filter matches on cluster AND role together — each entry in NETBOX_AGENT_DEVICE_FILTERS pairs one cluster_id with a role__in list (see deployment/charts/hub/values.yaml and src/fleet_manager/settings/agents/netbox.py), and the role is separately mapped to a device class through deviceRoleClassMapping. For OSL1 the whitelisted pair is cluster sys2-osl1 with roles gpu-node / openstack-cpu-hypervisors. So a spare fails this gate on either half, and gpu1001 currently fails on both: its role is gpu-spare, and its NetBox cluster is unset. Fixing only the role leaves it just as invisible — with no cluster it matches no filter entry whatever its role. It needs cluster = sys2-osl1 and role = gpu-node. That FM is blind to these devices is not an accident to be worked around: the role’s own NetBox description reads “Spare GPU nodes are assigned this role so they do not show in Fleet Manager”. Making a spare swappable means deliberately reversing that. Beyond existing, the spare must also:
  • pass the enrolment eligibility policy — the same policy the post-swap enrolment job will be held to, so not an inventory-only projection row, not a non-enrollable inventory or operational status, and an enrollable device class;
  • hold no allocation, current or future-dated. A spare under contract to a tenant is refused: step 5 moves the failed node’s memberships onto it;
  • not be named by another active job at the site, as a device or as another swap’s replacement.
See There is no spare role for the role half, and note that the cluster half also clears gate 3 on its own.

​
3. config_context.ironic_settings

gpu1001’s config_context is empty ({}). The preflight requires, for both devices, that config_context.ironic_settings exists and sets a non-empty raid_interface — checked up front precisely so a stock spare with no Ironic config fails while nothing is broken, rather than at the last step after the customer instance is deleted and NetBox is rewired. Do not hand-write this block: assign the cluster and NetBox will supply it. The context that gives OSL1 B200s their ironic_settings is sys2-osl1-ironic_settings_SR680a_v3, scoped by device type and cluster — role is not one of its criteria. gpu1001 already matches the device type; it misses only on cluster being unset. Setting cluster = sys2-osl1 inherits all fifteen keys that context carries — ironic_settings plus expected_hw_description and the bios_checks / bmc_checks / cpu_checks / disk_checks / gpu_checks / mellanox_checks / network_checks family the health gates read. Writing ironic_settings per device would satisfy this gate and silently leave the other fourteen missing. Re-roling does not populate anything here, and gains no unwanted context either: the only gpu-node-exclusive contexts are scoped to device types and sites that an OSL1 SR680a cannot match. Reference values from a production OSL1 B200, for verification after the cluster is set: Copy the whole block verbatim from a production sibling at the same site rather than hand-writing it. capabilities, bond, and the exact target_raid_config have to match the production image and are not guessable; a software-RAID node also deliberately omits root_device so IPA auto-selects the md array. The full field list and the rules for each are on DeviceIronicSettings in src/fleet_manager/core/clients/netbox/schemas.py. resource_class earns attention twice over: besides Ironic, it is the key the workflow presets are seeded under, so B200_8NIC is also what makes the swap dispatchable from the UI.

​
4. Frontend interfaces: right type, real MACs, matching names

gpu1001 has zero MAC-bearing interfaces, no cables, and no interface of the production interface_type at all. Three separate requirements sit here:
  • Type and MAC. At least one interface of ironic_settings.interface_type carrying a MAC address — 200gbase-x-qsfpdd at OSL1, 200gbase-x-qsfp56 at STA1. This is asked via list_frontend_interfaces_with_mac (src/fleet_manager/core/domain/ironic_enrolment.py), the same function enrolment itself calls, so it cannot drift from what the final step demands. Without it the preflight raises NoUsableFrontendInterfacesError. A B200 spare with no Ironic node gets no help from the swap here: it fills missing MACs only from a GB300 tray’s BMC, so a B200 spare without them is refused one step later, once its BMC has been configured and probed, under the same NETBOX_DEVICE_STATUS_INVALID.
  • Names must match the production node’s cabled ports, e.g. PCIe-9-200G-0 and PCIe-9-200G-1. The swap moves the production slot’s cables onto the spare’s identically-named interfaces; a cabled interface with no same-named counterpart on the other device is refused outright by the unmatched guard in swap_device_cabling_and_location.
  • MACs must be the real per-port hardware addresses. Ironic ports are registered from them, and post-swap inspection matches LLDP and DHCP against them — a plausible-looking placeholder passes the preflight and fails inspection after the hardware has physically moved.
The spare’s own cabling in the spares rack is not required by the swap: it inherits the production slot’s cables, and the failed node leaves with none. What matters is that the names line up.

​
5 and 6. Placement, pair identity, BMC

The remaining conditions are the same ones the GB300 page lists, and OSL1’s spares racks satisfy the rack condition naturally: A6 and B6 are distinct from every production rack, so a spare there is never in the same rack as the node it replaces. The spare also has to be the same device type and same site as the production node, and have a reachable BMC with an OOB IP in NetBox — both devices are probed before anything is drained. A box-fresh spare that NetBox has in a rack with no U follows Box-fresh spare with no rack position.

​
Verify before you dispatch

Run the NetBox data-contract check against the staged spare:
(ENVIRONMENT and OP_VAULT_SELECTOR must be set for the target site.) The lookup is an exact NetBox name match, so pass the fully-qualified device name — the short form the FM UI shows (gpu1001) will not resolve. One caveat: that script asserts the full production contract, so a spare that is not cabled in the spares rack also reports interfaces and cable_connectivity errors that the swap itself does not require (see gate 4). Everything else it flags is a genuine blocker. After that, the preflight is the safe rehearsal — there is no dry-run mode, but the preflight is step 1 and every rejection above happens before the failed node is drained or NetBox rewired. A rejected preflight costs nothing to fix and rerun.

​
There is no spare role

A spare carries the same NetBox role as the hardware it will replace. gpu-spare is not a convention to model against — it is an OSL1 anomaly, and re-roling to gpu-node is part of staging a spare, not a workaround. The GB300 fleet is the precedent, and it marks spares by rack name, not role: ICE2 has racks literally named Spare A and Spare C, and every compute tray in them carries the production compute-tray role. That is why a GB300 swap passes gates 1 and 2 with nothing special done to it. Role equality is not an incidental check the swap could relax — role resolves device class, and device class selects which Ironic instance the swap reads (see gate 1). gpu-spare (NetBox role 139) is held by exactly four devices, all at OSL1 — audited across every site, with no site filter: All four are NetBox status inventory, none is an FM device, and none has an Ironic node. Note only gpu1001 is a B200 — the other three are H200. Re-roling alone fixes nothing. Both the FM-ingest filter and the ironic_settings config context key on cluster, and gpu1001 has no cluster set. Staging it needs cluster = sys2-osl1 and role = gpu-node: the cluster clears gates 2 and 3, the role clears gates 1 and 2. Neither creates its MACs or its PCIe-9-200G-* interfaces — that stays per-device work. The three H200 rows are safe to re-role but gain nothing: they sit in cluster sys1-osl1, which is not whitelisted for ingest at all. Re-role them only to retire the role cleanly. Do not instead add gpu-spare to the NetBox agent’s deviceFilters and deviceRoleClassMapping. That is a fleet-wide deployment change, it pulls every spare into Fleet Manager inventory and site capacity counts, and on its own it still does not make a swap possible — validate_swap_pair compares roles, so a gpu-spare replacement remains refused against a gpu-node production node. Two things to settle before the role is deleted rather than merely emptied:
  • NetBox object permissions constraining on role 139 are unverified — that endpoint is not readable with a read-only token. Someone with NetBox admin should confirm none exists.
  • Device type 159’s interface template is the problem behind gate 4. It ships eth1/1 / eth1/2 as 400gbase-x-qsfp112 plus RDMA1-8; the 111 production nodes were each reworked by hand to PCIe-9-200G-0 / -1 on 200gbase-x-qsfpdd. gpu1001 is a pristine template instantiation, so every future B200 spare repeats this until the template is corrected.

​
Running the swap

Parameters and sequence are identical to GB300 — see Step 1 there for the full description, and the failure-code table for triage:
Dispatch on the failed node, with the spare named as the replacement. Flip force_reclaim to true only when the failed node still has a Nova instance bound; false is right for the usual case. Force-reclaim must be enabled at the site regardless. replacement_host_name is free text and takes the fully-qualified device name, not the short form the UI displays; pass the short form and the preflight rejects it, naming the fully-qualified device it thinks you meant. The UI path is the same as GB300: pick the failed node → Swap node → set the replacement name and tick force-reclaim if the node still carries a customer instance. A B200_8NIC-swap preset in src/fleet_manager/temporal/workflows/presets.yaml renders the action, keyed by resource class exactly as the GB300 preset is. Presets are seeded on deploy, not by a migration. To dispatch without the UI: POST /api/v1/jobs with the failed node’s device, NodeSwapWorkflow, and the parameters above. A spare with no NetBox cables gets the same topology handling as a GB300 cableless swap — see A spare with no NetBox cables. A cabled spare gets the GB300 cabled-swap handling — see A spare with NetBox cables. Everything after the workflow completes is unchanged from GB300: the technician makes the datahall match NetBox (Step 2), then the normal Enrolment workflow runs on each node (Step 3). Only the destinations are named differently: the replacement goes on to the customer, and the failed server goes to the spares rack and then into SLURM for diagnosis or RMA triage.
Why a stale SLURM record matters. slurmctld keeps a node’s record when its slurmd dies, and a dynamic node’s record is never refreshed: a later self-registration joins the existing record and its adopt-time features and topology win silently. Those are rack-derived, so a swapped server can keep advertising its old rack, pass registration, and then fail the first rack-scoped collective with BadConstraints — a rack-scale job rejected in about two minutes with no runtime. Both nodes are handled, and no manual step is needed. tear_down_ironic_activity (src/fleet_manager/temporal/activities/unenrollment.py) deletes a node’s SLURM node object as part of the Ironic teardown, and the swap tears down both nodes in step 6 — so the failed node and the spare are each cleared, including a spare that was previously adopted in the spares rack. force_reclaim_host (src/fleet_manager/core/domain/slurm_adoption.py) also deletes it during the drain, which is what covers a quarantine take-back that never tears down Ironic.