> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# B200 Node Replacement

> Operator runbook for swapping a failed B200 server with a pre-staged spare — where it can run, and what a spare must look like in NetBox before the NodeSwap workflow will accept it.

## Overview

A B200 replacement uses the same workflow as a GB300 tray swap — `NodeSwapWorkflow` — with no
hardware-specific branch anywhere in it. A B200 is a **whole 8-[GPU](../start-here/glossary#gpu) server**: one [NetBox](../start-here/glossary#netbox) [device](../start-here/glossary#node)
row, one Fleet Manager Device, one [Ironic](../start-here/glossary#ironic) node. There is no NVL72 rack fabric behind it, so a
B200 swap has no [NVLink](../start-here/glossary#nvlink)-domain or IMEX dimension: nothing outside the two servers being
exchanged is fenced, re-mapped or re-tested by the swap.

Because the workflow is hardware-independent, **the mechanics are documented once**, on
[GB300 Compute Tray Replacement](./gb300-tray-replacement): the numbered pre-swap steps, the
run-it-exactly-once warning, recovery mode, and the `failure_code` table. Read that page for
how the swap runs. This page covers only what is different for B200:

* which sites have B200 at all, and which one can actually be swapped today;
* what a B200 spare must look like in NetBox before the preflight will accept it.

> The workflow's parameter descriptions, log lines and error messages say **"tray"** and
> **"hospital rack"** — it was built for GB300 and the vocabulary stuck. For B200 read those as
> "server" and "spares rack"; it is the same code path, not a different one.

<Warning>
  **B200 swaps are blocked today.** The swap marks the [burn-in](../start-here/glossary#burn-in) runs that qualified both servers
  stale, and the test list for that comes from the `node_swap` entry in
  `stale_test_operations.yaml` — which currently covers **GB300 only**. A B200 swap therefore
  stops at step 2 with `No stale-test mapping defined for operation 'node_swap' on device
    type(s): …`.

  It stops before doing anything: the failed server is not cordoned, its instance is not
  deleted, and no [allocation](../start-here/glossary#allocation) or cabling is touched. The swap is simply refused, and can be rerun
  once a B200 mapping exists.

  This is not a regression: a B200 swap has not been run end-to-end yet either way (Oscar
  Savolainen, 2026-08-19: *"Node swap not tested on B200 yet, we'll get there, but not blocking
  on this PR"*). Unblocking it means adding a B200 `node_swap` entry to
  `src/fleet_manager/core/burn_in/stale_test_operations.yaml` — a hardware-policy call about
  which tests requalify a B200 — tracked in **FLEET-2432**. See
  [Hardware without a `node_swap` mapping](./gb300-tray-replacement#hardware-without-a-node_swap-mapping).
</Warning>

## Where a B200 swap can run

B200 exists at exactly two sites, and they are not equivalent.

| | OSL1 | STA1 |
| - | - | - |
| Fleet Manager site slug | `osl1` | `stavanger` |
| NetBox site slug | `osl1` | `sta1` |
| Production B200 | 111, 2 per rack across 56 racks | 255, 4 per rack across 65 racks |
| NetBox device type | `ThinkSystem SR680a V3 - B200 GPU` (id 159) | a different type (id 107) |
| Spares | racks **A6** and **B6** in location SU-6 | none |
| Spare B200 today | **one** — `gpu1001` (device 47770, rack A6 position 13) | zero |
| `quarantine_flavor` | `trait` | `legacy_slurm` |
| Swappable today | Yes, once the spare is staged | **No** |

**Note the slug divergence at STA1**: Fleet Manager calls the site `stavanger` while NetBox
calls it `sta1`. Anything scoped by FM site slug needs `stavanger`.

**STA1 cannot be swapped today.** There is no spare to swap in, and all 255 production B200 are
`in_use` under live customer allocations (NVIDIA, 233; Dynamo SemiAnalysis, 22). A swap needs an
*unallocated* spare sitting in a *different rack* from the node it replaces — pulling a
production node out of a live allocation to act as the spare is not a swap, it is an outage for
a second customer. Making STA1 swappable means racking and staging spares, not configuring
anything. **OSL1 is the viable target.**

### Force-reclaim must be enabled at the site

The swap drains the failed node, which can delete the customer's [Nova](../start-here/glossary#nova) instance, so it requires
the per-site opt-in `WorkflowAgentConfig.force_reclaim_allowed`
(`src/fleet_manager/settings/agents/workflow.py`) — plus `force_reclaim: true` on the job
itself when the failed node still has an instance bound. The flag is set per site in
`k8s-cluster-fleet-manager`, alongside ICE2 and LON14.

**OSL1 has it enabled; STA1 does not.** So OSL1 is blocked only on spare staging, while a run at
STA1 fails naming the site gate rather than the parameter — the per-site gate is evaluated before
the operator's confirmation. Enabling it is a deliberate decision: it authorises destroying a
running customer workload, and at a `trait` site it also unlocks quarantine's force-[reclaim](../start-here/glossary#reclaim)
take-back.

`NodeSwapWorkflow` additionally requires `quarantine_flavor = trait`, because the swap's only
cordon is the customer-ready trait and that trait does not exist at a `legacy_slurm` site. STA1 is
therefore rejected on the flavour before anything else is evaluated — one more reason it is not a
target. OSL1 is `trait` and unaffected.

## Spare pre-staging checklist

This is the part that actually blocks a B200 swap. `gpu1001`, the only B200 spare at OSL1, fails
four of these gates today, and every one of them is a NetBox precondition an operator can fix
before dispatching anything.

Check each against **the production node the spare will replace**, not against an abstract
standard — most gates are equality checks between the two devices.

| # | Precondition | Required value | Enforced by |
| - | - | - | - |
| 1 | Same NetBox device role as the production node | `gpu-node` (role 3) at OSL1 | `validate_swap_pair` |
| 2 | Present in Fleet Manager, enrollable, unallocated, not claimed by another job | NetBox **cluster** `sys2-osl1` **and** role `gpu-node` — the ingest filter needs both | `preflight_swap_activity` |
| 3 | `config_context.ironic_settings` present, with a non-empty `raid_interface` | inherited by setting the NetBox cluster, not written per device | `preflight_swap_activity` |
| 4 | Frontend interfaces of the site's `interface_type`, with real MACs, named exactly as the production node's cabled ports | `200gbase-x-qsfpdd` at OSL1 | `list_frontend_interfaces_with_mac`, then the cable-move guard |
| 5 | Racked with a position in a different rack, or in any rack with no U; same device type, same site | A6/B6 vs any production rack | `validate_swap_pair` |
| 6 | Reachable [BMC](../start-here/glossary#bmc) with an [OOB](../start-here/glossary#oob) IP in NetBox | as production | the workflow's pre-drain BMC probe |
| 7 | `ironic_settings` name the same `driver`, `boot_mode`, hardware interfaces, `resource_class`, node properties (`capabilities`, `cpu_arch`, `vendor`) and virtual-media settings; `ironic_driver_info` is not compared | the production node's, inherited by setting the cluster, as for gate 3 | `preflight_swap_activity` |

Gates 1, 2, 3 and 4 are the ones `gpu1001` fails.

### 1. Role must match the production node

`validate_swap_pair` (`src/fleet_manager/core/domain/tray_swap.py`) rejects a pair whose NetBox
`role` differs. Production B200 at OSL1 is `gpu-node` (role 3); `gpu1001` is `gpu-spare`
(role 139), so the pair is refused before anything else is looked at.

This is not cosmetic. Role — not device type — is what resolves a device's Fleet Manager device
class, and device class selects **which Ironic instance** the swap reads. A same-model device
under a different role can live on the other instance, where it reads as "no Ironic node" and is
indistinguishable from a never-enrolled spare — so the swap would mint a second node for it
pinned to the production UUID while its real node lingered elsewhere.

Clear it by re-roling the spare to `gpu-node` — see [There is no spare
role](#there-is-no-spare-role).

### 2. The spare must be a Fleet Manager device

The preflight refuses a replacement that is not an FM device at the site ("Pick a managed
spare", `src/fleet_manager/temporal/activities/tray_replacement.py`), because the swap ends by
re-adding both nodes and the operator then has to *enrol* the spare.

The NetBox agent's ingest filter matches on **cluster AND role together** — each entry in
`NETBOX_AGENT_DEVICE_FILTERS` pairs one `cluster_id` with a `role__in` list (see
`deployment/charts/hub/values.yaml` and `src/fleet_manager/settings/agents/netbox.py`), and the
role is separately mapped to a device class through `deviceRoleClassMapping`. For OSL1 the
whitelisted pair is cluster `sys2-osl1` with roles `gpu-node` / `openstack-cpu-hypervisors`.

So a spare fails this gate on **either** half, and `gpu1001` currently fails on both: its role is
`gpu-spare`, and its NetBox **cluster is unset**. Fixing only the role leaves it just as invisible
— with no cluster it matches no filter entry whatever its role. It needs `cluster = sys2-osl1`
**and** `role = gpu-node`.

That FM is blind to these devices is not an accident to be worked around: the role's own NetBox
description reads *"Spare GPU nodes are assigned this role so they do not show in Fleet Manager"*.
Making a spare swappable means deliberately reversing that.

Beyond existing, the spare must also:

* **pass the enrolment eligibility policy** — the same policy the post-swap enrolment job will
  be held to, so not an inventory-only projection row, not a non-enrollable inventory or
  operational status, and an enrollable device class;
* **hold no allocation**, current or future-dated. A spare under contract to a [tenant](../start-here/glossary#tenant) is
  refused: step 5 moves the failed node's memberships onto it;
* **not be named by another active job** at the site, as a device or as another swap's
  replacement.

See [There is no spare role](#there-is-no-spare-role) for the role half, and note that the
cluster half also clears [gate 3](#3-config_contextironic_settings) on its own.

### 3. `config_context.ironic_settings`

`gpu1001`'s `config_context` is empty (`{}`). The preflight requires, **for both devices**, that
`config_context.ironic_settings` exists and sets a non-empty `raid_interface` — checked up front
precisely so a stock spare with no Ironic config fails while nothing is broken, rather than at
the last step after the customer instance is deleted and NetBox is rewired.

**Do not hand-write this block: assign the cluster and NetBox will supply it.** The context that
gives OSL1 B200s their `ironic_settings` is `sys2-osl1-ironic_settings_SR680a_v3`, scoped by
device type **and cluster** — role is not one of its criteria. `gpu1001` already matches the
device type; it misses only on cluster being unset. Setting `cluster = sys2-osl1` inherits all
fifteen keys that context carries — `ironic_settings` plus `expected_hw_description` and the
`bios_checks` / `bmc_checks` / `cpu_checks` / `disk_checks` / `gpu_checks` / `mellanox_checks` /
`network_checks` family the health gates read. Writing `ironic_settings` per device would satisfy
this gate and silently leave the other fourteen missing.

Re-roling does **not** populate anything here, and gains no unwanted context either: the only
`gpu-node`-exclusive contexts are scoped to device types and sites that an OSL1 SR680a cannot
match.

Reference values from a production OSL1 B200, for verification after the cluster is set:

| Field | Production value |
| - | - |
| `driver` | `redfish` |
| `boot_mode` | `uefi` |
| `resource_class` | `B200_8NIC` |
| `raid_interface` | `agent` |
| `target_raid_config` | software RAID1 |

**Copy the whole block verbatim from a production sibling at the same site** rather than
hand-writing it. `capabilities`, `bond`, and the exact `target_raid_config` have to match the
production image and are not guessable; a software-RAID node also deliberately omits
`root_device` so [IPA](../start-here/glossary#ironic-python-agent) auto-selects the md array. The full field list and the rules for each are on
`DeviceIronicSettings` in `src/fleet_manager/core/clients/netbox/schemas.py`.

`resource_class` earns attention twice over: besides Ironic, it is the key the workflow presets
are seeded under, so `B200_8NIC` is also what makes the swap dispatchable from the UI.

### 4. Frontend interfaces: right type, real MACs, matching names

`gpu1001` has zero MAC-bearing interfaces, no cables, and no interface of the production
`interface_type` at all. Three separate requirements sit here:

* **Type and MAC.** At least one interface of `ironic_settings.interface_type` carrying a MAC
  address — `200gbase-x-qsfpdd` at OSL1, `200gbase-x-qsfp56` at STA1. This is asked via
  `list_frontend_interfaces_with_mac` (`src/fleet_manager/core/domain/ironic_enrolment.py`), the
  same function enrolment itself calls, so it cannot drift from what the final step demands.
  Without it the preflight raises `NoUsableFrontendInterfacesError`. A B200 spare with no
  Ironic node gets no help from the swap here: it fills missing MACs only from a GB300 tray's
  BMC, so a B200 spare without them is refused one step later, once its BMC has been
  configured and probed, under the same `NETBOX_DEVICE_STATUS_INVALID`.
* **Names must match the production node's cabled ports**, e.g. `PCIe-9-200G-0` and
  `PCIe-9-200G-1`. The swap moves the production slot's *cables* onto the spare's
  identically-named interfaces; a cabled interface with no same-named counterpart on the other
  device is refused outright by the `unmatched` guard in `swap_device_cabling_and_location`.
* **MACs must be the real per-port hardware addresses.** Ironic ports are registered from them,
  and post-swap inspection matches LLDP and [DHCP](../start-here/glossary#dhcp) against them — a plausible-looking placeholder
  passes the preflight and fails inspection after the hardware has physically moved.

The spare's **own** cabling in the spares rack is not required by the swap: it inherits the
production slot's cables, and the failed node leaves with none. What matters is that the names
line up.

### 5 and 6. Placement, pair identity, BMC

The remaining conditions are the same ones the GB300 page lists, and OSL1's spares racks satisfy
the rack condition naturally: A6 and B6 are distinct from every production rack, so a spare there
is never in the same rack as the node it replaces. The spare also has to be the **same device
type** and **same site** as the production node, and have a **reachable BMC with an OOB IP in
NetBox** — both devices are probed before anything is drained. A box-fresh spare that NetBox has
in a rack with no U follows
[Box-fresh spare with no rack position](./gb300-tray-replacement#box-fresh-spare-with-no-rack-position).

### Verify before you dispatch

Run the NetBox data-contract check against the staged spare:

```bash theme={null}
uv run python scripts/checks/netbox.py --devices <fully-qualified-spare-name>
```

(`ENVIRONMENT` and `OP_VAULT_SELECTOR` must be set for the target site.) The lookup is an exact
NetBox name match, so pass the **fully-qualified** device name — the short form the FM UI shows
(`gpu1001`) will not resolve. One caveat: that script
asserts the **full production** contract, so a spare that is not cabled in the spares rack also
reports `interfaces` and `cable_connectivity` errors that the swap itself does not require (see
gate 4). Everything else it flags is a genuine blocker.

After that, the preflight is the safe rehearsal — there is no dry-run mode, but the preflight is
step 1 and every rejection above happens **before** the failed node is drained or NetBox rewired.
A rejected preflight costs nothing to fix and rerun.

## There is no spare role

**A spare carries the same NetBox role as the hardware it will replace.** `gpu-spare` is not a
convention to model against — it is an OSL1 anomaly, and re-roling to `gpu-node` is part of
staging a spare, not a workaround.

The GB300 [fleet](../start-here/glossary#fleet) is the precedent, and it marks spares by **rack name**, not role: ICE2 has racks
literally named `Spare A` and `Spare C`, and every compute tray in them carries the production
`compute-tray` role. That is why a GB300 swap passes gates 1 and 2 with nothing special done to
it. Role equality is not an incidental check the swap could relax — role resolves device class,
and device class selects which Ironic instance the swap reads (see
[gate 1](#1-role-must-match-the-production-node)).

`gpu-spare` (NetBox role 139) is held by exactly **four** devices, all at OSL1 — audited across
every site, with no site filter:

| Device | Rack | Pos | Model | Cluster | NetBox id |
| - | - | - | - | - | - |
| `gpu1001` | A6 | 13 | SR680a V3 — **B200** | *unset* | 47770 |
| `gpu415` | A6 | 35 | SR680a — H200 | `sys1-osl1` | 67361 |
| `gpu418` | B6 | 13 | SR680a — H200 | `sys1-osl1` | 10999 |
| `gpu1000` | B6 | 35 | SR680a — H200 | `sys1-osl1` | 12476 |

All four are NetBox status `inventory`, none is an FM device, and none has an Ironic node. Note
only `gpu1001` is a B200 — the other three are H200.

**Re-roling alone fixes nothing.** Both the FM-ingest filter and the `ironic_settings` config
context key on **cluster**, and `gpu1001` has no cluster set. Staging it needs
`cluster = sys2-osl1` **and** `role = gpu-node`: the cluster clears gates 2 and 3, the role
clears gates 1 and 2. Neither creates its MACs or its `PCIe-9-200G-*` interfaces — that stays
per-device work.

The three H200 rows are safe to re-role but gain nothing: they sit in cluster `sys1-osl1`, which
is not whitelisted for ingest at all. Re-role them only to retire the role cleanly.

Do **not** instead add `gpu-spare` to the NetBox agent's `deviceFilters` and
`deviceRoleClassMapping`. That is a fleet-wide deployment change, it pulls every spare into
Fleet Manager inventory and site capacity counts, and **on its own it still does not make a swap
possible** — `validate_swap_pair` compares roles, so a `gpu-spare` replacement remains refused
against a `gpu-node` production node.

Two things to settle before the role is deleted rather than merely emptied:

* **NetBox object permissions constraining on role 139 are unverified** — that endpoint is not
  readable with a read-only token. Someone with NetBox admin should confirm none exists.
* **Device type 159's interface template is the problem behind gate 4.** It ships `eth1/1` /
  `eth1/2` as `400gbase-x-qsfp112` plus `RDMA1-8`; the 111 production nodes were each reworked by
  hand to `PCIe-9-200G-0` / `-1` on `200gbase-x-qsfpdd`. `gpu1001` is a pristine template
  instantiation, so **every future B200 spare repeats this** until the template is corrected.

## Running the swap

Parameters and sequence are identical to GB300 — see
[Step 1](./gb300-tray-replacement#step-1--run-the-swap-node-workflow) there for the full
description, and the [failure-code table](./gb300-tray-replacement#failure-handling) for triage:

```json theme={null}
{ "replacement_host_name": "<spare-device-name>", "force_reclaim": false }
```

Dispatch on the **failed** node, with the spare named as the replacement. Flip `force_reclaim`
to `true` only when the failed node still has a Nova instance bound; `false` is right for the
usual case. [Force-reclaim](#force-reclaim-must-be-enabled-at-the-site) must be enabled at the
site regardless.
`replacement_host_name` is free text and takes the **fully-qualified** device name, not the short
form the UI displays; pass the short form and the preflight rejects it, naming the
fully-qualified device it thinks you meant.

The UI path is the same as GB300: pick the failed node → *Swap node* → set the replacement name
and tick force-reclaim if the node still carries a customer instance. A `B200_8NIC-swap` preset in
`src/fleet_manager/temporal/workflows/presets.yaml` renders the action, keyed by resource class
exactly as the GB300 preset is. Presets are seeded on deploy, not by a migration. To dispatch
without the UI: `POST /api/v1/jobs` with the failed node's device, `NodeSwapWorkflow`, and the
parameters above.

A spare with no NetBox cables gets the same topology handling as a GB300 cableless swap — see
[A spare with no NetBox cables](./gb300-tray-replacement#a-spare-with-no-netbox-cables-cableless-swap).
A cabled spare gets the GB300 cabled-swap handling — see
[A spare with NetBox cables](./gb300-tray-replacement#a-spare-with-netbox-cables-cabled-swap).

Everything after the workflow completes is unchanged from GB300: the technician makes the
datahall match NetBox ([Step 2](./gb300-tray-replacement#step-2--physical-swap-datahall)), then
the normal Enrolment workflow runs on **each** node
([Step 3](./gb300-tray-replacement#step-3--enrol-each-node)). Only the destinations are named
differently: the replacement goes on to the customer, and the failed server goes to the spares
rack and then into [SLURM](../start-here/glossary#slurm) for diagnosis or RMA triage.

> **Why a stale SLURM record matters.** `slurmctld` keeps a node's record when its `slurmd` dies,
> and a dynamic node's record is never refreshed: a later self-registration joins the **existing**
> record and its adopt-time features and topology win silently. Those are rack-derived, so a
> swapped server can keep advertising its old rack, pass registration, and then fail the first
> rack-scoped collective with `BadConstraints` — a rack-scale job rejected in about two minutes
> with no runtime.
>
> **Both** nodes are handled, and no manual step is needed. `tear_down_ironic_activity`
> (`src/fleet_manager/temporal/activities/unenrollment.py`) deletes a node's SLURM node object
> as part of the Ironic teardown, and the swap tears down both nodes in step 6 — so the failed
> node and the spare are each cleared, including a spare that was previously adopted in the
> spares rack. `force_reclaim_host` (`src/fleet_manager/core/domain/slurm_adoption.py`) also
> deletes it during the drain, which is what covers a quarantine take-back that never tears
> down Ironic.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.