The Systems These Workflows Touch
Most workflows act on a machine by talking to one of the systems below. You do not need to be an expert in any of them to use this page — this table is here so the rest of the page reads clearly.| System | In plain terms |
|---|---|
| NetBox | The inventory record — what hardware exists and where it is in its lifecycle. |
| Ironic | Bare-metal control — powers a machine on and off and installs its base operating system. |
| SLURM | The scheduler that actually hands a node to customer jobs. |
| Placement / Nova | OpenStack allocation — which customer project owns a machine. |
| BMC / Redfish | The small management chip on every machine, reachable even when the machine’s OS is down. Redfish is the modern way of talking to it. |
| VRF | Network isolation — whose private network the machine’s ports sit in. |
| NVLink | The high-speed fabric connecting the GPUs inside a multi-GPU tray. |
How A Workflow Runs
Every workflow follows the same shape: you give it some machines and options, it does its work against the systems above, and you get a machine in a new state plus a job result. This is true whether the workflow reboots a chip or replaces a whole tray. The per-workflow tables below fill in the three boxes for each one: When you run it and its options are the input, What it does is the process, and Leaves the machine is the output.Anatomy Of A Workflow Run
Every per-workflow entry below answers four questions the same way. Read this once and the terse tables read clearly. Inputs — two kinds. You (or CHAD) supply only two things at job creation: which machine(s) and a small set of options (listed per workflow under Inputs). Everything else a workflow needs — the organization, site, device name, Ironic instance, and the site’s own settings (for example whether a fabric flip is real or simulated) — Fleet Manager fills in for you from its records. So when an entry says Inputs: none, it means you just pick the machine and run it. How you know it passed. Every workflow ends as a Job with a terminal status. Success iscompleted (what that means for the machine is under Leaves the machine per workflow). A run that
did not succeed ends in one of the other terminal statuses — failed, partially_failed,
cancelled, terminated, superseded, or timed_out — see the
status reference for the full list. Note the rule at the top of this page: a
completed job does not by itself mean the machine is customer-ready — check the machine’s status.
What happens when it fails. Fleet Manager workflows stop loudly: on a problem the Job ends
failed with a plain reason (a failure code plus a message), and — unless a workflow’s
entry says otherwise — the workflow does not silently undo its earlier steps or retry forever.
The fix is almost always: read the failure reason, clear the cause, and run the workflow again (most
are safe to re-run). Where a workflow is not safe to blindly re-run, or leaves the machine in a
partial state, its If it fails row says so explicitly.
A note for SOP authors. The Inputs rows use the real option names a Job takes (for example
target_slot_id, release_to_customer_pool) and the If it fails rows name the real failure codes
(for example NVLINK_PARTITION_FAILED) so an SOP can key on them directly — each is explained in
plain words alongside.
Which Workflow Do I Need?
Start here if you know the situation but not the workflow name.| Situation | Workflow |
|---|---|
| A new machine has been racked and needs to become usable | Enrollment |
| Pull a machine out of the customer pool (planned work, or it’s misbehaving) | Quarantine |
| Put a machine back into the customer pool | Release |
| Take a machine out of the scheduler temporarily, jobs finishing first | SLURM Drain |
| Put a drained machine back into the scheduler | SLURM Resume |
| The whole machine is wedged and needs a hard power cycle | Power Cycle |
| Take a machine out of service and leave it switched off | Power Off |
| The management chip (BMC) is unresponsive but the workload must keep running | Cold-Reset BMC |
| Do hands-on work and stop Ironic touching the machine | Set Maintenance Mode |
| Check whether a machine’s firmware matches what’s expected | Firmware Verification |
| Just see what firmware is on a rack, no pass/fail | Firmware Inventory |
| Check a machine can be reached | Connectivity Verification |
| Run burn-in / hardware tests on one or more machines | Burn-In (single or multi-node) |
| Pull a machine aside for a repair, then return it | Frontend VRF, Back-End VRF or NVLink isolate / return pair |
| Replace failed hardware with a pre-staged spare | Node Swap |
| Produce a report or CSV for a set of machines | Generate Report |
Workflow Index
Every workflow at a glance. “Read-only” means it inspects and reports but changes nothing on the machine.| Workflow | What it does | Paired with | Changes the machine? |
|---|---|---|---|
| Enrollment | Brings a racked machine into management, ready to deploy | Unenrollment | Yes |
| Unenrollment | Removes a machine from management so it can be re-enrolled | Enrollment | Yes |
| Quarantine | Pulls a machine out of the customer pool | Release | Yes |
| Release | Returns a machine to the customer pool | Quarantine | Yes |
| SLURM Drain | Stops new jobs landing on a machine, lets running ones finish | SLURM Resume | Yes |
| SLURM Resume | Lets a drained machine schedule jobs again | SLURM Drain | Yes |
| Power Cycle | Powers the whole machine off and on | — | Yes |
| Power Off | Drains a machine, then switches it off and leaves it off | — | Yes |
| Cold-Reset BMC | Reboots the management chip only; the machine keeps running | — | Yes (BMC only) |
| Set Maintenance Mode | Tells Ironic to stop acting on a machine | Disable Maintenance Mode | Yes |
| Disable Maintenance Mode | Tells Ironic it may act on a machine again | Set Maintenance Mode | Yes |
| Ironic Node Clean | Recovers a machine whose earlier clean-up failed | — | Yes |
| Connectivity Verification | Checks a machine can be reached | — | Read-only |
| Firmware Inventory | Records the firmware across a rack | — | Read-only |
| Firmware Verification | Checks a tray’s firmware against expected versions | — | Read-only |
| Generate Report | Produces reports / CSVs for a set of machines | — | Read-only |
| Single-Node Burn-In | Runs a chosen sequence of hardware tests on one machine | — | Runs tests |
| Multi-Node Burn-In (Collective) | Runs collective tests across a group of machines | — | Runs tests |
| Multi-Node Burn-In (Pairwise) | Runs one network test across every pair in a group | — | Runs tests |
| Prepare Hardware-Test Images | Installs the images the burn-in tests need onto a machine | — | Yes |
| Image Node | Installs an image on a machine, no customer handoff | Unimage Node | Yes |
| Unimage Node | Removes the image and returns the machine to available | Image Node | Yes |
| Frontend VRF Isolate | Moves a machine’s network ports from the customer’s network to Nscale’s maintenance network | Frontend VRF Return | Yes |
| Frontend VRF Return | Moves the ports back into the customer’s network (return to service) | Frontend VRF Isolate | Yes |
| NVLink Isolate | Moves a tray’s GPUs into a repair partition | NVLink Return | Yes |
| NVLink Return | Moves the GPUs back into the customer partition | NVLink Isolate | Yes |
| Back-End VRF Isolate | Moves a host’s RoCE back-end ports into the maintenance VRF (simulated unless the site allows real back-end writes) | Back-End VRF Return | Yes, where real writes are allowed |
| Back-End VRF Return | Moves the back-end ports back into the customer VRF (simulated unless the site allows real back-end writes) | Back-End VRF Isolate | Yes, where real writes are allowed |
| Node Swap | Prepares a failed machine and a spare for a physical swap | — | Yes (destructive) |
Enrolment Lifecycle
Enrollment
| When you run it | A newly-racked machine needs to become a managed, deployable machine — and again as the second half of a hardware swap. |
| Inputs | The machine, and four optional switches: run_inspection (on by default — discover the hardware), force_inspection (re-discover even if we already have good data), force_bmc_restart (reboot the management chip even if its stored password still works), and node_uuid (pin a specific Ironic identity so an existing slot’s reservations survive a swap; omit to mint a fresh one). Run with no options for a normal enrolment. |
| What it does | Takes the machine end-to-end: checks its inventory status, sets up its network and management chip, marks it live in inventory, then hands it to Ironic to discover its hardware and clean it. See the deep walkthrough on Enrollment Workflow. |
| Leaves the machine | Live in inventory and ready to be deployed (active in NetBox, available in Ironic). |
| If it fails | Job Failed, stops loudly, no auto-undo — the machine is left part-enrolled (its Ironic node may sit in manageable or clean failed). It is designed to be re-run: fix the cause and run it again. Common reasons: IRONIC_NODE_LOCKED (Ironic was busy with the node — retry), IRONIC_CLEANFAIL (cleaning timed out or stalled), ENROLLMENT_FINALIZATION_IDENTITY_UNAVAILABLE (inspection finished without resolving the machine’s identity). |
| Watch out for | Configuring the management chip may rotate its password and reboot it. Re-enrolling into an existing slot reuses that slot’s identity; a machine already in use cannot be taken over this way. |
Unenrollment
| When you run it | A machine’s records were wrong and all of its enrolment state needs tearing down so it can be re-enrolled from scratch. |
| Inputs | Just the machine — no options. |
| What it does | The exact reverse of enrolment — removes the machine from the scheduler, from Ironic, from allocation, resets the management chip to factory defaults, deletes its stored credentials, and moves it back to a staged state in inventory. When the machine is active in NetBox, it first declares the machine’s removal to topology-service (a Sync Intent named unenrollment-<machine>-<job id>) before changing anything, checks again just before moving it to staged (declaring then if it became active during the teardown, or if the first declaration found no Region), and triggers the Region’s topology sync once inventory says staged, so a Region that only discovers active machines publishes the removal instead of holding its sync. |
| Leaves the machine | Back to staged, with all management state removed, ready to be enrolled again. |
| If it fails | Job Failed, stops loudly, no auto-undo — the teardown stops at whichever step failed (so it is part torn-down). Safe to re-run end-to-end once the cause is fixed. It refuses up front if a live customer instance is still bound to the machine. TOPOLOGY_SYNC_INTENT_FAILED means the removal could not be declared to topology-service; nothing has been torn down yet (unless the machine became active during the teardown, in which case Ironic is torn down but inventory is unchanged), so re-run once the hub’s topology sync answers. |
| Watch out for | It refuses if a customer is still running on the machine. There is no automatic undo — if a step fails it stops loudly; fix the cause and run it again (it is safe to re-run). |
Customer Pool Custody
The descriptions below are the fully-managed (SLURM) case. Machines on other custody modes behave differently: a Metal-delegated machine is reclaimed through Metal with no scheduler step, and a self-managed (Google) machine runs no activities here at all because its own connector owns that work. If you are unsure which mode a machine is on, check before relying on the SLURM behaviour.Quarantine
| When you run it | You need to pull a machine out of the customer-schedulable pool — for planned work, or because it is misbehaving. |
| Inputs | The machine, plus at most one optional mode (they are mutually exclusive — combining them is rejected when the job is created): cordon_only, force_reclaim, or force_rebuild (all default off — see Watch out for). With no mode set it does a plain quarantine. The machine’s custody mode is filled in by Fleet Manager, not you. |
| What it does | Takes the machine out of the customer pool and moves it into the internal (SLURM) cluster so you can work on it. |
| Leaves the machine | Out of the customer pool and marked quarantined. |
| If it fails | Job Failed, stops loudly. The most common reason is QUARANTINE_TRAIT_REMOVAL_FAILED — a mode was used on a site that does not support it (cordon_only/force_reclaim need a TRAIT-flavour site; force_rebuild needs a fully-managed machine). A mid-run failure can leave the machine part-way (e.g. removed from the customer pool but not yet in SLURM); that is recoverable by running Release. |
| Watch out for | Three optional modes exist and are easy to confuse: cordon_only takes it out of rotation without disturbing a running customer (reversible), force_reclaim deletes the customer’s running instance first (destructive, opt-in), and force_rebuild recovers a machine that came up but never joined the scheduler. Only one applies at a time. cordon_only and force_reclaim only work on TRAIT-flavour sites — on a legacy-SLURM site they are rejected, leaving plain quarantine and force_rebuild. |
Release
| When you run it | Work is finished and the machine should go back to being customer-schedulable. |
| Inputs | The machine, and one optional switch: release_to_customer_pool (on by default). Set it to false to release from the internal cluster but deliberately keep the machine out of the customer pool for ongoing operator work. Custody mode is filled in by Fleet Manager. |
| What it does | The reverse of Quarantine — returns the machine to the customer pool and releases it from the internal cluster. For a provider-owned (Metal-delegated) machine this is the step that hands it back to the provider; for a self-managed (Google) machine it does nothing here — the customer’s own connector owns that. |
| Leaves the machine | Back in the customer pool and schedulable again. |
| If it fails | Job Failed, stops loudly, no auto-undo. A mid-run failure can leave the machine released from SLURM but not yet returned to the pool; re-run once the cause is fixed. A customer-pool release is refused while the machine may still be isolated on a fabric: RELEASE_PARTITION_LATCH_REFUSED (NVLink repair partition; run NVLink Return) or RELEASE_BACKEND_VRF_LATCH_REFUSED (back-end ports in the maintenance VRF; see the release wall in docs/runbooks/backend-vrf-flip.md). |
| Watch out for | You can release it from the internal cluster but deliberately keep it out of the customer pool (for ongoing operator work) by setting release_to_customer_pool=false. |
SLURM Node Drain / Resume
SLURM Drain
| When you run it | Before hands-on maintenance, when you want running jobs to finish but no new jobs to land. |
| Inputs | The machine, and one optional reason string recorded in the scheduler (defaults to a generic maintenance note). |
| What it does | Marks the machine “drained” in the scheduler so it stops accepting new work. |
| Leaves the machine | Draining in the scheduler; running jobs continue to completion. The job result tells you which of three things happened: it drained the machine, it was already drained, or the machine isn’t in the scheduler at all — all count as success. |
| If it fails | Job Failed only if the scheduler could not be reached or refused the drain. Safe to re-run (draining twice is harmless). |
| Watch out for | Safe to run repeatedly — it reports whether it drained the machine, it was already drained, or it isn’t in the scheduler at all. Pair it with SLURM Resume afterwards. |
SLURM Resume
| When you run it | Maintenance is done and the machine should schedule jobs again. |
| Inputs | Just the machine — no options. |
| What it does | Clears the “drained” mark so the scheduler can place jobs on it. |
| Leaves the machine | Back to scheduling jobs. |
| If it fails | Job Failed if the scheduler refuses the resume — that is the whole point of the job, so it fails loudly rather than pretending. Safe to re-run. |
| Watch out for | The job fails if the scheduler refuses the resume — resuming the machine is the whole point of the job. |
In-Service Hardware Operations
Power Cycle
| When you run it | The whole machine is wedged and needs a hard off-and-on. |
| Inputs | The machine, plus optional switches: force_shutoff (on by default — power off immediately, killing running jobs; set false to wait for jobs to finish first), drain_timeout_seconds (how long to wait for that, default 5 hours), skip_slurm_drain (skip the courtesy drain), and how to reset — power_cycle_method (ironic default, or redfish) and reset_kind (standard, or aux for a full A/C cycle, which requires redfish). |
| What it does | Powers the machine off then on. It first drains the machine in the scheduler as a courtesy, then resets it. |
| Leaves the machine | Powered back on. If it drained the machine and the reset completed cleanly, it resumes it in the scheduler; otherwise it leaves it drained for you to check. |
| If it fails | Job Failed, stops loudly. Typical reasons: IRONIC_NODE_POWER_STATE_ERROR (timed out waiting for the machine to reach off/on), IRONIC_DRIVER_MISMATCH (machine isn’t on the expected power driver), or an invalid option combo (aux reset with the ironic method). If it had drained the machine but never issued the reset, it resumes it for you; once the reset was issued and then failed, the machine is left drained for you to check. |
| Watch out for | By default it kills running jobs. Setting force_shutoff=false tries to let jobs finish first, but only on a best-effort basis — if the scheduler drain or the wait fails, it logs the problem and resets anyway. Do not rely on it to protect a running workload; confirm the machine is drained independently first. This resets the whole machine — for the management chip only, use Cold-Reset BMC instead. |
Power Off
| When you run it | You’re taking a machine out of service and it should stay switched off — planned decommission, a move, or hardware work that needs the power down. |
| Inputs | The machine, plus two optional overrides and a timeout: force_shutoff (off by default — set true to switch off as soon as the drain is issued, killing running jobs), skip_slurm_drain (skip the scheduler drain entirely, for sites with no scheduler), and drain_timeout_seconds (how long to wait for running jobs to finish, default 5 hours). |
| What it does | Drains the machine in the scheduler, waits for its running jobs to finish, sets Ironic’s maintenance flag, then switches the machine off. |
| Leaves the machine | Drained, in maintenance, and powered off — deliberately. Its operational status becomes quarantine, decided by the maintenance flag alone, so that is the end state whether or not the machine was in the scheduler. Its conditions read MaintenanceClear false, CustomerAllocated false, and the power axis off; a machine that was in the scheduler also reads SlurmSchedulable false while SlurmMember stays true — it remains a cluster member, just unschedulable. InstanceAllocated may still be true: a test-cluster machine keeps its maintenance-pool instance, and an attached instance is not the same thing as a customer holding the machine. Note this is not the operational status drained, which means something else entirely: a machine temporarily out of partner custody. There is no power-on and no resume; bring it back with Disable Maintenance Mode and SLURM Resume when you’re ready. |
| If it fails | Job Failed, stops loudly, and nothing is undone — a machine that was drained before the failure stays drained. Typical reasons: IRONIC_NODE_CUSTOMER_INSTANCE_ATTACHED (a customer holds the machine), SLURM_NODE_STATE_FAILED (the machine isn’t in the scheduler, or its drain was cancelled mid-wait, so quiescence can’t be proven), IRONIC_NODE_POWER_STATE_ERROR (timed out waiting for it to report off), IRONIC_DRIVER_MISMATCH (not on the expected power driver). |
| Watch out for | Unlike Power Cycle, the default here protects running work: it switches off only once the scheduler confirms the machine is drained and idle, and refuses to switch off at all if it can’t prove that. force_shutoff=true and skip_slurm_drain=true are the two deliberate opt-outs of that guarantee — both kill running jobs. A customer-held machine is refused outright, before anything is changed and again just before switch-off, with no override. There is no compensation, so a partial failure leaves the machine drained and out of the scheduler until you put it back. |
Cold-Reset BMC
| When you run it | The management chip (BMC) is unresponsive but the machine’s workload must keep running. |
| Inputs | The machine, and one optional transport (redfish by default, or ipmi as a fallback when the chip is too wedged to answer Redfish). |
| What it does | Reboots the management chip only. The machine stays powered on and any running work is unaffected. |
| Leaves the machine | Running exactly as before, with a freshly rebooted management chip. |
| If it fails | Job Failed, stops loudly; the machine’s workload is untouched throughout. Because the reset itself is a single attempt (not safely repeatable automatically), re-run it by hand if needed — consider switching transport to ipmi if Redfish didn’t answer. |
| Watch out for | This is a single attempt (the reset can’t be safely retried automatically). It resets only the chip — for the whole machine, use Power Cycle. |
Set Maintenance Mode
| When you run it | You want to do hands-on work and stop Ironic acting on the machine. |
| Inputs | The machine, plus optional reason (recorded on the maintenance flag) and power_on (off by default — set true to power the machine on after enabling maintenance). |
| What it does | Sets Ironic’s maintenance flag so automation leaves the machine alone; can also power the machine on afterwards. |
| Leaves the machine | In maintenance — isolated from Ironic automation. |
| If it fails | Job Failed, stops loudly; typically a timeout toggling the flag. Safe to re-run. |
| Watch out for | Remember to clear it with Disable Maintenance Mode when you’re done, or automation stays paused. |
Disable Maintenance Mode
| When you run it | Hands-on work is finished and Ironic should manage the machine again. |
| Inputs | Just the machine — no options. |
| What it does | Clears Ironic’s maintenance flag. |
| Leaves the machine | Back under normal Ironic automation. |
| If it fails | Job Failed, stops loudly; typically a timeout toggling the flag. Safe to re-run. |
| Watch out for | Nothing notable — it is the plain counterpart to Set Maintenance Mode. |
Ironic Node Clean
| When you run it | A machine’s earlier clean-up failed and it needs to be returned to a clean, available state. |
| Inputs | Just the machine — no options. |
| What it does | Re-runs Ironic’s clean-up, making sure the machine boots from the right network port along the way. |
| Leaves the machine | Clean and available (available in Ironic). If the machine isn’t in a state that can be cleaned, the job still Succeeds but reports that it skipped — it changes nothing. |
| If it fails | Job Failed, stops loudly, if the re-clean itself errors or times out. Safe to re-run. (An ineligible machine is a skip, not a failure — see above.) |
| Watch out for | It only proceeds if the machine is in a failed-clean or manageable state; from anywhere else it safely does nothing. |
Read-Only Checks & Reports
These inspect and report. They change nothing on the machine.Connectivity Verification
| When you run it | You want to confirm a machine can be reached. |
| Inputs | Just the machine — no options. |
| What it does | Runs the connectivity checks (today, whether the management chip answers) and returns the result. |
| Leaves the machine | Unchanged. Passes only when every check passes. |
| If it fails | Job Failed, with the reason naming which check failed and why — e.g. login rejected (wrong/rotated credentials), access denied (credentials lack access), no answer/timed out (chip unreachable), or not answering the expected protocol. Machine is unchanged. |
| Watch out for | The job only passes if every check passes; a single failure marks the job failed with per-check detail. |
Firmware Inventory
| When you run it | You want to see what firmware is installed across a rack. |
| Inputs | The machines, which must all be in one rack. No options today — the gather takes no parameters at job creation; it scans the rack and logs the firmware it finds. |
| What it does | Scans one rack and records the firmware it finds on each machine. |
| Leaves the machine | Unchanged. Produces a firmware listing; a machine it couldn’t read is recorded as such, and the job still Succeeds. |
| If it fails | Job Failed only for a scope mistake — FIRMWARE_INVENTORY_INVALID_SCOPE (no machines selected, a machine with no rack, or machines spanning more than one rack). It never fails on firmware differences — it only gathers. |
| Watch out for | It gathers only — it never compares against expected versions and never fails on a mismatch. Scope must be a single rack. For pass/fail, use Firmware Verification. |
Firmware Verification
| When you run it | You need to confirm a tray’s firmware matches the expected versions. |
| Inputs | Exactly one compute tray, plus the versions to check against: desired_target_level (the approved firmware level label) and components (the list of expected parts, each with its exact name and expected version). Both are required. |
| What it does | Collects the tray’s firmware, compares it against the versions you supply, and saves an evidence report. |
| Leaves the machine | Unchanged. Passes only when the tray is fully compliant; saves an evidence report either way. |
| If it fails | Job Failed with the reason: FIRMWARE_VERIFICATION_NON_COMPLIANT (versions didn’t match), FIRMWARE_INVENTORY_COLLECTION_FAILED (couldn’t read the firmware), or FIRMWARE_VERIFICATION_EVIDENCE_UNAVAILABLE (result was indeterminate). Machine is unchanged; the evidence report is still saved for the SOP to attach. |
| Watch out for | The job passes only if the tray is fully compliant; anything else fails the job. Scope is exactly one compute tray. |
Generate Report
| When you run it | You need a report or CSVs for a set of machines. |
| Inputs | The machines to cover, plus what to report on: filters (required — the date range, test slugs, and pass/fail filter selecting which historical runs to include), and optional sections (which parts of the report), test_selection_scope (all tests or a custom set), generate_logs (also bundle per-run logs), and org_slug/site_slug. |
| What it does | Produces report artifacts and a downloadable bundle for the machines you pick. |
| Leaves the machine | Unchanged — the machines are named only so the report knows what to cover. Produces a report plus CSVs and a downloadable bundle. |
| If it fails | Job Failed with a plain reason: no matching data for the filters, the site wasn’t found, an invalid section, or a bad bundle input. Machines are untouched; just fix the inputs and re-run. |
| Watch out for | Nothing notable; it holds no locks and runs alongside other work. |
Burn-In / Hardware Testing
Single-Node Burn-In
| When you run it | You want to run a chosen sequence of hardware tests on one machine. |
| Inputs | One machine, plus tests (required — the ordered list of tests to run, at least one), and optionally partition (which scheduler partition) and max_wait_time_seconds (how long to wait per run; unlimited by default). |
| What it does | Runs the tests you list, in order, on one machine and saves the combined results. |
| Leaves the machine | Unchanged in custody — this is a measurement run. Results are judged later. See Burn-in Testing. |
| If it fails | Job Failed with a plain reason — commonly BURN_IN_TEST_PREPARATION_FAILED (a test’s parameters were rejected, or a topology-aware test lacked the machine’s resource-class) or a test-execution failure. Custody is unchanged; fix and re-run. |
| Watch out for | You control exactly which tests run and in what order. |
Multi-Node Burn-In (Collective)
| When you run it | You want to run collective tests across a group of machines together. |
| Inputs | A group of two or more machines, plus tests (required — the ordered list of collective tests, no duplicates) and optionally partition and max_wait_time_seconds. |
| What it does | Runs the list of collective tests you pick across a group of two or more machines. |
| Leaves the machine | Unchanged in custody — a measurement run. The whole group is marked completed or failed together. |
| If it fails | Job Failed and the group is marked failed, with a plain reason — e.g. BURN_IN_INSUFFICIENT_DEVICES (fewer than two machines), BURN_IN_UNKNOWN_TEST_SLUG (a test name it doesn’t recognise), BURN_IN_MIXED_RESOURCE_CLASS (the group mixes machine types for a topology-aware test), or a preparation/execution failure. Custody unchanged. |
| Watch out for | All tests run against the same group; the group is marked completed or failed accordingly. |
Multi-Node Burn-In (Pairwise)
| When you run it | You want to test the network between every pair of machines in a group. |
| Inputs | A group of two or more machines, plus test_slug (required — the single test to run; one test per job) and optionally test_params, partition, and max_wait_time_seconds. |
| What it does | Runs one pairwise network test across all pairs in the group. |
| Leaves the machine | Unchanged in custody — a measurement run. All machines are marked completed together. |
| If it fails | Job Failed and the machines are marked failed, with a plain reason — e.g. BURN_IN_INSUFFICIENT_DEVICES (fewer than two machines), BURN_IN_UNKNOWN_TEST_SLUG, BURN_IN_MISSING_PAIR_METRICS (the test ran but produced no per-pair results), or a preparation/execution failure. Custody unchanged. |
| Watch out for | One test per job — to run several pairwise tests, start the workflow once per test. |
Prepare Hardware-Test Images
| When you run it | A machine needs the images the burn-in tests rely on installed or refreshed. |
| Inputs | One machine, plus one optional mode: ensure (default — download only images that are missing) or refresh (re-download and atomically replace every image). The image set itself is site configuration, not something you pass. |
| What it does | Validates, downloads, and installs the site’s hardware-test images onto one machine. |
| Leaves the machine | With the current test images in place. |
| If it fails | Job Failed with a plain reason (commonly HARDWARE_TEST_IMAGE_PREPARATION_FAILED — a download, verification, or install step failed, or the wait timed out). Partial-install caveat: each image is only swapped in after it verifies, so images completed earlier in the same run may already have been replaced. On failure it makes a best-effort attempt to cancel its scheduler job; if that comes back unconfirmed (or you see BURN_IN_DUPLICATE_SLURM_SUBMISSION), the scheduler job may still be running — check the recorded cleanup state or cancel it by hand before re-running. |
| Watch out for | If an install fails partway, images installed earlier in the same run may already have been replaced. On failure it tries to cancel its scheduler job, but only on a best-effort basis — if that cleanup comes back unconfirmed the job may still be running, so check the recorded cleanup state or cancel it manually before rerunning. |
Imaging (No Customer Handoff)
Image Node
| When you run it | You need a machine imaged for internal use, without registering it with the scheduler or handing it to a customer. |
| Inputs | Just the machine — no options. It must be a GPU-compute or CPU-baremetal machine. |
| What it does | Installs an image on the machine and waits until it is up. |
| Leaves the machine | Imaged and running the image (active). |
| If it fails | Job Failed with reason IMAGE_NODE_FAILED — the machine wasn’t an eligible type, or imaging didn’t come up in time. Re-run once the cause is fixed. |
| Watch out for | Only applies to GPU-compute or CPU-baremetal machines; anything else is refused. |
Unimage Node
| When you run it | You need to remove an image installed by Image Node. |
| Inputs | Just the machine — no options. Same machine-type limits as Image Node. |
| What it does | Removes the image server and returns the machine to an available state. |
| Leaves the machine | Back to available. |
| If it fails | Job Failed with reason UNIMAGE_NODE_FAILED — the machine wasn’t an eligible type, or it didn’t return to available in time. Re-run once the cause is fixed. |
| Watch out for | The counterpart to Image Node; same machine-type limits apply. |
Network And GPU Fabric Moves (Paired)
These move a machine’s front-end network (VRF) and its GPU fabric (NVLink) between the customer and Nscale. They come in pairs, one workflow per direction, and they do not undo themselves — always run the matching opposite workflow. Both directions re-check that the machine is still eligible before acting, and both stop loudly on any problem rather than silently reverting. Every fabric uses the same two verbs with the same meaning: isolate moves a machine from customer service into Nscale maintenance, and return moves it from maintenance back to customer service (“return to service”). Don’t confuse “return” with the business phrase “node return”, when a customer gives a node back to Nscale. That is when you run isolate.| Direction | Front-end network (VRF) | RoCE back-end network (VRF) | GPU fabric (NVLink) |
|---|---|---|---|
| Take the machine back from the customer (e.g. for repair) | Frontend VRF Isolate | Back-End VRF Isolate | NVLink Isolate |
| Return the machine to customer service | Frontend VRF Return | Back-End VRF Return | NVLink Return |
Until FLEET-2825 the front-end pair was named the other way round: Switch VRF Return is now
Frontend VRF Isolate (customer → maintenance), and Switch VRF Handover is now Frontend VRF
Return (maintenance → customer). A runbook or SOP that still says “Switch VRF Return” means
isolate. Don’t just swap the prefix.
These workflows do not automatically reverse. To put a machine back, run the opposite workflow from
the table above. On some sites they may run in a safe “simulated” mode until the site is switched on
for real changes — check the runbook for switch VRF, back-end VRF or NVLink if you are unsure whether a real change happened.
Frontend VRF Isolate
| When you run it | Taking a machine’s front-end network back from a customer — e.g. on node return, or before repair. |
| Inputs | The machine. Optional target_vrf overrides the site’s default maintenance VRF (switch_return_vrf). Real-or-simulated is set by the site (switch_vrf_flip_allowed), same as Frontend VRF Return. |
| What it does | Moves the front-end ports out of the customer’s network into Nscale’s maintenance network. |
| Leaves the machine | With its ports on Nscale’s network; sets the marker FrontendPortsInCustomerVrf = False (reason FrontendVrfIsolateVerified). A simulated run writes no marker and is suffixed “(SIMULATED …)” — important here, because a simulated isolate must not falsely clear the marker. |
| If it fails | Job Failed, stops loudly, no auto-reverse. Reasons: SWITCH_VRF_FLIP_FAILED (the switch change failed, or no target VRF could be resolved; the message lists what happened on each switch), SWITCH_VRF_PREFLIGHT_REFUSED (the machine wasn’t eligible, its front end isn’t exactly one ToR pair, or a switch had a change still being applied or unsaved configuration — nothing was written), SWITCH_WRITE_UNCONFIRMED (a switch change was sent but still hadn’t finished 20 minutes after the write step ended, so the switch may yet change — read the switch before rerunning), SWITCH_LOCK_TIMEOUT (other flips on the same ToR pair kept it busy for 4 hours; nothing was written; rerun), SWITCH_LOCK_LOST (the flip’s turn on the pair ran out before it wrote anything; rerun), or SWITCH_VRF_LATCH_WRITE_FAILED (the ports moved but the marker didn’t record — re-run the same job to write it). |
| Watch out for | Can run whether the customer still holds the machine or has just released it. Real flips of machines on the same ToR pair queue and run one at a time, about a minute each, so a whole rack takes a while; a queued job shows as running. A switch change whose outcome is unclear keeps the pair busy while the flip checks it for up to 20 minutes after the write step ends (the write step itself takes at most 10 minutes, so about half an hour in all), and a switch that failed after the other one changed is tried once more a minute later. |
Frontend VRF Return
| When you run it | Returning a machine to customer service — its front-end network moves into the customer’s network. |
| Inputs | The machine. Optional target_vrf overrides the site’s default customer VRF (switch_handover_vrf; leave empty to use the site default). Whether the flip is real or simulated is set by the site, not you (switch_vrf_flip_allowed) — on sites where it’s off, every run is simulated. (test_frontend_interfaces is test-only; omit in production.) |
| What it does | Moves the machine’s front-end switch ports from Nscale’s maintenance network into the customer’s private network (VRF). |
| Leaves the machine | With its ports in the customer’s network. Records the durable marker FrontendPortsInCustomerVrf = True (reason FrontendVrfReturnVerified). A simulated run configures no switch and writes no marker — its result is suffixed “(SIMULATED — no switch was configured)”, so trust the marker, not just a green job. |
| If it fails | Job Failed, stops loudly, and nothing flips back automatically — to reverse it you must run Frontend VRF Isolate. Same reasons as Frontend VRF Isolate: SWITCH_VRF_FLIP_FAILED, SWITCH_VRF_PREFLIGHT_REFUSED, SWITCH_WRITE_UNCONFIRMED, SWITCH_LOCK_TIMEOUT, SWITCH_LOCK_LOST, SWITCH_VRF_LATCH_WRITE_FAILED. |
| Watch out for | The order relative to the customer handover matters and is currently yours to get right — see the runbook. Real flips on one ToR pair run one at a time, as for Frontend VRF Isolate. Detail lives in Device State Machines and the switch VRF runbook. |
NVLink Isolate
| When you run it | Nscale has taken a compute tray back for a repair and its GPUs must be separated from the customer. |
| Inputs | The tray, and target_slot_id — required, no default: the exact NMX-M slot of the tray to move. Naming the wrong slot would eject a live customer’s GPUs, so it must be given explicitly. Real-or-simulated is set by the site (nvlink_partition_allowed). |
| What it does | Moves the tray’s four GPUs out of the customer’s GPU partition into a dedicated out-for-repair partition. |
| Leaves the machine | With its GPUs in the repair partition. Sets the marker NvLinkGpusInRepairPartition = True. A simulated run configures no fabric and writes no marker — suffixed “(SIMULATED — no fabric was configured)”. |
| If it fails | Job Failed, stops loudly, no auto-reverse. Reasons: NVLINK_PARTITION_INVALID_REQUEST (no target_slot_id given — nothing was attempted, just resupply it), NVLINK_PARTITION_FAILED (the fabric move failed — if the message says NMX-M rejected every create attempt, the tray is in no partition: re-run), NVLINK_PARTITION_CONTRACT_FAILED (the control plane rejected it — escalate), NVLINK_PARTITION_PREFLIGHT_REFUSED (tray wasn’t eligible — re-run once it is), or NVLINK_LATCH_WRITE_FAILED (GPUs moved but the marker didn’t record — re-run to write it). A create the control plane rejects with NMX_ST_BADPARAM (a known NetQ 5.1 issue) is retried automatically, up to three attempts, before the job fails. |
| Watch out for | Never run on a tray a customer still holds. You must name the exact tray (target_slot_id) — there is no default, and the wrong target would eject a live customer’s GPUs. A partition change alone does not fully re-wire the fabric; that happens on the tray’s repair reboot. See the NVLink runbook. |
NVLink Return
| When you run it | A repaired tray is about to rejoin the customer pool. |
| Inputs | The tray, and target_slot_id — required, no default (same as Isolate). Real-or-simulated is set by the site (nvlink_partition_allowed). |
| What it does | Moves the tray’s four GPUs back from the repair partition into the customer partition. |
| Leaves the machine | With its GPUs back in the customer partition; sets the marker NvLinkGpusInRepairPartition = False, which is what lets Release hand the machine back. A simulated run writes no marker (suffixed “(SIMULATED …)”). |
| If it fails | Job Failed, stops loudly, no auto-reverse. Same reasons as Isolate: NVLINK_PARTITION_INVALID_REQUEST, NVLINK_PARTITION_FAILED, NVLINK_PARTITION_CONTRACT_FAILED, NVLINK_PARTITION_PREFLIGHT_REFUSED, NVLINK_LATCH_WRITE_FAILED. If the marker is left unrecorded, re-run NVLink Return to record it — never clear it by hand, or Release would hand back a tray that isn’t really back. |
| Watch out for | You must name the exact tray (target_slot_id). This is the direction that may act on a machine still inside the customer window. |
Back-End VRF Isolate
| When you run it | Nscale has taken a GPU host back for a repair (after Quarantine) and its RoCE back-end ports must leave the customer’s back-end network. |
| Inputs | The host. Nothing else: the VRF pair (be-meta / be-nscale at GSO1), the port naming rules and how many leaves it works on at once are site settings. |
| What it does | Finds the host’s back-end ports and their leaves in NetBox, checks every leaf, then moves every port from the customer VRF to the maintenance VRF and checks each leaf again by reading it back. A leaf whose ports are already in the maintenance VRF is left alone. |
| Leaves the machine | At a site that allows real back-end writes (backendVrfFlipAllowed), every back-end port in the maintenance VRF and the BackendPortsInMaintenanceVrf marker recorded True, which blocks Release. The marker reads Unknown from the first leaf write until every leaf verifies. At every other site the run is simulated: no leaf is configured, no marker is recorded, and the result says “(SIMULATED — no leaf was configured)”. |
| If it fails | Job Failed, stops loudly, no auto-reverse. Before failing, a real run that wrote some leaves retries each unfinished leaf once, a minute later, if its earlier write has finished either way; a leaf whose write may still land is never retried. BACKEND_VRF_PREFLIGHT_REFUSED (the site has no back-end VRF pair or port naming rules or, for real writes, no leaf account or no in-flight Release check; NetBox does not describe the host’s back-end ports; or a leaf failed its check or could not be reached; nothing was written), BACKEND_VRF_APPLY_REFUSED (every leaf refused just before writing, including a leaf where another revision was still being applied, or whose fresh revision carried another writer’s changes and was deleted; nothing was written, but the marker may read Unknown because it is recorded just before the write; rerun to record a definite value), BACKEND_VRF_FLIP_FAILED (a leaf may have been written, and either not every leaf verified after that retry or the flip’s lock was lost before it finished; the marker stays Unknown; the message lists the verified, refused and failed leaves), SWITCH_WRITE_UNCONFIRMED (a leaf change was sent but still hadn’t finished 20 minutes after it was sent, so the leaf may yet change; the marker stays Unknown — read the leaf before rerunning), BACKEND_VRF_LATCH_WRITE_FAILED (every leaf verified but the marker was not recorded; rerun), SWITCH_LOCK_TIMEOUT (other back-end flips in the same scalable unit, which queue for its lock one at a time, kept it busy for 4 hours; nothing was written; rerun once they have finished) or SWITCH_LOCK_LOST (the flip’s turn on the lock ran out before any leaf was written; nothing was written; rerun). |
| Watch out for | Real runs in one scalable unit go one at a time, in the order they reach its lock: every host in an SU shares the same 16 leaves, so later flips queue behind the first, for up to 4 hours. Never run on a host a customer still holds. See the back-end VRF runbook. |
Back-End VRF Return
| When you run it | A repaired GPU host is going back to the customer and its RoCE back-end ports must rejoin the customer’s back-end network. Run it after NVLink Return and before Frontend VRF Return. Also run it before releasing a replacement tray from a Node Swap, which can inherit isolated ports. |
| Inputs | The host. Nothing else, as for Back-End VRF Isolate. |
| What it does | The reverse of Back-End VRF Isolate: moves every port from the maintenance VRF back to the customer VRF, then checks each leaf by reading it back. A leaf whose ports are already in the customer VRF is left alone. |
| Leaves the machine | At a site that allows real back-end writes, every back-end port in the customer VRF and the BackendPortsInMaintenanceVrf marker recorded False, which lets Release through. At every other site the run is simulated: no leaf is configured and the marker is not cleared, so it does not unblock a Release; the runbook says how to clear it by hand. |
| If it fails | Job Failed, stops loudly, no auto-reverse. Retries unfinished leaves once first, as Back-End VRF Isolate does. Same codes as Back-End VRF Isolate, including BACKEND_VRF_APPLY_REFUSED with a possible Unknown marker: BACKEND_VRF_PREFLIGHT_REFUSED, BACKEND_VRF_APPLY_REFUSED, BACKEND_VRF_FLIP_FAILED, SWITCH_WRITE_UNCONFIRMED, BACKEND_VRF_LATCH_WRITE_FAILED, SWITCH_LOCK_TIMEOUT or SWITCH_LOCK_LOST. |
| Watch out for | This is the direction that may act on a host still inside the customer window. See the back-end VRF runbook. |
Node Replacement
Node Swap
Node Swap is destructive and not safe to blindly re-run. It has two points of no return: a
force-reclaim that deletes the customer’s running instance, and a records rewrite that makes the
inventory describe the post-swap world. There are no automatic retries beyond the NetBox writes
and a never-enrolled spare’s BMC configuration — if a step fails it stops
loudly, so follow the recorded failure notes. A failure early on (before the records rewrite)
normally reruns the standard path;
recovery mode is only for a failure after the teardown has
begun, because it skips straight to re-pinning and assumes the earlier steps already applied.| When you run it | You are replacing a failed machine with a pre-staged spare, before the physical swap. |
| Inputs | Run it against the failed machine, and give replacement_host_name — required: the pre-staged spare that takes its slot. On a normal run, add force_reclaim only if a live customer instance must be deleted first, and failed_tray_unreachable if the failed machine’s chip is dead (verify power at the PDU — it is not powered off remotely). netbox_only is a separate day-0 mode that rewrites only the records. Recovery is not a plain mode you pick on its own: to re-run after a failure that already tore the machines down, pass recovery=true together with production_uuid (and hospital_uuid if one was used) from the failed run’s logs and force_reclaim — recovery by itself is rejected with NODE_SWAP_VALIDATION_FAILED. This workflow automatically updates NetBox, allocations, Ironic and the scheduler — the only manual step is the physical tray move. |
| What it does | Prepares both the failed machine and the spare up to the point a technician moves the hardware: configures a never-enrolled spare’s BMC as enrolment would, where the site’s enrolment configures BMCs (on a GB300 tray this creates its login, so no manual BMC step), fills a never-enrolled GB300 spare’s missing frontend MACs in NetBox from that BMC once it proves to be the spare’s own, drains the failed machine, moves records and allocations to the post-swap state, removes both from management, and parks both powered off in maintenance with their identities cross-pinned. The post-swap half is just a normal Enrollment run per machine. When a dead-tray swap’s spare has no NetBox cables, it also declares the failed machine’s removal to topology-service before its first write to Fleet Manager or the NetBox records, sets the failed machine NetBox offline, and triggers the Region’s topology sync — see a spare with no NetBox cables. When both machines are cabled, it declares the pair as swapped to topology-service straight after the NetBox rewrite and triggers the sync that publishes it — see a spare with NetBox cables. |
| Leaves the machine | Both trays powered off, in maintenance, cross-pinned, ready for the physical swap and re-enrolment. |
| If it fails | Job Failed, stops loudly, no automatic retries (bar the NetBox writes and a never-enrolled spare’s BMC configuration), no auto-reverse — this workflow is not safe to blindly re-run (see the warning above). The most common reason is NODE_SWAP_VALIDATION_FAILED (a precondition was not met — e.g. the site disallows force-reclaim, a live instance, or a dead-tray run whose failed machine still has its node, needs force_reclaim=true, or a mode combination is invalid); IRONIC_NODE_LOCKED (Ironic busy with a node — re-adding it, or still tearing down a dead tray’s deleted instance) and BMC_HTTP_SERVICE_UNAVAILABLE (a spare’s BMC busy while its host boots) mean wait and rerun. The job page records which side of the two points-of-no-return it stopped on (the force-reclaim delete, then the records rewrite). A failure before the records rewrite normally re-runs the standard path; a failure after the teardown began uses recovery mode with the identifiers from the failed run — never a fresh run. |
| Watch out for | See the modes table below. Applies to the GB300 tray and B200 server flows — see GB300 Tray Replacement and B200 Node Replacement. |
| Mode | What it does |
|---|---|
netbox_only | Rewrites only the inventory cabling/location and stops (a Day-0 records fix). |
recovery | Re-runs after a failure that already removed the machines, using identifiers from the failed run’s logs. |
failed_tray_unreachable | Declares the failed machine’s management chip dead — check power at the PDU before pulling it; it is not powered off remotely. |
force_reclaim | Confirms the customer’s instance may be deleted (opt-in per site). |
Internal & Retired Workflows
You may see these names in the API or logs. You do not run them directly.| Workflow | Why it’s here |
|---|---|
| Single Test | The internal step that runs one hardware test — used by the burn-in workflows above, not started on its own. |
| Customer Release | Folded into Release. Kept registered only so old job histories stay valid. |
| SLURM Adopt | Folded into Quarantine. Kept registered only so old job histories stay valid. |
| SLURM Release | Folded into Release. Kept registered only so old job histories stay valid. |