> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a New Burn-in Test

> How burn-in tests run end-to-end and how to add a new one.

Burn-in tests run hardware validation jobs on [SLURM](../../start-here/glossary#slurm)-managed compute and feed the results back into Fleet Manager. This page covers the high-level architecture and the steps to add a new test.

## Test categories

Three shapes of test exist; the first two are the focus today, the third is partial.

* **Single-node** — runs on exactly one compute node. Used for per-[device](../../start-here/glossary#node) validation (HPL, [NCCL](../../start-here/glossary#nccl) loopback, DCGMI, NVMe, node health). One SLURM job per device. Orchestrated by `SingleNodeTest`.
* **Multi-node collective** — runs across two or more nodes as a single coordinated job and produces aggregate metrics (multi-node NCCL all-reduce/all-gather/all-to-all, iperf matrix, distributed training, parallel storage benchmarks). One SLURM job for the whole cohort. Orchestrated by `MultiNodeTest`.
* **Multi-node pairwise** — fans out per-NIC-pair tests across a cohort and reports per-pair metrics (`PAIRWISE_NET_BW`, `PAIRWISE_NET_LAT`, [GPU](../../start-here/glossary#gpu) variants). Orchestrated by `PairwiseNetworkTestBase` (a `MultiNodeTest`). **Status: WIP.** The execution path works and reports include pairwise data, but the UI surfaces only single-node and multi-node collective results today; pairwise is reports-only.

## Architecture

Burn-in tests cleanly separate **orchestration** (in `fleet-manager`) from **test execution** (in the `hardware-tests-runner` library). Both live in this repo so they share types and version, but they sit in different Python packages so the dependency direction is enforced and the runner stays thin enough to extract if we ever need to ship it independently.

| Layer | Path | Responsibility |
| - | - | - |
| Orchestration | `src/fleet_manager/core/hardware_tests/` | `BaseTest` / `SingleNodeTest` / `MultiNodeTest`, registry, `RunContext`, sbatch rendering, artifact parsing, threshold evaluation. |
| Shared contract | `libs/hardware-tests-contract/` | `SlurmBatchConfig`, `ExecutionContext`, slug enums, params models, metrics models, artifact metadata. The only thing both sides depend on. |
| Execution | `libs/hardware-tests-runner/` | `hardware-tests` Typer CLI, `BaseRunner`, `WorkspaceManager`, concrete test executors under `tests/`. Runs on the compute node. |

The runner library never imports from `fleet_manager`. Fleet-manager never imports from `hardware_tests_runner`. They communicate through the JSON-serialised `ExecutionContext` and the artifacts written to S3.

For the repo-internal agent checklist and training-specific gotchas, see `.agents/skills/burn-in-test-creator/SKILL.md`.

### Runtime topology

```mermaid theme={null}
flowchart LR
  Op["Operator"]
  API["Fleet Manager API"]
  DB[("Fleet Manager DB")]
  S3[("Object Storage")]

  subgraph Satellite["Satellite (site-local)"]
    WA["Workflow Agent"]
    TW["Temporal BurnInWorkflow"]
    SC["SLURM restd"]
    CN["Compute Nodes"]
  end

  Op --> API
  API --> DB
  WA --> API
  WA --> TW
  TW --> SC
  TW --> API
  SC --> CN
  CN --> S3
  S3 --> TW
```

What each piece does:

* **Operator** kicks off a burn-in `Job` through the Fleet Manager API.
* **Fleet Manager API + DB** (hub) own the desired state (jobs, presets, thresholds) and the materialised results.
* **Workflow Agent** (per satellite) reconciles `Job`s into [Temporal](../../start-here/glossary#temporal) workflow executions.
* **Temporal `BurnInWorkflow`** (single-node, multi-node collective, or multi-node pairwise variant) renders an sbatch script per test, dispatches it, polls for completion, then pulls artifacts from object storage and writes results back to the API.
* **SLURM restd → compute nodes** runs the actual job and invokes the runner CLI.
* **Compute nodes** stream stdout/stderr and custom artifacts to **object storage** via presigned URLs.
* The workflow then **pulls from object storage** to evaluate metrics against thresholds and persist a `TestResult`.

Hub holds operator-facing state and the artifact store; satellites hold everything that talks to the hardware. This keeps trust and credentials site-local and lets sites be added or rotated independently.

### Life of a single test

1. The Temporal workflow asks the activity to render and submit a job for a given `(slug, params, RunContext)`.
2. The activity loads the test class via `registry.get_test_class`, validates params, builds a `SlurmBatchConfig` and an `ExecutionContext`, generates presigned S3 URLs, and renders an sbatch script from `shared/templates/sbatch.sh.j2`.
3. SLURM runs the script. The script invokes `hardware-tests run <slug> --workspace $WORKSPACE --config-json "$CONFIG_JSON"` on the compute node.
4. On the node, the runner instantiates the `BaseRunner` subclass for the slug and calls `async execute(...)`. `WorkspaceManager` captures stdout/stderr, writes `metadata.json`, packages `results/` into `custom-artifacts.tar.gz`, and uploads via the presigned URLs.
5. The workflow downloads the artifacts via `ArtifactCollector`, calls the test's `extract_metrics(...)` for typed metrics, runs `evaluate(...)` against the thresholds, and writes a `TestResult` back through the API.

## Author a new test

Use the Hello World test (`single_node/hello_world.py` and the matching runner file) as the canonical reference.

### 1. Extend the shared contract

In `libs/hardware-tests-contract/`:

* Add a slug to `slugs.py` (`SingleNodeTests` or `MultiNodeTests` enum + `Slugs.*` constant).
* Add a Pydantic params model under `params/<family>.py`, then re-export it from `params/__init__.py` so callers can import it from `hardware_tests_contract.params`. **Keep orchestration fields out** (partition, nodes, timeout, GPUs) — those come from `RunContext`. Params describe what the test does, not how SLURM should schedule it. Expose only genuine workload knobs — enough that real knobs aren't hard-coded, not so many that the surface sprawls — and **keep out any knob that would move the right threshold**. Thresholds are calibrated against one fixed configuration and are never switched on param values at runtime, so a threshold-moving param silently invalidates its gate; bake those into the recipe/config at the calibrated value instead.
* Add a `*Metrics` model to `metrics/models.py` extending `BaseMetrics` (from `metrics/base.py`), or use `PairwiseNetworkMetrics` for pairwise. Implement `to_metrics_response()` so the API/DB representation is explicit.
* Add the new slug to at least one operation's `test_slugs` list in `src/fleet_manager/core/burn_in/stale_test_operations.yaml` — the hardware operation(s) that should invalidate this test's evidence (e.g. node swap, firmware update, power cycle). There is no default mapping: a coverage-gate unit test (`tests/core/burn_in/test_stale_test_operations.py`) fails CI if a registered slug isn't listed under at least one operation.

### 2. Implement the worker-side runner

Under `libs/hardware-tests-runner/src/hardware_tests_runner/tests/`:

* Subclass `BaseRunner[MyParams]`, set `slug` and `params_class`, and implement `async execute(self, context, workspace_manager)`.
* Use `workspace_manager.run_command(...)` for shell workloads, `write_custom_artifact(...)` / `get_custom_artifact_path(...)` for outputs. `BaseRunner` takes care of log files, `metadata.json`, archiving, and uploads.
* Provide tooling via the node OS image PATH, a test container image, or explicit binary paths in params (e.g. `hpl_binary`).
* Register the runner in `hardware_tests_runner/registry.py`.

The runner only executes commands and writes artifacts. It does not parse metrics, evaluate thresholds, or know about Fleet Manager. The one call it does make is whether the test *ran*: a tool that exits non-zero because it **found a hardware fault** is a legitimate result — write the artifact and let evaluation fail the verdict — whereas an execution error (missing binary, SLURM/container problem) should fail the job loudly.

### 3. Implement the fleet-manager orchestration class

Place the file under `single_node/` or `multi_node/` (or extend `PairwiseNetworkTestBase` for pairwise) and inherit `SingleNodeTest` / `MultiNodeTest`. Required:

* `slug`, `display_name`, `params_class`, `metrics_class`, `thresholds_class`.
* `make_batch_config(self, context: RunContext) -> SlurmBatchConfig` — use `make_single_node_slurm_batch_config` / `make_multi_node_slurm_batch_config`. `RunContext` exposes `partition`, `timeout_minutes`, optional `nodelist`, and the `num_nodes` property — **always use `context.num_nodes`** for multi-node tests, not `context.nodes`. When a `nodelist` is supplied, `nodes` is forced to 1 and the real count comes from the list length; `num_nodes` resolves both cases. GPU allocation comes from params or class attributes (e.g. `gpus_per_node=0` for non-GPU tests).
* `extract_metrics(self, artifact_collector) -> BaseMetrics | PairwiseNetworkMetrics` — pure parsing, no I/O. Reach into `artifact_collector.paths[TestArtifactFile.STDOUT]` for logs, `artifact_collector.get_custom_file("pattern")` for unpacked custom artifacts, and `artifact_collector.metadata` for the standard `HardwareTestMetadata`.
* `evaluate(cls, metrics, thresholds, iteration_summary=None, resolved_params=None) -> EvaluationResult` — populate `metric_evaluations` with `TestMetricEvaluation` entries (use `metric_pass()` / `metric_fail(...)`), set a concise `reason`, and **never raise** — the activity wraps `EvaluationResult` into a typed `TestFailedEvaluationError`. Use `resolved_params` only for persisted per-run policy that genuinely changes the verdict; parse it with the test's params model and return a failed `EvaluationResult` if it is malformed.

Define the thresholds schema as a Pydantic model in **Python** (alongside the test class, or reuse `GenericThresholds` for plain numeric thresholds) and implement `catalogue_thresholds()` on it so the test catalogue can render numeric expectations. `BaseTest.__init_subclass__` enforces both — the YAML cannot satisfy these requirements because it has no methods. Then add the threshold *values* keyed by slug to `src/fleet_manager/core/hardware_tests/thresholds/thresholds.yaml` (or, for resource-class-specific tests, under a `defaults:` map keyed by resource class). At runtime `BaseTest.load_thresholds` reads the raw dict via the `thresholds.loader` module and calls `thresholds_class.model_validate(raw)`.

Then register the orchestration class in `_REGISTERED_TESTS` in `src/fleet_manager/core/hardware_tests/registry.py`.

### 4. Design the metrics for the UI

`metrics_class` is a UX surface, not a debug dump. Whatever you emit lands directly in the test results UI and the catalogue, so make every choice with the operator's attention in mind.

* **Only expose metrics operators should care about.** If a number isn't going to drive a pass/fail decision or a triage step, leave it out of the metrics model — write it to a custom artifact instead. Attention is a precious resource.
* **Order matters.** Field order in the Pydantic model is the display order. Put the most important metric first; less important context last.
* **Name metrics for humans.** Use clear, plain names (`bus_bandwidth`, `min_latency`) — not internal column names or test-runner jargon.
* **Don't repeat units in names.** That's what `MetricUnit` is for. `bus_bandwidth` with `MetricUnit.GB_S`, **not** `bus_bandwidth_gbps`. Same for latency, throughput, percentages.
* **Pick the right unit and stick with it across tests.** Use `MetricUnit.GB_S` for gigabytes/s (memory/bus) and `MetricUnit.GBPS` for gigabits/s (network). They are not interchangeable.
* **Match thresholds to the metrics you display.** If a metric isn't worth a threshold or a comparator, it probably isn't worth showing.
* **Hide non-decision fields, don't drop them.** Mark fields an operator wouldn't act on (problem size, block size, raw counts) `api_visible=False` — they still persist for evaluation and debugging but stay out of the UI.
* **Use `detail` for actionable failure context.** The per-`MetricValue` `detail` string (\~2 KB, shown on the UI) is the operator's first clue when a gate trips — name *what* and *where*, not just the number: the misconfigured NICs and their settings, the worst GPU pairs, the GPUs that failed ECC.

### 5. Verify locally

```bash theme={null}
# Discover and inspect
uv run fleetctl hardware-tests list
uv run fleetctl hardware-tests schema <stage>.<test-name>

# Render the sbatch script without submitting
uv run fleetctl hardware-tests generate-sbatch <stage>.<test-name> \
  --params '{...}' --partition testing --timeout 15 --show-script

# Submit end-to-end via the SLURM REST API and poll for results
uv run fleetctl hardware-tests dispatch <stage>.<test-name> \
  --params '{...}' --partition testing --timeout 15
```

Pass `--local` to `dispatch` to submit via the `sbatch` CLI instead of the REST API — useful for narrowing down REST submission bugs (resource parameter mapping, etc.).

For failed runs, `uv run fleetctl hardware-tests debug <workflow-id>` pulls the `DebugBundle` from the Temporal memo and produces presigned URLs to the artifacts.

Once it passes on real hardware, **attach the evidence to the PR** — the workflow id, SLURM state, and key metrics from a passing run — so reviewers can confirm it ran green on hardware, not just in unit tests.

### 6. Tests and pre-commit

Minimum coverage:

* Unit tests for `extract_metrics` against canned artifacts.
* Unit tests for `evaluate` against representative metrics + thresholds.
* A worker-side test for `execute` that exercises `WorkspaceManager` writes.
* Wiring assertions: the slug appears in `list_all_tests()` and `hardware_tests_runner.registry`.

Then:

```bash theme={null}
make pre-commit
make test
```

## Common mistakes

* Putting orchestration fields (`partition`, `nodes`, `timeout`, GPU counts) in the params model instead of `RunContext`.
* Exposing a param that changes the *correct* threshold. Thresholds don't switch on param values, so the gate silently goes meaningless when a caller flips it — bake threshold-moving knobs into the recipe/config at the calibrated value.
* Doing I/O inside `extract_metrics` — it must be pure parsing against `ArtifactCollector`.
* Scraping stdout for metrics when the runner could write a JSON artifact. Emit structured JSON and `extract_metrics` becomes a trivial `json.loads`; reserve log parsing for third-party tools whose output you don't control.
* Swallowing an error or defaulting a metric on bad input. A silent fallback hides a bigger fault and wastes time later — `extract_metrics` raises on a missing or malformed artifact instead of returning zeros that pass the gate.
* Raising inside `evaluate` instead of returning an `EvaluationResult`.
* Threading ad-hoc runtime params into `evaluate()`. It receives metrics + thresholds, optional `iteration_summary`, and the persisted execution `resolved_params`; if a value is a measured hardware fact rather than per-run policy, tag it onto the metrics payload host-side instead.
* Flipping a comparator. Latency and error-rate gate as upper bounds; bandwidth, throughput, and TFLOPs as lower bounds — a swapped comparator passes garbage green.
* Aborting the runner on a legitimate hardware-fault exit. A benchmark exiting non-zero because it found a real fault is data: capture it and let `evaluate` fail the verdict. Fail the runner loudly only for an execution error (missing binary, SLURM/container problem) where the test couldn't run.
* Importing from `fleet_manager` inside `hardware_tests_runner` (or vice versa). The contract package is the only shared dependency.
* Registering the runner but forgetting to register the fleet-manager orchestration class (or the other way round). Both registries must list the slug.
* Forgetting to add the new slug to `stale_test_operations.yaml`. The coverage-gate unit test catches this at CI time, but it's cheaper to decide up front which hardware operations should invalidate the test's evidence.
* Trying to define the thresholds *schema* in `thresholds.yaml`. The schema (and `catalogue_thresholds()`) lives in Python; the YAML only carries values.
* Forgetting to add the threshold values for the slug to `thresholds.yaml` — `BaseTest.load_thresholds` will fail at runtime.
* Reading `context.nodes` in a multi-node `make_batch_config`. Use `context.num_nodes` — `nodes` is forced to 1 when `nodelist` is supplied.
* Dumping every parsed value into `metrics_class`. Anything that isn't decision-grade belongs in a custom artifact, not the UI.
* Encoding units in metric names (`*_gbps`, `*_us`) instead of using `MetricUnit`.
* Ordering metrics by parsing convenience instead of operator importance.
* Coining a near-duplicate slug. Name it for the workload, stage-prefixed and consistent with siblings (`multi-node.nccl-all-reduce`), and reuse an existing family where one fits.
* Building a standalone test for a small node-level check. Extend `NodeHealthCheckTest` (an opt-in `MetricValue` field + a `checks` entry) instead of a new slug + runner + thresholds.
* Choosing pairwise network tests when a collective gives the signal. Collective is simpler and surfaces in the UI; pairwise fans out into per-pair executions off the standard metrics surface — use it only for per-NIC-pair isolation.

## Good references

* `src/fleet_manager/core/hardware_tests/single_node/hello_world.py` + `libs/hardware-tests-runner/src/hardware_tests_runner/tests/hello_world.py` — minimal end-to-end example.
* `src/fleet_manager/core/hardware_tests/single_node/hpl_gpu.py` — GPU single-node test with thresholds.
* `src/fleet_manager/core/hardware_tests/multi_node/nccl.py` — multi-node collective.
* `src/fleet_manager/core/hardware_tests/multi_node/pairwise_base.py` — pairwise base class (WIP in the UI, used for reports today).
* `.agents/skills/burn-in-test-creator/SKILL.md` — internal agent checklist for recipe-driven tests, fail-loud review constraints, SLURM gotchas, and calibration discipline.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.