Build a New Burn-in Test

​

Test categories

• Single-node — runs on exactly one compute node. Used for per-device validation (HPL, NCCL loopback, DCGMI, NVMe, node health). One SLURM job per device. Orchestrated by SingleNodeTest.
• Multi-node collective — runs across two or more nodes as a single coordinated job and produces aggregate metrics (multi-node NCCL all-reduce/all-gather/all-to-all, iperf matrix, distributed training, parallel storage benchmarks). One SLURM job for the whole cohort. Orchestrated by MultiNodeTest.
• Multi-node pairwise — fans out per-NIC-pair tests across a cohort and reports per-pair metrics (PAIRWISE_NET_BW, PAIRWISE_NET_LAT, GPU variants). Orchestrated by PairwiseNetworkTestBase (a MultiNodeTest). Status: WIP. The execution path works and reports include pairwise data, but the UI surfaces only single-node and multi-node collective results today; pairwise is reports-only.

​

Architecture

​

Runtime topology

• Operator kicks off a burn-in Job through the Fleet Manager API.
• Fleet Manager API + DB (hub) own the desired state (jobs, presets, thresholds) and the materialised results.
• Workflow Agent (per satellite) reconciles Jobs into Temporal workflow executions.
• Temporal BurnInWorkflow (single-node, multi-node collective, or multi-node pairwise variant) renders an sbatch script per test, dispatches it, polls for completion, then pulls artifacts from object storage and writes results back to the API.
• SLURM restd → compute nodes runs the actual job and invokes the runner CLI.
• Compute nodes stream stdout/stderr and custom artifacts to object storage via presigned URLs.
• The workflow then pulls from object storage to evaluate metrics against thresholds and persist a TestResult.

​

Life of a single test

1. The Temporal workflow asks the activity to render and submit a job for a given (slug, params, RunContext).
2. The activity loads the test class via registry.get_test_class, validates params, builds a SlurmBatchConfig and an ExecutionContext, generates presigned S3 URLs, and renders an sbatch script from shared/templates/sbatch.sh.j2.
3. SLURM runs the script. The script invokes hardware-tests run <slug> --workspace $WORKSPACE --config-json "$CONFIG_JSON" on the compute node.
4. On the node, the runner instantiates the BaseRunner subclass for the slug and calls async execute(...). WorkspaceManager captures stdout/stderr, writes metadata.json, packages results/ into custom-artifacts.tar.gz, and uploads via the presigned URLs.
5. The workflow downloads the artifacts via ArtifactCollector, calls the test’s extract_metrics(...) for typed metrics, runs evaluate(...) against the thresholds, and writes a TestResult back through the API.

​

Author a new test

​

1. Extend the shared contract

• Add a slug to slugs.py (SingleNodeTests or MultiNodeTests enum + Slugs.* constant).
• Add a Pydantic params model under params/<family>.py, then re-export it from params/__init__.py so callers can import it from hardware_tests_contract.params. Keep orchestration fields out (partition, nodes, timeout, GPUs) — those come from RunContext. Params describe what the test does, not how SLURM should schedule it. Expose only genuine workload knobs — enough that real knobs aren’t hard-coded, not so many that the surface sprawls — and keep out any knob that would move the right threshold. Thresholds are calibrated against one fixed configuration and are never switched on param values at runtime, so a threshold-moving param silently invalidates its gate; bake those into the recipe/config at the calibrated value instead.
• Add a *Metrics model to metrics/models.py extending BaseMetrics (from metrics/base.py), or use PairwiseNetworkMetrics for pairwise. Implement to_metrics_response() so the API/DB representation is explicit.
• Add the new slug to at least one operation’s test_slugs list in src/fleet_manager/core/burn_in/stale_test_operations.yaml — the hardware operation(s) that should invalidate this test’s evidence (e.g. node swap, firmware update, power cycle). There is no default mapping: a coverage-gate unit test (tests/core/burn_in/test_stale_test_operations.py) fails CI if a registered slug isn’t listed under at least one operation.

​

2. Implement the worker-side runner

• Subclass BaseRunner[MyParams], set slug and params_class, and implement async execute(self, context, workspace_manager).
• Use workspace_manager.run_command(...) for shell workloads, write_custom_artifact(...) / get_custom_artifact_path(...) for outputs. BaseRunner takes care of log files, metadata.json, archiving, and uploads.
• Provide tooling via the node OS image PATH, a test container image, or explicit binary paths in params (e.g. hpl_binary).
• Register the runner in hardware_tests_runner/registry.py.

​

3. Implement the fleet-manager orchestration class

• slug, display_name, params_class, metrics_class, thresholds_class.
• make_batch_config(self, context: RunContext) -> SlurmBatchConfig — use make_single_node_slurm_batch_config / make_multi_node_slurm_batch_config. RunContext exposes partition, timeout_minutes, optional nodelist, and the num_nodes property — always use context.num_nodes for multi-node tests, not context.nodes. When a nodelist is supplied, nodes is forced to 1 and the real count comes from the list length; num_nodes resolves both cases. GPU allocation comes from params or class attributes (e.g. gpus_per_node=0 for non-GPU tests).
• extract_metrics(self, artifact_collector) -> BaseMetrics | PairwiseNetworkMetrics — pure parsing, no I/O. Reach into artifact_collector.paths[TestArtifactFile.STDOUT] for logs, artifact_collector.get_custom_file("pattern") for unpacked custom artifacts, and artifact_collector.metadata for the standard HardwareTestMetadata.
• evaluate(cls, metrics, thresholds, iteration_summary=None, resolved_params=None) -> EvaluationResult — populate metric_evaluations with TestMetricEvaluation entries (use metric_pass() / metric_fail(...)), set a concise reason, and never raise — the activity wraps EvaluationResult into a typed TestFailedEvaluationError. Use resolved_params only for persisted per-run policy that genuinely changes the verdict; parse it with the test’s params model and return a failed EvaluationResult if it is malformed.

​

4. Design the metrics for the UI

• Only expose metrics operators should care about. If a number isn’t going to drive a pass/fail decision or a triage step, leave it out of the metrics model — write it to a custom artifact instead. Attention is a precious resource.
• Order matters. Field order in the Pydantic model is the display order. Put the most important metric first; less important context last.
• Name metrics for humans. Use clear, plain names (bus_bandwidth, min_latency) — not internal column names or test-runner jargon.
• Don’t repeat units in names. That’s what MetricUnit is for. bus_bandwidth with MetricUnit.GB_S, not bus_bandwidth_gbps. Same for latency, throughput, percentages.
• Pick the right unit and stick with it across tests. Use MetricUnit.GB_S for gigabytes/s (memory/bus) and MetricUnit.GBPS for gigabits/s (network). They are not interchangeable.
• Match thresholds to the metrics you display. If a metric isn’t worth a threshold or a comparator, it probably isn’t worth showing.
• Hide non-decision fields, don’t drop them. Mark fields an operator wouldn’t act on (problem size, block size, raw counts) api_visible=False — they still persist for evaluation and debugging but stay out of the UI.
• Use detail for actionable failure context. The per-MetricValue detail string (~2 KB, shown on the UI) is the operator’s first clue when a gate trips — name what and where, not just the number: the misconfigured NICs and their settings, the worst GPU pairs, the GPUs that failed ECC.

​

5. Verify locally

# Discover and inspect
uv run fleetctl hardware-tests list
uv run fleetctl hardware-tests schema <stage>.<test-name>

# Render the sbatch script without submitting
uv run fleetctl hardware-tests generate-sbatch <stage>.<test-name> \
  --params '{...}' --partition testing --timeout 15 --show-script

# Submit end-to-end via the SLURM REST API and poll for results
uv run fleetctl hardware-tests dispatch <stage>.<test-name> \
  --params '{...}' --partition testing --timeout 15

​

6. Tests and pre-commit

• Unit tests for extract_metrics against canned artifacts.
• Unit tests for evaluate against representative metrics + thresholds.
• A worker-side test for execute that exercises WorkspaceManager writes.
• Wiring assertions: the slug appears in list_all_tests() and hardware_tests_runner.registry.

make pre-commit
make test

​

Common mistakes

• Putting orchestration fields (partition, nodes, timeout, GPU counts) in the params model instead of RunContext.
• Exposing a param that changes the correct threshold. Thresholds don’t switch on param values, so the gate silently goes meaningless when a caller flips it — bake threshold-moving knobs into the recipe/config at the calibrated value.
• Doing I/O inside extract_metrics — it must be pure parsing against ArtifactCollector.
• Scraping stdout for metrics when the runner could write a JSON artifact. Emit structured JSON and extract_metrics becomes a trivial json.loads; reserve log parsing for third-party tools whose output you don’t control.
• Swallowing an error or defaulting a metric on bad input. A silent fallback hides a bigger fault and wastes time later — extract_metrics raises on a missing or malformed artifact instead of returning zeros that pass the gate.
• Raising inside evaluate instead of returning an EvaluationResult.
• Threading ad-hoc runtime params into evaluate(). It receives metrics + thresholds, optional iteration_summary, and the persisted execution resolved_params; if a value is a measured hardware fact rather than per-run policy, tag it onto the metrics payload host-side instead.
• Flipping a comparator. Latency and error-rate gate as upper bounds; bandwidth, throughput, and TFLOPs as lower bounds — a swapped comparator passes garbage green.
• Aborting the runner on a legitimate hardware-fault exit. A benchmark exiting non-zero because it found a real fault is data: capture it and let evaluate fail the verdict. Fail the runner loudly only for an execution error (missing binary, SLURM/container problem) where the test couldn’t run.
• Importing from fleet_manager inside hardware_tests_runner (or vice versa). The contract package is the only shared dependency.
• Registering the runner but forgetting to register the fleet-manager orchestration class (or the other way round). Both registries must list the slug.
• Forgetting to add the new slug to stale_test_operations.yaml. The coverage-gate unit test catches this at CI time, but it’s cheaper to decide up front which hardware operations should invalidate the test’s evidence.
• Trying to define the thresholds schema in thresholds.yaml. The schema (and catalogue_thresholds()) lives in Python; the YAML only carries values.
• Forgetting to add the threshold values for the slug to thresholds.yaml — BaseTest.load_thresholds will fail at runtime.
• Reading context.nodes in a multi-node make_batch_config. Use context.num_nodes — nodes is forced to 1 when nodelist is supplied.
• Dumping every parsed value into metrics_class. Anything that isn’t decision-grade belongs in a custom artifact, not the UI.
• Encoding units in metric names (*_gbps, *_us) instead of using MetricUnit.
• Ordering metrics by parsing convenience instead of operator importance.
• Coining a near-duplicate slug. Name it for the workload, stage-prefixed and consistent with siblings (multi-node.nccl-all-reduce), and reuse an existing family where one fits.
• Building a standalone test for a small node-level check. Extend NodeHealthCheckTest (an opt-in MetricValue field + a checks entry) instead of a new slug + runner + thresholds.
• Choosing pairwise network tests when a collective gives the signal. Collective is simpler and surfaces in the UI; pairwise fans out into per-pair executions off the standard metrics surface — use it only for per-NIC-pair isolation.

​

Good references

• src/fleet_manager/core/hardware_tests/single_node/hello_world.py + libs/hardware-tests-runner/src/hardware_tests_runner/tests/hello_world.py — minimal end-to-end example.
• src/fleet_manager/core/hardware_tests/single_node/hpl_gpu.py — GPU single-node test with thresholds.
• src/fleet_manager/core/hardware_tests/multi_node/nccl.py — multi-node collective.
• src/fleet_manager/core/hardware_tests/multi_node/pairwise_base.py — pairwise base class (WIP in the UI, used for reports today).
• .agents/skills/burn-in-test-creator/SKILL.md — internal agent checklist for recipe-driven tests, fail-loud review constraints, SLURM gotchas, and calibration discipline.