Skip to main content
Fleet Manager is tested in three layers, each trading speed for fidelity. The guiding rule: add coverage at the lowest layer that can catch the bug. Most coverage lives at the base; the higher layers stay small and focused on what only they can catch. Fleet Manager testing pyramid Toward the top: slower, fewer, higher-fidelity (real infrastructure). Toward the base: faster, more numerous, fully mocked.

​
Unit & API

The base layer, and where the bulk of coverage lives — add tests here by default. External services are mocked (no BMCs, no OpenStack, no Temporal server, no network) and Temporal activities are stubbed so workflow logic is exercised deterministically. It isn’t purely in-memory, though: API and serializer tests marked django_db run against a transactional test database, so the API contract is verified through real serialization. Catches: logic bugs, serializer/schema drift, workflow branching, regressions in core domain code. This is the only layer that gates the Docker image build — a failure here blocks the release.

​
Integration & contract

The middle tier exercises real component boundaries without standing up real infrastructure. Integration tests need a real local dependency — a database, SSH, or an API — rather than pure mocks, covering cross-component behaviour such as health endpoints, Temporal enrolment replay, and CLI commands. Most run on every PR (make test-integration-ci); tests marked mgmt_network require the management network and run only from there. Catches: wiring and concurrency bugs that mocks hide. Contract tests replay consumer-published pacts against the real API, so a breaking change to a response shape is caught before it reaches a consumer (the reservation service and others). Catches: API contract breaks against downstream consumers.

​
E2E

The top layer: full workflows, end to end, kept small and focused on critical paths. e2e is the layer marker; a second marker scopes where a test runs — local_docker_only or dev_only — so e2e is the layer, not an environment. Use the repository’s e2e-tests agent skill for the branch/PR test loop and for choosing between local Compose, app51, and shared dev infrastructure. Two environments, each catching what the other can’t:
  • Local Docker Compose — the fast local loop against the Compose stack. Plain e2e tests (no env scope) run here, plus local_docker_only tests. These include the “docker CI” NetBox suite (make docker-up-ci + make test-e2e-ci), which runs against a local seeded NetBox the harness owns and can safely mutate. Catches: upstream-data and freshness regressions the dev suite can’t touch safely, e.g. inventory-projection freshness between NetBox and the FM API, plus node-swap cable recreation and rack-slot mutation against NetBox’s real API constraints. The full suite runs nightly in the Docker CI E2E Tests workflow; the focused node-swap test also runs on relevant pull requests. For the local loop, make test-e2e-netbox starts only NetBox and runs the destructive mutation test against disposable topology.
  • Dev (dev_only) — full workflows against real dev infrastructure (Ironic, Nova, OpenStack, SLURM) on shared glo1. Catches: integration failures that only appear against real hardware and OpenStack — enrolment, burn-in, quarantine/release — that every lower layer mocks away. This suite also gates production promotion.

​
Run the isolated NetBox mutation test

Use the focused target while changing NetBox write paths such as node-swap cable recreation or device placement:
The target starts only the local NetBox Compose stack on port 8090; it does not need Harbor credentials, 1Password, the management network, or a Fleet Manager API container. The test client is pinned to the throwaway URL and token in tests/e2e/settings.toml, so this target cannot mutate dev or production NetBox. Each run creates a uniquely named topology, exercises the production NetBox domain function, and removes the topology afterward. The stack stays up to make reruns fast. Stop it when finished:
If startup or cleanup fails, inspect the local service and rerun after fixing it:

​
Run the VRF-flip emulator suites

Use this target while changing the front-end or back-end VRF flips, the switch client or their activities. It needs Harbor credentials and local copies of the tracked .env.example and deployment/docker/.env.example templates, which the Compose stack reads:
A worker runs only the VRF workflows and their real activities against an HTTPS NVUE emulator in its network namespace, and refuses any switch target that is not loopback. To keep the suite short, the worker shortens the real-time waits: the NVUE client’s request timeout and confirmation polling, and the workflows’ retry delay and ceiling on an unknown write outcome (three minutes, not 20), with the front-end lock’s lease cut to match. Production keeps its own values. The back-end suite flips the 16 PFs of gpu40, and of gpu41 when two flips share leaves, on leaves at 127.0.1.1–127.0.1.16. The emulator replays a revision’s snapshot when it saves, so overlapping applies revert each other as the per-SU lock assumes, and it also stubs the netbox_datacenter plugin’s peer mappings, which the local NetBox lacks. No CI workflow runs this target; a run takes about 14 minutes.
For how to run each suite, the CI workflows, and their schedules, see the project Makefile and the workflow definitions under .github/workflows/. For the dev environment and the production-promotion gate, see Testing in Dev and Releasing to Production.