> ## Documentation Index > Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt > Use this file to discover all available pages before exploring further. # Test Strategy > The three layers of Fleet Manager testing — what each layer covers, what it leaves to the layer above, and where a new test belongs. Fleet Manager is tested in three layers, each trading speed for fidelity. The guiding rule: **add coverage at the lowest layer that can catch the bug.** Most coverage lives at the base; the higher layers stay small and focused on what only they can catch. Fleet Manager testing pyramid Toward the top: slower, fewer, higher-fidelity (real infrastructure). Toward the base: faster, more numerous, fully mocked. | Layer | What it exercises | External I/O | Marker | | - | - | - | - | | **Unit & API** | Python logic, activities, serializers, DB-backed API views | External services mocked; Django test DB for `django_db` tests | *(default)* | | **Integration & contract** | Real component boundaries + API contracts vs. consumer pacts | Database, SSH, APIs | `integration`, `pact` | | **E2E** | Full workflows, end to end | Real or stack infrastructure | `e2e` (+ env scope) | ## Unit & API The base layer, and where the bulk of coverage lives — add tests here by default. External services are mocked (no [BMCs](../start-here/glossary#bmc), no OpenStack, no [Temporal](../start-here/glossary#temporal) server, no network) and Temporal activities are stubbed so workflow logic is exercised deterministically. It isn't purely in-memory, though: API and serializer tests marked `django_db` run against a transactional test database, so the API contract is verified through real serialization. **Catches:** logic bugs, serializer/schema drift, workflow branching, regressions in core domain code. This is the only layer that gates the Docker image build — a failure here blocks the release. ## Integration & contract The middle tier exercises real component boundaries without standing up real infrastructure. **Integration tests** need a real local dependency — a database, SSH, or an API — rather than pure mocks, covering cross-component behaviour such as health endpoints, Temporal enrolment replay, and CLI commands. Most run on every PR (`make test-integration-ci`); tests marked `mgmt_network` require the management network and run only from there. **Catches:** wiring and concurrency bugs that mocks hide. **Contract tests** replay consumer-published pacts against the real API, so a breaking change to a response shape is caught before it reaches a consumer (the reservation service and others). **Catches:** API contract breaks against downstream consumers. ## E2E The top layer: full workflows, end to end, kept small and focused on critical paths. `e2e` is the **layer** marker; a second marker scopes *where* a test runs — `local_docker_only` or `dev_only` — so `e2e` is the layer, not an environment. Use the repository's `e2e-tests` agent skill for the branch/PR test loop and for choosing between local Compose, app51, and shared dev infrastructure. Two environments, each catching what the other can't: * **Local Docker Compose** — the fast local loop against the Compose stack. Plain `e2e` tests (no env scope) run here, plus `local_docker_only` tests. These include the "docker CI" [NetBox](../start-here/glossary#netbox) suite (`make docker-up-ci` + `make test-e2e-ci`), which runs against a **local seeded NetBox** the harness owns and can safely mutate. **Catches:** upstream-data and freshness regressions the dev suite can't touch safely, e.g. inventory-projection freshness between NetBox and the FM API, plus node-swap cable recreation and rack-slot mutation against NetBox's real API constraints. The full suite runs nightly in the *Docker CI E2E Tests* workflow; the focused node-swap test also runs on relevant pull requests. For the local loop, `make test-e2e-netbox` starts only NetBox and runs the destructive mutation test against disposable topology. * **Dev** (`dev_only`) — full workflows against real dev infrastructure ([Ironic](../start-here/glossary#ironic), [Nova](../start-here/glossary#nova), OpenStack, [SLURM](../start-here/glossary#slurm)) on shared glo1. **Catches:** integration failures that only appear against real hardware and OpenStack — enrolment, [burn-in](../start-here/glossary#burn-in), quarantine/release — that every lower layer mocks away. This suite also gates production promotion. ### Run the isolated NetBox mutation test Use the focused target while changing NetBox write paths such as node-swap cable recreation or [device](../start-here/glossary#node) placement: ```bash theme={null} make test-e2e-netbox ``` The target starts only the local NetBox Compose stack on port 8090; it does not need [Harbor](../start-here/glossary#harbor) credentials, [1Password](../start-here/glossary#1password), the management network, or a Fleet Manager API container. The test client is pinned to the throwaway URL and token in `tests/e2e/settings.toml`, so this target cannot mutate dev or production NetBox. Each run creates a uniquely named topology, exercises the production NetBox domain function, and removes the topology afterward. The stack stays up to make reruns fast. Stop it when finished: ```bash theme={null} make docker-down-netbox ``` If startup or cleanup fails, inspect the local service and rerun after fixing it: ```bash theme={null} docker compose -p "fm-$(whoami | tr '.' '-')" \ -f deployment/docker/docker-compose.netbox.yml logs --tail=100 netbox ``` ### Run the VRF-flip emulator suites Use this target while changing the front-end or back-end VRF flips, the switch client or their activities. It needs Harbor credentials and local copies of the tracked `.env.example` and `deployment/docker/.env.example` templates, which the Compose stack reads: ```bash theme={null} ENVIRONMENT=local_docker OP_VAULT_SELECTOR=glo1 make test-e2e-vrf ``` A worker runs only the VRF workflows and their real activities against an HTTPS NVUE emulator in its network namespace, and refuses any switch target that is not loopback. To keep the suite short, the worker shortens the real-time waits: the NVUE client's request timeout and confirmation polling, and the workflows' retry delay and ceiling on an unknown write outcome (three minutes, not 20), with the front-end lock's lease cut to match. Production keeps its own values. The back-end suite flips the 16 PFs of `gpu40`, and of `gpu41` when two flips share leaves, on leaves at `127.0.1.1`–`127.0.1.16`. The emulator replays a revision's snapshot when it saves, so overlapping applies revert each other as the per-SU lock assumes, and it also stubs the `netbox_datacenter` plugin's peer mappings, which the local NetBox lacks. No CI workflow runs this target; a run takes about 14 minutes. *** For how to run each suite, the CI workflows, and their schedules, see the project `Makefile` and the workflow definitions under `.github/workflows/`. For the dev environment and the production-promotion gate, see [Testing in Dev](./testing-in-dev) and [Releasing to Production](./releasing-to-production#dev-e2e-gate). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.