Skip to main content
Fleet Manager depends on several external systems. A site is not ready for rollout until the core infrastructure, inventory, credentials, and workflow dependencies are in place.

​
Base Infrastructure

Before Fleet Manager itself is deployed, the site needs:
  • an agreed deployment identifier such as glo1 or sta1
  • dedicated Kubernetes clusters for the Fleet Manager environments
  • basic networking for those clusters
  • a DHCP server VM that Fleet Manager can reach over SSH
Kubernetes clusters usually follow this naming pattern:
Examples:

​
NetBox

NetBox is the source of truth for inventory and site metadata. Before rollout:
  • devices for the site should already exist in NetBox
  • devices intended for Fleet Manager ingestion should be marked as staging
  • deployment-specific config context should be present where required
The new-site runbook calls out config context as especially important for:
  • Mellanox and Broadcom NIC configuration
  • firmware expectations
  • BIOS settings

​
Secrets and Credentials

Fleet Manager expects deployment- and environment-scoped secrets. For each target environment, create:
  • Fleet Manager {deployment} {env} for runtime secrets
  • Fleet Manager {deployment}-operators {env} for operator-managed secrets such as BMC and IPMI credentials
These vaults typically hold:
  • NetBox API tokens with the required Object Permissions
  • OpenStack application credentials
  • DHCP SSH keys
  • SLURM JWT tokens
  • operator-managed credentials such as ipmi-exporter

​
OpenStack

Fleet Manager uses OpenStack for bare-metal and compute workflows. That means the site needs:
  • application credentials in the correct projects
  • the correct access rules for Ironic, Placement, and Nova
  • IPA kernel and ramdisk images uploaded to the target environment
Use OpenStack Application Credentials for the required access rules.

​
DHCP

The DHCP server must be ready for Fleet Manager’s automation model:
  • Fleet Manager needs SSH access to the host
  • the automation user must be able to run the required commands without password prompts
  • the restart lock file used to serialize dnsmasq restarts must exist on the server

​
SLURM

If the site will run burn-in and validation workflows, it also needs:
  • a reachable SLURM controller with the REST API enabled
  • network connectivity from the Fleet Manager Kubernetes cluster to the SLURM API
  • JWT credentials stored in the site’s runtime vault
  • the expected GPU partition and test environment on the cluster

​
Fabric Control Planes

Two workflows write to fabric hardware Fleet Manager does not enrol: NVLink partition moves through NMX-M, and front-end switch VRF flips. Each is gated by its own per-deployment flag, each defaults to off, and the two are independent — a site can enable either, both, or neither. NMX-M availability is not a precondition for VRF flipping, and a site that flips VRFs does not need an NMX-M control plane. Until a site opts in, the corresponding workflow runs against a simulated control plane. A simulated write is reported as simulated and must not be treated as satisfying a handover or return safety gate. Both read their service account from the site’s networking vault (Fleet Manager {deployment}-networking {env}, see Create the Fleet Manager Vaults). Enabling a flag without its credential item fails closed before the control plane is contacted. Every Helm value below is a satellite chart value — the hub chart has no workflowAgent or temporalWorker key. They split across those two: the opt-ins and their fabric configuration are workflowAgent values, read by the workflow agent at submission, while the credential item titles are temporalWorker values, read by the worker when it authenticates. Opt in with nvlink_partition_allowed (workflowAgent.nvlinkPartitionAllowed). The site also needs:
  • an NMX-M controller reachable from the Temporal worker for every rack that may be partitioned
  • an entry per FM rack name in nmx_hosts (workflowAgent.nmxHosts) — a bare host or IP, not a URL, plus the domainUuid read from GET /nmx/v1/services?type=CONTROLLER. The domain UUID is mandatory for real writes, so a controller pointed at the wrong domain fails before any partition is read or changed
  • an NMX-M service account in the networking vault (temporalWorker.nmxCredentialsItem, default svc-nmx-login); a rack needing its own account sets credentialsItem on its nmxHosts entry
  • workflowAgent.nmxActivityArgsReady set true. It gates the per-rack activity arguments and defaults to false, so a submission with the opt-in on but this false is refused. Flip it only once every Temporal worker accepts the arguments and the old ReplicaSet is at zero
NMX-M is a per-rack service and NMX slot IDs are unique only inside a domain, so with the flag on, a device whose rack has no nmxHosts entry — or an entry without a domainUuid — is refused at submission rather than sent to a guessed controller.

​
Front-end VRF Flips

Opt in with switch_vrf_flip_allowed (workflowAgent.switchVrfFlipAllowed). The site also needs:
  • the switch management network reachable from the Temporal worker, verified before enabling with a GET /nvue_v1/revision read against the site’s switch
  • an NVUE service account in the networking vault (temporalWorker.switchCredentialsItem, default svc-switch-login)
  • the site’s two VRF names in workflowAgent.switchHandoverVrf (the customer VRF that Frontend VRF Return moves ports into) and workflowAgent.switchReturnVrf (the maintenance VRF that Frontend VRF Isolate moves them into). The keys predate the FLEET-2825 workflow rename. These are switch configuration rather than a Fleet Manager concept and differ per site; leaving them empty means the site does not flip VRFs and the workflow fails loudly rather than guessing a routing table
  • workflowAgent.switchApiAddressField set to whichever NetBox field carries the switch management address at this site (oob_ip or primary_ip)
  • NetBox cabling that resolves every front-end host interface to a host-facing switch port, with switch-side LAG membership recorded for every bonded port. The flip decides per port whether to write the bond or the physical port, and it decides from NetBox — a port genuinely in a LAG but with no LAG parent recorded is flipped at the member, where the VRF does not live, and the read-back confirms the member anyway. That failure is silent and green
  • every front-end interface type declared on the device’s ironic_settings. The flip scopes on frontend_interface_types, which is interface_type plus additional_frontend_interface_types; interface_type alone is the provisioning bond. A host whose front end spans a second NetBox type must declare it, or the flip moves one type’s ports, re-reads only those, and reports a verified success with the rest still in the old VRF
The two NetBox items above are site data rather than Fleet Manager config, and both fail green rather than loudly, so confirm them before enabling the flag. The switch VRF runbook carries the full pre-flight checklist, including the worker-image rollout that must complete before the flag is flipped.

​
Back-end (RoCE) VRF Flips

Opt in with backend_vrf_flip_allowed (temporalWorker.backendVrfFlipAllowed), and only once every satellite worker at the site runs 0.289.0 or later, the first release that carries the per-SU lock (FLEET-2813). Never set it on 0.288.x, which writes real leaves without the lock. The back-end VRF runbook lists the site checks. The site also needs:
  • the back-end VRF pair in temporalWorker.backendCustomerVrf and temporalWorker.backendMaintenanceVrf (be-meta / be-nscale at GSO1)
  • a back-end leaf account in the networking vault (temporalWorker.switchBackendCredentialsItem, or the front-end item when empty) and the leaves’ management addresses in the NetBox field temporalWorker.backendSwitchApiAddressField names
  • workflowAgent.releaseActivityArgsReady, so Release re-reads the back-end latch in flight; a real back-end flip refuses without it
The back-end VRF runbook carries the checklist, including the worker-image rollout and the deploy-wrapper forwarding to confirm before the flag is flipped.

​
Observability

Site validation also assumes access to:
  • Grafana dashboards and Loki logs
  • Temporal workflow visibility
  • the Fleet Manager admin or API surface used to verify imported devices and jobs
The satellite probes the hub API through the observability stack’s blackbox exporter: the prometheus-blackbox-exporter Service in namespace prometheus-blackbox-exporter, with the http_2xx module. A satellite cluster without it never probes the hub, so FleetManagerHubApiUnreachable cannot fire for that site, and nothing reports the gap. The Observability Squad deploys it per cluster, from the region list in charts/observability/templates/otel/agent/applications/blackbox-exporter-application.yaml in nscaledev/k8s-deploy-observability; ask them to add the new site.