> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Site Dependencies & External Systems

> The systems and prerequisites that must exist before Fleet Manager can be rolled out at a site.

Fleet Manager depends on several external systems. A site is not ready for rollout until the
core infrastructure, inventory, credentials, and workflow dependencies are in place.

## Base Infrastructure

Before Fleet Manager itself is deployed, the site needs:

* an agreed deployment identifier such as `glo1` or `sta1`
* dedicated Kubernetes clusters for the Fleet Manager environments
* basic networking for those clusters
* a [DHCP](../../start-here/glossary#dhcp) server VM that Fleet Manager can reach over SSH

Kubernetes clusters usually follow this naming pattern:

```bash theme={null}
fleet-mgr-{ENV}-{DEPLOYMENT}
```

Examples:

```bash theme={null}
fleet-mgr-dev-glo1
fleet-mgr-prod-sta1
```

## NetBox

[NetBox](../../start-here/glossary#netbox) is the source of truth for inventory and site metadata.
Before rollout:

* [devices](../../start-here/glossary#node) for the site should already exist in NetBox
* devices intended for Fleet Manager ingestion should be marked as `staging`
* deployment-specific config context should be present where required

The new-site runbook calls out config context as especially important for:

* Mellanox and Broadcom NIC configuration
* firmware expectations
* BIOS settings

## Secrets and Credentials

Fleet Manager expects deployment- and environment-scoped secrets.
For each target environment, create:

* `Fleet Manager {deployment} {env}` for runtime secrets
* `Fleet Manager {deployment}-operators {env}` for operator-managed secrets such as [BMC](../../start-here/glossary#bmc) and [IPMI](../../start-here/glossary#ipmi) credentials

These vaults typically hold:

* [NetBox API tokens](./deploy-fleet-manager-at-a-new-site#create-the-netbox-api-token) with
  the required Object Permissions
* OpenStack application credentials
* DHCP SSH keys
* [SLURM](../../start-here/glossary#slurm) JWT tokens
* operator-managed credentials such as `ipmi-exporter`

## OpenStack

Fleet Manager uses OpenStack for bare-metal and compute workflows.
That means the site needs:

* application credentials in the correct projects
* the correct access rules for [Ironic](../../start-here/glossary#ironic), [Placement](../../start-here/glossary#placement), and [Nova](../../start-here/glossary#nova)
* [IPA](../../start-here/glossary#ironic-python-agent) kernel and ramdisk images uploaded to the target environment

Use [OpenStack Application Credentials](./openstack-application-credentials) for the required access rules.

## DHCP

The DHCP server must be ready for Fleet Manager's automation model:

* Fleet Manager needs SSH access to the host
* the automation user must be able to run the required commands without password prompts
* the restart lock file used to serialize dnsmasq restarts must exist on the server

## SLURM

If the site will run [burn-in](../../start-here/glossary#burn-in) and validation workflows, it also needs:

* a reachable SLURM controller with the REST API enabled
* network connectivity from the Fleet Manager Kubernetes cluster to the SLURM API
* JWT credentials stored in the site's runtime vault
* the expected [GPU](../../start-here/glossary#gpu) partition and test environment on the cluster

## Fabric Control Planes

Two workflows write to fabric hardware Fleet Manager does not enrol: [NVLink](../../start-here/glossary#nvlink) partition moves
through NMX-M, and front-end switch [VRF](../../start-here/glossary#vrf) flips. Each is gated by its own per-deployment flag,
each defaults to off, and **the two are independent** — a site can enable either, both, or
neither. NMX-M availability is not a precondition for VRF flipping, and a site that flips VRFs
does not need an NMX-M control plane.

Until a site opts in, the corresponding workflow runs against a simulated control plane. A
simulated write is reported as simulated and must not be treated as satisfying a [handover](../../start-here/glossary#handover) or
return safety gate.

Both read their service account from the site's networking vault
(`Fleet Manager {deployment}-networking {env}`, see
[Create the Fleet Manager Vaults](./deploy-fleet-manager-at-a-new-site#create-the-fleet-manager-vaults)).
Enabling a flag without its credential item fails closed before the control plane is contacted.

Every [Helm](../../start-here/glossary#helm) value below is a **satellite** chart value — the hub chart has no `workflowAgent` or
`temporalWorker` key. They split across those two: the opt-ins and their fabric configuration are
`workflowAgent` values, read by the workflow agent at submission, while the credential item titles
are `temporalWorker` values, read by the worker when it authenticates.

### NVLink Partitions (NMX-M)

Opt in with `nvlink_partition_allowed` (`workflowAgent.nvlinkPartitionAllowed`). The site also needs:

* an NMX-M controller reachable from the [Temporal](../../start-here/glossary#temporal) worker for every rack that may be partitioned
* an entry per FM rack name in `nmx_hosts` (`workflowAgent.nmxHosts`) — a bare host or IP, not a
  URL, plus the `domainUuid` read from `GET /nmx/v1/services?type=CONTROLLER`. The domain UUID is
  mandatory for real writes, so a controller pointed at the wrong domain fails before any partition
  is read or changed
* an NMX-M service account in the networking vault (`temporalWorker.nmxCredentialsItem`, default
  `svc-nmx-login`); a rack needing its own account sets `credentialsItem` on its `nmxHosts` entry
* `workflowAgent.nmxActivityArgsReady` set true. It gates the per-rack activity arguments and
  defaults to false, so a submission with the opt-in on but this false is refused. Flip it only
  once every Temporal worker accepts the arguments and the old ReplicaSet is at zero

NMX-M is a per-rack service and NMX slot IDs are unique only inside a domain, so with the flag on,
a device whose rack has no `nmxHosts` entry — or an entry without a `domainUuid` — is refused at
submission rather than sent to a guessed controller.

### Front-end VRF Flips

Opt in with `switch_vrf_flip_allowed` (`workflowAgent.switchVrfFlipAllowed`). The site also needs:

* the switch management network reachable from the Temporal worker, verified before enabling with a
  `GET /nvue_v1/revision` read against the site's switch
* an NVUE service account in the networking vault (`temporalWorker.switchCredentialsItem`, default
  `svc-switch-login`)
* the site's two VRF names in `workflowAgent.switchHandoverVrf` (the customer VRF that Frontend
  VRF Return moves ports into) and `workflowAgent.switchReturnVrf` (the maintenance VRF that
  Frontend VRF Isolate moves them into). The keys predate the FLEET-2825 workflow rename. These are switch configuration rather than a Fleet Manager concept and differ per site; leaving
  them empty means the site does not flip VRFs and the workflow fails loudly rather than guessing a
  routing table
* `workflowAgent.switchApiAddressField` set to whichever NetBox field carries the switch management
  address at this site (`oob_ip` or `primary_ip`)
* NetBox cabling that resolves every front-end host interface to a host-facing switch port, with
  switch-side LAG membership recorded for every bonded port. The flip decides per port whether to
  write the bond or the physical port, and it decides from NetBox — a port genuinely in a LAG but
  with no LAG parent recorded is flipped at the member, where the VRF does not live, and the
  read-back confirms the member anyway. That failure is silent and green
* every front-end interface type declared on the device's `ironic_settings`. The flip scopes on
  `frontend_interface_types`, which is `interface_type` **plus**
  `additional_frontend_interface_types`; `interface_type` alone is the provisioning bond. A host
  whose front end spans a second NetBox type must declare it, or the flip moves one type's ports,
  re-reads only those, and reports a verified success with the rest still in the old VRF

The two NetBox items above are site data rather than Fleet Manager config, and both fail green
rather than loudly, so confirm them before enabling the flag. The
[switch VRF runbook](https://github.com/nscaledev/fleet-manager/blob/main/docs/runbooks/switch-vrf-flip.md)
carries the full pre-flight checklist, including the worker-image rollout that must complete before
the flag is flipped.

### Back-end (RoCE) VRF Flips

Opt in with `backend_vrf_flip_allowed` (`temporalWorker.backendVrfFlipAllowed`), and only once
every satellite worker at the site runs 0.289.0 or later, the first release that carries the per-SU
lock (FLEET-2813). Never set it on 0.288.x, which writes real leaves without the lock. The
[back-end VRF runbook](https://github.com/nscaledev/fleet-manager/blob/main/docs/runbooks/backend-vrf-flip.md#enabling-real-writes)
lists the site checks. The site also needs:

* the back-end VRF pair in `temporalWorker.backendCustomerVrf` and
  `temporalWorker.backendMaintenanceVrf` (`be-meta` / `be-nscale` at GSO1)
* a back-end leaf account in the networking vault (`temporalWorker.switchBackendCredentialsItem`,
  or the front-end item when empty) and the leaves' management addresses in the NetBox field
  `temporalWorker.backendSwitchApiAddressField` names
* `workflowAgent.releaseActivityArgsReady`, so Release re-reads the back-end latch in flight; a real
  back-end flip refuses without it

The [back-end VRF runbook](https://github.com/nscaledev/fleet-manager/blob/main/docs/runbooks/backend-vrf-flip.md)
carries the checklist, including the worker-image rollout and the deploy-wrapper forwarding to
confirm before the flag is flipped.

## Observability

Site validation also assumes access to:

* Grafana dashboards and Loki logs
* Temporal workflow visibility
* the Fleet Manager admin or API surface used to verify imported devices and jobs

The satellite probes the hub API through the observability stack's blackbox exporter: the
`prometheus-blackbox-exporter` Service in namespace `prometheus-blackbox-exporter`, with the `http_2xx`
module. A satellite cluster without it never probes the hub, so
[FleetManagerHubApiUnreachable](../../operator-guide/sev1-runbook#fleetmanagerhubapiunreachable) cannot
fire for that site, and nothing reports the gap. The Observability Squad deploys it per cluster, from the
region list in `charts/observability/templates/otel/agent/applications/blackbox-exporter-application.yaml`
in `nscaledev/k8s-deploy-observability`; ask them to add the new site.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.