Deploy Fleet Manager at a New Site

Use this page as the main rollout guide for a new Fleet Manager site. It collects the deploy-time procedure in one place and links out only when a separate page is already the canonical source for that topic. Steps owned by other teams (Networking, Kubernetes, OpenStack) are called out as dependencies. When can I use Fleet Manager? You can log in at fleet.nscale.com once the Apply / Deploy step completes (Phase 7 — see Deploy and Argo Sync). The flow below shows the dependency order: grey is physical/DC-ops work, yellow gates are readiness owned by other teams, blue is Fleet Manager configuration, green is applying the change, and purple is validation. Dashed gates are per-site opt-ins most sites skip — see Fabric Control Planes. The detailed phase diagram breaks the same rollout into its full set of steps: green is handled by the standup automation, yellow is manual/owner work, blue is config-repo PRs, and purple is validation. Dashed nodes are per-deployment opt-ins most sites skip. The two dashed nodes are independent of each other: NVLink partition writes and switch VRF flips are separate per-deployment flags, both off by default, and a site can enable either, both, or neither. Their prerequisites are in Fabric Control Planes.

Prerequisites and Access

Set up general access first — GitHub, 1Password, HashiCorp Vault, NetBox, and Teleport — using the Getting Access guide. This section covers only what’s specific to standing up a new site. You need GitHub access (via the fleet-manager team) to these repos:

Beyond that standard access, the bootstrap automation requires these rights, requested via Servicely:

create 1Password vaults
create service-account (SA) tokens — request this via a Servicely ticket
apply existing group grants and create the site Infra Ops group

Bootstrap also needs:

the 1Password CLI (op) at 2.34.0 or newer
a HashiCorp Vault policy with KV-v2 create/update/read on fleet-manager/data/{env}/{deployment}/* and list/read on fleet-manager/metadata/{env}/{deployment}/* — request it from the Vault admins. Vault cluster-admin rights are not needed; the KV mount, AppRole auth method, and ExternalSecret store are platform-managed.

The deployment vaults do not exist before the first bootstrap. After bootstrap creates them, it may exit with code 2 while creating the service account. Enable Service Account Access on the runtime, operators, and networking vaults (Manage → Service Account Access), then follow the lower-level retry printed by the CLI. The service account reads the site’s local networking vault for switch-workflow credentials; it never receives access to the shared source vault.

Set Up the Site Infra Ops Group

Site operators use the site’s own 1Password group, named after the site tag — ICE2 Infra Ops, GSO1 Infra Ops. Agree its membership in the Slack deployment group before bootstrap. The config-driven bootstrap then:

resolves the site tag from the deployment slug’s site segment (sys1-ice2 → ICE2), asking you to confirm it. Pass --site-tag to set it outright when the slug and site tag differ
creates {site} Infra Ops if it is missing
assigns the configured group managers, plus anyone passed with --add-infra-ops-group-manager — use this for the site manager
grants the group view/copy/history/create/edit/export/share/print access to Fleet Manager {deployment}-operators {env} — item edits let site operators reset BMC passwords in place. Import, archive, delete, and move are not granted

If the signed-in operator cannot look up or create the group, bootstrap skips only that group and still applies every other configured grant. Adding other members is still manual; use a Servicely ticket for the agreed operator list.

Useful Contacts

Who to talk to for the site dependencies:

Topic	Team	Contact
NetBox, OpenStack	Infra Eng	Tom Clark, John Garbutt
IPMI	Infra Eng	Tom Clark or Grant Delaney
Kubernetes cluster	Cloud Native Platform Reliability Engineering	Drew Hudson-Viles, Joe Derbyshire
SLURM	HPC	Antony Cleave, James Vaughn

Kubernetes Cluster

The target Kubernetes cluster is a prerequisite for the FM satellite (control plane) work for a worked example, see the LON14 provisioning PR.

Establish the Site Identity and Platform

Start by confirming the basic deployment shape:

the site has an agreed identifier such as glo1 or sta1
the target Kubernetes clusters exist for the required environments
the site’s devices already exist in NetBox
the DHCP model for the site is known — externally managed by default (see DHCP)

Kubernetes clusters normally follow this pattern:

fleet-mgr-{ENV}-{DEPLOYMENT}

Examples:

fleet-mgr-dev-glo1
fleet-mgr-prod-sta1

Before a deployment can go live, the site also needs:

networking for the target Kubernetes clusters
any SLURM-side VM provisioning needed for burn-in workflows
coordination with the teams responsible for NetBox, Kubernetes, and network access

Run the Bootstrap Automation First

For the Vault/ExternalSecret happy path, use the standup CLI before creating 1Password, Vault, or OpenStack project-prep resources by hand:

uv run fleetctl deployment-standup automation bootstrap {deployment} {env} \
  --project-prep-config ./project-prep.yaml

Bootstrap reuses your active 1Password and HashiCorp Vault sessions, so authenticate first:

op signin
export VAULT_ADDR=https://vault.nscale.com
vault login -method=oidc

For the full command reference — helper commands, retries, local rehearsal, and auth details — see the operator reference at src/fleet_manager/cli/commands/deployment_standup/README.md. This page keeps the happy path; that README covers the low-level deployment-standup subcommands. Pass --project-prep-config only when the non-secret OpenStack project-prep file is ready. Without it, the OpenStack slice is skipped and the 1Password/Vault path still runs. Prod uses Fleet Manager shared prod as its default source for nscale-api-key, mimir-login, netbox-login, and pki-ca. It copies svc-switch-login from Fleet Manager shared-networking prod into the site’s networking vault. The operator needs read access to both shared source vaults; workers receive only their deployment-local copies. Stg has no default runtime-credential source vault: a default stg bootstrap creates only pki-ca from pki-ca-stg and skips all three runtime items as a group. Before deploying stg, run the lower-level 1Password slice with an approved source vault, then finish the OpenStack and Vault slices described in the operator reference:

uv run fleetctl deployment-standup onepassword bootstrap {deployment} stg \
  --runtime-credentials-source-vault "<approved-source-vault>"

The source uses netbox-login-stg and pki-ca-stg; the deployment items remain named netbox-login and pki-ca. The matching agents fail to start until the runtime items exist. automation bootstrap handles:

1Password runtime, operators, and networking vault creation or reuse
configured 1Password group grants
shared runtime item copies for nscale-api-key, mimir-login, and the configured NetBox source item when a source vault is configured
the shared-source svc-switch-login item copied into the site’s networking vault
deployment pki-ca generation from the configured source CA (pki-ca-stg for stg)
1Password service-account token creation and storage
optional OpenStack project prep when --project-prep-config is provided
transitional HashiCorp Vault runtime secret bootstrap for ExternalSecrets

The Vault slice is not a no-op: it attempts to write the satellite runtime Vault footprint and requires Vault access. There is no non-Vault route — Vault and the platform ExternalSecret are how runtime secrets reach the cluster, and no deployment-standup command writes Kubernetes Secrets. A site without Vault access cannot finish standup; request the KV-v2 policy in Prerequisites and Access first. Use the lower-level deployment-standup commands for retries, partial recovery, or non-default backend options. If bootstrap exits with code 2, stop at that slice and follow the printed handoff and retry command. The orchestrator has no --resume option; do not add lower-level options to it.

Items to Copy or Create Manually After Bootstrap

Bootstrap creates the secrets and env vars needed to stand up the FM cluster without an OpenStack dependency. The following items are not created by bootstrap:

slurm-api-prod — see Store the SLURM JWT in 1Password.
ipmi-exporter — create manually. Add to the operators vault Fleet Manager {deployment}-operators {env}. See Add IPMI Exporter Credentials.

Create the Secrets Footprint

The bootstrap command creates most of this footprint. Use this section as the target-state checklist and for the pieces that still need an external owner or manual setup.

Create the Fleet Manager Vaults

Handled by automation bootstrap. For each target environment it creates or reuses:

Fleet Manager {deployment} {env} for runtime secrets
Fleet Manager {deployment}-operators {env} for operator-managed secrets such as BMC and IPMI credentials
Fleet Manager {deployment}-networking {env} for site network engineering credentials

Real switch-VRF or NVLink partition writes use Fleet Manager {deployment}-networking {env}. Bootstrap creates that vault and its group access; separately grant the satellite service account Read and add only the service accounts needed by enabled real fabric operations. Use this split:

runtime vault: NetBox token, OpenStack/clouds config, SLURM/API tokens
operators vault: *-bmc, ipmi-exporter, and similar operator-managed credentials
networking vault: site-local network gear credentials, owned by network engineering. Switch workflows read the local svc-switch-login copy when real switch writes are enabled; they never read the shared source vault directly

To add the networking vault to a site that already exists, scope a bootstrap run to that role:

uv run fleetctl deployment-standup onepassword bootstrap {deployment} {env} --only networking

It creates the vault, applies the configured group access (Fleet Manager Admins, Fleet Manager Squad, FOE), copies the shared switch Login, and verifies the existing service-account token can read all three local vaults. It does not create or rotate the token, or run runtime-item or pki-ca work. For a deployment whose service account predates networking-vault access, an account Owner must first add Read access to Fleet Manager {deployment}-networking {env} to the existing service account. The focused command intentionally verifies that grant rather than rotating the existing token. Non-production runs still create the empty networking vault and grant token access, but copy a switch Login only when given --networking-credentials-source-vault.

Generate Service Account Tokens

Handled by automation bootstrap. It generates one service-account token per environment with access to all three deployment-local vaults:

Fleet Manager {deployment} {env}
Fleet Manager {deployment}-operators {env}
Fleet Manager {deployment}-networking {env}

Recommended permissions:

runtime vault: Read
operators vault: Read + Write
networking vault: Read

Store the token in the runtime vault as an API Credentials item named:

Service Account Auth Token: Fleet Manager {deployment} {env}

Satellite runtime config should set:

OP_VAULT_SELECTOR={deployment}

At deploy time the satellite reads this token from the Kubernetes Secret fleet-manager-satellite-op-token (namespace fleet-manager, key op-token) — every agent resolves its 1Password items through it. Vault/ExternalSecret delivers that Secret from the footprint bootstrap writes, so nothing is created on the cluster by hand. The standup CLI does not write Kubernetes Secrets; if that Secret is missing, fix the Vault value and the ExternalSecret rather than applying it manually.

Create the NetBox API Token

For prod, bootstrap expects netbox-login in the configured shared source vault and copies it into the deployment runtime vault. For stg, the configured source item is netbox-login-stg, and the target runtime-vault item still stays named netbox-login. Create a deployment-specific NetBox token only when the NetBox owner asks for one, or when the shared source item has not been prepared yet. Repeat the following for each target environment:

Access NetBox and log in.
Open Profile → API Tokens → Add a Token.
Create a token with:
- description: Fleet Manager {deployment} {env} API Access
- Write enabled: checked
- expiry: 31 December 2030
- allowed IPs: the Kubernetes cluster NAT gateway IP or CIDR

The token needs write access because Fleet Manager updates device status and tags during enrolment workflows. Bootstrap normally copies the same netbox-login token value into each deployment runtime vault, so hub and satellite services commonly use one shared token. Its NetBox Write enabled flag permits every POST, PATCH, PUT, and DELETE that the token owner’s Object Permissions authorise. Fleet Manager has no per-operation write switch. Grant that token the union of the Object Permissions below, and scope them to the deployment’s sites wherever the NetBox owner can safely do so.

Required NetBox Object Permissions

Fleet Manager reads the following resources while synchronising inventory and carrying out workflow preflight. The consumer column identifies which services need each view permission; it does not imply separate tokens:

Consumer	NetBox objects	Why Fleet Manager reads them
Hub and satellite	Sites (`dcim.site`)	Resolve site configuration and materialise sites in Fleet Manager
Hub agents	Locations (`dcim.location`), Racks (`dcim.rack`), Device Types (`dcim.devicetype`), Device Roles (`dcim.devicerole`)	Materialise the inventory hierarchy in Fleet Manager
Hub and satellite	Devices (`dcim.device`), Interfaces (`dcim.interface`), Cables (`dcim.cable`)	Sync devices and topology; resolve workflow targets and cabling
Hub agents	MAC Addresses (`dcim.macaddress`)	Requeue a device within a minute of an edit to one of its MAC addresses, which NetBox 4.2+ keeps apart from the interface. Without it the netbox agent’s change poll logs an HTTP 403 every minute, and MAC address edits wait for the resync. A constraint that leaves out a synced site hides that site’s MAC addresses without an error
Hub agents	Modules (`dcim.module`)	Build the per-component FRU data projection
Hub agents	Console Ports (`dcim.consoleport`), Console Server Ports (`dcim.consoleserverport`)	Project console links into inventory
Hub and satellite	IP Addresses (`ipam.ipaddress`)	Resolve management and network configuration
Hub agents	Prefixes (`ipam.prefix`)	Resolve network configuration for inventory projection
Hub and satellite	Tags (`extras.tag`)	Resolve the pre-existing tag before attaching it to a device
Satellite worker and operators, at a site with a RoCE back end	Interface peer mappings (`netbox_datacenter.inventoryinfointerfacepeermapping`, from the `netbox_datacenter` plugin)	Resolve which back-end leaf sub-port each host PF reaches through shuffle cassettes (`fleetctl netbox be-port-plan`, `be-switch-state`, and the back-end isolate and return workflows). Without it they fail with HTTP 403. In production NetBox it is part of the `Read-only` Object Permission, which the `fleetmanager` user already holds (confirmed by the plugin owner, 2026-10-08)

The following writes are conditional on the corresponding Fleet Manager operation, but must be included when that operation is enabled:

NetBox objects	Object Permissions	Fleet Manager operation
Devices (`dcim.device`)	`view`, `change`	Enrolment sets `staged` → `active`, and, once the site’s `platform_class_activity_ready` gate is on, records `custom_fields.platform_class`; `fleetctl netbox backfill-platform-class --apply` writes the same field; enrolment/BMC configuration attach existing tags; unenrolment restores `staged`; node swap moves rack locations, and a cableless swap sets the failed tray `offline`
Interfaces (`dcim.interface`)	`view`, `change`	`fleetctl netbox populate-ib-guids --apply` writes an InfiniBand port GUID to an interface label; node swap sets a never-enrolled spare’s frontend interfaces’ primary MAC
MAC Addresses (`dcim.macaddress`)	`view`, `add`	Node swap fills a never-enrolled GB300 spare’s missing frontend MACs from its BMC: it looks each MAC up across NetBox, refusing one recorded elsewhere, and creates the missing ones. Without these grants the swap fails at that step with HTTP 403 after configuring the spare’s BMC, having at most created a MAC object a rerun reuses, and before any write to Fleet Manager, Nova or Ironic
Cables (`dcim.cable`)	`view`, `add`, `delete`	GB300 tray replacement deletes and recreates cables while swapping terminations

Fleet Manager does not create or delete Devices, Interfaces, IP Addresses, Prefixes, Sites, Locations, Racks, Device Types, Device Roles, Console Ports, or Console Server Ports. Do not grant those permissions merely for Fleet Manager. Before rollout, the NetBox owner should validate the filters against the deployed NetBox version and run the token check. check-auth proves the token can read Sites from the operator’s source IP; a production token restricted to the cluster NAT still needs a cluster-side workflow or agent read to prove its allowed-IP rule:

uv run fleetctl deployment-standup netbox check-auth {deployment} {env}

Store the NetBox Token in 1Password

If bootstrap is copying the shared source item, keep the source item named netbox-login (netbox-login-stg for stg) and let the CLI copy it. If a manual per-deployment token is required, store a Login item called netbox-login in the runtime vault:

username: fleetmanager
website: https://netbox-{env}.mgt.{deployment}.nscale.com
auth-token: the token created in NetBox

Prefer the standup helper so the item shape stays consistent: Set NETBOX_STANDUP_TOKEN to the token from the previous step, or omit it and enter the token at the hidden prompt.

uv run fleetctl deployment-standup netbox store-token {deployment} {env} \
  --netbox-url "https://netbox-{env}.mgt.{deployment}.nscale.com"

Generate and Store the Site PKI Material

Handled by automation bootstrap when the configured source vault contains pki-ca. Use PKI & Certificate Trust for source CA setup, rotation, and manual fallback steps.

DHCP

DHCP is managed externally at current sites — Fleet Manager does not own DHCP configuration, so leave the enrolment step off:

fleetManager:
  satellite:
    temporalWorker:
      enrollmentSteps:
        configureDhcp: false

If a site instead relies on Fleet Manager-managed dnsmasq, keep configureDhcp enabled and follow DHCP server - required setup. If unsure, state your assumption in #squad-fleet-manager at go-time.

Add Runtime Service Credentials

These runtime-vault items are required — the matching agents crash on startup if they are missing. Bootstrap creates pki-ca from the configured source CA. When a runtime-credential source vault is configured, it also copies nscale-api-key, mimir-login, and the configured NetBox source item. If that copy was skipped, create the three runtime items with the standalone commands or rerun the lower-level onepassword bootstrap slice with an explicit source vault before deploying. For stg, the source item titles are netbox-login-stg and pki-ca-stg; the deployment item titles remain netbox-login and pki-ca.

1Password item	Used by	Required fields
`mimir-login`	Health-status agent (Mimir queries)	`username`, `password`
`nscale-api-key`	Diagnostics agent (LLM calls)	`credential`

One runtime item is optional. Put a Slack incoming-webhook URL in the credential field of fm-funny-job-names-slack-webhook and the diagnostics agent posts that channel every new job whose generated name the LLM finds funny. Without the item those alerts are off; the agent reads it at startup, so restart the agent after adding or removing it. The Workflow agent’s S3 credentials can be provided two ways, both supported today:

Manual object-storage item. Create an object-storage 1Password item in the runtime vault with access_key_id and secret_access_key (the satellite’s own S3 credentials, not a shared pair).
Controller-provisioned bucket (intended direction). Use the bucket-controller secret workflow-agent-object-storage-credentials (see Provision S3 Buckets).

Provision S3 Buckets

Fleet Manager needs two S3 buckets, provisioned by the s3.nscale.com controller (deployed via k8s-deploy-core). Enable the bucket manifests only once the controller is confirmed synced on the cluster. Add them to the deployment’s buckets.yaml — see this PR for an example:

Temporal Postgres backups — namespace temporal, secret temporal-postgres-s3-credentials.
Workflow report/artifact uploads — namespace fleet-manager, secret workflow-agent-object-storage-credentials (the Workflow agent’s runtime S3 credentials).

Key points:

Keep spec.name short. The controller always prepends cluster-env-region to spec.name, so do not pre-write the prefix or it doubles — for a production fleet-mgr template, postgres-temporal resolves to fleet-mgr-prod-mgt-sys1-lon14-postgres-temporal. metadata.name must also stay ≤ 63 characters, since access-key resource names are built from cluster + namespace + Bucket name.
credentialsOutput creates the named secret with keys AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, BUCKET_NAME, and S3_ENDPOINT. For controller-provisioned buckets, set fleetManager.satellite.workflowAgent.objectStorage.enableOperatorSecret: true in the deployment’s fleet-manager.yaml; the satellite chart then reads the credentials, bucket name, and endpoint from workflow-agent-object-storage-credentials. Leave the flag false for the manual object-storage 1Password flow. The umbrella does not forward manual bucket/endpoint overrides when the flag is false, so that path uses the inner chart’s environment defaults; add a pass-through before rollout if the site needs custom values.
deletionPolicy: Retain is intentional — Temporal Postgres backups must survive removal of the Bucket CR.

The SLURM JWT (slurm-api-prod) is created later under Store the SLURM JWT in 1Password.

Add IPMI Exporter Credentials

Store the deployment IPMI credentials in the operators vault as a Login item named ipmi-exporter. Example:

DEPLOYMENT="{deployment}"
ENV="{env}"
VAULT="Fleet Manager ${DEPLOYMENT}-operators ${ENV}"
IPMI_PASSWORD="[PROVIDED BY OPERATIONS TEAM]"

op item create \
  --category=login \
  --title="ipmi-exporter" \
  --vault="${VAULT}" \
  username=ipmi_exporter \
  "password=${IPMI_PASSWORD}"

automation bootstrap also runs the transitional HashiCorp Vault bootstrap used by ExternalSecrets. Only wire Vault paths manually when the chart paths differ from the standup convention or bootstrap prints a manual handoff.

Register Redfish Credentials in the Observability Chart

If the cluster uses Redfish hardware monitoring, wire its credentials in the k8s-deploy-observability repo as an ExternalSecret, under charts/observability/templates/otel/agent/secrets/redfish-secrets.yaml. This is separate from the ipmi-exporter item above — it feeds the Prometheus Redfish exporter env vars from Vault so it can authenticate against the cluster’s BMC interfaces. Copy an existing region’s ExternalSecret in redfish-secrets.yaml and adjust it for the new region — that file is the source of truth for the manifest shape, so follow it rather than a copy reproduced here. When adjusting it, the Fleet Manager-specific values are:

gate the resource on the target cluster and region: clusterName: fleet-mgr, region: <region>
Vault key path <region>/{{ .Values.environment }}/ipmi, properties username / password (e.g. sys1-ice2/prod/ipmi)
templated env vars in UPPER_SNAKE region form — SYS1_ICE2_REDFISH_EXPORTER_USERNAME / ..._PASSWORD — to match the exporter’s credential lookup

Prepare NetBox and Config Context

Validate the Source Data

Before rollout:

devices for the site should already exist in NetBox
devices intended for Fleet Manager ingestion should be marked as staging
required deployment-specific metadata should already be present

Run the validation script before proceeding:

export NETBOX_URL=https://netbox-dev.nscale.teleport.sh
export ENVIRONMENT=dev
export OP_VAULT_SELECTOR=glo1
export OP_SERVICE_ACCOUNT_TOKEN=...   # 1Password service-account token for the selected vault

uv run python scripts/checks/netbox.py

Use dev NetBox for pre-rollout validation and prod NetBox before deploying to prod. ENVIRONMENT, OP_VAULT_SELECTOR, and OP_SERVICE_ACCOUNT_TOKEN are only required here to satisfy the script’s runtime settings (it loads the same settings as the services and resolves the NetBox token from 1Password when one is not provided); use valid local values for your environment rather than treating dev and glo1 as deployment-specific rollout inputs.

Create the NetBox Config Context

Config context is the source of truth for hardware settings Fleet Manager expects to enforce. At minimum, prepare the deployment’s expected:

NIC configuration
firmware versions
BIOS settings

For NIC configuration:

Work with the datacenter and networking teams to identify the required settings.
Format the expected values into the config-context JSON structure.
Create a Config Context in NetBox under Provisioning → Config Contexts.
Scope it narrowly to the correct deployment and device type.
Save it and verify it appears on the targeted devices.

A typical structure looks like:

{
  "mlxconfig_config": {
    "port1": {},
    "port2": {}
  },
  "broadcom_config": {
    "nic_settings": {}
  },
  "bios_config": {},
  "firmware_config": {}
}

Keep the scope tight enough to avoid accidentally applying the same config context to the wrong site or hardware family.

Create the NetBox Cluster

Create a cluster for the deployment under Virtualization → Clusters. The NetBox agent uses this cluster to filter and ingest only the devices belonging to this deployment.

Configure OpenStack and IPA

Dependency: the OpenStack project and application credentials are provisioned by the OpenStack team (see Useful Contacts).

This phase depends on OpenStack being ready. Store only the credentials for integrations the site actually uses in the deployment runtime vault:

openstack-overcloud for every site
openstack-undercloud when the site has CPU_HYPERVISOR devices
openstack-slurm when SLURM adoption is configured

The k8s-deploy-fleet-manager umbrella takes the Ironic enrolment project by name; Fleet Manager resolves it to a Keystone project ID at runtime. Get the name from the OpenStack owner. If there is no undercloud, leave it disabled:

fleetManager:
  satellite:
    openstack:
      undercloudsEnabled: false
      placementEnabled: true # set false on standalone-Ironic sites with no Placement service
      ironic:
        enrolTargetProjectName: "[ENROL_TARGET_PROJECT_NAME]"

Prerequisite (standalone-Ironic sites) — automated disk cleaning must be on. Fleet Manager’s teardown (UnimageNodeWorkflow) drives Ironic’s deleted action and waits for the node to reach AVAILABLE, but the disk, config-drive, and injected-key wipe is Ironic cleaning’s work on that transition — FM does not force the erase. If the conductor has automated cleaning disabled or no disk-erase clean step, a torn-down node returns to the enrol pool with the previous tenant’s image and data intact while teardown still reports success. Confirm the conductor runs with [conductor]automated_clean=True and a disk-erase step in the site’s clean_steps before enabling the Ironic-direct IMAGE_NODE path.

The same umbrella currently requires real SLURM project, network, and REST endpoint values. A site with no SLURM cannot be configured safely by inserting placeholders; updating the umbrella chart to expose the inner chart’s no-SLURM mode is a rollout blocker. See Connect the Site to SLURM.

Project and Application Credentials (provisioned by the OpenStack team)

The Keystone project, the Fleet Manager user role, quotas, and the application credentials are created by the OpenStack provisioning team. Share OpenStack Application Credentials with them as the spec for the access rules Fleet Manager needs (baremetal / placement / compute) and the clouds.yaml shape. The finished credentials land in 1Password as the applicable items listed above. If FM engineers need to run project prep themselves, pass a reviewed non-secret project-prep config to bootstrap — it creates or reuses the Keystone project, grants the Fleet Manager user role, and sets the managed Nova, Cinder, and Neutron quotas. Start from the project-prep example, copy it to a local project-prep.yaml, and keep secrets out of the file. Use the lower-level deployment-standup openstack project-prep command only for retries or non-default credential sources.

Dual-Ironic Deployments

Some deployments (e.g., GB300) run two separate Ironic instances — overcloud and undercloud — each managing distinct hardware. Fleet Manager routes devices to the correct Ironic instance based on the device_class field on FM DeviceRoles, which is populated by the NetBox agent via NETBOX_AGENT_DEVICE_ROLE_CLASS_MAPPING. Routing is automatic — no Helm flag or list of slugs is needed. The OpenStack agent queries FM at startup to discover which device roles are in use at its configured site and starts one reflector per active instance:

Overcloud — GPU_COMPUTE and CPU_BAREMETAL device classes
Undercloud — CPU_HYPERVISOR device class

If no site devices map to an instance (e.g. no CPU_HYPERVISOR hardware at that site), that instance is skipped and no credential lookup is attempted.

1. Add 1Password credentials

Add an openstack-overcloud document to 1Password containing the overcloud clouds.yaml (see OpenStack Application Credentials).
If the deployment has CPU_HYPERVISOR nodes, also add an openstack-undercloud document for the undercloud.

2. Use the standard 1Password document names

The inner satellite chart supports these overrides:

Helm value	Default	Description
`openstack.overcloudsOpItem`	`openstack-overcloud`	1Password doc for overcloud
`openstack.undercloudsOpItem`	`openstack-undercloud`	1Password doc for undercloud

The current k8s-deploy-fleet-manager umbrella does not forward these values. New deployments must use the defaults; if different item names are required, add the umbrella pass-through before rollout.

3. Local development

For local dev, point directly at clouds.yaml files instead of 1Password:

openstack:
  undercloudsFile: "/path/to/undercloud-clouds.yaml"
  overcloudsFile: "/path/to/overcloud-clouds.yaml"

Or via environment variables: OPENSTACK_UNDERCLOUD_CLOUDS_FILE / OPENSTACK_OVERCLOUD_CLOUDS_FILE.

Verification

OpenStack agent logs show Starting reflector for Ironic instance 'overcloud' (and 'undercloud' if CPU_HYPERVISOR nodes are present).
Each reflector’s store is named ironic-nodes-overcloud / ironic-nodes-undercloud in metrics.
Nodes from each Ironic instance reconcile to FM devices with the correct roles.

Verify or Upload the IPA Images

Make sure the required IPA images exist in the target OpenStack environment. Common image families:

Architecture	GPU Support	Image Type	Required?
x86_64	NVIDIA	`ipa-nvidia`	For B200 deployments
aarch64	NVIDIA	`ipa-nvidia-arm`	For GB200 or GB300 deployments

Check what is already present:

openstack image list --name "ipa-*"

If you need to upload the images manually:

openstack image create \
  --disk-format aki \
  --container-format aki \
  --public \
  --file [KERNEL_FILE] \
  ipa-[VARIANT]-kernel-[VERSION]

openstack image create \
  --disk-format ari \
  --container-format ari \
  --public \
  --file [RAMDISK_FILE] \
  ipa-[VARIANT]-ramdisk-[VERSION]

Record the final UUIDs:

{deployment} IPA Images:
[VARIANT]:
  Kernel UUID: [TO BE FILLED]
  Ramdisk UUID: [TO BE FILLED]

Then wire them into Fleet Manager:

set fleetManager.satellite.openstack.ironic.deployKernel
set fleetManager.satellite.openstack.ironic.deployRamdisk

Create the Cluster Deployment Configuration

Once the Kubernetes cluster is provisioned and bootstrap has run, create the deployment from the maintained satellite template in k8s-cluster-fleet-manager. See this PR for a worked example. Create the deployment-specific configuration in k8s-cluster-fleet-manager:

git clone git@github.com:nscaledev/k8s-cluster-fleet-manager.git
cd k8s-cluster-fleet-manager
cp -r _templates/satellite {deployment}

Replace every placeholder in the copied production files. Values such as secretPrefix, Ceph IDs/monitors/pools, Velero buckets, and the Temporal/Loki object-storage targets are owned by the platform or storage teams; do not copy them from a live deployment. For a production satellite, update the files under {deployment}/production/, including:

observability.yaml
infra.yaml
core.yaml
fleet-manager.yaml

Typical deployment-specific changes include:

env- and deployment-specific paths
Ironic IPA image UUIDs
targetRevision for k8s-deploy-fleet-manager

The NetBox allowlist is hub configuration, not satellite configuration. Add the deployment’s cluster to fleetManager.hub.netboxAgent.deviceFilters in glo1/production/fleet-manager.yaml. If the site introduces a new NetBox device-role slug, also update fleetManager.hub.netboxAgent.deviceRoleClassMapping; device class controls Ironic routing and workflow eligibility. See the filter example and the role-mapping example. To configure the Topology agent, see Configure the Topology Service below — a new region needs topology-service dev work and NetBox readiness before it can be enabled in the manifests. See this PR for an example. Bootstrap prepares the 1Password and Vault footprint; k8s-cluster-fleet-manager still owns the namespace and chart wiring. If namespace or ExternalSecret generation is not yet automated there, create the namespace explicitly and wire the deployment to the expected secret paths before opening the pull request.

Set the Satellite Identity and Endpoints

Start from the maintained template and replace every placeholder. These identity and endpoint values are easy to misplace because operators configure the umbrella chart, not the inner satellite chart:

Helm value	What it is
`environment`	Deployment environment, normally `prod` for a new production satellite.
`satelliteName`	The deployment slug (e.g. `sys1-ice2`). Also the default 1Password vault selector, and the agents’ OAuth2 client IDs are templated from it (`workflow-agent-{satelliteName}`, …).
`siteSlug`	The Fleet Manager/NetBox site slug (e.g. `keflavik`). Confirm it from NetBox site data — do not derive it from `satelliteName`; they usually differ.
`fleetManager.netboxUrl`	The NetBox instance for this environment.
`fleetManager.satellite.hubUrl`	The hub Fleet Manager API URL the agents call.
`fleetManager.satellite.workflowAgent.logsExplorerUrl`	The Grafana Explore URL linked from workflow failures.
`fleetManager.satellite.healthStatusAgent.mimir.baseUrl`	The Mimir endpoint; required when the health-status agent is enabled.

The umbrella generates the inner temporal.host; it is not a site input. OpenStack and SLURM render requirements are covered in their sections below.

Enable the Hub to Accept the Satellite

The hub must be configured to accept the new satellite before it can register: add the new satelliteName to fleetManager.hub.expectedSatellites in the hub’s fleet-manager.yaml (under glo1/production for prod). The hub pre-creates one OAuth2 client per agent per entry (ironic-agent-{satellite} — the OpenStack agent’s unchanged wire identity, ADR-0036 — slurm-agent-{satellite}, workflow-agent-{satellite}, …), and the satellite agents authenticate with client IDs templated from satelliteName — so the entry must match it exactly, or every agent fails hub authentication. This is a separate change from the satellite’s own configuration and must be merged and synced before the satellite comes up. See this PR for an example.

Configure the Topology Service

The Topology service builds the NVLink/network topology graph for a region from NetBox data. A new region is not enabled by config alone — it needs development work in the topology-service repo first, then values changes in k8s-cluster-fleet-manager, and it depends on the region’s NetBox data being ready.

Do the topology-service dev work first

Registering a new region requires code and chart changes in topology-service first — follow that repo’s NEW_REGION_ONBOARDING.md for the full change-set (it owns that contract). Until the change lands and ships, the region cannot be enabled in the deploy manifests.

Check NetBox config readiness

The graph is only as good as the NetBox data. Before enabling a region, confirm in the satellite’s NetBox that:

the expected device roles exist (e.g. compute-tray, be-spine-switch, be-leaf-switch)
devices are in the expected statuses (e.g. active) and cabling is connected
you have the region’s NetBox site ID(s), cluster ID(s), and GraphQL URL — these are per-NetBox-instance, so pull them from the satellite’s NetBox rather than reusing another region’s values

Enable the topology agent

Once the region is registered in topology-service and NetBox is ready, add the region’s UNI name and FM site slug to fleetManager.hub.topologyAgent.targets in the hub’s glo1/production/fleet-manager.yaml. For a worked example, see the glo1 config — topologyAgent block and targets The API reads the same list to find a site’s Region: the topology-sync eligibility gate checks that Region before multi-node burn-ins, and POST /api/v1/sites/{id}/topology-sync/, which a cableless node swap and an unenrolment of an active tray use to declare the tray’s removal, answers 422 for a site missing from it. The hub chart’s api.extraTopologyTargets adds sites to that list for the API alone, which maps a Region without starting the topology agent’s ingestion for it; k8s-deploy-fleet-manager has to forward that value before a cluster repo can set it.

Configure Cross-Site Hardware Monitoring

When a cluster’s hardware is scraped by an exporter running at a different site — for example, sys1-ice2 hosts are scraped from a glo1 exporter — add a Prometheus Operator ScrapeConfig for that cluster in the k8s-deploy-observability repo, under charts/observability/templates/otel/agent/scrapeconfigs/<region>/. Copy an existing region’s files (e.g. sys1-ice2/infra-redfish-hgx.yaml for GB300/HGX hardware, plus the infra-redfish-exporter.yaml, infra-redfish-telemetry.yaml, and infra-netbox-ipmi.yaml siblings) and adjust them for the new region. Each file is a ScrapeConfig gated to the target cluster and region. Follow the reference files in k8s-deploy-observability for the manifest shape rather than a copy reproduced here; the Fleet Manager-specific points to get right (these replace the old disableSiteFilter values-list pattern) are:

Scope by &cluster={{ .Values.region }} on the NetBox prometheus-sd URL. This is what targets a remote cluster’s devices — there is no disableSiteFilter flag anymore. device_type selects the hardware class (GB300/HGX uses hg635n-v2; adjust for other families).
__param_site is set by a relabeling to {{ .Values.region | replace "-" "_" }}, so the underscores (sys1_ice2) are produced by the template rather than hand-written. This must still match the SYS1_ICE2_* credential env var naming, or authentication fails in ways that are hard to trace back to the config.

Connect the Site to SLURM

The inner Fleet Manager chart can run without SLURM. Enable the OpenStack integration explicitly with:

fleetManager.satellite.openstack.slurm.enabled

When that flag is true, configure the SLURM OpenStack project and provider network, as well as fleetManager.satellite.temporalWorker.slurm.baseUrl. These must be real values. A dummy URL starts SLURM clients and causes runtime failures. If the site has no SLURM, set the flag to false; the project and network values are ignored by Fleet Manager. The flag means “SLURM reached through OpenStack (Nova)”, not “this site has SLURM”. A standalone-Ironic site with no Nova keeps it false even if it has a SLURM cluster. With the flag false, the Release workflow’s SLURM step builds no Nova, Placement or SLURM client and does not read the openstack-slurm document: it only checks that the Ironic node exists, then continues to the hand-back.

Check the Prerequisites

For SLURM adoption, the site needs:

a SLURM project in the site’s OpenStack with an application credential — see OpenStack Application Credentials for the access rules and the openstack-slurm document. If the project does not exist yet, prepare it with automation bootstrap --project-prep-config.
a Nova keypair in the SLURM project named {deployment}-keypair (created by the tofu bootstrap; the slurm-adopt activity injects this keypair into the Nova VM as cloud-user)
the provider network the SLURM nodes attach to
a published ARM64 (or x86_64) image for the SLURM compute VMs
a Nova flavor for each device class the site adopts (e.g. GB300 → g.144.gb300.4)

The current umbrella also needs a SLURM controller, REST endpoint, and JWT so its agents can start. Burn-in additionally needs:

the expected GPU partition on the cluster
any required S3 credentials for artifact or benchmark flows
the hardware-test container images staged where the SLURM compute nodes can read them — if none are published yet for the site’s CPU architecture, building them needs a cluster node with Enroot, in practice a GPU node; see Stage the Hardware-Test Container Images

Configure OpenStack SLURM Adoption

These values feed the slurm-adopt workflow that creates the Nova VM on each Ironic node. Set enabled: true for a site with the SLURM OpenStack integration and provide the project name and provider network. The per-role image, flavor, GRES, and user-data values live in Hub SlurmBootstrapConfig rows keyed by site and DeviceRole; they are not Helm values or environment variables. SLURM enrolment is keyed on DeviceRole.slug rather than resource_class because resource_class does not distinguish CPU architectures (x86 vs ARM nodes share the same CPU_BAREMETAL device class), and SLURM compute images differ per arch — see ADR-0025 in the repo for the full rationale. Set them in the deployment’s fleet-manager.yaml:

fleetManager:
  satellite:
    openstack:
      slurm:
        enabled: true
        targetProjectName: "[SLURM_PROJECT_NAME]"
        network: "[PROVIDER_NETWORK_UUID]"
        keyName: "{deployment}-keypair"

Create the corresponding Hub row through the /api/v1/sites/{site_id}/slurm-bootstrap-configs/ API. Use the exact DeviceRole UUID for the role and provide the Nova image, flavor, optional GRES, and optional base64-encoded cloud-init user-data in that row. If you have the cloud-user SSH private key (used for ops to log in to SLURM compute VMs), also store it in the runtime vault as an SSH Key item — Fleet Manager does not consume it at runtime, but keeping it co-located with the rest of the site’s secrets keeps recovery simple.

Generate the SLURM JWT

The current umbrella requires the SLURM REST endpoint, and a working client also needs its JWT. The JWT signing key (/etc/slurm/jwt_hs256.key and the matching AuthAltTypes=auth/jwt config in slurm.conf) is provisioned automatically by the Fleet Manager SLURM appliance. If the site does not use the appliance, follow the upstream JWT setup on the controller before continuing. SSH to the SLURM login node as hpcadmin and create a long-lived token. lifespan is in seconds; the default of 1800 is too short for unattended Fleet Manager use, so set something effectively non-expiring:

sudo scontrol token username=hpcadmin lifespan=315360000  # 10 years

Tokens cannot be listed or revoked individually — to invalidate all outstanding tokens, rotate /etc/slurm/jwt_hs256.key on the controller and reissue.

Store the SLURM JWT in 1Password

Create a Login item called slurm-api-prod in the runtime vault — this is the name the worker’s jwtOpItem points at across sites.

DEPLOYMENT="{deployment}"
ENV="{env}"
VAULT="Fleet Manager ${DEPLOYMENT} ${ENV}"
SLURM_JWT="[PASTE TOKEN FROM THE PREVIOUS STEP]"

op item create \
  --category=login \
  --title="slurm-api-prod" \
  --vault="${VAULT}" \
  "jwt[password]=${SLURM_JWT}" \
  "notes=JWT token for SLURM REST API authentication"

Configure the Temporal Worker

Point the worker at the SLURM REST endpoint and the JWT item created above. The endpoint is the SLURM login node’s reachable address (typically a floating IP) on the SLURM REST port — use https:// if the controller terminates TLS, otherwise plain http://.

fleetManager:
  satellite:
    temporalWorker:
      slurm:
        baseUrl: "https://[SLURM_LOGIN_FLOATING_IP]:[SLURM_REST_PORT]"
        jwtOpItem: "slurm-api-prod"
        # Optional: deviceNamePrefix and apiVersion overrides.

Stage the Hardware-Test Container Images

This step is only needed for burn-in. Burn-in tests use Enroot squashfs (.sqsh) images. Check the published S3 artifacts for the site’s CPU architecture first; build only an artifact that is missing or changed. images/build.sh builds hardware-tests-default, gpu-fryer, and nvloom; other images follow their separate export path. Building requires a target-architecture host with Enroot, normally a GPU node. Follow the hardware-test image build runbook for the build, publication, image-to-setting map, and troubleshooting. The current k8s-deploy-fleet-manager umbrella forwards local *ContainerImage destinations but does not forward the inner chart’s imageArch or imageSources. Therefore, stage each .sqsh manually today. The build cache and the configured destination are different locations: the runbook builds under /vast/enroot-cache/<architecture>/, while /home and /vast are separate mounts on the SLURM appliance and Fleet Manager reads only the exact path configured below. After building, copy each .sqsh to the shared destination directory — current sites use /home/hpcadmin/shared/enroot-cache, which lives on the controller’s ~1 TB home volume — and configure that absolute path. Any filesystem the compute nodes all mount works as the destination: at sites with VAST set up, a path under /vast (PB-scale) is equally valid and avoids controller home-volume capacity pressure. What matters is that every *ContainerImage value points at exactly where the files were copied.

fleetManager:
  satellite:
    temporalWorker:
      hardwareTests:
        # Use the exact .sqsh basename the build produced — images/build.sh
        # prints each artifact path, and the runbook's image-to-setting map
        # lists which image feeds which setting.
        # Key the default image by CPU architecture. A single-architecture site
        # configures only its one key; a mixed-architecture site configures both.
        defaultContainerImage:
          x86_64: /home/hpcadmin/shared/enroot-cache/x86_64/[HARDWARE_TESTS_DEFAULT_SQSH_BASENAME]
          aarch64: /home/hpcadmin/shared/enroot-cache/aarch64/[HARDWARE_TESTS_DEFAULT_SQSH_BASENAME]
        gpuFryerContainerImage: /home/hpcadmin/shared/enroot-cache/[GPU_FRYER_SQSH_BASENAME]
        nvloomContainerImage: /home/hpcadmin/shared/enroot-cache/[NVLOOM_SQSH_BASENAME]

For an existing site migrating from the legacy scalar, deploy the compatible chart and Temporal worker first while the scalar remains, then apply the architecture-keyed map. Do not apply the map to an older chart: it renders the map as one scalar string and the old worker treats that text as an image path. Runner tarballs are selected at runtime from the compute node’s uname -m; do not set the deprecated runnerTarballArchitecture override. Architecture-keyed default images are currently supported for single-node hardware tests only. A multi-node test must retain the legacy scalar until its SLURM allocation can be constrained to one CPU architecture. Do not add imageArch or imageSources directly to this umbrella values file; they are currently dropped. With manual staging there is no workflow to run — copying the .sqsh files into place is the whole step. Once the umbrella forwards imageArch and imageSources, the S3 install path replaces manual staging: configure every source with its matching *ContainerImage destination, then run the Prepare hardware-test images workflow with the Ensure hardware-test images preset against one active compute device in validation, as defined by ADR-0033.

Validate Connectivity

Shell into a Temporal worker pod and query the SLURM API:

kubectl exec -it -n fleet-manager deployment/temporal-worker -- /bin/bash

SLURM_BASE_URL="[INSERT BASE URL]"
SLURM_JWT="[INSERT JWT TOKEN]"

curl -sk \
  -H "X-SLURM-USER-TOKEN: ${SLURM_JWT}" \
  -X GET "${SLURM_BASE_URL}/slurm/v0.0.43/nodes"

Deploy and Argo Sync

This is the Apply / Deploy step from the flow above — Fleet Manager goes live when it completes. Open the k8s-cluster-fleet-manager pull request with the deployment configuration from the previous phases, get it reviewed, and merge it. If the hub change from Enable the Hub to Accept the Satellite travels in a separate PR, merge and sync it first — the satellite agents cannot authenticate until the hub expects them. The maintained template enables autosync on the parent Application aoa-deploy-fleet-manager; that parent creates the child fleet-manager-satellite Application. If autosync is disabled, sync the parent against the target cluster:

argocd app sync aoa-deploy-fleet-manager

During the deploy, the chart’s ExternalSecrets copy the bootstrap-written Vault values into Kubernetes — no manual Secret creation is needed on the happy path (see Generate Service Account Tokens). Confirm the Application reports Synced and Healthy and the Fleet Manager pods start. Crash loops at this point usually mean a missing required Helm value (see Set the Satellite Identity and Endpoints) or a missing runtime-vault item (see Add Runtime Service Credentials). Once the deployment is live, log in at fleet.nscale.com and continue with validation.

Run Post-Deployment Validation

Treat validation as part of the deployment, not as a later cleanup step. At minimum, verify:

device ingestion from NetBox
end-to-end enrolment
Temporal workflow activity
Grafana logs and dashboards
burn-in smoke tests where SLURM is enabled

Use Site Validation for the concrete checks and promotion expectations.

Log In and Request Access

Once Fleet Manager is live (after Apply / Deploy), it’s at fleet.nscale.com (dev and prod). Anyone with an @nscale.com email gets read-only access automatically — just log in. Posting in #fleet-manager-chat triggers the “Fleet Manager - request user access” workflow bot, which replies with the steps:

Log in to Fleet Manager with your Nscale email to create your user.
- If you’re stuck on the Vercel screen in dev, use the access password from the bot message.
If you need access above read-only (e.g. to run jobs), fill out the form (Request User Access) to update your permissions.

If you already have an account and are getting errors, check whether there’s an incident before requesting access.

​Prerequisites and Access

​Set Up the Site Infra Ops Group

​Useful Contacts

​Kubernetes Cluster

​Establish the Site Identity and Platform

​Run the Bootstrap Automation First

​Items to Copy or Create Manually After Bootstrap

​Create the Secrets Footprint

​Create the Fleet Manager Vaults

​Generate Service Account Tokens

​Create the NetBox API Token

​Required NetBox Object Permissions

​Store the NetBox Token in 1Password

​Generate and Store the Site PKI Material

​DHCP

​Add Runtime Service Credentials

​Provision S3 Buckets

​Add IPMI Exporter Credentials

​Register Redfish Credentials in the Observability Chart

​Prepare NetBox and Config Context

​Validate the Source Data

​Create the NetBox Config Context

​Create the NetBox Cluster

​Configure OpenStack and IPA

​Project and Application Credentials (provisioned by the OpenStack team)

​Dual-Ironic Deployments

​1. Add 1Password credentials

​2. Use the standard 1Password document names

​3. Local development

​Verification

​Verify or Upload the IPA Images

​Create the Cluster Deployment Configuration

​Set the Satellite Identity and Endpoints

​Enable the Hub to Accept the Satellite

​Configure the Topology Service

​Do the topology-service dev work first

​Check NetBox config readiness

​Enable the topology agent

​Configure Cross-Site Hardware Monitoring

​Connect the Site to SLURM

​Check the Prerequisites

​Configure OpenStack SLURM Adoption

​Generate the SLURM JWT

​Store the SLURM JWT in 1Password

​Configure the Temporal Worker

​Stage the Hardware-Test Container Images

​Validate Connectivity

​Deploy and Argo Sync

​Run Post-Deployment Validation

​Log In and Request Access