> ## Documentation Index > Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt > Use this file to discover all available pages before exploring further. # Deploy Fleet Manager at a New Site > Actionable rollout guide for bringing Fleet Manager online at a new site. Use this page as the main rollout guide for a new Fleet Manager site. It collects the deploy-time procedure in one place and links out only when a separate page is already the canonical source for that topic. Steps owned by other teams (Networking, Kubernetes, OpenStack) are called out as dependencies. **When can I use Fleet Manager?** You can log in at [fleet.nscale.com](https://fleet.nscale.com) once the **Apply / Deploy** step completes (Phase 7 — see [Deploy and Argo Sync](#deploy-and-argo-sync)). The flow below shows the dependency order: grey is physical/DC-ops work, yellow gates are readiness owned by other teams, blue is Fleet Manager configuration, green is applying the change, and purple is validation. Dashed gates are per-site opt-ins most sites skip — see [Fabric Control Planes](./dependencies-and-external-systems#fabric-control-planes). ```mermaid theme={null} flowchart TD HW["Hardware installation · rack · cable · power
Deployment Team · Harold Coqmard"] K8S["K8s cluster ready
Infrastructure · Drew Hudson-Viles, Joe Derbyshire"] OS["OpenStack ready
OpenStack · Tom Clark, John Garbutt"] NB["NetBox ready · devices · config context
NetBox · Tom Clark, John Garbutt"] CFG["Configure Fleet Manager · secrets + templates
Fleet Manager Team· Vedran Šimunović"] APPLY["Apply changes · deploy · Argo sync
Fleet Manager Team · Vedran Šimunović"] LIVE["✅ Fleet Manager is LIVE
log in at fleet.nscale.com"] VAL["Validate
Fleet Manager Team · Vedran Šimunović"] SLURM["SLURM integration · current template dependency
burn-in optional · SLURM · Antony Cleave, James Vaughn"] NVL["NVLink partitions · NMX-M
only if the site writes real partitions · Network Engineering"] SWVRF["Switch VRF flips
only if the site flips real VRFs · Network Engineering"] HW --> K8S --> OS --> NB --> CFG --> SLURM --> APPLY --> LIVE --> VAL CFG -.-> NVL -.-> APPLY CFG -.-> SWVRF -.-> APPLY classDef phys fill:#e2e8f0,stroke:#475569,color:#0f172a; classDef dep fill:#fde68a,stroke:#b45309,color:#422006; classDef fm fill:#bfdbfe,stroke:#1d4ed8,color:#172554; classDef apply fill:#bbf7d0,stroke:#15803d,color:#052e16; classDef live fill:#4ade80,stroke:#166534,color:#052e16,stroke-width:3px; classDef verify fill:#e9d5ff,stroke:#7e22ce,color:#3b0764; classDef depopt fill:#fde68a,stroke:#b45309,color:#422006,stroke-dasharray: 4 3; class HW phys; class K8S,OS,NB dep; class CFG,SLURM fm; class APPLY apply; class LIVE live; class VAL verify; class NVL,SWVRF depopt; ``` The detailed phase diagram breaks the same rollout into its full set of steps: green is handled by the standup automation, yellow is manual/owner work, blue is config-repo PRs, and purple is validation. Dashed nodes are per-deployment opt-ins most sites skip. ```mermaid theme={null} flowchart TD P0["Phase 0 · Prerequisites & access"] subgraph SEC["Phase 1 · Secrets footprint"] direction LR S1["onepassword
bootstrap"] --> S2["openstack
project-prep
(if --config)"] --> S3["hashicorp-vault
bootstrap"] --> SM["manual items
slurm-api · ipmi"] end P2["Phase 2 · NetBox & config context"] P3["Phase 3 · OpenStack & IPA images"] P4["Phase 4 · Cluster config (k8s-cluster-fleet-manager)"] NMX["NVLink partitions · NMX-M
only if the site writes real partitions"] VRF["Switch VRF flips
only if the site flips real VRFs"] P5["Phase 5 · SLURM integration
current template dependency · burn-in optional"] P6["Phase 6 · Cross-site monitoring
only if scraped remotely"] P7["Phase 7 · Deploy & Argo sync
✅ FM LIVE · log in at fleet.nscale.com"] P8["Phase 8 · Post-deploy validation"] P0 --> SEC --> P2 SEC --> P3 P2 --> P4 P3 --> P4 P4 --> P5 --> P7 P4 --> P6 --> P7 P4 -.-> NMX -.-> P7 P4 -.-> VRF -.-> P7 P7 --> P8 classDef auto fill:#dcfce7,stroke:#16a34a,color:#14532d; classDef manual fill:#fef9c3,stroke:#ca8a04,color:#713f12; classDef repo fill:#dbeafe,stroke:#2563eb,color:#1e3a8a; classDef verify fill:#f3e8ff,stroke:#9333ea,color:#581c87; classDef optional fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-dasharray: 4 3; class SEC,S1,S2,S3 auto; class SM,P0,P7 manual; class P3,P4,P6 repo; class P2,P5 manual; class P8 verify; class NMX,VRF optional; ``` The two dashed nodes are independent of each other: [NVLink](../../start-here/glossary#nvlink) partition writes and switch [VRF](../../start-here/glossary#vrf) flips are separate per-deployment flags, both off by default, and a site can enable either, both, or neither. Their prerequisites are in [Fabric Control Planes](./dependencies-and-external-systems#fabric-control-planes). ## Prerequisites and Access Set up general access first — GitHub, [1Password](../../start-here/glossary#1password), HashiCorp [Vault](../../start-here/glossary#vault), [NetBox](../../start-here/glossary#netbox), and [Teleport](../../start-here/glossary#teleport) — using the [Getting Access](../../start-here/getting-access) guide. This section covers only what's specific to standing up a new site. You need GitHub access (via the `fleet-manager` team) to these repos: * [`k8s-cluster-fleet-manager`](https://github.com/nscaledev/k8s-cluster-fleet-manager) * [`fleet-manager`](https://github.com/nscaledev/fleet-manager) * [`k8s-deploy-fleet-manager`](https://github.com/nscaledev/k8s-deploy-fleet-manager) Beyond that standard access, the bootstrap automation requires these rights, requested via [Servicely](https://nscale.servicely.ai/): * create 1Password vaults * create service-account (SA) tokens — request this via a [Servicely ticket](https://nscale.servicely.ai/) * apply existing group grants and create the site Infra Ops group Bootstrap also needs: * the 1Password CLI (`op`) at 2.34.0 or newer * a HashiCorp Vault policy with KV-v2 create/update/read on `fleet-manager/data/{env}/{deployment}/*` and list/read on `fleet-manager/metadata/{env}/{deployment}/*` — request it from the Vault admins. Vault cluster-admin rights are not needed; the KV mount, AppRole auth method, and ExternalSecret store are platform-managed. The deployment vaults do not exist before the first bootstrap. After bootstrap creates them, it may exit with code `2` while creating the service account. Enable **Service Account Access** on the runtime, operators, and networking vaults (`Manage` → `Service Account Access`), then follow the lower-level retry printed by the CLI. The service account reads the site's local networking vault for switch-workflow credentials; it never receives access to the shared source vault. ### Set Up the Site Infra Ops Group Site operators use the site's own 1Password group, named after the site tag — `ICE2 Infra Ops`, `GSO1 Infra Ops`. Agree its membership in the Slack deployment group before bootstrap. The config-driven bootstrap then: 1. resolves the site tag from the deployment slug's site segment (`sys1-ice2` → `ICE2`), asking you to confirm it. Pass `--site-tag` to set it outright when the slug and site tag differ 2. creates `{site} Infra Ops` if it is missing 3. assigns the configured group managers, plus anyone passed with `--add-infra-ops-group-manager` — use this for the site manager 4. grants the group view/copy/history/create/edit/export/share/print access to `Fleet Manager {deployment}-operators {env}` — item edits let site operators reset [BMC](../../start-here/glossary#bmc) passwords in place. Import, archive, delete, and move are not granted If the signed-in operator cannot look up or create the group, bootstrap skips only that group and still applies every other configured grant. Adding other members is still manual; use a [Servicely ticket](https://nscale.servicely.ai/) for the agreed operator list. ## Useful Contacts Who to talk to for the site dependencies: | Topic | Team | Contact | | - | - | - | | NetBox, OpenStack | Infra Eng | Tom Clark, John Garbutt | | [IPMI](../../start-here/glossary#ipmi) | Infra Eng | Tom Clark or Grant Delaney | | Kubernetes cluster | Cloud Native Platform Reliability Engineering | Drew Hudson-Viles, Joe Derbyshire | | [SLURM](../../start-here/glossary#slurm) | HPC | Antony Cleave, James Vaughn | ### Kubernetes Cluster The target Kubernetes cluster is a prerequisite for the FM satellite (control plane) work for a worked example, see the [LON14 provisioning PR](https://github.com/nscaledev/k8s-cluster-fleet-manager/pull/483/changes). ## Establish the Site Identity and Platform Start by confirming the basic deployment shape: * the site has an agreed identifier such as `glo1` or `sta1` * the target Kubernetes clusters exist for the required environments * the site's [devices](../../start-here/glossary#node) already exist in NetBox * the [DHCP](../../start-here/glossary#dhcp) model for the site is known — externally managed by default (see [DHCP](#dhcp)) Kubernetes clusters normally follow this pattern: ```bash theme={null} fleet-mgr-{ENV}-{DEPLOYMENT} ``` Examples: ```bash theme={null} fleet-mgr-dev-glo1 fleet-mgr-prod-sta1 ``` Before a deployment can go live, the site also needs: * networking for the target Kubernetes clusters * any SLURM-side VM provisioning needed for [burn-in](../../start-here/glossary#burn-in) workflows * coordination with the teams responsible for NetBox, Kubernetes, and network access ## Run the Bootstrap Automation First For the Vault/ExternalSecret happy path, use the standup CLI before creating 1Password, Vault, or OpenStack project-prep resources by hand: ```bash theme={null} uv run fleetctl deployment-standup automation bootstrap {deployment} {env} \ --project-prep-config ./project-prep.yaml ``` Bootstrap reuses your active 1Password and HashiCorp Vault sessions, so authenticate first: ```bash theme={null} op signin export VAULT_ADDR=https://vault.nscale.com vault login -method=oidc ``` For the full command reference — helper commands, retries, local rehearsal, and auth details — see the operator reference at [`src/fleet_manager/cli/commands/deployment_standup/README.md`](https://github.com/nscaledev/fleet-manager/blob/main/src/fleet_manager/cli/commands/deployment_standup/README.md). This page keeps the happy path; that README covers the low-level `deployment-standup` subcommands. Pass `--project-prep-config` only when the non-secret OpenStack project-prep file is ready. Without it, the OpenStack slice is skipped and the 1Password/Vault path still runs. Prod uses **`Fleet Manager shared prod`** as its default source for `nscale-api-key`, `mimir-login`, `netbox-login`, and `pki-ca`. It copies `svc-switch-login` from **`Fleet Manager shared-networking prod`** into the site's networking vault. The operator needs read access to both shared source vaults; workers receive only their deployment-local copies. Stg has no default runtime-credential source vault: a default stg bootstrap creates only `pki-ca` from `pki-ca-stg` and skips all three runtime items as a group. Before deploying stg, run the lower-level 1Password slice with an approved source vault, then finish the OpenStack and Vault slices described in the operator reference: ```bash theme={null} uv run fleetctl deployment-standup onepassword bootstrap {deployment} stg \ --runtime-credentials-source-vault "" ``` The source uses `netbox-login-stg` and `pki-ca-stg`; the deployment items remain named `netbox-login` and `pki-ca`. The matching agents fail to start until the runtime items exist. `automation bootstrap` handles: * 1Password runtime, operators, and networking vault creation or reuse * configured 1Password group grants * shared runtime item copies for `nscale-api-key`, `mimir-login`, and the configured NetBox source item when a source vault is configured * the shared-source `svc-switch-login` item copied into the site's networking vault * deployment `pki-ca` generation from the configured source CA (`pki-ca-stg` for stg) * 1Password service-account token creation and storage * optional OpenStack project prep when `--project-prep-config` is provided * transitional HashiCorp Vault runtime secret bootstrap for ExternalSecrets The Vault slice is not a no-op: it attempts to write the satellite runtime Vault footprint and requires Vault access. There is no non-Vault route — Vault and the platform ExternalSecret are how runtime secrets reach the cluster, and no `deployment-standup` command writes Kubernetes Secrets. A site without Vault access cannot finish standup; request the KV-v2 policy in [Prerequisites and Access](#prerequisites-and-access) first. Use the lower-level `deployment-standup` commands for retries, partial recovery, or non-default backend options. If bootstrap exits with code `2`, stop at that slice and follow the printed handoff and retry command. The orchestrator has no `--resume` option; do not add lower-level options to it. ### Items to Copy or Create Manually After Bootstrap Bootstrap creates the secrets and env vars needed to stand up the FM cluster without an OpenStack dependency. The following items are **not** created by bootstrap: * **`slurm-api-prod`** — see [Store the SLURM JWT in 1Password](#store-the-slurm-jwt-in-1password). * **`ipmi-exporter`** — create manually. Add to the operators vault `Fleet Manager {deployment}-operators {env}`. See [Add IPMI Exporter Credentials](#add-ipmi-exporter-credentials). ## Create the Secrets Footprint The bootstrap command creates most of this footprint. Use this section as the target-state checklist and for the pieces that still need an external owner or manual setup. ### Create the Fleet Manager Vaults Handled by `automation bootstrap`. For each target environment it creates or reuses: * `Fleet Manager {deployment} {env}` for runtime secrets * `Fleet Manager {deployment}-operators {env}` for operator-managed secrets such as BMC and IPMI credentials * `Fleet Manager {deployment}-networking {env}` for site network engineering credentials Real switch-VRF or NVLink partition writes use `Fleet Manager {deployment}-networking {env}`. Bootstrap creates that vault and its group access; separately grant the satellite service account `Read` and add only the service accounts needed by enabled real fabric operations. Use this split: * runtime vault: NetBox token, OpenStack/clouds config, SLURM/API tokens * operators vault: `*-bmc`, `ipmi-exporter`, and similar operator-managed credentials * networking vault: site-local network gear credentials, owned by network engineering. Switch workflows read the local `svc-switch-login` copy when real switch writes are enabled; they never read the shared source vault directly To add the networking vault to a site that already exists, scope a bootstrap run to that role: ```bash theme={null} uv run fleetctl deployment-standup onepassword bootstrap {deployment} {env} --only networking ``` It creates the vault, applies the configured group access (`Fleet Manager Admins`, `Fleet Manager Squad`, `FOE`), copies the shared switch Login, and verifies the existing service-account token can read all three local vaults. It does not create or rotate the token, or run runtime-item or `pki-ca` work. For a deployment whose service account predates networking-vault access, an account Owner must first add `Read` access to `Fleet Manager {deployment}-networking {env}` to the existing service account. The focused command intentionally verifies that grant rather than rotating the existing token. Non-production runs still create the empty networking vault and grant token access, but copy a switch Login only when given `--networking-credentials-source-vault`. ### Generate Service Account Tokens Handled by `automation bootstrap`. It generates one service-account token per environment with access to all three deployment-local vaults: * `Fleet Manager {deployment} {env}` * `Fleet Manager {deployment}-operators {env}` * `Fleet Manager {deployment}-networking {env}` Recommended permissions: * runtime vault: `Read` * operators vault: `Read + Write` * networking vault: `Read` Store the token in the runtime vault as an `API Credentials` item named: ```text theme={null} Service Account Auth Token: Fleet Manager {deployment} {env} ``` Satellite runtime config should set: ```bash theme={null} OP_VAULT_SELECTOR={deployment} ``` At deploy time the satellite reads this token from the Kubernetes Secret `fleet-manager-satellite-op-token` (namespace `fleet-manager`, key `op-token`) — every agent resolves its 1Password items through it. Vault/ExternalSecret delivers that Secret from the footprint bootstrap writes, so nothing is created on the cluster by hand. The standup CLI does not write Kubernetes Secrets; if that Secret is missing, fix the Vault value and the ExternalSecret rather than applying it manually. ### Create the NetBox API Token For prod, bootstrap expects `netbox-login` in the configured shared source vault and copies it into the deployment runtime vault. For stg, the configured source item is `netbox-login-stg`, and the target runtime-vault item still stays named `netbox-login`. Create a deployment-specific NetBox token only when the NetBox owner asks for one, or when the shared source item has not been prepared yet. Repeat the following for each target environment: 1. Access NetBox and log in. 2. Open `Profile` → `API Tokens` → `Add a Token`. 3. Create a token with: * description: `Fleet Manager {deployment} {env} API Access` * `Write enabled`: checked * expiry: `31 December 2030` * allowed IPs: the Kubernetes cluster NAT gateway IP or CIDR The token needs write access because Fleet Manager updates device status and tags during enrolment workflows. Bootstrap normally copies the same `netbox-login` token value into each deployment runtime vault, so hub and satellite services commonly use one shared token. Its NetBox `Write enabled` flag permits every `POST`, `PATCH`, `PUT`, and `DELETE` that the token owner's Object Permissions authorise. Fleet Manager has no per-operation write switch. Grant that token the union of the Object Permissions below, and scope them to the deployment's sites wherever the NetBox owner can safely do so. #### Required NetBox Object Permissions Fleet Manager reads the following resources while synchronising inventory and carrying out workflow preflight. The consumer column identifies which services need each `view` permission; it does not imply separate tokens: | Consumer | NetBox objects | Why Fleet Manager reads them | | - | - | - | | Hub and satellite | Sites (`dcim.site`) | Resolve site configuration and materialise sites in Fleet Manager | | Hub agents | Locations (`dcim.location`), Racks (`dcim.rack`), Device Types (`dcim.devicetype`), Device Roles (`dcim.devicerole`) | Materialise the inventory hierarchy in Fleet Manager | | Hub and satellite | Devices (`dcim.device`), Interfaces (`dcim.interface`), Cables (`dcim.cable`) | Sync devices and topology; resolve workflow targets and cabling | | Hub agents | MAC Addresses (`dcim.macaddress`) | Requeue a device within a minute of an edit to one of its MAC addresses, which NetBox 4.2+ keeps apart from the interface. Without it the netbox agent's change poll logs an HTTP 403 every minute, and MAC address edits wait for the resync. A constraint that leaves out a synced site hides that site's MAC addresses without an error | | Hub agents | Modules (`dcim.module`) | Build the per-component FRU data projection | | Hub agents | Console Ports (`dcim.consoleport`), Console Server Ports (`dcim.consoleserverport`) | Project console links into inventory | | Hub and satellite | IP Addresses (`ipam.ipaddress`) | Resolve management and network configuration | | Hub agents | Prefixes (`ipam.prefix`) | Resolve network configuration for inventory projection | | Hub and satellite | Tags (`extras.tag`) | Resolve the pre-existing tag before attaching it to a device | | Satellite worker and operators, at a site with a RoCE back end | Interface peer mappings (`netbox_datacenter.inventoryinfointerfacepeermapping`, from the `netbox_datacenter` plugin) | Resolve which back-end leaf sub-port each host PF reaches through shuffle cassettes (`fleetctl netbox be-port-plan`, `be-switch-state`, and the back-end isolate and return workflows). Without it they fail with HTTP 403. In production NetBox it is part of the `Read-only` Object Permission, which the `fleetmanager` user already holds (confirmed by the plugin owner, 2026-10-08) | The following writes are conditional on the corresponding Fleet Manager operation, but must be included when that operation is enabled: | NetBox objects | Object Permissions | Fleet Manager operation | | - | - | - | | Devices (`dcim.device`) | `view`, `change` | Enrolment sets `staged` → `active`, and, once the site's `platform_class_activity_ready` gate is on, records `custom_fields.platform_class`; `fleetctl netbox backfill-platform-class --apply` writes the same field; enrolment/BMC configuration attach existing tags; unenrolment restores `staged`; node swap moves rack locations, and a cableless swap sets the failed tray `offline` | | Interfaces (`dcim.interface`) | `view`, `change` | `fleetctl netbox populate-ib-guids --apply` writes an InfiniBand port GUID to an interface label; node swap sets a never-enrolled spare's frontend interfaces' primary MAC | | MAC Addresses (`dcim.macaddress`) | `view`, `add` | Node swap fills a never-enrolled GB300 spare's missing frontend MACs from its BMC: it looks each MAC up across NetBox, refusing one recorded elsewhere, and creates the missing ones. Without these grants the swap fails at that step with HTTP 403 after configuring the spare's BMC, having at most created a MAC object a rerun reuses, and before any write to Fleet Manager, Nova or Ironic | | Cables (`dcim.cable`) | `view`, `add`, `delete` | GB300 tray replacement deletes and recreates cables while swapping terminations | Fleet Manager does **not** create or delete Devices, Interfaces, IP Addresses, Prefixes, Sites, Locations, Racks, Device Types, Device Roles, Console Ports, or Console Server Ports. Do not grant those permissions merely for Fleet Manager. Before rollout, the NetBox owner should validate the filters against the deployed NetBox version and run the token check. `check-auth` proves the token can read Sites from the operator's source IP; a production token restricted to the cluster NAT still needs a cluster-side workflow or agent read to prove its allowed-IP rule: ```bash theme={null} uv run fleetctl deployment-standup netbox check-auth {deployment} {env} ``` ### Store the NetBox Token in 1Password If bootstrap is copying the shared source item, keep the source item named `netbox-login` (`netbox-login-stg` for stg) and let the CLI copy it. If a manual per-deployment token is required, store a `Login` item called `netbox-login` in the runtime vault: * `username`: `fleetmanager` * `website`: `https://netbox-{env}.mgt.{deployment}.nscale.com` * `auth-token`: the token created in NetBox Prefer the standup helper so the item shape stays consistent: Set `NETBOX_STANDUP_TOKEN` to the token from the previous step, or omit it and enter the token at the hidden prompt. ```bash theme={null} uv run fleetctl deployment-standup netbox store-token {deployment} {env} \ --netbox-url "https://netbox-{env}.mgt.{deployment}.nscale.com" ``` ### Generate and Store the Site PKI Material Handled by `automation bootstrap` when the configured source vault contains `pki-ca`. Use [PKI & Certificate Trust](../pki-and-certificate-trust) for source CA setup, rotation, and manual fallback steps. ### DHCP DHCP is managed externally at current sites — Fleet Manager does not own DHCP configuration, so leave the enrolment step off: ```yaml theme={null} fleetManager: satellite: temporalWorker: enrollmentSteps: configureDhcp: false ``` If a site instead relies on Fleet Manager-managed `dnsmasq`, keep `configureDhcp` enabled and follow [DHCP server - required setup](https://github.com/nscaledev/fleet-manager/blob/main/docs/development-setup.md#dhcp-server---required-setup). If unsure, state your assumption in `#squad-fleet-manager` at go-time. ### Add Runtime Service Credentials These runtime-vault items are required — the matching agents crash on startup if they are missing. Bootstrap creates `pki-ca` from the configured source CA. When a runtime-credential source vault is configured, it also copies `nscale-api-key`, `mimir-login`, and the configured NetBox source item. If that copy was skipped, create the three runtime items with the standalone commands or rerun the lower-level `onepassword bootstrap` slice with an explicit source vault before deploying. For stg, the source item titles are `netbox-login-stg` and `pki-ca-stg`; the deployment item titles remain `netbox-login` and `pki-ca`. | 1Password item | Used by | Required fields | | - | - | - | | `mimir-login` | Health-status agent (Mimir queries) | `username`, `password` | | `nscale-api-key` | Diagnostics agent (LLM calls) | `credential` | One runtime item is optional. Put a Slack incoming-webhook URL in the `credential` field of `fm-funny-job-names-slack-webhook` and the diagnostics agent posts that channel every new job whose generated name the LLM finds funny. Without the item those alerts are off; the agent reads it at startup, so restart the agent after adding or removing it. The Workflow agent's S3 credentials can be provided two ways, both supported today: * **Manual `object-storage` item.** Create an `object-storage` 1Password item in the runtime vault with `access_key_id` and `secret_access_key` (the satellite's own S3 credentials, not a shared pair). * **Controller-provisioned bucket (intended direction).** Use the bucket-controller secret `workflow-agent-object-storage-credentials` (see [Provision S3 Buckets](#provision-s3-buckets)). ### Provision S3 Buckets Fleet Manager needs two S3 buckets, provisioned by the `s3.nscale.com` controller (deployed via `k8s-deploy-core`). Enable the bucket manifests only once the controller is confirmed synced on the cluster. Add them to the deployment's `buckets.yaml` — see [this PR](https://github.com/nscaledev/k8s-cluster-fleet-manager/pull/452/changes) for an example: * **[Temporal](../../start-here/glossary#temporal) Postgres backups** — namespace `temporal`, secret `temporal-postgres-s3-credentials`. * **Workflow report/artifact uploads** — namespace `fleet-manager`, secret `workflow-agent-object-storage-credentials` (the Workflow agent's runtime S3 credentials). Key points: * **Keep `spec.name` short.** The controller always prepends `cluster-env-region` to `spec.name`, so do not pre-write the prefix or it doubles — for a production fleet-mgr template, `postgres-temporal` resolves to `fleet-mgr-prod-mgt-sys1-lon14-postgres-temporal`. `metadata.name` must also stay ≤ 63 characters, since access-key resource names are built from cluster + namespace + Bucket name. * **`credentialsOutput`** creates the named secret with keys `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `BUCKET_NAME`, and `S3_ENDPOINT`. For controller-provisioned buckets, set `fleetManager.satellite.workflowAgent.objectStorage.enableOperatorSecret: true` in the deployment's `fleet-manager.yaml`; the satellite chart then reads the credentials, bucket name, and endpoint from `workflow-agent-object-storage-credentials`. Leave the flag false for the manual `object-storage` 1Password flow. The umbrella does not forward manual bucket/endpoint overrides when the flag is false, so that path uses the inner chart's environment defaults; add a pass-through before rollout if the site needs custom values. * **`deletionPolicy: Retain`** is intentional — Temporal Postgres backups must survive removal of the Bucket CR. The SLURM JWT (`slurm-api-prod`) is created later under [Store the SLURM JWT in 1Password](#store-the-slurm-jwt-in-1password). ### Add IPMI Exporter Credentials Store the deployment IPMI credentials in the operators vault as a `Login` item named `ipmi-exporter`. Example: ```bash theme={null} DEPLOYMENT="{deployment}" ENV="{env}" VAULT="Fleet Manager ${DEPLOYMENT}-operators ${ENV}" IPMI_PASSWORD="[PROVIDED BY OPERATIONS TEAM]" op item create \ --category=login \ --title="ipmi-exporter" \ --vault="${VAULT}" \ username=ipmi_exporter \ "password=${IPMI_PASSWORD}" ``` `automation bootstrap` also runs the transitional HashiCorp Vault bootstrap used by ExternalSecrets. Only wire Vault paths manually when the chart paths differ from the standup convention or bootstrap prints a manual handoff. ### Register Redfish Credentials in the Observability Chart If the cluster uses [Redfish](../../start-here/glossary#redfish) hardware monitoring, wire its credentials in the [`k8s-deploy-observability`](https://github.com/nscaledev/k8s-deploy-observability) repo as an `ExternalSecret`, under `charts/observability/templates/otel/agent/secrets/redfish-secrets.yaml`. This is separate from the `ipmi-exporter` item above — it feeds the Prometheus Redfish exporter env vars from Vault so it can authenticate against the cluster's BMC interfaces. Copy an existing region's `ExternalSecret` in `redfish-secrets.yaml` and adjust it for the new region — that file is the source of truth for the manifest shape, so follow it rather than a copy reproduced here. When adjusting it, the Fleet Manager-specific values are: * gate the resource on the target cluster and region: `clusterName: fleet-mgr`, `region: ` * Vault key path `/{{ .Values.environment }}/ipmi`, properties `username` / `password` (e.g. `sys1-ice2/prod/ipmi`) * templated env vars in UPPER\_SNAKE region form — `SYS1_ICE2_REDFISH_EXPORTER_USERNAME` / `..._PASSWORD` — to match the exporter's credential lookup ## Prepare NetBox and Config Context ### Validate the Source Data Before rollout: * devices for the site should already exist in NetBox * devices intended for Fleet Manager ingestion should be marked as `staging` * required deployment-specific metadata should already be present Run the validation script before proceeding: ```bash theme={null} export NETBOX_URL=https://netbox-dev.nscale.teleport.sh export ENVIRONMENT=dev export OP_VAULT_SELECTOR=glo1 export OP_SERVICE_ACCOUNT_TOKEN=... # 1Password service-account token for the selected vault uv run python scripts/checks/netbox.py ``` Use dev NetBox for pre-rollout validation and prod NetBox before deploying to prod. `ENVIRONMENT`, `OP_VAULT_SELECTOR`, and `OP_SERVICE_ACCOUNT_TOKEN` are only required here to satisfy the script's runtime settings (it loads the same settings as the services and resolves the NetBox token from 1Password when one is not provided); use valid local values for your environment rather than treating `dev` and `glo1` as deployment-specific rollout inputs. ### Create the NetBox Config Context Config context is the source of truth for hardware settings Fleet Manager expects to enforce. At minimum, prepare the deployment's expected: * NIC configuration * firmware versions * BIOS settings For NIC configuration: 1. Work with the datacenter and networking teams to identify the required settings. 2. Format the expected values into the config-context JSON structure. 3. Create a Config Context in NetBox under `Provisioning` → `Config Contexts`. 4. Scope it narrowly to the correct deployment and device type. 5. Save it and verify it appears on the targeted devices. A typical structure looks like: ```json theme={null} { "mlxconfig_config": { "port1": {}, "port2": {} }, "broadcom_config": { "nic_settings": {} }, "bios_config": {}, "firmware_config": {} } ``` Keep the scope tight enough to avoid accidentally applying the same config context to the wrong site or hardware family. ### Create the NetBox Cluster Create a cluster for the deployment under [Virtualization → Clusters](https://netbox.nscale.teleport.sh/virtualization/clusters/). The NetBox agent uses this cluster to filter and ingest only the devices belonging to this deployment. ## Configure OpenStack and IPA > **Dependency:** the OpenStack project and application credentials are provisioned by the > OpenStack team (see [Useful Contacts](#useful-contacts)). This phase depends on OpenStack being ready. Store only the credentials for integrations the site actually uses in the deployment runtime vault: * `openstack-overcloud` for every site * `openstack-undercloud` when the site has `CPU_HYPERVISOR` devices * `openstack-slurm` when SLURM adoption is configured The `k8s-deploy-fleet-manager` umbrella takes the [Ironic](../../start-here/glossary#ironic) enrolment project by name; Fleet Manager resolves it to a Keystone project ID at runtime. Get the name from the OpenStack owner. If there is no undercloud, leave it disabled: ```yaml theme={null} fleetManager: satellite: openstack: undercloudsEnabled: false placementEnabled: true # set false on standalone-Ironic sites with no Placement service ironic: enrolTargetProjectName: "[ENROL_TARGET_PROJECT_NAME]" ``` > **Prerequisite (standalone-Ironic sites) — automated disk cleaning must be on.** Fleet > Manager's teardown (`UnimageNodeWorkflow`) drives Ironic's `deleted` action and waits for the > node to reach `AVAILABLE`, but the disk, config-drive, and injected-key wipe is Ironic > cleaning's work on that transition — FM does not force the erase. If the conductor has > automated cleaning disabled or no disk-erase clean step, a torn-down node returns to the enrol > pool with the previous [tenant](../../start-here/glossary#tenant)'s image and data intact while teardown still reports success. > Confirm the conductor runs with `[conductor]automated_clean=True` and a disk-erase step in the > site's `clean_steps` before enabling the Ironic-direct IMAGE\_NODE path. The same umbrella currently requires real SLURM project, network, and REST endpoint values. A site with no SLURM cannot be configured safely by inserting placeholders; updating the umbrella chart to expose the inner chart's no-SLURM mode is a rollout blocker. See [Connect the Site to SLURM](#connect-the-site-to-slurm). ### Project and Application Credentials (provisioned by the OpenStack team) The Keystone project, the Fleet Manager user role, quotas, and the application credentials are created by the OpenStack provisioning team. Share [OpenStack Application Credentials](./openstack-application-credentials) with them as the spec for the access rules Fleet Manager needs (baremetal / placement / compute) and the `clouds.yaml` shape. The finished credentials land in 1Password as the applicable items listed above. If FM engineers need to run project prep themselves, pass a reviewed non-secret project-prep config to bootstrap — it creates or reuses the Keystone project, grants the Fleet Manager user role, and sets the managed [Nova](../../start-here/glossary#nova), Cinder, and Neutron quotas. Start from the [project-prep example](https://github.com/nscaledev/fleet-manager/blob/main/src/fleet_manager/cli/commands/deployment_standup/openstack/project-prep.example.yaml), copy it to a local `project-prep.yaml`, and keep secrets out of the file. Use the lower-level `deployment-standup openstack project-prep` command only for retries or non-default credential sources. ### Dual-Ironic Deployments Some deployments (e.g., GB300) run two separate Ironic instances — **overcloud** and **undercloud** — each managing distinct hardware. Fleet Manager routes devices to the correct Ironic instance based on the `device_class` field on FM DeviceRoles, which is populated by the NetBox agent via `NETBOX_AGENT_DEVICE_ROLE_CLASS_MAPPING`. Routing is automatic — no [Helm](../../start-here/glossary#helm) flag or list of slugs is needed. The OpenStack agent queries FM at startup to discover which device roles are in use at its configured site and starts one reflector per active instance: * **Overcloud** — GPU\_COMPUTE and CPU\_BAREMETAL device classes * **Undercloud** — CPU\_HYPERVISOR device class If no site devices map to an instance (e.g. no CPU\_HYPERVISOR hardware at that site), that instance is skipped and no credential lookup is attempted. #### 1. Add 1Password credentials * Add an `openstack-overcloud` document to 1Password containing the overcloud `clouds.yaml` (see [OpenStack Application Credentials](./openstack-application-credentials)). * If the deployment has CPU\_HYPERVISOR nodes, also add an `openstack-undercloud` document for the undercloud. #### 2. Use the standard 1Password document names The inner satellite chart supports these overrides: | Helm value | Default | Description | | - | - | - | | `openstack.overcloudsOpItem` | `openstack-overcloud` | 1Password doc for overcloud | | `openstack.undercloudsOpItem` | `openstack-undercloud` | 1Password doc for undercloud | The current `k8s-deploy-fleet-manager` umbrella does not forward these values. New deployments must use the defaults; if different item names are required, add the umbrella pass-through before rollout. #### 3. Local development For local dev, point directly at `clouds.yaml` files instead of 1Password: ```yaml theme={null} openstack: undercloudsFile: "/path/to/undercloud-clouds.yaml" overcloudsFile: "/path/to/overcloud-clouds.yaml" ``` Or via environment variables: `OPENSTACK_UNDERCLOUD_CLOUDS_FILE` / `OPENSTACK_OVERCLOUD_CLOUDS_FILE`. #### Verification * OpenStack agent logs show `Starting reflector for Ironic instance 'overcloud'` (and `'undercloud'` if CPU\_HYPERVISOR nodes are present). * Each reflector's store is named `ironic-nodes-overcloud` / `ironic-nodes-undercloud` in metrics. * Nodes from each Ironic instance reconcile to FM devices with the correct roles. ### Verify or Upload the IPA Images Make sure the required [IPA](../../start-here/glossary#ironic-python-agent) images exist in the target OpenStack environment. Common image families: | Architecture | [GPU](../../start-here/glossary#gpu) Support | Image Type | Required? | | - | - | - | - | | x86\_64 | NVIDIA | `ipa-nvidia` | For B200 deployments | | aarch64 | NVIDIA | `ipa-nvidia-arm` | For GB200 or GB300 deployments | Check what is already present: ```bash theme={null} openstack image list --name "ipa-*" ``` If you need to upload the images manually: ```bash theme={null} openstack image create \ --disk-format aki \ --container-format aki \ --public \ --file [KERNEL_FILE] \ ipa-[VARIANT]-kernel-[VERSION] openstack image create \ --disk-format ari \ --container-format ari \ --public \ --file [RAMDISK_FILE] \ ipa-[VARIANT]-ramdisk-[VERSION] ``` Record the final UUIDs: ```text theme={null} {deployment} IPA Images: [VARIANT]: Kernel UUID: [TO BE FILLED] Ramdisk UUID: [TO BE FILLED] ``` Then wire them into Fleet Manager: * set `fleetManager.satellite.openstack.ironic.deployKernel` * set `fleetManager.satellite.openstack.ironic.deployRamdisk` ## Create the Cluster Deployment Configuration Once the Kubernetes cluster is provisioned and bootstrap has run, create the deployment from the maintained satellite template in `k8s-cluster-fleet-manager`. See [this PR](https://github.com/nscaledev/k8s-cluster-fleet-manager/pull/452) for a worked example. Create the deployment-specific configuration in `k8s-cluster-fleet-manager`: ```bash theme={null} git clone git@github.com:nscaledev/k8s-cluster-fleet-manager.git cd k8s-cluster-fleet-manager cp -r _templates/satellite {deployment} ``` Replace every placeholder in the copied production files. Values such as `secretPrefix`, Ceph IDs/monitors/pools, Velero buckets, and the Temporal/Loki object-storage targets are owned by the platform or storage teams; do not copy them from a live deployment. For a production satellite, update the files under `{deployment}/production/`, including: * `observability.yaml` * `infra.yaml` * `core.yaml` * `fleet-manager.yaml` Typical deployment-specific changes include: * env- and deployment-specific paths * Ironic IPA image UUIDs * `targetRevision` for `k8s-deploy-fleet-manager` The NetBox allowlist is hub configuration, not satellite configuration. Add the deployment's cluster to `fleetManager.hub.netboxAgent.deviceFilters` in `glo1/production/fleet-manager.yaml`. If the site introduces a new NetBox device-role slug, also update `fleetManager.hub.netboxAgent.deviceRoleClassMapping`; device class controls Ironic routing and workflow eligibility. See [the filter example](https://github.com/nscaledev/k8s-cluster-fleet-manager/pull/207) and [the role-mapping example](https://github.com/nscaledev/k8s-cluster-fleet-manager/pull/220/changes). To configure the Topology agent, see [Configure the Topology Service](#configure-the-topology-service) below — a new region needs topology-service dev work and NetBox readiness before it can be enabled in the manifests. See [this PR](https://github.com/nscaledev/k8s-cluster-fleet-manager/pull/215) for an example. Bootstrap prepares the 1Password and Vault footprint; `k8s-cluster-fleet-manager` still owns the namespace and chart wiring. If namespace or ExternalSecret generation is not yet automated there, create the namespace explicitly and wire the deployment to the expected secret paths before opening the pull request. ### Set the Satellite Identity and Endpoints Start from the maintained template and replace every placeholder. These identity and endpoint values are easy to misplace because operators configure the umbrella chart, not the inner satellite chart: | Helm value | What it is | | - | - | | `environment` | Deployment environment, normally `prod` for a new production satellite. | | `satelliteName` | The deployment slug (e.g. `sys1-ice2`). Also the default 1Password vault selector, and the agents' OAuth2 client IDs are templated from it (`workflow-agent-{satelliteName}`, …). | | `siteSlug` | The Fleet Manager/NetBox **site** slug (e.g. `keflavik`). Confirm it from NetBox site data — do not derive it from `satelliteName`; they usually differ. | | `fleetManager.netboxUrl` | The NetBox instance for this environment. | | `fleetManager.satellite.hubUrl` | The hub Fleet Manager API URL the agents call. | | `fleetManager.satellite.workflowAgent.logsExplorerUrl` | The Grafana Explore URL linked from workflow failures. | | `fleetManager.satellite.healthStatusAgent.mimir.baseUrl` | The Mimir endpoint; required when the health-status agent is enabled. | The umbrella generates the inner `temporal.host`; it is not a site input. OpenStack and SLURM render requirements are covered in their sections below. ### Enable the Hub to Accept the Satellite The hub must be configured to accept the new satellite before it can register: add the new `satelliteName` to `fleetManager.hub.expectedSatellites` in the **hub's** `fleet-manager.yaml` (under `glo1/production` for prod). The hub pre-creates one OAuth2 client per agent per entry (`ironic-agent-{satellite}` — the OpenStack agent's unchanged wire identity, ADR-0036 — `slurm-agent-{satellite}`, `workflow-agent-{satellite}`, …), and the satellite agents authenticate with client IDs templated from `satelliteName` — so the entry must match it exactly, or every agent fails hub authentication. This is a separate change from the satellite's own configuration and must be merged and synced before the satellite comes up. See [this PR](https://github.com/nscaledev/k8s-cluster-fleet-manager/pull/477) for an example. ## Configure the Topology Service The Topology service builds the NVLink/network topology graph for a region from NetBox data. A new region is **not** enabled by config alone — it needs development work in the [`topology-service`](https://github.com/nscaledev/topology-service) repo first, then values changes in `k8s-cluster-fleet-manager`, and it depends on the region's NetBox data being ready. ### Do the topology-service dev work first Registering a new region requires code and chart changes in `topology-service` first — follow that repo's `NEW_REGION_ONBOARDING.md` for the full change-set (it owns that contract). Until the change lands and ships, the region cannot be enabled in the deploy manifests. ### Check NetBox config readiness The graph is only as good as the NetBox data. Before enabling a region, confirm in the satellite's NetBox that: * the expected device roles exist (e.g. `compute-tray`, `be-spine-switch`, `be-leaf-switch`) * devices are in the expected statuses (e.g. `active`) and cabling is `connected` * you have the region's NetBox **site ID(s)**, **cluster ID(s)**, and **GraphQL URL** — these are per-NetBox-instance, so pull them from the satellite's NetBox rather than reusing another region's values ### Enable the topology agent Once the region is registered in `topology-service` and NetBox is ready, add the region's UNI name and FM site slug to `fleetManager.hub.topologyAgent.targets` in the hub's `glo1/production/fleet-manager.yaml`. For a worked example, see the glo1 config — [`topologyAgent` block](https://github.com/nscaledev/k8s-cluster-fleet-manager/blob/2a9332b3b0714a90deeb69696130c55f34115bbb/glo1/production/fleet-manager.yaml#L273) and [`targets`](https://github.com/nscaledev/k8s-cluster-fleet-manager/pull/215) The API reads the same list to find a site's Region: the topology-sync eligibility gate checks that Region before multi-node burn-ins, and `POST /api/v1/sites/{id}/topology-sync/`, which a cableless node swap and an unenrolment of an `active` tray use to declare the tray's removal, answers 422 for a site missing from it. The hub chart's `api.extraTopologyTargets` adds sites to that list for the API alone, which maps a Region without starting the topology agent's ingestion for it; k8s-deploy-fleet-manager has to forward that value before a cluster repo can set it. ## Configure Cross-Site Hardware Monitoring When a cluster's hardware is scraped by an exporter running at a **different** site — for example, `sys1-ice2` hosts are scraped from a `glo1` exporter — add a Prometheus Operator `ScrapeConfig` for that cluster in the [`k8s-deploy-observability`](https://github.com/nscaledev/k8s-deploy-observability) repo, under `charts/observability/templates/otel/agent/scrapeconfigs//`. Copy an existing region's files (e.g. `sys1-ice2/infra-redfish-hgx.yaml` for GB300/HGX hardware, plus the `infra-redfish-exporter.yaml`, `infra-redfish-telemetry.yaml`, and `infra-netbox-ipmi.yaml` siblings) and adjust them for the new region. Each file is a `ScrapeConfig` gated to the target cluster and region. Follow the reference files in `k8s-deploy-observability` for the manifest shape rather than a copy reproduced here; the Fleet Manager-specific points to get right (these replace the old `disableSiteFilter` values-list pattern) are: * **Scope by `&cluster={{ .Values.region }}`** on the NetBox prometheus-sd URL. This is what targets a remote cluster's devices — there is no `disableSiteFilter` flag anymore. `device_type` selects the hardware class (GB300/HGX uses `hg635n-v2`; adjust for other families). * **`__param_site`** is set by a relabeling to `{{ .Values.region | replace "-" "_" }}`, so the underscores (`sys1_ice2`) are produced by the template rather than hand-written. This must still match the `SYS1_ICE2_*` credential env var naming, or authentication fails in ways that are hard to trace back to the config. ## Connect the Site to SLURM The inner Fleet Manager chart can run without SLURM. Enable the OpenStack integration explicitly with: * `fleetManager.satellite.openstack.slurm.enabled` When that flag is true, configure the SLURM OpenStack project and provider network, as well as `fleetManager.satellite.temporalWorker.slurm.baseUrl`. These must be real values. A dummy URL starts SLURM clients and causes runtime failures. If the site has no SLURM, set the flag to false; the project and network values are ignored by Fleet Manager. The flag means "SLURM reached through OpenStack (Nova)", not "this site has SLURM". A standalone-Ironic site with no Nova keeps it false even if it has a SLURM cluster. With the flag false, the Release workflow's SLURM step builds no Nova, Placement or SLURM client and does not read the `openstack-slurm` document: it only checks that the Ironic node exists, then continues to the hand-back. ### Check the Prerequisites For SLURM adoption, the site needs: * a SLURM project in the site's OpenStack with an application credential — see [OpenStack Application Credentials](./openstack-application-credentials) for the access rules and the `openstack-slurm` document. If the project does not exist yet, prepare it with `automation bootstrap --project-prep-config`. * a Nova keypair in the SLURM project named `{deployment}-keypair` (created by the tofu bootstrap; the slurm-adopt activity injects this keypair into the Nova VM as `cloud-user`) * the provider network the SLURM nodes attach to * a published ARM64 (or x86\_64) image for the SLURM compute VMs * a Nova flavor for each device class the site adopts (e.g. `GB300` → `g.144.gb300.4`) The current umbrella also needs a SLURM controller, REST endpoint, and JWT so its agents can start. Burn-in additionally needs: * the expected GPU partition on the cluster * any required S3 credentials for artifact or benchmark flows * the hardware-test container images staged where the SLURM compute nodes can read them — if none are published yet for the site's CPU architecture, building them needs a cluster node with Enroot, in practice a GPU node; see [Stage the Hardware-Test Container Images](#stage-the-hardware-test-container-images) ### Configure OpenStack SLURM Adoption These values feed the slurm-adopt workflow that creates the Nova VM on each Ironic node. Set `enabled: true` for a site with the SLURM OpenStack integration and provide the project name and provider network. The per-role image, flavor, GRES, and user-data values live in Hub `SlurmBootstrapConfig` rows keyed by site and `DeviceRole`; they are not Helm values or environment variables. SLURM enrolment is keyed on `DeviceRole.slug` rather than `resource_class` because `resource_class` does not distinguish CPU architectures (x86 vs ARM nodes share the same `CPU_BAREMETAL` device class), and SLURM compute images differ per arch — see ADR-0025 in the repo for the full rationale. Set them in the deployment's `fleet-manager.yaml`: ```yaml theme={null} fleetManager: satellite: openstack: slurm: enabled: true targetProjectName: "[SLURM_PROJECT_NAME]" network: "[PROVIDER_NETWORK_UUID]" keyName: "{deployment}-keypair" ``` Create the corresponding Hub row through the `/api/v1/sites/{site_id}/slurm-bootstrap-configs/` API. Use the exact DeviceRole UUID for the role and provide the Nova image, flavor, optional GRES, and optional base64-encoded cloud-init user-data in that row. If you have the `cloud-user` SSH private key (used for ops to log in to SLURM compute VMs), also store it in the runtime vault as an `SSH Key` item — Fleet Manager does not consume it at runtime, but keeping it co-located with the rest of the site's secrets keeps recovery simple. ### Generate the SLURM JWT The current umbrella requires the SLURM REST endpoint, and a working client also needs its JWT. The JWT signing key (`/etc/slurm/jwt_hs256.key` and the matching `AuthAltTypes=auth/jwt` config in `slurm.conf`) is provisioned automatically by the [Fleet Manager SLURM appliance](https://github.com/nscaledev/fleetmanager-slurm-appliance). If the site does not use the appliance, follow the [upstream JWT setup](https://slurm.schedmd.com/jwt.html) on the controller before continuing. SSH to the SLURM login node as `hpcadmin` and create a long-lived token. `lifespan` is in seconds; the default of `1800` is too short for unattended Fleet Manager use, so set something effectively non-expiring: ```bash theme={null} sudo scontrol token username=hpcadmin lifespan=315360000 # 10 years ``` Tokens cannot be listed or revoked individually — to invalidate all outstanding tokens, rotate `/etc/slurm/jwt_hs256.key` on the controller and reissue. ### Store the SLURM JWT in 1Password Create a `Login` item called `slurm-api-prod` in the runtime vault — this is the name the worker's `jwtOpItem` points at across sites. ```bash theme={null} DEPLOYMENT="{deployment}" ENV="{env}" VAULT="Fleet Manager ${DEPLOYMENT} ${ENV}" SLURM_JWT="[PASTE TOKEN FROM THE PREVIOUS STEP]" op item create \ --category=login \ --title="slurm-api-prod" \ --vault="${VAULT}" \ "jwt[password]=${SLURM_JWT}" \ "notes=JWT token for SLURM REST API authentication" ``` ### Configure the Temporal Worker Point the worker at the SLURM REST endpoint and the JWT item created above. The endpoint is the SLURM login node's reachable address (typically a floating IP) on the SLURM REST port — use `https://` if the controller terminates TLS, otherwise plain `http://`. ```yaml theme={null} fleetManager: satellite: temporalWorker: slurm: baseUrl: "https://[SLURM_LOGIN_FLOATING_IP]:[SLURM_REST_PORT]" jwtOpItem: "slurm-api-prod" # Optional: deviceNamePrefix and apiVersion overrides. ``` ### Stage the Hardware-Test Container Images This step is only needed for burn-in. Burn-in tests use Enroot squashfs (`.sqsh`) images. Check the published S3 artifacts for the site's CPU architecture first; build only an artifact that is missing or changed. `images/build.sh` builds `hardware-tests-default`, `gpu-fryer`, and `nvloom`; other images follow their separate export path. Building requires a target-architecture host with Enroot, normally a GPU node. Follow the [hardware-test image build runbook](https://github.com/nscaledev/fleet-manager/blob/main/docs/runbooks/hardware-test-image-builds.md) for the build, publication, image-to-setting map, and troubleshooting. The current `k8s-deploy-fleet-manager` umbrella forwards local `*ContainerImage` destinations but does **not** forward the inner chart's `imageArch` or `imageSources`. Therefore, stage each `.sqsh` manually today. The build cache and the configured destination are **different locations**: the runbook builds under `/vast/enroot-cache//`, while `/home` and `/vast` are separate mounts on the SLURM appliance and Fleet Manager reads only the exact path configured below. After building, copy each `.sqsh` to the shared destination directory — current sites use `/home/hpcadmin/shared/enroot-cache`, which lives on the controller's \~1 TB home volume — and configure that absolute path. Any filesystem the compute nodes all mount works as the destination: at sites with VAST set up, a path under `/vast` (PB-scale) is equally valid and avoids controller home-volume capacity pressure. What matters is that every `*ContainerImage` value points at exactly where the files were copied. ```yaml theme={null} fleetManager: satellite: temporalWorker: hardwareTests: # Use the exact .sqsh basename the build produced — images/build.sh # prints each artifact path, and the runbook's image-to-setting map # lists which image feeds which setting. # Key the default image by CPU architecture. A single-architecture site # configures only its one key; a mixed-architecture site configures both. defaultContainerImage: x86_64: /home/hpcadmin/shared/enroot-cache/x86_64/[HARDWARE_TESTS_DEFAULT_SQSH_BASENAME] aarch64: /home/hpcadmin/shared/enroot-cache/aarch64/[HARDWARE_TESTS_DEFAULT_SQSH_BASENAME] gpuFryerContainerImage: /home/hpcadmin/shared/enroot-cache/[GPU_FRYER_SQSH_BASENAME] nvloomContainerImage: /home/hpcadmin/shared/enroot-cache/[NVLOOM_SQSH_BASENAME] ``` For an existing site migrating from the legacy scalar, deploy the compatible chart and Temporal worker first while the scalar remains, then apply the architecture-keyed map. Do not apply the map to an older chart: it renders the map as one scalar string and the old worker treats that text as an image path. Runner tarballs are selected at runtime from the compute node's `uname -m`; do not set the deprecated `runnerTarballArchitecture` override. Architecture-keyed default images are currently supported for single-node hardware tests only. A multi-node test must retain the legacy scalar until its SLURM allocation can be constrained to one CPU architecture. Do not add `imageArch` or `imageSources` directly to this umbrella values file; they are currently dropped. With manual staging there is no workflow to run — copying the `.sqsh` files into place is the whole step. Once the umbrella forwards `imageArch` and `imageSources`, the S3 install path replaces manual staging: configure every source with its matching `*ContainerImage` destination, then run the `Prepare hardware-test images` workflow with the `Ensure hardware-test images` preset against one active compute device in `validation`, as defined by [ADR-0033](https://github.com/nscaledev/fleet-manager/blob/main/docs/adr/0033-s3-backed-hardware-test-images.md). ### Validate Connectivity Shell into a Temporal worker pod and query the SLURM API: ```bash theme={null} kubectl exec -it -n fleet-manager deployment/temporal-worker -- /bin/bash ``` ```bash theme={null} SLURM_BASE_URL="[INSERT BASE URL]" SLURM_JWT="[INSERT JWT TOKEN]" curl -sk \ -H "X-SLURM-USER-TOKEN: ${SLURM_JWT}" \ -X GET "${SLURM_BASE_URL}/slurm/v0.0.43/nodes" ``` ## Deploy and Argo Sync This is the **Apply / Deploy** step from the flow above — Fleet Manager goes live when it completes. Open the `k8s-cluster-fleet-manager` pull request with the deployment configuration from the previous phases, get it reviewed, and merge it. If the hub change from [Enable the Hub to Accept the Satellite](#enable-the-hub-to-accept-the-satellite) travels in a separate PR, merge and sync it first — the satellite agents cannot authenticate until the hub expects them. The maintained template enables autosync on the parent Application `aoa-deploy-fleet-manager`; that parent creates the child `fleet-manager-satellite` Application. If autosync is disabled, sync the parent against the target cluster: ```bash theme={null} argocd app sync aoa-deploy-fleet-manager ``` During the deploy, the chart's ExternalSecrets copy the bootstrap-written Vault values into Kubernetes — no manual Secret creation is needed on the happy path (see [Generate Service Account Tokens](#generate-service-account-tokens)). Confirm the Application reports Synced and Healthy and the Fleet Manager pods start. Crash loops at this point usually mean a missing required Helm value (see [Set the Satellite Identity and Endpoints](#set-the-satellite-identity-and-endpoints)) or a missing runtime-vault item (see [Add Runtime Service Credentials](#add-runtime-service-credentials)). Once the deployment is live, log in at [fleet.nscale.com](https://fleet.nscale.com) and continue with validation. ## Run Post-Deployment Validation Treat validation as part of the deployment, not as a later cleanup step. At minimum, verify: * device ingestion from NetBox * end-to-end enrolment * Temporal workflow activity * Grafana logs and dashboards * burn-in smoke tests where SLURM is enabled Use [Site Validation](./site-validation) for the concrete checks and promotion expectations. ## Log In and Request Access Once Fleet Manager is live (after Apply / Deploy), it's at [fleet.nscale.com](https://fleet.nscale.com) (dev and prod). Anyone with an `@nscale.com` email gets **read-only access automatically** — just log in. Posting in [`#fleet-manager-chat`](https://grid-nscaleworkspace.enterprise.slack.com/archives/C097P9ZLQTB) triggers the **"Fleet Manager - request user access"** workflow bot, which replies with the steps: 1. **Log in** to Fleet Manager with your Nscale email to create your user. * If you're stuck on the Vercel screen in dev, use the access password from the bot message. 2. **If you need access above read-only** (e.g. to run jobs), fill out the form (Request User Access) to update your permissions. If you already have an account and are getting errors, check whether there's an incident before requesting access. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.