Skip to main content
  • Kind: Observation (Fleet Manager also writes: Sync Intents and sync triggers for the trays its own workflows remove or swap, so this page is filed under the topology Fleet Manager reads)
  • Version: 3
  • Fleet Manager owner: Fleet Manager squad, #fleet-manager-chat (individual claimant pending)
  • Counterpart owner: Fleet Manager squad (the service moved to Fleet Manager ownership in July 2026)
  • Last reviewed: 2026-10-09

​
What flows

The hub topology-agent (src/fleet_manager/agents/topology/) reads active topology snapshots from the nscaledev/topology-service API for a one-to-one mapping of topology regions to Fleet Manager sites, polling every 30 min with a 10 min resync, and rebuilds TopologyGroup trees (root, super-spine, spine, pod, NVLink domain) with device membership. topology-service itself reads NetBox. The groups feed burn-in cohorts, the Radar projection’s parent resources (behind emitResourceHierarchy) and Reservation’s placement reads. NVLink-domain health does not consume them yet; that join is deferred (see mimir). The Fleet Manager hub API also reads the per-Region sync status. Its Helm pre-install/pre-upgrade gate waits for the topology-service /internal/v1/regions/{region}/sync contract before allowing the API release to proceed. Two satellite workflows write to topology-service through the hub, since satellites cannot reach it: POST /api/v1/sites/{id}/topology-sync/ forwards a Sync Intent naming the trays whose removal a later sync should publish rather than hold (expect_nodes_absent) to POST /internal/v1/regions/{region}/sync, which also triggers a Region sync. A cableless node swap declares its failed tray (node-swap-<job id>), and an unenrolment its tray when moving it to staged takes it out of discovery (unenrollment-<device>-<job id>). Each declares before the NetBox write that removes the tray (an unenrolment before its first write, rechecking just before that write) and posts the intent again after it to trigger the sync. A cabled node swap instead declares both trays as one swapped pair (expect_nodes_swapped, under node-swap-<job id>) once, straight after the NetBox rewrite: topology-service pins the pair to where both sit in the topology it is serving when the intent arrives, and the same call triggers the sync that publishes the exchange. The Day-0 netbox_only rewrite of two cabled trays does the same. Unlike a removal, a failed swap declaration does not fail the workflow; the Region holds until the pair is declared.

​
What breaks if this changes

A bad snapshot import blocks burn-in verdicts for a whole site (ADR-0029 fails loud on missing topology membership) and removes rack and pod parents from the Radar projection. A region-to-site mapping error attaches devices to the wrong site’s groups. When the sync trigger is unavailable, a cableless node swap and the unenrolment of an active tray fail before their first write as TOPOLOGY_SYNC_INTENT_FAILED.

​
Enforcement

  • Principal: topology-agent.
  • Mechanism: towards Fleet Manager, FM OAuth client (private-key JWT) in the hub chart roster; towards the topology API, an unauthenticated in-cluster HTTP client on the per-region base URL from src/fleet_manager/settings/agents/topology.py (src/fleet_manager/agents/topology/main.py); there is no credential to rotate on that side.
  • Hub roster client topology-agent (topology-group:create|update|rebuild) in deployment/charts/hub/values.yaml.
  • Hub roster client temporal-worker holds site:topology_sync for the sync-trigger proxy; the hub API then calls topology-service unauthenticated in-cluster, as the agent does (src/fleet_manager/core/clients/topology_service.py).
  • Settings: src/fleet_manager/settings/agents/topology.py (topology_api_base_url, region targets).
  • Releases of topology-service pin through k8s-deploy-fleet-manager.
  • The hub chart’s API contract-gate Job blocks its release until the sync-status endpoint is live.

​
Gaps

  • Both sides are the same squad, so the review discipline relies on the file rather than on a second team.

​
Change log

  • 2026-09-09 v1: created; describes present behaviour.
  • 2026-10-05 v2: the hub’s sync-trigger proxy, and the Sync Intents the cableless node swap and unenrolment declare through it.
  • 2026-10-09 v3: the cabled node swap (and netbox_only rewrite) declares the swapped pair.