Skip to main content
  • Kind: Observation (Fleet Manager also writes: the partition moves are its own workflows, so this page is filed under the state Fleet Manager derives from the fabric)
  • Version: 1
  • Fleet Manager owner: Fleet Manager squad, #fleet-manager-chat (individual claimant pending)
  • Counterpart owner: unassigned (the NMX-M controllers are site networking infrastructure)
  • Last reviewed: 2026-09-09

​
What flows

NvLinkIsolateWorkflow and NvLinkReturnWorkflow (src/fleet_manager/temporal/activities/nvlink.py) move a returned break-fix tray’s GPUs between the default NVLink partition and a repair partition on the rack’s NMX-M controller. The worker resolves the controller per Fleet Manager rack name from nmx_hosts and talks to https://<host>/nmx/v1 through the vendor-isolated client in src/fleet_manager/core/datacenter/nmx/vendors/nmxm/client.py (read the domain, GPUs and partitions, replace or create or delete a partition, poll the operation). Partition membership is read back from /nmx/v1/gpus after every move and recorded on the device as the durable NvLinkGpusInRepairPartition condition, source nvlink-fabric (src/fleet_manager/core/conditions/registry.py). That condition does not move operational status; eligibility reads it and release to the customer pool refuses while it is asserted (release_to_customer_pool in src/fleet_manager/core/eligibility/policies.py, failure code RELEASE_PARTITION_LATCH_REFUSED). Real writes are gated per site: temporalWorker.nvlinkPartitionAllowed in the satellite chart defaults false and nmx_activity_args_ready defaults false, so a deployment on chart defaults runs the moves against the simulated fabric and never contacts a controller. Where a site enables them, nmx_hosts must carry the controller’s domain_uuid and the move fails closed if the controller reports a different domain.

​
What breaks if this changes

An NMX-M API change (the client is pinned to the 4.021.34 shapes) or an unreachable controller fails the move workflows before mutation; a move that mutates but cannot be verified leaves the latch at Unknown, which blocks release until the same move is re-run. A wrong nmx_hosts entry points a rack’s moves at another rack’s controller, which the domain_uuid check is there to catch. If the latch write to Fleet Manager fails after the fabric moved, the activity fails and the operator re-runs the move, which no-ops on the fabric and records the latch.

​
Enforcement

  • Principal: the satellite temporal-worker (towards Fleet Manager) and the NMX-M service account (towards the controller).
  • Mechanism: towards Fleet Manager, the temporal-worker FM OAuth client (private-key JWT) in the satellite roster writing device-condition:create; towards NMX-M, HTTPS with the service account from the site’s networking 1Password vault item named by temporalWorker.nmxCredentialsItem (default svc-nmx-login, src/fleet_manager/settings/agents/temporal_worker.py), read only when real writes are enabled.
  • Settings: src/fleet_manager/settings/agents/workflow.py (nvlink_partition_allowed, nmx_hosts, nmx_activity_args_ready).

​
Gaps

  • No counterpart owner: nobody outside Fleet Manager is named for the controllers, their firmware or the service account.
  • No contract test against a real NMX-M; the client shapes were captured by hand from one firmware version.

​
Change log

  • 2026-09-09 v1: created after the repair-partition latch landed on main (#2818); describes present behaviour.