Skip to main content
  • Kind: Dependency
  • Version: 3
  • Fleet Manager owner: Fleet Manager squad, #fleet-manager-chat (individual claimant pending)
  • Counterpart owner: Fleet Manager squad operates the cluster; consumer teams (CHAD, Radar, Microsoft, Google) are listed on their own pages
  • Last reviewed: 2026-09-18

​
What flows

Fleet Manager runs its own NATS JetStream plane (ADR-0031, spec 0030, runbook docs/runbooks/nats-operations.md): nats.fleet-manager.glo1.nscale.com in production (3 servers, R3 streams) and nats.fleet-manager-dev.glo1.nscale.com in development (1 server, R1). Every Fleet Manager producer and consumer in glo1 production and development is pointed at it through the k8s-cluster-fleet-manager overlay: the NetBox inventory agent, the Matrix connector, the Radar ingestion agent and the CHAD event publisher as producers, CHAD’s consumer, and the Google connector in development. Radar’s radar-ingest consumer in the observability estate also binds here. The shared product-mgmt cluster is no longer in the path: the hub chart defaults every client’s nats.tls.secretName to its fleet-<client>-client-tls Secret and renders no certificate objects of its own, so returning to product-mgmt is a chart revert, not a values change. Streams in the fleet account, all byte-capped with discard: old: fleet_jobs (CHAD postcards, 90 d, 1 GiB), inventory_devices (device and cable events, 90 d, 1 GiB), radar_devices (7 d, 4 GiB) and radar_devices_dlq. Durable consumers are declared as NACK Consumer CRDs for the Matrix and Google connectors and for radar-ingest; CHAD self-creates its durables and its four CHAD_FM_*_DLQ streams client-side, and those are CHAD’s to operate. The connectors bind an existing durable and refuse to start if it is missing or its filter differs. Postcard production stays gated on chart defaults (see chad); the plane carries whatever the overlays enable.

​
What breaks if this changes

A plane outage stops the Radar projection, connector deliveries and postcards at once. Running connector pods fall back to their polling backstops and the outbox retains rows, but Radar goes stale. A connector pod that starts or restarts during the outage does not fall back: both src/fleet_manager/api/connectors/matrix/main.py and src/fleet_manager/api/connectors/google/main.py await the NATS connect before the backstop task is created, so the pod crash-loops until the plane is reachable again. Retention is min(maxAge, maxBytes): a stream sitting at its byte cap sheds oldest messages, so a consumer that lags past the effective window loses events silently. Development has no HA; a pod restart there is a short outage that clients reconnect through on their own. A stream or subject change needs every consumer page updated.

​
Enforcement

  • Principal: one client certificate per component, identity is the CN (fleet-<component>-client, for example fleet-matrix-connector-client, fleet-chad-client); no passwords, NKEYs or JWTs.
  • Mechanism: Fleet Manager is its own CA. cert-manager in the nats namespace issues the server certificate and one client Certificate per CN; each client cert is pushed to the Fleet Manager Vault mount fleet-manager at <env>/hub/nats/<client>/client-tls and materialised back into the fleet-manager namespace as fleet-<client>-client-tls (see hashicorp-vault). The fleet account’s CN allowlist is the whole authorisation boundary and it is flat: any allowed CN can publish, subscribe or purge, so the producer and consumer split is a convention. Subject grammar in src/fleet_manager/core/events/subjects.py.

​
Gaps

  • The CN allowlist has no per-role ACL (spec 0030 open question), so a compromised consumer certificate can publish.
  • denyPurgeStreams stays false in production until post-cutover validation; the legacy product streams are retained meanwhile.

​
Change log

  • 2026-09-18 v3: hub chart no longer renders the Product Vault NATS certificate path (FLEET-2692); rollback is a chart revert. ADR-0031 accepted.
  • 2026-09-16 v2: rewritten for the Fleet Manager-owned plane after the cutover; streams, identities and certificate flow from docs/runbooks/nats-operations.md.
  • 2026-09-09 v1: created; described the shared product-mgmt cluster.