Skip to main content
  • Kind: Actuation (CHAD is also a reader: it polls devices, jobs and test executions through the SDK, but this page is filed under the direction that changes hardware)
  • Version: 1
  • Fleet Manager owner: Fleet Manager squad, #fleet-manager-chat (individual claimant pending)
  • Counterpart owner: unassigned, SRE (CHAD) with FOE co-owning the scheduler, #chad-chat
  • Last reviewed: 2026-09-09

​
What flows

CHAD is the repair-and-return orchestrator. Fleet Manager can publish thin notify-and-pull events for it (ADR-0027) on the 6-segment subject grammar: device.standup-requested, job.state-updated, device.triage-requested and device.validation-requested. Every one of them is gated on the producer side: api.eventing.standupEnabled, triageRequestedEnabled and validationRequestedEnabled each default false, job events are emitted only for workflow types listed in api.eventing.jobWorkflowAllowlist (empty by default), and the outbox is drained to NATS only when chadEventPublisher.enabled is true (default false). A deployment on chart defaults publishes nothing; the per-environment overlays turn each event on. Per ADR-0027 the CHAD-side consumer is likewise a migration target that is disabled by default: CHAD reads job state by polling the API today, and standup, triage and validation still start from the existing manual or triggered paths. Its SOP recipes in nscaledev/chad-sops call Fleet Manager workflows by name and create jobs through POST /api/v1/jobs/, and CHAD’s own code talks to Fleet Manager through the Python SDK in libs/fleet-manager-sdk, which is its primary interface. Two clients write: chad-client (production) and chad-dev-client (development) each hold job:create, job:cancel, test-execution:mark_stale, stale-hold read and undo, device:update, and read scopes on devices, jobs, test executions, fault snapshots and impairment events (operations:read). chad-test is read-mostly: it holds device:read, device:update, job:read, job-device-status:read, test-execution:read and device-fault-snapshot:read, and cannot create or cancel jobs. Fleet Manager applies eligibility (ADR-0012) and custody to every job and returns failure codes per ADR-0014. Idempotency is caller-supplied: POST /api/v1/jobs/ accepts an optional idempotency_key scoped to the calling principal, and a request without one is not idempotent, so CHAD must send a key on every create it may retry. Fleet Manager does not restrict which workflow types a client may create; any workflow the device is eligible for can be dispatched by any client holding job:create.

​
What breaks if this changes

A change to job status semantics breaks CHAD’s polling of job state today, and a change to an event contract will break its consumer once enabled. Job admission already guards exclusive workflows: an incoming exclusive job is rejected under a device-row lock when any non-read-only job is active on those devices, and a non-exclusive job (burn-in, connectivity verification) is rejected when an exclusive job is active. Two non-exclusive jobs may still overlap on the same device set, so a new dispatcher racing operators on burn-in dispatch is the one unguarded case (src/fleet_manager/api/core/workflows/jobs.py, JobDeviceStatus.validate_no_active_workflow_conflicts). A device an active node swap names as its spare counts as that swap’s device, so a day-zero enrolment CHAD dispatches for it gets the same 400 conflict until the swap finishes.

​
Enforcement

  • Principal: chad-client, chad-dev-client, chad-test.
  • Mechanism: towards Fleet Manager, FM OAuth clients in the hub chart roster; the outbox publisher reaches NATS as fleet-chad-event-publisher-client and CHAD consumes as fleet-chad-client on the Fleet Manager-owned plane (see nats).
  • Hub roster clients chad-client, chad-dev-client, chad-test in deployment/charts/hub/values.yaml.
  • Event contracts: src/fleet_manager/core/events/notification.py; ADR-0027.
  • Failure codes: ADR-0014.

​
Gaps

  • No per-client allow-list of workflow types.
  • No overlap guard for two non-exclusive jobs on the same device set; exclusive workflows are already guarded.
  • CHAD’s own scheduler and its Fleet Manager wake-up consumer are being built; that path is not described here until it is enabled.

​
Change log

  • 2026-09-09 v1: created; describes present behaviour.