Build a New Agent

Agents in Fleet Manager pull scoped data from a source system, cache it in-process, and reconcile it into a target system. They are built on a small set of primitives in src/fleet_manager/agents/machinery and follow the shape of src/fleet_manager/agents/topology/ — the cleanest reference implementation.

source client list() -> ListReflector -> Store -> Controller -> Reconciler -> target client upsert()

Use one Store + ListReflector per source table, one Controller per reconciler, a shared WorkerPool, and a single Manager that runs the whole thing.

Use the /agent-creator skill. When building a new agent, invoke it — it knows this architecture, the file layout, and the canonical wiring, and will scaffold the agent end-to-end. This page is the reference behind that skill; read it if you need to deviate from the default shape or understand what the skill is producing.

Naming conventions — pick these first

Agent names drift across six places. Pick all of them deliberately before writing code; silent mismatches between the ServiceType value, cert CN, and ensureagentclient name will look like auth bugs at runtime. The cert CN ≡ ensureagentclient name ≡ FM_CLIENT_ID link comes from Fleet Manager’s OAuth flow — the client submits FM_CLIENT_ID as client_id, the API loads the Application by that value, then verifies the leaf cert CN matches it. See Client Authentication for the mechanism and Scopes and Permissions for how the scopes you pass to ensureagentclient are enforced. The pattern for most hub agents is:

Piece	Convention	Example (`netbox`)
Settings module	`src/fleet_manager/settings/agents/<stem>.py`	`netbox.py`
Service module	`src/fleet_manager/settings/services/<stem>_agent.py`	`netbox_agent.py`
`ServiceType` enum value	kebab-case, ends in `-agent`	`netbox-agent`
Chart folder	`deployment/charts/hub/templates/<name>-agent/`	`netbox-agent`
`ensureagentclient` name (= cert CN)	same as chart folder name	`netbox-agent`
Env var prefix (Pydantic settings)	uppercase of service module stem	`NETBOX_AGENT_`

Satellite agents (anything deployed into a satellite cluster) get one client per satellite — the name is suffixed with the satellite ID (workflow-agent-glo, workflow-agent-sta), declared once as a namePrefix under api.migration.satelliteClients and expanded per expectedSatellites entry. Hub agents get a single unqualified client.

1. Define the sync scope

Decide these before writing code:

Source tables and the target entities they map to.
A stable key per table (usually the source ID; fall back to composite keys only if necessary).
Filters per table (API filters, org scope, active-only). Filters belong on the agent settings, not hard-coded in main.py.
The target client (Fleet Manager, NetBox, Kusto, NATS, etc.).

If the agent is read-mostly and has no target writes (e.g. a ingestion-style agent), it still uses the same machinery — the reconciler just publishes instead of calling an upsert.

2. Know the machinery

The machinery lives in src/fleet_manager/agents/machinery/. You do not subclass it; you compose it. How it runs: a reflector polls the source and updates a store, which emits events. The controller translates each event into a set of reconcile keys and pushes them onto a dedup queue — the key stays in place if a duplicate is pushed, and the same key never dequeues twice concurrently. The queue has two lanes: changes go in the change lane, and re-checks of objects that have not changed (the periodic resync, a full refresh’s REFRESHED events, the store’s first listing after a restart) go in the resync lane. The change lane goes first, apart from every tenth turn, which goes to the resync lane so a busy change lane can’t starve it. A change moves a key already waiting in the resync lane. The controller hands the worker pool one key per worker, so the backlog stays in the queue where a change can still jump it. Workers pull keys, run the reconciler (which reads the latest state from the store), and ack when done. Failures requeue with clamped exponential backoff, in the lane they came from. This gives you a few useful properties for free:

Reconciles only run for objects that actually changed (with optional periodic resync).
A change never waits behind a resync sweep, however long the sweep is.
The reconciler always reads the freshest state from the store — queued keys can’t go stale.
A fast-changing object is throttled (one reconcile in flight per key) without starving other keys.
Total concurrency is bounded by the worker pool.

The primitives:

Store[K, V] (store.py) — typed in-memory cache. Emits CREATED / MODIFIED / DELETED events. Raises StoreNotInitialisedError if read before the initial load.
ListReflector[K, V] (reflector.py) — periodically calls an async list function and replaces the store contents. Handles transient failures with exponential backoff. Dies after max_retries + 1 consecutive failures, which kills the agent — this is intentional.
Controller (controller.py) — subscribes to stores and schedules reconciles through the worker pool. Handles requeue and backoff.
WorkerPool (worker_pool.py) — bounded concurrency for reconcile tasks. One pool per agent is usually enough.
Manager (manager.py) — owns reflectors, controllers, the worker pool, and telemetry. Call await manager.run() to start the agent.
AgentCollectorBundleBuilder (telemetry.py) — builds per-component telemetry collectors with a single agent name.
Result / RequeueRequest — how reconcilers signal success, requeue, or delay.

3. Create stores and reflectors

One Store + ListReflector per source table. If the agent has several tables that take the same setup, wrap them in a small helper that takes the list function and interval and injects the telemetry collectors — see create_nb_store in src/fleet_manager/agents/netbox/main.py. For agents with only two or three stores, constructing them inline (as src/fleet_manager/agents/topology/main.py does) is fine. Filters belong on the agent settings class, not hard-coded in main.py. For single-filter resources, pass the client list method straight to the reflector. For filter-per-cluster setups that need to fan out across many filter combinations, use the iter_devices_for_filters pattern in src/fleet_manager/agents/netbox/main.py.

4. Implement the reconciler

A reconciler is any Callable[[K], Awaitable[Result | None]] — a class with __call__, or a bare async function. It takes a key, reads whatever it needs from the stores, resolves dependencies (e.g. site → location → rack) and performs exactly one target write path.

Error handling — when to requeue

The controller distinguishes three outcomes. Pick the right one and the queue handles the rest; pick the wrong one and you either hot-loop or drop work silently.

Outcome	Signal	When to use
Success	return `None` or `Result(requeue=False)`	the reconcile finished (including intentional no-ops).
Known transient miss	raise `RequeueRequest(requeue_after=...)`	a dependency isn’t in its store yet (parent not reflected, downstream eventually-consistent read). Uses your delay verbatim — skip the controller’s backoff.
Unknown transient failure	let the exception bubble	timeouts, 5xx from the target, connection resets. The controller catches, logs, and requeues with clamped exponential backoff.
Permanent failure	raise `FatalError`	the input will never be valid (bad schema, contradictory state). The key is dropped, not requeued.

Two mistakes to avoid:

Swallowing transient errors. If you try/except Exception: return None around an upsert, the controller thinks the reconcile succeeded and the object never gets retried. Let transient errors bubble.
Raising RequeueRequest with a 0-second delay in a tight loop. If the dependency is chronically missing, the queue hot-loops. Pass a sensible requeue_after (the canonical agents use 1–5s) and — where possible — subscribe the controller to the dependency’s store so the reconcile is triggered again when the dependency arrives, not by polling.

See src/fleet_manager/agents/topology/reconciler.py for the canonical shape, and src/fleet_manager/agents/radar_ingestion_service/ for RequeueRequest on transient NATS errors.

5. Wire the controller

One Controller per reconciler. Each controller subscribes to the store(s) whose events should drive that reconciler, mapping each event to a list of reconcile keys (a single source event can fan out to multiple target keys). Set resync_interval > 0 on a subscription when the target can drift silently from the source (e.g. external edits); leave it 0 when store events alone are authoritative.

6. Compose the agent main

main.py is the wiring layer: instantiate clients, build the AgentCollectorBundleBuilder, create stores/reflectors/controllers/worker pool, hand everything to the Manager, call await manager.run(). Note that the service settings class nests the agent-specific config under a field named after the agent (e.g. settings.netbox_agent.*) — dereference through that field, not the top-level settings object. Don’t forget manager.add_collector(bundle.build()), or you get no telemetry.

7. Add agent and service settings

Create src/fleet_manager/settings/agents/<agent>.py with a Pydantic model for the agent’s own config (filters, intervals, worker count, cert paths, telemetry name, any NATS subject names). Create src/fleet_manager/settings/services/<agent>_agent.py that inherits BaseFleetManagerSettings and composes the agent config with its client settings. Mirror NetBoxAgentSettings in src/fleet_manager/settings/services/netbox_agent.py. If the agent talks to a new external system, add its client settings under src/fleet_manager/settings/clients/. Register the service in src/fleet_manager/settings/loaders.py:

Add a ServiceType enum entry.
Add the SERVICE_TYPE_MAP row.
Add a get_<agent>_settings() accessor that delegates to _get_cached_settings(...).

8. Wire the CLI

Add src/fleet_manager/cli/commands/infra/<agent>_agent.py with a Typer app that exposes a single run command invoking asyncio.run(main()). Register it from src/fleet_manager/cli/commands/infra/main.py with app.add_typer(...). Copy the shape from src/fleet_manager/cli/commands/infra/netbox_agent.py. Add a CLI test under tests/cli/commands/infra/ that patches asyncio.run and the agent’s main.

9. Wire local deployment

Local deploy = Docker Compose. Edit deployment/docker/docker-compose.app.yml. Two up-front decisions change what you need:

Does the agent call the Fleet Manager API? If yes (most agents), you need steps a, b, and the FM API auth vars in c. If the agent only writes to an external system (e.g. netbox-matrix-agent → Kusto), skip a and b and omit those vars.
Is the agent satellite-scoped? In local compose, only add per-satellite certs / clients / services when the stack already simulates multiple sites for that agent. Today workflow-agent and diagnostics-agent use suffixed local entries (-glo, -sta). Follow the existing pattern in compose rather than assuming every satellite agent has suffixed local services.

a. Generate the mTLS cert

Under fm-generate-certs, add one line per client the agent will use. The name becomes the cert filename and the leaf cert’s Common Name — which must match the ensureagentclient name in the next step and the FM_CLIENT_ID the agent submits in the token request.

generate_cert <client-name>          # hub agent, e.g. generate_cert netbox-agent
generate_cert <client-name>-glo      # satellite agent: one per simulated satellite
generate_cert <client-name>-sta

b. Register the FM API client

fm-api starts by running manage bootstrap, which registers every client listed in deployment/docker/bootstrap-clients.yaml. Add one entry per client with the exact scopes your agent needs (see Scopes and Permissions for how to pick them):

clients:
  # ... existing entries
  - name: <client-name>
    scopes:
      - "organisation:read"
      - "site:read"
      - "device:read"

This is separate from the helm roster in Section 10 — both must list the client. Without the compose manifest entry, local token acquisition at /o/token/ fails with 401 (no Application for the submitted client_id); without the helm entry, the same 401 happens in real clusters.

c. Add the agent service

Add a service stanza modeled on fm-netbox-agent:

fm-<agent>-agent:
  image: ${FLEET_MANAGER_IMAGE}
  depends_on:
    fm-generate-certs: { condition: service_completed_successfully }
    fm-api: { condition: service_healthy }
  command: ["uv", "run", "fleetctl", "infra", "<agent>-agent", "run"]
  environment:
    SERVICE_TYPE: <agent>-agent                     # must match ServiceType enum value
    ENVIRONMENT: "${ENVIRONMENT:?ENVIRONMENT must be set}"
    # Only if the agent calls the Fleet Manager API:
    FLEET_MANAGER_CLIENT_BASE_URL: http://fm-api:8000/
    <PREFIX>_FM_CLIENT_ID: <agent>-name              # must match ensureagentclient name
    <PREFIX>_FM_API_CERT_PATH: /etc/fleet-manager/certs/<agent>-name.crt
    <PREFIX>_FM_API_KEY_PATH: /etc/fleet-manager/certs/<agent>-name.key
  volumes:
    - fm_certs:/etc/fleet-manager/certs
  networks:
    - default

<PREFIX> is the Pydantic env prefix from your agent settings class (NETBOX_AGENT_, RADAR_INGESTION_, WORKFLOW_AGENT_, …). Get this wrong and the agent starts but can’t find its cert. The four FLEET_MANAGER_CLIENT_BASE_URL / *_FM_CLIENT_ID / *_FM_API_CERT_PATH / *_FM_API_KEY_PATH variables are only needed if the agent calls the FM API. Agents that write only to external systems (e.g. netbox-matrix-agent → Kusto) omit all four and the fm_certs volume mount.

d. Run it end-to-end

make docker-up
docker compose -f deployment/docker/docker-compose.app.yml logs -f fm-<agent>-agent

First-pass log signals to look for:

“Starting Fleet Manager <Agent> agent” — the CLI entry ran.
Reflector startup messages per store (“devices”, “sites”, …).
No StoreNotInitialisedError after the first poll interval.
Target writes / publishes showing up on the other side (Fleet Manager API, NATS, Kusto, …).

e. Common local-run failures

invalid SERVICE_TYPE at startup — the value in the compose stanza isn’t in the ServiceType enum. Add it to src/fleet_manager/settings/loaders.py first.
FileNotFoundError on the cert path — the generate_cert line is missing or the filename doesn’t match <PREFIX>_FM_API_CERT_PATH.
HTTP 401 at /o/token/ (token never issued) — one of: no entry for the submitted client_id in deployment/docker/bootstrap-clients.yaml (so no Application record exists), the cert CN doesn’t match the FM_CLIENT_ID, or the JWT’s iss/sub don’t match. Check fm-api logs — authenticate_client logs the specific reason.
HTTP 403 at an API endpoint — the token was issued but its scopes don’t cover the call. Add the missing scope to the agent’s scopes list in that manifest.

10. Decide the deployment topology

Confirm whether the agent runs in the hub cluster or a satellite cluster. Hub agents (netbox-agent, radar-agent, topology-agent) have a single client. Satellite agents (workflow-agent, openstack-agent, slurm-agent, diagnostics-agent, …) have one client per satellite. A process name and its client id can differ — the OpenStack agent authenticates as ironic-agent-{satellite} (ADR-0036).
If the agent calls the Fleet Manager API, add it to the client roster in deployment/charts/hub/values.yaml: hub agents go under api.migration.clients (name + scopes), satellite agents under api.migration.satelliteClients as a namePrefix — the chart expands that once per expectedSatellites entry into <namePrefix>-<satellite>. See Scopes and Permissions for picking the right scopes — missing scopes cause silent 403s.
Add a Helm template for the agent deployment under deployment/charts/hub/templates/<agent>/ (or deployment/charts/satellite/...). Mirror an existing agent (e.g. netbox-agent) for the Deployment, Service, and any ConfigMap/Secret references.

11. Roll it out through the deploy repos

Real-cluster deploys go through two more repositories. Both are Argo CD driven.

k8s-deploy-fleet-manager — wrapper Helm chart that feeds values into the upstream deployment/charts/hub / satellite charts. Also owns shared ExternalSecrets.
k8s-cluster-fleet-manager — per-cluster GitOps repo. One folder per cluster/environment (e.g. glo1/production/, sys2-sta1/production/) with an Argo Application that points at k8s-deploy-fleet-manager and supplies cluster-specific values.

In `k8s-deploy-fleet-manager`

Add defaults in charts/fleet-manager/values.yaml under fleetManager.hub.<yourAgent> (or fleetManager.satellite.<yourAgent>): workerCount, any agent-specific config fields, and resources.
Pass them through in charts/fleet-manager/templates/fleet-manager/application-hub.yaml (or application-satellite.yaml) so the upstream chart receives them.
If the agent needs Vault-backed credentials, add an ExternalSecret under charts/fleet-manager/templates/fleet-manager/ following external-secret-hub-op-token.yaml.
Add a representative values sample to ci/test-hub.yaml or ci/test-satellite.yaml so chart tests cover the new agent.
Bump fleetManager.chartVersion when cutting a new upstream chart release that includes the agent.

In `k8s-cluster-fleet-manager`

Open the target cluster/environment file, e.g. glo1/production/fleet-manager.yaml.
Enable the agent and set its real values under fleetManager.hub.<yourAgent> (or fleetManager.satellite.<yourAgent>).
If the agent is deployed as a standalone Argo Application (i.e. not part of the main fleet-manager Helm chart, like uni-filestorage-agent-*.yaml under sys2-sta1/production/), add a new manifest file and register it in kustomization.yaml.

End-to-end example — `netbox-agent`

This is the canonical path. Follow the same chain for a new agent.

fleet-manager — code and helm template:
- src/fleet_manager/agents/netbox/ (agent code, reconciler, main.py).
- src/fleet_manager/settings/services/netbox_agent.py and the ServiceType.NETBOX_AGENT entry in loaders.py.
- src/fleet_manager/cli/commands/infra/netbox_agent.py (fleetctl infra netbox-agent run).
- deployment/charts/hub/templates/netbox-agent/ (Deployment + Service + ConfigMap).
- deployment/charts/hub/values.yaml adds a netbox-agent entry under api.migration.clients for FM API scopes.
k8s-deploy-fleet-manager — chart defaults:
- charts/fleet-manager/values.yaml (fleetManager.hub.netboxAgent) — default workerCount, deviceFilters: [], deviceRoleClassMapping: {}, resources.
- charts/fleet-manager/templates/fleet-manager/application-hub.yaml passes those values into the upstream Helm chart.
- charts/fleet-manager/templates/fleet-manager/external-secret-ghcr.yaml provides image-pull creds; the 1Password token (external-secret-hub-op-token.yaml) is what the agent reads credentials from.
k8s-cluster-fleet-manager — real cluster values:
- glo1/production/fleet-manager.yaml sets fleetManager.hub.netboxAgent.workerCount, the production deviceFilters (cluster_id + role__in per cluster), and the deviceRoleClassMapping (gpu-node: gpu_compute, etc.).
- Argo CD picks up the change and syncs to the glo1 hub cluster.

Fleet Manager releases move configured agent image pins based on labels on the fleet-manager Release Please PR. The final tags are visible in deployment/charts/hub/release/agent-versions.yaml before merge and are bundled with the released hub chart; for example, the current policy maps netbox-agent to netboxAgent.image.tag and netbox-matrix-agent to netboxMatrixAgent.image.tag. Those labels are configured in .github/agent-release-policy.json, so another independently pinned agent should add its own policy entry instead of hard-coding workflow logic. The deploy wrapper only promotes the chart version in charts/fleet-manager/values.yaml (fleetManager.chartVersion).

12. Produce memory consumption estimates

Every new agent gets a memory_consumption_estimates.md next to its code. Follow the structure of src/fleet_manager/agents/radar_ingestion_service/memory_consumption_estimates.md:

State topology assumptions (number of sites, racks, devices, types, etc.).
Measure retained size of representative objects with sys.getsizeof + gc.get_referents.
Give formulas for total memory as a function of the assumptions.
Include a scaling table and a mitigation section (e.g. “partition by site using filters”).

This is how we decide replica counts and whether an agent needs sharding before it ships.

13. Tests

Follow the Unit Tests guidance in the repository root AGENTS.md. Minimum agent-specific coverage:

tests/agents/<agent>/test_reconciler.py — one test per reconcile branch (create, update, reuse, skip, requeue).
Fixtures for mock_src_client, mock_target_client, and Store instances seeded via a small protocol-based seed helper.
RequeueRequest assertions when required dependencies are missing.
Explicit verification of target client .create() / .update() calls, including payload fields.
A smoke test that wires the real Controller + WorkerPool (or their fakes in tests/agents/machinery/) and drives a reconcile via a real store event, asserting the outbound client was called.
tests/cli/commands/infra/test_<agent>_agent.py — verify the CLI app wiring.

14. Verification

uv run pytest tests/agents/<agent>/ tests/cli/commands/infra/test_<agent>_agent.py
make pre-commit

Then run the agent locally against the Docker Compose stack and confirm it performs one full reconcile pass without requeues piling up.

Common mistakes

Subscribing the controller to the wrong store (subscribe to the driving table, not its dependencies).
Missing resync_interval when the target can drift silently from the source.
Hard-coding filters in main.py instead of the agent settings class.
Forgetting manager.add_collector(bundle.build()) — you get no telemetry.
Reading from a store inside the reconciler without handling StoreNotInitialisedError / missing keys; raise RequeueRequest instead.
Registering the service settings class but forgetting the ServiceType enum or SERVICE_TYPE_MAP entry — get_<agent>_settings() will raise at runtime.
Adding certs to docker-compose.app.yml but not mounting the fm_certs volume into the agent container.
Missing scopes in the api.migration client roster (values.yaml) for agents that call the FM API.
Shipping without a memory_consumption_estimates.md.

Good references

src/fleet_manager/agents/topology/ — cleanest reference: compact main.py, a separate planner, and a focused reconciler.
src/fleet_manager/agents/netbox/ — larger two-way sync with filter fan-out (iter_devices_for_filters) for filter-per-cluster setups.
src/fleet_manager/agents/radar_ingestion_service/ — publish-style agent with RequeueRequest for transient NATS errors, plus a representative memory_consumption_estimates.md.
tests/agents/netbox/test_agent.py — BDD test structure.
tests/agents/machinery/test_controller.py — how to wire a real Controller + WorkerPool in a test.

Start Here

Operator Guide

Developer Guide

Relationships

Deployment Guide

Archived

API Reference

Build a New Agent

Naming conventions — pick these first

1. Define the sync scope

2. Know the machinery

3. Create stores and reflectors

4. Implement the reconciler

Error handling — when to requeue

5. Wire the controller

6. Compose the agent main

7. Add agent and service settings

8. Wire the CLI

9. Wire local deployment

a. Generate the mTLS cert

b. Register the FM API client

c. Add the agent service

d. Run it end-to-end

e. Common local-run failures

10. Decide the deployment topology

11. Roll it out through the deploy repos

In `k8s-deploy-fleet-manager`

In `k8s-cluster-fleet-manager`

End-to-end example — `netbox-agent`

12. Produce memory consumption estimates

13. Tests

14. Verification

Common mistakes

Good references

​Naming conventions — pick these first

​1. Define the sync scope

​2. Know the machinery

​3. Create stores and reflectors

​4. Implement the reconciler

​Error handling — when to requeue

​5. Wire the controller

​6. Compose the agent main

​7. Add agent and service settings

​8. Wire the CLI

​9. Wire local deployment

​a. Generate the mTLS cert

​b. Register the FM API client

​c. Add the agent service

​d. Run it end-to-end

​e. Common local-run failures

​10. Decide the deployment topology

​11. Roll it out through the deploy repos

​In k8s-deploy-fleet-manager

​In k8s-cluster-fleet-manager

​End-to-end example — netbox-agent

​12. Produce memory consumption estimates

​13. Tests

​14. Verification

​Common mistakes

​Good references

Naming conventions — pick these first

1. Define the sync scope

2. Know the machinery

3. Create stores and reflectors

4. Implement the reconciler

Error handling — when to requeue

5. Wire the controller

6. Compose the agent main

7. Add agent and service settings

8. Wire the CLI

9. Wire local deployment

a. Generate the mTLS cert

b. Register the FM API client

c. Add the agent service

d. Run it end-to-end

e. Common local-run failures

10. Decide the deployment topology

11. Roll it out through the deploy repos

In `k8s-deploy-fleet-manager`

In `k8s-cluster-fleet-manager`

End-to-end example — `netbox-agent`

12. Produce memory consumption estimates

13. Tests

14. Verification

Common mistakes

Good references