> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a New Agent

> Short guide for adding a new data-sync agent in Fleet Manager.

Agents in Fleet Manager pull scoped data from a source system, cache it in-process, and reconcile it into a target system. They are built on a small set of primitives in `src/fleet_manager/agents/machinery` and follow the shape of `src/fleet_manager/agents/topology/` — the cleanest reference implementation.

```
source client list() -> ListReflector -> Store -> Controller -> Reconciler -> target client upsert()
```

Use one `Store` + `ListReflector` per source table, one `Controller` per reconciler, a shared `WorkerPool`, and a single `Manager` that runs the whole thing.

<Tip>
  **Use the `/agent-creator` skill.** When building a new agent, invoke it — it knows this architecture, the file layout, and the canonical wiring, and will scaffold the agent end-to-end. This page is the reference behind that skill; read it if you need to deviate from the default shape or understand what the skill is producing.
</Tip>

## Naming conventions — pick these first

Agent names drift across six places. Pick all of them deliberately before writing code; silent mismatches between the `ServiceType` value, cert CN, and `ensureagentclient` name will look like auth bugs at runtime.

The cert CN ≡ `ensureagentclient` name ≡ `FM_CLIENT_ID` link comes from Fleet Manager's OAuth flow — the client submits `FM_CLIENT_ID` as `client_id`, the API loads the `Application` by that value, then verifies the leaf cert CN matches it. See [Client Authentication](../integrate-with-the-api/client-authentication) for the mechanism and [Scopes and Permissions](../integrate-with-the-api/scopes-and-permissions) for how the scopes you pass to `ensureagentclient` are enforced.

The pattern for most hub agents is:

| Piece | Convention | Example (`netbox`) |
| - | - | - |
| Settings module | `src/fleet_manager/settings/agents/<stem>.py` | `netbox.py` |
| Service module | `src/fleet_manager/settings/services/<stem>_agent.py` | `netbox_agent.py` |
| `ServiceType` enum value | kebab-case, ends in `-agent` | `netbox-agent` |
| Chart folder | `deployment/charts/hub/templates/<name>-agent/` | `netbox-agent` |
| `ensureagentclient` name (= cert CN) | same as chart folder name | `netbox-agent` |
| Env var prefix (Pydantic settings) | uppercase of service module stem | `NETBOX_AGENT_` |

Satellite agents (anything deployed into a satellite cluster) get one client per satellite — the name is suffixed with the satellite ID (`workflow-agent-glo`, `workflow-agent-sta`), declared once as a `namePrefix` under `api.migration.satelliteClients` and expanded per `expectedSatellites` entry. Hub agents get a single unqualified client.

## 1. Define the sync scope

Decide these before writing code:

* Source tables and the target entities they map to.
* A stable key per table (usually the source ID; fall back to composite keys only if necessary).
* Filters per table (API filters, org scope, active-only). Filters belong on the agent settings, not hard-coded in `main.py`.
* The target client (Fleet Manager, [NetBox](../../start-here/glossary#netbox), [Kusto](../../start-here/glossary#kusto), [NATS](../../start-here/glossary#nats), etc.).

If the agent is read-mostly and has no target writes (e.g. a ingestion-style agent), it still uses the same machinery — the reconciler just publishes instead of calling an upsert.

## 2. Know the machinery

The machinery lives in `src/fleet_manager/agents/machinery/`. You do not subclass it; you compose it.

**How it runs:** a reflector polls the source and updates a store, which emits events. The controller translates each event into a set of reconcile keys and pushes them onto a dedup queue — the key stays in place if a duplicate is pushed, and the same key never dequeues twice concurrently. The queue has two lanes: changes go in the change lane, and re-checks of objects that have not changed (the periodic resync, a full refresh's `REFRESHED` events, the store's first listing after a restart) go in the resync lane. The change lane goes first, apart from every tenth turn, which goes to the resync lane so a busy change lane can't starve it. A change moves a key already waiting in the resync lane. The controller hands the worker pool one key per worker, so the backlog stays in the queue where a change can still jump it. Workers pull keys, run the reconciler (which reads the latest state from the store), and ack when done. Failures requeue with clamped exponential backoff, in the lane they came from.

This gives you a few useful properties for free:

* Reconciles only run for objects that actually changed (with optional periodic resync).
* A change never waits behind a resync sweep, however long the sweep is.
* The reconciler always reads the freshest state from the store — queued keys can't go stale.
* A fast-changing object is throttled (one reconcile in flight per key) without starving other keys.
* Total concurrency is bounded by the worker pool.

The primitives:

* `Store[K, V]` (`store.py`) — typed in-memory cache. Emits `CREATED` / `MODIFIED` / `DELETED` events. Raises `StoreNotInitialisedError` if read before the initial load.
* `ListReflector[K, V]` (`reflector.py`) — periodically calls an async list function and replaces the store contents. Handles transient failures with exponential backoff. Dies after `max_retries + 1` consecutive failures, which kills the agent — this is intentional.
* `Controller` (`controller.py`) — subscribes to stores and schedules reconciles through the worker pool. Handles requeue and backoff.
* `WorkerPool` (`worker_pool.py`) — bounded concurrency for reconcile tasks. One pool per agent is usually enough.
* `Manager` (`manager.py`) — owns reflectors, controllers, the worker pool, and telemetry. Call `await manager.run()` to start the agent.
* `AgentCollectorBundleBuilder` (`telemetry.py`) — builds per-component telemetry collectors with a single agent name.
* `Result` / `RequeueRequest` — how reconcilers signal success, requeue, or delay.

## 3. Create stores and reflectors

One `Store` + `ListReflector` per source table. If the agent has several tables that take the same setup, wrap them in a small helper that takes the list function and interval and injects the telemetry collectors — see `create_nb_store` in `src/fleet_manager/agents/netbox/main.py`. For agents with only two or three stores, constructing them inline (as `src/fleet_manager/agents/topology/main.py` does) is fine.

Filters belong on the agent settings class, not hard-coded in `main.py`. For single-filter resources, pass the client list method straight to the reflector. For filter-per-cluster setups that need to fan out across many filter combinations, use the `iter_devices_for_filters` pattern in `src/fleet_manager/agents/netbox/main.py`.

## 4. Implement the reconciler

A reconciler is any `Callable[[K], Awaitable[Result | None]]` — a class with `__call__`, or a bare async function. It takes a key, reads whatever it needs from the stores, resolves dependencies (e.g. site → location → rack) and performs exactly one target write path.

### Error handling — when to requeue

The controller distinguishes three outcomes. Pick the right one and the queue handles the rest; pick the wrong one and you either hot-loop or drop work silently.

| Outcome | Signal | When to use |
| - | - | - |
| Success | return `None` or `Result(requeue=False)` | the reconcile finished (including intentional no-ops). |
| Known transient miss | raise `RequeueRequest(requeue_after=...)` | a dependency isn't in its store yet (parent not reflected, downstream eventually-consistent read). Uses *your* delay verbatim — skip the controller's backoff. |
| Unknown transient failure | let the exception bubble | timeouts, 5xx from the target, connection resets. The controller catches, logs, and requeues with clamped exponential backoff. |
| Permanent failure | raise `FatalError` | the input will never be valid (bad schema, contradictory state). The key is dropped, not requeued. |

Two mistakes to avoid:

* **Swallowing transient errors.** If you `try/except Exception: return None` around an upsert, the controller thinks the reconcile succeeded and the object never gets retried. Let transient errors bubble.
* **Raising `RequeueRequest` with a 0-second delay in a tight loop.** If the dependency is chronically missing, the queue hot-loops. Pass a sensible `requeue_after` (the canonical agents use 1–5s) and — where possible — subscribe the controller to the dependency's store so the reconcile is triggered again when the dependency arrives, not by polling.

See `src/fleet_manager/agents/topology/reconciler.py` for the canonical shape, and `src/fleet_manager/agents/radar_ingestion_service/` for `RequeueRequest` on transient NATS errors.

## 5. Wire the controller

One `Controller` per reconciler. Each controller subscribes to the store(s) whose events should drive that reconciler, mapping each event to a list of reconcile keys (a single source event can fan out to multiple target keys). Set `resync_interval > 0` on a subscription when the target can drift silently from the source (e.g. external edits); leave it `0` when store events alone are authoritative.

## 6. Compose the agent main

`main.py` is the wiring layer: instantiate clients, build the `AgentCollectorBundleBuilder`, create stores/reflectors/controllers/worker pool, hand everything to the `Manager`, call `await manager.run()`.

Note that the service settings class nests the agent-specific config under a field named after the agent (e.g. `settings.netbox_agent.*`) — dereference through that field, not the top-level settings object.

Don't forget `manager.add_collector(bundle.build())`, or you get no telemetry.

## 7. Add agent and service settings

Create `src/fleet_manager/settings/agents/<agent>.py` with a Pydantic model for the agent's own config (filters, intervals, worker count, cert paths, telemetry name, any NATS subject names).

Create `src/fleet_manager/settings/services/<agent>_agent.py` that inherits `BaseFleetManagerSettings` and composes the agent config with its client settings. Mirror `NetBoxAgentSettings` in `src/fleet_manager/settings/services/netbox_agent.py`.

If the agent talks to a new external system, add its client settings under `src/fleet_manager/settings/clients/`.

Register the service in `src/fleet_manager/settings/loaders.py`:

* Add a `ServiceType` enum entry.
* Add the `SERVICE_TYPE_MAP` row.
* Add a `get_<agent>_settings()` accessor that delegates to `_get_cached_settings(...)`.

## 8. Wire the CLI

Add `src/fleet_manager/cli/commands/infra/<agent>_agent.py` with a Typer app that exposes a single `run` command invoking `asyncio.run(main())`. Register it from `src/fleet_manager/cli/commands/infra/main.py` with `app.add_typer(...)`. Copy the shape from `src/fleet_manager/cli/commands/infra/netbox_agent.py`. Add a CLI test under `tests/cli/commands/infra/` that patches `asyncio.run` and the agent's `main`.

## 9. Wire local deployment

Local deploy = Docker Compose. Edit `deployment/docker/docker-compose.app.yml`.

Two up-front decisions change what you need:

* **Does the agent call the Fleet Manager API?** If yes (most agents), you need steps a, b, and the FM API auth vars in c. If the agent only writes to an external system (e.g. `netbox-matrix-agent` → Kusto), skip a and b and omit those vars.
* **Is the agent satellite-scoped?** In local compose, only add per-satellite certs / clients / services when the stack already simulates multiple sites for that agent. Today `workflow-agent` and `diagnostics-agent` use suffixed local entries (`-glo`, `-sta`). Follow the existing pattern in compose rather than assuming every satellite agent has suffixed local services.

### a. Generate the mTLS cert

Under `fm-generate-certs`, add one line per client the agent will use. The name becomes the cert filename and the leaf cert's Common Name — which must match the `ensureagentclient` name in the next step and the `FM_CLIENT_ID` the agent submits in the token request.

```yaml theme={null}
generate_cert <client-name>          # hub agent, e.g. generate_cert netbox-agent
generate_cert <client-name>-glo      # satellite agent: one per simulated satellite
generate_cert <client-name>-sta
```

### b. Register the FM API client

`fm-api` starts by running `manage bootstrap`, which registers every client listed in `deployment/docker/bootstrap-clients.yaml`. Add one entry per client with the exact scopes your agent needs (see [Scopes and Permissions](../integrate-with-the-api/scopes-and-permissions) for how to pick them):

```yaml theme={null}
clients:
  # ... existing entries
  - name: <client-name>
    scopes:
      - "organisation:read"
      - "site:read"
      - "device:read"
```

This is **separate** from the helm roster in Section 10 — both must list the client. Without the compose manifest entry, local token acquisition at `/o/token/` fails with 401 (no `Application` for the submitted `client_id`); without the helm entry, the same 401 happens in real clusters.

### c. Add the agent service

Add a service stanza modeled on `fm-netbox-agent`:

```yaml theme={null}
fm-<agent>-agent:
  image: ${FLEET_MANAGER_IMAGE}
  depends_on:
    fm-generate-certs: { condition: service_completed_successfully }
    fm-api: { condition: service_healthy }
  command: ["uv", "run", "fleetctl", "infra", "<agent>-agent", "run"]
  environment:
    SERVICE_TYPE: <agent>-agent                     # must match ServiceType enum value
    ENVIRONMENT: "${ENVIRONMENT:?ENVIRONMENT must be set}"
    # Only if the agent calls the Fleet Manager API:
    FLEET_MANAGER_CLIENT_BASE_URL: http://fm-api:8000/
    <PREFIX>_FM_CLIENT_ID: <agent>-name              # must match ensureagentclient name
    <PREFIX>_FM_API_CERT_PATH: /etc/fleet-manager/certs/<agent>-name.crt
    <PREFIX>_FM_API_KEY_PATH: /etc/fleet-manager/certs/<agent>-name.key
  volumes:
    - fm_certs:/etc/fleet-manager/certs
  networks:
    - default
```

`<PREFIX>` is the Pydantic env prefix from your agent settings class (`NETBOX_AGENT_`, `RADAR_INGESTION_`, `WORKFLOW_AGENT_`, …). Get this wrong and the agent starts but can't find its cert.

The four `FLEET_MANAGER_CLIENT_BASE_URL` / `*_FM_CLIENT_ID` / `*_FM_API_CERT_PATH` / `*_FM_API_KEY_PATH` variables are only needed if the agent calls the FM API. Agents that write only to external systems (e.g. `netbox-matrix-agent` → Kusto) omit all four and the `fm_certs` volume mount.

### d. Run it end-to-end

```bash theme={null}
make docker-up
docker compose -f deployment/docker/docker-compose.app.yml logs -f fm-<agent>-agent
```

First-pass log signals to look for:

* "Starting Fleet Manager `<Agent>` agent" — the CLI entry ran.
* Reflector startup messages per store ("[devices](../../start-here/glossary#node)", "sites", …).
* No `StoreNotInitialisedError` after the first poll interval.
* Target writes / publishes showing up on the other side (Fleet Manager API, NATS, Kusto, ...).

### e. Common local-run failures

* `invalid SERVICE_TYPE` at startup — the value in the compose stanza isn't in the `ServiceType` enum. Add it to `src/fleet_manager/settings/loaders.py` first.
* `FileNotFoundError` on the cert path — the `generate_cert` line is missing or the filename doesn't match `<PREFIX>_FM_API_CERT_PATH`.
* **HTTP 401 at `/o/token/`** (token never issued) — one of: no entry for the submitted `client_id` in `deployment/docker/bootstrap-clients.yaml` (so no `Application` record exists), the cert CN doesn't match the `FM_CLIENT_ID`, or the JWT's `iss`/`sub` don't match. Check `fm-api` logs — `authenticate_client` logs the specific reason.
* **HTTP 403 at an API endpoint** — the token was issued but its scopes don't cover the call. Add the missing scope to the agent's `scopes` list in that manifest.

## 10. Decide the deployment topology

* Confirm whether the agent runs in the hub cluster or a satellite cluster. Hub agents (`netbox-agent`, `radar-agent`, `topology-agent`) have a single client. Satellite agents (`workflow-agent`, `openstack-agent`, `slurm-agent`, `diagnostics-agent`, ...) have one client per satellite. A process name and its client id can differ — the OpenStack agent authenticates as `ironic-agent-{satellite}` (ADR-0036).
* If the agent calls the Fleet Manager API, add it to the client roster in `deployment/charts/hub/values.yaml`: hub agents go under `api.migration.clients` (`name` + `scopes`), satellite agents under `api.migration.satelliteClients` as a `namePrefix` — the chart expands that once per `expectedSatellites` entry into `<namePrefix>-<satellite>`. See [Scopes and Permissions](../integrate-with-the-api/scopes-and-permissions) for picking the right scopes — missing scopes cause silent 403s.
* Add a [Helm](../../start-here/glossary#helm) template for the agent deployment under `deployment/charts/hub/templates/<agent>/` (or `deployment/charts/satellite/...`). Mirror an existing agent (e.g. `netbox-agent`) for the Deployment, Service, and any ConfigMap/Secret references.

## 11. Roll it out through the deploy repos

Real-cluster deploys go through two more repositories. Both are [Argo CD](../../start-here/glossary#argo-cd) driven.

* `k8s-deploy-fleet-manager` — wrapper Helm chart that feeds values into the upstream `deployment/charts/hub` / `satellite` charts. Also owns shared `ExternalSecret`s.
* `k8s-cluster-fleet-manager` — per-cluster GitOps repo. One folder per cluster/environment (e.g. `glo1/production/`, `sys2-sta1/production/`) with an Argo `Application` that points at `k8s-deploy-fleet-manager` and supplies cluster-specific values.

### In `k8s-deploy-fleet-manager`

1. Add defaults in `charts/fleet-manager/values.yaml` under `fleetManager.hub.<yourAgent>` (or `fleetManager.satellite.<yourAgent>`): `workerCount`, any agent-specific config fields, and `resources`.
2. Pass them through in `charts/fleet-manager/templates/fleet-manager/application-hub.yaml` (or `application-satellite.yaml`) so the upstream chart receives them.
3. If the agent needs [Vault](../../start-here/glossary#vault)-backed credentials, add an `ExternalSecret` under `charts/fleet-manager/templates/fleet-manager/` following `external-secret-hub-op-token.yaml`.
4. Add a representative values sample to `ci/test-hub.yaml` or `ci/test-satellite.yaml` so chart tests cover the new agent.
5. Bump `fleetManager.chartVersion` when cutting a new upstream chart release that includes the agent.

### In `k8s-cluster-fleet-manager`

1. Open the target cluster/environment file, e.g. `glo1/production/fleet-manager.yaml`.
2. Enable the agent and set its real values under `fleetManager.hub.<yourAgent>` (or `fleetManager.satellite.<yourAgent>`).
3. If the agent is deployed as a standalone Argo `Application` (i.e. not part of the main fleet-manager Helm chart, like `uni-filestorage-agent-*.yaml` under `sys2-sta1/production/`), add a new manifest file and register it in `kustomization.yaml`.

### End-to-end example — `netbox-agent`

This is the canonical path. Follow the same chain for a new agent.

1. **`fleet-manager`** — code and helm template:
   * `src/fleet_manager/agents/netbox/` (agent code, reconciler, `main.py`).
   * `src/fleet_manager/settings/services/netbox_agent.py` and the `ServiceType.NETBOX_AGENT` entry in `loaders.py`.
   * `src/fleet_manager/cli/commands/infra/netbox_agent.py` (`fleetctl infra netbox-agent run`).
   * `deployment/charts/hub/templates/netbox-agent/` (Deployment + Service + ConfigMap).
   * `deployment/charts/hub/values.yaml` adds a `netbox-agent` entry under `api.migration.clients` for FM API scopes.

2. **`k8s-deploy-fleet-manager`** — chart defaults:
   * `charts/fleet-manager/values.yaml` (`fleetManager.hub.netboxAgent`) — default `workerCount`, `deviceFilters: []`, `deviceRoleClassMapping: {}`, `resources`.
   * `charts/fleet-manager/templates/fleet-manager/application-hub.yaml` passes those values into the upstream Helm chart.
   * `charts/fleet-manager/templates/fleet-manager/external-secret-ghcr.yaml` provides image-pull creds; the [1Password](../../start-here/glossary#1password) token (`external-secret-hub-op-token.yaml`) is what the agent reads credentials from.

3. **`k8s-cluster-fleet-manager`** — real cluster values:
   * `glo1/production/fleet-manager.yaml` sets `fleetManager.hub.netboxAgent.workerCount`, the production `deviceFilters` (`cluster_id` + `role__in` per cluster), and the `deviceRoleClassMapping` (`gpu-node: gpu_compute`, etc.).
   * Argo CD picks up the change and syncs to the `glo1` hub cluster.

Fleet Manager releases move configured agent image pins based on labels on the `fleet-manager` Release Please PR. The final tags are visible in `deployment/charts/hub/release/agent-versions.yaml` before merge and are bundled with the released hub chart; for example, the current policy maps `netbox-agent` to `netboxAgent.image.tag` and `netbox-matrix-agent` to `netboxMatrixAgent.image.tag`. Those labels are configured in `.github/agent-release-policy.json`, so another independently pinned agent should add its own policy entry instead of hard-coding workflow logic. The deploy wrapper only promotes the chart version in `charts/fleet-manager/values.yaml` (`fleetManager.chartVersion`).

## 12. Produce memory consumption estimates

Every new agent gets a `memory_consumption_estimates.md` next to its code. Follow the structure of `src/fleet_manager/agents/radar_ingestion_service/memory_consumption_estimates.md`:

* State topology assumptions (number of sites, racks, devices, types, etc.).
* Measure retained size of representative objects with `sys.getsizeof` + `gc.get_referents`.
* Give formulas for total memory as a function of the assumptions.
* Include a scaling table and a mitigation section (e.g. "partition by site using filters").

This is how we decide replica counts and whether an agent needs sharding before it ships.

## 13. Tests

Follow the Unit Tests guidance in the repository root `AGENTS.md`.

Minimum agent-specific coverage:

* `tests/agents/<agent>/test_reconciler.py` — one test per reconcile branch (create, update, reuse, skip, requeue).
* Fixtures for `mock_src_client`, `mock_target_client`, and `Store` instances seeded via a small protocol-based seed helper.
* `RequeueRequest` assertions when required dependencies are missing.
* Explicit verification of target client `.create()` / `.update()` calls, including payload fields.
* A smoke test that wires the real `Controller` + `WorkerPool` (or their fakes in `tests/agents/machinery/`) and drives a reconcile via a real store event, asserting the outbound client was called.
* `tests/cli/commands/infra/test_<agent>_agent.py` — verify the CLI app wiring.

## 14. Verification

```bash theme={null}
uv run pytest tests/agents/<agent>/ tests/cli/commands/infra/test_<agent>_agent.py
make pre-commit
```

Then run the agent locally against the Docker Compose stack and confirm it performs one full reconcile pass without requeues piling up.

## Common mistakes

* Subscribing the controller to the wrong store (subscribe to the *driving* table, not its dependencies).
* Missing `resync_interval` when the target can drift silently from the source.
* Hard-coding filters in `main.py` instead of the agent settings class.
* Forgetting `manager.add_collector(bundle.build())` — you get no telemetry.
* Reading from a store inside the reconciler without handling `StoreNotInitialisedError` / missing keys; raise `RequeueRequest` instead.
* Registering the service settings class but forgetting the `ServiceType` enum or `SERVICE_TYPE_MAP` entry — `get_<agent>_settings()` will raise at runtime.
* Adding certs to `docker-compose.app.yml` but not mounting the `fm_certs` volume into the agent container.
* Missing scopes in the `api.migration` client roster (`values.yaml`) for agents that call the FM API.
* Shipping without a `memory_consumption_estimates.md`.

## Good references

* `src/fleet_manager/agents/topology/` — cleanest reference: compact `main.py`, a separate planner, and a focused reconciler.
* `src/fleet_manager/agents/netbox/` — larger two-way sync with filter fan-out (`iter_devices_for_filters`) for filter-per-cluster setups.
* `src/fleet_manager/agents/radar_ingestion_service/` — publish-style agent with `RequeueRequest` for transient NATS errors, plus a representative `memory_consumption_estimates.md`.
* `tests/agents/netbox/test_agent.py` — BDD test structure.
* `tests/agents/machinery/test_controller.py` — how to wire a real `Controller` + `WorkerPool` in a test.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.