> ## Documentation Index > Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt > Use this file to discover all available pages before exploring further. # Client Authentication > How agents, CLIs, and other non-browser clients authenticate to the Fleet Manager API. Fleet Manager supports **two** OAuth 2.0 mechanisms for non-browser clients, both against the same `/o/token/` endpoint and the same `Application` registry: * **Private key JWT** (RFC 7523) — the default for Fleet Manager's own agents and anything else living inside FM-managed clusters, where `cert-manager` issues the certificates. A client proves its identity by signing a short-lived JWT with a private key whose certificate chain is trusted by the API. * **Client secret** (`client_credentials` + a confidential secret) — for external services that do not sit on FM's PKI. If you are integrating another product's backend with this API, this is the mechanism to ask for; see [External services: client secret](#external-services-client-secret) and do not build JWT assertion signing. The private-key path is not plain mTLS. Transport is regular HTTPS; the "mutual" part lives in the signed JWT assertion. ## End-to-end flow 1. An agent loads its certificate chain (`.crt`) and Ed25519 private key (`.key`) from disk. 2. It builds a client-assertion JWT with: * Standard claims (`iss`, `sub`, `aud`, `iat`, `exp`, `jti`). Both `iss` and `sub` are set to the client's own `client_id`. * An `x5c` header containing the base64-DER chain. * A signature from the leaf key. 3. It `POST`s a form-encoded body to `/o/token/` with four fields: | Field | Value | | - | - | | `grant_type` | `client_credentials` | | `client_id` | the client's own name (matches cert CN) | | `client_assertion_type` | `urn:ietf:params:oauth:client-assertion-type:jwt-bearer` | | `client_assertion` | the signed JWT from step 2 | 4. The API validates the request in order: 1. Look up `Application` by the submitted `client_id`. Not found → 401. 2. Extract the `x5c` chain from the JWT header and verify it terminates at one of the trusted CAs. Fails → 401. The trust file (`API_OAUTH2_PRIVATE_KEY_JWT_CA_PATH`, the `ca.crt` of the `pki-ca` item) is a PEM bundle: it usually holds one root, and holds the old and the new root together during a CA rotation (old root first, so an API replica still running a single-CA build keeps trusting the current leaves) so leaves from both keep working until every client is reissued and the old root is removed. The hub ExternalSecret refreshes hourly, so a new root is trusted up to an hour after the 1Password edit; satellite `pki-ca` items are intermediates minted from the hub root and must be re-minted for the new root before their agents can reissue. The full sequence is in [PKI and Certificate Trust → Rotate the Hub Root CA](../../deployment-guide/pki-and-certificate-trust#rotate-the-hub-root-ca). 3. Verify the leaf certificate's Common Name matches the submitted `client_id`. Mismatch → 401. 4. Verify the JWT signature using the leaf public key. Fails → 401. 5. Verify the JWT's `iss` and `sub` claims both equal the submitted `client_id`. Mismatch → 401. On success, the API issues a bearer token with the scopes on the `Application`. 5. The client uses the bearer token as `Authorization: Bearer ...` and refreshes when it is within \~60 seconds of expiry. ## Who uses which mechanism | Client | Mechanism | | - | - | | Fleet Manager agents (netbox, workflow, radar, ...) | Private key JWT → bearer token | | CI harnesses, and `fleetctl` acting as an agent | Private key JWT → bearer token (same flow) | | `fleetctl api login` (a human at a terminal) | Authorization code + PKCE against a public client → bearer token carrying that user's identity | | External service integrations (other products' backends) | `client_credentials` + client secret → bearer token | | Humans in the browsable API | Django session auth | | Humans via OAuth auth code (delegated access) | Bearer token with a user identity attached | Browser users and automated clients both land on the same `DefaultPermissions` class (see [Scopes and Permissions](./scopes-and-permissions)) — user permissions for sessions, token scopes for machine clients. ## Registering a client An agent does not exist to the API until an `Application` record is created for it. The underlying registration is the `ensureagentclient` Django management command — but it must run **inside the running API process** so it talks to the right Postgres. Running `uv run fleetctl infra api manage ensureagentclient ...` from your host fails: with no `API_DATABASE_URL` configured, Django falls back to a fresh local SQLite file with no migrations applied, and the command errors with `no such table: fleet_manager_auth_application`. Pick the entry point that matches your context: ### Permanent registration (deployed environments + standard local dev) Permanent agent clients are seeded by the hub API's **migration job** in deployed clusters and by `fm-api` startup in local Docker Compose. Both run the same `manage bootstrap` command, which reads a declarative client roster. **Both rosters must stay in sync** — without the helm entry, real clusters fail with 401; without the compose entry, local dev does. Add the agent to both rosters: * `deployment/charts/hub/values.yaml` — `api.migration.clients` for hub agents, or `api.migration.satelliteClients` (a `namePrefix` expanded per `expectedSatellites` entry) for satellite agents. The chart renders these into a hook ConfigMap that the migration job mounts; it runs once on every deploy, and this is how dev/prod clusters get their `Application` rows. * `deployment/docker/bootstrap-clients.yaml` — the local roster, mounted into `fm-api` and applied on each `make docker-up-api`. For local dev, recreating the container (`docker compose up -d --force-recreate fm-api`) picks up your new entry — the roster is mounted from the repo, so no rebuild is needed. For dev/prod, the migration job runs automatically on the next [Argo CD](../../start-here/glossary#argo-cd) sync. See [Build a New Agent](../authoring-guides/build-a-new-agent) for the full onboarding checklist. ### Ad-hoc local registration (experiments, one-off testing) To register a one-off client without editing compose files (e.g. while prototyping a new agent or scope), use the helper Makefile target. It generates the mTLS cert if missing and registers the OAuth client, both inside the `fm-api` container so they share the running database and `fm_certs` volume: ```bash theme={null} make register-agent NAME=my-agent SCOPES="device:read,site:read" ``` Requires `make docker-up-api` to have been run first. The target is idempotent: rerunning with new scopes updates the existing client. ### Browser login clients (`fleetctl api login`) Everything above provisions a machine client. An interactive human login is a different shape and needs `--kind=browser`: a **public** client running authorization-code + PKCE, with no secret to distribute, and a loopback redirect URI for the CLI to catch the callback on. **Local Compose roster only.** In `deployment/docker/bootstrap-clients.yaml`, that is two extra fields: ```yaml theme={null} - name: fleetctl-api kind: browser redirect_uris: - http://127.0.0.1:8765/callback scopes: - "*" ``` Do **not** add a browser entry to `api.migration.clients` in the hub chart. Unlike the Compose roster, that one is not used directly — `migration-clients-configmap.yaml` re-renders each entry and emits only `name` and `scopes`, so `kind` and `redirect_uris` are dropped. The entry then arrives at bootstrap as a service client asking for `*`, which is rejected, and because the migration Job is a `pre-install,pre-upgrade` hook that failure takes down the whole deploy. Deployed hubs register their `fleetctl-api` outside this roster; ask in `#fleet-manager` rather than adding one here. Three things have to line up or the login fails at the authorize step, before any token request: * **`name` must match the client id the CLI sends** — `DEFAULT_LOGIN_CLIENT_ID` in `src/fleet_manager/cli/commands/api/auth.py`, currently `fleetctl-api`. An unknown client id is rejected by `/o/authorize/`, and the CLI then blocks for its full `--timeout-seconds` (default 300) waiting for a callback that never comes. * **`redirect_uris` must contain the CLI's loopback callback** — `http://127.0.0.1:/callback` from the same module. A browser client with no redirect URI is rejected at save time. * **`kind: browser`** — omitting it silently gives you the default service client, so the authorization-code grant is not enabled and the login cannot work. `scopes: ["*"]` is the provisioning sentinel described in [Scopes and Permissions → Attaching scopes to a client](./scopes-and-permissions#attaching-scopes-to-a-client). It is **only** accepted for `kind: browser`; a service client must list its scopes. Ad-hoc registration works the same way, run inside the container so it reaches the right database: ```bash theme={null} docker compose -p fm-$(whoami | tr '.' '-') -f deployment/docker/docker-compose.app.yml \ exec fm-api uv run fleetctl infra api manage ensureagentclient \ fleetctl-api '*' --kind=browser --redirect-uris http://127.0.0.1:8765/callback ``` The `make register-agent` target does not cover this path — it registers service clients only. ### What the registration does `ensureagentclient` creates or updates an `Application`, with the client and grant type determined by `--kind` (`ClientKind` in `src/fleet_manager/api/auth/models.py`): | `--kind` | `client_type` | `grant_type` | Used by | | - | - | - | - | | `service` (default) | `confidential` | `client_credentials` | agents, connectors, external services | | `browser` | `public` | `authorization_code` (+ PKCE) | `fleetctl api login` | For a **service** client, the `` must match all three of: 1. the Common Name on the leaf certificate the agent presents, 2. the `client_id` the agent submits in the token request, 3. the JWT's `iss` and `sub` claims. Any mismatch fails token acquisition with 401. A **browser** client presents no certificate at all — PKCE replaces the client secret — so only the `client_id` and the registered redirect URI have to match. ## External services: client secret If your service runs outside FM-managed clusters and doesn't consume the FM Python SDK, ask for a **secret-based confidential client** instead of implementing the JWT assertion above. The server falls back to standard OAuth2 client authentication whenever no assertion is presented, so no special server-side setup is involved. One external team has already built (and then thrown away) Ed25519 `x5c` signing because this page didn't say so — don't repeat that. The exchange is the textbook one (an external integration runs it against production today): ```bash theme={null} curl -s https://api.fleet.nscale.com/o/token/ \ -d grant_type=client_credentials \ -d client_id= \ -d client_secret=$FLEET_MANAGER_CLIENT_SECRET ``` * Credentials in the form body and HTTP basic auth are both accepted. * Tokens last **10 hours**; standard libraries (`golang.org/x/oauth2/clientcredentials`, `httpx-oauth`, ...) handle caching and refresh. * The token response's `scope` field lists what was actually **granted**, which can be narrower than what you requested — log it once at first mint so a thin registration shows up immediately rather than as a mystery 403 weeks later. What to know about the secret itself: * **Registration never emits a secret.** The roster and `ensureagentclient` create the `Application` row only, and secrets are stored hashed (`hash_client_secret`), so a secret cannot be read back later. It is set once through Django admin by whoever provisions the client and captured at creation — or never. * **Each environment is a separate registry.** A dev `client_id` with a prod `client_secret` fails as `invalid_client`, which reads like a scope or grant-type problem and is not. * On your side, treat it like any other credential: a Kubernetes `Secret` / [1Password](../../start-here/glossary#1password) item, delivered via env var, never a flag or a repo file. Getting one is a two-step ask — no certificate, CA, or cert-manager involvement at any point: 1. **Register the client** with a PR adding your client name and minimum scopes to the `api.migration.clients` roster in `deployment/charts/hub/values.yaml`: ```yaml theme={null} - name: scopes: - "site:read" ``` Pick scopes from `_ALL_SCOPES` in `src/fleet_manager/api/auth/scopes.py` (see [Scopes and Permissions](./scopes-and-permissions)). The roster is the managed, recreatable source of truth: the migration job re-asserts every entry on each deploy, so roster clients survive an environment database rebuild. An ad-hoc row persists until someone removes it or the database is rebuilt — nothing recreates it. 2. **Have the secret set** — ask in `#fleet-manager`. Someone with access sets it once through Django admin and hands it to you at creation; per the custody rules above it cannot be read back afterwards, per environment. ## Where the cert comes from Registration creates the `Application` row. To actually call the API, the client also needs a leaf certificate + private key whose chain terminates at the CA the API trusts. Where that comes from depends on where the client runs. ### In-cluster agents (the standard path) If your client is a Fleet Manager agent (a pod running in the hub or a satellite cluster), you don't handle certs by hand. **`cert-manager` issues a leaf cert for each agent automatically**, signed by the satellite intermediate (or hub root, depending on placement), and writes it into a Kubernetes secret. Your agent's `Deployment` mounts that secret and points its `*_FM_API_CERT_PATH` / `*_FM_API_KEY_PATH` env vars at the mounted files. You enable this by adding a `Certificate` resource to your agent's chart subfolder under `deployment/charts/hub/templates//` (or `deployment/charts/satellite/...`). Mirror an existing agent's chart folder — `netbox-agent` is the canonical reference. ### Local Docker Compose `fm-generate-certs` runs once at `make docker-up-api`, creates a CA into the `fm_certs` Docker volume, then issues a leaf cert per agent listed in its `generate_cert ` block. Every container that needs an agent cert mounts the same `fm_certs` volume. To add a permanent local agent, add a `generate_cert ` line. To register one ad-hoc, use `make register-agent NAME=... SCOPES=...` — it issues the cert and registers the client in one step. ### External clients (dashboards, CI, services outside the FM cluster) The standard pattern — used today by the [burn-in](../../start-here/glossary#burn-in)-testing dashboard against `https://api.fleet.dev.nscale.com` — is **register once via PR, mint cert via cert-manager, store cert + key in 1Password, fetch with `op read` from any consumer**. Five concrete steps: #### 1. Register your client (PR into `fleet-manager`) Open a PR against the `fleet-manager` repo adding your client to the `api.migration.clients` roster in `deployment/charts/hub/values.yaml`: ```yaml theme={null} - name: scopes: - "device:read" - "site:read" ``` This is the **managed and recreatable way to register a client in dev or prod**. The migration job re-runs on every deploy and re-asserts every roster entry, so roster clients survive an environment database rebuild. Anything you registered ad-hoc (e.g. via `kubectl exec` directly into `fm-api`) is never deleted by the migration job — it persists until explicitly removed or the environment database is rebuilt — but nothing recreates it either, so treat ad-hoc rows as unmanaged. Argo CD picks the roster change up on the next sync. Real worked example — the `burn-in-testing` entry in that roster: ```yaml theme={null} - name: burn-in-testing scopes: - "organisation:read" - "site:read" - "location:read" - "rack:read" - "device:read" - "device-type:read" - "device-role:read" - "workflow:read" - "job:read" - "job:create" - "job:cancel" - "job-device-status:read" - "job-device-status:update" - "cohort:read" - "cohort:create" - "test-execution:read" - "test-execution:create" - "test-execution:update" - "test-execution:mark_stale" - "test-execution-stale-hold:read" - "test-execution-stale-hold:undo" ``` Pick scopes from `_ALL_SCOPES` in `src/fleet_manager/api/auth/scopes.py` (see [Scopes and Permissions](./scopes-and-permissions) for how each maps to endpoints) — request the minimum your client actually needs. Adding scopes later is another small PR; over-granting now is awkward to walk back. #### 2. Get a cert minted (one-time, requires cluster access) Ask in `#fleet-manager` — someone with cluster access will mint a leaf cert with `CN=` signed by the cluster's CA. The recommended path is a `cert-manager` `Certificate` resource declared in the chart; `cert-manager` issues a secret which you can read back with `kubectl get secret -tls -o yaml` and decode. The leaf cert's CN must equal the `client_id` you registered in step 1. External clients that copy PEMs into 1Password must not blindly inherit the 7-day lifetime used by in-cluster agents. `cert-manager` renewal updates the Kubernetes Secret only; it does not update the copied 1Password item. Either automate syncing the renewed Secret back to 1Password, or use a longer external client lifetime with an explicit operator rotation process. The `radar-service` credential uses a 365-day lifetime with a 30-day renewal window for this reason. #### 3. Stash cert + key in 1Password Convention: vault `Fleet Manager hub dev` (and `Fleet Manager hub prod` for prod), one item per client named `-creds`, with two fields: * `cert` — PEM-encoded certificate * `key` — PEM-encoded private key This initial setup is usually done by the same person who minted the cert in step 2. Treat the 1Password item as its own rotation target: update it from the renewed Kubernetes Secret before the copied certificate expires. #### 4. Fetch credentials with `op read` Any consumer (your laptop, CI, another cluster) pulls them on demand with the 1Password CLI. A `make auth-setup` target along these lines is the canonical shape: ```bash theme={null} op read --out-file ./certs/dev/.crt "op://Fleet Manager hub dev/-creds/cert" op read --out-file ./certs/dev/.key "op://Fleet Manager hub dev/-creds/key" chmod 600 ./certs/dev/.key ``` #### 5. Make calls with the FM Python SDK The SDK's `Auth` class handles JWT signing and bearer-token refresh for you: ```python theme={null} from pathlib import Path from fleet_manager_sdk import Auth, Client client = Client( base_url="https://api.fleet.dev.nscale.com", auth=Auth( base_url="https://api.fleet.dev.nscale.com", client_id="", cert_path=Path("./certs/dev/.crt"), key_path=Path("./certs/dev/.key"), ), timeout=30, ) devices = await client.list_devices() ``` If you can't depend on the FM Python SDK (different language, sandboxed runtime), the [End-to-end flow](#end-to-end-flow) at the top of this page describes the JWT assertion and `/o/token/` exchange in enough detail to implement directly. **Reference implementation**: the burn-in-testing repo (`burn-in-testing/Makefile` `auth-setup` target + `backend/.certs/` layout + `Auth` usage) is the canonical worked example. In every path above (in-cluster, local Docker Compose, external client), the leaf cert's Common Name is the `client_id`, and the same CA bundle (`OAUTH2_PRIVATE_KEY_JWT_CA_PATH`) terminates chain verification on the API side. For the full PKI tree, see [PKI and Certificate Trust](../../deployment-guide/pki-and-certificate-trust). ## Where the pieces live * `src/fleet_manager/core/clients/fleet_manager/auth.py` — client-side token acquisition and caching. * `src/fleet_manager/api/auth/oauth2.py` — server-side validator: chain verification, CN extraction, JWT signature check. * `src/fleet_manager/api/auth/management/commands/ensureagentclient.py` — the registration command. * `src/fleet_manager/api/auth/models.py` — the `Application` model. * `src/fleet_manager/api/auth/permissions.py` — `DefaultPermissions`, where token scopes are enforced per request. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.