Skip to main content
Fleet Manager supports two OAuth 2.0 mechanisms for non-browser clients, both against the same /o/token/ endpoint and the same Application registry:
  • Private key JWT (RFC 7523) — the default for Fleet Manager’s own agents and anything else living inside FM-managed clusters, where cert-manager issues the certificates. A client proves its identity by signing a short-lived JWT with a private key whose certificate chain is trusted by the API.
  • Client secret (client_credentials + a confidential secret) — for external services that do not sit on FM’s PKI. If you are integrating another product’s backend with this API, this is the mechanism to ask for; see External services: client secret and do not build JWT assertion signing.
The private-key path is not plain mTLS. Transport is regular HTTPS; the “mutual” part lives in the signed JWT assertion.

​
End-to-end flow

  1. An agent loads its certificate chain (.crt) and Ed25519 private key (.key) from disk.
  2. It builds a client-assertion JWT with:
    • Standard claims (iss, sub, aud, iat, exp, jti). Both iss and sub are set to the client’s own client_id.
    • An x5c header containing the base64-DER chain.
    • A signature from the leaf key.
  3. It POSTs a form-encoded body to /o/token/ with four fields:
  4. The API validates the request in order:
    1. Look up Application by the submitted client_id. Not found → 401.
    2. Extract the x5c chain from the JWT header and verify it terminates at one of the trusted CAs. Fails → 401. The trust file (API_OAUTH2_PRIVATE_KEY_JWT_CA_PATH, the ca.crt of the pki-ca item) is a PEM bundle: it usually holds one root, and holds the old and the new root together during a CA rotation (old root first, so an API replica still running a single-CA build keeps trusting the current leaves) so leaves from both keep working until every client is reissued and the old root is removed. The hub ExternalSecret refreshes hourly, so a new root is trusted up to an hour after the 1Password edit; satellite pki-ca items are intermediates minted from the hub root and must be re-minted for the new root before their agents can reissue. The full sequence is in PKI and Certificate Trust → Rotate the Hub Root CA.
    3. Verify the leaf certificate’s Common Name matches the submitted client_id. Mismatch → 401.
    4. Verify the JWT signature using the leaf public key. Fails → 401.
    5. Verify the JWT’s iss and sub claims both equal the submitted client_id. Mismatch → 401.
    On success, the API issues a bearer token with the scopes on the Application.
  5. The client uses the bearer token as Authorization: Bearer ... and refreshes when it is within ~60 seconds of expiry.

​
Who uses which mechanism

Browser users and automated clients both land on the same DefaultPermissions class (see Scopes and Permissions) — user permissions for sessions, token scopes for machine clients.

​
Registering a client

An agent does not exist to the API until an Application record is created for it. The underlying registration is the ensureagentclient Django management command — but it must run inside the running API process so it talks to the right Postgres. Running uv run fleetctl infra api manage ensureagentclient ... from your host fails: with no API_DATABASE_URL configured, Django falls back to a fresh local SQLite file with no migrations applied, and the command errors with no such table: fleet_manager_auth_application. Pick the entry point that matches your context:

​
Permanent registration (deployed environments + standard local dev)

Permanent agent clients are seeded by the hub API’s migration job in deployed clusters and by fm-api startup in local Docker Compose. Both run the same manage bootstrap command, which reads a declarative client roster. Both rosters must stay in sync — without the helm entry, real clusters fail with 401; without the compose entry, local dev does. Add the agent to both rosters:
  • deployment/charts/hub/values.yaml — api.migration.clients for hub agents, or api.migration.satelliteClients (a namePrefix expanded per expectedSatellites entry) for satellite agents. The chart renders these into a hook ConfigMap that the migration job mounts; it runs once on every deploy, and this is how dev/prod clusters get their Application rows.
  • deployment/docker/bootstrap-clients.yaml — the local roster, mounted into fm-api and applied on each make docker-up-api.
For local dev, recreating the container (docker compose up -d --force-recreate fm-api) picks up your new entry — the roster is mounted from the repo, so no rebuild is needed. For dev/prod, the migration job runs automatically on the next Argo CD sync. See Build a New Agent for the full onboarding checklist.

​
Ad-hoc local registration (experiments, one-off testing)

To register a one-off client without editing compose files (e.g. while prototyping a new agent or scope), use the helper Makefile target. It generates the mTLS cert if missing and registers the OAuth client, both inside the fm-api container so they share the running database and fm_certs volume:
Requires make docker-up-api to have been run first. The target is idempotent: rerunning with new scopes updates the existing client.

​
Browser login clients (fleetctl api login)

Everything above provisions a machine client. An interactive human login is a different shape and needs --kind=browser: a public client running authorization-code + PKCE, with no secret to distribute, and a loopback redirect URI for the CLI to catch the callback on. Local Compose roster only. In deployment/docker/bootstrap-clients.yaml, that is two extra fields:
Do not add a browser entry to api.migration.clients in the hub chart. Unlike the Compose roster, that one is not used directly — migration-clients-configmap.yaml re-renders each entry and emits only name and scopes, so kind and redirect_uris are dropped. The entry then arrives at bootstrap as a service client asking for *, which is rejected, and because the migration Job is a pre-install,pre-upgrade hook that failure takes down the whole deploy. Deployed hubs register their fleetctl-api outside this roster; ask in #fleet-manager rather than adding one here. Three things have to line up or the login fails at the authorize step, before any token request:
  • name must match the client id the CLI sends — DEFAULT_LOGIN_CLIENT_ID in src/fleet_manager/cli/commands/api/auth.py, currently fleetctl-api. An unknown client id is rejected by /o/authorize/, and the CLI then blocks for its full --timeout-seconds (default 300) waiting for a callback that never comes.
  • redirect_uris must contain the CLI’s loopback callback — http://127.0.0.1:<DEFAULT_CALLBACK_PORT>/callback from the same module. A browser client with no redirect URI is rejected at save time.
  • kind: browser — omitting it silently gives you the default service client, so the authorization-code grant is not enabled and the login cannot work.
scopes: ["*"] is the provisioning sentinel described in Scopes and Permissions → Attaching scopes to a client. It is only accepted for kind: browser; a service client must list its scopes. Ad-hoc registration works the same way, run inside the container so it reaches the right database:
The make register-agent target does not cover this path — it registers service clients only.

​
What the registration does

ensureagentclient creates or updates an Application, with the client and grant type determined by --kind (ClientKind in src/fleet_manager/api/auth/models.py): For a service client, the <client-name> must match all three of:
  1. the Common Name on the leaf certificate the agent presents,
  2. the client_id the agent submits in the token request,
  3. the JWT’s iss and sub claims.
Any mismatch fails token acquisition with 401. A browser client presents no certificate at all — PKCE replaces the client secret — so only the client_id and the registered redirect URI have to match.

​
External services: client secret

If your service runs outside FM-managed clusters and doesn’t consume the FM Python SDK, ask for a secret-based confidential client instead of implementing the JWT assertion above. The server falls back to standard OAuth2 client authentication whenever no assertion is presented, so no special server-side setup is involved. One external team has already built (and then thrown away) Ed25519 x5c signing because this page didn’t say so — don’t repeat that. The exchange is the textbook one (an external integration runs it against production today):
  • Credentials in the form body and HTTP basic auth are both accepted.
  • Tokens last 10 hours; standard libraries (golang.org/x/oauth2/clientcredentials, httpx-oauth, …) handle caching and refresh.
  • The token response’s scope field lists what was actually granted, which can be narrower than what you requested — log it once at first mint so a thin registration shows up immediately rather than as a mystery 403 weeks later.
What to know about the secret itself:
  • Registration never emits a secret. The roster and ensureagentclient create the Application row only, and secrets are stored hashed (hash_client_secret), so a secret cannot be read back later. It is set once through Django admin by whoever provisions the client and captured at creation — or never.
  • Each environment is a separate registry. A dev client_id with a prod client_secret fails as invalid_client, which reads like a scope or grant-type problem and is not.
  • On your side, treat it like any other credential: a Kubernetes Secret / 1Password item, delivered via env var, never a flag or a repo file.
Getting one is a two-step ask — no certificate, CA, or cert-manager involvement at any point:
  1. Register the client with a PR adding your client name and minimum scopes to the api.migration.clients roster in deployment/charts/hub/values.yaml:
    Pick scopes from _ALL_SCOPES in src/fleet_manager/api/auth/scopes.py (see Scopes and Permissions). The roster is the managed, recreatable source of truth: the migration job re-asserts every entry on each deploy, so roster clients survive an environment database rebuild. An ad-hoc row persists until someone removes it or the database is rebuilt — nothing recreates it.
  2. Have the secret set — ask in #fleet-manager. Someone with access sets it once through Django admin and hands it to you at creation; per the custody rules above it cannot be read back afterwards, per environment.

​
Where the cert comes from

Registration creates the Application row. To actually call the API, the client also needs a leaf certificate + private key whose chain terminates at the CA the API trusts. Where that comes from depends on where the client runs.

​
In-cluster agents (the standard path)

If your client is a Fleet Manager agent (a pod running in the hub or a satellite cluster), you don’t handle certs by hand. cert-manager issues a leaf cert for each agent automatically, signed by the satellite intermediate (or hub root, depending on placement), and writes it into a Kubernetes secret. Your agent’s Deployment mounts that secret and points its *_FM_API_CERT_PATH / *_FM_API_KEY_PATH env vars at the mounted files. You enable this by adding a Certificate resource to your agent’s chart subfolder under deployment/charts/hub/templates/<agent>/ (or deployment/charts/satellite/...). Mirror an existing agent’s chart folder — netbox-agent is the canonical reference.

​
Local Docker Compose

fm-generate-certs runs once at make docker-up-api, creates a CA into the fm_certs Docker volume, then issues a leaf cert per agent listed in its generate_cert <name> block. Every container that needs an agent cert mounts the same fm_certs volume. To add a permanent local agent, add a generate_cert <name> line. To register one ad-hoc, use make register-agent NAME=... SCOPES=... — it issues the cert and registers the client in one step.

​
External clients (dashboards, CI, services outside the FM cluster)

The standard pattern — used today by the burn-in-testing dashboard against https://api.fleet.dev.nscale.com — is register once via PR, mint cert via cert-manager, store cert + key in 1Password, fetch with op read from any consumer. Five concrete steps:

​
1. Register your client (PR into fleet-manager)

Open a PR against the fleet-manager repo adding your client to the api.migration.clients roster in deployment/charts/hub/values.yaml:
This is the managed and recreatable way to register a client in dev or prod. The migration job re-runs on every deploy and re-asserts every roster entry, so roster clients survive an environment database rebuild. Anything you registered ad-hoc (e.g. via kubectl exec directly into fm-api) is never deleted by the migration job — it persists until explicitly removed or the environment database is rebuilt — but nothing recreates it either, so treat ad-hoc rows as unmanaged. Argo CD picks the roster change up on the next sync. Real worked example — the burn-in-testing entry in that roster:
Pick scopes from _ALL_SCOPES in src/fleet_manager/api/auth/scopes.py (see Scopes and Permissions for how each maps to endpoints) — request the minimum your client actually needs. Adding scopes later is another small PR; over-granting now is awkward to walk back.

​
2. Get a cert minted (one-time, requires cluster access)

Ask in #fleet-manager — someone with cluster access will mint a leaf cert with CN=<your-client-name> signed by the cluster’s CA. The recommended path is a cert-manager Certificate resource declared in the chart; cert-manager issues a secret which you can read back with kubectl get secret <name>-tls -o yaml and decode. The leaf cert’s CN must equal the client_id you registered in step 1. External clients that copy PEMs into 1Password must not blindly inherit the 7-day lifetime used by in-cluster agents. cert-manager renewal updates the Kubernetes Secret only; it does not update the copied 1Password item. Either automate syncing the renewed Secret back to 1Password, or use a longer external client lifetime with an explicit operator rotation process. The radar-service credential uses a 365-day lifetime with a 30-day renewal window for this reason.

​
3. Stash cert + key in 1Password

Convention: vault Fleet Manager hub dev (and Fleet Manager hub prod for prod), one item per client named <your-client-name>-creds, with two fields:
  • cert — PEM-encoded certificate
  • key — PEM-encoded private key
This initial setup is usually done by the same person who minted the cert in step 2. Treat the 1Password item as its own rotation target: update it from the renewed Kubernetes Secret before the copied certificate expires.

​
4. Fetch credentials with op read

Any consumer (your laptop, CI, another cluster) pulls them on demand with the 1Password CLI. A make auth-setup target along these lines is the canonical shape:

​
5. Make calls with the FM Python SDK

The SDK’s Auth class handles JWT signing and bearer-token refresh for you:
If you can’t depend on the FM Python SDK (different language, sandboxed runtime), the End-to-end flow at the top of this page describes the JWT assertion and /o/token/ exchange in enough detail to implement directly. Reference implementation: the burn-in-testing repo (burn-in-testing/Makefile auth-setup target + backend/.certs/ layout + Auth usage) is the canonical worked example. In every path above (in-cluster, local Docker Compose, external client), the leaf cert’s Common Name is the client_id, and the same CA bundle (OAUTH2_PRIVATE_KEY_JWT_CA_PATH) terminates chain verification on the API side. For the full PKI tree, see PKI and Certificate Trust.

​
Where the pieces live

  • src/fleet_manager/core/clients/fleet_manager/auth.py — client-side token acquisition and caching.
  • src/fleet_manager/api/auth/oauth2.py — server-side validator: chain verification, CN extraction, JWT signature check.
  • src/fleet_manager/api/auth/management/commands/ensureagentclient.py — the registration command.
  • src/fleet_manager/api/auth/models.py — the Application model.
  • src/fleet_manager/api/auth/permissions.py — DefaultPermissions, where token scopes are enforced per request.