> ## Documentation Index > Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt > Use this file to discover all available pages before exploring further. # PKI & Certificate Trust > How Fleet Manager establishes hub-to-satellite trust with root and intermediate CAs. Fleet Manager uses a hub/satellite trust model. Each hub owns a root CA, and each satellite gets its own intermediate CA signed by that hub root. ## Trust Model * The **hub root CA** establishes the root of trust * Each **satellite intermediate CA** is signed by the corresponding hub root * Satellite-local components can then issue and rotate agent certificates without breaking the trust chain back to the hub This model matches the deployment split and keeps each site's trust material isolated. ## Certificate Lifetime The current runbook issues CA certificates for 10 years: * agent certificates can be rotated by `cert-manager` * the CA lifetime is intentionally long until there is a better mechanism for distributing satellite CA certificates ## Generate the Hub Root CA Create the root CA and upload it to the hub vault: ```bash theme={null} uv run fleetctl infra certs root-ca --days 3650 op item create \ --vault "Fleet Manager hub ${ENV}" \ --title pki-ca \ --category 'secure note' \ 'tls\.crt[file]=ca.crt' \ 'tls\.key[file]=ca.key' \ 'ca\.crt[file]=ca.crt' ``` ## Generate a Satellite Intermediate CA Read the hub CA from [1Password](../start-here/glossary#1password), generate the site CA, and upload it to the satellite vault: ```bash theme={null} op read --out-file ca.crt "op://Fleet Manager hub ${ENV}/pki-ca/tls.crt" op read --out-file ca.key "op://Fleet Manager hub ${ENV}/pki-ca/tls.key" uv run fleetctl infra certs satellite-ca ${NAME} --days 3650 op item create \ --vault "Fleet Manager ${NAME} ${ENV}" \ --title pki-ca \ --category 'secure note' \ "tls\.crt[file]=${NAME}.crt" \ "tls\.key[file]=${NAME}.key" \ 'ca\.crt[file]=ca.crt' ``` The satellite chart projects only `tls.crt`/`tls.key` into the cert-manager Issuer Secret; the item's `ca.crt` is kept as a record of the signing chain and is not read by anything on the satellite. ## Rotate the Hub Root CA Use this when the root key may have been exposed or on a planned rotation. The API trusts every root in the `ca.crt` field of the hub `pki-ca` item (a PEM bundle), so the rotation is rolling: trust both roots, move issuance to the new one, reissue every client, then drop the old root. Removing the old root is the only revocation the API offers, so the sequence matters more than the speed. 1. **Gate: bundle-aware API everywhere.** Every API replica in the environment must run a build that loads the trust file as a bundle. An older build loads only the first certificate, so the bundle order below is not optional. 2. **Mint the new root offline.** `fleetctl infra certs root-ca --cn "Fleet Manager Root CA" --days 3650 --output-dir ./new-root` produces `ca.crt`/`ca.key`. The root key is used only by the `sign` step below and then lives in 1Password; it never lands on an Issuer. 3. **Trust both, hub first.** `fleetctl deployment-standup onepassword pki-ca-root trust --new-root ./new-root/ca.crt` appends the new root to the hub `pki-ca` field `ca.crt`, **existing roots first**, and skips anything already present. Pass `--also-trust ` for a partner root that must be trusted alongside (for example UNI's own client-certificate root). `tls.crt`/`tls.key` are untouched. Force-sync the hub `pki-ca-trust` ExternalSecret (the trust bundle syncs into its own Secret; the Issuer's `pki-ca` Secret carries only the signing pair) and confirm on every API replica that `/etc/fleet-manager/certs/ca.crt` holds every root, that the `fleet_manager_api_trusted_ca_roots` gauge matches, and that the API log shows `trusted CA bundle loaded: N root(s)` with no bundle warnings. The API re-reads the file on change, no restart. 4. **Switch issuance.** `pki-ca-root sign --new-root ./new-root/ca.crt --new-root-key ./new-root/ca.key` mints a hub intermediate signed by the new root and replaces hub `tls.crt`/`tls.key`; it refuses until the new root is in `ca.crt`. Then `pki-ca --rotate` once per satellite re-mints that satellite's intermediate from the hub item and replaces its `pki-ca` fields in place. External issuers that hold a copy of the CA (for example UNI's `reservations/fm-client-ca`) receive their own CA material at this point. 5. **Reissue every client.** cert-manager does not reissue a leaf when its Issuer's Secret changes. Force renewal of every Certificate in `deployment/charts/hub/templates/*/certificate.yaml`, `deployment/charts/satellite/templates/*/certificate.yaml` and any out-of-cluster client, then prove each new leaf cryptographically against the new root, not by issuer name (a re-minted CA can keep the same subject): `openssl verify -CAfile new-root.pem leaf.pem` for hub leaves, `openssl verify -CAfile new-root.pem -untrusted intermediate.pem leaf.pem` for satellite leaves, and the same command with `old-root.pem` must fail. Then authenticate once against the API with each client class (a hub agent, a satellite agent, CHAD, an out-of-cluster client) and confirm a token is issued. Re-copy hand-distributed leaves into 1Password and reload their consumers. 6. **Prove, then revoke.** Only when every leaf on the inventory verifies against the new root and fails against the old one, and every client class has authenticated with its new chain, run `pki-ca-root revoke --keep-fingerprint ` once per root that must remain (the `trust` step logs every fingerprint; a prefix works only while it matches exactly one root, and the roots share a subject so fingerprints are the only handle). The command checks the signature, not the subject, and refuses to drop the root that signed the current hub `tls.crt`. Wait one access-token lifetime before declaring the rotation complete: a client still holding a token issued under the old root fails only when it next authenticates. **Rollback.** Before step 6 nothing has been distrusted: rolling the API back to a single-CA build is safe only while the old root is still first in `ca.crt`, and a new-root leaf issued in step 5 will fail against that build, so roll back leaves before rolling back the API. After step 6, re-adding the old root re-trusts a compromised key; the fix for a missed client is to reissue it, not to restore the root. ## Operational Notes * `ENV` should match the environment name used for the target hub and satellite vaults * `NAME` should match the satellite deployment identifier, such as `glo1` or `sta1` * PKI setup is a prerequisite for bringing a new satellite online in the published deployment flow For the broader site rollout sequence, continue with [Deploy Fleet Manager at a New Site](./new-site-deployment/deploy-fleet-manager-at-a-new-site). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.