Skip to main content
Fleet Manager uses a hub/satellite trust model. Each hub owns a root CA, and each satellite gets its own intermediate CA signed by that hub root.

​
Trust Model

  • The hub root CA establishes the root of trust
  • Each satellite intermediate CA is signed by the corresponding hub root
  • Satellite-local components can then issue and rotate agent certificates without breaking the trust chain back to the hub
This model matches the deployment split and keeps each site’s trust material isolated.

​
Certificate Lifetime

The current runbook issues CA certificates for 10 years:
  • agent certificates can be rotated by cert-manager
  • the CA lifetime is intentionally long until there is a better mechanism for distributing satellite CA certificates

​
Generate the Hub Root CA

Create the root CA and upload it to the hub vault:

​
Generate a Satellite Intermediate CA

Read the hub CA from 1Password, generate the site CA, and upload it to the satellite vault:
The satellite chart projects only tls.crt/tls.key into the cert-manager Issuer Secret; the item’s ca.crt is kept as a record of the signing chain and is not read by anything on the satellite.

​
Rotate the Hub Root CA

Use this when the root key may have been exposed or on a planned rotation. The API trusts every root in the ca.crt field of the hub pki-ca item (a PEM bundle), so the rotation is rolling: trust both roots, move issuance to the new one, reissue every client, then drop the old root. Removing the old root is the only revocation the API offers, so the sequence matters more than the speed.
  1. Gate: bundle-aware API everywhere. Every API replica in the environment must run a build that loads the trust file as a bundle. An older build loads only the first certificate, so the bundle order below is not optional.
  2. Mint the new root offline. fleetctl infra certs root-ca --cn "Fleet Manager Root CA" --days 3650 --output-dir ./new-root produces ca.crt/ca.key. The root key is used only by the sign step below and then lives in 1Password; it never lands on an Issuer.
  3. Trust both, hub first. fleetctl deployment-standup onepassword pki-ca-root trust <env> --new-root ./new-root/ca.crt appends the new root to the hub pki-ca field ca.crt, existing roots first, and skips anything already present. Pass --also-trust <pem> for a partner root that must be trusted alongside (for example UNI’s own client-certificate root). tls.crt/tls.key are untouched. Force-sync the hub pki-ca-trust ExternalSecret (the trust bundle syncs into its own Secret; the Issuer’s pki-ca Secret carries only the signing pair) and confirm on every API replica that /etc/fleet-manager/certs/ca.crt holds every root, that the fleet_manager_api_trusted_ca_roots gauge matches, and that the API log shows trusted CA bundle loaded: N root(s) with no bundle warnings. The API re-reads the file on change, no restart.
  4. Switch issuance. pki-ca-root sign <env> --new-root ./new-root/ca.crt --new-root-key ./new-root/ca.key mints a hub intermediate signed by the new root and replaces hub tls.crt/tls.key; it refuses until the new root is in ca.crt. Then pki-ca <satellite> <env> --rotate once per satellite re-mints that satellite’s intermediate from the hub item and replaces its pki-ca fields in place. External issuers that hold a copy of the CA (for example UNI’s reservations/fm-client-ca) receive their own CA material at this point.
  5. Reissue every client. cert-manager does not reissue a leaf when its Issuer’s Secret changes. Force renewal of every Certificate in deployment/charts/hub/templates/*/certificate.yaml, deployment/charts/satellite/templates/*/certificate.yaml and any out-of-cluster client, then prove each new leaf cryptographically against the new root, not by issuer name (a re-minted CA can keep the same subject): openssl verify -CAfile new-root.pem leaf.pem for hub leaves, openssl verify -CAfile new-root.pem -untrusted intermediate.pem leaf.pem for satellite leaves, and the same command with old-root.pem must fail. Then authenticate once against the API with each client class (a hub agent, a satellite agent, CHAD, an out-of-cluster client) and confirm a token is issued. Re-copy hand-distributed leaves into 1Password and reload their consumers.
  6. Prove, then revoke. Only when every leaf on the inventory verifies against the new root and fails against the old one, and every client class has authenticated with its new chain, run pki-ca-root revoke <env> --keep-fingerprint <sha256> once per root that must remain (the trust step logs every fingerprint; a prefix works only while it matches exactly one root, and the roots share a subject so fingerprints are the only handle). The command checks the signature, not the subject, and refuses to drop the root that signed the current hub tls.crt. Wait one access-token lifetime before declaring the rotation complete: a client still holding a token issued under the old root fails only when it next authenticates.
Rollback. Before step 6 nothing has been distrusted: rolling the API back to a single-CA build is safe only while the old root is still first in ca.crt, and a new-root leaf issued in step 5 will fail against that build, so roll back leaves before rolling back the API. After step 6, re-adding the old root re-trusts a compromised key; the fix for a missed client is to reissue it, not to restore the root.

​
Operational Notes

  • ENV should match the environment name used for the target hub and satellite vaults
  • NAME should match the satellite deployment identifier, such as glo1 or sta1
  • PKI setup is a prerequisite for bringing a new satellite online in the published deployment flow
For the broader site rollout sequence, continue with Deploy Fleet Manager at a New Site.