Skip to main content
Site validation should happen twice:
  • before cluster rollout, using the shared development environment
  • after deployment to the target Kubernetes environment

​
Local or Pre-Cluster Validation

Before shipping a site configuration to Kubernetes, verify that the development environment can:
  • ingest the target site’s devices from NetBox
  • start the expected Fleet Manager services
  • trigger an enrolment workflow
  • show workflow progress in Temporal and logs in Grafana
The current internal process uses the shared glo1 dev environment for this stage.

​
Kubernetes Health Checks

After deployment, start with the basic cluster checks:
These commands confirm that the Fleet Manager and Temporal components are present and healthy.

​
Device and Workflow Validation

Validate the core Fleet Manager behaviour end to end:
  • imported devices appear in Fleet Manager after NetBox sync
  • enrolment can be triggered from the API or UI
  • the device state moves as expected in NetBox
  • workflow history is visible in Temporal
If devices do not appear after sync, check the configured NetBox device filter and inspect the NetBox agent logs.

​
Grafana and Loki Checks

The deployment runbook specifically calls out checking logs for:
  • Temporal server activity
  • Temporal worker activity
  • workflow-agent activity
Useful LogQL starting points:

​
SLURM Validation

If the site supports hardware-test workflows, also validate the SLURM integration:
  • the Kubernetes cluster can reach the SLURM REST API
  • the Temporal worker can authenticate with the stored JWT token
  • the hardware-test CLI works in the expected environment
  • at least one smoke test can be launched and observed
The deployment process also expects validation of:
  • one single-node test
  • one pairwise test
  • one collective test

​
Monitoring and Dashboards

Check the expected dashboards in Grafana, especially:
  • Fleet Manager dashboards
  • NGINX Ingress Controller dashboards
  • Temporal Server Metrics
For a healthy rollout, you should be able to see activity from recent or running workflows for the new site. Check that the new satellite probes the hub: in Grafana Explore on metrics-all, probe_success{cluster="fleet-mgr-<region>",job="fleet-manager-satellite-hub-probe"} returns 1. The alerts are evaluated by the satellite’s own Prometheus, so check there too, through the fleet-mgr-<region>-prometheus Teleport app: the same series, and the fleet-manager-satellite-hub-probe group under Alerts. That is the only check for a site that does not ship metrics to central Mimir. No series means the site has no blackbox exporter, so a hub outage would go unreported from it.