> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Site Validation

> Validation checks for a new Fleet Manager deployment before and after rollout.

Site validation should happen twice:

* before cluster rollout, using the shared development environment
* after deployment to the target Kubernetes environment

## Local or Pre-Cluster Validation

Before shipping a site configuration to Kubernetes, verify that the development environment can:

* ingest the target site's [devices](../../start-here/glossary#node) from [NetBox](../../start-here/glossary#netbox)
* start the expected Fleet Manager services
* trigger an enrolment workflow
* show workflow progress in [Temporal](../../start-here/glossary#temporal) and logs in Grafana

The current internal process uses the shared `glo1 dev` environment for this stage.

## Kubernetes Health Checks

After deployment, start with the basic cluster checks:

```bash theme={null}
kubectl get pods -n fleet-manager
kubectl get services -n fleet-manager
kubectl get pods -n temporal
```

These commands confirm that the Fleet Manager and Temporal components are present and healthy.

## Device and Workflow Validation

Validate the core Fleet Manager behaviour end to end:

* imported devices appear in Fleet Manager after NetBox sync
* enrolment can be triggered from the API or UI
* the device state moves as expected in NetBox
* workflow history is visible in Temporal

If devices do not appear after sync, check the configured NetBox device filter and inspect the
NetBox agent logs.

## Grafana and Loki Checks

The deployment runbook specifically calls out checking logs for:

* Temporal server activity
* Temporal worker activity
* workflow-agent activity

Useful LogQL starting points:

```logql theme={null}
{namespace="temporal", app="temporal"} |= ``
```

```logql theme={null}
{namespace="fleet-manager", component="temporal-worker"} |= ``
```

```logql theme={null}
{namespace="fleet-manager", component="workflow-agent"} |= ``
```

## SLURM Validation

If the site supports hardware-test workflows, also validate the [SLURM](../../start-here/glossary#slurm) integration:

* the Kubernetes cluster can reach the SLURM REST API
* the Temporal worker can authenticate with the stored JWT token
* the hardware-test CLI works in the expected environment
* at least one smoke test can be launched and observed

The deployment process also expects validation of:

* one single-node test
* one pairwise test
* one collective test

## Monitoring and Dashboards

Check the expected dashboards in Grafana, especially:

* Fleet Manager dashboards
* NGINX Ingress Controller dashboards
* Temporal Server Metrics

For a healthy rollout, you should be able to see activity from recent or running workflows for
the new site.

Check that the new satellite probes the hub: in Grafana Explore on `metrics-all`,
`probe_success{cluster="fleet-mgr-<region>",job="fleet-manager-satellite-hub-probe"}` returns `1`. The
alerts are evaluated by the satellite's own Prometheus, so check there too, through the
`fleet-mgr-<region>-prometheus` Teleport app: the same series, and the
`fleet-manager-satellite-hub-probe` group under Alerts. That is the only check for a site that does not
ship metrics to central Mimir. No series means the site has no blackbox exporter, so a hub outage would
go unreported from it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.