> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Matrix Agent Operations

> Operator and support runbook for the netbox_matrix agent — health checks, Dev freshness and Prod replication verification, emergency manual promote, common failure modes, and FAQ.

<Warning>
  **Archived — the legacy `netbox_matrix` agent is retired after the prod cutover.** Kept for
  historical reference only. For the live pipeline, see [Matrix Connector Operations](../../operator-guide/matrix-connector-operations).
</Warning>

This page is the operational runbook for the Matrix agent. For architecture, schema design, and the sync cadence model, see the [Matrix Agent overview](./overview).

## Health checks

Start here when investigating a sync issue or responding to a customer query about freshness:

1. **[Grafana dashboard](https://fleet-mgr-glo1-grafana.nscale.teleport.sh/d/agent-machinery/agent-machinery?orgId=1\&var-agent=netbox-matrix)** — filter to the `netbox-matrix` agent. Shows reflector cycle counts, errors, latency, and Kusto ingestion volume.
2. **Pod logs** — `kubectl -n fleet-manager logs deploy/netbox-matrix-agent` in the `fleet-mgr-prod-glo1` cluster. Look for reconciler error logs or NetBox/Kusto client failures.
3. **Live Kusto state** — use the freshness query below against the Dev cluster.

## Verifying Dev freshness

Run against the [Dev cluster](https://portal.azure.com/?l=en.en-gb#@nscale.no/resource/subscriptions/b3a74c1f-25d4-4455-b34e-37bf7912ea94/resourceGroups/Matrix/providers/Microsoft.Kusto/Clusters/matrix-portugal-dev/Databases/nscale-inventory-dev-db/overview) to see the newest record per table:

```kql theme={null}
// Against materialized views: SyncTimestamp reflects the latest ingest per key.
// The 6h controller resync re-ingests every record in the store, so on a healthy
// pipeline every table's MaxSyncTimestamp should advance within ~6 hours.
union DataCenters, EthDevices, EthInterfaceLinks, IbInventory, IbSwitchLinks, NVSwitches, Servers
| summarize MaxSyncTimestamp = max(SyncTimestamp), RowCount = count() by TableName = $table
| extend AgeHours = datetime_diff('hour', now(), MaxSyncTimestamp)
| order by TableName asc
```

`AgeHours > 7` (one resync cycle + slack) points at a stalled controller or sink — pair with the dashboard to confirm. The 25 h ceiling mentioned in the [overview](./overview#implications-for-synctimestamp) is the worst case if both the resync *and* the next full refresh are also delayed; for routine health checks, treat anything older than \~7 h as a signal.

To inspect raw staging history (last 7 days) for one record:

```kql theme={null}
EthDevices_staging
| where NetboxDeviceId == <id>
| order by SyncTimestamp desc
```

## Verifying Prod replication

`kusto-copy` runs continuously (default `iterationPeriod` 2 min), so Prod parity should hold without operator intervention. To spot-check it, run the same freshness query against the [Prod cluster](https://portal.azure.com/?l=en.en-gb#@nscale.no/resource/subscriptions/b3a74c1f-25d4-4455-b34e-37bf7912ea94/resourceGroups/Matrix/providers/Microsoft.Kusto/Clusters/matrix-portugal-1/Databases/nscale-inventory-prod-db/overview) (`matrix-portugal-1` / `nscale-inventory-prod-db`). The `MaxSyncTimestamp` per table should trail Dev's by no more than a single iteration plus export/ingest time (a few minutes), and the materialized-view row counts on both clusters should match:

```kql theme={null}
// Run on both clusters and compare.
union DataCenters, EthDevices, EthInterfaceLinks, IbInventory, IbSwitchLinks, NVSwitches, Servers
| summarize RowCount = count(), MaxSyncTimestamp = max(SyncTimestamp) by TableName = $table
| order by TableName asc
```

For pod-level health, check the kusto-copy Deployment in the `fleet-mgr-prod-glo1` cluster:

```bash theme={null}
kubectl -n fleet-manager get deploy fleet-manager-hub-netbox-matrix-agent-kusto-copy
kubectl -n fleet-manager logs deploy/fleet-manager-hub-netbox-matrix-agent-kusto-copy -c kusto-copy --tail=200
```

A healthy steady-state log plans zero or a small number of new extents per iteration and ends each cycle with a checkpoint commit to the staging-storage path.

## Emergency manual promote

`kusto-copy` is the steady-state replication path, so most cases do not need a manual promote. The legacy [`matrix-kusto-promote` Azure Function](https://github.com/nscaledev/fleet-manager/tree/main/src/fleet_manager/agents/netbox_matrix/functions/kusto-promote) and its [GitHub Actions workflow](https://github.com/nscaledev/fleet-manager/actions/workflows/matrix-kusto-promote.yml) remain available via `workflow_dispatch` only — the twice-daily schedule was removed once kusto-copy was in steady state.

Reach for this only as a recovery lever — e.g. a full re-seed of Prod staging, or while `kusto-copy` itself is unrecoverable. Running it alongside a healthy kusto-copy creates a second writer against the same Prod surface, which can confuse the checkpointed cursor. If you do need to use it:

1. Pause kusto-copy first: `kubectl -n fleet-manager scale deploy fleet-manager-hub-netbox-matrix-agent-kusto-copy --replicas=0`.
2. Run the workflow (or invoke the function directly with the commands below).
3. When the promote completes, scale kusto-copy back to `1`. It resumes from its blob checkpoint; on next iteration it will pick up any new extents created since the checkpoint and converge.

To invoke the function directly rather than via GitHub Actions (requires an Azure admin with access to the `Matrix` resource group — the GitHub Actions secrets of the same name are write-only, so they are not a usable source):

```bash theme={null}
# URL — derive from the function app's default hostname
az functionapp show \
  --name matrix-kusto-promote \
  --resource-group Matrix \
  --query defaultHostName -o tsv
# -> matrix-kusto-promote.azurewebsites.net
# Full URL: https://<hostname>/api/promote

# Key — the function-level auth secret
az functionapp function keys list \
  --name matrix-kusto-promote \
  --resource-group Matrix \
  --function-name promote
```

```bash theme={null}
curl -X POST \
  -H "x-functions-key: <KUSTO_PROMOTE_FUNCTION_KEY>" \
  -H "Content-Type: application/json" \
  -d '{}' \
  https://<function-app-name>.azurewebsites.net/api/promote
```

Do *not* paste the function key into Slack or a ticket comment.

## Common failure modes

| Symptom | First thing to check |
| - | - |
| Customer says "data is missing in Prod" | Compare Dev vs Prod with the [parity query](#verifying-prod-replication). If Dev's MV has rows Prod's doesn't, check whether their backing `*_staging` rows still exist in Dev — rows aged past the 7-day staging retention cannot be replicated by kusto-copy. |
| `SyncTimestamp` for a table hasn't moved in >7 h | Controller or Kusto sink is stalled — the 6h resync should have re-ingested everything. Check pod logs and the Grafana dashboard for that reconciler. |
| Dashboard shows zero reconciler cycles | Agent pod is down or wedged — `kubectl get pods`, then `kubectl describe`. |
| Kusto ingestion latency spiked | Check Azure Data Explorer ingestion-queue metrics on the Dev cluster. Agent will keep batching; backlog eventually drains. |
| Schema migration failed during deploy | `backoffLimit: 0` is intentional — see the [schema README](https://github.com/nscaledev/fleet-manager/tree/main/src/fleet_manager/agents/netbox_matrix/schema) and [Materialized View Key Changes](./kusto-materialized-view-key-changes) for triage steps. |
| `kusto-copy` pod in `CrashLoopBackOff` with `Lease on blob already present` | A previous `kc` process exited without releasing its tracking lease. Break the lease on `stmatrixkustocopy/kusto-copy/prod/netbox-matrix/tracking/lock` using the `azure-kusto-admin` SP (the `az` CLI's user login does not have data-plane rights), then the pod recovers on next restart. |
| Prod lag > a few minutes | `kubectl logs` the kusto-copy Deployment. Iteration should be planning/exporting/checkpointing within `iterationPeriod`. Common causes: storage-account auth, source cluster throttling, or a long backfill iteration triggered by a big agent full-refresh (lag accumulates during long iterations rather than overlapping runs). |

## FAQ

These are the recurring customer / support questions — answers are captured here so support can self-serve.

**How often does data sync from NetBox to Kusto Dev?**
Roughly every 30 minutes for incremental changes (records modified since the last cycle), with a full re-fetch every 24 hours to catch deletions. See [Sync Cadence and Freshness](./overview#sync-cadence-and-freshness).

**Is there an SLA on that 30-minute cadence?**
No. It is a target, not an SLA — occasional delays from upstream latency, ingestion backpressure, or pod restarts are expected.

**Why is data missing in Prod?**
In steady state, it shouldn't be — `kusto-copy` replicates Dev → Prod continuously. If a record is in Dev's materialized view but not Prod's, first check whether its backing `*_staging` row still exists in Dev. Rows aged past the 7-day staging retention can't be re-replicated, which is the typical cause once cutover is past. Otherwise check the kusto-copy pod's health — see [Common failure modes](#common-failure-modes).

**What is the lag from Dev to Prod?**
Typically a few minutes. Steady-state lag is one `kusto-copy` iteration (default `iterationPeriod: 2 min`) plus export/ingest time. A big agent full-refresh produces a larger batch of extents and that iteration can take several minutes; subsequent iterations catch up rather than overlap.

**Why do different records show different `SyncTimestamp` values?**
A record's `SyncTimestamp` only advances when something touches it — an upstream modification, a 24h full refresh, or the 6h periodic resync. Records untouched in NetBox for days will carry an older timestamp until the next full refresh.

**Can the customer trigger Prod promotion themselves?**
Not needed in steady state — Prod tracks Dev continuously via `kusto-copy`. The legacy manual promote function is kept only as an internal emergency lever (see [Emergency manual promote](#emergency-manual-promote)).

**Is there real-time sync available?**
Not today. NetBox → Dev is a 30-minute periodic snapshot pipeline; Dev → Prod via kusto-copy adds another single-digit-minute hop on top of that. An event-driven, near real-time workflow (NetBox change events → ingest) is in the design phase and will eventually shorten the NetBox → Dev hop.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.