Matrix Agent Operations

Archived — the legacy netbox_matrix agent is retired after the prod cutover. Kept for historical reference only. For the live pipeline, see Matrix Connector Operations.

This page is the operational runbook for the Matrix agent. For architecture, schema design, and the sync cadence model, see the Matrix Agent overview.

Health checks

Start here when investigating a sync issue or responding to a customer query about freshness:

Grafana dashboard — filter to the netbox-matrix agent. Shows reflector cycle counts, errors, latency, and Kusto ingestion volume.
Pod logs — kubectl -n fleet-manager logs deploy/netbox-matrix-agent in the fleet-mgr-prod-glo1 cluster. Look for reconciler error logs or NetBox/Kusto client failures.
Live Kusto state — use the freshness query below against the Dev cluster.

Verifying Dev freshness

Run against the Dev cluster to see the newest record per table:

// Against materialized views: SyncTimestamp reflects the latest ingest per key.
// The 6h controller resync re-ingests every record in the store, so on a healthy
// pipeline every table's MaxSyncTimestamp should advance within ~6 hours.
union DataCenters, EthDevices, EthInterfaceLinks, IbInventory, IbSwitchLinks, NVSwitches, Servers
| summarize MaxSyncTimestamp = max(SyncTimestamp), RowCount = count() by TableName = $table
| extend AgeHours = datetime_diff('hour', now(), MaxSyncTimestamp)
| order by TableName asc

AgeHours > 7 (one resync cycle + slack) points at a stalled controller or sink — pair with the dashboard to confirm. The 25 h ceiling mentioned in the overview is the worst case if both the resync and the next full refresh are also delayed; for routine health checks, treat anything older than ~7 h as a signal. To inspect raw staging history (last 7 days) for one record:

EthDevices_staging
| where NetboxDeviceId == <id>
| order by SyncTimestamp desc

Verifying Prod replication

kusto-copy runs continuously (default iterationPeriod 2 min), so Prod parity should hold without operator intervention. To spot-check it, run the same freshness query against the Prod cluster (matrix-portugal-1 / nscale-inventory-prod-db). The MaxSyncTimestamp per table should trail Dev’s by no more than a single iteration plus export/ingest time (a few minutes), and the materialized-view row counts on both clusters should match:

// Run on both clusters and compare.
union DataCenters, EthDevices, EthInterfaceLinks, IbInventory, IbSwitchLinks, NVSwitches, Servers
| summarize RowCount = count(), MaxSyncTimestamp = max(SyncTimestamp) by TableName = $table
| order by TableName asc

For pod-level health, check the kusto-copy Deployment in the fleet-mgr-prod-glo1 cluster:

kubectl -n fleet-manager get deploy fleet-manager-hub-netbox-matrix-agent-kusto-copy
kubectl -n fleet-manager logs deploy/fleet-manager-hub-netbox-matrix-agent-kusto-copy -c kusto-copy --tail=200

A healthy steady-state log plans zero or a small number of new extents per iteration and ends each cycle with a checkpoint commit to the staging-storage path.

Emergency manual promote

kusto-copy is the steady-state replication path, so most cases do not need a manual promote. The legacy matrix-kusto-promote Azure Function and its GitHub Actions workflow remain available via workflow_dispatch only — the twice-daily schedule was removed once kusto-copy was in steady state. Reach for this only as a recovery lever — e.g. a full re-seed of Prod staging, or while kusto-copy itself is unrecoverable. Running it alongside a healthy kusto-copy creates a second writer against the same Prod surface, which can confuse the checkpointed cursor. If you do need to use it:

Pause kusto-copy first: kubectl -n fleet-manager scale deploy fleet-manager-hub-netbox-matrix-agent-kusto-copy --replicas=0.
Run the workflow (or invoke the function directly with the commands below).
When the promote completes, scale kusto-copy back to 1. It resumes from its blob checkpoint; on next iteration it will pick up any new extents created since the checkpoint and converge.

To invoke the function directly rather than via GitHub Actions (requires an Azure admin with access to the Matrix resource group — the GitHub Actions secrets of the same name are write-only, so they are not a usable source):

# URL — derive from the function app's default hostname
az functionapp show \
  --name matrix-kusto-promote \
  --resource-group Matrix \
  --query defaultHostName -o tsv
# -> matrix-kusto-promote.azurewebsites.net
# Full URL: https://<hostname>/api/promote

# Key — the function-level auth secret
az functionapp function keys list \
  --name matrix-kusto-promote \
  --resource-group Matrix \
  --function-name promote

curl -X POST \
  -H "x-functions-key: <KUSTO_PROMOTE_FUNCTION_KEY>" \
  -H "Content-Type: application/json" \
  -d '{}' \
  https://<function-app-name>.azurewebsites.net/api/promote

Do not paste the function key into Slack or a ticket comment.

Common failure modes

Symptom	First thing to check
Customer says “data is missing in Prod”	Compare Dev vs Prod with the parity query. If Dev’s MV has rows Prod’s doesn’t, check whether their backing `*_staging` rows still exist in Dev — rows aged past the 7-day staging retention cannot be replicated by kusto-copy.
`SyncTimestamp` for a table hasn’t moved in >7 h	Controller or Kusto sink is stalled — the 6h resync should have re-ingested everything. Check pod logs and the Grafana dashboard for that reconciler.
Dashboard shows zero reconciler cycles	Agent pod is down or wedged — `kubectl get pods`, then `kubectl describe`.
Kusto ingestion latency spiked	Check Azure Data Explorer ingestion-queue metrics on the Dev cluster. Agent will keep batching; backlog eventually drains.
Schema migration failed during deploy	`backoffLimit: 0` is intentional — see the schema README and Materialized View Key Changes for triage steps.
`kusto-copy` pod in `CrashLoopBackOff` with `Lease on blob already present`	A previous `kc` process exited without releasing its tracking lease. Break the lease on `stmatrixkustocopy/kusto-copy/prod/netbox-matrix/tracking/lock` using the `azure-kusto-admin` SP (the `az` CLI’s user login does not have data-plane rights), then the pod recovers on next restart.
Prod lag > a few minutes	`kubectl logs` the kusto-copy Deployment. Iteration should be planning/exporting/checkpointing within `iterationPeriod`. Common causes: storage-account auth, source cluster throttling, or a long backfill iteration triggered by a big agent full-refresh (lag accumulates during long iterations rather than overlapping runs).

FAQ

These are the recurring customer / support questions — answers are captured here so support can self-serve. How often does data sync from NetBox to Kusto Dev? Roughly every 30 minutes for incremental changes (records modified since the last cycle), with a full re-fetch every 24 hours to catch deletions. See Sync Cadence and Freshness. Is there an SLA on that 30-minute cadence? No. It is a target, not an SLA — occasional delays from upstream latency, ingestion backpressure, or pod restarts are expected. Why is data missing in Prod? In steady state, it shouldn’t be — kusto-copy replicates Dev → Prod continuously. If a record is in Dev’s materialized view but not Prod’s, first check whether its backing *_staging row still exists in Dev. Rows aged past the 7-day staging retention can’t be re-replicated, which is the typical cause once cutover is past. Otherwise check the kusto-copy pod’s health — see Common failure modes. What is the lag from Dev to Prod? Typically a few minutes. Steady-state lag is one kusto-copy iteration (default iterationPeriod: 2 min) plus export/ingest time. A big agent full-refresh produces a larger batch of extents and that iteration can take several minutes; subsequent iterations catch up rather than overlap. Why do different records show different SyncTimestamp values? A record’s SyncTimestamp only advances when something touches it — an upstream modification, a 24h full refresh, or the 6h periodic resync. Records untouched in NetBox for days will carry an older timestamp until the next full refresh. Can the customer trigger Prod promotion themselves? Not needed in steady state — Prod tracks Dev continuously via kusto-copy. The legacy manual promote function is kept only as an internal emergency lever (see Emergency manual promote). Is there real-time sync available? Not today. NetBox → Dev is a 30-minute periodic snapshot pipeline; Dev → Prod via kusto-copy adds another single-digit-minute hop on top of that. An event-driven, near real-time workflow (NetBox change events → ingest) is in the design phase and will eventually shorten the NetBox → Dev hop.

Start Here

Operator Guide

Developer Guide

Relationships

Deployment Guide

Archived

API Reference

Matrix Agent Operations

Health checks

Verifying Dev freshness

Verifying Prod replication

Emergency manual promote

Common failure modes

FAQ

​Health checks

​Verifying Dev freshness

​Verifying Prod replication

​Emergency manual promote

​Common failure modes

​FAQ

Health checks

Verifying Dev freshness

Verifying Prod replication

Emergency manual promote

Common failure modes

FAQ