Skip to main content
Archived — the legacy netbox_matrix agent is retired after the prod cutover. Kept for historical reference only. For the live pipeline, see Matrix Connector Operations.
The Matrix agent (netbox_matrix) is a fleet-manager agent that mirrors DCIM data from NetBox into a customer-facing Azure Data Explorer (Kusto) database on a fixed schedule. It follows the fleet-manager agent pattern to implement an Extract-Transform-Load flow:
  • Extract from NetBox via the HTTP API
  • Transform from the NetBox model into the customer’s required Matrix schema
  • Load into Azure Kusto staging tables, from which materialized views expose the current state

​
Requirements

The golden source of truth for the Matrix schema is the customer’s Inventory Data Requirements document on SharePoint. It supersedes the handful of earlier documents shared by various customer engineers and project managers during development. Access is via the customer’s OneDrive/SharePoint tenant.
For the NetBox-side requirements (what must be populated on a device for it to sync), see NetBox Data Requirements — that page stays live, since the same transforms now run in the Matrix connector.

​
Architecture

​
Kusto Schema Shape

The Kusto schema is a set of append-only <Table>_staging tables that are deduplicated by key and latest timestamp into materialized views — for example, the EthDevices materialized view is backed by the EthDevices_staging table.

​
Why append-only plus materialized views?

Aligns with Kusto’s architecture. Kusto is optimised for append-only writes (columnar, compressed). Updates and deletes are expensive and fight the engine’s design. This pattern leverages Kusto’s strengths rather than working against them. Simple and fast. The agent only writes point-in-time state. Kusto handles deduplication at materialization time, so there is no merge logic in the agent. Audit trail plus current state. The staging tables retain 7 days of history for troubleshooting. The views always show the latest state, auto-deduplicated. Performance and cost. Pre-computed deduplication makes queries fast. 7-day retention with hard delete keeps storage bounded. This is cheaper than query-time deduplication. Idiomatic. This is the standard Kusto pattern for periodic snapshots on a 30-minute cadence, and is much simpler than real-time CDC or merge patterns. Trade-off. We pay for 7 days of staging history, which is acceptable under the retention policy.

​
Schema Management

The Kusto schema is managed declaratively. A single source-of-truth file, schema.kql, describes the desired state of every staging table, materialized view, and retention policy. It contains only idempotent commands (.create-merge, .create-or-alter materialized-view, .alter table policy retention, .drop ... ifexists).

​
Apply flow

A Helm pre-install,pre-upgrade Job runs alongside every deploy of the agent and executes:
The runner wraps schema.kql in .execute database script with (ContinueOnErrors=false, ThrowOnErrors=true) <| ... and sends it to the cluster as a single management command (Microsoft’s recommended primitive for batched declarative schema apply against standalone ADX). On success, the runner re-snapshots the live schema with .show database schema as csl script and asserts no drift remains. The Job uses an admin Service Principal stored in 1Password as azure-kusto-admin, distinct from the ingest-only azure-kusto item used by the runtime agent. backoffLimit is 0 — partial application requires human triage rather than silent retry. The Job runs from the pinned netboxMatrixAgent image. Fleet Manager Release Please PRs include the netbox-matrix-agent label by default, which sets deployment/charts/hub/release/agent-versions.yaml to the release version for that configured pin. If a release changes the matrix schema files or the apply-schema execution path configured in .github/agent-release-policy.json, keep the netbox-matrix-agent label on the Release Please PR — but the release workflow only enforces this when skip-release-gates is not on the PR.
Release Please adds skip-release-gates to every release PR by default (via extra-label in release-please-config.json). While the label is on the PR, the matrix-schema gate is silently bypassed. If your release touches any of these paths, remove skip-release-gates from the Release Please PR before merging:
  • src/fleet_manager/agents/netbox_matrix/schema/ (any file)
  • src/fleet_manager/cli/commands/infra/netbox_matrix_agent.py
  • src/fleet_manager/core/clients/azure_kusto/management.py
  • src/fleet_manager/core/clients/azure_kusto/errors.py
Otherwise the release ships with a stale netboxMatrixAgent image, the pre-upgrade Job applies your new schema.kql from old runtime code, and schema.kql and the runtime image diverge. The release workflow cannot catch this for you when the override is active. See Releasing to Production for the rationale.

​
Runtime vs data environments

The migration Jobs run in GLO1 Prod Hub (the agent’s only runtime). By default the dev apply-schema Job targets the customer dev database (matrix-portugal-dev.eastus2/nscale-inventory-dev-db). When netboxMatrixAgent.applyToProd.enabled=true, a sibling Job applies the same schema.kql to the customer prod database. Both jobs use the admin Service Principal stored in 1Password as azure-kusto-admin. Operator-run schema commands can target either database explicitly:

​
Non-idempotent operations (transition scripts)

Some Kusto operations cannot be expressed declaratively because re-running them would fail: .rename table (no ifexists), .alter column ... type, materialized-view group-by-key changes (rejected outright by .create-or-alter), and any data backfill. These live as one-off scripts in schema/transitions/ and are run by an operator out-of-band:
This pattern is modelled after Atlas’s “declarative + pre-migration hooks” approach. Transition files stay in transitions/ after running — git history is the audit log, not a separate archive directory. The pre-commit lint enforces the whitelist on schema.kql and refuses to commit a .create-or-alter materialized-view body change without an accompanying newly-added transition file in the same change. The lint exists because Kusto’s .create-or-alter materialized-view rejects group-by-expression changes outright with .alter materialized-view does not support changing group-by expressions — without a paired transition that does .drop + .create with (backfill=true), the next deploy’s apply-schema fails. (An earlier version of this section claimed .create-or-alter would silently skip backfill on a group-by change. That was wrong; it’s a hard error.) For group-by-key changes specifically, the transition has to run after the column-add deploy populates the new key column, so the recreate-with-backfill operates on populated staging. That requires splitting the migration across two PRs — see Kusto: Materialized View Key Changes for the full pattern. The run-transition CLI extracts OperationIds from each .create async ... and polls .show operations until every operation reaches Completed (or any fails) before printing Transition applied. — so a successful return guarantees the live state matches the post-transition schema, not just that the script was queued. When the prod replica is enabled, run transitions against both Kusto databases explicitly. kusto-copy replicates staging-table inserts; it does not replicate management commands, materialized-view recreates, or deletes. The usual sequence is: deploy schema to both databases, wait for the primary reconcile cycle, run the transition on the primary, wait for kusto-copy to drain, then run the transition on the replica.

​
Authoring rules and runbooks

For authoring details (what is and isn’t safe to put in schema.kql), see the schema README. For operational runbooks that touch the schema, see:

​
Environments

The Matrix agent is only enabled in the GLO1 Prod Hub environment. It is not run in spokes or in dev.

​
Local development

  • Reads from a local NetBox copy (filtered by device in the Docker Compose config)
  • Transforms locally
  • Pushes to the Azure Data Explorer emulator (emulator setup is still TODO)

​
Production — GLO1 Hub

  • Reads from the NetBox production sin1 site
  • Transforms in the agent pod running in the fleet-mgr-prod-glo1 Kubernetes cluster
  • Pushes (batched) to the Kusto “dev” database, which is the customer’s development environment
  • The customer’s “prod” Kusto database is kept in sync continuously by Microsoft’s kusto-copy tool running alongside the agent — see Replicating Kusto Dev to Prod

​
Sync Cadence and Freshness

Three intervals govern how data moves from NetBox to the customer Dev database. They are independent and serve different purposes — operators get asked about these regularly.

​
Implications for SyncTimestamp

SyncTimestamp is set per record at ingest time. A record’s timestamp only advances when one of the following touches it:
  • the record was modified in NetBox during an incremental cycle, or
  • a full refresh ran (every ~24 h), or
  • a periodic resync re-enqueued the store (every ~6 h).
In the worst case, a record that hasn’t been modified in NetBox for several days will carry a SyncTimestamp up to roughly 25 hours old before the next full refresh refreshes it. Different records will show different timestamps — that is expected, not a bug.

​
No SLA

The 30-minute incremental cadence is a target, not an SLA. It can drift longer for various reasons (NetBox response latency, Kusto ingestion queue backpressure, pod restarts). When in doubt, check the agent dashboard.

​
Replicating Kusto Dev to Prod

The agent only writes to the customer Dev database. The customer Prod database is kept in sync by Microsoft’s kusto-copy tool, which runs as a single-replica Deployment (fleet-manager-hub-netbox-matrix-agent-kusto-copy) in the same fleet-mgr-prod-glo1 cluster as the agent. Every iterationPeriod (default 2 min) it polls Dev for new *_staging extents, exports them via an Azure Blob staging path, and ingests them into Prod. The Prod materialized views then refresh themselves from the new staging rows the same way Dev’s do.

​
Operating model

  • Prod is “Dev with a small lag”. Steady-state replication lag is roughly one iterationPeriod plus export/ingest time — typically a few minutes, not hours or days.
  • Forward-only and checkpointed. kusto-copy runs in BackfillAndNew mode against a stable staging-storage path (stmatrixkustocopy/kusto-copy/prod/netbox-matrix). The cursor is persisted in blob storage, so a pod restart resumes from the last checkpoint — downtime becomes lag, not data loss. Don’t change the staging path casually; a fresh path means a fresh stream.
  • Single writer. Only kusto-copy writes Prod staging in steady state. The chart enforces replicas: 1 with strategy: Recreate to avoid two kc processes contending over the same tracking lease.
  • Data only. kusto-copy replicates staging-table inserts. It does not replicate schema DDL, materialized-view recreates, .delete commands, or transitions — those still apply per-cluster (see Schema Management).
  • Manual recovery lever. The legacy matrix-kusto-promote Azure Function and its GitHub Actions workflow remain available via workflow_dispatch for emergency re-seed only. The twice-daily schedule was removed once kusto-copy reached steady state — running both writers would race the checkpointed cursor.
For health checks, replication monitoring, emergency promote, common failure modes, and FAQ, see Matrix Agent Operations.

​
GraphQL PoV — Not Viable Until NetBox Upgrade

We investigated whether moving the NetBox extract from REST to GraphQL would reduce load on NetBox by combining multiple queries into fewer round-trips and composing the data directly into the shape we need.
NetBox’s GraphQL implementation does not support pagination in our version, which is a non-starter — there is far too much data to pull without pagination. Pagination was introduced in NetBox 4.3.0; we are currently on 4.2.9.