> ## Documentation Index > Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt > Use this file to discover all available pages before exploring further. # Matrix Agent Overview > What the netbox_matrix agent does, how it syncs data from NetBox to Azure Kusto, and where it is deployed. **Archived — the legacy `netbox_matrix` agent is retired after the prod cutover.** Kept for historical reference only. For the live pipeline, see [Matrix Connector Operations](../../operator-guide/matrix-connector-operations). The Matrix agent (`netbox_matrix`) is a fleet-manager agent that mirrors DCIM data from NetBox into a customer-facing Azure Data Explorer (Kusto) database on a fixed schedule. It follows the fleet-manager agent pattern to implement an **Extract-Transform-Load** flow: * **Extract** from NetBox via the HTTP API * **Transform** from the NetBox model into the customer's required Matrix schema * **Load** into Azure Kusto staging tables, from which materialized views expose the current state ## Quick Links * [netbox-matrix agent source](https://github.com/nscaledev/fleet-manager/tree/main/src/fleet_manager/agents/netbox_matrix) * [Schema](https://github.com/nscaledev/fleet-manager/tree/main/src/fleet_manager/agents/netbox_matrix/schema) — declarative `schema.kql` applied automatically on deploy * [Fleet Manager Grafana dashboard](https://fleet-mgr-glo1-grafana.nscale.teleport.sh/d/agent-machinery/agent-machinery?orgId=1\&var-agent=netbox-matrix) — filter to the `netbox-matrix` agent * [GPUCap Vendor Inventory Linear project](https://linear.app/nscale-workspace/project/matrix-gpucap-vendor-inventory-8b47507a48b6/issues?layout=list\&ordering=priority\&grouping=workflowState\&subGrouping=none\&showCompletedIssues=all\&showSubIssues=true\&showTriageIssues=true) ## Requirements The golden source of truth for the Matrix schema is the customer's [Inventory Data Requirements document](https://microsoft.sharepoint.com/:w:/r/teams/nscale/Shared%20Documents/Inventory/Inventory%20Data%20Requirements%20-%20Compute,%20IB,%20ethernet%20-%20Copy.docx?d=w540725639a504e27a25e885de5bddee4\&csf=1\&web=1\&e=HOBFP0) on SharePoint. It supersedes the handful of earlier documents shared by various customer engineers and project managers during development. Access is via the customer's OneDrive/SharePoint tenant. For the NetBox-side requirements (what must be populated on a device for it to sync), see [NetBox Data Requirements](../../operator-guide/netbox-data-requirements) — that page stays live, since the same transforms now run in the Matrix connector. ## Architecture ```mermaid theme={null} --- title: NetBox Matrix Agent - Production Architecture --- graph TB subgraph "Management Environment (Kubernetes)" subgraph "NetBox Matrix Agent Pod" AGENT[Agent Process
fleetctl infra netbox-matrix-agent run] subgraph "Core Components" CLIENT[NetBox Client
HTTP API Consumer] TRANSFORM[Transform Layer
NetBox → Kusto Format] RECONCILER[Reconciler
Change Detection & Sync] KUSTO_CLIENT[Azure Kusto Client
Service Principal Auth] end CONFIG[Configuration
ConfigMap + Secrets] end NB[NetBox Service
DCIM System
Sites, Devices, Racks] end subgraph "External Services" OP[OnePassword
Credentials Vault
Service Account Token] end subgraph "Azure Cloud" AAD[Azure AD
Service Principal Auth] subgraph "Azure Kusto Cluster" STAGING[Staging Tables
e.g. DataCenters_staging] VIEWS[Materialized Views
e.g. DataCenters] RETENTION[Retention Policy
7 days on staging table] end end subgraph "Deployment" HELM[Helm Chart
netbox-matrix-agent] K8S[Kubernetes API] end HELM -->|deploys| K8S K8S -->|creates| AGENT K8S -->|mounts| CONFIG CONFIG -->|env vars| AGENT OP -->|credentials via API| AGENT AGENT -->|orchestrates| RECONCILER RECONCILER -->|fetches via| CLIENT CLIENT -->|HTTP GET /api/dcim/*| NB RECONCILER -->|transforms via| TRANSFORM RECONCILER -->|ingests via| KUSTO_CLIENT KUSTO_CLIENT -->|authenticates| AAD KUSTO_CLIENT -->|ingests| STAGING STAGING -->|latest by sync timestamp| VIEWS RETENTION -.->|enforces 7-day TTL| STAGING classDef agent fill:#e1f5ff,stroke:#0066cc,stroke-width:2px classDef azure fill:#fff4e1,stroke:#ff9900,stroke-width:2px classDef data fill:#e8f5e9,stroke:#4caf50,stroke-width:2px classDef infra fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px class AGENT,CLIENT,TRANSFORM,RECONCILER,KUSTO_CLIENT agent class AAD,STAGING,VIEWS,RETENTION azure class NB,OP data class HELM,K8S,CONFIG infra ``` ```mermaid theme={null} --- title: NetBox Matrix Agent - Production Sync Sequence --- sequenceDiagram autonumber participant K8s as Kubernetes participant Agent as NetBox Matrix Agent Pod participant Config as Configuration participant OP as OnePassword API participant NBClient as NetBox Client participant NB as NetBox API participant Transform as Transform Layer participant Reconciler as Reconciler participant Kusto as Azure Kusto Client participant AAD as Azure AD participant ADX as Azure Kusto K8s->>Agent: Start pod (Helm deployment) Agent->>Config: Load env vars from ConfigMap/Secrets Note over Config: ENVIRONMENT, OP_VAULT_SELECTOR,
NETBOX_URL, NETBOX_TOKEN Config->>OP: Fetch Azure Kusto SP credentials OP-->>Config: Return client_id/client_secret/tenant_id Config-->>Agent: Configuration ready Agent->>Kusto: Initialize Kusto client Kusto->>AAD: Authenticate with Service Principal AAD-->>Kusto: Return access token (JWT) Agent->>NBClient: Initialize NetBox client NBClient->>NB: Test connection NB-->>NBClient: 200 OK loop Every sync interval (configurable, default 1800s / 30 min) Agent->>Reconciler: Start reconciliation cycle alt Incremental cycle (every poll_interval, default 30 min) Reconciler->>NBClient: Fetch sites changed since last cursor NBClient->>NB: GET /api/dcim/sites/?last_updated__gte= NB-->>NBClient: Sites changed since last cursor else Full refresh (every full_refresh_interval, default 24 h) Reconciler->>NBClient: Fetch all sites NBClient->>NB: GET /api/dcim/sites/?limit=1000 NB-->>NBClient: Full site list (catches deletions) end NBClient-->>Reconciler: List[NetBoxSite] Reconciler->>Transform: Transform to DataCenters format Note over Transform: Extract DC Region Code (site slug)
Package full site data as Vendor Data JSON
Add SyncTimestamp Transform-->>Reconciler: List[DataCenterRecord] Reconciler->>Kusto: Ingest to DataCenters_staging Kusto->>ADX: Queued ingestion
DC Region Code, Vendor Data, SyncTimestamp ADX-->>Kusto: Ack Note over ADX: DataCenters materialized view
shows latest by SyncTimestamp
Deduplicates to show current state only Reconciler-->>Agent: Reconciliation cycle complete Note over Agent: Sleep until next interval end Note over Agent,ADX: Metrics: /metrics endpoint (Prometheus format)
Logs: JSON structured logs to stdout
Errors: Automatic retry on next cycle Note over ADX: Staging table: 7-day retention (auto-cleanup)
Materialized view: Permanent (current state only) ``` ## Kusto Schema Shape The Kusto schema is a set of append-only `_staging` tables that are deduplicated by key and latest timestamp into materialized views — for example, the `EthDevices` materialized view is backed by the `EthDevices_staging` table. ### Why append-only plus materialized views? **Aligns with Kusto's architecture.** Kusto is optimised for append-only writes (columnar, compressed). Updates and deletes are expensive and fight the engine's design. This pattern leverages Kusto's strengths rather than working against them. **Simple and fast.** The agent only writes point-in-time state. Kusto handles deduplication at materialization time, so there is no merge logic in the agent. **Audit trail plus current state.** The staging tables retain 7 days of history for troubleshooting. The views always show the latest state, auto-deduplicated. **Performance and cost.** Pre-computed deduplication makes queries fast. 7-day retention with hard delete keeps storage bounded. This is cheaper than query-time deduplication. **Idiomatic.** This is the standard Kusto pattern for periodic snapshots on a 30-minute cadence, and is much simpler than real-time CDC or merge patterns. **Trade-off.** We pay for 7 days of staging history, which is acceptable under the retention policy. ## Schema Management The Kusto schema is managed declaratively. A single source-of-truth file, [`schema.kql`](https://github.com/nscaledev/fleet-manager/tree/main/src/fleet_manager/agents/netbox_matrix/schema), describes the desired state of every staging table, materialized view, and retention policy. It contains only idempotent commands (`.create-merge`, `.create-or-alter materialized-view`, `.alter table policy retention`, `.drop ... ifexists`). ### Apply flow A Helm `pre-install,pre-upgrade` Job runs alongside every deploy of the agent and executes: ```bash theme={null} fleetctl infra netbox-matrix-agent apply-schema ``` The runner wraps `schema.kql` in `.execute database script with (ContinueOnErrors=false, ThrowOnErrors=true) <| ...` and sends it to the cluster as a single management command (Microsoft's recommended primitive for batched declarative schema apply against standalone ADX). On success, the runner re-snapshots the live schema with `.show database schema as csl script` and asserts no drift remains. The Job uses an admin Service Principal stored in 1Password as `azure-kusto-admin`, distinct from the ingest-only `azure-kusto` item used by the runtime agent. `backoffLimit` is `0` — partial application requires human triage rather than silent retry. The Job runs from the pinned `netboxMatrixAgent` image. Fleet Manager Release Please PRs include the `netbox-matrix-agent` label by default, which sets `deployment/charts/hub/release/agent-versions.yaml` to the release version for that configured pin. If a release changes the matrix schema files or the `apply-schema` execution path configured in `.github/agent-release-policy.json`, keep the `netbox-matrix-agent` label on the Release Please PR — but the release workflow only enforces this when `skip-release-gates` is **not** on the PR. Release Please adds `skip-release-gates` to every release PR by default (via `extra-label` in `release-please-config.json`). While the label is on the PR, the matrix-schema gate is silently bypassed. **If your release touches any of these paths, remove `skip-release-gates` from the Release Please PR before merging:** * `src/fleet_manager/agents/netbox_matrix/schema/` (any file) * `src/fleet_manager/cli/commands/infra/netbox_matrix_agent.py` * `src/fleet_manager/core/clients/azure_kusto/management.py` * `src/fleet_manager/core/clients/azure_kusto/errors.py` Otherwise the release ships with a stale `netboxMatrixAgent` image, the pre-upgrade Job applies your new `schema.kql` from old runtime code, and `schema.kql` and the runtime image diverge. The release workflow cannot catch this for you when the override is active. See [Releasing to Production](../../developer-guide/releasing-to-production#release-gates-are-bypassed-by-default--remove-skip-release-gates-when-your-release-needs-them) for the rationale. ### Runtime vs data environments The migration Jobs run in **GLO1 Prod Hub** (the agent's only runtime). By default the dev apply-schema Job targets the **customer dev** database (`matrix-portugal-dev.eastus2/nscale-inventory-dev-db`). When `netboxMatrixAgent.applyToProd.enabled=true`, a sibling Job applies the same `schema.kql` to the customer prod database. Both jobs use the admin Service Principal stored in 1Password as `azure-kusto-admin`. Operator-run schema commands can target either database explicitly: ```bash theme={null} fleetctl infra netbox-matrix-agent plan \ --cluster-url https://.kusto.windows.net \ --database \ --op-item-name azure-kusto-admin ``` ### Non-idempotent operations (transition scripts) Some Kusto operations cannot be expressed declaratively because re-running them would fail: `.rename table` (no `ifexists`), `.alter column ... type`, materialized-view group-by-key changes (rejected outright by `.create-or-alter`), and any data backfill. These live as one-off scripts in [`schema/transitions/`](https://github.com/nscaledev/fleet-manager/tree/main/src/fleet_manager/agents/netbox_matrix/schema/transitions) and are run by an operator out-of-band: ```bash theme={null} fleetctl infra netbox-matrix-agent run-transition YYYYMMDD_short_description.kql \ --cluster-url https://.kusto.windows.net \ --database \ --op-item-name azure-kusto-admin ``` This pattern is modelled after [Atlas](https://atlasgo.io/)'s "declarative + pre-migration hooks" approach. Transition files stay in `transitions/` after running — git history is the audit log, not a separate archive directory. The pre-commit lint enforces the whitelist on `schema.kql` and refuses to commit a `.create-or-alter materialized-view` body change without an accompanying *newly-added* transition file in the same change. The lint exists because Kusto's `.create-or-alter materialized-view` **rejects group-by-expression changes outright** with `.alter materialized-view does not support changing group-by expressions` — without a paired transition that does `.drop` + `.create with (backfill=true)`, the next deploy's apply-schema fails. (An earlier version of this section claimed `.create-or-alter` would silently skip backfill on a group-by change. That was wrong; it's a hard error.) For group-by-key changes specifically, the transition has to run *after* the column-add deploy populates the new key column, so the recreate-with-backfill operates on populated staging. That requires splitting the migration across two PRs — see [Kusto: Materialized View Key Changes](./kusto-materialized-view-key-changes) for the full pattern. The `run-transition` CLI extracts `OperationId`s from each `.create async ...` and polls `.show operations` until every operation reaches `Completed` (or any fails) before printing `Transition applied.` — so a successful return guarantees the live state matches the post-transition schema, not just that the script was queued. When the prod replica is enabled, run transitions against both Kusto databases explicitly. kusto-copy replicates staging-table inserts; it does not replicate management commands, materialized-view recreates, or deletes. The usual sequence is: deploy schema to both databases, wait for the primary reconcile cycle, run the transition on the primary, wait for kusto-copy to drain, then run the transition on the replica. ### Authoring rules and runbooks For authoring details (what is and isn't safe to put in `schema.kql`), see the [schema README](https://github.com/nscaledev/fleet-manager/tree/main/src/fleet_manager/agents/netbox_matrix/schema). For operational runbooks that touch the schema, see: * [Kusto: Deleting Records](./kusto-deleting-records) * [Kusto: Materialized View Key Changes](./kusto-materialized-view-key-changes) ## Environments The Matrix agent is only enabled in the **GLO1 Prod Hub** environment. It is not run in spokes or in dev. ### Local development * Reads from a local NetBox copy (filtered by device in the Docker Compose config) * Transforms locally * Pushes to the [Azure Data Explorer emulator](https://learn.microsoft.com/en-us/azure/data-explorer/kusto-emulator-install) (emulator setup is still TODO) ### Production — GLO1 Hub * Reads from the NetBox production `sin1` site * Transforms in the agent pod running in the `fleet-mgr-prod-glo1` Kubernetes cluster * Pushes (batched) to the [Kusto "dev" database](https://portal.azure.com/?l=en.en-gb#@nscale.no/resource/subscriptions/b3a74c1f-25d4-4455-b34e-37bf7912ea94/resourceGroups/Matrix/providers/Microsoft.Kusto/Clusters/matrix-portugal-dev/Databases/nscale-inventory-dev-db/overview), which is the customer's *development* environment * The customer's "prod" Kusto database is kept in sync continuously by Microsoft's [`kusto-copy`](https://github.com/Azure/kusto-copy) tool running alongside the agent — see [Replicating Kusto Dev to Prod](#replicating-kusto-dev-to-prod) ## Sync Cadence and Freshness Three intervals govern how data moves from NetBox to the customer Dev database. They are independent and serve different purposes — operators get asked about these regularly. | Setting | Default | What it does | | - | - | - | | `NETBOX_MATRIX_AGENT_POLL_INTERVAL` | 30 min (1800 s) | Delta cadence. Each cycle fetches NetBox objects modified since the previous cursor via `last_updated__gte`. | | `NETBOX_MATRIX_AGENT_RESYNC_INTERVAL` | 6 h (21600 s) | Controller-level store → Kusto sink re-enqueue. Refreshes staging rows (7-day retention) even when the source data hasn't changed. | | `NETBOX_MATRIX_AGENT_FULL_REFRESH_INTERVAL` | 24 h (86400 s) | Reflector-level full re-fetch from NetBox. Required to catch deletions, since deletes don't show up in `last_updated__gte` results. | ### Implications for `SyncTimestamp` `SyncTimestamp` is set per record at ingest time. A record's timestamp only advances when one of the following touches it: * the record was modified in NetBox during an incremental cycle, or * a full refresh ran (every \~24 h), or * a periodic resync re-enqueued the store (every \~6 h). In the worst case, a record that hasn't been modified in NetBox for several days will carry a `SyncTimestamp` up to roughly **25 hours old** before the next full refresh refreshes it. Different records will show different timestamps — that is expected, not a bug. ### No SLA The 30-minute incremental cadence is a target, not an SLA. It can drift longer for various reasons (NetBox response latency, Kusto ingestion queue backpressure, pod restarts). When in doubt, check the [agent dashboard](https://fleet-mgr-glo1-grafana.nscale.teleport.sh/d/agent-machinery/agent-machinery?orgId=1\&var-agent=netbox-matrix). ## Replicating Kusto Dev to Prod The agent only writes to the customer **Dev** database. The customer **Prod** database is kept in sync by Microsoft's [`kusto-copy`](https://github.com/Azure/kusto-copy) tool, which runs as a single-replica `Deployment` (`fleet-manager-hub-netbox-matrix-agent-kusto-copy`) in the same `fleet-mgr-prod-glo1` cluster as the agent. Every `iterationPeriod` (default 2 min) it polls Dev for new `*_staging` extents, exports them via an Azure Blob staging path, and ingests them into Prod. The Prod materialized views then refresh themselves from the new staging rows the same way Dev's do. ### Operating model * **Prod is "Dev with a small lag".** Steady-state replication lag is roughly one `iterationPeriod` plus export/ingest time — typically a few minutes, not hours or days. * **Forward-only and checkpointed.** `kusto-copy` runs in `BackfillAndNew` mode against a stable staging-storage path (`stmatrixkustocopy/kusto-copy/prod/netbox-matrix`). The cursor is persisted in blob storage, so a pod restart resumes from the last checkpoint — downtime becomes lag, not data loss. Don't change the staging path casually; a fresh path means a fresh stream. * **Single writer.** Only kusto-copy writes Prod staging in steady state. The chart enforces `replicas: 1` with `strategy: Recreate` to avoid two `kc` processes contending over the same tracking lease. * **Data only.** kusto-copy replicates staging-table inserts. It does **not** replicate schema DDL, materialized-view recreates, `.delete` commands, or transitions — those still apply per-cluster (see [Schema Management](#schema-management)). * **Manual recovery lever.** The legacy `matrix-kusto-promote` Azure Function and its [GitHub Actions workflow](https://github.com/nscaledev/fleet-manager/actions/workflows/matrix-kusto-promote.yml) remain available via `workflow_dispatch` for emergency re-seed only. The twice-daily schedule was removed once kusto-copy reached steady state — running both writers would race the checkpointed cursor. For health checks, replication monitoring, emergency promote, common failure modes, and FAQ, see [Matrix Agent Operations](./operations). ## GraphQL PoV — Not Viable Until NetBox Upgrade We investigated whether moving the NetBox extract from REST to GraphQL would reduce load on NetBox by combining multiple queries into fewer round-trips and composing the data directly into the shape we need. NetBox's GraphQL implementation does not support pagination in our version, which is a non-starter — there is far too much data to pull without pagination. Pagination was introduced in NetBox **4.3.0**; we are currently on **4.2.9**. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.