Skip to main content
The matrix-connector ships NetBox inventory to the customer’s “Matrix” Kusto databases: it reads change events (and runs a polling backstop to catch anything missed), turns them into rows, and writes them to Kusto. Since the prod cutover (k8s-cluster-fleet-manager #638) it writes both customer databases directly — nscale-inventory-prod-db (the primary, which drives delivery success/retry and the 5-minute SLA) and nscale-inventory-dev-db (a best-effort mirror that gets the same rows at the same moment). The legacy netbox-matrix-agent, kusto-copy, and kusto-monitor are retired; rollback for the whole cutover is reverting that one PR, which restores the legacy pipeline and points the connector back at the shadow DB.
  • For operators — tell if it is healthy and, if not, who to call. No deep knowledge needed.
  • For engineers — every error class the connector can hit, each with a ready-to-paste sentinel prompt, the raw queries, the cause, and the fix.
Golden rule for operators: if the health check is not green, escalate — do not scale the connector, delete pods, or run engineer commands. When in doubt, call the Fleet Manager team.
For what a device needs in NetBox to sync at all (naming conventions, required fields, role-to-table mapping), see NetBox Data Requirements.

​
Instances

The connector is multi-instance: one process per delivery scope, sharing identical code. There is no routing inside a single pod — a second scope means a second Deployment, with its own delivery-queue partition and JetStream durable cursor, so the two never share processing state and one instance’s incident cannot stall the other’s deliveries. Every command and query below is written for sin1/Portugal, the one instance live today; substitute the column values from this table to target a different instance. Each process delivers to exactly one primary Kusto database, but two instances may be pointed at the same one when a customer wants several sites consolidated there. Instances are separated by NetBox site, so their device and cable rows occupy disjoint key spaces even in a shared database (see Disabling or rolling back one instance). The one table two instances can both write is DataCenters_staging — each also emits a row per cable far-end site, which may be the other instance’s — and there the two rows are identical in content, so the MV’s arg_max dedup resolves them. A shared database does mean both sites’ inventory is visible to anyone who can read it.
sin1 kept its pre-multi-instance resource names on purpose (useLegacyResourceNames: true in the chart) so the multi-instance rollout did not recreate its Deployment — renaming a live Deployment makes Argo delete-and-recreate rather than roll it, which would have re-opened the cold-start freshness gap the readiness gate exists to close. Every other instance, Ward included, gets its component and resource names suffixed with its instance key (-wdc1) instead.
Ward County (wdc1) status: the chart and connector code support it today, but it is not yet serving traffic — its instance entry ships with enabled: false until its Kusto delivery target (its own database, or Portugal’s) and a production Fleet Manager Site exist. Nothing in this guide’s Ward column is actionable until Fleet Manager confirms the cutover; if you see a matrix-wdc1-prod alert or a matrix-connector-wdc1 pod before that announcement, treat it as unexpected and escalate.

​
Who to call

Work down this table — most incidents end at the first row. Escalation order is always Fleet Manager → OBS Squad → customer. Escalate to Fleet Manager first; they decide when the customer is told.

​
The dashboard

The Matrix Connector dashboard (prod-hub Grafana — run tsh apps login fleet-mgr-glo1-grafana if it asks you to authenticate) is the fastest way to see the connector’s health. It has four rows; Freshness and SLA is the one that matters.
Use the Connector instance dropdown at the top of the dashboard to scope every panel in the Freshness/SLA and Backstop rows to one instance (matrix-prod / matrix-wdc1-prod — see Instances). Leave it on All to see both at once: freshness and backstop panels break their lines out by instance automatically, so a single-instance incident is visible as one red line, not a blended average that hides it.
The SLA is: inventory in Kusto within 5 minutes. The dashboard tracks the age of the oldest inventory change not yet shipped — under 5 minutes (300 s) is green, at or over is red.
If Freshness now or Worst in the last 1 hour is red, the 5-minute SLA is being missed — this is customer-visible. Escalate to Fleet Manager immediately.
The other three rows are engineer detail: HTTP reads & writes (request rate, non-2xx error ratio, latency), Projection engine (the upstream netbox-inventory-agent), and Resources (CPU, memory, file descriptors).

​
Alerts

A PrometheusRule fires the following, all evaluated by (connector) so a firing alert already names the instance in its connector label — no separate rule per instance to maintain, and AlertManager can route or silence on that label like any other. All of these route through the same Who to call table regardless of which instance’s connector label they carry — a Ward-only page is handled exactly like a sin1 one, just scoped to that customer’s data when you investigate.

​
For operators

​
Health check (30 seconds)

1

Look at the dashboard

Open the Matrix Connector dashboard. On the Freshness and SLA row, Freshness now should be green (under 5 minutes) and Backstop sweeps should show a steady rate. No cluster login needed.
2

(If needed) check the pod(s)

Log in once with tsh kube login fleet-mgr-prod-glo1, then list every connector pod at once (matches both instances’ component selectors — see Instances):
Expect one pod per enabled instance, Running and 1/1 — today that is one pod for sin1 only, since Ward is not yet enabled (a matrix-connector-wdc1 pod appearing before that announcement is unexpected). Two pods for the same instance for a few minutes during a deploy is the safe rolling update and is normal.
If the SLA panel is green for every enabled instance, the Dead-lettered jobs stat is 0, and every expected pod is up, the connector is healthy. A green SLA with a non-zero dead-letter count is NOT healthy — the freshness panel deliberately excludes terminally-failed jobs.

​
If it is not green

Everything below is for the engineer who picks up the escalation.

​
For engineers

​
Access

The Matrix Connector dashboard (prod-hub Grafana, tsh apps login fleet-mgr-glo1-grafana) is the quickest overview — start there before ad-hoc queries. Note it is a different Grafana from the central one used for the queries below.
tsh apps login observability-grafana, then open observability-grafana.nscale.teleport.sh → Explore. For a promql query pick the metrics-all (Prometheus) data source; for a logql query pick product-loki (Loki). Paste the query and set the time range top-right.
tsh kube login fleet-mgr-prod-glo1 makes the prod hub your current context, so the kubectl -n fleet-manager … commands below work with no --context flag. (The full context name is nscale.teleport.sh-fleet-mgr-prod-glo1.)
From a Fleet Manager checkout, source .agents/skills/sentinel/scripts/grafana.sh defines promql '<query>' and loki_pretty '<query>' 1h 100 so you can run the queries from a terminal instead of Grafana Explore.
Constants used throughout: Loki cluster label fleet-mgr-glo1 (no -prod-) · namespace fleet-manager. Every example below targets sin1 — deployment fleet-manager-hub-matrix-connector, connector slug matrix-prod — since it is the only instance live today; swap in the wdc1 column from Instances once Ward is enabled. service_name is fleet-manager-matrix-connector for every instance (OTel’s OTEL_SERVICE_NAME is not per-instance), so Loki queries below already cover both — Prometheus queries need the connector label from the table to tell them apart.

​
Troubleshooting with sentinel

Each error class below carries a sentinel prompt — a copy-paste task for the read-only sentinel skill (Claude Code), which queries Loki / Prometheus / Kubernetes and summarises the answer so you do not have to remember every query. Open Claude Code, invoke sentinel, and paste the prompt. No sentinel? Run the raw query in the same block.
Every prompt already names the scope — cluster fleet-mgr-glo1, namespace fleet-manager, the connector’s service_name, and a time window. Keep those when you adapt one, or sentinel searches the wrong stream.

​
Error classes

Each is the same shape: Symptom → Ask sentinel → Or check manually → Cause & fix → Escalate.
Symptom. Pod CrashLoopBackOff / Error, or 0/1 beyond the ~10-minute warm-up.Ask sentinel:
Or check manually:
Cause & fix.
  • ShadowRetentionError in the traceback → the Kusto retention policy has drifted. The connector refuses to start on purpose. Do not restart-loop it — fix the Kusto SoftDeletePeriod (must be 7 days) first.
  • A site-slug / config ValueError at startup → a bad site_filter or slug map; needs a config fix.
  • OOMKilled in events → the memory limit needs raising.
  • No error, still warming → warm-up is allowed ~10 minutes; wait before treating it as a crashloop.
If it is genuinely wedged (not warming, no retention/config error), restart it the safe way — this example is sin1; swap the Deployment name from Instances to restart Ward instead:
Never kubectl delete pod the connector and never scale any instance above 1 replica — one replica per instance is by design (each has its own durable cursor). rollout restart keeps the old pod serving until the new one is ready. Restarting one instance’s Deployment never touches the other’s — they are independent processes with independent delivery queues, so a sin1 restart cannot affect Ward’s freshness and vice versa.
Escalate. ShadowRetentionError, a config error, or a pod that does not recover after one rollout restart.
Symptom. The pod is Running but 0/1 for more than ~10 minutes.Ask sentinel:
Or check manually:
Cause & fix. Reading the exact /readyz reason needs cluster access operators do not have, so you route on 0/1 + the backlog + the logs. The reasons are bootstrap_pending / queue_unsampled (still starting — wait), queue_backlog:<seconds>s (see the Delivery-queue backlog class), or netbox_direct_full_pending (only if NetBox-direct is on — see the NetBox-direct note).Escalate. A pod that never crosses to 1/1 and is not backlog-bound.
Symptom. No events processed when NetBox changes are expected — events_total is flat.Ask sentinel:
Or check manually:
Cause & fix. Failed to subscribe to subject means it could not bind its durable consumer (often a subject/durable mismatch). Restart re-attaches it:
An empty events_total on its own is not a fault — it just means no inventory changed. Only treat it as an incident alongside Failed to subscribe to subject or a stalled heartbeat.
Escalate. Failed to subscribe to subject that returns immediately after a restart — the durable consumer is misprovisioned.
Symptom. /readyz reports queue_backlog:<n>s (pod 0/1), or the backlog gauge climbs and stays above 120 seconds.Ask sentinel:
Or check manually:
Cause & fix.
  • Matrix worker pool cycle failed — retrying → a transient DB blip; self-heals.
  • Recovered orphaned IN_PROGRESS jobs → normal self-repair after a restart.
  • Delivery job dead-lettered → a job failed repeatedly and was parked. The Dead-lettered jobs stat on the dashboard counts these rows and stays red while any exist — the freshness panel deliberately excludes them, so a green freshness with a red dead-letter count means terminally failed work is sitting unshipped. A dead-lettered delete is permanent customer staleness (the backstop regenerates live rows, never deletes). Note its error_code (see the Kusto ingestion and Devices skipped classes); this needs a data or downstream fix, not a restart.
The backlog gauge always reports a value (0 = empty = healthy). No result at all is a broken metrics scrape, not “quiet” — investigate scraping / escalate to OBS; do not assume the queue is empty.
Escalate. A backlog climbing past a few minutes, or any Delivery job dead-lettered (with the error_code).
Symptom. Rows not reaching Kusto; delivery jobs dead-letter with error_code=kusto_ingest_failed.Ask sentinel:
Or check manually:
Cause & fix. Kusto ingest is queued/async (1–5 min latency is normal), so a bad row surfaces here as a dead-lettered job rather than a crash. Sustained kusto_ingest_failed usually means Azure SP auth (the ClientSecretCredential), Kusto throttling, or a schema mismatch — all need Fleet Manager / Azure.Escalate. Sustained kusto_ingest_failed, with the quoted error.
Symptom. fleet_manager_azure_kusto_mirror_ingest_failures_total is increasing, or warning logs Mirror Kusto ingest failed. Prod-db freshness stays green. Read the reason label first: dropped (warning log Mirror upload dropped) is pending-byte-budget back-pressure — excluded from the alert and self-healing on the next sweep. Only failed / tombstone_timeout mean the mirror is not accepting rows. The alert deliberately ignores dropped, so a mirror that is persistently behind the budget (every window drops, nothing fails) is not paged — watch sum by (mirror_database)(rate(fleet_manager_azure_kusto_mirror_ingest_failures_total{reason="dropped"}[1h])) on the dashboard; a sustained non-zero rate means the dev mirror is falling behind, not failing.What it means. The connector could not queue rows into the mirror destination (nscale-inventory-dev-db on the dev cluster). For live rows this is deliberately contained — the delivery succeeds and the prod SLA is unaffected (most commonly an outage or throttling on the customer’s dev Kusto cluster, a different Azure region, which nothing on our side can write through anyway). Two tombstone (IsDeleted=true) subtleties: the legacy-schema dev-DB mirror is deliberately skipped for tombstones (supportsTombstones: false — its tables lack the IsDeleted column, so a mirrored tombstone would land as a blank live-looking row and blank the real device in its MV; deletes keep legacy semantics there, stale rows linger). For a mirror that does carry the V2 schema, a tombstone batch that fails on it fails the delivery job on purpose and retries until the mirror accepts it — deletes are never regenerated by the backstop (the repeated primary ingest is absorbed by the MV dedup).Ask sentinel:
Cause & fix. No manual replay is needed, but the repair cadence differs by table. The polling backstop re-delivers every in-scope site each sweep, so the projection tables (Servers, NVSwitches, DataCenters) and both link tables self-heal on the first sweep after the dev cluster recovers (staging duplicates are absorbed by MV dedup). The NetBox-direct tables (EthDevices, IbInventory) are NOT backstop-fed — their mirror rows repair on the NetBox-direct keep-alive re-stamp (default 6 hours, CONNECTOR_NETBOX_DIRECT_KEEPALIVE_INTERVAL) or the next NetBox change, whichever comes first. A “mirror healed” conclusion within minutes of recovery is only true for the projection and link tables. Sustained failures while the dev cluster is healthy mean auth (the mirror uses the ingest-only azure-kusto Service Principal, granted on the dev cluster only) or a schema problem — Fleet Manager / Azure.Escalate. Only if failures persist after the dev cluster is confirmed healthy, or the customer reports dev-db staleness. Live-row mirror failures are never a prod-SLA incident by themselves; tombstone retry only applies to V2-schema mirrors (the dev-DB mirror skips tombstones), where a long mirror outage shows as dead-lettered delete jobs (error_code=kusto_ingest_failed) that recover on redelivery once the cluster returns.
Symptom. Specific devices missing in Kusto while the rest of a site is present, or MatrixConnectorDevicesSkippedTransformError firing.Ask sentinel:
Or check manually:
Cause & fix. A device whose transform hits a required-field gap (for example a newly onboarded device with no asset_tag, which maps to NodeName) is skipped for that table so the rest of the site still delivers; the job marks done, so this never shows as a dead letter. The skip is metered on devices_skipped_total with error_code set to the transform exception class (ServerValidationError, NVSwitchValidationError, MatrixDeviceNotReadyError — a bounded set) and logged at ERROR as Skipped device with transform error carrying device_id, table and reason. MatrixDeviceNotReadyError is the one transient case (interfaces not projected yet) — it logs at WARNING and a later sweep delivers the device, so the alert excludes it. Every other error_code is bad or incomplete NetBox data and stays missing from Kusto until the record is fixed in NetBox — the next backstop sweep then delivers it. It is not a connector bug. If every device at the site is rejected the job still marks done with nothing ingested; only a site-level DataCenterValidationError (a site missing its description) fails the job with transform_failed and dead-letters — see the Dead-lettered jobs row above.Escalate. Only if the NetBox record is not yours to fix — route it to whoever owns it.
Symptom. Data looks stale for a window right after a rollout.Ask sentinel:
Or check manually:
Cause & fix. A brief lag while a new pod warms is expected (the rolling update holds the old pod until the new one is ready, so it should be minimal). Once the pod is 1/1 and the heartbeat is above 0, it has recovered.Escalate. A lag that does not clear within ~15 minutes.

​
Disabling or rolling back one instance

Two different things share the word “rollback” here — do not confuse them:
  • Disabling one customer instance (e.g. pulling Ward out of service without touching sin1) is a values change: set that instance’s enabled: false in the overlay and redeploy. The chart renders nothing for a disabled instance — no Deployment, no ConfigMap, no schema-apply Job — and every other enabled instance’s resources are untouched, because each instance’s rendered config only depends on its own values plus the shared defaults block. This is the fast, safe, per-customer lever; it is an engineer action (values + redeploy), not something an operator does ad hoc.
  • Rolling back the whole connector to the legacy pipeline (netbox-matrix-agent / kusto-copy / kusto-monitor) is the unrelated, chart-version-level rollback described in the intro above — reverting the cutover PR. It affects every instance at once and is a last resort, not the tool for isolating a single customer’s incident.
A config change that shares processing state between instances (for example moving siteFilter into matrixConnector.defaults instead of one instance) fails the Helm render outright rather than deploying — the chart validates that no two enabled instances can resolve to the same NetBox site, connector slug, or NATS durable consumer. If a deploy is rejected citing one of those, it is a values-authoring mistake, not a cluster problem; do not force it through.Sharing a Kusto database is the exception: two instances may deliberately deliver into one database (consolidating several sites for a customer who wants them in one place), so the chart does not reject it. Nothing in the render will catch a mistyped azureKusto.database that points one customer’s instance at another customer’s database — that one is caught only in values review, so treat any change to an instance’s database as a customer-data change.

​
If NetBox-direct is switched on

During the prod soak, NetBox-direct mode is off, so the classes above are the complete picture. If Fleet Manager later switches it on, /readyz can also report netbox_direct_full_pending while it warms, freshness metrics prefixed fleet_manager_matrix_connector_netbox_direct_ start being exported, and the log SyncLoop cycle failed — supervisor restarting appears on a failing cycle. Those are Fleet Manager signals — as an operator, treat netbox_direct_full_pending like any other “still starting up” state and escalate if it does not clear.

​
Appendix

​
Where the connector logs live

Filter Loki by service_name, always with cluster="fleet-mgr-glo1" and namespace="fleet-manager":

​
Customer heads-up template (Fleet Manager / OBS only)

Use this only if Fleet Manager decides the customer should be notified — operators do not send it:
Subject: Fleet Manager inventory pipeline — status update We are seeing [brief symptom, e.g. delayed inventory freshness] on the inventory data feed as of [time, UTC]. The Fleet Manager and Observability teams are engaged and investigating. We will update by [time]. No action is required on your side.