> ## Documentation Index > Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt > Use this file to discover all available pages before exploring further. # Matrix Connector Operations > Operations and troubleshooting runbook for the matrix-connector inventory pipeline — a quick health check and escalation path for operators, and a full error-class troubleshooting guide (with sentinel prompts) for engineers. The **matrix-connector** ships [NetBox](../start-here/glossary#netbox) inventory to the customer's "Matrix" [Kusto](../start-here/glossary#kusto) databases: it reads change events (and runs a polling backstop to catch anything missed), turns them into rows, and writes them to Kusto. Since the prod cutover (`k8s-cluster-fleet-manager` #638) it writes **both customer databases directly** — `nscale-inventory-prod-db` (the **primary**, which drives delivery success/retry and the 5-minute [SLA](../start-here/glossary#sla)) and `nscale-inventory-dev-db` (a **best-effort mirror** that gets the same rows at the same moment). The legacy `netbox-matrix-agent`, `kusto-copy`, and `kusto-monitor` are retired; **rollback for the whole cutover is reverting that one PR**, which restores the legacy pipeline and points the connector back at the shadow DB. * **[For operators](#for-operators)** — tell if it is healthy and, if not, who to call. No deep knowledge needed. * **[For engineers](#for-engineers)** — every error class the connector can hit, each with a ready-to-paste **sentinel** prompt, the raw queries, the cause, and the fix. **Golden rule for operators:** if the health check is not green, **escalate** — do not scale the connector, delete pods, or run engineer commands. When in doubt, call the Fleet Manager team. For what a [device](../start-here/glossary#node) needs in NetBox to sync at all (naming conventions, required fields, role-to-table mapping), see [NetBox Data Requirements](./netbox-data-requirements).

Instances

**The connector is multi-instance: one process per delivery scope, sharing identical code.** There is no routing inside a single pod — a second scope means a second Deployment, with its own delivery-queue partition and JetStream durable cursor, so the two never share processing state and one instance's incident cannot stall the other's deliveries. Every command and query below is written for **sin1/Portugal**, the one instance live today; substitute the column values from this table to target a different instance. Each process delivers to exactly one primary Kusto database, but two instances **may** be pointed at the same one when a customer wants several sites consolidated there. Instances are separated by NetBox site, so their device and cable rows occupy disjoint key spaces even in a shared database (see [Disabling or rolling back one instance](#instance-rollback)). The one table two instances can both write is `DataCenters_staging` — each also emits a row per cable **far-end** site, which may be the other instance's — and there the two rows are identical in content, so the MV's `arg_max` dedup resolves them. A shared database does mean both sites' inventory is visible to anyone who can read it. | | **sin1 (Portugal)** | **wdc1 (Ward County)** | | - | - | - | | Status | **Live** | Staged, not yet enabled — see [prod cutover status](#ward-status) | | Connector slug (`connector` label on every metric) | `matrix-prod` | `matrix-wdc1-prod` | | Component selector | `app.kubernetes.io/component=matrix-connector` | `app.kubernetes.io/component=matrix-connector-wdc1` | | Deployment / ConfigMap / Service name | `fleet-manager-hub-matrix-connector` | `fleet-manager-hub-mtx-conn-wdc1` | | Schema-apply Job name | `fleet-manager-hub-matrix-connector-migration` | `fleet-manager-hub-mtx-conn-mig-wdc1` | | Kusto database (primary) | `nscale-inventory-prod-db` | *(pending — either its own database or Portugal's, once the customer decides)* | | NetBox site(s) | `sin1` | `wdc1` | sin1 kept its pre-multi-instance resource names on purpose (`useLegacyResourceNames: true` in the chart) so the multi-instance rollout did not recreate its Deployment — renaming a live Deployment makes Argo delete-and-recreate rather than roll it, which would have re-opened the cold-start freshness gap the readiness gate exists to close. Every other instance, Ward included, gets its component and resource names suffixed with its instance key (`-wdc1`) instead. **Ward County (wdc1) status:** the chart and connector code support it today, but it is not yet serving traffic — its instance entry ships with `enabled: false` until its Kusto delivery target (its own database, or Portugal's) and a production Fleet Manager Site exist. Nothing in this guide's Ward column is actionable until Fleet Manager confirms the cutover; if you see a `matrix-wdc1-prod` alert or a `matrix-connector-wdc1` pod before that announcement, treat it as unexpected and escalate.

Who to call

Work down this table — most incidents end at the first row. | Situation | What to do | | - | - | | Anything customer-visible, or you are unsure | Post in Slack **`#fleetmanager-alerts-prod`**; to raise an incident, page **`@pagerduty`** there or Slack **`@fm`**. | | You need a named person | **Sulabh** or **Vedran** (Fleet Manager team). | | Alerts / dashboards / metric routing look broken (not the connector) | Escalate to the **OBS Squad**. | | A customer freshness impact is confirmed | Fleet Manager + OBS notify the **customer (Steven / Microsoft)**. Operators do **not** message the customer directly. | Escalation order is always **Fleet Manager → OBS Squad → customer**. Escalate to Fleet Manager first; they decide when the customer is told.

The dashboard

The **[Matrix Connector dashboard](https://fleet-mgr-glo1-grafana.nscale.teleport.sh/d/matrix-connector-ship/matrix-connector?orgId=1\&refresh=30s)** (prod-hub Grafana — run `tsh apps login fleet-mgr-glo1-grafana` if it asks you to authenticate) is the fastest way to see the connector's health. It has four rows; **Freshness and SLA** is the one that matters. Use the **Connector instance** dropdown at the top of the dashboard to scope every panel in the Freshness/SLA and Backstop rows to one instance (`matrix-prod` / `matrix-wdc1-prod` — see [Instances](#instances)). Leave it on **All** to see both at once: freshness and backstop panels break their lines out by instance automatically, so a single-instance incident is visible as one red line, not a blended average that hides it. **The SLA is: inventory in Kusto within 5 minutes.** The dashboard tracks the age of the oldest inventory change not yet shipped — under 5 minutes (300 s) is **green**, at or over is **red**. | Panel | What it means | Healthy | | - | - | - | | **Freshness now** | The SLA number right now (oldest undelivered change). | Green, under 5 min. | | **Worst in the last 1 h / 24 h** | The peak in that window — catches a breach that has recovered. | Green. | | **Freshness over time** | The trend; the red line is the 5-minute SLA. | Stays below the red line. | | **Keepalive stall check** | A *separate* NetBox-direct clock (7 h), **not** the SLA. Not yet split per instance — it reflects whichever instance is worse. | Ignore during the soak — may show no data. | | **Backstop sweeps** | The connector's heartbeat, one line per instance. | A steady `success` rate for every enabled instance. | If **Freshness now** or **Worst in the last 1 hour** is red, the 5-minute SLA is being missed — this is customer-visible. Escalate to Fleet Manager immediately. The other three rows are engineer detail: **HTTP reads & writes** (request rate, non-2xx error ratio, latency), **Projection engine** (the upstream netbox-inventory-agent), and **Resources** (CPU, memory, file descriptors).

Alerts

A `PrometheusRule` fires the following, all evaluated `by (connector)` so a firing alert already names the instance in its `connector` label — no separate rule per instance to maintain, and AlertManager can route or silence on that label like any other. | Alert | Severity | Fires when | | - | - | - | | `MatrixConnectorFreshnessSlaWarning` | warning | An instance's oldest undelivered job is 240–300s old. | | `MatrixConnectorFreshnessSlaBreached` | critical | An instance's oldest undelivered job exceeds the 300s SLA. | | `MatrixConnectorDeadLettersPresent` | warning | An instance has any `DEAD_LETTER` delivery-queue row. | | `MatrixConnectorEventErrorRateHigh` | warning | An instance fails to route more than 5% of consumed events over 15m. | | `MatrixConnectorBackstopSweepsFailing` | warning | More than a quarter of an instance's polling backstop sweeps have errored for 20m. | | `MatrixConnectorBackstopSweepsAbsent` | warning | An instance has had no successful backstop sweep in 30m. | | `MatrixConnectorDevicesSkippedTransformError` | warning | A connector pod (`pod`, `table`, `error_code` labels) has skipped devices with required-field gaps for 15m — those devices are missing from Kusto until the NetBox data is fixed (see the **Devices skipped** runbook). | | `MatrixConnectorKustoMirrorIngestFailing` | warning | A mirror database (`mirror_database` label) has had ingest failures in every 15m window for 20m, excluding pending-byte-budget drops (`reason="dropped"`), which self-heal on the next sweep. | | `MatrixConnectorPodRestarting` | warning | A connector pod (`pod` label) restarted more than 3 times in an hour. | | `MatrixConnectorScrapeTargetsAbsent` | critical | No connector pod in the namespace is exporting metrics at all — namespace-wide, does not identify which instance (see the dashboard note on the same gap). | All of these route through the same [Who to call](#who-to-call) table regardless of which instance's `connector` label they carry — a Ward-only page is handled exactly like a sin1 one, just scoped to that customer's data when you investigate.

For operators

### Health check (30 seconds) Open the [Matrix Connector dashboard](#dashboard). On the **Freshness and SLA** row, **Freshness now** should be green (under 5 minutes) and **Backstop sweeps** should show a steady rate. No cluster login needed. Log in once with `tsh kube login fleet-mgr-prod-glo1`, then list every connector pod at once (matches both instances' component selectors — see [Instances](#instances)): ```bash theme={null} kubectl -n fleet-manager get pods -l 'app.kubernetes.io/component in (matrix-connector,matrix-connector-wdc1)' ``` Expect one pod **per enabled instance**, `Running` and `1/1` — today that is one pod for sin1 only, since Ward is not yet enabled (a `matrix-connector-wdc1` pod appearing before that announcement is unexpected). Two pods for the *same* instance for a few minutes during a deploy is the safe rolling update and is normal. If the SLA panel is green for every enabled instance, the **Dead-lettered jobs** stat is **0**, and every expected pod is up, the connector is healthy. A green SLA with a non-zero dead-letter count is NOT healthy — the freshness panel deliberately excludes terminally-failed jobs. ### If it is not green | What you see | What to do | | - | - | | The **Freshness** (SLA) panel is red | The 5-minute SLA is being missed and it is customer-visible — **escalate immediately** (`@fm` / page `@pagerduty`). | | A pod `CrashLoopBackOff` / `Error`, or `0/1` for more than \~10 minutes | **Escalate to Fleet Manager.** Do not restart or scale it yourself. | | The **Dead-lettered jobs** stat is above 0 | Terminally-failed deliveries are parked — a dead-lettered delete is permanent customer staleness. **Escalate to Fleet Manager** with the `error_code` from the `Delivery job dead-lettered` log line. | | A customer reports missing or wrong inventory data | Capture what they reported and **escalate**. Do not run comparison scripts yourself. | | Freshness briefly delayed right after a deploy | Expected while the new pod warms up. If it does not recover within \~15 minutes, capture the time and escalate. | Everything below is for the engineer who picks up the escalation.

For engineers

### Access The [Matrix Connector dashboard](#dashboard) (prod-hub Grafana, `tsh apps login fleet-mgr-glo1-grafana`) is the quickest overview — start there before ad-hoc queries. Note it is a **different** Grafana from the central one used for the queries below. `tsh apps login observability-grafana`, then open [observability-grafana.nscale.teleport.sh](https://observability-grafana.nscale.teleport.sh) → **Explore**. For a `promql` query pick the **`metrics-all`** (Prometheus) data source; for a `logql` query pick **`product-loki`** (Loki). Paste the query and set the time range top-right. `tsh kube login fleet-mgr-prod-glo1` makes the prod hub your current context, so the `kubectl -n fleet-manager …` commands below work with no `--context` flag. (The full context name is `nscale.teleport.sh-fleet-mgr-prod-glo1`.) From a Fleet Manager checkout, `source .agents/skills/sentinel/scripts/grafana.sh` defines `promql ''` and `loki_pretty '' 1h 100` so you can run the queries from a terminal instead of Grafana Explore. **Constants used throughout:** Loki cluster label **`fleet-mgr-glo1`** (no `-prod-`) · namespace **`fleet-manager`**. Every example below targets **sin1** — deployment **`fleet-manager-hub-matrix-connector`**, connector slug **`matrix-prod`** — since it is the only instance live today; swap in the **wdc1** column from [Instances](#instances) once Ward is enabled. `service_name` is **`fleet-manager-matrix-connector`** for every instance (OTel's `OTEL_SERVICE_NAME` is not per-instance), so Loki queries below already cover both — Prometheus queries need the `connector` label from the table to tell them apart. ### Troubleshooting with sentinel Each error class below carries a **sentinel prompt** — a copy-paste task for the read-only **sentinel** skill (Claude Code), which queries Loki / Prometheus / Kubernetes and summarises the answer so you do not have to remember every query. Open Claude Code, invoke `sentinel`, and paste the prompt. No sentinel? Run the **raw query** in the same block. Every prompt already names the scope — cluster `fleet-mgr-glo1`, namespace `fleet-manager`, the connector's `service_name`, and a time window. Keep those when you adapt one, or sentinel searches the wrong stream. ### Error classes Each is the same shape: **Symptom → Ask sentinel → Or check manually → Cause & fix → Escalate.** **Symptom.** Pod `CrashLoopBackOff` / `Error`, or `0/1` beyond the \~10-minute warm-up. **Ask sentinel:** ```text theme={null} On the fleet-mgr-glo1 prod hub, namespace fleet-manager, investigate the matrix-connector pod (label app.kubernetes.io/component=matrix-connector). Get its status and restart count, and pull the last 200 log lines from service_name="fleet-manager-matrix-connector". Tell me if you see ShadowRetentionError, a config/site-slug error, an OOMKilled event, or an image-pull error, and quote the failing line. ``` **Or check manually:** ```bash theme={null} kubectl -n fleet-manager describe pod -l app.kubernetes.io/component=matrix-connector kubectl -n fleet-manager logs -l app.kubernetes.io/component=matrix-connector --tail=200 ``` **Cause & fix.** * **`ShadowRetentionError`** in the traceback → the Kusto retention policy has drifted. The connector refuses to start on purpose. **Do not restart-loop it** — fix the Kusto `SoftDeletePeriod` (must be 7 days) first. * A **site-slug / config `ValueError`** at startup → a bad `site_filter` or slug map; needs a config fix. * **`OOMKilled`** in events → the memory limit needs raising. * **No error, still warming** → warm-up is allowed \~10 minutes; wait before treating it as a crashloop. If it is genuinely wedged (not warming, no retention/config error), restart it the safe way — this example is sin1; swap the Deployment name from [Instances](#instances) to restart Ward instead: ```bash theme={null} kubectl -n fleet-manager rollout restart deploy/fleet-manager-hub-matrix-connector ``` **Never** `kubectl delete pod` the connector and **never** scale any instance above **1** replica — one replica per instance is by design (each has its own durable cursor). `rollout restart` keeps the old pod serving until the new one is ready. Restarting one instance's Deployment never touches the other's — they are independent processes with independent delivery queues, so a sin1 restart cannot affect Ward's freshness and vice versa. **Escalate.** `ShadowRetentionError`, a config error, or a pod that does not recover after one `rollout restart`. **Symptom.** The pod is `Running` but `0/1` for more than \~10 minutes. **Ask sentinel:** ```text theme={null} On fleet-mgr-glo1 / fleet-manager, the matrix-connector pod is 0/1. Describe the pod and report the Readiness probe events, and check the delivery-queue backlog metric fleet_manager_matrix_connector_delivery_queue_oldest_undelivered_age_seconds and the backstop heartbeat increase(fleet_manager_matrix_connector_backstop_sweeps_total{connector="matrix-prod",outcome="success"}[15m]). Tell me whether the backlog is the cause. ``` **Or check manually:** ```bash theme={null} kubectl -n fleet-manager describe pod -l app.kubernetes.io/component=matrix-connector ``` **Cause & fix.** Reading the exact `/readyz` reason needs cluster access operators do not have, so you route on `0/1` + the backlog + the logs. The reasons are `bootstrap_pending` / `queue_unsampled` (still starting — wait), `queue_backlog:s` (see the **Delivery-queue backlog** class), or `netbox_direct_full_pending` (only if NetBox-direct is on — see the [NetBox-direct note](#netbox-direct)). **Escalate.** A pod that never crosses to `1/1` and is not backlog-bound. **Symptom.** No events processed when NetBox changes are expected — `events_total` is flat. **Ask sentinel:** ```text theme={null} On fleet-mgr-glo1 / fleet-manager, service_name="fleet-manager-matrix-connector", search the last 1h for the log "Failed to subscribe to subject". Also give me the 1h rate of fleet_manager_matrix_connector_events_total{connector="matrix-prod"} grouped by outcome. Tell me if the connector is failing to attach to its NATS stream or just idle. ``` **Or check manually:** ```logql theme={null} {cluster="fleet-mgr-glo1",namespace="fleet-manager",service_name="fleet-manager-matrix-connector"} |= "Failed to subscribe to subject" ``` **Cause & fix.** `Failed to subscribe to subject` means it could not bind its durable consumer (often a subject/durable mismatch). Restart re-attaches it: ```bash theme={null} kubectl -n fleet-manager rollout restart deploy/fleet-manager-hub-matrix-connector ``` An empty `events_total` on its own is **not** a fault — it just means no inventory changed. Only treat it as an incident alongside `Failed to subscribe to subject` or a stalled heartbeat. **Escalate.** `Failed to subscribe to subject` that returns immediately after a restart — the durable consumer is misprovisioned. **Symptom.** `/readyz` reports `queue_backlog:s` (pod `0/1`), or the backlog gauge climbs and stays above **120** seconds. **Ask sentinel:** ```text theme={null} On fleet-mgr-glo1 / fleet-manager, check the matrix-connector delivery queue. Query fleet_manager_matrix_connector_delivery_queue_oldest_undelivered_age_seconds (climbing or draining?), and grep service_name="fleet-manager-matrix-connector" over the last 2h for "Delivery job dead-lettered", "Recovered orphaned IN_PROGRESS jobs", and "Matrix worker pool cycle failed". Report the backlog age and any dead-letter error_codes. ``` **Or check manually:** ```promql theme={null} fleet_manager_matrix_connector_delivery_queue_oldest_undelivered_age_seconds ``` ```logql theme={null} {cluster="fleet-mgr-glo1",namespace="fleet-manager",service_name="fleet-manager-matrix-connector"} |~ "dead-lettered|Recovered orphaned|worker pool cycle failed" ``` **Cause & fix.** * `Matrix worker pool cycle failed — retrying` → a transient DB blip; **self-heals**. * `Recovered orphaned IN_PROGRESS jobs` → normal self-repair after a restart. * `Delivery job dead-lettered` → a job failed repeatedly and was parked. The **Dead-lettered jobs** stat on the dashboard counts these rows and stays red while any exist — the freshness panel deliberately excludes them, so a green freshness with a red dead-letter count means terminally failed work is sitting unshipped. A dead-lettered **delete** is permanent customer staleness (the backstop regenerates live rows, never deletes). Note its `error_code` (see the **Kusto ingestion** and **Devices skipped** classes); this needs a data or downstream fix, not a restart. The backlog gauge **always** reports a value (`0` = empty = healthy). **No result at all** is a broken metrics scrape, not "quiet" — investigate scraping / escalate to OBS; do not assume the queue is empty. **Escalate.** A backlog climbing past a few minutes, or any `Delivery job dead-lettered` (with the `error_code`). **Symptom.** Rows not reaching Kusto; delivery jobs dead-letter with `error_code=kusto_ingest_failed`. **Ask sentinel:** ```text theme={null} On fleet-mgr-glo1 / fleet-manager, service_name="fleet-manager-matrix-connector", search the last 2h for "Failed to ingest" (regex "Failed to ingest .* to Kusto") and for "kusto_ingest_failed". Quote the errors and tell me if they look like Kusto/Azure auth (ClientSecretCredential), throttling, or a schema/ingestion error. ``` **Or check manually:** ```logql theme={null} {cluster="fleet-mgr-glo1",namespace="fleet-manager",service_name="fleet-manager-matrix-connector"} |~ "Failed to ingest .* to Kusto|kusto_ingest_failed" ``` **Cause & fix.** Kusto ingest is queued/async (1–5 min latency is normal), so a bad row surfaces here as a dead-lettered job rather than a crash. Sustained `kusto_ingest_failed` usually means Azure SP auth (the `ClientSecretCredential`), Kusto throttling, or a schema mismatch — all need Fleet Manager / Azure. **Escalate.** Sustained `kusto_ingest_failed`, with the quoted error. **Symptom.** `fleet_manager_azure_kusto_mirror_ingest_failures_total` is increasing, or warning logs `Mirror Kusto ingest failed`. Prod-db freshness stays green. Read the `reason` label first: `dropped` (warning log `Mirror upload dropped`) is pending-byte-budget back-pressure — excluded from the alert and self-healing on the next sweep. Only `failed` / `tombstone_timeout` mean the mirror is not accepting rows. The alert deliberately ignores `dropped`, so a mirror that is *persistently* behind the budget (every window drops, nothing fails) is not paged — watch `sum by (mirror_database)(rate(fleet_manager_azure_kusto_mirror_ingest_failures_total{reason="dropped"}[1h]))` on the dashboard; a sustained non-zero rate means the dev mirror is falling behind, not failing. **What it means.** The connector could not queue rows into the **mirror** destination (`nscale-inventory-dev-db` on the dev cluster). For **live rows** this is deliberately contained — the delivery succeeds and the prod SLA is unaffected (most commonly an outage or throttling on the customer's dev Kusto cluster, a different Azure region, which nothing on our side can write through anyway). Two tombstone (`IsDeleted=true`) subtleties: the legacy-schema dev-DB mirror is **deliberately skipped for tombstones** (`supportsTombstones: false` — its tables lack the `IsDeleted` column, so a mirrored tombstone would land as a blank live-looking row and blank the real device in its MV; deletes keep legacy semantics there, stale rows linger). For a mirror that **does** carry the V2 schema, a tombstone batch that fails on it **fails the delivery job on purpose** and retries until the mirror accepts it — deletes are never regenerated by the backstop (the repeated primary ingest is absorbed by the MV dedup). **Ask sentinel:** ```text theme={null} On fleet-mgr-glo1 / fleet-manager, service_name="fleet-manager-matrix-connector", search the last 2h for "Mirror Kusto ingest failed" and quote the mirror_cluster/mirror_database and the underlying error. Also plot rate(fleet_manager_azure_kusto_mirror_ingest_failures_total[15m]) by (table, reason), treating reason="dropped" as benign back-pressure. ``` **Cause & fix.** No manual replay is needed, but the repair cadence differs by table. The polling backstop re-delivers every in-scope site each sweep, so the **projection tables (Servers, NVSwitches, DataCenters) and both link tables self-heal on the first sweep after the dev cluster recovers** (staging duplicates are absorbed by MV dedup). The **NetBox-direct tables (EthDevices, IbInventory) are NOT backstop-fed** — their mirror rows repair on the NetBox-direct keep-alive re-stamp (default 6 hours, `CONNECTOR_NETBOX_DIRECT_KEEPALIVE_INTERVAL`) or the next NetBox change, whichever comes first. A "mirror healed" conclusion within minutes of recovery is only true for the projection and link tables. Sustained failures while the dev cluster is healthy mean auth (the mirror uses the ingest-only `azure-kusto` Service Principal, granted on the dev cluster only) or a schema problem — Fleet Manager / Azure. **Escalate.** Only if failures persist after the dev cluster is confirmed healthy, or the customer reports dev-db staleness. Live-row mirror failures are never a prod-SLA incident by themselves; tombstone retry only applies to V2-schema mirrors (the dev-DB mirror skips tombstones), where a long mirror outage shows as dead-lettered delete jobs (`error_code=kusto_ingest_failed`) that recover on redelivery once the cluster returns. **Symptom.** Specific devices missing in Kusto while the rest of a site is present, or `MatrixConnectorDevicesSkippedTransformError` firing. **Ask sentinel:** ```text theme={null} On fleet-mgr-glo1 / fleet-manager, query fleet_manager_matrix_connector_devices_skipped_total grouped by table and error_code, and grep service_name="fleet-manager-matrix-connector" for "Skipped device with transform error" over the last 6h. Tell me which error_codes dominate and list the device_id and reason from each log line. ``` **Or check manually:** ```promql theme={null} sum by (table, error_code) (increase(fleet_manager_matrix_connector_devices_skipped_total[6h])) ``` **Cause & fix.** A device whose transform hits a required-field gap (for example a newly onboarded device with no `asset_tag`, which maps to `NodeName`) is skipped for that table so the rest of the site still delivers; the job marks done, so this never shows as a dead letter. The skip is metered on `devices_skipped_total` with `error_code` set to the transform exception class (`ServerValidationError`, `NVSwitchValidationError`, `MatrixDeviceNotReadyError` — a bounded set) and logged at ERROR as `Skipped device with transform error` carrying `device_id`, `table` and `reason`. `MatrixDeviceNotReadyError` is the one transient case (interfaces not projected yet) — it logs at WARNING and a later sweep delivers the device, so the alert excludes it. Every other `error_code` is **bad or incomplete NetBox data** and stays missing from Kusto until the record is fixed in NetBox — the next backstop sweep then delivers it. It is not a connector bug. If every device at the site is rejected the job still marks done with nothing ingested; only a site-level `DataCenterValidationError` (a site missing its description) fails the job with `transform_failed` and dead-letters — see the **Dead-lettered jobs** row above. **Escalate.** Only if the NetBox record is not yours to fix — route it to whoever owns it. **Symptom.** Cross-DC links show a missing or wrong far-end datacenter / device / interface. **Ask sentinel:** ```text theme={null} On fleet-mgr-glo1 / fleet-manager, service_name="fleet-manager-netbox-inventory-agent", search the last 6h for "Validation failed - record skipped" and the requeue warnings "site_not_in_store"/"device_not_in_store"/"interface_not_in_store"/"cable_not_in_store" — the link resolvers run inside the inventory agent, so resolution failures surface THERE, before a cable projection ever reaches the connector. Also check sum by (reason) (increase(fleet_manager_inventory_agent_link_resolution_skips_total[6h])). On the connector side search only for "cable_projection_invalid" (a malformed projection that was delivered anyway). Tell me if far-end sites are failing to resolve and where. ``` **Or check manually:** ```logql theme={null} {cluster="fleet-mgr-glo1",namespace="fleet-manager",service_name="fleet-manager-matrix-connector"} |~ "cable_projection_invalid|not_in_store|Validation failed - record skipped" ``` **Cause & fix.** `*_not_in_store` warnings are transient — the far side is retried next cycle, so a few are normal. **Escalate.** Persistent requeues where far-end sites never resolve, or `cable_projection_invalid` dead-letters (the cross-DC far-end resolution path) — include an example cable. **Symptom.** Data looks stale for a window right after a rollout. **Ask sentinel:** ```text theme={null} On fleet-mgr-glo1 / fleet-manager, check when the matrix-connector pod last restarted and whether /readyz is warm now (READY 1/1). Query the backstop heartbeat increase(fleet_manager_matrix_connector_backstop_sweeps_total{connector="matrix-prod",outcome="success"}[15m]) and tell me if it recovered after the restart. ``` **Or check manually:** ```bash theme={null} kubectl -n fleet-manager get pods -l app.kubernetes.io/component=matrix-connector ``` ```promql theme={null} increase(fleet_manager_matrix_connector_backstop_sweeps_total{connector="matrix-prod",outcome="success"}[15m]) ``` **Cause & fix.** A brief lag while a new pod warms is expected (the rolling update holds the old pod until the new one is ready, so it should be minimal). Once the pod is `1/1` and the heartbeat is above 0, it has recovered. **Escalate.** A lag that does not clear within \~15 minutes.

Disabling or rolling back one instance

Two different things share the word "rollback" here — do not confuse them: * **Disabling one customer instance** (e.g. pulling Ward out of service without touching sin1) is a values change: set that instance's `enabled: false` in the overlay and redeploy. The chart renders **nothing** for a disabled instance — no Deployment, no ConfigMap, no schema-apply Job — and every other enabled instance's resources are untouched, because each instance's rendered config only depends on its own values plus the shared `defaults` block. This is the fast, safe, per-customer lever; it is an engineer action (values + redeploy), not something an operator does ad hoc. * **Rolling back the whole connector to the legacy pipeline** (`netbox-matrix-agent` / `kusto-copy` / `kusto-monitor`) is the unrelated, chart-version-level rollback described in the intro above — reverting the cutover PR. It affects every instance at once and is a last resort, not the tool for isolating a single customer's incident. A config change that shares **processing** state between instances (for example moving `siteFilter` into `matrixConnector.defaults` instead of one instance) fails the [Helm](../start-here/glossary#helm) render outright rather than deploying — the chart validates that no two enabled instances can resolve to the same NetBox site, connector slug, or [NATS](../start-here/glossary#nats) durable consumer. If a deploy is rejected citing one of those, it is a values-authoring mistake, not a cluster problem; do not force it through. Sharing a **Kusto database** is the exception: two instances may deliberately deliver into one database (consolidating several sites for a customer who wants them in one place), so the chart does **not** reject it. Nothing in the render will catch a mistyped `azureKusto.database` that points one customer's instance at another customer's database — that one is caught only in values review, so treat any change to an instance's `database` as a customer-data change.

If NetBox-direct is switched on

During the prod soak, NetBox-direct mode is **off**, so the classes above are the complete picture. If Fleet Manager later switches it on, `/readyz` can also report `netbox_direct_full_pending` while it warms, freshness metrics prefixed `fleet_manager_matrix_connector_netbox_direct_` start being exported, and the log `SyncLoop cycle failed — supervisor restarting` appears on a failing cycle. Those are Fleet Manager signals — as an operator, treat `netbox_direct_full_pending` like any other "still starting up" state and escalate if it does not clear. ## Appendix ### Where the connector logs live Filter Loki by **`service_name`**, always with `cluster="fleet-mgr-glo1"` and `namespace="fleet-manager"`: | Component | Loki `service_name` | | - | - | | Matrix connector | `fleet-manager-matrix-connector` | ### Customer heads-up template (Fleet Manager / OBS only) Use this **only** if Fleet Manager decides the customer should be notified — operators do not send it: > Subject: Fleet Manager inventory pipeline — status update > > We are seeing \[brief symptom, e.g. delayed inventory freshness] on the inventory data feed as of > \[time, UTC]. The Fleet Manager and Observability teams are engaged and investigating. We will > update by \[time]. No action is required on your side. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.