> ## Documentation Index
> Fetch the complete documentation index at: https://docs.internal.nscale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Claim workflow cancellation work (internal)

> Claim this cycle's cancellation-dispatch work for one replica.

Internal endpoint: replaces the plain CANCELLATION_REQUESTED list read so
each pending cancellation is dispatched to Temporal by exactly one replica
per TTL window. The response lease lets the controller discard stale
cached work after another replica reclaims it.



## OpenAPI

````yaml /openapi/fleet-manager-openapi.yaml get /api/v1/jobs/cancel-work/
openapi: 3.0.3
info:
  title: Fleet Manager API
  version: 1.0.0
  description: API for the Nscale Fleet Manager
servers: []
security: []
paths:
  /api/v1/jobs/cancel-work/:
    get:
      tags:
        - Cancel Work
      summary: Claim workflow cancellation work (internal)
      description: >-
        Claim this cycle's cancellation-dispatch work for one replica.


        Internal endpoint: replaces the plain CANCELLATION_REQUESTED list read
        so

        each pending cancellation is dispatched to Temporal by exactly one
        replica

        per TTL window. The response lease lets the controller discard stale

        cached work after another replica reclaims it.
      operationId: api_v1_jobs_cancel_work_list
      parameters:
        - in: query
          name: limit
          schema:
            type: integer
            maximum: 200
            minimum: 1
            default: 200
        - in: query
          name: site_id
          schema:
            type: string
            format: uuid
        - in: query
          name: site_slug
          schema:
            type: string
            minLength: 1
        - in: query
          name: ttl_seconds
          schema:
            type: integer
            maximum: 3600
            minimum: 5
          description: >-
            Claim lease duration. Pass the agent's poll interval so claim expiry
            doubles as the poll schedule; defaults are per operation.
      responses:
        '200':
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/JobWorkClaimItem'
          description: ''
        '400':
          description: Validation error.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '401':
          description: Authentication credentials were not provided or are invalid.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '403':
          description: You do not have permission to perform this action.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
      security:
        - oauth2:
            - job:update
        - cookieAuth: []
components:
  schemas:
    JobWorkClaimItem:
      type: object
      description: >-
        Internal recurring-work item carrying its operation lease.


        These values are query annotations from the just-acquired JobWorkClaim
        row.

        They stay off ordinary job reads, but let a controller reject an expired

        cached item after another replica has taken the next lease.
      properties:
        id:
          type: string
          format: uuid
          readOnly: true
        workflow:
          allOf:
            - $ref: '#/components/schemas/WorkflowSummary'
          readOnly: true
        created_by:
          type: string
          nullable: true
          description: 'Return the creator: username, app client_id with prefix, or None.'
          readOnly: true
        ai_failure_summary:
          allOf:
            - $ref: '#/components/schemas/AIJobFailureSummary'
          nullable: true
          readOnly: true
          description: >-
            AI-generated failure summary for jobs that ended in FAILED or
            PARTIALLY_FAILED
        test_slugs:
          type: array
          items:
            $ref: '#/components/schemas/TestSlugEnum'
          readOnly: true
          description: Distinct test slugs from associated test executions
        hostlist:
          type: string
          readOnly: true
          description: Compressed hostlist derived from the job's associated device names
        chad_sop:
          allOf:
            - $ref: '#/components/schemas/ChadSop'
          nullable: true
          readOnly: true
          description: >-
            The CHAD SOP recipe that dispatched this job; null when the job was
            not dispatched by CHAD.
        links:
          allOf:
            - $ref: '#/components/schemas/JobLinks'
          readOnly: true
        devices:
          type: array
          items:
            $ref: '#/components/schemas/JobDeviceSummary'
          readOnly: true
          description: >-
            Per-device name + physical location (rack, U, networking pod) for
            the job's devices, ordered by name — a slim summary the UI
            aggregates into rack/pod chips. Carried by the jobs list only;
            omitted from responses that do not project it. The full device
            objects are at links.devices.
        device_count:
          type: integer
          description: >-
            Number of devices the job runs on. A convenience projection of the
            same set `devices` lists and `hostlist` compresses, so all three
            always agree. Carried by the jobs list only.
          readOnly: true
        failed_device_count:
          type: integer
          description: >-
            Number of the job's devices that failed burn-in evaluation, counting
            each device once however many of the job's tests it failed. Sourced
            from the evaluation outcome of the job's test executions, so it
            reflects the threshold verdict rather than the Temporal result: a
            burn-in job whose devices all report `completed` still counts them
            here when their metrics breached thresholds. A test that has not
            been evaluated yet is not counted, and a job that runs no burn-in
            tests reports 0 — read `unevaluated_device_count` to tell those two
            apart. A failing multi-device test implicates every device in its
            cohort, so one bad device in a collective run counts the whole
            cohort. Counts debug-mode and stale-marked evidence too, matching
            what the test-executions list returns for the job. Carried by the
            jobs list only.
          readOnly: true
        unevaluated_device_count:
          type: integer
          description: >-
            Number of the job's devices whose burn-in evidence finished but has
            never been evaluated, so `failed_device_count` cannot speak for
            them. Non-zero means this row's failure count is incomplete rather
            than clean: the test-executions list evaluates these rows on read,
            so it can show a failure the count does not. Counted once per
            device, and 0 for a job that runs no burn-in tests. Carried by the
            jobs list only.
          readOnly: true
        work_claimed_at:
          type: string
          format: date-time
          readOnly: true
        work_claim_expires_at:
          type: string
          format: date-time
          readOnly: true
        created_at:
          type: string
          format: date-time
          readOnly: true
        updated_at:
          type: string
          format: date-time
          readOnly: true
        name:
          type: string
          maxLength: 50
          pattern: ^[-a-zA-Z0-9_]+$
        parameters:
          readOnly: true
        max_attempts_override:
          type: integer
          readOnly: true
          nullable: true
          description: >-
            Per-job override for Temporal activity retry budgets. NULL uses the
            worker's ActivityRetryConfigs defaults; 1 fails fast (no retries)
            for OpenStack debugging. Must be >= 1 — Temporal treats
            maximum_attempts=0 as unlimited retries.
        status:
          $ref: '#/components/schemas/JobStatusEnum'
        started_at:
          type: string
          format: date-time
          nullable: true
        completed_at:
          type: string
          format: date-time
          nullable: true
        temporal_workflow_id:
          type: string
          nullable: true
          maxLength: 255
        source_test_execution:
          type: string
          format: uuid
          readOnly: true
          nullable: true
          description: >-
            Set when this job was created via a single-test re-run; points to
            the source execution.
      required:
        - ai_failure_summary
        - chad_sop
        - created_at
        - created_by
        - device_count
        - devices
        - failed_device_count
        - hostlist
        - id
        - links
        - max_attempts_override
        - parameters
        - source_test_execution
        - test_slugs
        - unevaluated_device_count
        - updated_at
        - work_claim_expires_at
        - work_claimed_at
        - workflow
    ErrorResponse:
      type: object
      description: >-
        Uniform error body for every 4xx/5xx response. `detail` is a
        human-readable message, always present, safe to display to end users.
        `code` is the machine-readable identifier. `field_errors` lists
        per-input-field validation messages and is empty for non-field errors
        (auth, permission, not-found, conflict, server).
      required:
        - detail
        - code
        - field_errors
      properties:
        detail:
          type: string
        code:
          $ref: '#/components/schemas/ErrorCodeEnum'
        field_errors:
          type: array
          items:
            $ref: '#/components/schemas/FieldError'
    WorkflowSummary:
      type: object
      description: >-
        Compact workflow reference embedded in every job response.


        Includes the fields required by the fleet-manager client's WorkflowRef

        model while skipping expensive computed fields like parameters_presets.


        `category` is here because the UI reads it off a job's workflow, and it
        costs

        nothing: `Workflow.category` is a dict lookup on
        `temporal_workflow_name`, not a

        column or a query. Declared exactly as `WorkflowSerializer` declares it
        so both

        resolve to one named enum component instead of splitting into two
        generated types.
      properties:
        id:
          type: string
          format: uuid
          readOnly: true
        name:
          type: string
          maxLength: 255
        temporal_workflow_name:
          $ref: '#/components/schemas/TemporalWorkflowNameEnum'
        organisation:
          type: string
          format: uuid
        category:
          allOf:
            - $ref: '#/components/schemas/WorkflowCategoryEnum'
          readOnly: true
      required:
        - category
        - id
        - name
        - organisation
        - temporal_workflow_name
    AIJobFailureSummary:
      type: object
      description: Serializer for AIJobFailureSummary.
      properties:
        headline:
          type: string
          description: Concise job-level summary (max ~80 chars)
        explanation:
          type: string
          description: Narrative of what happened across devices
        failure_categories:
          type: object
          additionalProperties:
            type: integer
          description: Map of failure category to count of affected devices
      required:
        - explanation
        - failure_categories
        - headline
    TestSlugEnum:
      enum:
        - multi-node.hello-world
        - multi-node.iperf-matrix
        - multi-node.nccl-all-gather
        - multi-node.nccl-all-reduce
        - multi-node.nccl-all-reduce-multi-rack
        - multi-node.nccl-all-to-all
        - multi-node.nvloom
        - multi-node.pairwise-net-gpu.bw
        - multi-node.pairwise-net-gpu.lat
        - multi-node.pairwise-net.bw
        - multi-node.pairwise-net.lat
        - multi-node.s3-elbencho
        - multi-node.storage-elbencho
        - multi-node.training-gpt-oss-120b
        - multi-node.training-llama3-1-405b
        - multi-node.training-llama3-70b
        - multi-node.training-nemotron-24x2h
        - multi-node.training-nemotron-checkpoint
        - multi-node.training-nemotron-h
        - multi-node.training-nemotron-h-multi-rack
        - multi-node.training-qwen3-235b
        - single-node.babelstream
        - single-node.dcgmi-diag
        - single-node.gpu-fryer
        - single-node.hello-world
        - single-node.hpl-cpu
        - single-node.hpl-gpu
        - single-node.lmbench-lat-mem-rd
        - single-node.memtester
        - single-node.nccl-all-gather
        - single-node.nccl-all-reduce
        - single-node.nccl-all-reduce-loopback
        - single-node.nccl-all-to-all
        - single-node.node-health-check
        - single-node.nvme-fio
        - single-node.nvme-self-test
        - single-node.storage-elbencho
        - single-node.stream
        - single-node.stress-ng-cpu
        - single-node.training-llama3-8b
        - single-node.training-nemotronh-8b
      type: string
      description: >-
        * `multi-node.hello-world` - multi-node.hello-world

        * `multi-node.iperf-matrix` - multi-node.iperf-matrix

        * `multi-node.nccl-all-gather` - multi-node.nccl-all-gather

        * `multi-node.nccl-all-reduce` - multi-node.nccl-all-reduce

        * `multi-node.nccl-all-reduce-multi-rack` -
        multi-node.nccl-all-reduce-multi-rack

        * `multi-node.nccl-all-to-all` - multi-node.nccl-all-to-all

        * `multi-node.nvloom` - multi-node.nvloom

        * `multi-node.pairwise-net-gpu.bw` - multi-node.pairwise-net-gpu.bw

        * `multi-node.pairwise-net-gpu.lat` - multi-node.pairwise-net-gpu.lat

        * `multi-node.pairwise-net.bw` - multi-node.pairwise-net.bw

        * `multi-node.pairwise-net.lat` - multi-node.pairwise-net.lat

        * `multi-node.s3-elbencho` - multi-node.s3-elbencho

        * `multi-node.storage-elbencho` - multi-node.storage-elbencho

        * `multi-node.training-gpt-oss-120b` - multi-node.training-gpt-oss-120b

        * `multi-node.training-llama3-1-405b` -
        multi-node.training-llama3-1-405b

        * `multi-node.training-llama3-70b` - multi-node.training-llama3-70b

        * `multi-node.training-nemotron-24x2h` -
        multi-node.training-nemotron-24x2h

        * `multi-node.training-nemotron-checkpoint` -
        multi-node.training-nemotron-checkpoint

        * `multi-node.training-nemotron-h` - multi-node.training-nemotron-h

        * `multi-node.training-nemotron-h-multi-rack` -
        multi-node.training-nemotron-h-multi-rack

        * `multi-node.training-qwen3-235b` - multi-node.training-qwen3-235b

        * `single-node.babelstream` - single-node.babelstream

        * `single-node.dcgmi-diag` - single-node.dcgmi-diag

        * `single-node.gpu-fryer` - single-node.gpu-fryer

        * `single-node.hello-world` - single-node.hello-world

        * `single-node.hpl-cpu` - single-node.hpl-cpu

        * `single-node.hpl-gpu` - single-node.hpl-gpu

        * `single-node.lmbench-lat-mem-rd` - single-node.lmbench-lat-mem-rd

        * `single-node.memtester` - single-node.memtester

        * `single-node.nccl-all-gather` - single-node.nccl-all-gather

        * `single-node.nccl-all-reduce` - single-node.nccl-all-reduce

        * `single-node.nccl-all-reduce-loopback` -
        single-node.nccl-all-reduce-loopback

        * `single-node.nccl-all-to-all` - single-node.nccl-all-to-all

        * `single-node.node-health-check` - single-node.node-health-check

        * `single-node.nvme-fio` - single-node.nvme-fio

        * `single-node.nvme-self-test` - single-node.nvme-self-test

        * `single-node.storage-elbencho` - single-node.storage-elbencho

        * `single-node.stream` - single-node.stream

        * `single-node.stress-ng-cpu` - single-node.stress-ng-cpu

        * `single-node.training-llama3-8b` - single-node.training-llama3-8b

        * `single-node.training-nemotronh-8b` -
        single-node.training-nemotronh-8b
    ChadSop:
      type: object
      description: >-
        The chad-sops recipe that dispatched a job.


        Provenance reported by CHAD, not a Fleet Manager entity: the recipe
        catalogue lives

        outside FM, so FM stores what the caller reports and validates only the
        shape. The

        three fields are set and cleared together — a job never carries a
        partial reference.
      properties:
        id:
          type: string
          description: >-
            chad-sops recipe id, e.g. `sop.gpu_compute.gb200_diag_reseat_swap`.
            A label, not a key: the same id can appear in more than one site's
            recipe, so do not treat it as unique.
          maxLength: 255
        name:
          type: string
          description: >-
            Human-readable recipe name, e.g. `GB200 GPU Diagnostics and Tray
            Reseat/Swap`. This is what operator-facing surfaces should display.
          maxLength: 255
        version:
          type: integer
          maximum: 2147483647
          minimum: 1
          description: >-
            Recipe version that ran, so a job can be tied to the exact
            procedure.
      required:
        - id
        - name
        - version
    JobLinks:
      type: object
      description: Serializer for job links.
      properties:
        self:
          type: string
          format: uri
          readOnly: true
        collection:
          type: string
          format: uri
          readOnly: true
        workflow:
          type: string
          format: uri
          readOnly: true
        devices:
          type: string
          format: uri
          readOnly: true
        job_device_statuses:
          type: string
          format: uri
          readOnly: true
        temporal:
          type: string
          format: uri
          readOnly: true
          nullable: true
        logs:
          type: string
          format: uri
          readOnly: true
          nullable: true
      required:
        - collection
        - devices
        - job_device_statuses
        - logs
        - self
        - temporal
        - workflow
    JobDeviceSummary:
      type: object
      description: A job device's name and physical location (rack + U + networking pod).
      properties:
        name:
          type: string
          readOnly: true
          description: Device name.
        rack_id:
          type: string
          format: uuid
          readOnly: true
          description: Rack id.
        rack_name:
          type: string
          readOnly: true
          description: Rack name.
        position:
          type: number
          format: double
          readOnly: true
          nullable: true
          description: Rack U position. Null when the device is in its rack without a U.
        pod:
          type: string
          readOnly: true
          description: >-
            Networking pod label for the device; empty if it has no networking
            topology.
      required:
        - name
        - pod
        - position
        - rack_id
        - rack_name
    JobStatusEnum:
      enum:
        - pending
        - running
        - cancellation_requested
        - completed
        - failed
        - partially_failed
        - cancelled
        - terminated
        - superseded
        - timed_out
      type: string
      description: |-
        * `pending` - Pending
        * `running` - Running
        * `cancellation_requested` - Cancellation Requested
        * `completed` - Completed
        * `failed` - Failed
        * `partially_failed` - Partially Failed
        * `cancelled` - Cancelled
        * `terminated` - Terminated
        * `superseded` - Superseded
        * `timed_out` - Timed Out
    ErrorCodeEnum:
      type: string
      description: >-
        Stable machine-readable error identifier. New members may be added
        within this API version; clients must tolerate unknown values.
      enum:
        - error
        - invalid
        - parse_error
        - authentication_failed
        - not_authenticated
        - permission_denied
        - not_found
        - method_not_allowed
        - not_acceptable
        - unsupported_media_type
        - conflict
        - enrollment_infrastructure_reference_pending
        - enrollment_infrastructure_reference_changed
        - unprocessable_entity
        - throttled
        - service_unavailable
    FieldError:
      type: object
      description: One input field's validation messages.
      required:
        - field
        - messages
      properties:
        field:
          type: string
          description: >-
            Dotted path to the offending input field, or to the collection row
            when the whole row is invalid (e.g. `filters.end_date`,
            `memberships.0.role`, `memberships.0`).
        messages:
          type: array
          minItems: 1
          items:
            type: string
    TemporalWorkflowNameEnum:
      enum:
        - EnrollmentWorkflow
        - SingleNodeBurnInWorkflow
        - MultiNodeBurnInPairwiseWorkflow
        - MultiNodeBurnInCollectiveWorkflow
        - PrepareHardwareTestImagesWorkflow
        - ReportGenerationWorkflow
        - SlurmAdoptWorkflow
        - SlurmReleaseWorkflow
        - CustomerReleaseWorkflow
        - IronicNodeCleanWorkflow
        - QuarantineWorkflow
        - ReleaseWorkflow
        - UnenrollmentWorkflow
        - ConnectivityVerificationWorkflow
        - FirmwareInventoryWorkflow
        - FirmwareVerificationWorkflow
        - FirmwarePolicyVerificationWorkflow
        - NVSwitchFirmwarePolicyVerificationWorkflow
        - NodeSwapWorkflow
        - PowerCycleWorkflow
        - PowerOffWorkflow
        - SetMaintenanceModeWorkflow
        - DisableMaintenanceModeWorkflow
        - ColdResetBmcWorkflow
        - SlurmDrainNodeWorkflow
        - SlurmResumeNodeWorkflow
        - FrontendVrfReturnWorkflow
        - FrontendVrfIsolateWorkflow
        - NvLinkIsolateWorkflow
        - NvLinkReturnWorkflow
        - BackendVrfIsolateWorkflow
        - BackendVrfReturnWorkflow
        - ImageNodeWorkflow
        - UnimageNodeWorkflow
        - NodeDiagnosticsWorkflow
      type: string
      description: >-
        * `EnrollmentWorkflow` - Enrollment Workflow

        * `SingleNodeBurnInWorkflow` - Single Node Burn-In Workflow

        * `MultiNodeBurnInPairwiseWorkflow` - Multi-Node Burn-In Pairwise
        Workflow

        * `MultiNodeBurnInCollectiveWorkflow` - Multi-Node Burn-In Collective
        Workflow

        * `PrepareHardwareTestImagesWorkflow` - Prepare Hardware-Test Images
        Workflow

        * `ReportGenerationWorkflow` - Report Generation Workflow

        * `SlurmAdoptWorkflow` - Add to Test Cluster Workflow

        * `SlurmReleaseWorkflow` - Release from Test Cluster Workflow

        * `CustomerReleaseWorkflow` - Customer Release Workflow

        * `IronicNodeCleanWorkflow` - Ironic Node Clean Workflow

        * `QuarantineWorkflow` - Quarantine Workflow

        * `ReleaseWorkflow` - Release Workflow

        * `UnenrollmentWorkflow` - Unenrollment Workflow

        * `ConnectivityVerificationWorkflow` - Connectivity Verification
        Workflow

        * `FirmwareInventoryWorkflow` - Firmware Inventory Workflow

        * `FirmwareVerificationWorkflow` - Firmware Verification Workflow

        * `FirmwarePolicyVerificationWorkflow` - Firmware Policy Verification
        Workflow

        * `NVSwitchFirmwarePolicyVerificationWorkflow` - NVSwitch Firmware
        Policy Verification Workflow

        * `NodeSwapWorkflow` - Node Swap Workflow

        * `PowerCycleWorkflow` - Power Cycle Workflow

        * `PowerOffWorkflow` - Power Off Workflow

        * `SetMaintenanceModeWorkflow` - Set Maintenance Mode Workflow

        * `DisableMaintenanceModeWorkflow` - Disable Maintenance Mode Workflow

        * `ColdResetBmcWorkflow` - Cold Reset BMC Workflow

        * `SlurmDrainNodeWorkflow` - Slurm Drain Node Workflow

        * `SlurmResumeNodeWorkflow` - Slurm Resume Drained Node Workflow

        * `FrontendVrfReturnWorkflow` - Frontend VRF Return Workflow

        * `FrontendVrfIsolateWorkflow` - Frontend VRF Isolate Workflow

        * `NvLinkIsolateWorkflow` - NVLink Isolate Workflow

        * `NvLinkReturnWorkflow` - NVLink Return Workflow

        * `BackendVrfIsolateWorkflow` - Back-end VRF Isolate Workflow

        * `BackendVrfReturnWorkflow` - Back-end VRF Return Workflow

        * `ImageNodeWorkflow` - Image Node Workflow

        * `UnimageNodeWorkflow` - Unimage Node Workflow

        * `NodeDiagnosticsWorkflow` - Node Diagnostics Workflow
    WorkflowCategoryEnum:
      enum:
        - enrolment
        - burn_in_testing
        - customer_release
        - in_service_ops
      type: string
      description: |-
        * `enrolment` - Enrolment
        * `burn_in_testing` - Burn-in testing
        * `customer_release` - Customer release
        * `in_service_ops` - In-service ops
  securitySchemes:
    oauth2:
      type: oauth2
      flows: {}
    cookieAuth:
      type: apiKey
      in: cookie
      name: sessionid

````

This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.