Work Queue Specification

Priority and fairness for the gh-aw work queue

Section titled “Priority and fairness for the gh-aw work queue”

For deployment and operator commands, see Work queues. Executable contracts, models, fixtures and verification instructions remain in specs/work-queue/.

0. Executive walkthrough (no protocol expertise required)

Section titled “0. Executive walkthrough (no protocol expertise required)”

Think of the work queue as a shared task board that remembers who was assigned what, what actually succeeded, and what should happen next. Agents contribute tasks and plans. The queue decides which ready tasks get an assignment. Workers do the work, and checked outputs let the next steps proceed.

This is the intended experience of the proposed design, not a feature already implemented. This section is an intuitive overview; the later sections define the precise rules and limits.

flowchart LR
    Plan["Agent proposes tasks and plans"] --> Queue["Queue chooses the next ready tasks"]
    Queue --> Worker["Worker receives an assignment"]
    Worker --> Check["Queue checks and records each task's outcome"]
    Check -->|Verified results| Next["Follow-ups with satisfied prerequisites"]
    Next --> Queue
    Queue --> History[("One shared history")]
    Check --> History

A developer’s day: one large-repository migration

Section titled “A developer’s day: one large-repository migration”

A developer asks an agent to plan an SDK migration in a large repository. Instead of one long, fragile run, the agent breaks the campaign into useful tasks: inspect the existing usage, propose code changes, propose documentation changes, and review the combined proposals.

flowchart LR
    Issue{{"Design Issue: completed"}} --> Inspect["Inspect current SDK usage"]
    Inspect --> Code["Propose code changes"]
    Inspect --> Docs["Propose documentation changes"]
    Code --> Review["Review both verified proposals"]
    Docs --> Review
    PR{{"Existing upstream SDK PR: merged"}} --> Review
    Review --> Handoff["Submit changes for human review"]

Once the design Issue is confirmed completed, inspection is the first ready task. Its checked report makes the code and documentation tasks ready; neither has to wait for the other. The combined review waits for both verified proposals and the existing upstream SDK PR to be merged. It cannot proceed on an agent’s assertion that the merge probably happened.

Issues and PRs are nodes in the dependency graph, not just links on tasks. They represent milestones that can hold up or unlock several tasks, including across repositories the queue is authorized to read. Unlike a task, an Issue or PR milestone does not need a worker or consume an assignment turn: a trusted check observes its condition. An Issue closed as “not planned” is not completed; a PR closed without merging is not merged.

If an agent needs to investigate an Issue, fix its code, or update a PR, that action is a separate worker task attached to the resource. It still gets a normal assignment and task-specific output permissions. The milestone node tracks what happened to the resource; it is not a free way to run agent work.

Out of the box, ready tasks are assigned in the order the queue accepted them, like a normal queue. The queue looks past tasks already assigned, waiting on prerequisites, or waiting for a retry, and can assign unrelated ready work when capacity permits. Several workers can run at once, so the first task assigned need not be the first task finished. GitHub still determines when a reserved worker run actually starts.

Fairness: a large backlog does not buy all the turns

Section titled “Fairness: a large backlog does not buy all the turns”

The ordinary FIFO default is not an automatic quota for each team. If several teams need agreed shares of assignment opportunities, an operator explicitly configures their groups and shares; the queue does not secretly invent groups from whoever submitted the tasks.

Imagine the migration expands to 500 ready tasks for Platform, while Security has 10 ready checks. With both teams in the same priority level and explicitly configured equal shares, the target is roughly half the assignments from that level for each team while both can accept more work and capacity is available. Platform does not get fifty times the turns just because its backlog is fifty times larger. This is a share over time, not a promise of perfect alternation, an exact ratio in every short window, or an immediate start.

flowchart LR
    Platform["Platform: 500 ready migration tasks"] --> Policy["Same priority and explicitly configured equal shares"]
    Security["Security: 10 ready checks"] --> Policy
    Policy --> PlatformTurns["Platform receives its share of assignment turns"]
    Policy --> SecurityTurns["Security receives its share of assignment turns"]

Fair does not have to mean equal. If Platform is allocated twice Security’s share, the intended balance is about two Platform assignments for each Security assignment at that priority over time. The agreed allocation, not backlog size or an agent’s insistence, determines the share. Each team’s oldest eligible task goes first within its own turn.

If Security has no tasks ready or is at its assignment limit, those opportunities can go to eligible work instead. Security does not bank unused turns to demand a catch-up burst later. Adding dispatchers, splitting a campaign into more tasks, or renaming its revision does not create a new share: the trusted policy controls the grouping, and earlier assignments remain accounted for.

Priority and fairness answer different questions. Priority determines how assignment opportunities are divided between urgency levels; team shares divide opportunities within each level. Under the default weighted priority policy, urgent work gets more turns but ordinary work still has a share. An explicitly chosen strict-priority policy instead lets urgent work take every opportunity while it remains eligible, so ordinary work can wait indefinitely. Neither policy stops a job already running.

Assignment fairness is not equal computing time or cost. A one-minute task and a one-hour task each count as one assignment. Equal team shares can still produce very different worker-minutes and completion rates. Likewise, three tasks sharing one setup-heavy worker run still count as three turns, and a failed assignment is not refunded; a new retry consumes another turn.

The agent remains useful: it decides what tasks to propose, prepares their plans, and reacts to results. It does not get to bypass the queue, invent extra turns for itself, or turn its own “done” message into verified success.

Save setup time without making everything all-or-nothing

Section titled “Save setup time without making everything all-or-nothing”

Normally, a worker receives one task. For a repository that takes ten minutes to prepare, an approved worker can explicitly receive several compatible, already-ready tasks and reuse that setup. The code and documentation proposals could share a run; inspection and its not-yet-ready follow-ups cannot.

Each task keeps its own outcome and permission to produce outputs. If the code proposal succeeds but the documentation task is cancelled, the code result survives. Documentation can get another turn later, and the combined review continues to wait for it. With one task, the queue attaches outputs to that task automatically; with several, the worker must say which task each output belongs to. Each assigned task still counts as its own turn for fairness. A shared run keeps its worker reservation until the run has demonstrably stopped.

What happens when the day does not go to plan

Section titled “What happens when the day does not go to plan”

An outage or accidental cancellation does not erase the task board. Recovery checks the shared history and the actual worker run. Tasks without recorded completion can be retried through the queue when safe, with a delay and a finite retry budget. Already completed tasks are not blindly run again: their outputs are checked, and missing or uncertain delivery stays visibly blocked or failed until it is resolved safely. The queue does not treat silence as proof that an old worker stopped.

Changing the campaign halfway through does not change what existing workers were asked to do. By default, they finish their saved plans; revised tasks are new entries in the same competition. If old and new versions cannot safely overlap, an operator pauses new work and waits for the old runs and output checks to settle before switching over. Earlier assignments stay recorded; a revision or retry is not a shortcut to extra turns.

For a developer or PM, the useful questions have concrete answers:

QuestionWhat the queue explains
Why has this task not started?Its prerequisite, retry delay, capacity/priority limit, or queued/unconfirmed GitHub launch
Why did another team get the next turn?The configured priority/team shares and recorded assignments, not simply which team has the bigger backlog
What survived a failed run?Each task’s recorded completion, verified outputs, and remaining delivery problems
What will happen next?Which tasks are ready, which need another attempt, and which require operator action
Did the campaign update lose old progress?The old plans/results remain visible alongside the newly admitted tasks

The record of these decisions is one shared history, not an agent’s memory or an optional telemetry dashboard. In the technical sections, a task is Work, the recorded permission to attempt one task is a Claim, and its checked output is a Result. The operational recovery scenarios walk through the failure cases in detail.

Research date: 2026-10-05

Repository examined: github/gh-aw, commit 81891f23dfb58b88bd90c9736887880234bffbe5

Status: specification with implementation in progress; the coverage table in section 9.1 identifies verified, unfinished, and deferred requirements.

Protocol review revision: 2026-10-05. Separates fair selection from batch packing, specifies delivery/launch recovery and operational controls, defines batch trust boundaries and replay limits, and distinguishes required guarantees from the existing bounded model evidence.

Scope: the durable tools.work-queue protocol, including the Git and Issues backends, with particular attention to HPC workflow dispatch and scheduling literature.

Required storage constraint: all authoritative scheduling decisions and their policy, charge, reservation, launch-binding, and release transactions remain in the same canonical work-queue.jsonl transaction log as Work and Claim transactions. No separate scheduling log, authoritative outbox, database, cursor file, manifest, or state sidecar is introduced. In-memory projections and activation snapshots are disposable derivatives of that one log.

Required scheduling constraint: every queue MUST enforce scheduling. There is no opt-out, legacy queue mode, advisory-only operating mode, or backward-compatible transaction reader. The proposed protocol replaces the existing behavior; unsupported old records or backends fail explicitly rather than falling back to an unscheduled queue.

Required default behavior: without explicit priority or fairness grouping, the queue MUST feel like a normal FIFO work queue: one priority, one accounting bucket, and oldest-available Work selected next. This is the mandatory scheduler’s default behavior, not a legacy mode or a scheduling opt-out.

Required worker batching: one worker dispatch MAY carry several scheduler-selected Claims. The inbound assignment is a version-3 object containing a bounded claims array, even for one Claim. Each Claim has independent finish intent, terminal state, and effect authorization; completing one must not require completing the rest.

Required output attribution: every safe-output message from a queue worker MUST resolve to exactly one assigned Claim before any handler runs. Trusted processing supplies the sole handle automatically for an immutable one-Claim assignment; a multi-Claim assignment requires an explicit handle on every message. This is enforced authorization, not a prompt convention.

Required DAG support: Work, Issue, and Pull Request are first-class DAG node kinds. Work is scheduled; Issue/PR nodes are observed condition gates, not implicit worker assignments. Immutable typed predecessor edges and trusted result/Issue/PR observations determine the ready frontier before priority/fairness selection. Independent Work has an empty dependency list and retains queue-like defaults.

Diagram convention: use Mermaid throughout relationship, information-flow, and lifecycle explanations. Keep diagrams consistent with the normative tables, guards, and algorithms; diagrams do not introduce separate state or authority.

Introduce a mandatory, trusted, hierarchical grant scheduler for every queue, rather than making priority and fairness additional agent-controlled sort keys:

flowchart LR
    Ready["Dependency and logical-capacity eligibility"] --> Class["Priority-class policy"]
    Class --> Key["Weighted accounting-key selection"]
    Key --> FIFO["Oldest eligible node in that bucket"]
    FIFO --> Pack["Admit the packable fair prefix"]
    Pack --> Grant["Commit Claims: grant, charge, reserve"]
    Grant --> Launch["Dispatch and bind native run"]
    Launch --> Gate["Per-Claim finish and safe-output gate"]

Out of the box, resolve every submission to priority 3 and one default accounting bucket, then select the oldest available Work. The mandatory hierarchical scheduler therefore behaves like an ordinary FIFO queue without requiring configuration.

When priority levels or accounting keys are explicitly requested, use weighted priority classes with positive shares by default, weighted tenant fairness within each class, and bounded outstanding reservations. Offer strict priority as a separately named scheduling policy whose documented consequence is possible starvation of lower classes. FIFO remains the ordering within every selected bucket, including the single default bucket.

Use a deterministic, stride-inspired virtual-pass algorithm with exact integer ticks for the first implementation. Account in durable Claims, not worker launches or completed jobs. Each Claim grants and reserves one logical Work slot; compatible Claims share one worker-dispatch reservation. Both projections come from the same Claim records, with no separate reservation file.

Keep dispatchers agentic through staged submission and dispatch_next intents: agents decide what useful work exists, prepare its immutable plan, and choose when to request a bounded batch. The protocol selects the next Work; the selected worker then executes that Work’s plan. The agent never needs to implement the fairness algorithm or guess that its snapshot still names the winner.

The most important HPC lesson is that workflow readiness, cross-user entitlement, and resource placement are different decisions. DAGMan and Pegasus manage workflow structure; HTCondor and Slurm manage competing users and available resources. Within-workflow critical-path priority is not a substitute for cross-workflow fairness. H4, H5, H6, H7, H8, L3, L4

The proposed minimum guarantee is therefore:

Within one configured scheduling pool, durable grants follow the configured priority policy and weighted accounting-key policy, using fresh authoritative state. Fairness is over grants while keys remain eligible, not over execution order, completion order, runner-seconds, or external effects.

This is stronger than the current protocol’s advisory ordering and requires a protocol/ADR extension. It must not be presented as something a comparator change can deliver.

1.1 Simplicity, debuggability, and agentic-dispatch review

Section titled “1.1 Simplicity, debuggability, and agentic-dispatch review”

The earlier specification had the right trust boundary, but too much protocol machinery and too little explanation of how an agentic dispatcher could use it. The revised design replaces that machinery rather than layering another protocol on top.

Review findingDecision
A scheduler chain plus source-view digests/deltas duplicated history and complicated replayUse one causal QueueCommit envelope for all mutations; its predecessor defines the complete pre-decision state
Separate grant, Claim, and reservation facts created companion-validation and partial-state casesA Claim grants/charges a logical slot; its dispatch group shares a native slot, both derived from the same operations
Wall-clock enqueue keys obscured what “FIFO” meant during concurrent publicationOrder Work by its first committed (commit ordinal, operation index); timestamps measure age, not queue position
Rational wire state, arbitrary hierarchy, and live weight rebasing expanded the first releaseUse exact integer ticks, two scheduling levels, and policy changes only after the queue drains
An agent could prepare inputs for a snapshot winner that changes before publicationStore Work-specific plans at submission; stage dispatch_next(pool, max_claims, max_dispatches) and bind selected Work late
Multiple Claims in one worker could be confused with one fairness unit or one terminal resultCharge every Claim, reserve one native dispatch slot, and finalize/effect-gate each assigned handle independently
Single-Claim users should not manage redundant attribution, while mixed outcomes must remain validAutomatically scope one-Claim outputs; enforce explicit multi-Claim scopes and settle completed/cancelled members independently through DAG Results
“Read,” “staged,” “granted,” and “launched” could look equally successfulGive each an explicit status; expose partial batches, blocking reasons, and uncertain launches
Tracing risked becoming a second authority or a new observability subsystemReuse existing OTLP/context plumbing; correlate by request/commit/Claim IDs and derive receipts from the canonical log
Packing restrictions could silently change who competesSelect from scheduling eligibility first; admit only a packable fair prefix, without advancing debt for a rejected winner
Completion without delivered outputs could strand a DAG indefinitelyDiagnose unresolved delivery, reconcile within bounds, then publish Result or an explicit terminal DeliveryFailure; replacements get new immutable nodes
Drain-only changes could strand ordinary worker evolutionKeep scheduling economics quiescent; evolve compatible worker deployments prospectively without resetting debt or revoking outstanding Claims
Per-Claim handles could be mistaken for tenant isolationBatch only within an approved common trust domain; validate credential, data, and effect scopes before grouping
Full-history replay and daily partial searches could look like scalable guaranteesBound admission and replay costs; label algorithmic, protocol, runtime, and evidence-collection results separately

The first release should have one codec/replayer, one pure planNext selector, one checked publication path, and one trusted worker binder. It should not include adaptive scheduling, a third fairness level, arbitrary selection filters, live weight edits, or a new external coordinator.

All repository links below are pinned to the examined commit. “Current” in this report means this checkout, not whatever subsequently lands on main.

EvidenceWhat it establishes
Work-queue guidance, lines 7-15MCP reads an immutable activation snapshot. sort changes a recommendation, not trusted Claim selection or authority. Dispatch takes an explicit Work identity.
Runtime replay, lines 37-55 and 122-161Transaction shapes are closed. Work permits enqueue metadata, but no priority, accounting key, weight, or scheduling state. Duplicate conflicting Work facts are invalid.
Runtime replay, lines 201-279Ownership is determined by stable Claim identity; available Work is sorted by immutable enqueue time and UTF-8 Work identity. Ownership arbitration and queue ordering are separate.
Runtime selection helpers, lines 382-415Oldest-available selection uses the caller’s local view. A fixed Claim is revalidated for safety, not FIFO.
MCP server, lines 48-132Read sort keys are id, enqueued, and id_length; next_work is not an acquired Claim.
Dispatch handler, lines 253-263 and 340-370Trusted dispatch checks that a selected Work is available, publishes a Claim, checks effectiveness, and supplies the trusted worker assignment. It does not compare the selection against a scheduling policy.
Git store, lines 150-215Writes refresh the log and retry fast-forward-only ref updates. The retry loop reapplies the same supplied intents; it is not a general “choose the next fair Work” callback.
Issues store, lines 81-139 and 167-226Issues/comments are separately published signed records. Refreshes span multiple API reads. There is no queue-wide atomic compare-and-swap matching the Git branch’s publication point.
Operator selection, lines 20-55 and operator claim, lines 163-203The operator CLI has oldest-available selection, but keeps the initially selected Work across retries. Its wire format and branch differ from the runtime’s.
TypeSpec contracts, lines 8-128Operator Transaction and workflow WorkQueueTransaction are separate contracts. Workflow records currently use protocol version 2.
ADR-64955, protocol commitmentsThe intended current guarantee is best-effort oldest-available ordering, not global FIFO, a global sequence allocator, or a completion barrier.
Protocol-upgrade ADRMigrations are deterministic; historical Work gets no fabricated enqueue timestamp. Transaction version and snapshot-envelope version are independent.
TLA+ model, lines 133-176The model constrains local staging to oldest-known Work, but retries reapply serialized intents. Existing safety modeling does not establish a weighted scheduling liveness guarantee.

The original work-coordinator issue #64852 also requires deterministic replay of the same accepted fact set, trusted Claim authority, terminal-state protection, a single canonical authoritative file, and compaction preserving future semantics. The implementation documentation explicitly describes the MCP as a snapshot reader, rather than the live mutable MCP surface originally envisioned in the issue.

Conclusion: there is presently no built-in priority/fairness contract in this queue. Agent prompts or read-time sorting can express preferences, but cannot establish enforcement across independent dispatchers.

The existing upgrade, ordering, and compatibility behavior above is evidence about the examined implementation, not a compatibility requirement for the proposed protocol. The new design intentionally removes unscheduled operation and old-record support.

2.2 Two storage and command distinctions matter

Section titled “2.2 Two storage and command distinctions matter”

The examined workflow runtime uses work-queue and versioned workflow facts. The examined operator CLI defaults to gh-aw-work-queue and preserves payload/run-provenance fields in a different format. Those old formats are not interchangeable. The replacement MUST use one current QueueCommit contract and the exact algorithm for both surfaces. Native Go and JavaScript engines are deliberately retained: generated schemas and independent, exact cross-language conformance fixtures enforce identical replay, selection, packing, and request semantics. Each language’s consumers reuse its engine rather than implementing separate CLI, publisher, snapshot, or explanation selectors. Separately configured queue branches remain separate authorities; sharing a format does not merge their entitlement, permissions, or worker-effect authorization. Neither surface retains an unscheduled mode. R11, R12

Git and Issues expose similar tools, but do not offer identical consistency primitives. HMAC signatures authenticate Issues records; they do not turn a sequence of issue/comment reads and writes into an atomic queue-wide scheduling transaction. A design requiring a single global scheduling decision must acknowledge that difference. R7, R8

3. Specify the meaning of fairness before choosing an algorithm

Section titled “3. Specify the meaning of fairness before choosing an algorithm”

“Fair” can describe several incompatible objectives.

ObjectiveUnit being distributedAppropriate mechanismWhat it does not imply
FIFOPosition among eligible WorkFirst-committed enqueue positionTenant isolation; equal resource consumption
Dispatch fairnessGrants or startsWeighted round robin, stride, virtual-time schedulingEqual running mix or completion times
Concurrent allocation fairnessOutstanding slots/resourcesFair-share allocation and capacity admissionEqual historical consumption
Historical resource fairnessCPU-seconds, runner-seconds, billed resource unitsUsage accounting, often with decayA hard near-term share for every tenant
Multi-resource fairnessCPU, memory, GPUs, licenses jointlyDRF or resource-aware allocationPlacement feasibility or workflow deadlines
Workflow outcome fairnessTurnaround or slowdownWorkflow-aware multi-DAG schedulingA simple fixed dispatch ratio
Deadline priorityUrgency/slackEDF, deadline admission, reservationsFairness or feasibility under overload

Temporal’s fairness is primarily task-dispatch fairness. Slurm fair-share is a usage-based influence on job priority. HTCondor user priority influences resource allocation, separately from a user’s ordering of their own jobs. Multi-DAG literature also considers fairness of workflow slowdown. These are related, but not synonymous. T1, H1, H5, L3

For example, equal numbers of grants to one tenant whose jobs take 60 minutes and another whose jobs take one minute produce about 98.36% versus 1.64% of total runner-time, assuming equal runner allocation per job and complete execution. Even equal concurrent slots need not give equal completions per hour.

Likewise, priority at dispatch is not preemption. If every runner is occupied, selecting an urgent job first does not free a runner. Meaningful urgent-start guarantees require bounded runtimes, reserved capacity, or an independently specified preemption/checkpoint mechanism.

4. What HPC systems and literature contribute

Section titled “4. What HPC systems and literature contribute”

4.1 The architectural separation: workflow engine versus resource scheduler

Section titled “4.1 The architectural separation: workflow engine versus resource scheduler”

In a scientific DAG, a task becomes ready only when its required predecessors have succeeded. A workflow engine identifies ready tasks, handles retries and data movement, and submits work to a resource scheduler. A resource scheduler decides which competing accounts receive capacity and where jobs fit.

Pegasus represents scientific workflows as DAGs, maps abstract work to execution environments, and can reorder, group, and prioritize tasks. Its optimization documentation discusses clustering short tasks to amortize submission and execution overhead. These transformations improve efficiency, but do not independently guarantee fairness among different users. H7, H8

DAGMan’s node priorities affect submission among simultaneously ready nodes and set HTCondor JobPrio. They cannot override dependencies or guarantee execution order. After submission, JobPrio orders jobs of the same user; user allocation is handled separately. This is a particularly useful model for gh-aw: a dispatcher’s preference among its own tasks should not increase its account’s entitlement relative to other dispatchers. H4, H5

DAGMan also propagates accounting-group metadata into DAG jobs and sub-DAGs. That supports stable charging across fan-out, instead of letting each new child become a fresh resource claimant. H6

Translation to gh-aw: separate:

  1. Eligibility: Work exists, is available, is executable by the configured worker pool, and satisfies limits.
  2. Entitlement: an authorized account/project receives a share of grants.
  3. Local ordering: select an eligible Work within that entitlement.
  4. Execution placement: GitHub Actions allocates actual runners and applies its own concurrency rules.

The examined runtime is not yet a DAG dependency engine. The replacement makes that layer first-class: immutable predecessor edges, verified results, and typed cross-repository Issue/PR gates define readiness. Do not infer readiness from timestamps, issue links, or an agent’s statement that a task is ready.

4.2 Slurm: multifactor priority and hierarchical historical fair-share

Section titled “4.2 Slurm: multifactor priority and hierarchical historical fair-share”

Slurm’s multifactor plugin combines enabled contributions such as age, fair-share, job size, partition, QoS, association, site policy, and trackable resources, with a nice adjustment. Larger computed job-priority values are better. Scheduler consideration additionally accounts for preemption, advanced reservations, and partition priority tiers. Consequently, “highest priority” is not just one universal scalar. H1

Its fair-share contribution compares assigned shares with historical resource consumption. Usage can decay with a configurable half-life, and billing can include multiple resource types. It is not a fixed quota that stops a user once consumed: over-served users may use otherwise idle resources. H1

Fair Tree makes that entitlement hierarchical. Sibling accounts are ranked using normalized shares and usage; an account’s ranking controls its descendants’ fair-share factors. The relation Level Fairshare = normalized shares / normalized usage captures under- versus over-service at each level. This is valuable when a project contains many users or workflows: child proliferation must not create more project entitlement. H2

However, Fair Tree ranks a fair-share factor, not an unconditional final job order or exact percentage of future starts. Other configured factors and resource feasibility still matter. Age also saturates, and dependent/ineligible jobs do not accrue age in the same way as eligible pending jobs. A bounded age term alone is not a proof that every low-priority job eventually runs. H1, H2

Adopt: stable accounting identities, hierarchy, auditable policy factors, usage accounting as a future independent layer, and borrowing unused capacity.

Do not copy blindly: the full multifactor tuning surface, CPU accounting when gh-aw cannot measure it reliably, or a claim that a weighted priority score creates an enforceable minimum service share.

4.3 Backfilling: resource feasibility without unnecessary head-of-line blocking

Section titled “4.3 Backfilling: resource feasibility without unnecessary head-of-line blocking”

Lifka’s ANL/IBM SP scheduling work is the classic EASY backfilling reference. The later Mu’alem/Feitelson work examines utilization, predictability, workloads, and runtime estimates. L1, L2

The conventional distinction is:

PolicyProtected reservationTrade-off
EASY-style backfillingThe head job’s reservationMore aggressive use of gaps; weaker protection for other pending jobs
Conservative backfillingReservations for all considered pending jobsStronger predictability; potentially less flexibility and more planning

Slurm’s documented backfill scheduler can start lower-priority jobs when doing so does not delay the expected start of higher-priority jobs. It considers resource requests and runtime limits, and reserves future resources. Accurate enough time limits and an adequate planning horizon are essential. H3

This is not equivalent to arbitrarily skipping a blocked urgent job. A scheduler claiming safe backfill must be able to demonstrate that the borrowed resources will be available by the protected start.

Translation to gh-aw: initially skip unavailable, incompatible, retry-delayed, and capped keys so they do not block independent eligible work. Do not describe this as EASY backfilling: the current queue does not own runner placement or a reservation calendar.

Reserve runtime-aware backfilling for a future scheduler that knows capacities, execution bounds, and reservation times. Separate resource pools are preferable to pretending that a GPU-blocked job and a CPU-ready job compete for the same interchangeable resource.

Also, HPC systems are not universally nonpreemptive: Slurm supports configured cancellation, requeue, suspension, and gang-scheduling modes. The relevant limitation is that this gh-aw proposal does not add safe preemption, not that Slurm cannot preempt. H9

4.4 DAGMan throttles and Slurm arrays: bound workflow fan-out

Section titled “4.4 DAGMan throttles and Slurm arrays: bound workflow fan-out”

DAGMan provides per-DAG submitted-node and idle-job limits, plus category-specific MAXJOBS. These limits prevent a single workflow from overwhelming staging capacity or the scheduler. They are not, by themselves, cross-user fair-share policies, and a submitted cluster may contain multiple jobs. H10

Slurm arrays offer an analogous practical control: an array expression such as 0-15%4 limits simultaneously running array tasks to four. Array tasks remain subject to ordinary job limits. H11

Translation to gh-aw: a fair dispatch order can still produce poor responsiveness if one workflow preloads hundreds of accepted or pending Actions runs. Limit outstanding reservations, not just calls per dispatcher invocation. Count reserved, launching, queued, and running work until trusted reconciliation releases the reservation.

Distinguish:

  • Share: a relative entitlement while multiple keys are eligible.
  • Outstanding cap: an absolute admission limit, potentially leaving capacity unused.
  • Rate limit: a maximum number of grants per time interval.
  • Queue/admission bound: a maximum amount of submitted backlog.

They address different failure modes. A per-run batch limit cannot establish a global cap when many dispatchers run concurrently.

4.5 HEFT: valuable local priority, not inter-workflow fairness