Work Queue Specification
Priority and fairness for the gh-aw work queue
Section titled “Priority and fairness for the gh-aw work queue”For deployment and operator commands, see Work queues.
Executable contracts, models, fixtures and verification instructions remain in
specs/work-queue/.
0. Executive walkthrough (no protocol expertise required)
Section titled “0. Executive walkthrough (no protocol expertise required)”Think of the work queue as a shared task board that remembers who was assigned what, what actually succeeded, and what should happen next. Agents contribute tasks and plans. The queue decides which ready tasks get an assignment. Workers do the work, and checked outputs let the next steps proceed.
This is the intended experience of the proposed design, not a feature already implemented. This section is an intuitive overview; the later sections define the precise rules and limits.
flowchart LR
Plan["Agent proposes tasks and plans"] --> Queue["Queue chooses the next ready tasks"]
Queue --> Worker["Worker receives an assignment"]
Worker --> Check["Queue checks and records each task's outcome"]
Check -->|Verified results| Next["Follow-ups with satisfied prerequisites"]
Next --> Queue
Queue --> History[("One shared history")]
Check --> History
A developer’s day: one large-repository migration
Section titled “A developer’s day: one large-repository migration”A developer asks an agent to plan an SDK migration in a large repository. Instead of one long, fragile run, the agent breaks the campaign into useful tasks: inspect the existing usage, propose code changes, propose documentation changes, and review the combined proposals.
flowchart LR
Issue{{"Design Issue: completed"}} --> Inspect["Inspect current SDK usage"]
Inspect --> Code["Propose code changes"]
Inspect --> Docs["Propose documentation changes"]
Code --> Review["Review both verified proposals"]
Docs --> Review
PR{{"Existing upstream SDK PR: merged"}} --> Review
Review --> Handoff["Submit changes for human review"]
Once the design Issue is confirmed completed, inspection is the first ready task. Its checked report makes the code and documentation tasks ready; neither has to wait for the other. The combined review waits for both verified proposals and the existing upstream SDK PR to be merged. It cannot proceed on an agent’s assertion that the merge probably happened.
Issues and PRs are nodes in the dependency graph, not just links on tasks. They represent milestones that can hold up or unlock several tasks, including across repositories the queue is authorized to read. Unlike a task, an Issue or PR milestone does not need a worker or consume an assignment turn: a trusted check observes its condition. An Issue closed as “not planned” is not completed; a PR closed without merging is not merged.
If an agent needs to investigate an Issue, fix its code, or update a PR, that action is a separate worker task attached to the resource. It still gets a normal assignment and task-specific output permissions. The milestone node tracks what happened to the resource; it is not a free way to run agent work.
Out of the box, ready tasks are assigned in the order the queue accepted them, like a normal queue. The queue looks past tasks already assigned, waiting on prerequisites, or waiting for a retry, and can assign unrelated ready work when capacity permits. Several workers can run at once, so the first task assigned need not be the first task finished. GitHub still determines when a reserved worker run actually starts.
Fairness: a large backlog does not buy all the turns
Section titled “Fairness: a large backlog does not buy all the turns”The ordinary FIFO default is not an automatic quota for each team. If several teams need agreed shares of assignment opportunities, an operator explicitly configures their groups and shares; the queue does not secretly invent groups from whoever submitted the tasks.
Imagine the migration expands to 500 ready tasks for Platform, while Security has 10 ready checks. With both teams in the same priority level and explicitly configured equal shares, the target is roughly half the assignments from that level for each team while both can accept more work and capacity is available. Platform does not get fifty times the turns just because its backlog is fifty times larger. This is a share over time, not a promise of perfect alternation, an exact ratio in every short window, or an immediate start.
flowchart LR
Platform["Platform: 500 ready migration tasks"] --> Policy["Same priority and explicitly configured equal shares"]
Security["Security: 10 ready checks"] --> Policy
Policy --> PlatformTurns["Platform receives its share of assignment turns"]
Policy --> SecurityTurns["Security receives its share of assignment turns"]
Fair does not have to mean equal. If Platform is allocated twice Security’s share, the intended balance is about two Platform assignments for each Security assignment at that priority over time. The agreed allocation, not backlog size or an agent’s insistence, determines the share. Each team’s oldest eligible task goes first within its own turn.
If Security has no tasks ready or is at its assignment limit, those opportunities can go to eligible work instead. Security does not bank unused turns to demand a catch-up burst later. Adding dispatchers, splitting a campaign into more tasks, or renaming its revision does not create a new share: the trusted policy controls the grouping, and earlier assignments remain accounted for.
Priority and fairness answer different questions. Priority determines how assignment opportunities are divided between urgency levels; team shares divide opportunities within each level. Under the default weighted priority policy, urgent work gets more turns but ordinary work still has a share. An explicitly chosen strict-priority policy instead lets urgent work take every opportunity while it remains eligible, so ordinary work can wait indefinitely. Neither policy stops a job already running.
Assignment fairness is not equal computing time or cost. A one-minute task and a one-hour task each count as one assignment. Equal team shares can still produce very different worker-minutes and completion rates. Likewise, three tasks sharing one setup-heavy worker run still count as three turns, and a failed assignment is not refunded; a new retry consumes another turn.
The agent remains useful: it decides what tasks to propose, prepares their plans, and reacts to results. It does not get to bypass the queue, invent extra turns for itself, or turn its own “done” message into verified success.
Save setup time without making everything all-or-nothing
Section titled “Save setup time without making everything all-or-nothing”Normally, a worker receives one task. For a repository that takes ten minutes to prepare, an approved worker can explicitly receive several compatible, already-ready tasks and reuse that setup. The code and documentation proposals could share a run; inspection and its not-yet-ready follow-ups cannot.
Each task keeps its own outcome and permission to produce outputs. If the code proposal succeeds but the documentation task is cancelled, the code result survives. Documentation can get another turn later, and the combined review continues to wait for it. With one task, the queue attaches outputs to that task automatically; with several, the worker must say which task each output belongs to. Each assigned task still counts as its own turn for fairness. A shared run keeps its worker reservation until the run has demonstrably stopped.
What happens when the day does not go to plan
Section titled “What happens when the day does not go to plan”An outage or accidental cancellation does not erase the task board. Recovery checks the shared history and the actual worker run. Tasks without recorded completion can be retried through the queue when safe, with a delay and a finite retry budget. Already completed tasks are not blindly run again: their outputs are checked, and missing or uncertain delivery stays visibly blocked or failed until it is resolved safely. The queue does not treat silence as proof that an old worker stopped.
Changing the campaign halfway through does not change what existing workers were asked to do. By default, they finish their saved plans; revised tasks are new entries in the same competition. If old and new versions cannot safely overlap, an operator pauses new work and waits for the old runs and output checks to settle before switching over. Earlier assignments stay recorded; a revision or retry is not a shortcut to extra turns.
For a developer or PM, the useful questions have concrete answers:
| Question | What the queue explains |
|---|---|
| Why has this task not started? | Its prerequisite, retry delay, capacity/priority limit, or queued/unconfirmed GitHub launch |
| Why did another team get the next turn? | The configured priority/team shares and recorded assignments, not simply which team has the bigger backlog |
| What survived a failed run? | Each task’s recorded completion, verified outputs, and remaining delivery problems |
| What will happen next? | Which tasks are ready, which need another attempt, and which require operator action |
| Did the campaign update lose old progress? | The old plans/results remain visible alongside the newly admitted tasks |
The record of these decisions is one shared history, not an agent’s memory or an optional telemetry dashboard. In the technical sections, a task is Work, the recorded permission to attempt one task is a Claim, and its checked output is a Result. The operational recovery scenarios walk through the failure cases in detail.
1. Recommendation
Section titled “1. Recommendation”Research date: 2026-10-05
Repository examined: github/gh-aw, commit 81891f23dfb58b88bd90c9736887880234bffbe5
Status: specification with implementation in progress; the coverage table in section 9.1 identifies verified, unfinished, and deferred requirements.
Protocol review revision: 2026-10-05. Separates fair selection from batch packing, specifies delivery/launch recovery and operational controls, defines batch trust boundaries and replay limits, and distinguishes required guarantees from the existing bounded model evidence.
Scope: the durable tools.work-queue protocol, including the Git and Issues backends, with particular attention to HPC workflow dispatch and scheduling literature.
Required storage constraint: all authoritative scheduling decisions and their policy, charge, reservation, launch-binding, and release transactions remain in the same canonical work-queue.jsonl transaction log as Work and Claim transactions. No separate scheduling log, authoritative outbox, database, cursor file, manifest, or state sidecar is introduced. In-memory projections and activation snapshots are disposable derivatives of that one log.
Required scheduling constraint: every queue MUST enforce scheduling. There is no opt-out, legacy queue mode, advisory-only operating mode, or backward-compatible transaction reader. The proposed protocol replaces the existing behavior; unsupported old records or backends fail explicitly rather than falling back to an unscheduled queue.
Required default behavior: without explicit priority or fairness grouping, the queue MUST feel like a normal FIFO work queue: one priority, one accounting bucket, and oldest-available Work selected next. This is the mandatory scheduler’s default behavior, not a legacy mode or a scheduling opt-out.
Required worker batching: one worker dispatch MAY carry several scheduler-selected Claims. The inbound assignment is a version-3 object containing a bounded claims array, even for one Claim. Each Claim has independent finish intent, terminal state, and effect authorization; completing one must not require completing the rest.
Required output attribution: every safe-output message from a queue worker MUST resolve to exactly one assigned Claim before any handler runs. Trusted processing supplies the sole handle automatically for an immutable one-Claim assignment; a multi-Claim assignment requires an explicit handle on every message. This is enforced authorization, not a prompt convention.
Required DAG support: Work, Issue, and Pull Request are first-class DAG node kinds. Work is scheduled; Issue/PR nodes are observed condition gates, not implicit worker assignments. Immutable typed predecessor edges and trusted result/Issue/PR observations determine the ready frontier before priority/fairness selection. Independent Work has an empty dependency list and retains queue-like defaults.
Diagram convention: use Mermaid throughout relationship, information-flow, and lifecycle explanations. Keep diagrams consistent with the normative tables, guards, and algorithms; diagrams do not introduce separate state or authority.
Introduce a mandatory, trusted, hierarchical grant scheduler for every queue, rather than making priority and fairness additional agent-controlled sort keys:
flowchart LR
Ready["Dependency and logical-capacity eligibility"] --> Class["Priority-class policy"]
Class --> Key["Weighted accounting-key selection"]
Key --> FIFO["Oldest eligible node in that bucket"]
FIFO --> Pack["Admit the packable fair prefix"]
Pack --> Grant["Commit Claims: grant, charge, reserve"]
Grant --> Launch["Dispatch and bind native run"]
Launch --> Gate["Per-Claim finish and safe-output gate"]
Out of the box, resolve every submission to priority 3 and one default accounting bucket, then select the oldest available Work. The mandatory hierarchical scheduler therefore behaves like an ordinary FIFO queue without requiring configuration.
When priority levels or accounting keys are explicitly requested, use weighted priority classes with positive shares by default, weighted tenant fairness within each class, and bounded outstanding reservations. Offer strict priority as a separately named scheduling policy whose documented consequence is possible starvation of lower classes. FIFO remains the ordering within every selected bucket, including the single default bucket.
Use a deterministic, stride-inspired virtual-pass algorithm with exact integer ticks for the first implementation. Account in durable Claims, not worker launches or completed jobs. Each Claim grants and reserves one logical Work slot; compatible Claims share one worker-dispatch reservation. Both projections come from the same Claim records, with no separate reservation file.
Keep dispatchers agentic through staged submission and dispatch_next intents: agents decide what useful work exists, prepare its immutable plan, and choose when to request a bounded batch. The protocol selects the next Work; the selected worker then executes that Work’s plan. The agent never needs to implement the fairness algorithm or guess that its snapshot still names the winner.
The most important HPC lesson is that workflow readiness, cross-user entitlement, and resource placement are different decisions. DAGMan and Pegasus manage workflow structure; HTCondor and Slurm manage competing users and available resources. Within-workflow critical-path priority is not a substitute for cross-workflow fairness. H4, H5, H6, H7, H8, L3, L4
The proposed minimum guarantee is therefore:
Within one configured scheduling pool, durable grants follow the configured priority policy and weighted accounting-key policy, using fresh authoritative state. Fairness is over grants while keys remain eligible, not over execution order, completion order, runner-seconds, or external effects.
This is stronger than the current protocol’s advisory ordering and requires a protocol/ADR extension. It must not be presented as something a comparator change can deliver.
1.1 Simplicity, debuggability, and agentic-dispatch review
Section titled “1.1 Simplicity, debuggability, and agentic-dispatch review”The earlier specification had the right trust boundary, but too much protocol machinery and too little explanation of how an agentic dispatcher could use it. The revised design replaces that machinery rather than layering another protocol on top.
| Review finding | Decision |
|---|---|
| A scheduler chain plus source-view digests/deltas duplicated history and complicated replay | Use one causal QueueCommit envelope for all mutations; its predecessor defines the complete pre-decision state |
| Separate grant, Claim, and reservation facts created companion-validation and partial-state cases | A Claim grants/charges a logical slot; its dispatch group shares a native slot, both derived from the same operations |
| Wall-clock enqueue keys obscured what “FIFO” meant during concurrent publication | Order Work by its first committed (commit ordinal, operation index); timestamps measure age, not queue position |
| Rational wire state, arbitrary hierarchy, and live weight rebasing expanded the first release | Use exact integer ticks, two scheduling levels, and policy changes only after the queue drains |
| An agent could prepare inputs for a snapshot winner that changes before publication | Store Work-specific plans at submission; stage dispatch_next(pool, max_claims, max_dispatches) and bind selected Work late |
| Multiple Claims in one worker could be confused with one fairness unit or one terminal result | Charge every Claim, reserve one native dispatch slot, and finalize/effect-gate each assigned handle independently |
| Single-Claim users should not manage redundant attribution, while mixed outcomes must remain valid | Automatically scope one-Claim outputs; enforce explicit multi-Claim scopes and settle completed/cancelled members independently through DAG Results |
| “Read,” “staged,” “granted,” and “launched” could look equally successful | Give each an explicit status; expose partial batches, blocking reasons, and uncertain launches |
| Tracing risked becoming a second authority or a new observability subsystem | Reuse existing OTLP/context plumbing; correlate by request/commit/Claim IDs and derive receipts from the canonical log |
| Packing restrictions could silently change who competes | Select from scheduling eligibility first; admit only a packable fair prefix, without advancing debt for a rejected winner |
| Completion without delivered outputs could strand a DAG indefinitely | Diagnose unresolved delivery, reconcile within bounds, then publish Result or an explicit terminal DeliveryFailure; replacements get new immutable nodes |
| Drain-only changes could strand ordinary worker evolution | Keep scheduling economics quiescent; evolve compatible worker deployments prospectively without resetting debt or revoking outstanding Claims |
| Per-Claim handles could be mistaken for tenant isolation | Batch only within an approved common trust domain; validate credential, data, and effect scopes before grouping |
| Full-history replay and daily partial searches could look like scalable guarantees | Bound admission and replay costs; label algorithmic, protocol, runtime, and evidence-collection results separately |
The first release should have one codec/replayer, one pure planNext selector, one checked publication path, and one trusted worker binder. It should not include adaptive scheduling, a third fairness level, arbitrary selection filters, live weight edits, or a new external coordinator.
2. Evidence and current implementation
Section titled “2. Evidence and current implementation”2.1 Repository findings
Section titled “2.1 Repository findings”All repository links below are pinned to the examined commit. “Current” in this report means this checkout, not whatever subsequently lands on main.
| Evidence | What it establishes |
|---|---|
| Work-queue guidance, lines 7-15 | MCP reads an immutable activation snapshot. sort changes a recommendation, not trusted Claim selection or authority. Dispatch takes an explicit Work identity. |
| Runtime replay, lines 37-55 and 122-161 | Transaction shapes are closed. Work permits enqueue metadata, but no priority, accounting key, weight, or scheduling state. Duplicate conflicting Work facts are invalid. |
| Runtime replay, lines 201-279 | Ownership is determined by stable Claim identity; available Work is sorted by immutable enqueue time and UTF-8 Work identity. Ownership arbitration and queue ordering are separate. |
| Runtime selection helpers, lines 382-415 | Oldest-available selection uses the caller’s local view. A fixed Claim is revalidated for safety, not FIFO. |
| MCP server, lines 48-132 | Read sort keys are id, enqueued, and id_length; next_work is not an acquired Claim. |
| Dispatch handler, lines 253-263 and 340-370 | Trusted dispatch checks that a selected Work is available, publishes a Claim, checks effectiveness, and supplies the trusted worker assignment. It does not compare the selection against a scheduling policy. |
| Git store, lines 150-215 | Writes refresh the log and retry fast-forward-only ref updates. The retry loop reapplies the same supplied intents; it is not a general “choose the next fair Work” callback. |
| Issues store, lines 81-139 and 167-226 | Issues/comments are separately published signed records. Refreshes span multiple API reads. There is no queue-wide atomic compare-and-swap matching the Git branch’s publication point. |
| Operator selection, lines 20-55 and operator claim, lines 163-203 | The operator CLI has oldest-available selection, but keeps the initially selected Work across retries. Its wire format and branch differ from the runtime’s. |
| TypeSpec contracts, lines 8-128 | Operator Transaction and workflow WorkQueueTransaction are separate contracts. Workflow records currently use protocol version 2. |
| ADR-64955, protocol commitments | The intended current guarantee is best-effort oldest-available ordering, not global FIFO, a global sequence allocator, or a completion barrier. |
| Protocol-upgrade ADR | Migrations are deterministic; historical Work gets no fabricated enqueue timestamp. Transaction version and snapshot-envelope version are independent. |
| TLA+ model, lines 133-176 | The model constrains local staging to oldest-known Work, but retries reapply serialized intents. Existing safety modeling does not establish a weighted scheduling liveness guarantee. |
The original work-coordinator issue #64852 also requires deterministic replay of the same accepted fact set, trusted Claim authority, terminal-state protection, a single canonical authoritative file, and compaction preserving future semantics. The implementation documentation explicitly describes the MCP as a snapshot reader, rather than the live mutable MCP surface originally envisioned in the issue.
Conclusion: there is presently no built-in priority/fairness contract in this queue. Agent prompts or read-time sorting can express preferences, but cannot establish enforcement across independent dispatchers.
The existing upgrade, ordering, and compatibility behavior above is evidence about the examined implementation, not a compatibility requirement for the proposed protocol. The new design intentionally removes unscheduled operation and old-record support.
2.2 Two storage and command distinctions matter
Section titled “2.2 Two storage and command distinctions matter”The examined workflow runtime uses work-queue and versioned workflow facts. The examined operator CLI defaults to gh-aw-work-queue and preserves payload/run-provenance fields in a different format. Those old formats are not interchangeable. The replacement MUST use one current QueueCommit contract and the exact algorithm for both surfaces. Native Go and JavaScript engines are deliberately retained: generated schemas and independent, exact cross-language conformance fixtures enforce identical replay, selection, packing, and request semantics. Each language’s consumers reuse its engine rather than implementing separate CLI, publisher, snapshot, or explanation selectors. Separately configured queue branches remain separate authorities; sharing a format does not merge their entitlement, permissions, or worker-effect authorization. Neither surface retains an unscheduled mode. R11, R12
Git and Issues expose similar tools, but do not offer identical consistency primitives. HMAC signatures authenticate Issues records; they do not turn a sequence of issue/comment reads and writes into an atomic queue-wide scheduling transaction. A design requiring a single global scheduling decision must acknowledge that difference. R7, R8
3. Specify the meaning of fairness before choosing an algorithm
Section titled “3. Specify the meaning of fairness before choosing an algorithm”“Fair” can describe several incompatible objectives.
| Objective | Unit being distributed | Appropriate mechanism | What it does not imply |
|---|---|---|---|
| FIFO | Position among eligible Work | First-committed enqueue position | Tenant isolation; equal resource consumption |
| Dispatch fairness | Grants or starts | Weighted round robin, stride, virtual-time scheduling | Equal running mix or completion times |
| Concurrent allocation fairness | Outstanding slots/resources | Fair-share allocation and capacity admission | Equal historical consumption |
| Historical resource fairness | CPU-seconds, runner-seconds, billed resource units | Usage accounting, often with decay | A hard near-term share for every tenant |
| Multi-resource fairness | CPU, memory, GPUs, licenses jointly | DRF or resource-aware allocation | Placement feasibility or workflow deadlines |
| Workflow outcome fairness | Turnaround or slowdown | Workflow-aware multi-DAG scheduling | A simple fixed dispatch ratio |
| Deadline priority | Urgency/slack | EDF, deadline admission, reservations | Fairness or feasibility under overload |
Temporal’s fairness is primarily task-dispatch fairness. Slurm fair-share is a usage-based influence on job priority. HTCondor user priority influences resource allocation, separately from a user’s ordering of their own jobs. Multi-DAG literature also considers fairness of workflow slowdown. These are related, but not synonymous. T1, H1, H5, L3
For example, equal numbers of grants to one tenant whose jobs take 60 minutes and another whose jobs take one minute produce about 98.36% versus 1.64% of total runner-time, assuming equal runner allocation per job and complete execution. Even equal concurrent slots need not give equal completions per hour.
Likewise, priority at dispatch is not preemption. If every runner is occupied, selecting an urgent job first does not free a runner. Meaningful urgent-start guarantees require bounded runtimes, reserved capacity, or an independently specified preemption/checkpoint mechanism.
4. What HPC systems and literature contribute
Section titled “4. What HPC systems and literature contribute”4.1 The architectural separation: workflow engine versus resource scheduler
Section titled “4.1 The architectural separation: workflow engine versus resource scheduler”In a scientific DAG, a task becomes ready only when its required predecessors have succeeded. A workflow engine identifies ready tasks, handles retries and data movement, and submits work to a resource scheduler. A resource scheduler decides which competing accounts receive capacity and where jobs fit.
Pegasus represents scientific workflows as DAGs, maps abstract work to execution environments, and can reorder, group, and prioritize tasks. Its optimization documentation discusses clustering short tasks to amortize submission and execution overhead. These transformations improve efficiency, but do not independently guarantee fairness among different users. H7, H8
DAGMan’s node priorities affect submission among simultaneously ready nodes and set HTCondor JobPrio. They cannot override dependencies or guarantee execution order. After submission, JobPrio orders jobs of the same user; user allocation is handled separately. This is a particularly useful model for gh-aw: a dispatcher’s preference among its own tasks should not increase its account’s entitlement relative to other dispatchers. H4, H5
DAGMan also propagates accounting-group metadata into DAG jobs and sub-DAGs. That supports stable charging across fan-out, instead of letting each new child become a fresh resource claimant. H6
Translation to gh-aw: separate:
- Eligibility: Work exists, is available, is executable by the configured worker pool, and satisfies limits.
- Entitlement: an authorized account/project receives a share of grants.
- Local ordering: select an eligible Work within that entitlement.
- Execution placement: GitHub Actions allocates actual runners and applies its own concurrency rules.
The examined runtime is not yet a DAG dependency engine. The replacement makes that layer first-class: immutable predecessor edges, verified results, and typed cross-repository Issue/PR gates define readiness. Do not infer readiness from timestamps, issue links, or an agent’s statement that a task is ready.
4.2 Slurm: multifactor priority and hierarchical historical fair-share
Section titled “4.2 Slurm: multifactor priority and hierarchical historical fair-share”Slurm’s multifactor plugin combines enabled contributions such as age, fair-share, job size, partition, QoS, association, site policy, and trackable resources, with a nice adjustment. Larger computed job-priority values are better. Scheduler consideration additionally accounts for preemption, advanced reservations, and partition priority tiers. Consequently, “highest priority” is not just one universal scalar. H1
Its fair-share contribution compares assigned shares with historical resource consumption. Usage can decay with a configurable half-life, and billing can include multiple resource types. It is not a fixed quota that stops a user once consumed: over-served users may use otherwise idle resources. H1
Fair Tree makes that entitlement hierarchical. Sibling accounts are ranked using normalized shares and usage; an account’s ranking controls its descendants’ fair-share factors. The relation Level Fairshare = normalized shares / normalized usage captures under- versus over-service at each level. This is valuable when a project contains many users or workflows: child proliferation must not create more project entitlement. H2
However, Fair Tree ranks a fair-share factor, not an unconditional final job order or exact percentage of future starts. Other configured factors and resource feasibility still matter. Age also saturates, and dependent/ineligible jobs do not accrue age in the same way as eligible pending jobs. A bounded age term alone is not a proof that every low-priority job eventually runs. H1, H2
Adopt: stable accounting identities, hierarchy, auditable policy factors, usage accounting as a future independent layer, and borrowing unused capacity.
Do not copy blindly: the full multifactor tuning surface, CPU accounting when gh-aw cannot measure it reliably, or a claim that a weighted priority score creates an enforceable minimum service share.
4.3 Backfilling: resource feasibility without unnecessary head-of-line blocking
Section titled “4.3 Backfilling: resource feasibility without unnecessary head-of-line blocking”Lifka’s ANL/IBM SP scheduling work is the classic EASY backfilling reference. The later Mu’alem/Feitelson work examines utilization, predictability, workloads, and runtime estimates. L1, L2
The conventional distinction is:
| Policy | Protected reservation | Trade-off |
|---|---|---|
| EASY-style backfilling | The head job’s reservation | More aggressive use of gaps; weaker protection for other pending jobs |
| Conservative backfilling | Reservations for all considered pending jobs | Stronger predictability; potentially less flexibility and more planning |
Slurm’s documented backfill scheduler can start lower-priority jobs when doing so does not delay the expected start of higher-priority jobs. It considers resource requests and runtime limits, and reserves future resources. Accurate enough time limits and an adequate planning horizon are essential. H3
This is not equivalent to arbitrarily skipping a blocked urgent job. A scheduler claiming safe backfill must be able to demonstrate that the borrowed resources will be available by the protected start.
Translation to gh-aw: initially skip unavailable, incompatible, retry-delayed, and capped keys so they do not block independent eligible work. Do not describe this as EASY backfilling: the current queue does not own runner placement or a reservation calendar.
Reserve runtime-aware backfilling for a future scheduler that knows capacities, execution bounds, and reservation times. Separate resource pools are preferable to pretending that a GPU-blocked job and a CPU-ready job compete for the same interchangeable resource.
Also, HPC systems are not universally nonpreemptive: Slurm supports configured cancellation, requeue, suspension, and gang-scheduling modes. The relevant limitation is that this gh-aw proposal does not add safe preemption, not that Slurm cannot preempt. H9
4.4 DAGMan throttles and Slurm arrays: bound workflow fan-out
Section titled “4.4 DAGMan throttles and Slurm arrays: bound workflow fan-out”DAGMan provides per-DAG submitted-node and idle-job limits, plus category-specific MAXJOBS. These limits prevent a single workflow from overwhelming staging capacity or the scheduler. They are not, by themselves, cross-user fair-share policies, and a submitted cluster may contain multiple jobs. H10
Slurm arrays offer an analogous practical control: an array expression such as 0-15%4 limits simultaneously running array tasks to four. Array tasks remain subject to ordinary job limits. H11
Translation to gh-aw: a fair dispatch order can still produce poor responsiveness if one workflow preloads hundreds of accepted or pending Actions runs. Limit outstanding reservations, not just calls per dispatcher invocation. Count reserved, launching, queued, and running work until trusted reconciliation releases the reservation.
Distinguish:
- Share: a relative entitlement while multiple keys are eligible.
- Outstanding cap: an absolute admission limit, potentially leaving capacity unused.
- Rate limit: a maximum number of grants per time interval.
- Queue/admission bound: a maximum amount of submitted backlog.
They address different failure modes. A per-run batch limit cannot establish a global cap when many dispatchers run concurrently.