Queue and Scheduling Observability

Fuzzball exposes the scheduler queue so you can see what is waiting, what is running, and — for any allocation that is not yet running — why. Three commands cover this:

fuzzball queue show — list allocations in the scheduler queue.
fuzzball queue stats — aggregate scheduler counters and tick timings.
fuzzball workflow why — the typed reason a specific allocation is in its current state.

queue show (cluster-wide) and queue stats are cluster-admin commands. Any authenticated user can list their own queued allocations with queue show --mine, and workflow owners can run workflow why on their own allocations without admin rights.

Prerequisites

The cluster-wide views require an admin context:


$ fuzzball context create <context_name> <api_url>

$ fuzzball context login -u <admin_username> -p '<admin_password>'

queue show --mine and workflow why on your own allocations work with an ordinary user context.

View the Queue

The fuzzball queue show command lists allocations tracked by the scheduler, ordered by effective scheduling priority.

Basic Usage


$ fuzzball queue show

Example Output


ID                                   | WORKFLOW                             | JOB  | QUEUE   | STATUS  | PRIORITY | WEIGHT | NODE           | PREEMPTIBLE | QUEUE TIME            | OWNER
217e5086-3f7f-5f03-a58e-c0f1b2156ff6 | 0d87ddcb-881c-4011-bea7-760bdf136447 | occ  | running | running | 0        | 1      | 10.0.0.46/7331 | false       | 2026-07-02 10:23:17AM | c30a8e57-cccb-448c-85c7-e49f197ceed4
cf043e9a-2cdf-59de-9ee3-8c41986ffd9d | 75ec995e-ce3f-4ab5-b946-5c789d7aeb70 | head | ready   | -       | 0        | 1      |                | false       | 2026-07-02 10:33:41AM | c30a8e57-cccb-448c-85c7-e49f197ceed4

Allocations in the pending queue are blocked on dependencies and not yet scored, so their priority shows as -.

Filtering

Flag	Description
`--queue`	Restrict to one queue: `pending`, `ready`, `running`, or `finished`.
`--priority-min`	Only show allocations at or above the given effective priority.
`--owner`	Restrict to allocations owned by the given identity id (admin only).
`--mine`	Restrict to the calling user's own allocations.
`--cluster`	Query specific owning clusters by id (federate deployments); repeatable or comma-separated. Omit to query all.

Viewing Your Own Queue

Non-admin users can see where their own allocations sit in the queue with --mine:


$ fuzzball queue show --mine

The owner scope is enforced by the server, so --mine only ever returns the caller's own allocations. Because the scope is fixed to the caller, --mine cannot be combined with --owner or --cluster.

Note

Without --mine, queue show returns the cluster-wide queue and requires a cluster-admin context. --mine works for any authenticated user.

Queue Statistics

The fuzzball queue stats command (cluster-admin) reports aggregate scheduler counters, queue depths, tick timings, and the current backfill reservation.


$ fuzzball queue stats

Example Output


Queue depths
  pending              0
  ready                1
  running              2
  finished             3
Tick duration
  total ticks          3732
  p50                  0.006s
  p99                  0.008s
Preemption / backfill
  preemptions total    0
  backfilled total     1
Reservation
  active               true
  allocation id        cf043e9a-2cdf-59de-9ee3-8c41986ffd9d
  computed at          2026-07-02T17:34:17Z

In a federate deployment, pass --cluster to select owning clusters; queue depths and counters are aggregated across clusters, while tick timings and reservation state are reported per cluster.

Explain an Allocation

When an allocation is not running, fuzzball workflow why returns a typed reason and a human-readable detail string. It takes the workflow id and the allocation id (both UUIDs — get them from fuzzball workflow describe or queue show):


$ fuzzball workflow why <workflow-id> <allocation-id>

Example Output


Workflow: 75ec995e-ce3f-4ab5-b946-5c789d7aeb70
Allocation: cf043e9a-2cdf-59de-9ee3-8c41986ffd9d
Reason: insufficient resources
Detail: insufficient resources: head-of-queue but no node can fit (est_start=2026-07-03T01:54:25Z)

Workflow owners can explain their own allocations; explaining another user's allocation requires the appropriate permission. workflow why operates on allocations that are still in the queue — once an allocation finishes it is no longer explainable.

Reasons

Reason	Meaning
`READY_TO_SCHEDULE`	Eligible; expected to place on an upcoming tick.
`WAITING_DEPENDENCY`	Blocked on other stages it depends on (the detail lists them).
`AWAITING_PROVISIONING`	A provisioning request is in flight for it.
`PROVISIONING_BLOCKED`	Cannot provision a node right now (for example, a dynamic definition at its `maxNodes` cap).
`INSUFFICIENT_RESOURCES`	Head of the queue but no node currently fits; the detail includes an estimated start time when the reservation has one. When no start can be estimated (for example, services with no walltime hold the nodes, occupancy is unattributable, or the allocation's shape can never fit), the detail reports node availability as unestimable instead.
`BACKFILL_INELIGIBLE_NO_WALLTIME`	Cannot backfill because the allocation has no walltime (for example, a service).
`BACKFILL_INELIGIBLE_WOULD_DELAY_HEAD`	Its walltime would run past the reserved start of the blocked head, so backfilling it would delay that job.
`BACKFILL_INELIGIBLE_INTERNAL_NO_SURPLUS`	An internal staging job (image or data fetch) whose walltime ceiling runs past the reserved start and whose resources do not fit within the spare capacity the blocked head leaves on the reserved node.
`BACKFILL_NO_FIT`	Its walltime fits the gap, but its resource shape does not fit the free capacity before the reservation. Also reported for a candidate that is eligible behind an unestimable reservation (no estimated head start) but has not placed yet.
`RUNNING`	Currently running.
`FINISHED`	Completed.
`CANCELLED`	Cancelled.
`PREEMPTED`	Evicted to free capacity for a higher-priority allocation.

Priority, Preemption, and Backfill Debugging