2026-08-09. Found while validating the claimable-supply admission gate (wip/gate-*). The gate is the trigger; the deadlock is in machinery that predates it.
balance-reclaim-test.sh, "tree b: fill to ENOSPC into the reclaimed space" phase, on the gate kernel (#23 = testing + claimable counter + admission gate). Reproduced on two hosts:
btrfs_retire_open_stripes runs BEFORE the commit sets UNBLOCKED. It waits for the endio workers' runs to drain; the endio workers wait for UNBLOCKED; UNBLOCKED waits for retire. Circular.
while (ordered_extent->stripe_seq) {
u64 transid = trans->transid;
u64 retired;
spin_lock(&fs_info->open_stripe_lock);
retired = trans->transaction->stripe_retire_seq;
spin_unlock(&fs_info->open_stripe_lock);
if (!retired || ordered_extent->stripe_seq - 1 < retired)
break;
btrfs_end_transaction(trans);
btrfs_wait_transid_unblocked(fs_info, transid); /* <-- blocks */
trans = btrfs_join_transaction(root);
...
}
The deferral is meant to push an extent from a not-yet-retired run window to a later transaction (design invariant: no file extent in a window the commit did not drain). The hazard: the endio worker holds its ordered extent's stripe-run inflight accounting undrained WHILE it waits here, and the very commit it waits on is the one whose retire is waiting for that drain.
Design doc (covers/stripe-alloc.txt) claims the retire drain "is a pure data-IO wait, never an ordered-extent wait -- the latter needs a join too." This deadlock is the counter-example: the ordered extent's data IO is done, but its stripe-run write-done report has not been made (it happens later in finish_one_ordered), so retire's data-IO wait is transitively an ordered-extent wait after all.
The retire path, the stripe_seq deferral, and wait_transid_unblocked all predate the claimable work -- they are in topics/stripe-alloc base and the counter kernel (#22), which PASSED balance-reclaim once earlier the same day. The gate adds no code to any stuck stack. What it adds is near-full admission backpressure: many writes queue on the gate, flush to delalloc together, and produce a burst of ordered extents in the commit's join window -- exactly the population that defers via wait_transid_unblocked. So the working hypothesis is a LATENT pre-existing deadlock that the gate makes reliably reproducible.
Attribution experiment (bhive, 2026-08-09, CONCLUDED): counter-only kernel #125 (gate symbol absent, confirmed), balance-reclaim in a loop. Run 1 PASSED (~25 min). Run 2 DEADLOCKED at ~45 min in, identical signature: endio-write workers in D + retire_open_stripes waiting, sysrq-w showed 4 tasks in btrfs_wait_transid_unblocked. PRE-EXISTING CONFIRMED: the deadlock occurs WITHOUT the gate, probabilistically. The gate raises the probability sharply (hung on the FIRST balance-reclaim, every time; counter took a second loop iteration). The bug lives below the counter patch, in the base stripe_alloc retire/finish-ordered machinery, so it affects the ENTIRE series -- topics/stripe-alloc and everything merged over it, both master and 6.18 lines -- independent of any claimable work.
Reproduced on the gate kernel (wip/gate-testing 614890a5fca4) and probed with drgn against a full-DWARF vmlinux. Two prior claims in this file are now revised:
It is NOT (this run) a frozen circular deadlock. While a task sat in btrfs_retire_open_stripes (D) and python3 sat in wait_current_trans (D) -- the exact "deadlock signature" -- fs_info->stripe_retire_seq advanced continuously (12 -> 15 -> 20 -> ... -> 144) the whole time. The commit pipeline is PROGRESSING, slowly. The balance test was killed by its own 2400s timeout (rc=124) mid "tree b: fill to ENOSPC", NOT hung. A 6-sample (6 min) freeze detector never fired.
What retire actually waits on: at the seq=12 snapshot the two un-settled runs' ENTIRE inflight (32768 + 118784 bytes) was owned by 5 PARKED partial-stripe rbios sitting inside those runs -- left parked by retire's own btrfs_flush_parked_rbios(..., force=false). The four btrfs-endio-write workers in wait_transid_unblocked hold NO inflight (see correction below); they are co-victims, not the cause.
Why parked rbios stall retire: a parked rbio is "ready" (parked_rbio_ready, raid56.c:836) once its gathered bios cover [stripe_start, alloc_ceiling); the pad fills the dead tail. A NOT-ready rbio is missing an allocated neighbour's in-flight bio. retire uses force=false precisely so it does NOT pad over a sector an in-flight allocated bio still owns (that would be data loss / write hole -- the thing stripe_alloc prevents). So retire waits out the ~1s park_stuck_deadline (measured: timeout_ms~=100, stuck=10x=~1s) for each such rbio. Park stats over one killed run: parked=30511, stuck=8010, data_rmw=5549, padded=10354. Under fill-to- ENOSPC every run has an unfillable final partial stripe, so this recurs every commit.
Severity reclassification: this is a SEVERE THROUGHPUT PATHOLOGY, not a proven hard deadlock. The earlier "40-min hang / dhive hung_task panic" observations are consistent with (a) the 300s hung_task watchdog firing on one long-but-finite retire wait, or (b) the test simply not finishing in time -- both look like a hang to a liveness check. A genuinely frozen circular-deadlock MODE is NOT confirmed and NOT reproduced here; prior "PRE-EXISTING deadlock CONFIRMED" is downgraded to "pre-existing severe slowdown; hard-freeze mode unconfirmed." (An attribution run with the park timeout forced low is in flight to separate parking-wait cost from the gate's intended near-full admission backpressure.)
Fix caution: the naive "force-flush parked rbios at retire" is UNSAFE -- it can pad over in-flight allocated data (data loss). Any real fix must accelerate the missing arrivals or exclude only provably-dead space, and it lives in Zygo's retire/park invariant machinery. Do not apply blindly.
Prototyped the mechanism-matching fix (wip, off gate for signal): replace retire's single wait_var_event with the force + 10ms-wait + retry drain the fsync/ordered waiters use (btrfs_flush_parked_rbios(..., true) in a loop until bg_open_stripes_settled). Diff: /root/retire-fix.diff on bhive (fs/btrfs/block-group.c, +20/-2). Built on the gate config, A/B vs baseline gate, balance-reclaim-test.sh:
baseline gate (default park timeout): rc=124 timeout@2400s; stuck=8010 kicked=2233 gate + retire fix (default park timeout): rc=124 timeout@3001s; stuck=7760 kicked=6773 gate, park=3ms knob (no code change): rc=0 COMPLETE @1232s, both variants PASS
The fix worked as designed -- retire force-kicks tripled (kicked 2233->6773) -- but total time was UNCHANGED and it still timed out. stuck barely moved (8010->7760). => retire's drain was NEVER the throughput bottleneck.
Root cause (revised, final): the fill-to-ENOSPC slowness is dominated by WRITE-PATH stripe parking latency, not the commit drain. Every partial write parks up to ~1s (stripe_park_timeout_ms, stuck=10x) hoping to merge into a full stripe; at fill-to-ENOSPC the merges cannot complete (final partial stripes) so it is pure latency, ~8000 rbios eating the stuck deadline. Lowering the GLOBAL park timeout (3ms knob) fixes throughput (1232s, PASS) because it shortens every write's park; the retire-only fix cannot, because it only shortens the commit drain.
Data integrity held on EVERY run (tree a/b verify: 0 eio/zero/wrong/ missing) -- this is purely throughput, never loss. No hard deadlock was ever reproduced (retire_seq advanced monotonically throughout).
Recommendation for Zygo (all his call -- park policy is his design):
Re-reading the base series (topics/stripe-alloc counter tip 3e598f945328) before prototyping candidate 1 falsifies the mechanism this document assumed. The stripe-run write-done is NOT reported inside btrfs_finish_one_ordered (where the deferral wait lives); it is reported at BIO COMPLETION, one call earlier:
ordered-data.c can_finish_ordered_extent() (bytes_left hits 0)
line 450: btrfs_ordered_stripe_write_done(ordered)
-> ordered->stripe_run = NULL
-> btrfs_open_stripe_write_done_run(run, disk_num_bytes)
then queues finish_ordered_fn -> btrfs_finish_one_ordered
So by the time an endio-write worker reaches btrfs_wait_transid_unblocked (inode.c:3128), ordered->stripe_run is already NULL and the run inflight for THIS extent is already decremented. The four btrfs-endio-write workers in the sysrq dump therefore do NOT hold the inflight that btrfs_retire_open_stripes is waiting to drain -- they are co-victims blocked on the same commit, not the cause of retire's wait.
Consequences:
The blocking inflight belongs to an ordered extent still IN FLIGHT. The leading suspect is a PARKED partial-stripe rbio that retire's own flush left parked: btrfs_retire_open_stripes calls btrfs_flush_parked_rbios(fs_info, 0, U64_MAX, force=false); with force=false a not-ready parked rbio is left parked (raid56.c:986), expecting "the arrivals will complete it." At fill-to-ENOSPC the arrivals never come (all writers blocked in wait_current_trans), so the stripe drains only when the park stuck-deadline (timeout_ms*10) fires -- unless something prevents even that. Ground truth needed: which run is un-settled, its inflight_bytes/open/open_seq, and the parked-rbio table state, captured by drgn on a live reproduced hang (no rebuild-with-printk required; build host carries full-DWARF vmlinux + BTF).
The gate itself is correct in intent and passed its own acceptance (zero writeback loss, both dataloss modes); it is held unpublished (wip/gate-*) until the underlying deadlock is fixed, because it turns a rare hang into a frequent one.
The counter patch (published: topics/stripe-alloc, both lines) does NOT add the gate and does not create this backpressure, but it shares the latent retire/deferral machinery. If the attribution run shows the counter kernel hangs too, the whole stripe_alloc series carries this latent deadlock independent of any of the claimable work, and that is the headline for Zygo.