btrfs_merge_delayed_refs(): soft lockup on one large metadata head

2026-09-17. Found in Zygo's own logs from waya, not on the test rigs. Upstream btrfs, unmodified. Material for an upstream report.

Sources: ~/share/linux-stable-6.18/lockup-2026-09-17-10.txt (6.18.52-956e91f4e9c44+ #29) and the mid-crash companion dmesg-2026-09-17.txt (#30).

What happened

watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [btrfs-transacti:4699]
CPU#3 Utilization every 4000ms during lockup:
    #1..#5:  97% system, 0% softirq, ~2% hardirq, 0% idle
RIP: 0010:rb_next+0x27/0x80
Call Trace:
 ? btrfs_merge_delayed_refs+0xb7/0x340
 __btrfs_run_delayed_refs+0xf4/0x1260
 btrfs_run_delayed_refs+0x40/0x1c0
 btrfs_start_dirty_block_groups+0x2f4/0x600
 btrfs_commit_transaction+0xeb/0x1020
 transaction_kthread+0x169/0x1d0

Not a stuck wait: 97% system time in every sampling window, and the register state shows the ordinary leftmost-descent loop inside rb_next(). The transaction kthread is working, for longer than the watchdog tolerates.

Earlier in the same boot, at 10:31, the same grind (then owned by a snapshot CLI, btrfs:24141) had already produced a 622-second hung-task pileup: four bees worker threads in openat2 and eight btrfs commands.

Why it is quadratic

fs/btrfs/delayed-ref.c, identical to upstream v6.18:

again:
	for (node = rb_first_cached(&head->ref_tree); node; node = rb_next(node)) {
		ref = rb_entry(node, struct btrfs_delayed_ref_node, ref_node);
		if (seq && ref->seq >= seq)
			continue;
		if (merge_ref(fs_info, delayed_refs, head, ref, seq))
			goto again;
	}

Two properties combine badly:

  1. Every successful merge restarts the walk from rb_first_cached(). A head with N refs and M merges costs O(M*N) rb_next() steps.
  2. There is no cond_resched() in the function. The only one in the file is in the cleanup path. With PREEMPT(voluntary) the walk therefore holds the CPU until it finishes.

Data heads are skipped outright (if (head->is_data) return;, commented "we don't have too many refs to merge for data"), so the head that grinds is a metadata head. Relocation's "move data extents" stage rewrites the extent tree and concentrates large numbers of metadata refs on individual heads, which is exactly how N gets big enough to cross 26 seconds.

Trigger

Degraded raid5 testfs. The mount auto-resumes a balance, which relocates one block group holding 184,351 extents:

BTRFS info (device dm-1): balance: resume -dvrange=0..4363918508032
BTRFS info (device dm-1): relocating block group 4359623540736 flags data|raid5
BTRFS info (device dm-1): found 184351 extents, loops 1, stage: move data extents

A snapshot-heavy workload (bees) runs against the same filesystem.

Why it surfaced as hung tasks and not merely slow commits

btrfs_mksubvol() takes down_write_killable_nested(&dir->i_rwsem, I_MUTEX_PARENT) on the parent directory and releases it only at out_unlock, so it holds that directory exclusively across create_snapshot() and the transaction commit inside it. Anything opening a path through that directory blocks for the whole grind. That is the bees workers, stuck in openat2 -> path_openat -> down_read on &type->i_mutex_dir_key#8.

Correction: do NOT read dmesg silence as "stripe_alloc off"

This note first claimed the allocator was off on that filesystem because no stripe_alloc string appears in either log. That was wrong. getfattr shows btrfs.stripe_alloc="1" on /media/testfs, and only one of the two enable paths is loud:

So silence means "on via the mount option" or "the property never applied", and never "off". The authoritative live check is

grep ' /media/testfs ' /proc/mounts | tr , '\n' | grep stripe

because both paths set the same option bit and btrfs_show_options() prints it.

The delayed-ref quadratic below is still upstream code, byte-identical, and the lane touches delayed-ref.c only via an unrelated qgroup fix. But the claim that our allocator is not a factor is withdrawn: if it is active there it is in play, and the August analysis already flagged stripe_alloc fragmentation as a plausible amplifier of delayed-ref debt.

What the log does and does not say about the allocator

Zero occurrences of stripe_alloc, stripe_meta or stripe_park in either log -- which, per the correction above, tells us nothing either way. Mount options were degraded, ssd, flushoncommit, async discard, free space tree, autodefrag, zstd. The lane carries one commit touching delayed-ref.c (364685c4c2d9, an unrelated qgroup double-free fix) and none touching this function. The only patch of ours visible is balance-resume, doing its job: it resumed at its cursor and relocated a group below it, and it narrows the resumed range rather than widening it.

Distinct from the stall we already diagnosed

The 2026-08-18 finding (see the memory note on the silent commit stall) put NMI samples in __btrfs_run_delayed_refs -> __btrfs_free_extent -> lookup_extent_backref / btrfs_add_to_free_space_tree -> btrfs_search_slot, the free-extent and backref grind. This is the merge restart instead. Same family, delayed-reference debt, different hot spot.

Our rigs, for comparison

bhive #217 and dhive #78, stripe_alloc on, corrupt-model raid5, at the time of this analysis: zero soft lockups, zero hung tasks, no btrfs_merge_delayed_refs in dmesg, taint 0. bhive's worst commit out of 7167 was 10.1 s, mean about 223 ms. The octopus never relocates a block group near 184k extents, so it does not reach this.

Fix direction