Consolidating metadata updates into full-stripe writes

Design options, 2026-08-05. Goal: give raid56 metadata the property stripe_alloc gives data — no sub-stripe write ever lands in a stripe holding committed data — so a crash on a degraded array cannot tear a tree block.

What the ground looks like

Facts the options have to live with (all verified in the tree, master testing):

1. Metadata never claims a stripe today. btrfs_is_stripe_alloc_bg() requires the DATA flag, so metadata block groups take the ordinary clustered path. fetch_cluster_info() gives metadata an empty_cluster of 2M (SSD) or 64K, but a cluster is a locality hint with no stripe alignment, and it survives across transactions. Measured on the gate test: meta_rmw = 13-15 sub-stripe metadata RMWs in a few seconds of small-file churn on -m raid5.

2. Writeback is already batched. btrfs_write_and_wait_transaction() pushes the whole dirty extent buffer set in one go (btrfs_write_marked_extents(dirty_pages)). The timing that data had to engineer, metadata gets for free — the problem is purely placement.

3. The retire point is in the wrong place for metadata. btrfs_retire_open_stripes() runs at transaction.c:2455 and waits for claimed runs to drain; tree blocks are written at 2636. A metadata run claimed in the current transaction therefore cannot settle before the commit reaches the point that would settle it. This is not theoretical: it is the deadlock that mixed block groups hit (see the commit "btrfs: stripe_alloc: do not claim stripes in mixed block groups"). Any metadata scheme must move metadata retirement after tree writeback.

4. Metadata ENOSPC is a transaction abort, not a failed write. A claim that cannot be satisfied returns ENOSPC to the caller; for data that fails one write, for metadata the filesystem goes read-only. Whatever claims stripes for metadata needs a reservation margin before it can be turned on, in the same shape as the data-side pessimistic margin.

5. The geometry is favourable, and tunable. BTRFS_STRIPE_LEN is 64K, default nodesize is 16K:

   devices (raid5)	data columns	full stripe	tree blocks per stripe
   3	2	128K	8
   5	4	256K	16
   3, nodesize=64K	2	128K	2
   5, nodesize=64K	4	256K	4

   A commit that dirties a few dozen tree blocks fills several stripes at 16K nodes; at 64K nodes it fills them almost trivially.

6. Metadata is always CoW. No nocow class, no prealloc, no persistence of run state across mounts — the whole allow_rmw apparatus and the nocow-run persistence the data side needs simply do not apply. Metadata is simpler than data in every respect except reservations.

7. Reconstructed metadata is not verified inside raid56. fill_data_csums() returns early for non-DATA rbios, so csum_bitmap is NULL and verify_one_sector() returns 0 without checking anything. Detection happens one layer up in validate_extent_buffer(), and recovery is btrfs_read_extent_buffer() retrying mirror_num 1..N.

Options

A. Claimed metadata runs (direct analogue of the data path)

Metadata block groups claim full stripes exactly as data does: btrfs_is_stripe_alloc_bg() accepts METADATA, do_allocation_stripe() serves tree block allocations from an open run, rmw_try_pad_full() pads the tail at writeback.

• Guarantee: the same one data has. Strongest option.
• Needs: the retire reordering (fact 3); a metadata reservation margin (fact 4); a decision about the tail (see E); a policy for the log tree (see the log section).
• Risk: highest, because it puts a new failure mode (ENOSPC at claim time) in front of the one operation that aborts transactions.
• Reuses: everything — claim, run tracking, padding, the invariant checker, balance-based reclaim of stripe_unusable.

B. Stripe-aligned metadata cluster, reset every transaction

Do not claim. Instead make the existing metadata cluster (meta_alloc_cluster) stripe-sized and stripe-aligned, and drop it at transaction commit so the next transaction starts on a fresh stripe.

• Guarantee: weaker but real. Within a transaction, tree blocks pack into whole stripes; across transactions, only the boundary stripe (the one the cluster was in when the commit happened) can be shared.
• Needs: an alignment constraint in btrfs_find_space_cluster() and a reset hook at commit. No retire reordering, no reservation margin, no ENOSPC-at-claim: if the aligned cluster cannot be found, the allocator falls back to the ordinary path exactly as it does today.
• Risk: low. It cannot abort a transaction, cannot deadlock, and degrades to current behaviour under pressure.
• Cost: does not guarantee the invariant, so the write-hole checker would still fire on the boundary stripe. Best understood as "most of the benefit for almost none of the risk", and as a way to measure how much of the benefit is available before committing to A.

C. Writeback-side consolidation only

Leave allocation alone; at commit, group dirty extent buffers by full stripe and issue whole-stripe writes where the stripe happens to be fully dirty.

• Verdict: insufficient alone, useful as instrumentation. Padding a partially dirty stripe requires the uncovered columns to be free space; if allocation did not reserve the stripe, those columns may hold live tree blocks from earlier transactions, and writing the stripe would rewrite committed data — the write hole by another name. So C can only exploit stripes that A/B/D would have produced anyway.
• Worth building first as a counter: how many commits already dirty whole stripes by luck? That number sizes the benefit of B.

D. Per-transaction append frontier

Metadata allocation within a block group becomes a bump pointer for the duration of a transaction: each transaction starts at a stripe boundary and appends. Freed blocks go back to the free space tree as usual but are not reused until a later transaction.

• Guarantee: as strong as A, with a simpler invariant to state and audit ("a transaction's tree blocks occupy a contiguous, stripe-aligned extent").
• Needs: the same retire reordering; the tail question (E) becomes the central one, because every commit ends mid-stripe by construction.
• Interesting property: it makes metadata layout sequential, which is independently good for read locality — and makes the write pattern look like a log, which is what raid56 is best at.

E. What to do with the tail

Every option that aligns to stripes leaves a partially filled stripe when the transaction ends. Four ways to handle it, usable in combination:

1. Strand it (what data does). The remainder becomes stripe_unusable, reclaimed by balance. Simple and already accounted for; the concern is that metadata is ~1% of the filesystem, so stranding up to 7/8 of a 128K stripe per commit is proportionally larger than it is for data. At one commit per 30s that is bounded but not free.
2. Pad it (zero-fill and write the whole stripe). Costs the write bandwidth of the padding, gains a self-consistent stripe immediately. rmw_try_pad_full() already does exactly this.
3. Complete it with useful blocks. Opportunistically CoW cold tree blocks into the tail so the stripe finishes full. Attractive — it turns waste into defragmentation — but CoWing a block dirties its parent, which allocates again, which may open another stripe. Needs a hard bound (e.g. only relocate leaves whose parent is already dirty in this transaction) or it cascades.
4. Shrink the geometry for metadata. Allocate metadata chunks with fewer data columns (narrower raid5) so a full stripe is smaller and the tail is proportionally cheaper. A chunk-allocator policy change, no effect on the write path.

F. Tune nodesize (no kernel change)

With nodesize equal to the stripe length (64K), a 3-device raid5 full stripe is two tree blocks and a 5-device one is four. Any of A/B/D becomes dramatically easier to satisfy, and the tail in E shrinks to at most one node. This is an mkfs-time property, so it is a recommendation rather than an implementation — but it may be the single highest-leverage change for anyone who actually wants raid56 metadata.

G. Baseline: keep metadata off raid56

raid1c3/raid1c4 at metadata's ~1% footprint, which is what the series currently recommends at mount. Every option above has to justify itself against this, and for most users it will not: the space saving is negligible and the complexity is not. The case for doing the work is filesystems where metadata is large in absolute terms, or where the mirrored-metadata write amplification actually matters.

The verification gap (independent of the option chosen)

None of A-F make reconstructed metadata verifiable. Today a metadata rbio reconstructs a lost column and sets the uptodate bits on unverified data; if the reconstruction is wrong, validate_extent_buffer() catches it later and the read is retried with the next mirror_num. Two consequences:

• raid6 with two lost columns has several candidate reconstructions, and for data the csum bitmap lets raid56 pick the right one per sector. For metadata it cannot, so the search is driven from outside, one mirror at a time.
• Unverified reconstructed metadata can enter the stripe cache and later feed a parity computation — the same shape as the stale-cache bug fixed for data, with no csum path available to catch it.

Proposed fix, worth doing regardless: teach raid56 that a metadata rbio's verification unit is a nodesize-aligned extent buffer rather than a sector. A tree block is nodesize bytes, nodesize <= BTRFS_STRIPE_LEN and both are powers of two, so an eb always lies entirely within one column — after reconstructing that column, the eb header csum can be checked in place. verify_one_eb() alongside verify_one_sector(), called from recover_vertical() when map_type & METADATA.

Suggested order of work

1. Counter first (C). Add a "commit dirtied a whole stripe" counter next to meta_rmw. Cheap, and it tells us how much of the benefit is already there by accident — which decides whether B is enough.
2. verify_one_eb(). Independent of allocation, fixes a real gap in what reconstruction can be trusted to produce, and is a prerequisite for trusting anything built on top.
3. B, behind its own mount option. Low risk, no new abort paths, measurable against the counter from step 1.
4. The retire reordering. Required by both A and D; do it on its own, with the mixed-bg deadlock as the regression test.
5. A or D, once metadata reservations have a pessimistic margin in the same shape as the data side. D if the sequential-layout property is wanted; A if reusing the data path verbatim matters more.

The log tree is a separate decision throughout: its writes happen at fsync, outside the commit, and are latency-critical. Either give it a dedicated run (mirroring the per-inode runs the data side keeps for log-active inodes), or leave log tree blocks on the legacy path and accept that the fsync window is not covered — which is what the series already documents for data.

Companion notes: NOTE-stripe-alloc-reservation-and-reclaim.md (the metadata reservation analysis), NOTE-directed-writehole-parity.md.