Solved: the deterministic tree-'b' EIO cluster (stale rbio-cache parity)

Post-mortem of the kernel #82 rolling-failure run's tree-'b' anomaly (32 EIO blocks that scrub cannot repair; 112/48/34/32 across builds #77-#82), from the preserved end-of-run member images in /mnt/wh/rolling on bhive. 2026-08-03.

Verdict

Series bug in the interaction between the upstream rbio stripe cache and stripe_alloc's claimed-stripe write path. A partial write into a freshly claimed (fully-free) stripe stole an ancient cached rbio whose pages held the stripe's PREVIOUS life -- content from before the extents were freed and before the harness wiped a member underneath the fs. The all-uptodate stolen pages made need_read_stripe_sectors() return false, which short-circuits BOTH the RMW disk read AND rmw_try_pad_full() in rmw_rbio():

if (!rbio_is_full(rbio) && need_read_stripe_sectors(rbio) &&
    !rmw_try_pad_full(rbio)) {   /* read phase */

Parity was then generated from the stale cached view while only the bio-covered data sectors + parity were written to disk. Result: a live data column protected by a parity that disagrees with the actual on-disk free-space columns -- latent, invisible (data reads direct, csums pass, tree 'b' verified all-correct at step 6), and fatal the moment a later wipe removed the live column. Reconstruction then XORs the REAL free-space garbage with the parity's MEMORY of it and produces trash: uncorrectable csum error, unrepairable by scrub, sysadmin-must-delete -- exactly one 16-sector column in run #82.

The affected stripe, byte for byte

All 32 EIO blocks are the last 128K of one 512K file, b/f00526 (inode 783), logical 1554841600..1554972671 in the DATA|RAID5 chunk at 862191616 (8 stripes, 7 data, per-item chunk parse, mapping verified against map_blocks_raid56_read()). That range spans a full-stripe boundary:

• full stripe 1509, data col 6 -> devid 8 (rf7, NEVER wiped)
• full stripe 1510, data col 0 -> devid 3 (rf2, wiped 3rd)

fs1509 is completely healthy on the final images: every data column matches the oracle bytes, XOR(data)==P, and the wiped devid-3 column (col 1) was repaired by scrub. Its 16 EIO blocks are read-path collateral: the buffered read of the file's tail extent fails at larger-than-sector granularity, so 16 undamaged blocks report EIO alongside the 16 truly damaged ones. True damage = 16 sectors = exactly scrub's "csum=16 Uncorrectable" on devid 3 (only device with errors in scrub-stripe_alloc-1/2.txt).

fs1510 is the real casualty. Disk state (read-only loops, images untouched):

• col0 (devid 3): wipe-3 keystream (verified against the generator). Was f00526's tail; the fresh 'b'-era data write landed and verified at step 6, destroyed at step 7.

• col1 (devid 1): wipe-1 keystream. Free space; never rewritten after the step-3 wipe.

• cols2-6 (devids 5,6,7,8,4): stale tree-'a' data, identified block-exactly as a/f06913 blocks 64..127 and a/f06920 blocks 0..15 by brute-force lookup against every deterministic oracle block of trees a, b, c (1,491,494 blocks indexed). Both files were freed in a's half-deletion, so the stripe was fully free -- the claim-rule invariant HELD when 'b' claimed it.

• P (devid 2): not wipe garbage => written during 'b'. Its value solves exactly and uniquely:

  P == XOR( f00526 tail data,          <- the new write, included
            a/f06913 blocks 48..63,    <- PRE-WIPE content of col1
            disk cols2..6 )            <- stale 'a' data (as disk)
  

  The a/f06913-blocks-48-63 bytes existed NOWHERE on disk after the step-3 wipe (raid5, single copy, parity site also wiped). Only a memory copy -- the rbio stripe cache -- could have supplied them. Every rival hypothesis was tested and refuted per-sector 0/16: safe-RMW-of-disk, zeros-assumed padding, untouched 'a'-era parity, poison/wipe streams, misdirected parity (no fsn in the whole chunk has XOR(data)==this P; census: 1140/1536 stripes inconsistent, all the rest benign free-space stripes with wiped columns).

Mechanism, end to end

1. 'a'-era: fs1510 written (f06913 cols0-5, f06920 col6); a partial write on that stripe left a cached rbio (full rbios are never cached -- upstream policy -- so the cache population itself came from a sub-stripe write, and under stripe_alloc those are rare, which is why an entry can survive: cache insertions, not time, evict the 1024-entry LRU).
2. Step 2: both files freed. Nothing invalidates the rbio cache on extent free (upstream behavior).
3. Step 3: devids 1+2 wiped underneath the mounted fs. Cache still holds pre-wipe pages.
4. Step 5: stripe_alloc claims fs1510 (fully free -- legal), f00526's tail becomes a partial write of col0. lock_stripe_add() finds the stale cached rbio, steal_rbio() moves its all-uptodate data pages in, need_read_stripe_sectors() sees nothing to read, and the padding path never runs. generate_pq_vertical() computes parity from bio data + stolen stale pages; rmw_assemble_write_bios() writes ONLY bio sectors + parity. Disk: fresh col0, fresh poisoned parity, untouched garbage cols1-6.
5. Step 6 verify: col0 reads directly, csum OK. All green.
6. Step 7: wipe 3 destroys col0. Reconstruction = P ^ disk c1..c6 = data ^ (aC1 ^ wipe1) = trash. EIO forever; scrub cannot help (the information is genuinely gone); tree-'b' gate stays red by 0.9% until the file is deleted.

Per-build determinism and variation (112/48/34/32): the workload is deterministic, so which stripes carry surviving cache entries and which partial writes land on re-claimed stripes is fixed per build, but shifts as the parking/padding machinery changes between builds. The counts also mix true damage with the read-granularity collateral (e.g. #82 = 16 real + 16 collateral).

Why this matters beyond the harness

The wipe models silent device corruption -- squarely in the test's scope ("btrfs keeps running, corrects single-device corruption"). Stock btrfs has the same cache-steal (the cache and steal_rbio are upstream); on a healthy device the cached pages always match disk, so stock is only exposed when a device corrupts underneath -- and then stock's sub-stripe RMW writes near the corruption bake the stale view into parity the same way. But stripe_alloc makes the dangerous case STRUCTURAL: every claimed stripe's first partial write is exactly a "partial write into a region whose prior on-disk life the cache may remember". The series' core invariant -- a claimed stripe's write depends on nothing on disk -- was silently violated by depending on the cache's memory OF the disk.

Fix directions (series)

1. Minimal: swap the order so padding wins over the cache -- if (!rbio_is_full() && !rmw_try_pad_full() && need_read...). The dangerous case is precisely the first write of a claimed stripe, which is always fully paddable (uncovered sectors are at or past the run frontier); padding zero-fills AND WRITES the uncovered columns, restoring disk/parity consistency regardless of what the cache remembered. Later same-run partial writes steal only same-run cache entries (the first write consumed any stale pre-claim entry), which are fresh.
2. Belt-and-suspenders: invalidate any cached rbio covering a stripe at claim time -- the claim is a declaration that the stripe's prior contents are dead.
3. Upstream-worthy (separate discussion): drop cached rbios when their stripe's extents are freed, closing the same latent window for stock RMW after silent corruption.

Prediction: with (1) or (2), the rolling test's tree-'b' post-wipe-3 verify goes fully clean (only the fill-edge warned drops remain), and scrub-2 reports no errors.

CONFIRMED on kernel #85 (fix (1) as fixup! btrfs: raid56: pad sub-stripe writes to full stripes in open runs): the full rolling run's tree 'b' verified eio=0/short=0 at every step including after the dev-3 wipe, and BOTH scrubs reported "no errors found" -- the first fully clean rolling pass (the fill-edge drops were eliminated in the same kernel by the round-5 reservation margin, see NOTE-stripe-alloc-reservation-and-reclaim.md). Fix directions (2) (invalidate cached rbios at claim) and (3) (upstream: drop cached rbios on extent free) remain open options.

Forensic method notes

Everything read-only: losetup -r on the images, btrfs device scan --forget before and after, ro mount for the EIO list + fiemap + logical-resolve, then raw image reads. Mapping was verified by content (oracle-block search at predicted member offsets) before any conclusion, per the two prior false-conclusion lessons. The brute-force identification of unknown 64K columns against the full deterministic oracle block set (16-byte prefix index, then full-block confirm) is the tool that cracked it: it turned "parity is random garbage" into "parity is exactly XOR(new data, one specific dead file's blocks, disk)".