Upstream bug: degraded raid56 RMW writes fail with spurious EIO

Found 2026-08-01 by the harness's new degraded-write continuation probe (WH_DEGRADED_CONTINUE=1) on its first run: on the legacy allocator, 29 of 56 degraded read-write continuations died with fsync EIO -- many on clean images (consistent state minus one device), where degraded operation is unambiguously supposed to work.

Symptom

Writing to a degraded raid5/6 array, any sub-stripe (RMW) write whose target stripe's missing-device column is only partially covered by data csums fails with EIO:

BTRFS error (device loop3): dropping unwritten extent at root 5
  ino 1061 offset [0,4095] disk bytenr 638771200 length 4096

No device IO errors, no corruption counted, nothing else logged (the "dropping unwritten extent" line is our own logging patch; stock kernels drop the extent silently). Deterministic per (fs layout, missing device, allocation target): a workload either always dies at the same write or always survives, which produced the confusing per-(cut,drop) failure pattern in the matrix.

Root cause

verify_one_sector() (fs/btrfs/raid56.c) checks that the rbio has a csum_buf/csum_bitmap at all, but never tests the bitmap bit for the sector it verifies. Sectors without csums -- free space, or sectors whose ordered extent has not yet committed its csums, including the very sector the sub-stripe write is replacing -- are compared against an all-zero csum_buf slot and fail.

Mechanism of a failing write, traced with kprobes:

1. 4 KiB write allocates at logical L, which maps to the missing device (or any RMW into a partially csummed stripe).
2. RMW read phase (rmw_read_wait_recover) marks every sector of the missing device as an error, reads the rest.
3. recover_sectors() rebuilds the missing column row by row and calls verify_one_sector() on each rebuilt sector.
4. First rebuilt sector with no csum (e.g. the free sector being overwritten): computed csum vs zeroed slot -> -EIO -> rbio fails -> ordered extent IOERR -> extent dropped, fsync returns EIO.

The whole-stripe early-outs explain why some degraded RMWs survive: if the stripe has zero csums anywhere, fill_data_csums() frees the buffers (no_csum:) and verification is skipped entirely; if the missing column is fully csummed, every verify passes. Only the partial case dies -- hence layout-dependent, "flaky-looking" failures.

verify_bio_data_sectors(), the sibling that verifies directly-read sectors on the same RMW path, has the test_bit skip. The recovery path never got one.

Upstream since the function's introduction; present in v6.3, v6.6, v6.12, v6.18, and linus/master (2026-08). Not caused by the stripe_alloc series (kernel #76 carries it, but the function is byte-equivalent to upstream).

Fix

/* No csum for this sector, nothing to verify against. */
if (!test_bit(stripe_nr * rbio->stripe_nsectors + sector_nr,
              rbio->csum_bitmap))
        return 0;

after the P/Q early-out in verify_one_sector().

Patch: "btrfs: raid56: only verify recovered sectors that have a checksum", Fixes: 7a3150723061 ("btrfs: raid56: do data csum verification during RMW cycle"), CC stable 6.2+.

Relation to stripe_alloc

• The protect configuration went 56/56 degraded continuations clean on the unfixed kernel: stripe-exclusive allocation never RMWs, so it structurally never enters the buggy path. A nice independent confirmation of the design, and the reason the bug went unnoticed in all previous stripe_alloc testing.
• The legacy configuration (and allow_rmw policies, which deliberately RMW) need this fix before degraded-write behavior can be evaluated at all.
• Second upstream degraded-raid56 bug found by this harness, after "btrfs: propagate a split bio's error when the failing split completes last" (silent zeros instead of EIO on degraded reads).

Repro

/root/manual-repro.sh on bhive: replay a clean 4-device raid5 image minus devid 2, mount degraded, write files with fsync. Fails at the first 4 KiB file within seconds on unfixed kernels.

Results on the fixed kernel (#77, raid5/4, barrier model)

• Manual repro: 300/300 files written+fsynced degraded, no errors.
• Legacy: 55/56 continuations complete, 0 damaged pre-crash data, 0 damaged new data. The single remaining failure is the case where the crash-time write hole had already damaged committed data (cut=1 tail=2 drop=2, the 1/48 armed hit): the continuation's RMW reads the genuinely inconsistent stripe, verification correctly refuses it, and the write fails with loud csum warnings instead of baking reconstructed garbage over the damage. With the fix, a degraded RMW either succeeds correctly or fails detectably -- silent bake-in of wrong data over csummed extents is prevented by the (now correctly scoped) verification.
• Protect: 56/56 continuations clean on both the fixed and UNFIXED kernels (no RMW, structurally immune).

Open question (harmless post-fix, noted for completeness): the clean cut=2 image contained exactly two stripe rows, both csum-less free space, where a single nonzero data column coexists with zero parity. Likely a discard interaction in the original run (async discard is on by default on loop devices and zeroes free data sectors without updating parity). No live data was covered in either row; a future audit could stamp free space and watch discard traffic to confirm.