Root-caused and fixed -- see "Cause and fix" below.
Found 2026-08-01 while investigating fsync-window damage in the
stripe_alloc crash matrix. Not caused by the stripe_alloc series and
not a 7.2 regression: reproduced identically on stock linus/master
(fc02acf6ac0c, "7.2.0-rc5+") and on stable/linux-6.18.y (6.18.39), both
built with the zb64 config (CONFIG_BTRFS_DEBUG=y, CONFIG_BTRFS_ASSERT=y,
CONFIG_PROVE_LOCKING=y).
On a degraded raid5 array, reading a data extent whose reconstructed sectors fail checksum verification returns a different answer depending on the size of the read:
BTRFS_STRIPE_LEN) or more return the reconstructed
bytes -- zeros -- with no error;EIO, but only for some of the same sectors.btrfs detects the corruption in both cases: it logs
csum failed root 5 ino 257 off 0 csum 0xf7710cc3 expected csum 0x1da2fbfa mirror 2 for the affected offsets. It then hands the bad data to
userspace anyway.
scripts/torn-parity-repro.sh (no dm-log-writes, no crash simulation):
mkfs.btrfs -d raid5 -m raid1 over five 512 MiB loop images, write one
1 MiB file of stamped blocks, sync, unmount;torn-parity-plan.py;
members do not share a physical base, so the location is proved, not
assumed) -- and overwrite it with zeros, leaving all data intact;mount -o degraded.Reconstructing the missing column now produces wrong bytes for that column only; every other column still verifies. That mixed state is what a crash leaves behind, and it is what exposes the bug -- see "What is not affected".
Results, identical on 7.2.0-rc5+ and 6.18.39:
| read shape | result | zero blocks returned |
|---|---|---|
| buffered whole file (1 MiB) | OK | 16 of 256 |
pread() 1 MiB |
OK, 0 EIO | 16 |
pread() 64 KiB |
OK, 0 EIO | 16 |
pread() 4 KiB |
4 chunks EIO | 12 |
O_DIRECT 1 MiB |
EIO | -- |
The 16 zero blocks are exactly the missing column's 64 KiB unit in the torn
row. Note the 4 KiB path is not consistent either: only 4 of the 16
damaged sectors report EIO.
The same behaviour appears in the crash-replay matrix, where a torn log
replayed onto a degraded array resurrects references to data whose stripe
writes never completed (scripts/../fsync-block-probe.sh on a
WH_FSYNC_WINDOW=1 workdir): there a 1 MiB file returned 48 zero blocks
for every read of 64 KiB or more, and 36 EIO / 12 zeros for 4 KiB reads,
with the checksum tree fully covering the extent
(btrfs inspect-internal dump-tree -t 7: uncovered_bytes == 0).
scripts/readpath-suite.sh covers the ordinary error paths, and they are
all correct on the same kernels:
-d single -m dup, only copy corrupt: EIO for both read shapes;-d raid1 -m raid1, both copies corrupt: EIO for both;EIO for both read shapes.So the bug needs a torn stripe and a missing member and a read larger than a sector -- part of the stripe must verify while part does not. That is presumably why it has gone unnoticed.
Silent data corruption on a degraded raid56 array recovering from a crash: zero-filled data is returned for sectors whose checksums do not match, with no error at the syscall, while the checksum failure is logged to dmesg. Ordinary applications, which read in chunks far larger than a sector, get the bad data; only a program doing 4 KiB reads sees any error at all.
It also means "silent mismatch" verdicts from crash-consistency tooling on
raid56 can be an artifact of this behaviour rather than of the corruption
under test: with correct error propagation those reads would be detectable
EIO.
Not in raid56 at all. btrfs_bio_end_io() in fs/btrfs/bio.c saves the
first error of any split bio in the original btrfs_bio's ->status, and
loads it into ->bio.bi_status when the last split completes -- but only
if that last completion itself succeeded:
if (atomic_dec_and_test(&bbio->pending_ios)) {
/* Load split bio's error which might be set above. */
if (status == BLK_STS_OK)
bbio->bio.bi_status = READ_ONCE(bbio->status);
The guard assumes a failing final completion already stored its error in
the original bio at the top of the function. That is true only when the
bio was never split: for a split, bbio->bio.bi_status = status lands on
the clone, which is then freed and bbio switched to the original, whose
bi_status keeps whatever an earlier successful split left there.
Everything observed follows from that:
btrfs_submit_chunk() splits a read at every stripe boundary, so a read
of BTRFS_STRIPE_LEN or more on raid56 becomes clones, while a 4 KiB
read fits in one stripe unit and is never split -- hence the size
dependence;btrfs_data_csum_ok()
(fs/btrfs/inode.c) zeroes a sector whose checksum fails before
returning false, so only the error signal was missing, not the data
hygiene;end_bbio_data_read() marks folios uptodate iff !bio->bi_status, so a
lost error also poisons the page cache -- which is why a later 4 KiB
read of a sector already covered by a large readahead returned zeros
without error too.Fix: load the saved status unconditionally. Committed as "btrfs:
propagate a split bio's error when the failing split completes last"
(Fixes: d48e1dea3931, CC stable 6.6+) on v6.18.41 and cherry-picked onto
topics/raid56-fixes.
Verified on a 6.18.41 kernel with the patch:
| read shape | before | after |
|---|---|---|
| buffered whole file | OK, 16 zero blocks | EIO |
pread() 1 MiB |
OK, 0 EIO | EIO |
pread() 64 KiB |
OK, 0 EIO | 1 chunk EIO, other 240 blocks correct |
pread() 4 KiB |
4 EIO, 12 zeros | all 16 damaged sectors EIO |
O_DIRECT 1 MiB |
EIO | EIO |
No zero-filled block is returned by any read shape, and the undamaged 240
blocks still read correctly. readpath-suite.sh is unchanged on the
fixed kernel: single/dup and raid1 EIO, repairable raid5 still repaired
silently and correctly, unrecoverable raid5/raid6 EIO.