raid56 stripe_alloc — 6.18 octopus validation test plan (rev 3)

Status: DRAFT for review. §11 answers and rev-2 review notes folded in. When approved this becomes an execution script + fault-injector helpers; each run regenerates a pass/fail report that is a live map of what's done and what needs attention.

1. Objective

Validate that the raid56 stripe-exclusive-allocation write-hole series (6.18 backport octopus) meets the exit criteria below with the fault present for the whole run, under a total disk failure and under ongoing silent corruption of one drive, with no data loss and no crashes. 6.18 avoids unrelated 7.2-rc regressions.

2. System under test

item	value
kernel	`zygo-6.18.x-zb64` @ `47891c21c94f` (6.18.42 + `6.18/testing` octopus `1bd016ba68d9` + zb64 config; includes the adaptive-park patch)
btrfs-progs	v6.14 (`+ZSTD +LZO`)
allocator	mount `-o stripe_alloc` (primary pass); stock allocator is a later pass (§7)
profile	`mkfs.btrfs -d raid5 -m raid1 -K`; mount `-o nodiscard[,stripe_alloc]`
harness	`~/share/raid56-wh-harness` (canonical git repo; `bhive` is a git remote of it, push-to-deploy). `rolling-oracle.py` generator, `fsstress` at `/root/xfstests/ltp/fsstress`

Report records uname -r, branch commit, and progs version at run start.

Deploy: push 47891c21c94f to both VMs and build+install it. dhive's /root/harness has diverged from the canonical repo and must be reconciled before the run (bhive is already synced at harness 1107659).

3. VM assignment and fault models

Both VMs run the same suite; only the injector and nodatacow-inclusion differ. The fault is present for every test; it is removed only by the replace step (T12), after which the offline check (T13) runs on a healed array.

VM	fault	nodatacow tests	rationale
bhive = `FAIL`	one drive returns EIO on every read and write	included	a dead disk must not affect nodatacow correctness — data is still raid5-redundant
dhive = `CORRUPT`	one drive silently drops writes and returns random garbage on read, its garbage rotating so a block never reads the same twice	excluded	nodatacow has no checksum; silent corruption is undetectable there by design

4. Filesystem geometry

6 loop-backed devices, 3 GiB each, mkfs.btrfs -d raid5 -m raid1 -K, mounted -o nodiscard[,stripe_alloc]. -K/nodiscard are mandatory (discard would erase injected garbage and the pre-fill poison).
One extra spare device per VM as the replace target (T12).
Devices are pre-poisoned (wh_poison_img) before mkfs so a dropped write can never masquerade as a zero sector.
T6 (ENOSPC) does not add small devices; it shrinks the live filesystem with btrfs filesystem resize (§6).
A single-device fault is inside the redundancy of both profiles, so the correct expectation everywhere is full recovery, no application-visible loss.

5. Fault injectors

5.1 `FAIL` — dead disk via the dm `error` target

Stack each member loop → dm-linear at mkfs. After mount + initial writes, kill one data device live:

dmsetup suspend fail-devK
dmsetup reload  fail-devK --table "0 <sectors> error"
dmsetup resume  fail-devK

All subsequent I/O to fail-devK returns EIO — a dead disk, not a -o degraded remount. (dm error v1.7.0 confirmed present.)

5.2 `CORRUPT` — worst-case silent device via `dm-flakey drop_writes`

dm-flakey (v1.5.0, confirmed present) gives a device that accepts and discards every write and always returns garbage. Pre-fill a backing image with /dev/urandom, then front it with dm-flakey in permanent drop_writes mode:

dd if=/dev/urandom of=corruptK.img bs=1M ...
loopK=$(losetup -f --show corruptK.img)
dmsetup create corrupt-devK --table "0 <sectors> flakey $loopK 0 0 1 1 drop_writes"

Verified on the VM: a write returns success but the block re-reads unchanged (dropped), and reads return the random backing — there is no window where a recently-written block is readable. Every read of an affected stripe csum-fails on that column and rebuilds from parity; metadata falls back to its other raid1 mirror.

Rotation (ON by default). A dm-linear in front of a large random device whose start offset is rotated every second (dmsetup suspend/reload/resume) makes the same block address return different garbage over time — the true worst case: no lucky-consistent read, and no stable content for reconstruction to accidentally match. It is on by default because if the worst case passes, the easier (static-garbage, dd-overwrite) cases pass too. The one new risk it introduces is a low-memory deadlock while the dm device is suspended during the reload; T8's watchdog watches for exactly that (a stall with a member suspended).

Rejected alternative — periodic dd if=/dev/urandom onto the raw device. Slower (rewrites the device each cycle), and worse: between the good write and the next dd there is a window where the just-written block reads back correct, so reconstruction is only sometimes exercised and a bug that mishandles a corrupt column can hide. drop_writes never persists the good copy at all, so reconstruction is exercised deterministically on the first read. Not used.

6. Test suite

T1–T11 run with the fault active; T8 runs continuously. T12 removes the fault (replace); T13 checks the healed array.

Oracle classes: correct / eio / missing / zero / wrong / short. wrong and zero are always fatal. A short/missing file is fatal unless its writer got a write(2)/fsync EIO or ENOSPC for that region — otherwise it is a zero/missing in disguise; the write phase records per-file outcomes so verify can tell. Within single-fault tolerance no write error is expected, so short/missing must be 0 except at the ENOSPC edge in T6.

#	Criterion	Method	PASS
T1	read-after-write w/ flush	`oracle write` → `sync;drop_caches` → `verify --expect all-correct` (datacow)	`wrong=zero=eio=0`; `short=missing=0`
T2	no IO errors visible to apps	app return codes + oracle `eio` during T1/T5/T11	apps succeed; `eio=0` (internal csum/EIO must be recovered, not surfaced)
T3	scrub	`btrfs scrub start -Bd`; per-device status + `Error summary`; re-verify	scrub completes on every device, no early exit/abort, and no corruption visible to apps (post-scrub all-correct). Errors/uncorrectable on the faulted device are expected — target is completion + app-visible correctness, not zero errors
T4	balance	`btrfs balance start -d`; re-verify	completes with post-balance all-correct, or fails-with-a-reported-error-and-stops. FAIL = continues and relocates corrupted data (e.g. still RMWing under the fault)
T5	fsstress	`fsstress -d $MNT/stress -n <ops> -p 8` under the fault	no crash/hang; exits 0; no app-visible EIO
T6	ENOSPC, no data loss (after T4)	delete most of the tree (so the shrink can succeed), `btrfs filesystem resize <devid>:512M` on each device to make a small live fs, fill to ENOSPC with the oracle (+`enospc-dataloss.sh`) under the fault, then `resize <devid>:max` back	every acked write verifies; ENOSPC at `write(2)`; `short`/`missing` only at the fill edge where the writer got ENOSPC — never silent mid-file loss. Also: the live shrink/grow succeed (relocation, already confirmed by T4)
T7	compressed (zstd)	mount/subvol `compress=zstd`; `oracle --compressible` (4-bit-alphabet data, verified ~54% ratio) write/verify under the fault	all-correct and compression engaged (`compsize` shows ~50%), so the compress→raid56→reconstruct→decompress path really runs
T8	kernel health (continuous)	dmesg + D-state watchdog for the whole run	zero `oops/BUG/RIP/hung task/softlockup`; zero write-hole-invariant WARNs + no unexpected `dropping unwritten extent`/csum escalation; zero KASAN/`refcount_t`/kmemleak; no task stuck in D (incl. no stall with a member dm-suspended during rotation)
T9	nodatacow under disk failure (FAIL only)	`chattr +C` subvol; write → fault active → drop_caches → verify	nodatacow data all-correct
T10	fsync durability across the fault	write+`fsync` files under the fault → drop_caches → verify each	fsync'd bytes reconstruct; `wrong=zero=short=0`
T11	fsync-heavy log-tree stress	parallel threads: tight `fsync` loops plus `rename A over B` (both drive the log tree), under the fault	no crash/hang; renamed/committed state consistent on readback; no lost/torn fsync'd data
T12	device replace / rebuild (both VMs)	`btrfs replace start` the faulted device with the spare → wait → verify	replace completes; reconstructed data all-correct. Removes the fault for T13
T13	offline consistency, post-replace	after T12: unmount → `btrfs check --readonly`	no errors. Run after replace so every remaining/rebuilt drive is healthy — "the disks are healthy again, did the metadata survive?" — not `btrfs check` against a live fault

7. Execution passes

The suite repeats over allocator × fault-presentation, in priority order:

Pass 1 — online failure, stripe_alloc (primary): the live injectors of §5.
Pass 2 — degraded mount, stripe_alloc: the faulting device withheld, mount -o degraded from the start (harness dm-log-writes + withheld-member).
Pass 3 — online failure, stock allocator (patched kernel): Pass 1 with stripe_alloc off, on the same SUT (v6.18.42 + topics/raid56-fixes + the stripe series). This is the raid56-fixes control: it isolates stripe_alloc-specific regressions, and — because the SUT carries the raid56 fixes — its passing is informative in its own right: it shows the crash and corruption fixes carry the stock allocator through the same faults. (Any genuine stock FAIL is still marked FAIL(exp).) 3b. Pass 3b — completely stock kernel, no patches (positive control): the P3_stock suite on a pristine stable/linux-6.18.y build (v6.18.42, no zb64 topics; stripe_alloc does not exist there). It proves the suite is actually sensitive to the bugs topics/raid56-fixes fixes, rather than passing because the tests are too weak. Expected to FAIL — T1 (degraded-read-returns-zeros), T3 (scrub), T4 (balance corruption) — and/or crash (T8: the raid56 rbio use-after-free under write load), so it is a deliberately SHORT run (soaks trimmed or skipped, moderate FILES, one loop): long enough to trip the bugs, and a crash ends it early. Run on both fault models (a separately-built+booted pristine kernel per VM). A green P3b would be a warning that the suite is not catching the bugs and must be sharpened. Distinguished from P3 in the report by the kernel: line (uname -r lacks the zb64 + local-version suffix).

Pass 4 (stock + degraded) is skipped — no additional information over Pass 3.

8. Execution model

Boot the SUT; record uname -r, branch commit, progs version.
Detect-not-panic sysctls: hung_task_panic=0, softlockup_panic=0, panic_on_oops=0, panic_on_warn=0.
Start the T8 watchdog for the whole run.
Per pass: build fs, pre-fill, inject the fault (or mount degraded), run T1–T11 (fault-active tests such as T5/T11 soak for ~1 hour), then T12 replace, then T13 check; tear down.
On any failure: record and CONTINUE to the next test.
Loop the whole suite for 24 hours, or stop early on the first unexpected failure (an unexpected FAIL = any FAIL that is not an expected stock-allocator FAIL(exp)). The 24 h soak catches intermittents.
Diagnosis pass at the end: reproduce each recorded failure in isolation (minimal geometry, deterministic fault), capture repro command, dmesg, btrfs check, oracle detail lines, root-cause hypothesis.

9. Report (regenerated live)

The report table is regenerated after every test/stage, so the markdown file is a live status map while the run proceeds — a reader sees exactly which tests have passed, which failed, and what is still pending, without waiting for the run to finish.

# 6.18 octopus validation — <VM> (<fault>) — <date>  [loop N, elapsed HH:MM]
kernel: <uname>  branch: zygo-6.18.x-zb64 @ <sha>  progs: <ver>

| test | criterion                 | P1 online | P2 degraded | P3 stock | notes |
|------|---------------------------|-----------|-------------|----------|-------|
| T1   | read-after-write          | PASS      | PASS        | FAIL(exp)|       |
| T4   | balance no silent corrupt | RUNNING   | pending     | pending  |       |
| ...  |                           |           |             |          |       |

Loop N: pass=.. fail=.. skip=..   (cumulative over 24h: ..)
RESULT_6.18[bhive/FAIL/P1]: pass=.. fail=.. skip=..

## Failures needing attention
### D1 — T4 balance (P1): relocated 2 extents read as garbage
  repro: <cmd> ; dmesg: <excerpt> ; hypothesis: <...>

PASS/FAIL/SKIP/RUNNING/pending per cell; SKIP explained (nodatacow on CORRUPT); expected stock FAILs marked FAIL(exp).

10. Prerequisites / checklist

[ ] 47891c21c94f pushed to both VMs; built, installed, booted.
[ ] dhive /root/harness reconciled with canonical ~/share/raid56-wh-harness (bhive already synced).
[ ] dm-mod(error), dm-flakey(drop_writes), dm-linear, dm-log-writes, loop present (error/flakey/linear confirmed).
[x] Oracle --compressible mode (4-bit alphabet, ~54% via compsize) landed + round-trip verified; write-outcome recording for the short/missing↔EIO correlation still to add.
[ ] compsize, /root/xfstests/ltp/fsstress present (both confirmed on bhive).
[ ] 7 loop devices' worth of space per VM (6 members + 1 spare).
[ ] panic_on_oops/warn off; watchdog armed.

11. Decisions (from review)

Geometry 6 × 3 GiB (+1 spare); T6 shrinks the live fs instead of using small devices.
Fault-active soak ~1 hour; whole suite loops for 24 h or until the first unexpected failure.
Pass 4 skipped.
CORRUPT rotation ON by default (worst case; watch for the suspended-device low-memory deadlock).
Report regenerated after each stage (live status).

12. Rev 4 addenda (2026-09-12)

Expected outcomes restated as the spec: P1 and P2 (stripe_alloc) — all 13 rows PASS; P3 (stock) — FAIL(exp) allowed; T9 SKIP under CORRUPT. Everything else is unexpected and stops the loop. A "known" failure is never made expected: if the geometry causes it (T6/T12 metadata ENOSPC), the suite changes the geometry.
T6 shrink target is computed from what the filesystem stores (raw data at its parity ratio + metadata at its copy count, spread over the members, plus T6_SHRINK_SLACK_MB=768 per member; T6_SHRINK_MB=768 is the floor). An impossible shrink grinds relocation until the one raid1 metadata chunk starves and the fs aborts in merge_reloc_roots — in stock mode too (2026-09-07 control) — so the test must never ask for one. The fixed 768M target (rev 3.1, 2026-09-09) was never run under FAIL/P1 until 2026-09-12, when it still aborted.
Directed stage: scripts exit 77 = not applicable on this kernel (SKIPPED, not FAIL); kernel WARN/BUG during a script fails the stage unless declared in directed-expected-warns.txt.