raid56 stripe_alloc — 6.18 octopus validation test plan (rev 5)

Status: the specification run-6.18-validation.sh implements, rewritten 2026-09-17 against the suite as it runs today. Rev 5 moved the plan from ~/share/btrfs-forensics/ into this repository (the rev 3 + rev 4 text is kept at forensics/history/6.18-octopus-test-plan-rev4.md), folded the §11 decisions and the rev 4 addenda into the sections they govern, and added what the earlier revisions did not describe: the profile-parametric geometry and the T6 sizing rule, the metadata headroom chunks, the declared expected outcomes as the spec, the driver stages around the octopus, and where raid6 and raid56-metadata coverage actually live (they are not octopus passes).

1. Objective

Validate that the raid56 stripe-exclusive-allocation write-hole series (the 6.18 backport octopus) meets the exit criteria below with the fault present for the whole run, under a total disk failure and under ongoing silent corruption of one drive, with no data loss and no crashes.

The octopus is the umbrella acceptance suite (T1–T13, three passes). It is one stage of the rebase revalidation driver (§11); crash consistency, fsync forensics, the directed regression tests, raid6 (§12) and raid56 metadata (§13) are covered by the other stages and drivers named there.

2. System under test

3. VM assignment and fault models

Both VMs run the same suite; only the injector and nodatacow-inclusion differ. The fault is present for every test; it is removed only by the replace step (T12), after which the offline check (T13) runs on a healed array.

4. Filesystem geometry

• 6 loop-backed members × 3 GiB (NDEV=6, DEV_MB=3072) plus one 3 GiB spare as the replace target (T12). mkfs.btrfs -K, mount nodiscard: discard would erase injected garbage and the pre-fill poison.
• Members are pre-poisoned (wh_poison_img) before mkfs so a dropped write can never masquerade as a zero sector.
• Profiles are knobs, the acceptance shape is fixed. The suite runs PROFILE_D=raid5 PROFILE_M=raid1. The driver is written for any pairing (the T6 sizing below counts raid6 at two parities and raid1c3/raid1c4 at their copy counts), but no raid6 or raid56-metadata octopus pass has been run: that coverage comes from the drivers in §12 and §13, and an octopus pass at those profiles is an optional addition, not part of the acceptance (see §17).
• Metadata headroom (meta_buffer). A single raid1 metadata chunk is the one thing this small geometry starves: an impossible T6 shrink or a replace that locks a group grinds relocation until the metadata reservation fails and the filesystem aborts in merge_reloc_roots (-28) — in stock mode too (2026-09-07), and Zygo's ruling (2026-09-08) is that metadata ENOSPC on a small shrink/replace is the geometry, not the allocator. So the suite buys the headroom instead of expecting the abort: +2 metadata chunks after setup (for T6) and +1 before replace (T12), via /sys/fs/btrfs/<uuid>/allocation/metadata/force_chunk_alloc (debug sysfs; META_BUFFER=0 disables it for A/B against the old shape).
• T6 shrink target is computed, never fixed. It is what the filesystem actually stores, spread over the members that can hold data: raw = data_used × NDEV/(NDEV−parity) + (metadata_used + system_used) × copies, then target = raw/live + 1024/live + T6_SHRINK_SLACK_MB (768) per member, floored at T6_SHRINK_MB (768), where live = NDEV (or NDEV−1 in the degraded pass, whose fault member is missing). The 1024/live term is the relocation destination chunk; the slack covers the reloc-root merge and the fill. History: a fixed 768 MiB target (rev 3.1) aborted every FAIL/P1 run from 2026-09-07 to 2026-09-12, and a stored-bytes target over all six members aborted P2 on 2026-09-13 (10.5 GiB raw over five live members) — hence live.
• A single-device fault is inside the redundancy of both profiles, so the correct expectation everywhere is full recovery, no application-visible loss.

5. Fault injectors

5.1 FAIL — dead disk via the dm error target

Each member is loop → dm-linear at mkfs. After mount and the baseline writes, one data member is killed live: dmsetup suspend / reload --table "0 <sectors> error" / resume. All subsequent I/O to it returns EIO — a dead disk, not a -o degraded remount.

5.2 CORRUPT — worst-case silent device via dm-flakey drop_writes

dm-flakey in permanent drop_writes mode over a /dev/urandom backing: a write returns success but is discarded, and every read returns the random backing — there is no window where a recently written block is readable. Every read of an affected stripe csum-fails on that column and rebuilds from parity; metadata falls back to its other raid1 mirror.

Rotation (ON by default, CORRUPT_ROTATE=1, ROTATE_SECS=1). A dm-linear in front of the random backing whose start offset moves every second makes the same block return different garbage over time: no lucky-consistent read, no stable content for reconstruction to accidentally match. If the worst case passes, the easier cases (static garbage, dd overwrite) pass too. The one risk it adds — a low-memory stall while a member is dm-suspended for the reload — is what T8's watchdog looks for.

(Periodic dd if=/dev/urandom over the raw member was rejected: slower, and it leaves a window where the just-written block reads back correct, so reconstruction is exercised only sometimes.)

6. Test suite

Execution order within a pass: baseline tree dc (the T12 verify set) → inject the fault → T1 T3 T4 T5 T6 T7 T9 T10 T11, then T2 (it scans every writelog, so it runs once they all exist), T8, then T12 (removes the fault) and T13. T8 is re-checked at the end of the pass so a WARN/abort during T12–T13 cannot slip through (it did once, 2026-09-13).

Oracle classes: correct / eio / missing / zero / wrong / short. wrong and zero are always fatal. A short/missing file is fatal unless its writer got a write(2)/fsync EIO or ENOSPC for that region — otherwise it is a zero/missing in disguise; the write phase records per-file outcomes so verify can tell. Within single-fault tolerance no write error is expected, so short/missing must be 0 except at the ENOSPC edge in T6.

7. Execution passes

PASSES_RUN="P1_online P2_degraded P3_stock", in that order, every loop:

1. P1 online failure, stripe_alloc (primary) — the live injectors of §5.
2. P2 degraded mount, stripe_alloc — the fault member is downed to error before the first mount and the filesystem mounted -o degraded (P2_STOCK=1 runs it with the stock allocator too, as an A/B).
3. P3 online failure, stock allocator — P1 with stripe_alloc off on the same kernel (which carries topics/raid56-fixes). It isolates stripe_alloc-specific regressions, and its passing shows the crash and corruption fixes carry the stock allocator through the same faults; a genuine stock FAIL is marked FAIL(exp).

P3b, positive control (done 2026-08-13, forensics/p3b/): the same suite on a pristine stable/linux-6.18.y build (v6.18.42, no topics). It proved the suite is sensitive to the bugs the topics fix: FAIL model — T6 and T12 fail; CORRUPT — T2 and T10 fail. Re-run only if the suite's discriminating power is in doubt. Pass 4 (stock + degraded) is skipped — no information over P3.

8. Execution model

1. Boot the SUT; record uname -r, tree sha, progs version in the report.
2. Detect-not-panic sysctls: hung_task_panic=0 softlockup_panic=0 panic_on_oops=0 panic_on_warn=0.
3. Arm the T8 watchdog; kernel messages between loops (the previous loop's unmount, this one's setup) are reported but charged to no test.
4. Per pass: build the filesystem, meta_buffer +2, baseline dc, inject the fault (or mount degraded), run the tests in §6 order, replace, offline check, tear down (dm devices, loops).
5. On any failure: record and CONTINUE to the next test.
6. Loop the whole suite for BUDGET_SECS — 24 h standalone, 4 h when the rebase driver calls it — or stop on the first unexpected failure (§10). CONTINUE_ON_FAIL=1 keeps looping for the budget when a known, separate WARN would otherwise cut an A/B soak short; the final status still reflects the failure.
7. Diagnosis afterwards: reproduce each recorded failure in isolation (minimal geometry, deterministic fault); the record goes under forensics/<what>-<date>/.

9. Report (regenerated live)

The table is regenerated after every test, so /root/6.18-validation-report.md is a live status map while the run proceeds (the driver copies it to its run directory as octopus-report.md):

# 6.18 octopus validation — <VM> (<fault>) — <date>  [loop N, elapsed HH:MM]
kernel: <uname>  branch: zygo-6.18.x-zb64 @ <sha>  progs: <ver>

| test | criterion                 | P1 online | P2 degraded | P3 stock | notes |
| T1   | read-after-write          | PASS      | PASS        | FAIL(exp)|       |
| T6   | ENOSPC no data loss       | PASS      | RUNNING     | pending  | enospc edge write_errors=.. shrunk=5/5 to 2524M |
...
Loop N: pass=.. fail=.. skip=..
RESULT_6.18[bhive/FAIL/P1]: pass=.. fail=.. skip=..

PASS / FAIL / FAIL(exp) / SKIP / RUNNING / pending per cell, with a note.

10. Declared expected outcomes — the spec

• P1 and P2 (stripe_alloc): all 13 rows PASS. A single-device fault is inside the redundancy, so the only acceptable app-visible error is the ENOSPC edge in T6.
• P3 (stock): FAIL(exp) allowed on any row; it never stops the loop.
• T9: SKIP under CORRUPT.
• T8 FAIL(exp) on older 6.18 builds only for the expedited-RCU-stall NMI backtrace at unmount: that was CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=20 ms from an oldconfig (lockdep zap_class in workqueue teardown), set to 0 on the lane since 2026-09-13; a kernel built since must not show it.
• Everything else is unexpected and stops the loop. A "known" failure is never promoted to expected: if the geometry causes it (the T6/T12 metadata ENOSPC of §4), the suite changes the geometry.

11. Where the octopus sits: the rebase revalidation driver

run-rebase-validation.sh (run on each VM after a rebased kernel boots) sequences the regression-relevant subset of TEST-CATALOG.md and writes a live report under /root/rebase-validation-<date>/:

Two VMs run the same driver under their two fault models; the host-side heartbeat monitor (scripts/vm-heartbeat-monitor.sh) reports loss of signal, taint changes, D-state hangs, stage changes and hourly progress, and reads the serial console captures in ~/share/btrfs-forensics/console-logs/ for the crash signature when a VM goes away.

12. raid6 coverage

The octopus acceptance runs raid5. raid6 is covered by run-r6c3-validation.sh — the recommended production pairing **raid6 data

• raid1c3 metadata** — across the three failure axes:

raid1c3 is what makes the raid6 drop axis honest: under raid1 metadata a two-device drop can take both copies of a metadata chunk and the case dies as "unmountable" instead of testing data recovery (the 2026-08-01 geometry sweep's raid6-over-raid1 rows are weaker than they look for this reason).

Results: all six stages PASS on both VMs 2026-08-28/29; 7/7 on both 2026-09-15 (matrix stock RED 72 rows with 27/27 controls, stripe_alloc GREEN 43/43 continuations). That run also found the lib/raid6 x1 bug: the sse2x1/avx2x1/avx512x1 gen_syndrome variants read pointers[-1] at three disks (the 3-wide raid6 chunks the cdev stage and uneven devices produce) — fixed as topics/raid6-fixes on both lanes, with a test-only raid6_pq.algo= knob and a grub entry that pins the x1 variant so the coverage runs on the code path that had the bug (REFERENCE.md §3), and the 3-device probes raid6-3dev-test.sh / raid6-3dev-parity.py as the positive control. bg-gate-test.sh also asserts the claim rule at raid6 (data + raid1, data + raid6 metadata).

An octopus pass at raid6 (PROFILE_D=raid6 PROFILE_M=raid1c3, or raid1) has never been run; the driver supports it (§4). It is listed as an optional addition in §17, not as acceptance.

13. raid56 metadata coverage

stripe_alloc applies to raid56 metadata automatically — there is no separate option or property (Zygo, 2026-08-27: if the metadata is raid56 the policy covers it; if it is raid1 it has no effect there). Tree blocks are allocated by whole stripes, metadata runs retire per transaction (the adjacent-generation canary went 45 → 0), and the trapped/claimable accounting, the reserve that keeps tree blocks landing, and reclaim of trapped groups all apply to metadata groups. SYSTEM chunks are the one raid56 shape still uncovered — the mount warns "does not cover raid56 system chunks", and the RMW audit shows they take real write holes (forensics/, 2026-08-27). Mixed block groups are refused, permanently (mount option and property alike).

Coverage lives in the directed stage and the round-1 review tests, not in an octopus pass:

Measured (2026-09-16, four passes: raid5/raid6 × uniform/late cuts, 162 armed cases): 0 committed-data damage, 0 torn stripe that held a committed tree block, 0 stripes mixing log and non-log tree blocks; 2 raid5 log-tear refusals, both recovered intact by mount -o ro,degraded,rescue=nologreplay (the degraded recovery — btrfs rescue zero-log needs a writable, all-devices mount and cannot open a degraded raid56). Conclusion: the window is a detectable, log-only loss; committed metadata is never harmed. Settling metadata runs at log commit, which would close the window, is queued after the other open issues (Zygo, 2026-09-17).

An octopus pass with raid56 metadata (PROFILE_M=raid5 or raid6) has never been run — optional addition, §17.

14. Instrumented modes

• lockdep is on in every acceptance build; sanity and the end of the octopus stage check debug_locks (lockdep disables itself on its first report or table exhaustion, and a PASS earned blind is not a PASS).
• KASAN: vm-scripts/dhive/build-kasan.sh rebuilds the running tree with KASAN_GENERIC/INLINE + STACKDEPOT (64-frame alloc/free stacks), and after-boot-kasan.sh loops sanity octopus with the stripe-park knobs collapsed by a 200 ms sysfs setter (stripe_park_timeout_ms=2, stripe_park_sync_timeout_ms=1, stripe_park_congestion=1; the knobs are per-filesystem, so a setter loop is needed while the suite mounts fresh filesystems) and stops on the first report. First use 2026-09-16/17: the parked-rbio use-after-free that wedged dhive #77 forty minutes into P1 CORRUPT (forensics/dhive-rmw-listdel-oops-20260916.txt).
• raid6_pq.algo= pins the raid6 syndrome variant (§12).

15. Prerequisites / checklist (per VM, before every launch)

• [ ] the intended kernel booted (uname -r, #build in the heartbeat), taint 0
• [ ] /root/harness rsynced from the canonical repo; vm-heartbeat service active; host monitor armed
• [ ] /mnt/wh ≥ 20 GiB free: remove leftover loop images from earlier tests (a full /mnt/wh produced a whole false-red run on 2026-09-16)
• [ ] no stale @reboot cron launcher (a double launch cost a run on 2026-09-11); confirm the launch by pgrep and the next observable step
• [ ] dm-mod (error, linear), dm-flakey (drop_writes), dm-log-writes, loop
• [ ] /root/xfstests/ltp/fsstress, compsize, setfattr
• [ ] 7 loop devices' worth of space (6 members + spare); debug sysfs allocation/metadata/force_chunk_alloc present (else meta_buffer skips and the T6/T12 geometry abort is back)
• [ ] panic_on_oops/warn off (the driver sets them)

16. Results to date

The evidence table, by run, is REFERENCE.md §6. In brief: FAIL model GREEN 13/13 (first 2026-08); CORRUPT 24 h soak 2026-08-13 (forensics/); P3b positive control 2026-08-13; the ENOSPC chain candidates cand48/49 FAIL 3 × 39/39 and CORRUPT 12/12/12 (2026-09-13/14); cand53 (the landed chain + seven round-1 fixes) CORRUPT all-green and FAIL at the known T6 geometry abort before the sizing fix (2026-09-16); the rebased 6.18 lane (v6.18.52) under validation on both VMs 2026-09-16/17, interrupted on dhive by the parked-rbio UAF (§14) — not a rebase regression.

17. Decisions and changelog

• Geometry 6 × 3 GiB + spare; T6 shrinks the live filesystem rather than adding small members (rev 3); target computed from stored bytes over the live members (rev 4, 2026-09-12; degraded fix 2026-09-13).
• meta_buffer metadata headroom instead of expecting the geometry abort (2026-09-08 ruling).
• Fault-active soak ~1 h; suite loops for the budget or until the first unexpected failure; Pass 4 skipped; CORRUPT rotation ON; report regenerated after each test (rev 3).
• Expected outcomes are the spec (§10); directed statuses 77 = SKIP, undeclared WARN = FAIL (rev 4).
• Optional, not yet run: octopus passes at PROFILE_D=raid6 (+ raid1c3) and at raid56 metadata (PROFILE_M=raid5|raid6). Both are one environment setting away; scheduling them is Zygo's call once the current fix queue (parked-rbio UAF, misc-next one-pass port, then metadata runs at log commit) is through.
• Rev 5 (2026-09-17): moved into the harness repo; §4, §10–§17 new; §11 decisions and §12 addenda folded in.