raid56 stripe_alloc — test catalog and revalidation guide

Everything we have run against the stripe-exclusive-allocation (stripe_alloc) series and its supporting fixes, from single-purpose forensic probes to multi-day soak tests, with where each test lives and what it proved. Companion driver: run-rebase-validation.sh (this directory) re-runs the regression-relevant subset after a rebase onto a new upstream revision.

Environments: two test VMs (bhive, dhive), root ssh, kernels built in /mnt/linux/linux on the VM (bhive's tree is the build master), SUT booted via grub-reboot with the full menuentry title ("kernel-only vmlinuz (no initramfs, dm-mod.create root)"). This harness repo is canonical and is deployed to the VMs by git push or rsync; ~/share/raid56-hang-debug/ (the hang/soak kit, not in git) is deployed with rsync to /root/raid56-hang-debug. Push files to VMs with rsync only (atomic rename), never scp/cp.

1. Crash consistency: logged-write replay (Stage 0)

The foundation: every member of a raid5 array is a dm-log-writes target, so the write-hole can be synthesized at replay time instead of by crash timing. run-workload.sh runs committed phases (create/delete/fsync mix) with per-member log marks; replay-matrix.sh replays cut × torn-tail × dropped-device combinations, mounts degraded, and verify.py compares every committed file against a sha256 manifest. See README.md for the full method and expectations table.

• Scripts: setup-replay-log.sh, run-workload.sh, replay-matrix.sh, verify.py
• Crash models (report BOTH, they answer different questions):
  • entries — per-member cut at arbitrary log entries; harsher than real hardware (can tear inside a FUA barrier) but exhaustive.
  • barrier — cuts only at flush boundaries; flush-consistent and honest about what hardware can produce, lower hit rate.
• Results: write hole demonstrated RED on the stock kernel (exit criterion for trusting the harness); GREEN with stripe_alloc. Control cases (tail=any/drop=none, tail=clean/drop=any) clean, which validates the harness itself on every run.
• run-r6c3-validation.sh — coverage driver for the recommended production pairing, raid6 data + raid1c3 metadata: both matrix arms (stock RED / stripe_alloc GREEN) with 2-device drops and degraded-continue, the strict fsync-window matrix under the barrier model, and the corrupt-device family (within- and beyond-tolerance). raid1c3 is load-bearing here: raid1 metadata tolerates one lost device, so under the raid6 drop axis (two devices) a -m raid1 case can lose both copies of a metadata chunk and die as "unmountable" instead of testing data recovery — which quietly weakened the old geometry sweep's raid6 rows. First full run 2026-08-28, bhive #151 + dhive #37 (6.18 zb64), all six stages PASS on both hosts: stock RED (write hole observed, 27/27 controls clean, 72 rows) / stripe_alloc GREEN (0 damage, 43/43 degraded continuations intact vs stock's 12 incomplete); strict fsync-window PASS; recovery 30/30 (detected+recovered+repaired, plus corrupt+missing combos); cdev and cstripe stock-red/stripe_alloc-green (cstripe: 540 dropped writes on stock, 0 on stripe_alloc, identical fixture). Run cost five harness defects, all fixed: CSV drop-field comma shift (also invalidates historical raid6 sweep summaries), scrub "no errors found" parse, WH_NDEV default preemption in cstripe, punch-frees-nothing (bookend extents), and stripe placement assumptions replaced by chunk-tree math. The known stripe_open_remainder WARN fired once on bhive (-256K, first raid6 datapoint).

2. fsync correctness and torn-log forensics

What survives a crash+degraded mount when the fsync log tree itself is torn.

• Scripts: fsync-case-csum.sh, fsync-case-forensic.sh, fsync-csum-coverage.sh, fsync-block-probe.sh, setup-replay-log.sh
• Findings:
  • Torn-log replay endpoint is raid1-mirror-dependent (which mirror of the log tree survives decides how far replay gets) — documented forensics, not a stripe_alloc defect.
  • fsync-csum-coverage.sh: a torn log can replay a file extent whose csum items never landed, leaving the range unverifiable — the path by which reconstructed garbage reaches userspace without EIO.
  • Suspected upstream read-path bug: zeros returned instead of EIO on csum-failed sectors in degraded raid56 reads (NOTE-raid56-degraded-read-silent-zeros.md).

3. Directed write-hole and parity surgery

Deterministic single-stripe demonstrations replacing statistics with surgery.

• directed-writehole.sh — minimal 3-device raid5 geometry, named-column surgery with the raid5 rotation computed explicitly.
• scripts/two-drive-writehole.sh, scripts/torn-parity-repro.sh — variants.
• Level-2 (parity-dropped) write hole: latent variant, EIO for csummed victims and silent without csums; confined by stripe_alloc.
• NOTE-directed-writehole-parity.md records the analysis.

4. Corruption-under-load (the "b" EIO cluster and friends)

• corrupt-churn-test.sh — cross-commit sub-stripe RMW on a stripe already holding 2 unrecoverable corrupt blocks drops the new write; shows stripe_alloc prevents it. Status: A/B REOPENED — the earlier csum_failures_seen=0 dismissal was measured while RMW-read verification was broken by an upstream 6.18-rc bug (fixed in 6.18.44); redo on the new base with content verification for silent parity poisoning.
• corrupt-devices-test.sh, corrupt-stripe-test.sh — device- and stripe-scoped variants. Both (and corrupt-churn-test.sh) take a C?_META_PROFILE knob (default raid1); corrupt-devices-test.sh sizes its large-device count from meta_copies() so raid1c3 metadata still lands wholly on unwiped devices.
• corrupt-recovery-test.sh — the within-tolerance complement: corrupt up to parity whole devices (raid6 data + raid1c3 metadata by default) and require detected recovery — intact read-back, csum counters visibly moved (an all-zero detector fails the case as blind), scrub uncorrectable=0, cache-cold post-scrub re-read clean. Includes a one-corrupt-plus-one-missing combo family (recovery+detection only; scrub semantics on a degraded array are murkier).
• Stale rbio-cache parity: cache steal baked stale memory into claimed-stripe parity — root cause of the "b" EIO cluster; fixed (#85 pad-before-cache). Latent upstream-relevant window exists in stock.
• rolling-failure-test.sh + rolling-oracle.py — rolling device failure under a generated workload with an oracle manifest.

5. Allocator-specific regression tests

Each of these encodes a bug the series had and must not regrow:

• Reservation/ENOSPC accounting (#85): pessimistic reservation margin + per-add fragment counting; stripe-pessimism-measure.sh quantifies the claimable-only gate's refused space; also the stock 256K alignment fact (find_free_space full-stripe alignment since 53b381b3abeb).
• Entry-spanning ENOSPC (#115): v2 grid-walk slow path; fill-edge-debug.sh exercised the fill edge (raid6 clean, 0 drops).
• raid6/raid5 balance zero-fill: grow-frontier allocations never set liveness bits so pad zeroed cross-stripe spill; fix 1cba2c546a14; re-validate on rebase (was pending as #116; rebaseline116.sh, run116*.sh).
• Degraded RMW spurious EIO (upstream since 6.2): verify_one_sector skips the csum_bitmap test; one-line fix; stripe_alloc immune (NOTE-raid56-degraded-rmw-spurious-eio.md).
• RMW provenance audits: why-rmw.sh, why-rmw-aligned.sh, why-rmw-alloc.sh, why-rmw-decline.sh, why-rmw-ext.sh — attribute every RMW; frontier-stripe partial writes declining padding explain ~95% (goes away with stripe-aligned file sizes). RMW write-hole audit proves from disk which metadata RMWs are real holes (stripe_meta residual = 0 committed blocks).
• ENOSPC behavior: enospc-dataloss.sh, enospc-umount.sh; admission gating rollout is staged separately (write ENOSPC ⟺ statfs avail==0).
• balance-reclaim-test.sh, bg-gate-test.sh — mixed-bg commit deadlock fix (gate) and reclaim behavior.
• nocow-strand-test.sh — whole stripes stranded behind persistent nocow runs (2026-09-11): freed stripes inside a cached inode's private NOCOW run were counted claimable but refused to every claim, so the gate admitted writes that writeback dropped (~100k/loop under fsstress, stock 0). Fixes: the scan counts them trapped (bytes_stripe_stranded), private NOCOW runs only exist while stripe_alloc_allow_rmw lets the kind be written in place; stripe_run_stats counters prove the paths ran. Default policy must strand nothing; the waived policy must strand, refuse at write() rather than drop, and recover the space when the inodes are evicted.
• nocow-ro-drain-test.sh — pending NOCOW writes vs a block group going read-only (2026-09-11, upstream-relevant, not stripe_alloc specific): the write() decides NOCOW without a reservation, scrub/balance flip the group RO without flushing, the writeback falls back to an un-admitted COW and drops the data at ENOSPC. Fix drains the group first; the test scrubs with 16M dirty in a nodatacow file, fills to ENOSPC with fsync'd COW files, then syncs: content, in-place extents, 0 cow failures, nocow_ro_drains counter moved. Plain and stripe_alloc+waiver variants.
• legacy-conversion-repro.sh — enabling stripe_alloc on pre-existing legacy raid56 data trips a (now rate-limited) self-healing WARN.
• punch-hole-warn-repro.sh, stripe-claimable-warn-repro.sh, stripe-warn-sweep.sh — WARN reproducers.
• Metadata read-path verification: verify_one_eb() (topics/raid56-fixes) — reconstructed metadata was never verified upstream; read path only.
• Benchmarks (not pass/fail): rbio-cache-bench.sh, stripe-meta-bench.sh, profile-claim-test.sh (claim-scan CPU: 1447µs→0.8µs avg claim, 48→218 files/s after the per-bg negative cache, A/B-proven).

6. Octopus validation suite (T1–T13)

run-6.18-validation.sh — the umbrella acceptance suite for the 6.18 backport octopus: 13 tests (read-after-write, no-visible-IO-errors, scrub, balance, fsstress, ENOSPC, compression, kernel health, nodatacow, fsync durability, fsync-heavy log tree, device replace/rebuild, offline check) run under a fault present for the whole run, across three passes (online stripe_alloc, degraded stripe_alloc, stock allocator), looping for a wall-clock budget. Roles: bhive = FAIL (dm-error member, nodatacow included), dhive = CORRUPT (silent drop + rotating garbage, nodatacow excluded). Plan: ~/share/raid56-wh-harness/6.18-octopus-test-plan.md.

• Results: FAIL model GREEN 13/13 (first run; the dry-run failures were 6 harness bugs). CORRUPT 24h budget loop done 2026-08-13 (forensics/dhive-CORRUPT-24h-final-report.md); later runs, incl. the ENOSPC-chain candidates and the rebased-lane validations, are tabulated in REFERENCE.md §6. The plan (rev 5, 2026-09-17) also records what the octopus does NOT cover — raid6 (r6c3 driver) and raid56 metadata (round-1 directed tests + the meta-fsync tear matrix) — and the optional raid6 / raid56-metadata octopus passes that have never been run.

Expected outcomes (the spec; see the plan §4, §7, §8). Every stripe_alloc pass (P1 online, P2 degraded) is expected to PASS all 13 rows: a single device fault is inside the redundancy, so the only acceptable app-visible error is the ENOSPC edge in T6. P3 (stock allocator) may fail and is marked FAIL(exp). T9 is SKIP under CORRUPT (nodatacow loss is undetectable there). Anything else is an unexpected failure and stops the loop. Known shapes that are NOT allocator bugs: T6/T12 metadata ENOSPC on this small geometry (Zygo 2026-09-08) — the suite must keep them from happening (meta_buffer chunks; T6 sizes its shrink from what is stored, 2026-09-12) rather than expect them. History: FAIL/P1 T6 aborted the fs in merge_reloc_roots (-28) from 2026-09-07 (cand22 fill change) through 2026-09-12 — the 09-09 fixed 768M target was never run under FAIL/P1 until the misc-next lane run of 2026-09-12 showed it still impossible.

Directed stage statuses. A script exits 77 when the feature it tests is absent from the kernel (e.g. nocow-strand-test.sh needs the wip stranded-stripe counters): reported as SKIPPED, not a failure. A kernel WARNING/BUG/Oops seen while a script runs fails the stage unless the script declares it in directed-expected-warns.txt (script|pattern|why) — only the two legacy-layout repros declare the rate-limited "incremental stripe_claimable above scanned" WARN, which is what they measure.

7. Deadlock hunts and soak tests (~/share/raid56-hang-debug/)

repro-folio-hang.sh — the workload rig used for every hang: 8 LVs (one removed → degraded), optional dm-crypt under btrfs (CRYPT=1, prod-shaped kcryptd bounce-page stack), mount degraded,compress=zstd:3,flushoncommit,stripe_alloc,discard=async; workers: churn writers, deduper, snapshot loop, balance loop, mass rm; heartbeat loop watches commit progress; ENOSPC aborts recycle the fs and continue; a genuine stall trips a capture (SysRq w/l/t, /proc stacks, ftrace ring, drgn probes folio_probe.py/rbio_walk.py). Quiet "light captures" fire during any >90s commit without stopping the hunt. Env: CRYPT=1 MAX=<secs> VMLINUX=<matching vmlinux>. Real log is /root/folio-repro.log (the script re-execs into it).

Bugs found, fixed, and soak-verified with this rig:

• retire⟷finish-ordered deadlock (2026-08-17 prod hang): fixed both sides; verified on dhive #142 — ffdefer kprobe on the defer path counted continuous hits (254/10min at peak) across days of clean soak.
• carry/writeback folio deadlock: noflush soft-skip fix; drgn-proven live before the fix, clean soak after.
• flushoncommit+dedupe deadlock: part (a) landed (topics/reflink-deadlock); variant B remains (pre-existing delalloc under the dedupe extent lock; part (b) delalloc pre-check drafted, never shipped) — open item, lead candidate for the original production hang.
• Commit-stall family (SOLVED, benign): 60–233s self-resolving commit stalls on BOTH kernels and BOTH crypt configs are a giant delayed-ref backlog ground synchronously (__btrfs_run_delayed_refs → __btrfs_free_extent b-tree grind), NMI-proven via 31 SysRq-L spin samples; the grinder rotates (committer via start_dirty_block_groups, or an open-handle sync CLI / preempt-reclaim kworker = the extwriter pinning the committer at transaction.c:2389). Generic upstream delayed-ref-debt pathology, NOT a stripe_alloc bug. Trip thresholds are set above its tail (ccm >360s, frozen ≥40 heartbeats) so soaks don't trip on it.
• min-dedupe-deadlock.sh — minimal dedupe deadlock reproducer.

Soak verdict as of 2026-08-19: 48+ hours with no new deadlock on either VM or the production canary. Monitoring pattern: 12h watcher windows polling commit progress, WARN classification (known-benign: WARNING: CPU.*__btrfs_free_extent = ENOSPC abort), recycle count, light captures, ffdefer counter; poll hourly and check D-state/progress, not just liveness.

8. Known-benign signatures (do not re-diagnose)

• Delayed-ref commit stalls up to ~233s, self-resolving (see §7).
• __btrfs_free_extent WARN = ENOSPC transaction abort at fill edge (rate-limited, expected during fill-to-ENOSPC hunts).
• Bare sync(2) on a VM stalls minutes draining the virtiofs host share — use sync -f "$MNT".
• freeze/thaw while a device is failed forces whole-fs RO (intended device-change guard; keep --nolockfs).
• fsstress/xfstests live in ~/share/xfstests (VM copies under /root/xfstests); setup-replay-log.sh builds dm-log-writes tooling from it.

9. Revalidation on a new upstream base

Run ./run-rebase-validation.sh on each VM (see its header). Order: sanity → Stage 0 replay matrix (both crash models) → fsync suite → directed/regression set → octopus suite → soak. Deploy first: build the rebased SUT on bhive's /mnt/linux/linux, install on both VMs (grub full title), rsync this repo and raid56-hang-debug/ to /root/, and keep a matching vmlinux path for drgn (VMLINUX= env, honored since the 2026-08-19 fix).