raid56 stripe_alloc -- reference: harnesses, artifacts, results, and where they live

Last reverified 2026-09-15; review round 1 results added 2026-09-16 04:10. Three hosts: aigendu (the share host: kernel trees, harness repositories, forensics, Claude's memory), bhive (test VM, the FAIL fault model, kernel build master) and dhive (test VM, the CORRUPT fault model, benchmark host). Companion documents: TEST-CATALOG.md (every test campaign, what it proved), README.md (the Stage 0 replay method), the octopus test plan (~/share/raid56-wh-harness/6.18-octopus-test-plan.md) and the cover letters (covers/ on zygo-misc-next-zb64).

Conventions that apply everywhere below: push files to the VMs with rsync (atomic rename), never scp/cp; scripts arrive 0644, so chmod +x locally first; launch long VM payloads detached (setsid nohup ... &), never through systemd-run; confirm a launch with pgrep and by its next observable step; a bare sync on a VM stalls for minutes draining the virtiofs share -- use sync -f <mnt>; keep loop images on /mnt/wh or /mnt/linux, never /var/tmp or /; clean /mnt/wh leftovers before a suite (preflight needs 20-25 G free); after every VM reboot check /etc/cron.d and pgrep for stale launchers.

1. Hosts

Port triage: 22 dead but 23/24 open = guest wedged/booting, read the console; all three dead = host or LAN. A guest panic leaves 23 answering.

2. Kernel trees, lanes and refs (aigendu)

One git object store, several worktrees (git worktree list in ~/share/linux):

Branch naming per lane: LANE/topics/<topic> (patch series), LANE/testing (octopus merge of every topic on the lane base), zygo-LANE-zb64 (testing + Zygo's personal/config commits = what gets built). 6.18 lane refs have the 6.18/ prefix, misc-next misc-next/, master none. Priorities: 6.18 = production, misc-next = submission (covers/ lives there), master = formality and may drift (not given the raid6 fix).

Lane state 2026-09-17:

Other ref namespaces:

• refs/wip/<name>/<lane>-... -- candidates and work in progress (~95). Key ones: refs/wip/cand52/6.18-zb64 (efb22a1e554b = current zb64 + the raid6_pq.algo knob; bhive #212, dhive #73), refs/wip/cand49/6.18-zb64 (848031137e86, pre-review: lane + v2 + knob; bhive #209, dhive #69), refs/wip/cand50/cand51 (intermediate round-1 builds; cand50 BUGs at a runtime enable, cand51 leaves a stale open remainder), refs/wip/review-fixes/6.18-stripe-alloc (the six round-1 fix commits before landing), refs/wip/enospc/6.18-stripe-alloc-v2 (b73e951b0183, the chain as reviewed; its message-edited copy landed), refs/wip/cand48p/6.18-zb64 (positive control for the lib/raid6 bug), refs/wip/rbio-steal/*, refs/wip/scrub-ro/*, refs/wip/quiesce/* (already landed), refs/wip/enospc/misc-next-stripe-alloc (older, pre-v2 misc-next port -- the real port is still to do).
• refs/heads/6.18/reviews/stripe-alloc-v2 -- the chain as it was reviewed (25 commits after one fold, two message rewrites); superseded by the topic since the landing, kept for the review record.
• refs/backup/<tag>/<lane>/... -- every ref a lane script moved, before it moved (e.g. refs/backup/raid6x1-2026-09-14/{6.18,misc-next}/{testing,zb64}).
• refs/heads/<lane>/historic/<name> -- retired debug probes and experiments with a descriptive top commit, never built or submitted (6.18/historic/2026-09-13-cand47-debug, 6.18/historic/2026-09-13-enospc-chain-v1).
• wip/rmw-audit-instr -- the metadata RMW resubmit/logmix instrumentation (diagnostic only; ask before building).

Checkout-free lane tooling: scripts/lane/ in this repository (README there): lanetopic.sh (fast-forward a topic + re-make the octopus + replay zb64), lanerewrite.sh, laneretag.sh, lanefix.sh, laneaddtopic.sh (new one-patch topic), chainv2.sh (rebuild the ENOSPC chain with a drop/fold plan), rebuildtopics618.sh, enable-quiesce.py (blob-edit example), kloc.py (code-line deltas), blobedit.py (exact-string source edits on a ref -> commit, how the round-1 fixes were written). Every script backs refs up under refs/backup/ and prints a content oracle -- read it before trusting a result.

Upstream-ready patch files: ~/share/patches/2026-09-12-misc-next/{delalloc-fixes,raid56-rbio-fixes}/ and ~/share/patches/2026-09-14-raid6-x1/ (both path variants + SEND.txt).

3. Kernel builds on the VMs

• Trees: /mnt/linux/linux on both (bhive's is the build master; dhive's is a separate clone -- push branches with git push root@<vm>:/mnt/linux/linux <branch>, the VMs cannot fetch from the share). /mnt/linux/misc-next on both for the 7.3 lane. Candidate branches are named zb64-candNN (cand49 is checked out on both now). Tracked .config copies: /mnt/linux/config-*.
• Build + install: /root/build58.sh (bzImage -> /boot/vmlinuz, previous to /boot/vmlinuz.prev, then a one-shot grub-reboot with the FULL title "kernel-only vmlinuz (no initramfs, dm-mod.create root)" -- a short title silently boots stock Debian 6.12). Build numbers are per VM (#NNN in uname -v): bhive #212 = cand52, #211 = cand51, #210 = cand50, #209 = cand49, #208 = cand48p, #207 = cand48; dhive #73 = cand52 (#72 was the same tree, rebuilt after a transient fixdep failure), #71 = cand51, #70 = cand50, #69 = cand49, #68 = cand48, #66 = 6.18 lane (/boot/vmlinuz.lane618, own grub entry). panic=15 falls back to stock on a crash.
• Extra grub entries (both VMs): mitigations=off, maxcpus=1, cpuhp.parallel=0, and raid6_pq.algo=avx2x1 (forces the x1 SIMD gen_syndrome that the lib/raid6 bug lives in; appended directly to /boot/grub/grub.cfg because update-grub fails under the dm-mod.create root; backup grub.cfg.pre-raid6). The knob is the test-only commit lib/raid6: testing aid: raid6_pq.algo= on the cand branches, never on a topic.
• vmlinux copies for drgn/faddr2line: /mnt/linux/vmlinux-<build> (bhive: 207/208/209/abba/armA/armB/lockdep; dhive: 142/abba). VMLINUX= env for the drivers.
• Console capture (persistent, both VMs): ~/share/btrfs-forensics/console-logs/{bhive,dhive}-console.log on aigendu (serial port 23).

4. Harness repositories (aigendu -> VMs)

5. Test suites and drivers

Run on each VM as root; every driver writes a live report.md plus per-stage logs into a timestamped directory (.aborted suffix if it died early).

6. Results -- where the evidence is

Lane tip 2026-09-17: zygo-6.18.x-zb64 = 22886dbec236 (testing f0f47dd9eb1d), tree-identical to cand54 (refs/wip/cand54/6.18-zb64 = eb742bb0de84, bhive #215): the rebased lane + b72dc2da. Validated: the rebased lane (4ec82e513870, #214) passed the full acceptance on bhive 2026-09-17 (sanity/stage0/fsync PASS, octopus P1/P2/P3 13/13 each, no unexpected failure) and P1/P2 13/13 under KASAN on dhive #78; cand54 passed the directed stage (bg-gate property cases GREEN, balance-resume PASS). Pending candidate: refs/cand/parkref/{topic,testing,zb64} (zb64 5f3b8f082462, rebuilt on the post-knob lane) adds 1459c2b83afb "btrfs: stripe_alloc: hold a reference on parked rbios", the fix for the parked-rbio use-after-free root-caused in forensics/parked-rbio-uaf-rootcause-20260917.md. Built as bhive #217 with KASAN; the lane refs do not move until it is green (candidate made with scripts/lane/candtopic.sh, which is lanetopic.sh without the ref updates). bhive built the pre-re-message commit 8f67cef3e02c; the tree is byte-identical (5e6c05d6fbba), so that run validates the candidate as it stands. The history below is the round-1 candidate sequence.

Current candidate = cand53 (refs/wip/cand53/6.18-zb64 = 6.18 zb64 [lane incl. the landed chain and the SEVEN round-1 fixes] + the raid6_pq.algo= knob), bhive #213 / dhive #74. cand52 (2b724c29822c, bhive #212 / dhive #73) had six of the seven; the seventh (btrfs_free_open_stripe_runs teardown) came from a mount-fail assertion the tear run hit on cand52. cand49 (848031137e86, bhive #209 / dhive #69) was the pre-review candidate: lane + v2 + knob, booted with avx2x1 forced; cand48 = the same without the raid6 fix.

Older run directories are under /root/archive-runs/ on each VM; /root/*.log (204 on bhive, 113 on dhive) are queue, after-boot and stage logs named after their candidate (octopus-candNN*.log, rebase-validation-candNN.log, queue-*.log, after-boot-*.out). The launchers that produced them are versioned in vm-scripts.

7. Forensics and records (~/share/raid56-wh-harness/forensics/, in git since 2026-09-17)

Consolidated from ~/share/btrfs-forensics/ on 2026-09-17; only the bulk stays there, outside git: console-logs/{bhive,dhive}-console.log (live serial captures, GBs, read by the heartbeat monitor) and bulk/dhive-rbio-cache-oops-2026-09-04/dhive-boot-1-kernel.log (167 MB). The current test plan is ~/share/raid56-wh-harness/6.18-octopus-test-plan.md (rev 5); the rev-4 text is forensics/history/.

Hang captures: ~/share/raid56-hang-debug/captures/ (bhive carry-log-tail deadlock 2026-08-16 with TRIAGE.md, dedupe deadlock, variant B, enospc spiral, rmw-park lockup, dhive carry control, stock-6.12 minimal) plus the two reports at the top of that repo. On the VMs: /root/share/root-{bhive,dhive}/ (old /root forensics moved there 2026-08-29: folio captures, spin samples, bees logs), bhive:/mnt/linux/forensics-2026-09-12/ (saved cand32-39 journals, r6c3-cand48-crash-20260914/), dhive:/mnt/linux/forensics-archive/ (folio captures 2026-08-15..19), bhive:/root/br-stall-02{24,44}.txt (the one-off balance-reclaim stall stacks).

8. Open items (2026-09-16)

• Review round 1 done except the decision on the raid56-metadata log epoch: settle metadata runs at log commit (a log-class metadata run, like the per-inode data runs) or state the raid56-metadata fsync window as detectable loss. The measurement is complete (four passes, above): the loss is log-only, committed metadata is never harmed, no mixed stripe occurs, and ro,degraded,rescue=nologreplay recovers the committed set; either resolution is backed by the data, and the recommended raid1c3 metadata is unaffected.
• Port the landed 6.18 topic (72 commits) to misc-next (mnres.py resolutions in the job tmp), fold the roadmap in the cover letter, rebuild the misc-next octopus, revalidate on the 7.3 base.
• Full acceptance (octopus FAIL/CORRUPT, r6c3) on cand52: only the toggle, prealloc and directed sets have run on it so far.
• Upstream sends awaiting the go-ahead: lib/raid6 x1 fix (SEND.txt), the delalloc-fixes and raid56-rbio-fixes series, compress-fixes; the MICROCODE_DBG AP-bringup report; stable backport request for Qu's 8ceaad6cd6e7 (6.18).
• corrupt-churn residual (32-37 drops vs 360) unattributed; the stripe_alloc arm's earlier "scrub no errors" was the scrub-RO bug, now fixed.
• Run raid6 coverage once with raid6_pq.algo=sse2x1 as well (only avx2x1 has been forced so far).
• eb-leak A/B inconclusive (probes parked in historic/); nocow B2 questions (uniform close vs owner re-open, low-space forced close, dead-run free); claim-scan follow-ups (dirty-window rescan, bounded cursor); system chunks still uncovered; metadata stripe_park_stats ghost noise on reclaimed stripes (Zygo's call).
• Harness: r6c3-validation-latest symlink is stale; results/ in this repo was untracked until this commit; raid6-3dev-repro.sh and sys-chunk-rmw-probe.sh likewise.