rbio cache: measured cost of invalidate-on-extent-free (2026-08-03)

Question (Zygo): the proposed "drop cached rbios when their stripes' extents are freed" patch makes the next sub-stripe write into a reused stripe re-read the full stripe. How often does that actually happen, with true numbers, on stock behavior?

Method

Instrumented stock kernel (branch bench/rbio-cache-stats, stock fc02acf6ac0c + zygo: raid56: instrument rbio cache steal/stale statistics, kernel 7.2.0-rc5+ #2): at unpin, cached rbios overlapping freed extents are MARKED (identical walk and overlap test to the proposed patch) instead of dropped; steal_rbio() propagates the mark with the stolen pages; a sub-stripe RMW that skips its full-stripe read thanks to stale-marked pages counts as exactly one full-stripe read (data + P/Q -- rmw_read_wait_recover() reads everything) the patch would have added. The mark clears once counted, because the patched world would have re-read and re-cached fresh pages at that point, making descendants legitimate. Counters in /sys/fs/btrfs//rbio_cache_stats; benchmark rbio-cache-bench.sh.

Setup is a deliberate worst case for the patch: 5-dev raid5 for both data AND metadata (metadata RMW is the cache's sweet spot), small sub-stripe files, constant delete/overwrite churn so frees overlap cached stripes as often as possible. 90s phases.

Results

phase steals/s skips/s stale-skips/s patch cost stale share churn_sync 0.92 0.91 0.13 0.041 MB/s 14.6% churn_fsync 42.66 42.66 0.15 0.046 MB/s 0.3% odirect_rw 20.51 20.51 0.37 0.116 MB/s 1.8%

Cumulative: 7347 inserts, 6431 steals, 6430 RMW read-skips, 64 stale-enabled skips (1.0%), 20 MiB total would-be extra reads over ~5 minutes.

Internal consistency (true-numbers checks): rmw_partial - rmw_reads = rmw_skips exactly (7358 - 928 = 6430); steals == skips to within 1; stale_marked (928) vs stale steals (65): 93% of stale-marked entries die in the LRU without ever being stolen -- the patch's drops mostly hit entries that would never have been reused anyway.

Reading

• The cache itself is very effective here: 87% of sub-stripe RMWs skip their read. The patch preserves 99% of that benefit.
• The patch's worst-case cost is 0.04-0.12 MB/s of extra reads under free-heavy churn -- noise against the workload's own 0.3-0.7 MB/s writes, let alone array read bandwidth. In exchange it closes a silent parity-poisoning window (NOTE-b-eio-cluster.md) whose trigger (device corruption of freed space) healthy arrays hit a few times per year, but whose blast radius is unrecoverable multi-sector loss at reconstruction time.
• The fsync-heavy phase -- the cache's best case (42 skips/s) -- has the LOWEST stale share (0.3%): hot stripes are re-cached so fast that a freed-extent mark rarely survives to the next steal. Cost concentrates in the O_DIRECT rewrite phase (1.8%), still trivial in absolute terms.

Numbers ready for the upstream patch's cover text.