speedtest-hd/CLAUDE.md

speedtest-hd

Single-file Bash CLI (speedtest-hd.sh) that benchmarks any mounted path (HDD/SSD/NVMe, local or NFS/ZFS) using fio, presented as a CrystalDiskMark-style ASCII table. Falls back to dd when fio is absent. Goal: a quick, generic, "point it at a mount" disk tester — not a ZFS/NVMe-specific tool.

Usage

./speedtest-hd.sh <path> [--fio|--dd|--slog] [flags]

• Default: fio if installed, else dd. --fio forces CDM mode, --slog forces sync-latency mode.
• Flags: --engine=io_uring|libaio|posixaio|sync (default auto), --direct/--buffered (default auto-probe), --runtime=SEC (default 5), --size=SIZE (default 1g), --verbose (also dumps raw fio output to stderr).

Modes & layout

• CDM mode (cdm_speedtest): runs CrystalDiskMark's default tests, each read+write, prints a 5-col table (Read/Write MB/s, Read/Write IOPS). Tests: SEQ1M Q8T1, SEQ1M Q1T1, RND4K Q32T1, RND4K Q32T16. Q = --iodepth, T = --numjobs.
• SLOG mode (slog_speedtest): 4K synchronous randwrite sweep at T1/T4/T8/T16 to profile ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 commit latency. Requires fio + python3 (JSON percentile parsing).
• dd mode (dd_speedtest): legacy cached/uncached read/write fallback.

Key implementation notes

• detect_io_settings auto-picks the engine (io_uring → libaio → posixaio → sync) and probes whether O_DIRECT works, via tiny throwaway fio_probe jobs. Falls back to buffered with a warning if O_DIRECT is rejected (older OpenZFS <2.3, some NFS). libaio is only truly async with --direct=1 — that's why io_uring is preferred. O_DIRECT bypasses the page cache so we measure the device, not RAM (buffered results, esp. reads, can reflect ARC/page cache).
• One shared test file <path>/speedtest-hd.bench is reused across all runs (laid out once), removed at the end. Footprint = --size (default 1G), matching the startup notice.
• run_fio (CDM) returns "MB/s IOPS"; run_fio_sync (SLOG) forces --ioengine=psync --sync=1 (O_SYNC) so every write is a ZIL commit regardless of dataset sync property, and parses fio JSON to aggregate across jobs (sum IOPS/bw, avg p50, worst-case p99). CDM mode parses fio's text output: fio_bw_mbps takes the parenthetical SI MB/s (matches CDM's decimal MB/s, normalizes kB/MB/GB); fio_iops expands k/M suffixes.
• Write tests append --end_fsync=1 so cached writes can't inflate numbers.
• ASCII tables: cell formats and dash-segment widths must stay in sync (tbl_* = 18/16 dashes, slog_* = 18/14). Verify alignment after editing.

History / decisions

• Originated from the Ars Technica fio guide. Original 4K test used numjobs=1 iodepth=1, which measures single-op latency, not throughput — ~12 MB/s on fast NVMe is correct for QD1, not a bug. Refactored toward parallel/deep-queue tests to show real device capability, then fully reshaped into the CrystalDiskMark profile above. --simple flag was removed; the table is now the only fio output.
• Note: the current RND4K rows are Q32T1 + Q32T16. CrystalDiskMark's actual latest default profile is Q32T16 + Q1T1 (single-queue random is the meaningful low-end number). If aligning strictly to CDM, the Q32T1 row should become Q1T1 (iodepth=1 numjobs=1).

SLOG performance context (why --slog exists)

Built to diagnose TrueNAS SCALE box linvault1 (Dell R630, Xeon E5-2680 v3; pool nvme-ultra-r10 = 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset vm-root sync=always). Poor sync writes were CPU power management, not the SLOG:

• Fix (biggest last): Dell BIOS profile DAPC → Performance (~2×); cstate kernel args; and the big one — CPU governor performance (was intel_cpufreq+schedutil, which parked cores at 1.2 GHz because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post Init: echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor.
• Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor).
• Diagnostic: zpool iostat -vl <pool> 1 during fio showed the Optane logs vdev at ~90µs disk_wait — proving the SLOG was fine and latency was upstream (CPU).
• Healthy Optane SLOG single-stream (T1) target: ~15–25k IOPS, p50 ~40–65µs. Much higher usually = C-states / PCIe ASPM / BIOS power profile throttling.