speedtest-hd

speedtest-hd.py — a single-file Python 3 CLI that benchmarks any mounted path (HDD/SSD/NVMe, local or NFS/ZFS) using fio, presented as a CrystalDiskMark-style ASCII table. Falls back to dd when fio is absent. Goal: a quick, generic, "point it at a mount" disk tester — not a ZFS/NVMe-specific tool. README.md is kept in sync with the CLI.

Usage

./speedtest-hd.py <path> [--fio|--dd|--slog] [flags]      # or: python3 speedtest-hd.py ...

Default mode: fio if installed (→ cdm), else dd. --fio/--dd/--slog force a mode (mutually exclusive).
Flags: --engine {io_uring,libaio,posixaio,sync} (default auto), --direct/--buffered (default auto-probe), --runtime SEC (default 5), --size SIZE (default 1g), --verbose (dump raw fio output to stderr), -y/--yes (skip confirm prompt). argparse → both --flag value and --flag=value work.

Requirements

python3 (3.7+) — required for everything; uses only the standard library (argparse, json, dataclasses, statistics). No third-party packages.
fio — recommended; without it, auto mode falls back to dd. --fio/--slog hard-require it.
sudo — fio is run via sudo (for O_DIRECT + device-cache flush).

Modes / profiles (functions)

cdm (cdm_profile): CrystalDiskMark's default tests, each read+write, 5-col table (Read/Write MB/s, Read/Write IOPS). Tests defined as data in CDM_TESTS (CrystalDiskMark's default profile, in its display order): SEQ1M Q8T1, SEQ1M Q1T1, RND4K Q32T16, RND4K Q1T1. Q=iodepth, T=numjobs. Write runs use end_fsync.
slog (slog_profile): 4K synchronous randwrite sweep at T1/T4/T8/T16 (SLOG_THREADS) to profile ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 commit latency. Forces --ioengine=psync --sync=1 (O_SYNC) regardless of dataset sync property.
dd (dd_profile): dependency-free fallback; sequential write + uncached read + cached(RAM) read, timed in Python.

Key implementation

detect_io_settings auto-picks engine (ENGINE_CANDIDATES = io_uring→libaio→posixaio→sync) and probes O_DIRECT support via tiny throwaway fio_probe jobs; falls back to buffered (with a red banner warning) when O_DIRECT is rejected (older OpenZFS <2.3, some NFS). libaio is only truly async with --direct=1 → io_uring preferred. O_DIRECT bypasses the page cache so we measure the device, not RAM (buffered reads can reflect ARC/page cache).
Everything parses fio --output-format=json (robust, unit-safe metrics, no text scraping): run_fio → _aggregate sums bw/IOPS across jobs, averages per-job p50 (median), takes worst-case p99. fio is run without --group_reporting; we aggregate ourselves to avoid fio's group-merge quirks. FioResult holds bw_mbps (decimal MB/s, matches CDM's SI figure), iops, p50/p99/mean (µs).
One shared bench file <path>/speedtest-hd.bench, reused across all runs, removed at end (_cleanup). Footprint = --size (default 1G).
Output styling: Painter/ANSI (basic 16-color), honors NO_COLOR/FORCE_COLOR/TERM=dumb, per-stream isatty. stdout = results (banner + tables); stderr = progress + verbose dumps, each with its own color flag so piping one works. render_table auto-sizes columns on plain text (color applied after padding, so escapes don't skew widths).
Config dataclass holds resolved settings; confirm prompts unless --yes.

History / decisions

Originated from the Ars Technica fio guide. Original 4K test used QD1/1-job, which measures single-op latency, not throughput — ~12 MB/s on fast NVMe is correct for QD1, not a bug. Evolved to parallel/deep-queue CDM tests, then rewritten in Python for robust JSON parsing, color, and the SLOG profile.
CDM_TESTS is aligned to CrystalDiskMark's true default profile (Q32T16 + Q1T1); an earlier iteration used Q32T1 + Q32T16. The TrueNAS case study in README.md was captured with that earlier profile — its Q32T1 vs Q32T16 comparison is the reason the default changed.

SLOG performance context (why `--slog` exists; full case study in README.md)

Built to diagnose TrueNAS SCALE box linvault1 (Dell R630, Xeon E5-2680 v3; pool nvme-ultra-r10 = 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset vm-root sync=always). "Slow" sync writes were CPU power management, not the SLOG:

Fixes (biggest last): Dell BIOS DAPC → Performance (~2×); cstate/ASPM kernel args; and the big one — CPU governor performance (was intel_cpufreq+schedutil, parking cores at 1.2 GHz because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post Init: echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor.
Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor); T16 → ~78k IOPS / 319 MB/s, scaling regression gone.
Decisive diagnostic: zpool iostat -vl <pool> 1 during fio showed the Optane logs vdev at ~90µs disk_wait — proving the SLOG was fine and latency was upstream (CPU). Also: large sync writes bypass the SLOG (indirect ZIL >32K) — only small 4K sync writes exercise it, which is what --slog does. Healthy Optane SLOG T1 target: ~15–25k IOPS, p50 ~40–65µs.

5.2 KiB Raw Permalink Blame History Unescape Escape