76 lines
5.2 KiB
Markdown
76 lines
5.2 KiB
Markdown
# speedtest-hd
|
||
|
||
`speedtest-hd.py` — a single-file Python 3 CLI that benchmarks any mounted path
|
||
(HDD/SSD/NVMe, local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII
|
||
table. Falls back to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount"
|
||
disk tester — not a ZFS/NVMe-specific tool. `README.md` is kept in sync with the CLI.
|
||
|
||
## Usage
|
||
```
|
||
./speedtest-hd.py <path> [--fio|--dd|--slog] [flags] # or: python3 speedtest-hd.py ...
|
||
```
|
||
- Default mode: fio if installed (→ cdm), else dd. `--fio`/`--dd`/`--slog` force a mode (mutually exclusive).
|
||
- Flags: `--engine {io_uring,libaio,posixaio,sync}` (default auto), `--direct`/`--buffered`
|
||
(default auto-probe), `--runtime SEC` (default 5), `--size SIZE` (default 1g), `--verbose`
|
||
(dump raw fio output to stderr), `-y`/`--yes` (skip confirm prompt). argparse → both
|
||
`--flag value` and `--flag=value` work.
|
||
|
||
## Requirements
|
||
- **python3 (3.7+)** — required for everything; uses only the standard library (argparse, json,
|
||
dataclasses, statistics). No third-party packages.
|
||
- **fio** — recommended; without it, auto mode falls back to `dd`. `--fio`/`--slog` hard-require it.
|
||
- **sudo** — fio is run via `sudo` (for O_DIRECT + device-cache flush).
|
||
|
||
## Modes / profiles (functions)
|
||
- **cdm** (`cdm_profile`): CrystalDiskMark's default tests, each read+write, 5-col table
|
||
(Read/Write MB/s, Read/Write IOPS). Tests defined as data in `CDM_TESTS` (CrystalDiskMark's
|
||
default profile, in its display order): `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T16`, `RND4K Q1T1`.
|
||
Q=iodepth, T=numjobs. Write runs use `end_fsync`.
|
||
- **slog** (`slog_profile`): 4K synchronous randwrite sweep at T1/T4/T8/T16 (`SLOG_THREADS`) to
|
||
profile ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99
|
||
commit latency. Forces `--ioengine=psync --sync=1` (O_SYNC) regardless of dataset sync property.
|
||
- **dd** (`dd_profile`): dependency-free fallback; sequential write + uncached read + cached(RAM)
|
||
read, timed in Python.
|
||
|
||
## Key implementation
|
||
- `detect_io_settings` auto-picks engine (`ENGINE_CANDIDATES` = io_uring→libaio→posixaio→sync) and
|
||
probes O_DIRECT support via tiny throwaway `fio_probe` jobs; falls back to buffered (with a red
|
||
banner warning) when O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly
|
||
async with `--direct=1`** → io_uring preferred. O_DIRECT bypasses the page cache so we measure
|
||
the device, not RAM (buffered reads can reflect ARC/page cache).
|
||
- **Everything parses fio `--output-format=json`** (robust, unit-safe metrics, no text scraping):
|
||
`run_fio` → `_aggregate` sums bw/IOPS across jobs, averages per-job p50 (median), takes worst-case
|
||
p99. fio is run *without* `--group_reporting`; we aggregate ourselves to avoid fio's group-merge
|
||
quirks. `FioResult` holds bw_mbps (decimal MB/s, matches CDM's SI figure), iops, p50/p99/mean (µs).
|
||
- One shared bench file `<path>/speedtest-hd.bench`, reused across all runs, removed at end
|
||
(`_cleanup`). Footprint = `--size` (default 1G).
|
||
- Output styling: `Painter`/ANSI (basic 16-color), honors `NO_COLOR`/`FORCE_COLOR`/`TERM=dumb`,
|
||
per-stream isatty. **stdout = results (banner + tables); stderr = progress + verbose dumps**, each
|
||
with its own color flag so piping one works. `render_table` auto-sizes columns on *plain* text
|
||
(color applied after padding, so escapes don't skew widths).
|
||
- `Config` dataclass holds resolved settings; `confirm` prompts unless `--yes`.
|
||
|
||
## History / decisions
|
||
- Originated from the Ars Technica fio guide. Original 4K test used QD1/1-job,
|
||
which measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1,
|
||
not a bug. Evolved to parallel/deep-queue CDM tests, then rewritten in Python for robust JSON
|
||
parsing, color, and the SLOG profile.
|
||
- `CDM_TESTS` is aligned to CrystalDiskMark's true default profile (`Q32T16` + `Q1T1`); an earlier
|
||
iteration used `Q32T1` + `Q32T16`. The TrueNAS case study in README.md was captured with that
|
||
earlier profile — its `Q32T1` vs `Q32T16` comparison is the reason the default changed.
|
||
|
||
## SLOG performance context (why `--slog` exists; full case study in README.md)
|
||
Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10`
|
||
= 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). "Slow" sync
|
||
writes were **CPU power management**, not the SLOG:
|
||
- Fixes (biggest last): Dell BIOS DAPC → **Performance** (~2×); cstate/ASPM kernel args; and the big
|
||
one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, parking cores at 1.2 GHz
|
||
because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post Init:
|
||
`echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`.
|
||
- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor); T16 → ~78k
|
||
IOPS / 319 MB/s, scaling regression gone.
|
||
- Decisive diagnostic: `zpool iostat -vl <pool> 1` during fio showed the Optane `logs` vdev at ~90µs
|
||
disk_wait — proving the SLOG was fine and latency was upstream (CPU). Also: large sync writes
|
||
bypass the SLOG (indirect ZIL >32K) — only small 4K sync writes exercise it, which is what `--slog`
|
||
does. Healthy Optane SLOG T1 target: ~15–25k IOPS, p50 ~40–65µs.
|