# speedtest-hd

`speedtest-hd.py` — a single-file Python 3 CLI that benchmarks any mounted path
(HDD/SSD/NVMe, local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII
table. Falls back to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount"
disk tester — not a ZFS/NVMe-specific tool. `README.md` is kept in sync with the CLI.

## Usage
```
./speedtest-hd.py <path> [--fio|--dd|--slog] [flags]      # or: python3 speedtest-hd.py ...
```
- Default mode: fio if installed (→ cdm), else dd. `--fio`/`--dd`/`--slog` force a mode (mutually exclusive).
- Flags: `--engine {io_uring,libaio,posixaio,sync}` (default auto), `--direct`/`--buffered`
  (default auto-probe), `--runtime SEC` (default 5), `--size SIZE` (default 1g), `--verbose`
  (dump raw fio output to stderr), `-y`/`--yes` (skip confirm prompt). argparse → both
  `--flag value` and `--flag=value` work.

## Requirements
- **python3 (3.7+)** — required for everything; uses only the standard library (argparse, json,
  dataclasses, statistics). No third-party packages.
- **fio** — recommended; without it, auto mode falls back to `dd`. `--fio`/`--slog` hard-require it.
- **sudo** — fio is run via `sudo` (for O_DIRECT + device-cache flush).

## Modes / profiles (functions)
- **cdm** (`cdm_profile`): CrystalDiskMark's default tests, each read+write, 5-col table
  (Read/Write MB/s, Read/Write IOPS). Tests defined as data in `CDM_TESTS` (CrystalDiskMark's
  default profile, in its display order): `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T16`, `RND4K Q1T1`.
  Q=iodepth, T=numjobs. Write runs use `end_fsync`.
- **slog** (`slog_profile`): 4K synchronous randwrite sweep at T1/T4/T8/T16 (`SLOG_THREADS`) to
  profile ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99
  commit latency. Forces `--ioengine=psync --sync=1` (O_SYNC) regardless of dataset sync property.
- **dd** (`dd_profile`): dependency-free fallback; sequential write + uncached read + cached(RAM)
  read, timed in Python.

## Key implementation
- `detect_io_settings` auto-picks engine (`ENGINE_CANDIDATES` = io_uring→libaio→posixaio→sync) and
  probes O_DIRECT support via tiny throwaway `fio_probe` jobs; falls back to buffered (with a red
  banner warning) when O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly
  async with `--direct=1`** → io_uring preferred. O_DIRECT bypasses the page cache so we measure
  the device, not RAM (buffered reads can reflect ARC/page cache).
- **Everything parses fio `--output-format=json`** (robust, unit-safe metrics, no text scraping):
  `run_fio` → `_aggregate` sums bw/IOPS across jobs, averages per-job p50 (median), takes worst-case
  p99. fio is run *without* `--group_reporting`; we aggregate ourselves to avoid fio's group-merge
  quirks. `FioResult` holds bw_mbps (decimal MB/s, matches CDM's SI figure), iops, p50/p99/mean (µs).
- One shared bench file `<path>/speedtest-hd.bench`, reused across all runs, removed at end
  (`_cleanup`). Footprint = `--size` (default 1G).
- Output styling: `Painter`/ANSI (basic 16-color), honors `NO_COLOR`/`FORCE_COLOR`/`TERM=dumb`,
  per-stream isatty. **stdout = results (banner + tables); stderr = progress + verbose dumps**, each
  with its own color flag so piping one works. `render_table` auto-sizes columns on *plain* text
  (color applied after padding, so escapes don't skew widths).
- `Config` dataclass holds resolved settings; `confirm` prompts unless `--yes`.

## History / decisions
- Originated from the Ars Technica fio guide. Original 4K test used QD1/1-job,
  which measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1,
  not a bug. Evolved to parallel/deep-queue CDM tests, then rewritten in Python for robust JSON
  parsing, color, and the SLOG profile.
- `CDM_TESTS` is aligned to CrystalDiskMark's true default profile (`Q32T16` + `Q1T1`); an earlier
  iteration used `Q32T1` + `Q32T16`. The TrueNAS case study in README.md was captured with that
  earlier profile — its `Q32T1` vs `Q32T16` comparison is the reason the default changed.

## SLOG performance context (why `--slog` exists; full case study in README.md)
Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10`
= 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). "Slow" sync
writes were **CPU power management**, not the SLOG:
- Fixes (biggest last): Dell BIOS DAPC → **Performance** (~2×); cstate/ASPM kernel args; and the big
  one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, parking cores at 1.2 GHz
  because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post Init:
  `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`.
- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor); T16 → ~78k
  IOPS / 319 MB/s, scaling regression gone.
- Decisive diagnostic: `zpool iostat -vl <pool> 1` during fio showed the Optane `logs` vdev at ~90µs
  disk_wait — proving the SLOG was fine and latency was upstream (CPU). Also: large sync writes
  bypass the SLOG (indirect ZIL >32K) — only small 4K sync writes exercise it, which is what `--slog`
  does. Healthy Optane SLOG T1 target: ~15–25k IOPS, p50 ~40–65µs.