66 lines
4.4 KiB
Markdown
66 lines
4.4 KiB
Markdown
# speedtest-hd
|
||
|
||
Single-file Bash CLI (`speedtest-hd.sh`) that benchmarks any mounted path (HDD/SSD/NVMe,
|
||
local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII table. Falls back
|
||
to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount" disk tester — not
|
||
a ZFS/NVMe-specific tool.
|
||
|
||
## Usage
|
||
```
|
||
./speedtest-hd.sh <path> [--fio|--dd|--slog] [flags]
|
||
```
|
||
- Default: fio if installed, else dd. `--fio` forces CDM mode, `--slog` forces sync-latency mode.
|
||
- Flags: `--engine=io_uring|libaio|posixaio|sync` (default auto), `--direct`/`--buffered`
|
||
(default auto-probe), `--runtime=SEC` (default 5), `--size=SIZE` (default 1g), `--verbose`
|
||
(also dumps raw fio output to stderr).
|
||
|
||
## Modes & layout
|
||
- **CDM mode** (`cdm_speedtest`): runs CrystalDiskMark's default tests, each read+write,
|
||
prints a 5-col table (Read/Write MB/s, Read/Write IOPS). Tests: `SEQ1M Q8T1`, `SEQ1M Q1T1`,
|
||
`RND4K Q32T1`, `RND4K Q32T16`. Q = `--iodepth`, T = `--numjobs`.
|
||
- **SLOG mode** (`slog_speedtest`): 4K synchronous randwrite sweep at T1/T4/T8/T16 to profile
|
||
ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 commit
|
||
latency. Requires `fio` + `python3` (JSON percentile parsing).
|
||
- **dd mode** (`dd_speedtest`): legacy cached/uncached read/write fallback.
|
||
|
||
## Key implementation notes
|
||
- `detect_io_settings` auto-picks the engine (io_uring → libaio → posixaio → sync) and probes
|
||
whether O_DIRECT works, via tiny throwaway `fio_probe` jobs. Falls back to buffered with a
|
||
warning if O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly async
|
||
with `--direct=1`** — that's why io_uring is preferred. O_DIRECT bypasses the page cache so we
|
||
measure the device, not RAM (buffered results, esp. reads, can reflect ARC/page cache).
|
||
- One shared test file `<path>/speedtest-hd.bench` is reused across all runs (laid out once),
|
||
removed at the end. Footprint = `--size` (default 1G), matching the startup notice.
|
||
- `run_fio` (CDM) returns `"MB/s IOPS"`; `run_fio_sync` (SLOG) forces `--ioengine=psync --sync=1`
|
||
(O_SYNC) so every write is a ZIL commit regardless of dataset sync property, and parses fio
|
||
**JSON** to aggregate across jobs (sum IOPS/bw, avg p50, worst-case p99). CDM mode parses
|
||
fio's text output: `fio_bw_mbps` takes the parenthetical **SI MB/s** (matches CDM's decimal
|
||
MB/s, normalizes kB/MB/GB); `fio_iops` expands k/M suffixes.
|
||
- Write tests append `--end_fsync=1` so cached writes can't inflate numbers.
|
||
- ASCII tables: cell formats and dash-segment widths must stay in sync (`tbl_*` = 18/16 dashes,
|
||
`slog_*` = 18/14). Verify alignment after editing.
|
||
|
||
## History / decisions
|
||
- Originated from the Ars Technica fio guide. Original 4K test used `numjobs=1 iodepth=1`, which
|
||
measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1, not
|
||
a bug. Refactored toward parallel/deep-queue tests to show real device capability, then fully
|
||
reshaped into the CrystalDiskMark profile above. `--simple` flag was removed; the table is now
|
||
the only fio output.
|
||
- Note: the current `RND4K` rows are `Q32T1` + `Q32T16`. CrystalDiskMark's actual latest default
|
||
profile is `Q32T16` + `Q1T1` (single-queue random is the meaningful low-end number). If aligning
|
||
strictly to CDM, the `Q32T1` row should become `Q1T1` (`iodepth=1 numjobs=1`).
|
||
|
||
## SLOG performance context (why `--slog` exists)
|
||
Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10`
|
||
= 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). Poor sync
|
||
writes were **CPU power management**, not the SLOG:
|
||
- Fix (biggest last): Dell BIOS profile DAPC → **Performance** (~2×); cstate kernel args; and the
|
||
big one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, which parked cores at
|
||
1.2 GHz because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post
|
||
Init: `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`.
|
||
- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor).
|
||
- Diagnostic: `zpool iostat -vl <pool> 1` during fio showed the Optane `logs` vdev at ~90µs
|
||
disk_wait — proving the SLOG was fine and latency was upstream (CPU).
|
||
- Healthy Optane SLOG single-stream (T1) target: ~15–25k IOPS, p50 ~40–65µs. Much higher usually
|
||
= C-states / PCIe ASPM / BIOS power profile throttling.
|