mreschke/speedtest-hd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove the old bash script and update README.md and CLAUDE.md

Browse Source
This commit is contained in:
Matthew Reschke
2026-06-21 21:16:47 -06:00
parent 5561a95a94
commit 363379b2c2
4 changed files with 100 additions and 535 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+62 -52
View File
@@ -1,65 +1,75 @@
# speedtest-hd # speedtest-hd
Single-file Bash CLI (`speedtest-hd.sh`) that benchmarks any mounted path (HDD/SSD/NVMe, `speedtest-hd.py` — a single-file Python 3 CLI that benchmarks any mounted path
local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII table. Falls back (HDD/SSD/NVMe, local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII
to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount" disk tester — not table. Falls back to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount"
a ZFS/NVMe-specific tool. disk tester — not a ZFS/NVMe-specific tool. `README.md` is kept in sync with the CLI.
## Usage ## Usage
``` ```
./speedtest-hd.sh <path> [--fio|--dd|--slog] [flags] ./speedtest-hd.py <path> [--fio|--dd|--slog] [flags] # or: python3 speedtest-hd.py ...
``` ```
- Default: fio if installed, else dd. `--fio` forces CDM mode, `--slog` forces sync-latency mode. - Default mode: fio if installed (→ cdm), else dd. `--fio`/`--dd`/`--slog` force a mode (mutually exclusive).
- Flags: `--engine=io_uring|libaio|posixaio|sync` (default auto), `--direct`/`--buffered` - Flags: `--engine {io_uring,libaio,posixaio,sync}` (default auto), `--direct`/`--buffered`
(default auto-probe), `--runtime=SEC` (default 5), `--size=SIZE` (default 1g), `--verbose` (default auto-probe), `--runtime SEC` (default 5), `--size SIZE` (default 1g), `--verbose`
(also dumps raw fio output to stderr). (dump raw fio output to stderr), `-y`/`--yes` (skip confirm prompt). argparse → both
`--flag value` and `--flag=value` work.
## Modes & layout ## Requirements
- **CDM mode** (`cdm_speedtest`): runs CrystalDiskMark's default tests, each read+write, - **python3 (3.7+)** — required for everything; uses only the standard library (argparse, json,
prints a 5-col table (Read/Write MB/s, Read/Write IOPS). Tests: `SEQ1M Q8T1`, `SEQ1M Q1T1`, dataclasses, statistics). No third-party packages.
`RND4K Q32T1`, `RND4K Q32T16`. Q = `--iodepth`, T = `--numjobs`. - **fio** — recommended; without it, auto mode falls back to `dd`. `--fio`/`--slog` hard-require it.
- **SLOG mode** (`slog_speedtest`): 4K synchronous randwrite sweep at T1/T4/T8/T16 to profile - **sudo** — fio is run via `sudo` (for O_DIRECT + device-cache flush).
ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 commit
latency. Requires `fio` + `python3` (JSON percentile parsing).
- **dd mode** (`dd_speedtest`): legacy cached/uncached read/write fallback.
## Key implementation notes ## Modes / profiles (functions)
- `detect_io_settings` auto-picks the engine (io_uring → libaio → posixaio → sync) and probes - **cdm** (`cdm_profile`): CrystalDiskMark's default tests, each read+write, 5-col table
whether O_DIRECT works, via tiny throwaway `fio_probe` jobs. Falls back to buffered with a (Read/Write MB/s, Read/Write IOPS). Tests defined as data in `CDM_TESTS` (CrystalDiskMark's
warning if O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly async default profile, in its display order): `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T16`, `RND4K Q1T1`.
with `--direct=1`** — that's why io_uring is preferred. O_DIRECT bypasses the page cache so we Q=iodepth, T=numjobs. Write runs use `end_fsync`.
measure the device, not RAM (buffered results, esp. reads, can reflect ARC/page cache). - **slog** (`slog_profile`): 4K synchronous randwrite sweep at T1/T4/T8/T16 (`SLOG_THREADS`) to
- One shared test file `<path>/speedtest-hd.bench` is reused across all runs (laid out once), profile ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99
removed at the end. Footprint = `--size` (default 1G), matching the startup notice. commit latency. Forces `--ioengine=psync --sync=1` (O_SYNC) regardless of dataset sync property.
- `run_fio` (CDM) returns `"MB/s IOPS"`; `run_fio_sync` (SLOG) forces `--ioengine=psync --sync=1` - **dd** (`dd_profile`): dependency-free fallback; sequential write + uncached read + cached(RAM)
(O_SYNC) so every write is a ZIL commit regardless of dataset sync property, and parses fio read, timed in Python.
**JSON** to aggregate across jobs (sum IOPS/bw, avg p50, worst-case p99). CDM mode parses
fio's text output: `fio_bw_mbps` takes the parenthetical **SI MB/s** (matches CDM's decimal ## Key implementation
MB/s, normalizes kB/MB/GB); `fio_iops` expands k/M suffixes. - `detect_io_settings` auto-picks engine (`ENGINE_CANDIDATES` = io_uring→libaio→posixaio→sync) and
- Write tests append `--end_fsync=1` so cached writes can't inflate numbers. probes O_DIRECT support via tiny throwaway `fio_probe` jobs; falls back to buffered (with a red
- ASCII tables: cell formats and dash-segment widths must stay in sync (`tbl_*` = 18/16 dashes, banner warning) when O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly
`slog_*` = 18/14). Verify alignment after editing. async with `--direct=1`** → io_uring preferred. O_DIRECT bypasses the page cache so we measure
the device, not RAM (buffered reads can reflect ARC/page cache).
- **Everything parses fio `--output-format=json`** (robust, unit-safe metrics, no text scraping):
`run_fio` → `_aggregate` sums bw/IOPS across jobs, averages per-job p50 (median), takes worst-case
p99. fio is run *without* `--group_reporting`; we aggregate ourselves to avoid fio's group-merge
quirks. `FioResult` holds bw_mbps (decimal MB/s, matches CDM's SI figure), iops, p50/p99/mean (µs).
- One shared bench file `<path>/speedtest-hd.bench`, reused across all runs, removed at end
(`_cleanup`). Footprint = `--size` (default 1G).
- Output styling: `Painter`/ANSI (basic 16-color), honors `NO_COLOR`/`FORCE_COLOR`/`TERM=dumb`,
per-stream isatty. **stdout = results (banner + tables); stderr = progress + verbose dumps**, each
with its own color flag so piping one works. `render_table` auto-sizes columns on *plain* text
(color applied after padding, so escapes don't skew widths).
- `Config` dataclass holds resolved settings; `confirm` prompts unless `--yes`.
## History / decisions ## History / decisions
- Originated from the Ars Technica fio guide. Original 4K test used `numjobs=1 iodepth=1`, which - Originated from the Ars Technica fio guide. Original 4K test used QD1/1-job,
measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1, not which measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1,
a bug. Refactored toward parallel/deep-queue tests to show real device capability, then fully not a bug. Evolved to parallel/deep-queue CDM tests, then rewritten in Python for robust JSON
reshaped into the CrystalDiskMark profile above. `--simple` flag was removed; the table is now parsing, color, and the SLOG profile.
the only fio output. - `CDM_TESTS` is aligned to CrystalDiskMark's true default profile (`Q32T16` + `Q1T1`); an earlier
- Note: the current `RND4K` rows are `Q32T1` + `Q32T16`. CrystalDiskMark's actual latest default iteration used `Q32T1` + `Q32T16`. The TrueNAS case study in README.md was captured with that
profile is `Q32T16` + `Q1T1` (single-queue random is the meaningful low-end number). If aligning earlier profile — its `Q32T1` vs `Q32T16` comparison is the reason the default changed.
strictly to CDM, the `Q32T1` row should become `Q1T1` (`iodepth=1 numjobs=1`).
## SLOG performance context (why `--slog` exists) ## SLOG performance context (why `--slog` exists; full case study in README.md)
Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10` Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10`
= 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). Poor sync = 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). "Slow" sync
writes were **CPU power management**, not the SLOG: writes were **CPU power management**, not the SLOG:
- Fix (biggest last): Dell BIOS profile DAPC → **Performance** (~2×); cstate kernel args; and the - Fixes (biggest last): Dell BIOS DAPC → **Performance** (~2×); cstate/ASPM kernel args; and the big
big one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, which parked cores at one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, parking cores at 1.2 GHz
1.2 GHz because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post Init:
Init: `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`. `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`.
- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor). - Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor); T16 → ~78k
- Diagnostic: `zpool iostat -vl <pool> 1` during fio showed the Optane `logs` vdev at ~90µs IOPS / 319 MB/s, scaling regression gone.
disk_wait — proving the SLOG was fine and latency was upstream (CPU). - Decisive diagnostic: `zpool iostat -vl <pool> 1` during fio showed the Optane `logs` vdev at ~90µs
- Healthy Optane SLOG single-stream (T1) target: ~15–25k IOPS, p50 ~40–65µs. Much higher usually disk_wait — proving the SLOG was fine and latency was upstream (CPU). Also: large sync writes
= C-states / PCIe ASPM / BIOS power profile throttling. bypass the SLOG (indirect ZIL >32K) — only small 4K sync writes exercise it, which is what `--slog`
does. Healthy Optane SLOG T1 target: ~15–25k IOPS, p50 ~40–65µs.
README.md
+34 -22
View File
@@ -20,7 +20,7 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
## Features ## Features
- **CrystalDiskMark‑style profile** — `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T1`, `RND4K Q32T16`, each measured for Read **and** Write, reported in both **MB/s** and **IOPS**. - **CrystalDiskMark‑style profile** — `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T16`, `RND4K Q1T1` (CrystalDiskMark's default profile), each measured for Read **and** Write, reported in both **MB/s** and **IOPS**.
- **SLOG / sync‑write latency profile** (`--slog`) — synchronous 4K writes at T1/T4/T8/T16 reporting **IOPS, MB/s, and p50/p99 commit latency**. This is the load a ZFS SLOG actually sees. - **SLOG / sync‑write latency profile** (`--slog`) — synchronous 4K writes at T1/T4/T8/T16 reporting **IOPS, MB/s, and p50/p99 commit latency**. This is the load a ZFS SLOG actually sees.
- **Auto‑detection** — picks the fastest available IO engine (`io_uring` → `libaio` → `posixaio` → `sync`) and probes whether `O_DIRECT` works on the filesystem, falling back to buffered IO when it doesn't (e.g. older OpenZFS, some NFS mounts). - **Auto‑detection** — picks the fastest available IO engine (`io_uring` → `libaio` → `posixaio` → `sync`) and probes whether `O_DIRECT` works on the filesystem, falling back to buffered IO when it doesn't (e.g. older OpenZFS, some NFS mounts).
- **`dd` fallback** — if `fio` isn't present, runs a basic write/read test so you still get a number. - **`dd` fallback** — if `fio` isn't present, runs a basic write/read test so you still get a number.
@@ -30,9 +30,8 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
## Requirements ## Requirements
- `bash` - `python3` (3.7+) — the tool itself. It parses `fio`'s JSON output using only the standard library (no third‑party packages to install).
- [`fio`](https://fio.readthedocs.io/) — recommended (`apt install fio` / `pacman -S fio`). Without it, the tool falls back to `dd`. - [`fio`](https://fio.readthedocs.io/) — recommended (`apt install fio` / `pacman -S fio`). Without it, the tool falls back to a basic `dd` test.
- `python3` — **only** required for `--slog` (used to parse `fio`'s JSON output for latency percentiles). The normal profile needs only `fio` + `grep`/`awk`.
- `sudo` — `fio` is invoked via `sudo` so it can use `O_DIRECT` and flush device caches. - `sudo` — `fio` is invoked via `sudo` so it can use `O_DIRECT` and flush device caches.
--- ---
@@ -40,7 +39,8 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
## Usage ## Usage
```bash ```bash
./speedtest-hd.sh <path> [options] ./speedtest-hd.py <path> [options]
# or: python3 speedtest-hd.py <path> [options]
``` ```
`<path>` is the directory (or mount) to benchmark. Use `.` for the current directory. The tool creates a single test file (default 1 GiB) on the target and removes it afterward, so ensure enough free space. `<path>` is the directory (or mount) to benchmark. Use `.` for the current directory. The tool creates a single test file (default 1 GiB) on the target and removes it afterward, so ensure enough free space.
@@ -49,36 +49,44 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
| Invocation | What it does | | Invocation | What it does |
|---|---| |---|---|
| `./speedtest-hd.sh /mnt/disk` | Auto: uses `fio` if installed, else `dd` | | `./speedtest-hd.py /mnt/disk` | Auto: uses `fio` if installed, else `dd` |
| `./speedtest-hd.sh /mnt/disk --fio` | Force the `fio` CrystalDiskMark‑style profile | | `./speedtest-hd.py /mnt/disk --fio` | Force the `fio` CrystalDiskMark‑style profile |
| `./speedtest-hd.sh /mnt/disk --dd` | Force the basic `dd` test | | `./speedtest-hd.py /mnt/disk --dd` | Force the basic `dd` test |
| `./speedtest-hd.sh /mnt/disk --slog` | SLOG / sync‑write latency profile | | `./speedtest-hd.py /mnt/disk --slog` | SLOG / sync‑write latency profile |
`--fio`, `--dd`, and `--slog` are mutually exclusive.
### Tuning flags ### Tuning flags
| Flag | Effect | | Flag | Effect |
|---|---| |---|---|
| `--engine=io_uring\|libaio\|posixaio\|sync` | Force a specific IO engine (default: auto) | | `--engine {io_uring,libaio,posixaio,sync}` | Force a specific IO engine (default: auto) |
| `--direct` | Force `O_DIRECT` (bypass page cache) | | `--direct` | Force `O_DIRECT` (bypass page cache) |
| `--buffered` | Force buffered IO (e.g. when `O_DIRECT` is unsupported) | | `--buffered` | Force buffered IO (e.g. when `O_DIRECT` is unsupported) |
| `--runtime=SEC` | Seconds per run (default: 5, like CrystalDiskMark) | | `--runtime SEC` | Seconds per run (default: 5, like CrystalDiskMark) |
| `--size=SIZE` | Test file size (default: `1g`) | | `--size SIZE` | Test file size (default: `1g`) |
| `--verbose` | Also print the full `fio` output for every run (summary table unchanged) | | `--verbose` | Also print the full `fio` output for every run (summary table unchanged) |
| `-y`, `--yes` | Skip the confirmation prompt (for scripting/automation) |
> All flags accept either `--flag value` or `--flag=value` (argparse). `--direct`/`--buffered` are mutually exclusive.
### Examples ### Examples
```bash ```bash
# CrystalDiskMark-style test of the current directory # CrystalDiskMark-style test of the current directory
./speedtest-hd.sh . ./speedtest-hd.py .
# Larger file, longer runs, on an NVMe pool # Larger file, longer runs, on an NVMe pool
./speedtest-hd.sh /mnt/nvmepool --runtime=10 --size=4g ./speedtest-hd.py /mnt/nvmepool --runtime=10 --size=4g
# Buffered (e.g. an NFS share that doesn't support O_DIRECT) # Buffered (e.g. an NFS share that doesn't support O_DIRECT)
./speedtest-hd.sh /mnt/nfsshare --buffered ./speedtest-hd.py /mnt/nfsshare --buffered
# SLOG / sync latency profile, 30s per run # SLOG / sync latency profile, 30s per run
./speedtest-hd.sh /mnt/nvme-ultra-r10/vm-root --slog --runtime=30 ./speedtest-hd.py /mnt/nvme-ultra-r10/vm-root --slog --runtime=30
# Unattended (no prompt), forcing a specific engine
./speedtest-hd.py /mnt/nvmepool --yes --engine io_uring
``` ```
> **Tip:** when running `--slog` against a ZFS dataset, watch the SLOG live in another shell: > **Tip:** when running `--slog` against a ZFS dataset, watch the SLOG live in another shell:
@@ -92,14 +100,16 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
### CrystalDiskMark‑style profile ### CrystalDiskMark‑style profile
Representative output from a healthy local NVMe (your numbers will differ):
``` ```
+------------------+----------------+----------------+----------------+----------------+ +------------------+----------------+----------------+----------------+----------------+
| Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) | | Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) |
+------------------+----------------+----------------+----------------+----------------+ +------------------+----------------+----------------+----------------+----------------+
| SEQ1M Q8T1 | 6873.00 | 9.30 | 6873 | 9 | | SEQ1M Q8T1 | 3650.00 | 3120.00 | 3482 | 2976 |
| SEQ1M Q1T1 | 1608.00 | 20.00 | 1608 | 20 | | SEQ1M Q1T1 | 2680.00 | 2510.00 | 2556 | 2394 |
| RND4K Q32T1 | 538.00 | 10.80 | 137728 | 2764 | | RND4K Q32T16 | 2950.00 | 2240.00 | 720215 | 546875 |
| RND4K Q32T16 | 689.00 | 261.00 | 176384 | 66816 | | RND4K Q1T1 | 78.00 | 64.00 | 19043 | 15625 |
+------------------+----------------+----------------+----------------+----------------+ +------------------+----------------+----------------+----------------+----------------+
``` ```
@@ -126,8 +136,8 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
|---|---|---|---| |---|---|---|---|
| `SEQ1M Q8T1` | Sequential 1 MiB | 8 | 1 | | `SEQ1M Q8T1` | Sequential 1 MiB | 8 | 1 |
| `SEQ1M Q1T1` | Sequential 1 MiB | 1 | 1 | | `SEQ1M Q1T1` | Sequential 1 MiB | 1 | 1 |
| `RND4K Q32T1` | Random 4 KiB | 32 | 1 |
| `RND4K Q32T16` | Random 4 KiB | 32 | 16 | | `RND4K Q32T16` | Random 4 KiB | 32 | 16 |
| `RND4K Q1T1` | Random 4 KiB | 1 | 1 |
`Q` = queue depth (`--iodepth`), `T` = threads (`--numjobs`). Note that `--iodepth` only produces real queue depth when the IO is truly asynchronous (async engine **and** `O_DIRECT`). On filesystems where that isn't available, queue depth effectively collapses toward 1 and concurrency comes only from threads (`T`). `Q` = queue depth (`--iodepth`), `T` = threads (`--numjobs`). Note that `--iodepth` only produces real queue depth when the IO is truly asynchronous (async engine **and** `O_DIRECT`). On filesystems where that isn't available, queue depth effectively collapses toward 1 and concurrency comes only from threads (`T`).
@@ -174,6 +184,8 @@ The standard benchmark looked alarming — huge reads, tiny writes:
9.3 MB/s sequential write on an Optane‑backed NVMe pool looks broken. 9.3 MB/s sequential write on an Optane‑backed NVMe pool looks broken.
> **Note:** this investigation was captured with an earlier test profile that used `RND4K Q32T1` in place of today's `RND4K Q1T1`. The `Q32T1` vs `Q32T16` comparison below is exactly why the default later changed — see finding #3.
### Investigation ### Investigation
**1. The reads are RAM, not disk.** With a 1 GiB test file on ZFS, reads come straight from ARC (RAM cache). The huge read/write asymmetry is the tell — ignore the read column for judging the disks. **1. The reads are RAM, not disk.** With a 1 GiB test file on ZFS, reads come straight from ARC (RAM cache). The huge read/write asymmetry is the tell — ignore the read column for judging the disks.
@@ -287,5 +299,5 @@ The residual gap from raw Optane (~15 µs) is ZFS ZIL‑commit overhead plus the
- **`sudo`** is used for `fio` so it can apply `O_DIRECT` and flush device write caches at the end of write runs (`--end_fsync=1`), so cached writes can't inflate results. - **`sudo`** is used for `fio` so it can apply `O_DIRECT` and flush device write caches at the end of write runs (`--end_fsync=1`), so cached writes can't inflate results.
- **`O_DIRECT` is auto‑detected.** If the banner shows `O_DIRECT: DISABLED (buffered ...)`, results may reflect the page cache (RAM) rather than the device. - **`O_DIRECT` is auto‑detected.** If the banner shows `O_DIRECT: DISABLED (buffered ...)`, results may reflect the page cache (RAM) rather than the device.
- **A single shared test file** is reused across runs to keep the footprint to one file. - **A single shared test file** is reused across runs to keep the footprint to one file.
- **`--slog` requires `python3`** (for JSON latency‑percentile parsing); the standard profile does not. - **All profiles parse `fio`'s JSON output** (`--output-format=json`) with Python's standard library — robust, unit‑safe metrics with no fragile text scraping.
- The `--slog` profile forces synchronous IO and is intended for ZFS ZIL / SLOG and other sync‑write (NFS/iSCSI/VM) investigations. - The `--slog` profile forces synchronous IO and is intended for ZFS ZIL / SLOG and other sync‑write (NFS/iSCSI/VM) investigations.
speedtest-hd.py
+4 -4
View File
@@ -8,7 +8,7 @@ for diagnosing ZFS ZIL performance (NFS / iSCSI / VM sync workloads).
Three profiles Three profiles
-------------- --------------
* **cdm** — the four CrystalDiskMark default tests (``SEQ1M Q8T1``, ``SEQ1M * **cdm** — the four CrystalDiskMark default tests (``SEQ1M Q8T1``, ``SEQ1M
Q1T1``, ``RND4K Q32T1``, ``RND4K Q32T16``), each measured for read *and* Q1T1``, ``RND4K Q32T16``, ``RND4K Q1T1``), each measured for read *and*
write, reported in MB/s and IOPS. write, reported in MB/s and IOPS.
* **slog** — synchronous 4K random writes swept across thread counts, reporting * **slog** — synchronous 4K random writes swept across thread counts, reporting
IOPS, MB/s and p50/p99 *commit latency*. This is the load a ZFS SLOG sees. IOPS, MB/s and p50/p99 *commit latency*. This is the load a ZFS SLOG sees.
@@ -124,13 +124,13 @@ class CdmTest:
return "write" if self.seq else "randwrite" return "write" if self.seq else "randwrite"
# The four CrystalDiskMark default tests, as data. Q = queue depth (iodepth), # The four CrystalDiskMark default tests, as data, in CrystalDiskMark's own
# T = threads (numjobs). # display order. Q = queue depth (iodepth), T = threads (numjobs).
CDM_TESTS: tuple[CdmTest, ...] = ( CDM_TESTS: tuple[CdmTest, ...] = (
CdmTest("SEQ1M Q8T1", bs="1m", iodepth=8, numjobs=1, seq=True), CdmTest("SEQ1M Q8T1", bs="1m", iodepth=8, numjobs=1, seq=True),
CdmTest("SEQ1M Q1T1", bs="1m", iodepth=1, numjobs=1, seq=True), CdmTest("SEQ1M Q1T1", bs="1m", iodepth=1, numjobs=1, seq=True),
CdmTest("RND4K Q32T1", bs="4k", iodepth=32, numjobs=1, seq=False),
CdmTest("RND4K Q32T16", bs="4k", iodepth=32, numjobs=16, seq=False), CdmTest("RND4K Q32T16", bs="4k", iodepth=32, numjobs=16, seq=False),
CdmTest("RND4K Q1T1", bs="4k", iodepth=1, numjobs=1, seq=False),
) )
# Thread counts for the SLOG sweep. T1 is the headline single-stream latency; # Thread counts for the SLOG sweep. T1 is the headline single-stream latency;
speedtest-hd.sh
-457
View File
@@ -1,457 +0,0 @@
#!/usr/bin/env bash
# Robust HD/SSD/NVMe performance CLI utility
# A CrystalDiskMark-style storage benchmark for Linux, built on fio.
# Runs the exact same four tests CrystalDiskMark does (SEQ1M Q8T1, SEQ1M Q1T1,
# RND4K Q32T1, RND4K Q32T16), each measured for both Read and Write, and prints
# the results in a CrystalDiskMark-style table.
# Dependencies: fio (apt install fio / pacman -S fio). Falls back to dd if missing.
# See: https://cloud.google.com/compute/docs/disks/benchmarking-pd-performance
# See: https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/
# mReschke 2024-01-18
# CLI Parameters
path="$1"
option=""
engine="" # IO engine; empty = auto-detect (io_uring > libaio > posixaio > sync)
direct="" # O_DIRECT; empty = auto-detect; 1 = bypass page cache, 0 = buffered
runtime=5 # seconds per measurement (CrystalDiskMark default is 5s)
size="1g" # test file size (CrystalDiskMark default is 1GiB)
verbose=0 # 1 = also print the full fio output for every run
for arg in "${@:2}"; do
case "$arg" in
--dd) option="--dd" ;;
--fio) option="--fio" ;;
--slog) option="--slog" ;;
--buffered) direct=0 ;;
--direct) direct=1 ;;
--verbose) verbose=1 ;;
--engine=*) engine="${arg#*=}" ;;
--direct=*) direct="${arg#*=}" ;;
--runtime=*) runtime="${arg#*=}" ;;
--size=*) size="${arg#*=}" ;;
esac
done
# Main application flow
function main {
# Show usage if no params
if [ ! "$path" ]; then
usage
fi
# Understand . path
if [ "$path" == '.' ]; then
path=$(pwd)
fi
# Check if path exists
if [ ! -e "$path" ]; then
echo "Path $path does not exist"
exit 1
fi
# Must type y or n THEN press enter (which I like better)
echo "NOTICE: ${size^^} free space on '$path' is required to perform the benchmark."
echo -n "Are you ready to start a storage benchmark against '$path' ? "; read answer
if [ "$answer" != "${answer#[Yy]}" ]; then
echo "Great! Starting benchmark now!"
else
echo "Ok, cancelled!"
exit 0
fi
# Use dd or fio based on param or defaults
if [ "$option" == "--dd" ]; then
dd_speedtest
elif [ "$option" == "--fio" ]; then
cdm_speedtest
elif [ "$option" == "--slog" ]; then
slog_speedtest
elif [ "$option" == "" ]; then
# If fio is installed, use it, else use dd
if ! command -v fio &> /dev/null; then
echo ""
echo "fio is not installed -- falling back to basic dd test."
echo "Install fio for the full CrystalDiskMark-style benchmark."
dd_speedtest
else
cdm_speedtest
fi
fi
}
function fio_probe {
# Run a tiny throwaway fio job to see if a given (engine, direct) combo
# actually works on this path/filesystem. Returns 0 on success. This is how
# we stay accurate AND portable across ext4, xfs, btrfs, ZFS, NFS, etc.
# without the caller needing to know what each filesystem supports.
local eng="$1" dir="$2"
sudo fio --name=probe \
--directory="$path" \
--ioengine="$eng" \
--rw=write --bs=4k --size=1m \
--direct="$dir" \
--time_based --runtime=1 \
> /dev/null 2>&1
local rc=$?
rm -rf "$path"/probe* 2>/dev/null
return $rc
}
function detect_io_settings {
# Pick the fastest available IO engine unless the user forced one with
# --engine=. io_uring is the modern, lowest-overhead Linux engine; libaio
# is older (and only truly async with O_DIRECT); posixaio/sync are the
# portable fallbacks (eg some NFS mounts).
if [ -z "$engine" ]; then
for eng in io_uring libaio posixaio sync; do
if fio_probe "$eng" 0; then engine="$eng"; break; fi
done
[ -z "$engine" ] && engine="sync"
fi
# Decide O_DIRECT unless forced with --direct/--buffered. O_DIRECT bypasses
# the OS page cache so we measure the device instead of RAM. Not every
# filesystem supports it (eg older OpenZFS < 2.3, some NFS mounts), so we
# probe and fall back to buffered rather than erroring out.
if [ -z "$direct" ]; then
if fio_probe "$engine" 1; then direct=1; else direct=0; fi
fi
}
function fio_bw_mbps {
# Extract bandwidth in decimal MB/s (matching CrystalDiskMark) from a fio
# run. fio prints eg " READ: bw=3357MiB/s (3521MB/s), ..." -- we take the
# parenthetical SI figure and normalise kB/MB/GB into MB/s.
local out="$1"
local raw num unit
raw=$(echo "$out" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | head -n1)
[ -z "$raw" ] && { echo "0.00"; return; }
num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1)
unit=$(echo "$raw" | /usr/bin/grep -oP '[A-Za-z]+/s' | head -n1)
case "$unit" in
B/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000000}' ;;
kB/s|KB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000}' ;;
MB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;;
GB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n*1000}' ;;
*) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;;
esac
}
function fio_iops {
# Extract IOPS from a fio run. fio prints eg " read: IOPS=12.3k, BW=..." --
# the value may carry a k/M suffix, which we expand to a whole number.
local out="$1"
local raw num unit
raw=$(echo "$out" | /usr/bin/grep -oP 'IOPS=\K[0-9.]+[kKmM]?' | head -n1)
[ -z "$raw" ] && { echo "0"; return; }
num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1)
unit=$(echo "$raw" | /usr/bin/grep -oP '[kKmM]' | head -n1)
case "$unit" in
k|K) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000}' ;;
m|M) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000000}' ;;
*) awk -v n="$num" 'BEGIN{printf "%.0f", n}' ;;
esac
}
function run_fio {
# run_fio <rw> <bs> <iodepth> <numjobs> -> echoes "MB/s IOPS" (space-sep)
# A single shared 1GB test file is reused across every run (laid out once),
# which keeps the footprint at one file and avoids re-creating it each time.
local rw="$1" bs="$2" qd="$3" jobs="$4"
local args=(
--name=speedtest
--filename="$benchfile"
--ioengine="$engine"
--direct="$direct"
--rw="$rw"
--bs="$bs"
--size="$size"
--numjobs="$jobs"
--iodepth="$qd"
--time_based --runtime="$runtime"
--group_reporting=1
)
# Flush device cache at the end of write tests so cached writes can't lie.
case "$rw" in
*write) args+=(--end_fsync=1) ;;
esac
# Capture the full fio output. In verbose mode we also keep fio's stderr
# (2>&1); otherwise we discard it. Either way only the parsed bandwidth is
# echoed to stdout, so the captured value stays clean.
local out
if [ "$verbose" = 1 ]; then
out=$(sudo fio "${args[@]}" 2>&1)
else
out=$(sudo fio "${args[@]}" 2>/dev/null)
fi
# Dump the raw fio output (to stderr, so it never pollutes the captured
# values on stdout) when running verbose.
if [ "$verbose" = 1 ]; then
echo " --- fio: $rw bs=$bs iodepth=$qd numjobs=$jobs ---" >&2
echo "$out" >&2
echo "" >&2
fi
echo "$(fio_bw_mbps "$out") $(fio_iops "$out")"
}
# Table drawing -------------------------------------------------------------
# Label cell is "| %-16s |" (18 dashes); the four numeric cells are "| %14s |"
# (16 dashes each).
TBL_SEG_LBL="------------------" # 18 dashes -> matches "| %-16s |"
TBL_SEG_NUM="----------------" # 16 dashes -> matches "| %14s |"
function tbl_rule {
printf "+%s+%s+%s+%s+%s+\n" "$TBL_SEG_LBL" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM"
}
function tbl_row {
# tbl_row <test> <read-mb/s> <write-mb/s> <read-iops> <write-iops>
# (col1 left-aligned, the four numeric cols right-aligned)
printf "| %-16s | %14s | %14s | %14s | %14s |\n" "$1" "$2" "$3" "$4" "$5"
}
# Five-column table for the --slog latency profile.
# "| %-16s |" -> 18 dashes ; "| %12s |" -> 14 dashes
SLOG_SEG16="------------------" # 18 dashes
SLOG_SEG12="--------------" # 14 dashes
function slog_rule {
printf "+%s+%s+%s+%s+%s+\n" "$SLOG_SEG16" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12"
}
function slog_row {
# slog_row <test> <iops> <mb/s> <p50> <p99> (col1 left, rest right-aligned)
printf "| %-16s | %12s | %12s | %12s | %12s |\n" "$1" "$2" "$3" "$4" "$5"
}
function run_fio_sync {
# run_fio_sync <bs> <numjobs>
# -> echoes "IOPS MB/s p50_us p99_us mean_us" (one space-separated line)
# Forces synchronous IO (--sync=1 => O_SYNC) with the portable psync engine so
# every write is a ZIL commit -- i.e. it exercises the SLOG exactly the way a
# sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own
# sync property. We deliberately DON'T use --group_reporting here: each job is
# reported independently in the JSON and we aggregate ourselves (sum IOPS/bw,
# average p50, take the worst p99), which sidesteps fio's group-merge quirks.
local bs="$1" jobs="$2"
local args=(
--name=slog
--filename="$benchfile"
--ioengine=psync
--rw=randwrite
--bs="$bs"
--size="$size"
--numjobs="$jobs"
--sync=1
--time_based --runtime="$runtime"
--output-format=json
)
local out
out=$(sudo fio "${args[@]}" 2>/dev/null)
if [ "$verbose" = 1 ]; then
echo " --- fio json: randwrite bs=$bs numjobs=$jobs sync=1 ---" >&2
echo "$out" >&2
echo "" >&2
fi
# Parse the JSON: sum IOPS and bandwidth across jobs, aggregate completion
# latency percentiles (clat is in ns -> convert to us). bw_bytes is bytes/s.
echo "$out" | python3 -c '
import json, sys
try:
d = json.load(sys.stdin)
jobs = d["jobs"]
except Exception:
print("0 0.00 0.0 0.0 0.0"); sys.exit(0)
iops = sum(j["write"]["iops"] for j in jobs)
mbps = sum(j["write"]["bw_bytes"] for j in jobs) / 1e6
def pctl(j, p):
return j["write"]["clat_ns"]["percentile"].get(p)
p50 = [pctl(j, "50.000000") for j in jobs if pctl(j, "50.000000") is not None]
p99 = [pctl(j, "99.000000") for j in jobs if pctl(j, "99.000000") is not None]
mean = [j["write"]["clat_ns"]["mean"] for j in jobs]
p50_us = (sum(p50)/len(p50))/1000.0 if p50 else 0.0 # average of per-job medians
p99_us = (max(p99))/1000.0 if p99 else 0.0 # worst-case tail across jobs
mean_us = (sum(mean)/len(mean))/1000.0 if mean else 0.0
print("%.0f %.2f %.1f %.1f %.1f" % (iops, mbps, p50_us, p99_us, mean_us))
'
}
function slog_speedtest {
# SLOG / sync-write latency profile. Detect engine/direct only for the banner;
# the actual runs always force psync + O_SYNC (see run_fio_sync).
detect_io_settings
benchfile="$path/speedtest-hd.bench"
if ! command -v fio &> /dev/null; then
echo "ERROR: --slog requires fio (apt install fio / pacman -S fio)." >&2
exit 1
fi
if ! command -v python3 &> /dev/null; then
echo "ERROR: --slog requires python3 (used to parse fio JSON latency percentiles)." >&2
exit 1
fi
echo ""
echo " speedtest-hd : SLOG / sync-write latency profile"
echo " Target : $path"
echo " Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine"
echo " Profile : runtime=${runtime}s/run size=${size^^}"
echo " Note : every write is a synchronous ZIL commit -- this is the load"
echo " your SLOG actually sees. Watch it live in another shell with:"
echo " zpool iostat -vl <pool> 1"
echo ""
# T1 is the headline single-stream latency; the sweep shows how the SLOG
# scales as concurrent sync writers (NFS/iSCSI/VM threads) pile on. Run every
# measurement FIRST (progress goes to stderr), collect the results, then draw
# the whole table in one block so the "measuring..." lines can't interleave
# into the middle of the table.
local jobs_list="1 4 8 16"
local -a rows
local j res iops bw p50 p99 mean
for j in $jobs_list; do
echo " measuring 4K sync randwrite T$j ..." >&2
res=$(run_fio_sync 4k "$j")
read -r iops bw p50 p99 mean <<< "$res"
rows+=("$(slog_row "4K sync T$j" "$iops" "$bw" "$p50" "$p99")")
done
rm -f "$benchfile"
# Render the assembled table
echo ""
slog_rule
slog_row "Test" "IOPS" "MB/s" "p50 lat(us)" "p99 lat(us)"
slog_rule
printf '%s\n' "${rows[@]}"
slog_rule
echo ""
echo " Healthy Optane SLOG (eg P1600X) single-stream (T1) target:"
echo " ~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means"
echo " CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling."
echo ""
}
function cdm_speedtest {
# Auto-detect the best engine and whether O_DIRECT works on this path.
detect_io_settings
benchfile="$path/speedtest-hd.bench"
local direct_label="enabled (device)"
[ "$direct" != 1 ] && direct_label="DISABLED (buffered -- may reflect RAM cache!)"
# Banner
echo ""
echo " speedtest-hd : CrystalDiskMark-style storage benchmark"
echo " Target : $path"
echo " Engine : $engine O_DIRECT: $direct_label"
echo " Profile : size=${size^^} runtime=${runtime}s/run (8 runs)"
echo ""
# The four CrystalDiskMark default tests. Q = queue depth (--iodepth),
# T = threads (--numjobs). The first three are T1 (single thread); the last
# is T16. A single shared test file is reused -- with numjobs>1 every job
# issues IO against the same file, which is fine.
# Each run now returns both MB/s and IOPS ("bw iops"), so we capture the pair.
# label read-rw write-rw bs Q T
echo " measuring SEQ1M Q8T1 ..." >&2
local s8r_bw s8r_io; read -r s8r_bw s8r_io <<< "$(run_fio read 1m 8 1)"
local s8w_bw s8w_io; read -r s8w_bw s8w_io <<< "$(run_fio write 1m 8 1)"
echo " measuring SEQ1M Q1T1 ..." >&2
local s1r_bw s1r_io; read -r s1r_bw s1r_io <<< "$(run_fio read 1m 1 1)"
local s1w_bw s1w_io; read -r s1w_bw s1w_io <<< "$(run_fio write 1m 1 1)"
echo " measuring RND4K Q32T1 ..." >&2
local r32r_bw r32r_io; read -r r32r_bw r32r_io <<< "$(run_fio randread 4k 32 1)"
local r32w_bw r32w_io; read -r r32w_bw r32w_io <<< "$(run_fio randwrite 4k 32 1)"
echo " measuring RND4K Q32T16 ..." >&2
local r1r_bw r1r_io; read -r r1r_bw r1r_io <<< "$(run_fio randread 4k 32 16)"
local r1w_bw r1w_io; read -r r1w_bw r1w_io <<< "$(run_fio randwrite 4k 32 16)"
# Cleanup the shared test file
rm -f "$benchfile"
# Render the CrystalDiskMark-style table
echo ""
tbl_rule
tbl_row "Test" "Read (MB/s)" "Write (MB/s)" "Read (IOPS)" "Write (IOPS)"
tbl_rule
tbl_row "SEQ1M Q8T1" "$s8r_bw" "$s8w_bw" "$s8r_io" "$s8w_io"
tbl_row "SEQ1M Q1T1" "$s1r_bw" "$s1w_bw" "$s1r_io" "$s1w_io"
tbl_row "RND4K Q32T1" "$r32r_bw" "$r32w_bw" "$r32r_io" "$r32w_io"
tbl_row "RND4K Q32T16" "$r1r_bw" "$r1w_bw" "$r1r_io" "$r1w_io"
tbl_rule
echo ""
}
function dd_speedtest {
# Basic HD speed test using dd (fallback when fio is not installed)
# mReschke 2017-07-11
local file ddsize
file=$path/bigfile
ddsize=1024
echo "Running dd based HD/SSD/NVMe Benchmarks"
echo "---------------------------------------"
printf "Cached write speed...\n"
dd if=/dev/zero of=$file bs=1M count=$ddsize
printf "\nUncached write speed...\n"
dd if=/dev/zero of=$file bs=1M count=$ddsize conv=fdatasync,notrunc
printf "\nUncached read speed...\n"
echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
dd if=$file of=/dev/null bs=1M count=$ddsize
printf "\nCached read speed...\n"
dd if=$file of=/dev/null bs=1M count=$ddsize
rm $file
printf "\nDone\n"
}
# Show help and usage information
function usage {
echo "speedtest-hd : CrystalDiskMark-style storage benchmark"
echo " Runs the same four tests as CrystalDiskMark (each Read + Write):"
echo " SEQ1M Q8T1 Sequential 1MiB, queue depth 8, 1 thread"
echo " SEQ1M Q1T1 Sequential 1MiB, queue depth 1, 1 thread"
echo " RND4K Q32T1 Random 4KiB, queue depth 32, 1 thread"
echo " RND4K Q32T16 Random 4KiB, queue depth 32, 16 threads"
echo " Uses fio if installed (recommended), else basic dd."
echo "mReschke 2024-01-18"
echo ""
echo "NOTICE: creates one test file (default 1GB) on the destination disk."
echo "Please ensure you have write access with enough free space."
echo ""
echo "Usage:"
echo " Auto (fio if installed, else dd):"
echo " ./speedtest-hd /mnt/somedisk"
echo " ./speedtest-hd ."
echo ""
echo " Force fio / force dd:"
echo " ./speedtest-hd /mnt/somedisk --fio"
echo " ./speedtest-hd /mnt/somedisk --dd"
echo ""
echo " SLOG / sync-write latency profile (ZFS ZIL, NFS/iSCSI/VM sync IO):"
echo " ./speedtest-hd /mnt/zpool/dataset --slog"
echo " 4K synchronous writes at T1/T4/T8/T16; reports IOPS, MB/s, p50/p99"
echo " commit latency. Requires fio + python3. (Tip: watch the SLOG with"
echo " 'zpool iostat -vl <pool> 1' in another shell.)"
echo ""
echo " Tuning flags (auto-detected by default, override as needed):"
echo " --engine=io_uring|libaio|posixaio|sync IO engine (default: auto)"
echo " --direct Force O_DIRECT (bypass page cache)"
echo " --buffered Force buffered IO (eg if O_DIRECT is unsupported)"
echo " --runtime=SEC Seconds per run (default: 5, like CrystalDiskMark)"
echo " --size=SIZE Test file size (default: 1g)"
echo " --verbose Print full fio output for every run (summary table unchanged)"
echo " Examples:"
echo " ./speedtest-hd /mnt/nvmepool --runtime=10 --size=4g"
echo " ./speedtest-hd /mnt/nfsshare --buffered"
echo " ./speedtest-hd /mnt/nvmepool --verbose"
echo " ./speedtest-hd /mnt/nvme-ultra-r10/vm-root --slog --runtime=30"
exit 0
}
# Go
main