diff --git a/CLAUDE.md b/CLAUDE.md index 377139a..98d265e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,65 +1,75 @@ # speedtest-hd -Single-file Bash CLI (`speedtest-hd.sh`) that benchmarks any mounted path (HDD/SSD/NVMe, -local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII table. Falls back -to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount" disk tester — not -a ZFS/NVMe-specific tool. +`speedtest-hd.py` — a single-file Python 3 CLI that benchmarks any mounted path +(HDD/SSD/NVMe, local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII +table. Falls back to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount" +disk tester — not a ZFS/NVMe-specific tool. `README.md` is kept in sync with the CLI. ## Usage ``` -./speedtest-hd.sh [--fio|--dd|--slog] [flags] +./speedtest-hd.py [--fio|--dd|--slog] [flags] # or: python3 speedtest-hd.py ... ``` -- Default: fio if installed, else dd. `--fio` forces CDM mode, `--slog` forces sync-latency mode. -- Flags: `--engine=io_uring|libaio|posixaio|sync` (default auto), `--direct`/`--buffered` - (default auto-probe), `--runtime=SEC` (default 5), `--size=SIZE` (default 1g), `--verbose` - (also dumps raw fio output to stderr). +- Default mode: fio if installed (→ cdm), else dd. `--fio`/`--dd`/`--slog` force a mode (mutually exclusive). +- Flags: `--engine {io_uring,libaio,posixaio,sync}` (default auto), `--direct`/`--buffered` + (default auto-probe), `--runtime SEC` (default 5), `--size SIZE` (default 1g), `--verbose` + (dump raw fio output to stderr), `-y`/`--yes` (skip confirm prompt). argparse → both + `--flag value` and `--flag=value` work. -## Modes & layout -- **CDM mode** (`cdm_speedtest`): runs CrystalDiskMark's default tests, each read+write, - prints a 5-col table (Read/Write MB/s, Read/Write IOPS). Tests: `SEQ1M Q8T1`, `SEQ1M Q1T1`, - `RND4K Q32T1`, `RND4K Q32T16`. Q = `--iodepth`, T = `--numjobs`. -- **SLOG mode** (`slog_speedtest`): 4K synchronous randwrite sweep at T1/T4/T8/T16 to profile - ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 commit - latency. Requires `fio` + `python3` (JSON percentile parsing). -- **dd mode** (`dd_speedtest`): legacy cached/uncached read/write fallback. +## Requirements +- **python3 (3.7+)** — required for everything; uses only the standard library (argparse, json, + dataclasses, statistics). No third-party packages. +- **fio** — recommended; without it, auto mode falls back to `dd`. `--fio`/`--slog` hard-require it. +- **sudo** — fio is run via `sudo` (for O_DIRECT + device-cache flush). -## Key implementation notes -- `detect_io_settings` auto-picks the engine (io_uring → libaio → posixaio → sync) and probes - whether O_DIRECT works, via tiny throwaway `fio_probe` jobs. Falls back to buffered with a - warning if O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly async - with `--direct=1`** — that's why io_uring is preferred. O_DIRECT bypasses the page cache so we - measure the device, not RAM (buffered results, esp. reads, can reflect ARC/page cache). -- One shared test file `/speedtest-hd.bench` is reused across all runs (laid out once), - removed at the end. Footprint = `--size` (default 1G), matching the startup notice. -- `run_fio` (CDM) returns `"MB/s IOPS"`; `run_fio_sync` (SLOG) forces `--ioengine=psync --sync=1` - (O_SYNC) so every write is a ZIL commit regardless of dataset sync property, and parses fio - **JSON** to aggregate across jobs (sum IOPS/bw, avg p50, worst-case p99). CDM mode parses - fio's text output: `fio_bw_mbps` takes the parenthetical **SI MB/s** (matches CDM's decimal - MB/s, normalizes kB/MB/GB); `fio_iops` expands k/M suffixes. -- Write tests append `--end_fsync=1` so cached writes can't inflate numbers. -- ASCII tables: cell formats and dash-segment widths must stay in sync (`tbl_*` = 18/16 dashes, - `slog_*` = 18/14). Verify alignment after editing. +## Modes / profiles (functions) +- **cdm** (`cdm_profile`): CrystalDiskMark's default tests, each read+write, 5-col table + (Read/Write MB/s, Read/Write IOPS). Tests defined as data in `CDM_TESTS` (CrystalDiskMark's + default profile, in its display order): `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T16`, `RND4K Q1T1`. + Q=iodepth, T=numjobs. Write runs use `end_fsync`. +- **slog** (`slog_profile`): 4K synchronous randwrite sweep at T1/T4/T8/T16 (`SLOG_THREADS`) to + profile ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 + commit latency. Forces `--ioengine=psync --sync=1` (O_SYNC) regardless of dataset sync property. +- **dd** (`dd_profile`): dependency-free fallback; sequential write + uncached read + cached(RAM) + read, timed in Python. + +## Key implementation +- `detect_io_settings` auto-picks engine (`ENGINE_CANDIDATES` = io_uring→libaio→posixaio→sync) and + probes O_DIRECT support via tiny throwaway `fio_probe` jobs; falls back to buffered (with a red + banner warning) when O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly + async with `--direct=1`** → io_uring preferred. O_DIRECT bypasses the page cache so we measure + the device, not RAM (buffered reads can reflect ARC/page cache). +- **Everything parses fio `--output-format=json`** (robust, unit-safe metrics, no text scraping): + `run_fio` → `_aggregate` sums bw/IOPS across jobs, averages per-job p50 (median), takes worst-case + p99. fio is run *without* `--group_reporting`; we aggregate ourselves to avoid fio's group-merge + quirks. `FioResult` holds bw_mbps (decimal MB/s, matches CDM's SI figure), iops, p50/p99/mean (µs). +- One shared bench file `/speedtest-hd.bench`, reused across all runs, removed at end + (`_cleanup`). Footprint = `--size` (default 1G). +- Output styling: `Painter`/ANSI (basic 16-color), honors `NO_COLOR`/`FORCE_COLOR`/`TERM=dumb`, + per-stream isatty. **stdout = results (banner + tables); stderr = progress + verbose dumps**, each + with its own color flag so piping one works. `render_table` auto-sizes columns on *plain* text + (color applied after padding, so escapes don't skew widths). +- `Config` dataclass holds resolved settings; `confirm` prompts unless `--yes`. ## History / decisions -- Originated from the Ars Technica fio guide. Original 4K test used `numjobs=1 iodepth=1`, which - measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1, not - a bug. Refactored toward parallel/deep-queue tests to show real device capability, then fully - reshaped into the CrystalDiskMark profile above. `--simple` flag was removed; the table is now - the only fio output. -- Note: the current `RND4K` rows are `Q32T1` + `Q32T16`. CrystalDiskMark's actual latest default - profile is `Q32T16` + `Q1T1` (single-queue random is the meaningful low-end number). If aligning - strictly to CDM, the `Q32T1` row should become `Q1T1` (`iodepth=1 numjobs=1`). +- Originated from the Ars Technica fio guide. Original 4K test used QD1/1-job, + which measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1, + not a bug. Evolved to parallel/deep-queue CDM tests, then rewritten in Python for robust JSON + parsing, color, and the SLOG profile. +- `CDM_TESTS` is aligned to CrystalDiskMark's true default profile (`Q32T16` + `Q1T1`); an earlier + iteration used `Q32T1` + `Q32T16`. The TrueNAS case study in README.md was captured with that + earlier profile — its `Q32T1` vs `Q32T16` comparison is the reason the default changed. -## SLOG performance context (why `--slog` exists) +## SLOG performance context (why `--slog` exists; full case study in README.md) Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10` -= 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). Poor sync += 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). "Slow" sync writes were **CPU power management**, not the SLOG: -- Fix (biggest last): Dell BIOS profile DAPC → **Performance** (~2×); cstate kernel args; and the - big one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, which parked cores at - 1.2 GHz because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post - Init: `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`. -- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor). -- Diagnostic: `zpool iostat -vl 1` during fio showed the Optane `logs` vdev at ~90µs - disk_wait — proving the SLOG was fine and latency was upstream (CPU). -- Healthy Optane SLOG single-stream (T1) target: ~15–25k IOPS, p50 ~40–65µs. Much higher usually - = C-states / PCIe ASPM / BIOS power profile throttling. +- Fixes (biggest last): Dell BIOS DAPC → **Performance** (~2×); cstate/ASPM kernel args; and the big + one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, parking cores at 1.2 GHz + because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post Init: + `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`. +- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor); T16 → ~78k + IOPS / 319 MB/s, scaling regression gone. +- Decisive diagnostic: `zpool iostat -vl 1` during fio showed the Optane `logs` vdev at ~90µs + disk_wait — proving the SLOG was fine and latency was upstream (CPU). Also: large sync writes + bypass the SLOG (indirect ZIL >32K) — only small 4K sync writes exercise it, which is what `--slog` + does. Healthy Optane SLOG T1 target: ~15–25k IOPS, p50 ~40–65µs. diff --git a/README.md b/README.md index 8192c30..ba9149d 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync ## Features -- **CrystalDiskMark‑style profile** — `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T1`, `RND4K Q32T16`, each measured for Read **and** Write, reported in both **MB/s** and **IOPS**. +- **CrystalDiskMark‑style profile** — `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T16`, `RND4K Q1T1` (CrystalDiskMark's default profile), each measured for Read **and** Write, reported in both **MB/s** and **IOPS**. - **SLOG / sync‑write latency profile** (`--slog`) — synchronous 4K writes at T1/T4/T8/T16 reporting **IOPS, MB/s, and p50/p99 commit latency**. This is the load a ZFS SLOG actually sees. - **Auto‑detection** — picks the fastest available IO engine (`io_uring` → `libaio` → `posixaio` → `sync`) and probes whether `O_DIRECT` works on the filesystem, falling back to buffered IO when it doesn't (e.g. older OpenZFS, some NFS mounts). - **`dd` fallback** — if `fio` isn't present, runs a basic write/read test so you still get a number. @@ -30,9 +30,8 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync ## Requirements -- `bash` -- [`fio`](https://fio.readthedocs.io/) — recommended (`apt install fio` / `pacman -S fio`). Without it, the tool falls back to `dd`. -- `python3` — **only** required for `--slog` (used to parse `fio`'s JSON output for latency percentiles). The normal profile needs only `fio` + `grep`/`awk`. +- `python3` (3.7+) — the tool itself. It parses `fio`'s JSON output using only the standard library (no third‑party packages to install). +- [`fio`](https://fio.readthedocs.io/) — recommended (`apt install fio` / `pacman -S fio`). Without it, the tool falls back to a basic `dd` test. - `sudo` — `fio` is invoked via `sudo` so it can use `O_DIRECT` and flush device caches. --- @@ -40,7 +39,8 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync ## Usage ```bash -./speedtest-hd.sh [options] +./speedtest-hd.py [options] +# or: python3 speedtest-hd.py [options] ``` `` is the directory (or mount) to benchmark. Use `.` for the current directory. The tool creates a single test file (default 1 GiB) on the target and removes it afterward, so ensure enough free space. @@ -49,36 +49,44 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync | Invocation | What it does | |---|---| -| `./speedtest-hd.sh /mnt/disk` | Auto: uses `fio` if installed, else `dd` | -| `./speedtest-hd.sh /mnt/disk --fio` | Force the `fio` CrystalDiskMark‑style profile | -| `./speedtest-hd.sh /mnt/disk --dd` | Force the basic `dd` test | -| `./speedtest-hd.sh /mnt/disk --slog` | SLOG / sync‑write latency profile | +| `./speedtest-hd.py /mnt/disk` | Auto: uses `fio` if installed, else `dd` | +| `./speedtest-hd.py /mnt/disk --fio` | Force the `fio` CrystalDiskMark‑style profile | +| `./speedtest-hd.py /mnt/disk --dd` | Force the basic `dd` test | +| `./speedtest-hd.py /mnt/disk --slog` | SLOG / sync‑write latency profile | + +`--fio`, `--dd`, and `--slog` are mutually exclusive. ### Tuning flags | Flag | Effect | |---|---| -| `--engine=io_uring\|libaio\|posixaio\|sync` | Force a specific IO engine (default: auto) | +| `--engine {io_uring,libaio,posixaio,sync}` | Force a specific IO engine (default: auto) | | `--direct` | Force `O_DIRECT` (bypass page cache) | | `--buffered` | Force buffered IO (e.g. when `O_DIRECT` is unsupported) | -| `--runtime=SEC` | Seconds per run (default: 5, like CrystalDiskMark) | -| `--size=SIZE` | Test file size (default: `1g`) | +| `--runtime SEC` | Seconds per run (default: 5, like CrystalDiskMark) | +| `--size SIZE` | Test file size (default: `1g`) | | `--verbose` | Also print the full `fio` output for every run (summary table unchanged) | +| `-y`, `--yes` | Skip the confirmation prompt (for scripting/automation) | + +> All flags accept either `--flag value` or `--flag=value` (argparse). `--direct`/`--buffered` are mutually exclusive. ### Examples ```bash # CrystalDiskMark-style test of the current directory -./speedtest-hd.sh . +./speedtest-hd.py . # Larger file, longer runs, on an NVMe pool -./speedtest-hd.sh /mnt/nvmepool --runtime=10 --size=4g +./speedtest-hd.py /mnt/nvmepool --runtime=10 --size=4g # Buffered (e.g. an NFS share that doesn't support O_DIRECT) -./speedtest-hd.sh /mnt/nfsshare --buffered +./speedtest-hd.py /mnt/nfsshare --buffered # SLOG / sync latency profile, 30s per run -./speedtest-hd.sh /mnt/nvme-ultra-r10/vm-root --slog --runtime=30 +./speedtest-hd.py /mnt/nvme-ultra-r10/vm-root --slog --runtime=30 + +# Unattended (no prompt), forcing a specific engine +./speedtest-hd.py /mnt/nvmepool --yes --engine io_uring ``` > **Tip:** when running `--slog` against a ZFS dataset, watch the SLOG live in another shell: @@ -92,14 +100,16 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync ### CrystalDiskMark‑style profile +Representative output from a healthy local NVMe (your numbers will differ): + ``` +------------------+----------------+----------------+----------------+----------------+ | Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) | +------------------+----------------+----------------+----------------+----------------+ -| SEQ1M Q8T1 | 6873.00 | 9.30 | 6873 | 9 | -| SEQ1M Q1T1 | 1608.00 | 20.00 | 1608 | 20 | -| RND4K Q32T1 | 538.00 | 10.80 | 137728 | 2764 | -| RND4K Q32T16 | 689.00 | 261.00 | 176384 | 66816 | +| SEQ1M Q8T1 | 3650.00 | 3120.00 | 3482 | 2976 | +| SEQ1M Q1T1 | 2680.00 | 2510.00 | 2556 | 2394 | +| RND4K Q32T16 | 2950.00 | 2240.00 | 720215 | 546875 | +| RND4K Q1T1 | 78.00 | 64.00 | 19043 | 15625 | +------------------+----------------+----------------+----------------+----------------+ ``` @@ -126,8 +136,8 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync |---|---|---|---| | `SEQ1M Q8T1` | Sequential 1 MiB | 8 | 1 | | `SEQ1M Q1T1` | Sequential 1 MiB | 1 | 1 | -| `RND4K Q32T1` | Random 4 KiB | 32 | 1 | | `RND4K Q32T16` | Random 4 KiB | 32 | 16 | +| `RND4K Q1T1` | Random 4 KiB | 1 | 1 | `Q` = queue depth (`--iodepth`), `T` = threads (`--numjobs`). Note that `--iodepth` only produces real queue depth when the IO is truly asynchronous (async engine **and** `O_DIRECT`). On filesystems where that isn't available, queue depth effectively collapses toward 1 and concurrency comes only from threads (`T`). @@ -174,6 +184,8 @@ The standard benchmark looked alarming — huge reads, tiny writes: 9.3 MB/s sequential write on an Optane‑backed NVMe pool looks broken. +> **Note:** this investigation was captured with an earlier test profile that used `RND4K Q32T1` in place of today's `RND4K Q1T1`. The `Q32T1` vs `Q32T16` comparison below is exactly why the default later changed — see finding #3. + ### Investigation **1. The reads are RAM, not disk.** With a 1 GiB test file on ZFS, reads come straight from ARC (RAM cache). The huge read/write asymmetry is the tell — ignore the read column for judging the disks. @@ -287,5 +299,5 @@ The residual gap from raw Optane (~15 µs) is ZFS ZIL‑commit overhead plus the - **`sudo`** is used for `fio` so it can apply `O_DIRECT` and flush device write caches at the end of write runs (`--end_fsync=1`), so cached writes can't inflate results. - **`O_DIRECT` is auto‑detected.** If the banner shows `O_DIRECT: DISABLED (buffered ...)`, results may reflect the page cache (RAM) rather than the device. - **A single shared test file** is reused across runs to keep the footprint to one file. -- **`--slog` requires `python3`** (for JSON latency‑percentile parsing); the standard profile does not. +- **All profiles parse `fio`'s JSON output** (`--output-format=json`) with Python's standard library — robust, unit‑safe metrics with no fragile text scraping. - The `--slog` profile forces synchronous IO and is intended for ZFS ZIL / SLOG and other sync‑write (NFS/iSCSI/VM) investigations. diff --git a/speedtest-hd.py b/speedtest-hd.py index b2f6251..96090fc 100755 --- a/speedtest-hd.py +++ b/speedtest-hd.py @@ -8,7 +8,7 @@ for diagnosing ZFS ZIL performance (NFS / iSCSI / VM sync workloads). Three profiles -------------- * **cdm** — the four CrystalDiskMark default tests (``SEQ1M Q8T1``, ``SEQ1M - Q1T1``, ``RND4K Q32T1``, ``RND4K Q32T16``), each measured for read *and* + Q1T1``, ``RND4K Q32T16``, ``RND4K Q1T1``), each measured for read *and* write, reported in MB/s and IOPS. * **slog** — synchronous 4K random writes swept across thread counts, reporting IOPS, MB/s and p50/p99 *commit latency*. This is the load a ZFS SLOG sees. @@ -124,13 +124,13 @@ class CdmTest: return "write" if self.seq else "randwrite" -# The four CrystalDiskMark default tests, as data. Q = queue depth (iodepth), -# T = threads (numjobs). +# The four CrystalDiskMark default tests, as data, in CrystalDiskMark's own +# display order. Q = queue depth (iodepth), T = threads (numjobs). CDM_TESTS: tuple[CdmTest, ...] = ( CdmTest("SEQ1M Q8T1", bs="1m", iodepth=8, numjobs=1, seq=True), CdmTest("SEQ1M Q1T1", bs="1m", iodepth=1, numjobs=1, seq=True), - CdmTest("RND4K Q32T1", bs="4k", iodepth=32, numjobs=1, seq=False), CdmTest("RND4K Q32T16", bs="4k", iodepth=32, numjobs=16, seq=False), + CdmTest("RND4K Q1T1", bs="4k", iodepth=1, numjobs=1, seq=False), ) # Thread counts for the SLOG sweep. T1 is the headline single-stream latency; diff --git a/speedtest-hd.sh b/speedtest-hd.sh deleted file mode 100755 index 205fa0c..0000000 --- a/speedtest-hd.sh +++ /dev/null @@ -1,457 +0,0 @@ -#!/usr/bin/env bash - -# Robust HD/SSD/NVMe performance CLI utility -# A CrystalDiskMark-style storage benchmark for Linux, built on fio. -# Runs the exact same four tests CrystalDiskMark does (SEQ1M Q8T1, SEQ1M Q1T1, -# RND4K Q32T1, RND4K Q32T16), each measured for both Read and Write, and prints -# the results in a CrystalDiskMark-style table. -# Dependencies: fio (apt install fio / pacman -S fio). Falls back to dd if missing. -# See: https://cloud.google.com/compute/docs/disks/benchmarking-pd-performance -# See: https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/ -# mReschke 2024-01-18 - -# CLI Parameters -path="$1" -option="" -engine="" # IO engine; empty = auto-detect (io_uring > libaio > posixaio > sync) -direct="" # O_DIRECT; empty = auto-detect; 1 = bypass page cache, 0 = buffered -runtime=5 # seconds per measurement (CrystalDiskMark default is 5s) -size="1g" # test file size (CrystalDiskMark default is 1GiB) -verbose=0 # 1 = also print the full fio output for every run -for arg in "${@:2}"; do - case "$arg" in - --dd) option="--dd" ;; - --fio) option="--fio" ;; - --slog) option="--slog" ;; - --buffered) direct=0 ;; - --direct) direct=1 ;; - --verbose) verbose=1 ;; - --engine=*) engine="${arg#*=}" ;; - --direct=*) direct="${arg#*=}" ;; - --runtime=*) runtime="${arg#*=}" ;; - --size=*) size="${arg#*=}" ;; - esac -done - -# Main application flow -function main { - - # Show usage if no params - if [ ! "$path" ]; then - usage - fi - - # Understand . path - if [ "$path" == '.' ]; then - path=$(pwd) - fi - - # Check if path exists - if [ ! -e "$path" ]; then - echo "Path $path does not exist" - exit 1 - fi - - # Must type y or n THEN press enter (which I like better) - echo "NOTICE: ${size^^} free space on '$path' is required to perform the benchmark." - echo -n "Are you ready to start a storage benchmark against '$path' ? "; read answer - if [ "$answer" != "${answer#[Yy]}" ]; then - echo "Great! Starting benchmark now!" - else - echo "Ok, cancelled!" - exit 0 - fi - - # Use dd or fio based on param or defaults - if [ "$option" == "--dd" ]; then - dd_speedtest - elif [ "$option" == "--fio" ]; then - cdm_speedtest - elif [ "$option" == "--slog" ]; then - slog_speedtest - elif [ "$option" == "" ]; then - # If fio is installed, use it, else use dd - if ! command -v fio &> /dev/null; then - echo "" - echo "fio is not installed -- falling back to basic dd test." - echo "Install fio for the full CrystalDiskMark-style benchmark." - dd_speedtest - else - cdm_speedtest - fi - fi -} - -function fio_probe { - # Run a tiny throwaway fio job to see if a given (engine, direct) combo - # actually works on this path/filesystem. Returns 0 on success. This is how - # we stay accurate AND portable across ext4, xfs, btrfs, ZFS, NFS, etc. - # without the caller needing to know what each filesystem supports. - local eng="$1" dir="$2" - sudo fio --name=probe \ - --directory="$path" \ - --ioengine="$eng" \ - --rw=write --bs=4k --size=1m \ - --direct="$dir" \ - --time_based --runtime=1 \ - > /dev/null 2>&1 - local rc=$? - rm -rf "$path"/probe* 2>/dev/null - return $rc -} - -function detect_io_settings { - # Pick the fastest available IO engine unless the user forced one with - # --engine=. io_uring is the modern, lowest-overhead Linux engine; libaio - # is older (and only truly async with O_DIRECT); posixaio/sync are the - # portable fallbacks (eg some NFS mounts). - if [ -z "$engine" ]; then - for eng in io_uring libaio posixaio sync; do - if fio_probe "$eng" 0; then engine="$eng"; break; fi - done - [ -z "$engine" ] && engine="sync" - fi - - # Decide O_DIRECT unless forced with --direct/--buffered. O_DIRECT bypasses - # the OS page cache so we measure the device instead of RAM. Not every - # filesystem supports it (eg older OpenZFS < 2.3, some NFS mounts), so we - # probe and fall back to buffered rather than erroring out. - if [ -z "$direct" ]; then - if fio_probe "$engine" 1; then direct=1; else direct=0; fi - fi -} - -function fio_bw_mbps { - # Extract bandwidth in decimal MB/s (matching CrystalDiskMark) from a fio - # run. fio prints eg " READ: bw=3357MiB/s (3521MB/s), ..." -- we take the - # parenthetical SI figure and normalise kB/MB/GB into MB/s. - local out="$1" - local raw num unit - raw=$(echo "$out" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | head -n1) - [ -z "$raw" ] && { echo "0.00"; return; } - num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1) - unit=$(echo "$raw" | /usr/bin/grep -oP '[A-Za-z]+/s' | head -n1) - case "$unit" in - B/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000000}' ;; - kB/s|KB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000}' ;; - MB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;; - GB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n*1000}' ;; - *) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;; - esac -} - -function fio_iops { - # Extract IOPS from a fio run. fio prints eg " read: IOPS=12.3k, BW=..." -- - # the value may carry a k/M suffix, which we expand to a whole number. - local out="$1" - local raw num unit - raw=$(echo "$out" | /usr/bin/grep -oP 'IOPS=\K[0-9.]+[kKmM]?' | head -n1) - [ -z "$raw" ] && { echo "0"; return; } - num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1) - unit=$(echo "$raw" | /usr/bin/grep -oP '[kKmM]' | head -n1) - case "$unit" in - k|K) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000}' ;; - m|M) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000000}' ;; - *) awk -v n="$num" 'BEGIN{printf "%.0f", n}' ;; - esac -} - -function run_fio { - # run_fio -> echoes "MB/s IOPS" (space-sep) - # A single shared 1GB test file is reused across every run (laid out once), - # which keeps the footprint at one file and avoids re-creating it each time. - local rw="$1" bs="$2" qd="$3" jobs="$4" - local args=( - --name=speedtest - --filename="$benchfile" - --ioengine="$engine" - --direct="$direct" - --rw="$rw" - --bs="$bs" - --size="$size" - --numjobs="$jobs" - --iodepth="$qd" - --time_based --runtime="$runtime" - --group_reporting=1 - ) - # Flush device cache at the end of write tests so cached writes can't lie. - case "$rw" in - *write) args+=(--end_fsync=1) ;; - esac - # Capture the full fio output. In verbose mode we also keep fio's stderr - # (2>&1); otherwise we discard it. Either way only the parsed bandwidth is - # echoed to stdout, so the captured value stays clean. - local out - if [ "$verbose" = 1 ]; then - out=$(sudo fio "${args[@]}" 2>&1) - else - out=$(sudo fio "${args[@]}" 2>/dev/null) - fi - # Dump the raw fio output (to stderr, so it never pollutes the captured - # values on stdout) when running verbose. - if [ "$verbose" = 1 ]; then - echo " --- fio: $rw bs=$bs iodepth=$qd numjobs=$jobs ---" >&2 - echo "$out" >&2 - echo "" >&2 - fi - echo "$(fio_bw_mbps "$out") $(fio_iops "$out")" -} - -# Table drawing ------------------------------------------------------------- -# Label cell is "| %-16s |" (18 dashes); the four numeric cells are "| %14s |" -# (16 dashes each). -TBL_SEG_LBL="------------------" # 18 dashes -> matches "| %-16s |" -TBL_SEG_NUM="----------------" # 16 dashes -> matches "| %14s |" -function tbl_rule { - printf "+%s+%s+%s+%s+%s+\n" "$TBL_SEG_LBL" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM" -} -function tbl_row { - # tbl_row - # (col1 left-aligned, the four numeric cols right-aligned) - printf "| %-16s | %14s | %14s | %14s | %14s |\n" "$1" "$2" "$3" "$4" "$5" -} - -# Five-column table for the --slog latency profile. -# "| %-16s |" -> 18 dashes ; "| %12s |" -> 14 dashes -SLOG_SEG16="------------------" # 18 dashes -SLOG_SEG12="--------------" # 14 dashes -function slog_rule { - printf "+%s+%s+%s+%s+%s+\n" "$SLOG_SEG16" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12" -} -function slog_row { - # slog_row (col1 left, rest right-aligned) - printf "| %-16s | %12s | %12s | %12s | %12s |\n" "$1" "$2" "$3" "$4" "$5" -} - -function run_fio_sync { - # run_fio_sync - # -> echoes "IOPS MB/s p50_us p99_us mean_us" (one space-separated line) - # Forces synchronous IO (--sync=1 => O_SYNC) with the portable psync engine so - # every write is a ZIL commit -- i.e. it exercises the SLOG exactly the way a - # sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own - # sync property. We deliberately DON'T use --group_reporting here: each job is - # reported independently in the JSON and we aggregate ourselves (sum IOPS/bw, - # average p50, take the worst p99), which sidesteps fio's group-merge quirks. - local bs="$1" jobs="$2" - local args=( - --name=slog - --filename="$benchfile" - --ioengine=psync - --rw=randwrite - --bs="$bs" - --size="$size" - --numjobs="$jobs" - --sync=1 - --time_based --runtime="$runtime" - --output-format=json - ) - local out - out=$(sudo fio "${args[@]}" 2>/dev/null) - if [ "$verbose" = 1 ]; then - echo " --- fio json: randwrite bs=$bs numjobs=$jobs sync=1 ---" >&2 - echo "$out" >&2 - echo "" >&2 - fi - # Parse the JSON: sum IOPS and bandwidth across jobs, aggregate completion - # latency percentiles (clat is in ns -> convert to us). bw_bytes is bytes/s. - echo "$out" | python3 -c ' -import json, sys -try: - d = json.load(sys.stdin) - jobs = d["jobs"] -except Exception: - print("0 0.00 0.0 0.0 0.0"); sys.exit(0) -iops = sum(j["write"]["iops"] for j in jobs) -mbps = sum(j["write"]["bw_bytes"] for j in jobs) / 1e6 -def pctl(j, p): - return j["write"]["clat_ns"]["percentile"].get(p) -p50 = [pctl(j, "50.000000") for j in jobs if pctl(j, "50.000000") is not None] -p99 = [pctl(j, "99.000000") for j in jobs if pctl(j, "99.000000") is not None] -mean = [j["write"]["clat_ns"]["mean"] for j in jobs] -p50_us = (sum(p50)/len(p50))/1000.0 if p50 else 0.0 # average of per-job medians -p99_us = (max(p99))/1000.0 if p99 else 0.0 # worst-case tail across jobs -mean_us = (sum(mean)/len(mean))/1000.0 if mean else 0.0 -print("%.0f %.2f %.1f %.1f %.1f" % (iops, mbps, p50_us, p99_us, mean_us)) -' -} - -function slog_speedtest { - # SLOG / sync-write latency profile. Detect engine/direct only for the banner; - # the actual runs always force psync + O_SYNC (see run_fio_sync). - detect_io_settings - benchfile="$path/speedtest-hd.bench" - - if ! command -v fio &> /dev/null; then - echo "ERROR: --slog requires fio (apt install fio / pacman -S fio)." >&2 - exit 1 - fi - if ! command -v python3 &> /dev/null; then - echo "ERROR: --slog requires python3 (used to parse fio JSON latency percentiles)." >&2 - exit 1 - fi - - echo "" - echo " speedtest-hd : SLOG / sync-write latency profile" - echo " Target : $path" - echo " Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine" - echo " Profile : runtime=${runtime}s/run size=${size^^}" - echo " Note : every write is a synchronous ZIL commit -- this is the load" - echo " your SLOG actually sees. Watch it live in another shell with:" - echo " zpool iostat -vl 1" - echo "" - - # T1 is the headline single-stream latency; the sweep shows how the SLOG - # scales as concurrent sync writers (NFS/iSCSI/VM threads) pile on. Run every - # measurement FIRST (progress goes to stderr), collect the results, then draw - # the whole table in one block so the "measuring..." lines can't interleave - # into the middle of the table. - local jobs_list="1 4 8 16" - local -a rows - local j res iops bw p50 p99 mean - for j in $jobs_list; do - echo " measuring 4K sync randwrite T$j ..." >&2 - res=$(run_fio_sync 4k "$j") - read -r iops bw p50 p99 mean <<< "$res" - rows+=("$(slog_row "4K sync T$j" "$iops" "$bw" "$p50" "$p99")") - done - - rm -f "$benchfile" - - # Render the assembled table - echo "" - slog_rule - slog_row "Test" "IOPS" "MB/s" "p50 lat(us)" "p99 lat(us)" - slog_rule - printf '%s\n' "${rows[@]}" - slog_rule - echo "" - echo " Healthy Optane SLOG (eg P1600X) single-stream (T1) target:" - echo " ~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means" - echo " CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling." - echo "" -} - -function cdm_speedtest { - # Auto-detect the best engine and whether O_DIRECT works on this path. - detect_io_settings - benchfile="$path/speedtest-hd.bench" - - local direct_label="enabled (device)" - [ "$direct" != 1 ] && direct_label="DISABLED (buffered -- may reflect RAM cache!)" - - # Banner - echo "" - echo " speedtest-hd : CrystalDiskMark-style storage benchmark" - echo " Target : $path" - echo " Engine : $engine O_DIRECT: $direct_label" - echo " Profile : size=${size^^} runtime=${runtime}s/run (8 runs)" - echo "" - - # The four CrystalDiskMark default tests. Q = queue depth (--iodepth), - # T = threads (--numjobs). The first three are T1 (single thread); the last - # is T16. A single shared test file is reused -- with numjobs>1 every job - # issues IO against the same file, which is fine. - # Each run now returns both MB/s and IOPS ("bw iops"), so we capture the pair. - # label read-rw write-rw bs Q T - echo " measuring SEQ1M Q8T1 ..." >&2 - local s8r_bw s8r_io; read -r s8r_bw s8r_io <<< "$(run_fio read 1m 8 1)" - local s8w_bw s8w_io; read -r s8w_bw s8w_io <<< "$(run_fio write 1m 8 1)" - echo " measuring SEQ1M Q1T1 ..." >&2 - local s1r_bw s1r_io; read -r s1r_bw s1r_io <<< "$(run_fio read 1m 1 1)" - local s1w_bw s1w_io; read -r s1w_bw s1w_io <<< "$(run_fio write 1m 1 1)" - echo " measuring RND4K Q32T1 ..." >&2 - local r32r_bw r32r_io; read -r r32r_bw r32r_io <<< "$(run_fio randread 4k 32 1)" - local r32w_bw r32w_io; read -r r32w_bw r32w_io <<< "$(run_fio randwrite 4k 32 1)" - echo " measuring RND4K Q32T16 ..." >&2 - local r1r_bw r1r_io; read -r r1r_bw r1r_io <<< "$(run_fio randread 4k 32 16)" - local r1w_bw r1w_io; read -r r1w_bw r1w_io <<< "$(run_fio randwrite 4k 32 16)" - - # Cleanup the shared test file - rm -f "$benchfile" - - # Render the CrystalDiskMark-style table - echo "" - tbl_rule - tbl_row "Test" "Read (MB/s)" "Write (MB/s)" "Read (IOPS)" "Write (IOPS)" - tbl_rule - tbl_row "SEQ1M Q8T1" "$s8r_bw" "$s8w_bw" "$s8r_io" "$s8w_io" - tbl_row "SEQ1M Q1T1" "$s1r_bw" "$s1w_bw" "$s1r_io" "$s1w_io" - tbl_row "RND4K Q32T1" "$r32r_bw" "$r32w_bw" "$r32r_io" "$r32w_io" - tbl_row "RND4K Q32T16" "$r1r_bw" "$r1w_bw" "$r1r_io" "$r1w_io" - tbl_rule - echo "" -} - -function dd_speedtest { - # Basic HD speed test using dd (fallback when fio is not installed) - # mReschke 2017-07-11 - local file ddsize - file=$path/bigfile - ddsize=1024 - - echo "Running dd based HD/SSD/NVMe Benchmarks" - echo "---------------------------------------" - - printf "Cached write speed...\n" - dd if=/dev/zero of=$file bs=1M count=$ddsize - - printf "\nUncached write speed...\n" - dd if=/dev/zero of=$file bs=1M count=$ddsize conv=fdatasync,notrunc - - printf "\nUncached read speed...\n" - echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null - dd if=$file of=/dev/null bs=1M count=$ddsize - - printf "\nCached read speed...\n" - dd if=$file of=/dev/null bs=1M count=$ddsize - - rm $file - printf "\nDone\n" -} - -# Show help and usage information -function usage { - echo "speedtest-hd : CrystalDiskMark-style storage benchmark" - echo " Runs the same four tests as CrystalDiskMark (each Read + Write):" - echo " SEQ1M Q8T1 Sequential 1MiB, queue depth 8, 1 thread" - echo " SEQ1M Q1T1 Sequential 1MiB, queue depth 1, 1 thread" - echo " RND4K Q32T1 Random 4KiB, queue depth 32, 1 thread" - echo " RND4K Q32T16 Random 4KiB, queue depth 32, 16 threads" - echo " Uses fio if installed (recommended), else basic dd." - echo "mReschke 2024-01-18" - echo "" - echo "NOTICE: creates one test file (default 1GB) on the destination disk." - echo "Please ensure you have write access with enough free space." - echo "" - echo "Usage:" - echo " Auto (fio if installed, else dd):" - echo " ./speedtest-hd /mnt/somedisk" - echo " ./speedtest-hd ." - echo "" - echo " Force fio / force dd:" - echo " ./speedtest-hd /mnt/somedisk --fio" - echo " ./speedtest-hd /mnt/somedisk --dd" - echo "" - echo " SLOG / sync-write latency profile (ZFS ZIL, NFS/iSCSI/VM sync IO):" - echo " ./speedtest-hd /mnt/zpool/dataset --slog" - echo " 4K synchronous writes at T1/T4/T8/T16; reports IOPS, MB/s, p50/p99" - echo " commit latency. Requires fio + python3. (Tip: watch the SLOG with" - echo " 'zpool iostat -vl 1' in another shell.)" - echo "" - echo " Tuning flags (auto-detected by default, override as needed):" - echo " --engine=io_uring|libaio|posixaio|sync IO engine (default: auto)" - echo " --direct Force O_DIRECT (bypass page cache)" - echo " --buffered Force buffered IO (eg if O_DIRECT is unsupported)" - echo " --runtime=SEC Seconds per run (default: 5, like CrystalDiskMark)" - echo " --size=SIZE Test file size (default: 1g)" - echo " --verbose Print full fio output for every run (summary table unchanged)" - echo " Examples:" - echo " ./speedtest-hd /mnt/nvmepool --runtime=10 --size=4g" - echo " ./speedtest-hd /mnt/nfsshare --buffered" - echo " ./speedtest-hd /mnt/nvmepool --verbose" - echo " ./speedtest-hd /mnt/nvme-ultra-r10/vm-root --slog --runtime=30" - exit 0 -} - -# Go -main