mreschke/speedtest-hd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove the old bash script and update README.md and CLAUDE.md

Browse Source
This commit is contained in:
Matthew Reschke
2026-06-21 21:16:47 -06:00
parent 5561a95a94
commit 363379b2c2
4 changed files with 100 additions and 535 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+62 -52
View File
@@ -1,65 +1,75 @@
# speedtest-hd
Single-file Bash CLI (`speedtest-hd.sh`) that benchmarks any mounted path (HDD/SSD/NVMe,
local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII table. Falls back
to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount" disk tester — not
a ZFS/NVMe-specific tool.
`speedtest-hd.py` — a single-file Python 3 CLI that benchmarks any mounted path
(HDD/SSD/NVMe, local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII
table. Falls back to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount"
disk tester — not a ZFS/NVMe-specific tool. `README.md` is kept in sync with the CLI.
## Usage
```
./speedtest-hd.sh <path> [--fio|--dd|--slog] [flags]
./speedtest-hd.py <path> [--fio|--dd|--slog] [flags] # or: python3 speedtest-hd.py ...
```
- Default: fio if installed, else dd. `--fio` forces CDM mode, `--slog` forces sync-latency mode.
- Flags: `--engine=io_uring|libaio|posixaio|sync` (default auto), `--direct`/`--buffered`
(default auto-probe), `--runtime=SEC` (default 5), `--size=SIZE` (default 1g), `--verbose`
(also dumps raw fio output to stderr).
- Default mode: fio if installed (→ cdm), else dd. `--fio`/`--dd`/`--slog` force a mode (mutually exclusive).
- Flags: `--engine {io_uring,libaio,posixaio,sync}` (default auto), `--direct`/`--buffered`
(default auto-probe), `--runtime SEC` (default 5), `--size SIZE` (default 1g), `--verbose`
(dump raw fio output to stderr), `-y`/`--yes` (skip confirm prompt). argparse → both
`--flag value` and `--flag=value` work.
## Modes & layout
- **CDM mode** (`cdm_speedtest`): runs CrystalDiskMark's default tests, each read+write,
prints a 5-col table (Read/Write MB/s, Read/Write IOPS). Tests: `SEQ1M Q8T1`, `SEQ1M Q1T1`,
`RND4K Q32T1`, `RND4K Q32T16`. Q = `--iodepth`, T = `--numjobs`.
- **SLOG mode** (`slog_speedtest`): 4K synchronous randwrite sweep at T1/T4/T8/T16 to profile
ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 commit
latency. Requires `fio` + `python3` (JSON percentile parsing).
- **dd mode** (`dd_speedtest`): legacy cached/uncached read/write fallback.
## Requirements
- **python3 (3.7+)** — required for everything; uses only the standard library (argparse, json,
dataclasses, statistics). No third-party packages.
- **fio** — recommended; without it, auto mode falls back to `dd`. `--fio`/`--slog` hard-require it.
- **sudo** — fio is run via `sudo` (for O_DIRECT + device-cache flush).
## Key implementation notes
- `detect_io_settings` auto-picks the engine (io_uring → libaio → posixaio → sync) and probes
whether O_DIRECT works, via tiny throwaway `fio_probe` jobs. Falls back to buffered with a
warning if O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly async
with `--direct=1`** — that's why io_uring is preferred. O_DIRECT bypasses the page cache so we
measure the device, not RAM (buffered results, esp. reads, can reflect ARC/page cache).
- One shared test file `<path>/speedtest-hd.bench` is reused across all runs (laid out once),
removed at the end. Footprint = `--size` (default 1G), matching the startup notice.
- `run_fio` (CDM) returns `"MB/s IOPS"`; `run_fio_sync` (SLOG) forces `--ioengine=psync --sync=1`
(O_SYNC) so every write is a ZIL commit regardless of dataset sync property, and parses fio
**JSON** to aggregate across jobs (sum IOPS/bw, avg p50, worst-case p99). CDM mode parses
fio's text output: `fio_bw_mbps` takes the parenthetical **SI MB/s** (matches CDM's decimal
MB/s, normalizes kB/MB/GB); `fio_iops` expands k/M suffixes.
- Write tests append `--end_fsync=1` so cached writes can't inflate numbers.
- ASCII tables: cell formats and dash-segment widths must stay in sync (`tbl_*` = 18/16 dashes,
`slog_*` = 18/14). Verify alignment after editing.
## Modes / profiles (functions)
- **cdm** (`cdm_profile`): CrystalDiskMark's default tests, each read+write, 5-col table
(Read/Write MB/s, Read/Write IOPS). Tests defined as data in `CDM_TESTS` (CrystalDiskMark's
default profile, in its display order): `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T16`, `RND4K Q1T1`.
Q=iodepth, T=numjobs. Write runs use `end_fsync`.
- **slog** (`slog_profile`): 4K synchronous randwrite sweep at T1/T4/T8/T16 (`SLOG_THREADS`) to
profile ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99
commit latency. Forces `--ioengine=psync --sync=1` (O_SYNC) regardless of dataset sync property.
- **dd** (`dd_profile`): dependency-free fallback; sequential write + uncached read + cached(RAM)
read, timed in Python.
## Key implementation
- `detect_io_settings` auto-picks engine (`ENGINE_CANDIDATES` = io_uring→libaio→posixaio→sync) and
probes O_DIRECT support via tiny throwaway `fio_probe` jobs; falls back to buffered (with a red
banner warning) when O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly
async with `--direct=1`** → io_uring preferred. O_DIRECT bypasses the page cache so we measure
the device, not RAM (buffered reads can reflect ARC/page cache).
- **Everything parses fio `--output-format=json`** (robust, unit-safe metrics, no text scraping):
`run_fio` → `_aggregate` sums bw/IOPS across jobs, averages per-job p50 (median), takes worst-case
p99. fio is run *without* `--group_reporting`; we aggregate ourselves to avoid fio's group-merge
quirks. `FioResult` holds bw_mbps (decimal MB/s, matches CDM's SI figure), iops, p50/p99/mean (µs).
- One shared bench file `<path>/speedtest-hd.bench`, reused across all runs, removed at end
(`_cleanup`). Footprint = `--size` (default 1G).
- Output styling: `Painter`/ANSI (basic 16-color), honors `NO_COLOR`/`FORCE_COLOR`/`TERM=dumb`,
per-stream isatty. **stdout = results (banner + tables); stderr = progress + verbose dumps**, each
with its own color flag so piping one works. `render_table` auto-sizes columns on *plain* text
(color applied after padding, so escapes don't skew widths).
- `Config` dataclass holds resolved settings; `confirm` prompts unless `--yes`.
## History / decisions
- Originated from the Ars Technica fio guide. Original 4K test used `numjobs=1 iodepth=1`, which
measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1, not
a bug. Refactored toward parallel/deep-queue tests to show real device capability, then fully
reshaped into the CrystalDiskMark profile above. `--simple` flag was removed; the table is now
the only fio output.
- Note: the current `RND4K` rows are `Q32T1` + `Q32T16`. CrystalDiskMark's actual latest default
profile is `Q32T16` + `Q1T1` (single-queue random is the meaningful low-end number). If aligning
strictly to CDM, the `Q32T1` row should become `Q1T1` (`iodepth=1 numjobs=1`).
- Originated from the Ars Technica fio guide. Original 4K test used QD1/1-job,
which measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1,
not a bug. Evolved to parallel/deep-queue CDM tests, then rewritten in Python for robust JSON
parsing, color, and the SLOG profile.
- `CDM_TESTS` is aligned to CrystalDiskMark's true default profile (`Q32T16` + `Q1T1`); an earlier
iteration used `Q32T1` + `Q32T16`. The TrueNAS case study in README.md was captured with that
earlier profile — its `Q32T1` vs `Q32T16` comparison is the reason the default changed.
## SLOG performance context (why `--slog` exists)
## SLOG performance context (why `--slog` exists; full case study in README.md)
Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10`
= 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). Poor sync
= 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). "Slow" sync
writes were **CPU power management**, not the SLOG:
- Fix (biggest last): Dell BIOS profile DAPC → **Performance** (~2×); cstate kernel args; and the
big one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, which parked cores at
1.2 GHz because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post
Init: `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`.
- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor).
- Diagnostic: `zpool iostat -vl <pool> 1` during fio showed the Optane `logs` vdev at ~90µs
disk_wait — proving the SLOG was fine and latency was upstream (CPU).
- Healthy Optane SLOG single-stream (T1) target: ~15–25k IOPS, p50 ~40–65µs. Much higher usually
= C-states / PCIe ASPM / BIOS power profile throttling.
- Fixes (biggest last): Dell BIOS DAPC → **Performance** (~2×); cstate/ASPM kernel args; and the big
one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, parking cores at 1.2 GHz
because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post Init:
`echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`.
- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor); T16 → ~78k
IOPS / 319 MB/s, scaling regression gone.
- Decisive diagnostic: `zpool iostat -vl <pool> 1` during fio showed the Optane `logs` vdev at ~90µs
disk_wait — proving the SLOG was fine and latency was upstream (CPU). Also: large sync writes
bypass the SLOG (indirect ZIL >32K) — only small 4K sync writes exercise it, which is what `--slog`
does. Healthy Optane SLOG T1 target: ~15–25k IOPS, p50 ~40–65µs.
README.md
+34 -22
View File
@@ -20,7 +20,7 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
## Features
- **CrystalDiskMark‑style profile** — `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T1`, `RND4K Q32T16`, each measured for Read **and** Write, reported in both **MB/s** and **IOPS**.
- **CrystalDiskMark‑style profile** — `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T16`, `RND4K Q1T1` (CrystalDiskMark's default profile), each measured for Read **and** Write, reported in both **MB/s** and **IOPS**.
- **SLOG / sync‑write latency profile** (`--slog`) — synchronous 4K writes at T1/T4/T8/T16 reporting **IOPS, MB/s, and p50/p99 commit latency**. This is the load a ZFS SLOG actually sees.
- **Auto‑detection** — picks the fastest available IO engine (`io_uring` → `libaio` → `posixaio` → `sync`) and probes whether `O_DIRECT` works on the filesystem, falling back to buffered IO when it doesn't (e.g. older OpenZFS, some NFS mounts).
- **`dd` fallback** — if `fio` isn't present, runs a basic write/read test so you still get a number.
@@ -30,9 +30,8 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
## Requirements
- `bash`
- [`fio`](https://fio.readthedocs.io/) — recommended (`apt install fio` / `pacman -S fio`). Without it, the tool falls back to `dd`.
- `python3` — **only** required for `--slog` (used to parse `fio`'s JSON output for latency percentiles). The normal profile needs only `fio` + `grep`/`awk`.
- `python3` (3.7+) — the tool itself. It parses `fio`'s JSON output using only the standard library (no third‑party packages to install).
- [`fio`](https://fio.readthedocs.io/) — recommended (`apt install fio` / `pacman -S fio`). Without it, the tool falls back to a basic `dd` test.
- `sudo` — `fio` is invoked via `sudo` so it can use `O_DIRECT` and flush device caches.
---
@@ -40,7 +39,8 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
## Usage
```bash
./speedtest-hd.sh <path> [options]
./speedtest-hd.py <path> [options]
# or: python3 speedtest-hd.py <path> [options]
```
`<path>` is the directory (or mount) to benchmark. Use `.` for the current directory. The tool creates a single test file (default 1 GiB) on the target and removes it afterward, so ensure enough free space.
@@ -49,36 +49,44 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
| Invocation | What it does |
|---|---|
| `./speedtest-hd.sh /mnt/disk` | Auto: uses `fio` if installed, else `dd` |
| `./speedtest-hd.sh /mnt/disk --fio` | Force the `fio` CrystalDiskMark‑style profile |
| `./speedtest-hd.sh /mnt/disk --dd` | Force the basic `dd` test |
| `./speedtest-hd.sh /mnt/disk --slog` | SLOG / sync‑write latency profile |
| `./speedtest-hd.py /mnt/disk` | Auto: uses `fio` if installed, else `dd` |
| `./speedtest-hd.py /mnt/disk --fio` | Force the `fio` CrystalDiskMark‑style profile |
| `./speedtest-hd.py /mnt/disk --dd` | Force the basic `dd` test |
| `./speedtest-hd.py /mnt/disk --slog` | SLOG / sync‑write latency profile |
`--fio`, `--dd`, and `--slog` are mutually exclusive.
### Tuning flags
| Flag | Effect |
|---|---|
| `--engine=io_uring\|libaio\|posixaio\|sync` | Force a specific IO engine (default: auto) |
| `--engine {io_uring,libaio,posixaio,sync}` | Force a specific IO engine (default: auto) |
| `--direct` | Force `O_DIRECT` (bypass page cache) |
| `--buffered` | Force buffered IO (e.g. when `O_DIRECT` is unsupported) |
| `--runtime=SEC` | Seconds per run (default: 5, like CrystalDiskMark) |
| `--size=SIZE` | Test file size (default: `1g`) |
| `--runtime SEC` | Seconds per run (default: 5, like CrystalDiskMark) |
| `--size SIZE` | Test file size (default: `1g`) |
| `--verbose` | Also print the full `fio` output for every run (summary table unchanged) |
| `-y`, `--yes` | Skip the confirmation prompt (for scripting/automation) |
> All flags accept either `--flag value` or `--flag=value` (argparse). `--direct`/`--buffered` are mutually exclusive.
### Examples
```bash
# CrystalDiskMark-style test of the current directory
./speedtest-hd.sh .
./speedtest-hd.py .
# Larger file, longer runs, on an NVMe pool
./speedtest-hd.sh /mnt/nvmepool --runtime=10 --size=4g
./speedtest-hd.py /mnt/nvmepool --runtime=10 --size=4g
# Buffered (e.g. an NFS share that doesn't support O_DIRECT)
./speedtest-hd.sh /mnt/nfsshare --buffered
./speedtest-hd.py /mnt/nfsshare --buffered
# SLOG / sync latency profile, 30s per run
./speedtest-hd.sh /mnt/nvme-ultra-r10/vm-root --slog --runtime=30
./speedtest-hd.py /mnt/nvme-ultra-r10/vm-root --slog --runtime=30
# Unattended (no prompt), forcing a specific engine
./speedtest-hd.py /mnt/nvmepool --yes --engine io_uring
```
> **Tip:** when running `--slog` against a ZFS dataset, watch the SLOG live in another shell:
@@ -92,14 +100,16 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
### CrystalDiskMark‑style profile
Representative output from a healthy local NVMe (your numbers will differ):
```
+------------------+----------------+----------------+----------------+----------------+
| Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) |
+------------------+----------------+----------------+----------------+----------------+
| SEQ1M Q8T1 | 6873.00 | 9.30 | 6873 | 9 |
| SEQ1M Q1T1 | 1608.00 | 20.00 | 1608 | 20 |
| RND4K Q32T1 | 538.00 | 10.80 | 137728 | 2764 |
| RND4K Q32T16 | 689.00 | 261.00 | 176384 | 66816 |
| SEQ1M Q8T1 | 3650.00 | 3120.00 | 3482 | 2976 |
| SEQ1M Q1T1 | 2680.00 | 2510.00 | 2556 | 2394 |
| RND4K Q32T16 | 2950.00 | 2240.00 | 720215 | 546875 |
| RND4K Q1T1 | 78.00 | 64.00 | 19043 | 15625 |
+------------------+----------------+----------------+----------------+----------------+
```
@@ -126,8 +136,8 @@ It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync
|---|---|---|---|
| `SEQ1M Q8T1` | Sequential 1 MiB | 8 | 1 |
| `SEQ1M Q1T1` | Sequential 1 MiB | 1 | 1 |
| `RND4K Q32T1` | Random 4 KiB | 32 | 1 |
| `RND4K Q32T16` | Random 4 KiB | 32 | 16 |
| `RND4K Q1T1` | Random 4 KiB | 1 | 1 |
`Q` = queue depth (`--iodepth`), `T` = threads (`--numjobs`). Note that `--iodepth` only produces real queue depth when the IO is truly asynchronous (async engine **and** `O_DIRECT`). On filesystems where that isn't available, queue depth effectively collapses toward 1 and concurrency comes only from threads (`T`).
@@ -174,6 +184,8 @@ The standard benchmark looked alarming — huge reads, tiny writes:
9.3 MB/s sequential write on an Optane‑backed NVMe pool looks broken.
> **Note:** this investigation was captured with an earlier test profile that used `RND4K Q32T1` in place of today's `RND4K Q1T1`. The `Q32T1` vs `Q32T16` comparison below is exactly why the default later changed — see finding #3.
### Investigation
**1. The reads are RAM, not disk.** With a 1 GiB test file on ZFS, reads come straight from ARC (RAM cache). The huge read/write asymmetry is the tell — ignore the read column for judging the disks.
@@ -287,5 +299,5 @@ The residual gap from raw Optane (~15 µs) is ZFS ZIL‑commit overhead plus the
- **`sudo`** is used for `fio` so it can apply `O_DIRECT` and flush device write caches at the end of write runs (`--end_fsync=1`), so cached writes can't inflate results.
- **`O_DIRECT` is auto‑detected.** If the banner shows `O_DIRECT: DISABLED (buffered ...)`, results may reflect the page cache (RAM) rather than the device.
- **A single shared test file** is reused across runs to keep the footprint to one file.
- **`--slog` requires `python3`** (for JSON latency‑percentile parsing); the standard profile does not.
- **All profiles parse `fio`'s JSON output** (`--output-format=json`) with Python's standard library — robust, unit‑safe metrics with no fragile text scraping.
- The `--slog` profile forces synchronous IO and is intended for ZFS ZIL / SLOG and other sync‑write (NFS/iSCSI/VM) investigations.
speedtest-hd.py
+4 -4
View File
@@ -8,7 +8,7 @@ for diagnosing ZFS ZIL performance (NFS / iSCSI / VM sync workloads).
Three profiles
--------------
* **cdm** — the four CrystalDiskMark default tests (``SEQ1M Q8T1``, ``SEQ1M
Q1T1``, ``RND4K Q32T1``, ``RND4K Q32T16``), each measured for read *and*
Q1T1``, ``RND4K Q32T16``, ``RND4K Q1T1``), each measured for read *and*
write, reported in MB/s and IOPS.
* **slog** — synchronous 4K random writes swept across thread counts, reporting
IOPS, MB/s and p50/p99 *commit latency*. This is the load a ZFS SLOG sees.
@@ -124,13 +124,13 @@ class CdmTest:
return "write" if self.seq else "randwrite"
# The four CrystalDiskMark default tests, as data. Q = queue depth (iodepth),
# T = threads (numjobs).
# The four CrystalDiskMark default tests, as data, in CrystalDiskMark's own
# display order. Q = queue depth (iodepth), T = threads (numjobs).
CDM_TESTS: tuple[CdmTest, ...] = (
CdmTest("SEQ1M Q8T1", bs="1m", iodepth=8, numjobs=1, seq=True),
CdmTest("SEQ1M Q1T1", bs="1m", iodepth=1, numjobs=1, seq=True),
CdmTest("RND4K Q32T1", bs="4k", iodepth=32, numjobs=1, seq=False),
CdmTest("RND4K Q32T16", bs="4k", iodepth=32, numjobs=16, seq=False),
CdmTest("RND4K Q1T1", bs="4k", iodepth=1, numjobs=1, seq=False),
)
# Thread counts for the SLOG sweep. T1 is the headline single-stream latency;
speedtest-hd.sh
-457
View File
@@ -1,457 +0,0 @@
#!/usr/bin/env bash
# Robust HD/SSD/NVMe performance CLI utility
# A CrystalDiskMark-style storage benchmark for Linux, built on fio.
# Runs the exact same four tests CrystalDiskMark does (SEQ1M Q8T1, SEQ1M Q1T1,
# RND4K Q32T1, RND4K Q32T16), each measured for both Read and Write, and prints
# the results in a CrystalDiskMark-style table.
# Dependencies: fio (apt install fio / pacman -S fio). Falls back to dd if missing.
# See: https://cloud.google.com/compute/docs/disks/benchmarking-pd-performance
# See: https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/
# mReschke 2024-01-18
# CLI Parameters
path="$1"
option=""
engine="" # IO engine; empty = auto-detect (io_uring > libaio > posixaio > sync)
direct="" # O_DIRECT; empty = auto-detect; 1 = bypass page cache, 0 = buffered
runtime=5 # seconds per measurement (CrystalDiskMark default is 5s)
size="1g" # test file size (CrystalDiskMark default is 1GiB)
verbose=0 # 1 = also print the full fio output for every run
for arg in "${@:2}"; do
case "$arg" in
--dd) option="--dd" ;;
--fio) option="--fio" ;;
--slog) option="--slog" ;;
--buffered) direct=0 ;;
--direct) direct=1 ;;
--verbose) verbose=1 ;;
--engine=*) engine="${arg#*=}" ;;
--direct=*) direct="${arg#*=}" ;;
--runtime=*) runtime="${arg#*=}" ;;
--size=*) size="${arg#*=}" ;;
esac
done
# Main application flow
function main {
# Show usage if no params
if [ ! "$path" ]; then
usage
fi
# Understand . path
if [ "$path" == '.' ]; then
path=$(pwd)
fi
# Check if path exists
if [ ! -e "$path" ]; then
echo "Path $path does not exist"
exit 1
fi
# Must type y or n THEN press enter (which I like better)
echo "NOTICE: ${size^^} free space on '$path' is required to perform the benchmark."
echo -n "Are you ready to start a storage benchmark against '$path' ? "; read answer
if [ "$answer" != "${answer#[Yy]}" ]; then
echo "Great! Starting benchmark now!"
else
echo "Ok, cancelled!"
exit 0
fi
# Use dd or fio based on param or defaults
if [ "$option" == "--dd" ]; then
dd_speedtest
elif [ "$option" == "--fio" ]; then
cdm_speedtest
elif [ "$option" == "--slog" ]; then
slog_speedtest
elif [ "$option" == "" ]; then
# If fio is installed, use it, else use dd
if ! command -v fio &> /dev/null; then
echo ""
echo "fio is not installed -- falling back to basic dd test."
echo "Install fio for the full CrystalDiskMark-style benchmark."
dd_speedtest
else
cdm_speedtest
fi
fi
}
function fio_probe {
# Run a tiny throwaway fio job to see if a given (engine, direct) combo
# actually works on this path/filesystem. Returns 0 on success. This is how
# we stay accurate AND portable across ext4, xfs, btrfs, ZFS, NFS, etc.
# without the caller needing to know what each filesystem supports.
local eng="$1" dir="$2"
sudo fio --name=probe \
--directory="$path" \
--ioengine="$eng" \
--rw=write --bs=4k --size=1m \
--direct="$dir" \
--time_based --runtime=1 \
> /dev/null 2>&1
local rc=$?
rm -rf "$path"/probe* 2>/dev/null
return $rc
}
function detect_io_settings {
# Pick the fastest available IO engine unless the user forced one with
# --engine=. io_uring is the modern, lowest-overhead Linux engine; libaio
# is older (and only truly async with O_DIRECT); posixaio/sync are the
# portable fallbacks (eg some NFS mounts).
if [ -z "$engine" ]; then
for eng in io_uring libaio posixaio sync; do
if fio_probe "$eng" 0; then engine="$eng"; break; fi
done
[ -z "$engine" ] && engine="sync"
fi
# Decide O_DIRECT unless forced with --direct/--buffered. O_DIRECT bypasses
# the OS page cache so we measure the device instead of RAM. Not every
# filesystem supports it (eg older OpenZFS < 2.3, some NFS mounts), so we
# probe and fall back to buffered rather than erroring out.
if [ -z "$direct" ]; then
if fio_probe "$engine" 1; then direct=1; else direct=0; fi
fi
}
function fio_bw_mbps {
# Extract bandwidth in decimal MB/s (matching CrystalDiskMark) from a fio
# run. fio prints eg " READ: bw=3357MiB/s (3521MB/s), ..." -- we take the
# parenthetical SI figure and normalise kB/MB/GB into MB/s.
local out="$1"
local raw num unit
raw=$(echo "$out" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | head -n1)
[ -z "$raw" ] && { echo "0.00"; return; }
num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1)
unit=$(echo "$raw" | /usr/bin/grep -oP '[A-Za-z]+/s' | head -n1)
case "$unit" in
B/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000000}' ;;
kB/s|KB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000}' ;;
MB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;;
GB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n*1000}' ;;
*) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;;
esac
}
function fio_iops {
# Extract IOPS from a fio run. fio prints eg " read: IOPS=12.3k, BW=..." --
# the value may carry a k/M suffix, which we expand to a whole number.
local out="$1"
local raw num unit
raw=$(echo "$out" | /usr/bin/grep -oP 'IOPS=\K[0-9.]+[kKmM]?' | head -n1)
[ -z "$raw" ] && { echo "0"; return; }
num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1)
unit=$(echo "$raw" | /usr/bin/grep -oP '[kKmM]' | head -n1)
case "$unit" in
k|K) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000}' ;;
m|M) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000000}' ;;
*) awk -v n="$num" 'BEGIN{printf "%.0f", n}' ;;
esac
}
function run_fio {
# run_fio <rw> <bs> <iodepth> <numjobs> -> echoes "MB/s IOPS" (space-sep)
# A single shared 1GB test file is reused across every run (laid out once),
# which keeps the footprint at one file and avoids re-creating it each time.
local rw="$1" bs="$2" qd="$3" jobs="$4"
local args=(
--name=speedtest
--filename="$benchfile"
--ioengine="$engine"
--direct="$direct"
--rw="$rw"
--bs="$bs"
--size="$size"
--numjobs="$jobs"
--iodepth="$qd"
--time_based --runtime="$runtime"
--group_reporting=1
)
# Flush device cache at the end of write tests so cached writes can't lie.
case "$rw" in
*write) args+=(--end_fsync=1) ;;
esac
# Capture the full fio output. In verbose mode we also keep fio's stderr
# (2>&1); otherwise we discard it. Either way only the parsed bandwidth is
# echoed to stdout, so the captured value stays clean.
local out
if [ "$verbose" = 1 ]; then
out=$(sudo fio "${args[@]}" 2>&1)
else
out=$(sudo fio "${args[@]}" 2>/dev/null)
fi
# Dump the raw fio output (to stderr, so it never pollutes the captured
# values on stdout) when running verbose.
if [ "$verbose" = 1 ]; then
echo " --- fio: $rw bs=$bs iodepth=$qd numjobs=$jobs ---" >&2
echo "$out" >&2
echo "" >&2
fi
echo "$(fio_bw_mbps "$out") $(fio_iops "$out")"
}
# Table drawing -------------------------------------------------------------
# Label cell is "| %-16s |" (18 dashes); the four numeric cells are "| %14s |"
# (16 dashes each).
TBL_SEG_LBL="------------------" # 18 dashes -> matches "| %-16s |"
TBL_SEG_NUM="----------------" # 16 dashes -> matches "| %14s |"
function tbl_rule {
printf "+%s+%s+%s+%s+%s+\n" "$TBL_SEG_LBL" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM"
}
function tbl_row {
# tbl_row <test> <read-mb/s> <write-mb/s> <read-iops> <write-iops>
# (col1 left-aligned, the four numeric cols right-aligned)
printf "| %-16s | %14s | %14s | %14s | %14s |\n" "$1" "$2" "$3" "$4" "$5"
}
# Five-column table for the --slog latency profile.
# "| %-16s |" -> 18 dashes ; "| %12s |" -> 14 dashes
SLOG_SEG16="------------------" # 18 dashes
SLOG_SEG12="--------------" # 14 dashes
function slog_rule {
printf "+%s+%s+%s+%s+%s+\n" "$SLOG_SEG16" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12"
}
function slog_row {
# slog_row <test> <iops> <mb/s> <p50> <p99> (col1 left, rest right-aligned)
printf "| %-16s | %12s | %12s | %12s | %12s |\n" "$1" "$2" "$3" "$4" "$5"
}
function run_fio_sync {
# run_fio_sync <bs> <numjobs>
# -> echoes "IOPS MB/s p50_us p99_us mean_us" (one space-separated line)
# Forces synchronous IO (--sync=1 => O_SYNC) with the portable psync engine so
# every write is a ZIL commit -- i.e. it exercises the SLOG exactly the way a
# sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own
# sync property. We deliberately DON'T use --group_reporting here: each job is
# reported independently in the JSON and we aggregate ourselves (sum IOPS/bw,
# average p50, take the worst p99), which sidesteps fio's group-merge quirks.
local bs="$1" jobs="$2"
local args=(
--name=slog
--filename="$benchfile"
--ioengine=psync
--rw=randwrite
--bs="$bs"
--size="$size"
--numjobs="$jobs"
--sync=1
--time_based --runtime="$runtime"
--output-format=json
)
local out
out=$(sudo fio "${args[@]}" 2>/dev/null)
if [ "$verbose" = 1 ]; then
echo " --- fio json: randwrite bs=$bs numjobs=$jobs sync=1 ---" >&2
echo "$out" >&2
echo "" >&2
fi
# Parse the JSON: sum IOPS and bandwidth across jobs, aggregate completion
# latency percentiles (clat is in ns -> convert to us). bw_bytes is bytes/s.
echo "$out" | python3 -c '
import json, sys
try:
d = json.load(sys.stdin)
jobs = d["jobs"]
except Exception:
print("0 0.00 0.0 0.0 0.0"); sys.exit(0)
iops = sum(j["write"]["iops"] for j in jobs)
mbps = sum(j["write"]["bw_bytes"] for j in jobs) / 1e6
def pctl(j, p):
return j["write"]["clat_ns"]["percentile"].get(p)
p50 = [pctl(j, "50.000000") for j in jobs if pctl(j, "50.000000") is not None]
p99 = [pctl(j, "99.000000") for j in jobs if pctl(j, "99.000000") is not None]
mean = [j["write"]["clat_ns"]["mean"] for j in jobs]
p50_us = (sum(p50)/len(p50))/1000.0 if p50 else 0.0 # average of per-job medians
p99_us = (max(p99))/1000.0 if p99 else 0.0 # worst-case tail across jobs
mean_us = (sum(mean)/len(mean))/1000.0 if mean else 0.0
print("%.0f %.2f %.1f %.1f %.1f" % (iops, mbps, p50_us, p99_us, mean_us))
'
}
function slog_speedtest {
# SLOG / sync-write latency profile. Detect engine/direct only for the banner;
# the actual runs always force psync + O_SYNC (see run_fio_sync).
detect_io_settings
benchfile="$path/speedtest-hd.bench"
if ! command -v fio &> /dev/null; then
echo "ERROR: --slog requires fio (apt install fio / pacman -S fio)." >&2
exit 1
fi
if ! command -v python3 &> /dev/null; then
echo "ERROR: --slog requires python3 (used to parse fio JSON latency percentiles)." >&2
exit 1
fi
echo ""
echo " speedtest-hd : SLOG / sync-write latency profile"
echo " Target : $path"
echo " Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine"
echo " Profile : runtime=${runtime}s/run size=${size^^}"
echo " Note : every write is a synchronous ZIL commit -- this is the load"
echo " your SLOG actually sees. Watch it live in another shell with:"
echo " zpool iostat -vl <pool> 1"
echo ""
# T1 is the headline single-stream latency; the sweep shows how the SLOG
# scales as concurrent sync writers (NFS/iSCSI/VM threads) pile on. Run every
# measurement FIRST (progress goes to stderr), collect the results, then draw
# the whole table in one block so the "measuring..." lines can't interleave
# into the middle of the table.
local jobs_list="1 4 8 16"
local -a rows
local j res iops bw p50 p99 mean
for j in $jobs_list; do
echo " measuring 4K sync randwrite T$j ..." >&2
res=$(run_fio_sync 4k "$j")
read -r iops bw p50 p99 mean <<< "$res"
rows+=("$(slog_row "4K sync T$j" "$iops" "$bw" "$p50" "$p99")")
done
rm -f "$benchfile"
# Render the assembled table
echo ""
slog_rule
slog_row "Test" "IOPS" "MB/s" "p50 lat(us)" "p99 lat(us)"
slog_rule
printf '%s\n' "${rows[@]}"
slog_rule
echo ""
echo " Healthy Optane SLOG (eg P1600X) single-stream (T1) target:"
echo " ~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means"
echo " CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling."
echo ""
}
function cdm_speedtest {
# Auto-detect the best engine and whether O_DIRECT works on this path.
detect_io_settings
benchfile="$path/speedtest-hd.bench"
local direct_label="enabled (device)"
[ "$direct" != 1 ] && direct_label="DISABLED (buffered -- may reflect RAM cache!)"
# Banner
echo ""
echo " speedtest-hd : CrystalDiskMark-style storage benchmark"
echo " Target : $path"
echo " Engine : $engine O_DIRECT: $direct_label"
echo " Profile : size=${size^^} runtime=${runtime}s/run (8 runs)"
echo ""
# The four CrystalDiskMark default tests. Q = queue depth (--iodepth),
# T = threads (--numjobs). The first three are T1 (single thread); the last
# is T16. A single shared test file is reused -- with numjobs>1 every job
# issues IO against the same file, which is fine.
# Each run now returns both MB/s and IOPS ("bw iops"), so we capture the pair.
# label read-rw write-rw bs Q T
echo " measuring SEQ1M Q8T1 ..." >&2
local s8r_bw s8r_io; read -r s8r_bw s8r_io <<< "$(run_fio read 1m 8 1)"
local s8w_bw s8w_io; read -r s8w_bw s8w_io <<< "$(run_fio write 1m 8 1)"
echo " measuring SEQ1M Q1T1 ..." >&2
local s1r_bw s1r_io; read -r s1r_bw s1r_io <<< "$(run_fio read 1m 1 1)"
local s1w_bw s1w_io; read -r s1w_bw s1w_io <<< "$(run_fio write 1m 1 1)"
echo " measuring RND4K Q32T1 ..." >&2
local r32r_bw r32r_io; read -r r32r_bw r32r_io <<< "$(run_fio randread 4k 32 1)"
local r32w_bw r32w_io; read -r r32w_bw r32w_io <<< "$(run_fio randwrite 4k 32 1)"
echo " measuring RND4K Q32T16 ..." >&2
local r1r_bw r1r_io; read -r r1r_bw r1r_io <<< "$(run_fio randread 4k 32 16)"
local r1w_bw r1w_io; read -r r1w_bw r1w_io <<< "$(run_fio randwrite 4k 32 16)"
# Cleanup the shared test file
rm -f "$benchfile"
# Render the CrystalDiskMark-style table
echo ""
tbl_rule
tbl_row "Test" "Read (MB/s)" "Write (MB/s)" "Read (IOPS)" "Write (IOPS)"
tbl_rule
tbl_row "SEQ1M Q8T1" "$s8r_bw" "$s8w_bw" "$s8r_io" "$s8w_io"
tbl_row "SEQ1M Q1T1" "$s1r_bw" "$s1w_bw" "$s1r_io" "$s1w_io"
tbl_row "RND4K Q32T1" "$r32r_bw" "$r32w_bw" "$r32r_io" "$r32w_io"
tbl_row "RND4K Q32T16" "$r1r_bw" "$r1w_bw" "$r1r_io" "$r1w_io"
tbl_rule
echo ""
}
function dd_speedtest {
# Basic HD speed test using dd (fallback when fio is not installed)
# mReschke 2017-07-11
local file ddsize
file=$path/bigfile
ddsize=1024
echo "Running dd based HD/SSD/NVMe Benchmarks"
echo "---------------------------------------"
printf "Cached write speed...\n"
dd if=/dev/zero of=$file bs=1M count=$ddsize
printf "\nUncached write speed...\n"
dd if=/dev/zero of=$file bs=1M count=$ddsize conv=fdatasync,notrunc
printf "\nUncached read speed...\n"
echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
dd if=$file of=/dev/null bs=1M count=$ddsize
printf "\nCached read speed...\n"
dd if=$file of=/dev/null bs=1M count=$ddsize
rm $file
printf "\nDone\n"
}
# Show help and usage information
function usage {
echo "speedtest-hd : CrystalDiskMark-style storage benchmark"
echo " Runs the same four tests as CrystalDiskMark (each Read + Write):"
echo " SEQ1M Q8T1 Sequential 1MiB, queue depth 8, 1 thread"
echo " SEQ1M Q1T1 Sequential 1MiB, queue depth 1, 1 thread"
echo " RND4K Q32T1 Random 4KiB, queue depth 32, 1 thread"
echo " RND4K Q32T16 Random 4KiB, queue depth 32, 16 threads"
echo " Uses fio if installed (recommended), else basic dd."
echo "mReschke 2024-01-18"
echo ""
echo "NOTICE: creates one test file (default 1GB) on the destination disk."
echo "Please ensure you have write access with enough free space."
echo ""
echo "Usage:"
echo " Auto (fio if installed, else dd):"
echo " ./speedtest-hd /mnt/somedisk"
echo " ./speedtest-hd ."
echo ""
echo " Force fio / force dd:"
echo " ./speedtest-hd /mnt/somedisk --fio"
echo " ./speedtest-hd /mnt/somedisk --dd"
echo ""
echo " SLOG / sync-write latency profile (ZFS ZIL, NFS/iSCSI/VM sync IO):"
echo " ./speedtest-hd /mnt/zpool/dataset --slog"
echo " 4K synchronous writes at T1/T4/T8/T16; reports IOPS, MB/s, p50/p99"
echo " commit latency. Requires fio + python3. (Tip: watch the SLOG with"
echo " 'zpool iostat -vl <pool> 1' in another shell.)"
echo ""
echo " Tuning flags (auto-detected by default, override as needed):"
echo " --engine=io_uring|libaio|posixaio|sync IO engine (default: auto)"
echo " --direct Force O_DIRECT (bypass page cache)"
echo " --buffered Force buffered IO (eg if O_DIRECT is unsupported)"
echo " --runtime=SEC Seconds per run (default: 5, like CrystalDiskMark)"
echo " --size=SIZE Test file size (default: 1g)"
echo " --verbose Print full fio output for every run (summary table unchanged)"
echo " Examples:"
echo " ./speedtest-hd /mnt/nvmepool --runtime=10 --size=4g"
echo " ./speedtest-hd /mnt/nfsshare --buffered"
echo " ./speedtest-hd /mnt/nvmepool --verbose"
echo " ./speedtest-hd /mnt/nvme-ultra-r10/vm-root --slog --runtime=30"
exit 0
}
# Go
main