New python speedtest to replace bash. New CLAUDE.md

CLAUDE.md

@@ -0,0 +1,65 @@
+# speedtest-hd
+
+Single-file Bash CLI (`speedtest-hd.sh`) that benchmarks any mounted path (HDD/SSD/NVMe,
+local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII table. Falls back
+to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount" disk tester — not
+a ZFS/NVMe-specific tool.
+
+## Usage
+```
+./speedtest-hd.sh <path> [--fio|--dd|--slog] [flags]
+```
+- Default: fio if installed, else dd. `--fio` forces CDM mode, `--slog` forces sync-latency mode.
+- Flags: `--engine=io_uring|libaio|posixaio|sync` (default auto), `--direct`/`--buffered`
+  (default auto-probe), `--runtime=SEC` (default 5), `--size=SIZE` (default 1g), `--verbose`
+  (also dumps raw fio output to stderr).
+
+## Modes & layout
+- **CDM mode** (`cdm_speedtest`): runs CrystalDiskMark's default tests, each read+write,
+  prints a 5-col table (Read/Write MB/s, Read/Write IOPS). Tests: `SEQ1M Q8T1`, `SEQ1M Q1T1`,
+  `RND4K Q32T1`, `RND4K Q32T16`. Q = `--iodepth`, T = `--numjobs`.
+- **SLOG mode** (`slog_speedtest`): 4K synchronous randwrite sweep at T1/T4/T8/T16 to profile
+  ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 commit
+  latency. Requires `fio` + `python3` (JSON percentile parsing).
+- **dd mode** (`dd_speedtest`): legacy cached/uncached read/write fallback.
+
+## Key implementation notes
+- `detect_io_settings` auto-picks the engine (io_uring → libaio → posixaio → sync) and probes
+  whether O_DIRECT works, via tiny throwaway `fio_probe` jobs. Falls back to buffered with a
+  warning if O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly async
+  with `--direct=1`** — that's why io_uring is preferred. O_DIRECT bypasses the page cache so we
+  measure the device, not RAM (buffered results, esp. reads, can reflect ARC/page cache).
+- One shared test file `<path>/speedtest-hd.bench` is reused across all runs (laid out once),
+  removed at the end. Footprint = `--size` (default 1G), matching the startup notice.
+- `run_fio` (CDM) returns `"MB/s IOPS"`; `run_fio_sync` (SLOG) forces `--ioengine=psync --sync=1`
+  (O_SYNC) so every write is a ZIL commit regardless of dataset sync property, and parses fio
+  **JSON** to aggregate across jobs (sum IOPS/bw, avg p50, worst-case p99). CDM mode parses
+  fio's text output: `fio_bw_mbps` takes the parenthetical **SI MB/s** (matches CDM's decimal
+  MB/s, normalizes kB/MB/GB); `fio_iops` expands k/M suffixes.
+- Write tests append `--end_fsync=1` so cached writes can't inflate numbers.
+- ASCII tables: cell formats and dash-segment widths must stay in sync (`tbl_*` = 18/16 dashes,
+  `slog_*` = 18/14). Verify alignment after editing.
+
+## History / decisions
+- Originated from the Ars Technica fio guide. Original 4K test used `numjobs=1 iodepth=1`, which
+  measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1, not
+  a bug. Refactored toward parallel/deep-queue tests to show real device capability, then fully
+  reshaped into the CrystalDiskMark profile above. `--simple` flag was removed; the table is now
+  the only fio output.
+- Note: the current `RND4K` rows are `Q32T1` + `Q32T16`. CrystalDiskMark's actual latest default
+  profile is `Q32T16` + `Q1T1` (single-queue random is the meaningful low-end number). If aligning
+  strictly to CDM, the `Q32T1` row should become `Q1T1` (`iodepth=1 numjobs=1`).
+
+## SLOG performance context (why `--slog` exists)
+Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10`
+= 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). Poor sync
+writes were **CPU power management**, not the SLOG:
+- Fix (biggest last): Dell BIOS profile DAPC → **Performance** (~2×); cstate kernel args; and the
+  big one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, which parked cores at
+  1.2 GHz because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post
+  Init: `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`.
+- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor).
+- Diagnostic: `zpool iostat -vl <pool> 1` during fio showed the Optane `logs` vdev at ~90µs
+  disk_wait — proving the SLOG was fine and latency was upstream (CPU).
+- Healthy Optane SLOG single-stream (T1) target: ~15–25k IOPS, p50 ~40–65µs. Much higher usually
+  = C-states / PCIe ASPM / BIOS power profile throttling.

README.md

@@ -0,0 +1,291 @@
+# speedtest-hd
+
+A robust, CrystalDiskMark‑style storage benchmark for Linux, built on [`fio`](https://fio.readthedocs.io/) by mReschke and my buddy Claude Opus.
+
+It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync‑write latency profile** for diagnosing ZFS ZIL performance (NFS / iSCSI / VM sync workloads). It auto‑detects the best IO engine and whether `O_DIRECT` works on the target, and falls back to a basic `dd` test when `fio` isn't installed.
+
+---
+
+## Table of contents
+
+- [Features](#features)
+- [Requirements](#requirements)
+- [Usage](#usage)
+- [Output](#output)
+- [Understanding the tests](#understanding-the-tests)
+- [Case study: diagnosing a "slow" Optane SLOG on TrueNAS](#case-study-diagnosing-a-slow-optane-slog-on-truenas)
+- [Notes & caveats](#notes--caveats)
+
+---
+
+## Features
+
+- **CrystalDiskMark‑style profile** — `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T1`, `RND4K Q32T16`, each measured for Read **and** Write, reported in both **MB/s** and **IOPS**.
+- **SLOG / sync‑write latency profile** (`--slog`) — synchronous 4K writes at T1/T4/T8/T16 reporting **IOPS, MB/s, and p50/p99 commit latency**. This is the load a ZFS SLOG actually sees.
+- **Auto‑detection** — picks the fastest available IO engine (`io_uring` → `libaio` → `posixaio` → `sync`) and probes whether `O_DIRECT` works on the filesystem, falling back to buffered IO when it doesn't (e.g. older OpenZFS, some NFS mounts).
+- **`dd` fallback** — if `fio` isn't present, runs a basic write/read test so you still get a number.
+- **Verbose mode** — `--verbose` dumps the full raw `fio` output for every run while keeping the summary table intact.
+
+---
+
+## Requirements
+
+- `bash`
+- [`fio`](https://fio.readthedocs.io/) — recommended (`apt install fio` / `pacman -S fio`). Without it, the tool falls back to `dd`.
+- `python3` — **only** required for `--slog` (used to parse `fio`'s JSON output for latency percentiles). The normal profile needs only `fio` + `grep`/`awk`.
+- `sudo` — `fio` is invoked via `sudo` so it can use `O_DIRECT` and flush device caches.
+
+---
+
+## Usage
+
+```bash
+./speedtest-hd.sh <path> [options]
+```
+
+`<path>` is the directory (or mount) to benchmark. Use `.` for the current directory. The tool creates a single test file (default 1 GiB) on the target and removes it afterward, so ensure enough free space.
+
+### Modes
+
+| Invocation | What it does |
+|---|---|
+| `./speedtest-hd.sh /mnt/disk` | Auto: uses `fio` if installed, else `dd` |
+| `./speedtest-hd.sh /mnt/disk --fio` | Force the `fio` CrystalDiskMark‑style profile |
+| `./speedtest-hd.sh /mnt/disk --dd` | Force the basic `dd` test |
+| `./speedtest-hd.sh /mnt/disk --slog` | SLOG / sync‑write latency profile |
+
+### Tuning flags
+
+| Flag | Effect |
+|---|---|
+| `--engine=io_uring\|libaio\|posixaio\|sync` | Force a specific IO engine (default: auto) |
+| `--direct` | Force `O_DIRECT` (bypass page cache) |
+| `--buffered` | Force buffered IO (e.g. when `O_DIRECT` is unsupported) |
+| `--runtime=SEC` | Seconds per run (default: 5, like CrystalDiskMark) |
+| `--size=SIZE` | Test file size (default: `1g`) |
+| `--verbose` | Also print the full `fio` output for every run (summary table unchanged) |
+
+### Examples
+
+```bash
+# CrystalDiskMark-style test of the current directory
+./speedtest-hd.sh .
+
+# Larger file, longer runs, on an NVMe pool
+./speedtest-hd.sh /mnt/nvmepool --runtime=10 --size=4g
+
+# Buffered (e.g. an NFS share that doesn't support O_DIRECT)
+./speedtest-hd.sh /mnt/nfsshare --buffered
+
+# SLOG / sync latency profile, 30s per run
+./speedtest-hd.sh /mnt/nvme-ultra-r10/vm-root --slog --runtime=30
+```
+
+> **Tip:** when running `--slog` against a ZFS dataset, watch the SLOG live in another shell:
+> ```bash
+> zpool iostat -vl <pool> 1
+> ```
+
+---
+
+## Output
+
+### CrystalDiskMark‑style profile
+
+```
++------------------+----------------+----------------+----------------+----------------+
+| Test             |    Read (MB/s) |   Write (MB/s) |    Read (IOPS) |   Write (IOPS) |
++------------------+----------------+----------------+----------------+----------------+
+| SEQ1M  Q8T1      |        6873.00 |           9.30 |           6873 |              9 |
+| SEQ1M  Q1T1      |        1608.00 |          20.00 |           1608 |             20 |
+| RND4K  Q32T1     |         538.00 |          10.80 |         137728 |           2764 |
+| RND4K  Q32T16    |         689.00 |         261.00 |         176384 |          66816 |
++------------------+----------------+----------------+----------------+----------------+
+```
+
+### SLOG / sync‑write latency profile (`--slog`)
+
+```
++------------------+--------------+--------------+--------------+--------------+
+| Test             |         IOPS |         MB/s |  p50 lat(us) |  p99 lat(us) |
++------------------+--------------+--------------+--------------+--------------+
+| 4K sync T1       |        10687 |        43.77 |         85.5 |        185.3 |
+| 4K sync T4       |        29873 |       122.36 |        117.8 |        317.4 |
+| 4K sync T8       |        52612 |       215.50 |        136.2 |        391.2 |
+| 4K sync T16      |        77939 |       319.24 |        180.0 |        505.9 |
++------------------+--------------+--------------+--------------+--------------+
+```
+
+---
+
+## Understanding the tests
+
+### The CrystalDiskMark profile
+
+| Test | Pattern | Queue depth | Threads |
+|---|---|---|---|
+| `SEQ1M Q8T1` | Sequential 1 MiB | 8 | 1 |
+| `SEQ1M Q1T1` | Sequential 1 MiB | 1 | 1 |
+| `RND4K Q32T1` | Random 4 KiB | 32 | 1 |
+| `RND4K Q32T16` | Random 4 KiB | 32 | 16 |
+
+`Q` = queue depth (`--iodepth`), `T` = threads (`--numjobs`). Note that `--iodepth` only produces real queue depth when the IO is truly asynchronous (async engine **and** `O_DIRECT`). On filesystems where that isn't available, queue depth effectively collapses toward 1 and concurrency comes only from threads (`T`).
+
+### The SLOG profile
+
+`--slog` forces **synchronous** 4K random writes (`--sync=1` → `O_SYNC`) via the portable `psync` engine. Every write becomes a durable commit, so it traverses the ZFS ZIL / SLOG commit path exactly the way a `sync=always` dataset (NFS, iSCSI, VM storage) does — regardless of the dataset's own `sync` property. It sweeps thread counts (T1 → T4 → T8 → T16):
+
+- **T1** is the headline single‑stream latency (e.g. one database committing in a tight loop).
+- **The sweep** shows how the SLOG scales as concurrent sync writers pile on (multiple VMs / NFS clients) — usually the more important number for a virtualization host.
+
+A healthy Optane SLOG (e.g. P1600X) single‑stream target is roughly **15–25k IOPS, p50 ~40–65 µs**. Much higher latency usually points at **CPU C‑states / PCIe ASPM / a BIOS power profile** throttling the host — see the case study below.
+
+---
+
+## Case study: diagnosing a "slow" Optane SLOG on TrueNAS
+
+A real investigation that this tool's `--slog` mode was built to support. **Spoiler: the Optane SSD was healthy the entire time. The bottleneck was CPU power management.**
+
+### The setup
+
+| Component | Detail |
+|---|---|
+| Server | Dell PowerEdge R630 |
+| CPU | Intel Xeon E5‑2680 v3 (Haswell‑EP, 12C/24T, 2.5 GHz base, 3.3 GHz turbo) |
+| OS | TrueNAS SCALE 25.10 (OpenZFS 2.3) |
+| Pool | `nvme-ultra-r10` — 6× 4 TB KingSpec XG7000 NVMe in RAID10 (3 mirror vdevs) |
+| SLOG | Intel Optane **P1600X** |
+| Dataset | `/mnt/nvme-ultra-r10/vm-root`, `sync=always` |
+
+### The symptom
+
+The standard benchmark looked alarming — huge reads, tiny writes:
+
+```
++------------------+------------------+------------------+
+| Test             |      Read (MB/s) |     Write (MB/s) |
++------------------+------------------+------------------+
+| SEQ1M  Q8T1      |          6873.00 |             9.30 |
+| SEQ1M  Q1T1      |          1608.00 |            20.00 |
+| RND4K  Q32T1     |           538.00 |            10.80 |
+| RND4K  Q32T16    |           689.00 |           261.00 |
++------------------+------------------+------------------+
+```
+
+9.3 MB/s sequential write on an Optane‑backed NVMe pool looks broken.
+
+### Investigation
+
+**1. The reads are RAM, not disk.** With a 1 GiB test file on ZFS, reads come straight from ARC (RAM cache). The huge read/write asymmetry is the tell — ignore the read column for judging the disks.
+
+**2. `sync=always` makes writes latency‑bound.** Every write must be durably committed to the ZIL before it's acknowledged, so throughput ≈ (block size) ÷ (per‑commit latency). Anything running at low effective concurrency looks slow regardless of raw device speed.
+
+**3. The `Q8`/`Q32` labels were misleading.** On this ZFS setup `--iodepth` didn't produce real queue depth, so most rows effectively ran at QD1. Proof: `RND4K Q32T1` (10.8 MB/s) vs `RND4K Q32T16` (261 MB/s) — the 24× jump came entirely from threads (`numjobs=16`), not queue depth.
+
+**4. Large sequential sync writes bypass the SLOG.** With ZFS's default `logbias=latency`, writes larger than `zfs_immediate_write_sz` (32 KB) use an *indirect* ZIL record — the data goes straight to the main pool and only a pointer hits the Optane. So the `SEQ1M` write test was measuring the **consumer KingSpec pool's** forced‑sync performance, not the SLOG. Only small (4K) sync writes exercise the Optane.
+
+**5. Confirm with `zpool iostat -vl <pool> 1` during a 4K sync test.** This was decisive:
+
+- The `logs` (Optane) vdev took **all** the sync writes (~2.3–2.6k ops, ~18–20 MB/s); the data vdevs were idle between txg flushes. → **SLOG configured correctly, `logbias` fine, not bypassed.**
+- But the Optane's own `disk_wait` was **~90 µs** (a P1600X should be ~10–15 µs), and the fio‑level commit latency was **~328 µs** — meaning ~238 µs was being spent *above* the device, in the host/ZFS/CPU path.
+
+That "faster when busy, slow when idle" device latency plus huge host overhead is the classic signature of **power‑saving idle states** on a latency‑bound, QD1 workload.
+
+**6. Find the throttle.** Checking the CPU revealed the cores pinned at **1.2 GHz** — the E5‑2680 v3's *minimum* P‑state — on a chip rated for 2.5–3.3 GHz:
+
+```
+$ grep MHz /proc/cpuinfo | sort -u
+cpu MHz : 1200.069
+...
+
+$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
+intel_cpufreq                # = intel_pstate in passive mode
+$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
+schedutil                    # picks frequency from CPU utilization
+```
+
+The vicious loop: a QD1 sync workload spends each commit **blocked** waiting on the SLOG → the `schedutil` governor sees near‑zero utilization → parks the cores at 1.2 GHz → the ZFS commit code path runs ~2–3× slower → latency climbs.
+
+### Root cause
+
+**CPU power management, in two layers — not the SSD, pool, or PCIe link:**
+
+1. **BIOS System Profile = "Performance Per Watt Optimized (DAPC)"** — Dell Active Power Controller manages C‑states/P‑states in firmware and largely ignores the OS, keeping cores idling deep and clocked low.
+2. **OS `schedutil` governor** (TrueNAS SCALE default) — pinned cores at the 1.2 GHz floor for this bursty, IO‑blocked workload.
+
+### The fixes
+
+Applied in order, biggest impact last:
+
+**1. BIOS System Profile → Performance** (disables C‑states/C1E, raises P‑states):
+```
+# In BIOS (F2): System BIOS → System Profile Settings → System Profile → Performance
+# Or via iDRAC:
+racadm set BIOS.SysProfileSettings.SysProfile PerfOptimized
+racadm jobqueue create BIOS.Setup.1-1
+# reboot to apply
+```
+
+**2. Kernel parameters** (target PCIe/NVMe link power saving + residual C‑states). On TrueNAS: *System → Advanced Settings → Kernel Arguments*, then reboot:
+```
+intel_idle.max_cstate=1 processor.max_cstate=1 pcie_aspm=off nvme_core.default_ps_max_latency_us=0
+```
+
+**3. CPU governor → `performance`** *(the single biggest win)*:
+```bash
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+```
+Make it persistent on TrueNAS — *System → Advanced Settings → Init/Shutdown Scripts*, add a **Post Init** *Command*:
+```
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+```
+> ⚠️ Without the Post Init script the governor reverts to `schedutil` on every reboot, silently dropping you back to slow numbers.
+
+### Results — before & after
+
+**4K synchronous random writes** (`--slog`), per stage of the fix:
+
+| Stage | T1 IOPS | T1 p50 | T1 MB/s | T16 IOPS | T16 MB/s | T16 p99 |
+|---|---|---|---|---|---|---|
+| **DAPC** (start) | ~3,050 | ~328 µs | ~12.5 | — | 38 *(regressing)* | thrashing |
+| BIOS → Performance | 5,849 | 160.8 µs | 23.96 | 46,009 | 188.45 | 1057 µs |
+| + kernel parameters | 6,217 | 150.5 µs | 25.46 | 47,166 | 193.19 | 684 µs |
+| **+ `performance` governor** | **10,687** | **85.5 µs** | **43.77** | **77,939** | **319.24** | **506 µs** |
+
+**Full final result:**
+
+```
++------------------+--------------+--------------+--------------+--------------+
+| Test             |         IOPS |         MB/s |  p50 lat(us) |  p99 lat(us) |
++------------------+--------------+--------------+--------------+--------------+
+| 4K sync T1       |        10687 |        43.77 |         85.5 |        185.3 |
+| 4K sync T4       |        29873 |       122.36 |        117.8 |        317.4 |
+| 4K sync T8       |        52612 |       215.50 |        136.2 |        391.2 |
+| 4K sync T16      |        77939 |       319.24 |        180.0 |        505.9 |
++------------------+--------------+--------------+--------------+--------------+
+```
+
+**Net improvement:** ~**3.5× IOPS**, ~**3.8× lower latency** at T1, and ~**8.4× aggregate throughput** at T16 — with the scaling regression eliminated entirely.
+
+### Lessons learned
+
+- **An Optane SLOG showing high latency is usually a host‑side power‑management problem, not the device.** Confirm where the time goes before blaming hardware.
+- **`zpool iostat -vl <pool> 1` is the key diagnostic** — it shows whether the `logs` vdev is actually taking the writes and splits device latency (`disk_wait`) from host/ZFS overhead (`total_wait`).
+- **Latency‑bound QD1 sync workloads are the worst case for power saving.** The CPU looks idle (blocked on IO), so governors and firmware clock it down — which directly inflates the latency you're trying to measure.
+- **On TrueNAS SCALE, the default `schedutil` governor cripples sync‑write latency.** Set `performance` (and persist it).
+- **Reads from a small test file measure ARC (RAM), not the disk.** Watch the read/write asymmetry.
+- **Large sync writes bypass the SLOG** (indirect ZIL) — to actually test a SLOG, use small (4K) sync writes, which is exactly what `--slog` does.
+
+### `~85 µs` is roughly the floor here
+
+The residual gap from raw Optane (~15 µs) is ZFS ZIL‑commit overhead plus the per‑op cost of a 2014‑era Haswell core. Closing it further would need a newer/faster CPU for sharply diminishing returns. For a virtualization host the aggregate (78k IOPS / 319 MB/s) is what the workload feels, and it's healthy.
+
+---
+
+## Notes & caveats
+
+- **`sudo`** is used for `fio` so it can apply `O_DIRECT` and flush device write caches at the end of write runs (`--end_fsync=1`), so cached writes can't inflate results.
+- **`O_DIRECT` is auto‑detected.** If the banner shows `O_DIRECT: DISABLED (buffered ...)`, results may reflect the page cache (RAM) rather than the device.
+- **A single shared test file** is reused across runs to keep the footprint to one file.
+- **`--slog` requires `python3`** (for JSON latency‑percentile parsing); the standard profile does not.
+- The `--slog` profile forces synchronous IO and is intended for ZFS ZIL / SLOG and other sync‑write (NFS/iSCSI/VM) investigations.

speedtest-hd.py

@@ -0,0 +1,727 @@
+#!/usr/bin/env python3
+"""speedtest-hd — a CrystalDiskMark-style storage benchmark for Linux, on fio.
+
+This is the Python successor to ``speedtest-hd.sh``. It measures storage the way
+CrystalDiskMark does, and adds a dedicated **SLOG / sync-write latency profile**
+for diagnosing ZFS ZIL performance (NFS / iSCSI / VM sync workloads).
+
+Three profiles
+--------------
+* **cdm**  — the four CrystalDiskMark default tests (``SEQ1M Q8T1``, ``SEQ1M
+  Q1T1``, ``RND4K Q32T1``, ``RND4K Q32T16``), each measured for read *and*
+  write, reported in MB/s and IOPS.
+* **slog** — synchronous 4K random writes swept across thread counts, reporting
+  IOPS, MB/s and p50/p99 *commit latency*. This is the load a ZFS SLOG sees.
+* **dd**   — a dependency-free fallback when ``fio`` isn't installed.
+
+Why fio JSON?
+-------------
+Every measurement asks fio for ``--output-format=json`` and parses it with the
+standard library. That's the whole reason this is Python and not bash: robust,
+unit-safe parsing of bandwidth / IOPS / latency percentiles with no fragile
+text scraping.
+
+See README.md for the full case study (diagnosing a "slow" Optane SLOG that
+turned out to be CPU power management, not the disk).
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import os
+import shutil
+import statistics
+import subprocess
+import sys
+import time
+from dataclasses import dataclass, field
+from typing import Callable, Optional, Sequence
+
+# --------------------------------------------------------------------------- #
+# Color / styling (standard library only -- raw ANSI escape codes)
+# --------------------------------------------------------------------------- #
+
+# SGR escape sequences. We stick to the basic 16-color set so output respects
+# the user's terminal theme instead of hard-coding RGB.
+RESET = "\033[0m"
+BOLD = "\033[1m"
+DIM = "\033[2m"
+ITALIC = "\033[3m"
+RED = "\033[31m"
+GREEN = "\033[32m"
+YELLOW = "\033[33m"
+BLUE = "\033[34m"
+MAGENTA = "\033[35m"
+CYAN = "\033[36m"
+WHITE = "\033[37m"
+BRIGHT_CYAN = "\033[96m"
+
+
+def supports_color(stream) -> bool:
+    """Decide per-stream whether to emit ANSI codes.
+
+    Honors the de-facto ``NO_COLOR`` / ``FORCE_COLOR`` conventions and otherwise
+    colors only when the stream is an interactive terminal.
+    """
+    if os.environ.get("NO_COLOR"):
+        return False
+    if os.environ.get("FORCE_COLOR"):
+        return True
+    if os.environ.get("TERM") == "dumb":
+        return False
+    return bool(getattr(stream, "isatty", None)) and stream.isatty()
+
+
+class Painter:
+    """Wraps text in SGR codes, or returns it untouched when color is off."""
+
+    def __init__(self, enabled: bool):
+        self.enabled = enabled
+
+    def paint(self, text: str, *codes: str) -> str:
+        if not self.enabled or not codes:
+            return text
+        return "".join(codes) + text + RESET
+
+
+# stdout carries results (banner + tables); stderr carries progress + verbose
+# fio dumps. Each gets its own enable flag so piping one but not the other works.
+OUT = Painter(supports_color(sys.stdout))
+ERR = Painter(supports_color(sys.stderr))
+
+
+# --------------------------------------------------------------------------- #
+# Constants
+# --------------------------------------------------------------------------- #
+
+# Preference order for the IO engine. io_uring is the modern, lowest-overhead
+# Linux engine; libaio is older (and only truly async with O_DIRECT); posixaio
+# and sync are the portable fallbacks (e.g. some NFS mounts).
+ENGINE_CANDIDATES = ("io_uring", "libaio", "posixaio", "sync")
+
+# Binary unit multipliers (fio treats "1g" as 1 GiB, so we match that).
+_SIZE_UNITS = {"k": 1024, "m": 1024**2, "g": 1024**3, "t": 1024**4}
+
+
+@dataclass(frozen=True)
+class CdmTest:
+    """One CrystalDiskMark test. ``seq`` picks sequential vs random patterns."""
+
+    label: str
+    bs: str
+    iodepth: int
+    numjobs: int
+    seq: bool
+
+    @property
+    def read_rw(self) -> str:
+        return "read" if self.seq else "randread"
+
+    @property
+    def write_rw(self) -> str:
+        return "write" if self.seq else "randwrite"
+
+
+# The four CrystalDiskMark default tests, as data. Q = queue depth (iodepth),
+# T = threads (numjobs).
+CDM_TESTS: tuple[CdmTest, ...] = (
+    CdmTest("SEQ1M  Q8T1",   bs="1m", iodepth=8,  numjobs=1,  seq=True),
+    CdmTest("SEQ1M  Q1T1",   bs="1m", iodepth=1,  numjobs=1,  seq=True),
+    CdmTest("RND4K  Q32T1",  bs="4k", iodepth=32, numjobs=1,  seq=False),
+    CdmTest("RND4K  Q32T16", bs="4k", iodepth=32, numjobs=16, seq=False),
+)
+
+# Thread counts for the SLOG sweep. T1 is the headline single-stream latency;
+# the higher counts show how the SLOG scales as concurrent sync writers pile on.
+SLOG_THREADS: tuple[int, ...] = (1, 4, 8, 16)
+
+
+# --------------------------------------------------------------------------- #
+# Configuration
+# --------------------------------------------------------------------------- #
+
+
+@dataclass
+class Config:
+    """Resolved run settings. ``engine`` and ``direct`` may start unset and get
+    filled in by :func:`detect_io_settings`."""
+
+    path: str
+    mode: Optional[str]            # "cdm" | "slog" | "dd" | None (auto)
+    engine: Optional[str]          # None => auto-detect
+    direct: Optional[bool]         # None => auto-detect
+    runtime: int
+    size: str
+    verbose: bool
+    assume_yes: bool
+    benchfile: str = field(default="")
+
+    @property
+    def direct_flag(self) -> int:
+        """fio's --direct takes 0/1; default to 0 until detection runs."""
+        return 1 if self.direct else 0
+
+
+# --------------------------------------------------------------------------- #
+# Small helpers
+# --------------------------------------------------------------------------- #
+
+
+def log(message: str) -> None:
+    """Progress/diagnostic line -> stderr, so stdout stays pure results."""
+    print(message, file=sys.stderr, flush=True)
+
+
+def step(message: str) -> None:
+    """A single colored progress line ('measuring ...') -> stderr."""
+    log(f"  {ERR.paint('▶', BOLD, CYAN)} {ERR.paint(message, DIM)}")
+
+
+def vsection(title: str) -> None:
+    """A clear, ruled header for one --verbose fio section -> stderr."""
+    width = 78
+    head = f"─── {title} "
+    head += "─" * max(3, width - len(head))
+    log("")
+    log(ERR.paint(head, BOLD, CYAN))
+
+
+def parse_size_bytes(size: str) -> int:
+    """Turn an fio-style size ("4k", "1g", "512m", "2048") into bytes."""
+    s = size.strip().lower()
+    if s and s[-1] in _SIZE_UNITS:
+        return int(float(s[:-1]) * _SIZE_UNITS[s[-1]])
+    return int(s)
+
+
+def render_table(
+    headers: Sequence[str],
+    rows: Sequence[Sequence[str]],
+    aligns: Optional[Sequence[str]] = None,
+    col_styles: Optional[Sequence[Optional[str]]] = None,
+) -> str:
+    """Render a bordered ASCII table whose columns auto-size to their content.
+
+    ``aligns`` is a per-column "<" (left) or ">" (right); by default the first
+    column is left-aligned (the label) and the rest are right-aligned (numbers).
+    ``col_styles`` is an optional per-column ANSI style applied to body cells
+    (header is always bold, borders dim). Widths are computed on the *plain*
+    text and color is applied after padding, so escape codes never skew layout.
+    """
+    ncols = len(headers)
+    if aligns is None:
+        aligns = ["<"] + [">"] * (ncols - 1)
+    if col_styles is None:
+        col_styles = [None] * ncols
+
+    # Widest cell (header or any row) sets each column's width.
+    widths = [
+        max([len(str(headers[c]))] + [len(str(r[c])) for r in rows])
+        for c in range(ncols)
+    ]
+
+    bar = OUT.paint("|", DIM)
+    sep = OUT.paint(" | ", DIM)
+
+    def rule() -> str:
+        return OUT.paint("+" + "+".join("-" * (w + 2) for w in widths) + "+", DIM)
+
+    def line(cells: Sequence[str], *, header: bool = False) -> str:
+        parts = []
+        for c in range(ncols):
+            padded = f"{str(cells[c]):{aligns[c]}{widths[c]}}"
+            if header:
+                parts.append(OUT.paint(padded, BOLD))
+            elif col_styles[c]:
+                parts.append(OUT.paint(padded, col_styles[c]))
+            else:
+                parts.append(padded)
+        return f"{bar} " + sep.join(parts) + f" {bar}"
+
+    out = [rule(), line(headers, header=True), rule()]
+    out += [line(r) for r in rows]
+    out.append(rule())
+    return "\n".join(out)
+
+
+# --------------------------------------------------------------------------- #
+# fio: results, invocation, parsing
+# --------------------------------------------------------------------------- #
+
+
+@dataclass(frozen=True)
+class FioResult:
+    """Parsed metrics for one direction (read or write) of an fio run.
+
+    Latencies are completion latency (clat) in microseconds; for synchronous
+    writes that is effectively the durable-commit latency.
+    """
+
+    bw_mbps: float        # decimal MB/s, matching CrystalDiskMark's SI figure
+    iops: float
+    p50_us: float
+    p99_us: float
+    mean_us: float
+
+    @classmethod
+    def zero(cls) -> "FioResult":
+        return cls(0.0, 0.0, 0.0, 0.0, 0.0)
+
+
+def _direction_of(rw: str) -> str:
+    """fio reports 'read' and 'write' sub-objects; pick the active one."""
+    return "read" if "read" in rw else "write"
+
+
+def _aggregate(jobs: list[dict], direction: str) -> FioResult:
+    """Combine per-job fio JSON stats into a single :class:`FioResult`.
+
+    We deliberately run fio *without* --group_reporting and aggregate ourselves,
+    which sidesteps fio's group-merge quirks: sum throughput, average the median
+    latency, and take the worst-case tail (p99) across jobs.
+    """
+
+    def section(job: dict) -> dict:
+        return job.get(direction, {})
+
+    bw_mbps = sum(section(j).get("bw_bytes", 0) for j in jobs) / 1e6
+    iops = sum(section(j).get("iops", 0.0) for j in jobs)
+
+    def percentiles(key: str) -> list[float]:
+        vals = []
+        for j in jobs:
+            pct = section(j).get("clat_ns", {}).get("percentile", {})
+            v = pct.get(key)
+            if v is not None:
+                vals.append(v)
+        return vals
+
+    p50_ns = percentiles("50.000000")
+    p99_ns = percentiles("99.000000")
+    mean_ns = [section(j).get("clat_ns", {}).get("mean", 0.0) for j in jobs]
+
+    return FioResult(
+        bw_mbps=bw_mbps,
+        iops=iops,
+        p50_us=(statistics.mean(p50_ns) / 1000.0) if p50_ns else 0.0,   # avg of medians
+        p99_us=(max(p99_ns) / 1000.0) if p99_ns else 0.0,               # worst tail
+        mean_us=(statistics.mean(mean_ns) / 1000.0) if mean_ns else 0.0,
+    )
+
+
+def run_fio(
+    cfg: Config,
+    *,
+    rw: str,
+    bs: str,
+    iodepth: int,
+    numjobs: int,
+    engine: Optional[str] = None,
+    direct: Optional[bool] = None,
+    sync: bool = False,
+    end_fsync: bool = False,
+) -> FioResult:
+    """Run a single fio job against the shared bench file and parse its JSON.
+
+    ``engine``/``direct`` default to the detected config values; the SLOG path
+    overrides them (psync + buffered + O_SYNC). ``end_fsync`` flushes the device
+    cache at the end of a write run so cached writes can't inflate the result.
+    """
+    engine = engine if engine is not None else cfg.engine
+    direct = direct if direct is not None else cfg.direct
+
+    cmd = [
+        "sudo", "fio",
+        "--name=speedtest",
+        f"--filename={cfg.benchfile}",
+        f"--ioengine={engine}",
+        f"--direct={1 if direct else 0}",
+        f"--rw={rw}",
+        f"--bs={bs}",
+        f"--size={cfg.size}",
+        f"--numjobs={numjobs}",
+        f"--iodepth={iodepth}",
+        "--time_based",
+        f"--runtime={cfg.runtime}",
+        "--output-format=json",
+    ]
+    if sync:
+        cmd.append("--sync=1")        # O_SYNC: every write is a durable commit
+    if end_fsync:
+        cmd.append("--end_fsync=1")   # flush device cache once at the end
+
+    proc = subprocess.run(cmd, capture_output=True, text=True)
+
+    if cfg.verbose:
+        sync_tag = "  sync=1" if sync else ""
+        vsection(f"fio · {rw}  bs={bs}  iodepth={iodepth}  numjobs={numjobs}{sync_tag}")
+        log("  " + ERR.paint("$ " + " ".join(cmd[1:]), DIM, ITALIC))
+        log(ERR.paint(proc.stdout.strip() or "(no stdout)", DIM))
+        if proc.stderr.strip():
+            log(ERR.paint(proc.stderr.strip(), YELLOW))
+
+    try:
+        data = json.loads(proc.stdout)
+        return _aggregate(data["jobs"], _direction_of(rw))
+    except (json.JSONDecodeError, KeyError, ValueError):
+        # fio failed or produced no parseable JSON; report zeros rather than
+        # crashing the whole run (verbose mode above shows what went wrong).
+        return FioResult.zero()
+
+
+def fio_probe(path: str, engine: str, direct: bool) -> bool:
+    """Throwaway 1s fio job to see if an (engine, O_DIRECT) combo works here.
+
+    This is how we stay accurate *and* portable across ext4/xfs/btrfs/ZFS/NFS
+    without the caller needing to know what each filesystem supports.
+    """
+    cmd = [
+        "sudo", "fio", "--name=probe",
+        f"--directory={path}",
+        f"--ioengine={engine}",
+        "--rw=write", "--bs=4k", "--size=1m",
+        f"--direct={1 if direct else 0}",
+        "--time_based", "--runtime=1",
+    ]
+    proc = subprocess.run(cmd, capture_output=True, text=True)
+    for leftover in glob.glob(os.path.join(path, "probe*")):
+        try:
+            os.remove(leftover)
+        except OSError:
+            pass
+    return proc.returncode == 0
+
+
+def detect_io_settings(cfg: Config) -> None:
+    """Fill in ``cfg.engine`` and ``cfg.direct`` if the user didn't force them.
+
+    O_DIRECT bypasses the OS page cache so we measure the device, not RAM. Not
+    every filesystem supports it (older OpenZFS, some NFS), so we probe and fall
+    back to buffered rather than erroring out.
+    """
+    if cfg.engine is None:
+        for engine in ENGINE_CANDIDATES:
+            if fio_probe(cfg.path, engine, direct=False):
+                cfg.engine = engine
+                break
+        else:
+            cfg.engine = "sync"
+
+    if cfg.direct is None:
+        cfg.direct = fio_probe(cfg.path, cfg.engine, direct=True)
+
+
+# --------------------------------------------------------------------------- #
+# Profiles
+# --------------------------------------------------------------------------- #
+
+
+def _meta_line(text: str) -> str:
+    """Color the 'Label :' prefix of a banner metadata line, leave value as-is.
+
+    The value may already contain its own ANSI codes (e.g. a red O_DIRECT
+    warning); since these lines aren't width-aligned that's harmless.
+    """
+    key, sep, val = text.partition(":")
+    if not sep:
+        return "  " + text
+    return "  " + OUT.paint(key + ":", BOLD, BRIGHT_CYAN) + val
+
+
+def banner(title: str, cfg: Config, extra: Sequence[str] = ()) -> None:
+    full = f"speedtest-hd  ·  {title}"
+    inner = len(full) + 4
+
+    print()
+    print(OUT.paint("╭" + "─" * inner + "╮", BOLD, CYAN))
+    print(
+        OUT.paint("│", BOLD, CYAN)
+        + "  " + OUT.paint(full, BOLD, WHITE) + "  "
+        + OUT.paint("│", BOLD, CYAN)
+    )
+    print(OUT.paint("╰" + "─" * inner + "╯", BOLD, CYAN))
+    print(_meta_line(f"Target  : {OUT.paint(cfg.path, CYAN)}"))
+    for line_ in extra:
+        print(_meta_line(line_))
+    print()
+
+
+def cdm_profile(cfg: Config) -> None:
+    """The CrystalDiskMark-style profile: 4 tests x (read, write), MB/s + IOPS."""
+    detect_io_settings(cfg)
+
+    if cfg.direct:
+        direct_label = OUT.paint("enabled (device)", GREEN)
+    else:
+        direct_label = OUT.paint(
+            "DISABLED (buffered -- may reflect RAM cache!)", BOLD, RED
+        )
+
+    banner(
+        "CrystalDiskMark-style storage benchmark",
+        cfg,
+        extra=[
+            f"Engine  : {cfg.engine}    O_DIRECT: {direct_label}",
+            f"Profile : size={cfg.size.upper()}  runtime={cfg.runtime}s/run  (8 runs)",
+        ],
+    )
+
+    rows: list[list[str]] = []
+    for test in CDM_TESTS:
+        step(f"measuring {test.label.strip()} ...")
+        r = run_fio(cfg, rw=test.read_rw, bs=test.bs,
+                    iodepth=test.iodepth, numjobs=test.numjobs)
+        w = run_fio(cfg, rw=test.write_rw, bs=test.bs,
+                    iodepth=test.iodepth, numjobs=test.numjobs, end_fsync=True)
+        rows.append([
+            test.label,
+            f"{r.bw_mbps:.2f}", f"{w.bw_mbps:.2f}",
+            f"{r.iops:.0f}", f"{w.iops:.0f}",
+        ])
+
+    _cleanup(cfg)
+
+    print()
+    print(render_table(
+        ["Test", "Read (MB/s)", "Write (MB/s)", "Read (IOPS)", "Write (IOPS)"],
+        rows,
+        col_styles=[CYAN, GREEN, YELLOW, GREEN, YELLOW],
+    ))
+    print()
+
+
+def slog_profile(cfg: Config) -> None:
+    """SLOG / sync-write latency profile.
+
+    Forces synchronous 4K random writes (O_SYNC) via the portable psync engine,
+    so every write is a ZIL commit -- exercising a ZFS SLOG exactly the way a
+    sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own
+    sync property. Engine/direct detection is only for the banner here.
+    """
+    detect_io_settings(cfg)
+
+    banner(
+        "SLOG / sync-write latency profile",
+        cfg,
+        extra=[
+            "Method  : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine",
+            f"Profile : runtime={cfg.runtime}s/run  size={cfg.size.upper()}",
+            "Note    : every write is a synchronous ZIL commit -- the load your",
+            "          SLOG actually sees. Watch it live in another shell with:",
+            "            zpool iostat -vl <pool> 1",
+        ],
+    )
+
+    # Measure everything first (progress -> stderr), then draw the whole table,
+    # so the "measuring..." lines can't interleave into the middle of it.
+    rows: list[list[str]] = []
+    for threads in SLOG_THREADS:
+        step(f"measuring 4K sync randwrite T{threads} ...")
+        res = run_fio(
+            cfg, rw="randwrite", bs="4k", iodepth=1, numjobs=threads,
+            engine="psync", direct=False, sync=True,
+        )
+        rows.append([
+            f"4K sync T{threads}",
+            f"{res.iops:.0f}", f"{res.bw_mbps:.2f}",
+            f"{res.p50_us:.1f}", f"{res.p99_us:.1f}",
+        ])
+
+    _cleanup(cfg)
+
+    print()
+    print(render_table(
+        ["Test", "IOPS", "MB/s", "p50 lat(us)", "p99 lat(us)"],
+        rows,
+        col_styles=[CYAN, GREEN, GREEN, YELLOW, MAGENTA],
+    ))
+    print()
+    print(OUT.paint("  Healthy Optane SLOG (eg P1600X) single-stream (T1) target:", BOLD))
+    print(OUT.paint("    ~15-25k IOPS, p50 latency ~40-65us.", GREEN)
+          + OUT.paint(" Much higher latency usually means", DIM))
+    print(OUT.paint("    CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling.", DIM))
+    print()
+
+
+def dd_profile(cfg: Config) -> None:
+    """Dependency-free fallback when fio isn't installed.
+
+    Far cruder than fio: a single sequential stream, and the cached-read figure
+    will reflect RAM. Use it only as a rough sanity check.
+    """
+    count_mib = max(1, parse_size_bytes(cfg.size) // (1024**2))
+    nbytes = count_mib * 1024**2
+
+    banner(
+        "basic dd benchmark (fio not installed)",
+        cfg,
+        extra=[f"Profile : {count_mib} MiB sequential stream"],
+    )
+
+    def timed_dd(args: list[str]) -> float:
+        start = time.monotonic()
+        subprocess.run(args, capture_output=True, text=True)
+        elapsed = time.monotonic() - start
+        return (nbytes / elapsed / 1e6) if elapsed > 0 else 0.0  # decimal MB/s
+
+    step("measuring sequential write ...")
+    write_mbps = timed_dd([
+        "dd", "if=/dev/zero", f"of={cfg.benchfile}",
+        "bs=1M", f"count={count_mib}", "conv=fdatasync,notrunc",
+    ])
+
+    step("dropping caches for uncached read ...")
+    subprocess.run(["sudo", "sh", "-c", "echo 3 > /proc/sys/vm/drop_caches"],
+                   capture_output=True, text=True)
+
+    step("measuring uncached read ...")
+    uncached_mbps = timed_dd([
+        "dd", f"if={cfg.benchfile}", "of=/dev/null", "bs=1M", f"count={count_mib}",
+    ])
+
+    step("measuring cached read ...")
+    cached_mbps = timed_dd([
+        "dd", f"if={cfg.benchfile}", "of=/dev/null", "bs=1M", f"count={count_mib}",
+    ])
+
+    _cleanup(cfg)
+
+    print()
+    print(render_table(
+        ["Test", "MB/s"],
+        [
+            ["Sequential write", f"{write_mbps:.2f}"],
+            ["Uncached read", f"{uncached_mbps:.2f}"],
+            ["Cached read (RAM)", f"{cached_mbps:.2f}"],
+        ],
+        col_styles=[CYAN, GREEN],
+    ))
+    print()
+
+
+def _cleanup(cfg: Config) -> None:
+    """Remove the shared bench file."""
+    try:
+        os.remove(cfg.benchfile)
+    except OSError:
+        pass
+
+
+# --------------------------------------------------------------------------- #
+# CLI
+# --------------------------------------------------------------------------- #
+
+
+def build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="speedtest-hd.py",
+        description="CrystalDiskMark-style storage benchmark built on fio.",
+        epilog=(
+            "Examples:\n"
+            "  speedtest-hd.py .\n"
+            "  speedtest-hd.py /mnt/nvmepool --runtime=10 --size=4g\n"
+            "  speedtest-hd.py /mnt/nfsshare --buffered\n"
+            "  speedtest-hd.py /mnt/nvme-ultra-r10/vm-root --slog --runtime=30\n"
+        ),
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    p.add_argument("path", help="directory/mount to benchmark ('.' for cwd)")
+
+    mode = p.add_mutually_exclusive_group()
+    mode.add_argument("--fio", action="store_const", const="cdm", dest="mode",
+                      help="force the fio CrystalDiskMark-style profile")
+    mode.add_argument("--dd", action="store_const", const="dd", dest="mode",
+                      help="force the basic dd fallback test")
+    mode.add_argument("--slog", action="store_const", const="slog", dest="mode",
+                      help="SLOG / sync-write latency profile (ZFS ZIL)")
+
+    direct = p.add_mutually_exclusive_group()
+    direct.add_argument("--direct", action="store_const", const=True, dest="direct",
+                        help="force O_DIRECT (bypass page cache)")
+    direct.add_argument("--buffered", action="store_const", const=False, dest="direct",
+                        help="force buffered IO (e.g. if O_DIRECT unsupported)")
+
+    p.add_argument("--engine", choices=ENGINE_CANDIDATES,
+                   help="force a specific IO engine (default: auto-detect)")
+    p.add_argument("--runtime", type=int, default=5, metavar="SEC",
+                   help="seconds per run (default: 5, like CrystalDiskMark)")
+    p.add_argument("--size", default="1g", metavar="SIZE",
+                   help="test file size (default: 1g)")
+    p.add_argument("--verbose", action="store_true",
+                   help="also print the full fio output for every run")
+    p.add_argument("-y", "--yes", action="store_true", dest="assume_yes",
+                   help="skip the confirmation prompt")
+
+    p.set_defaults(mode=None, direct=None)
+    return p
+
+
+def confirm(cfg: Config) -> None:
+    """Guard prompt -- we're about to write a multi-GB file to the target."""
+    if cfg.assume_yes:
+        return
+    print(OUT.paint("NOTICE:", BOLD, YELLOW)
+          + f" {cfg.size.upper()} free space on "
+          + OUT.paint(f"'{cfg.path}'", CYAN)
+          + " is required to perform the benchmark.")
+    answer = input(f"Are you ready to start a storage benchmark against "
+                   f"'{cfg.path}' ? ")
+    if not answer.strip().lower().startswith("y"):
+        print(OUT.paint("Ok, cancelled!", YELLOW))
+        sys.exit(0)
+    print(OUT.paint("Great! Starting benchmark now!", BOLD, GREEN))
+
+
+def main(argv: Optional[Sequence[str]] = None) -> int:
+    args = build_parser().parse_args(argv)
+
+    path = os.getcwd() if args.path == "." else args.path
+    if not os.path.exists(path):
+        print(f"Path {path} does not exist", file=sys.stderr)
+        return 1
+
+    cfg = Config(
+        path=path,
+        mode=args.mode,
+        engine=args.engine,
+        direct=args.direct,
+        runtime=args.runtime,
+        size=args.size,
+        verbose=args.verbose,
+        assume_yes=args.assume_yes,
+        benchfile=os.path.join(path, "speedtest-hd.bench"),
+    )
+
+    have_fio = shutil.which("fio") is not None
+
+    # Resolve the mode: explicit flag wins; otherwise fio if available, else dd.
+    mode = cfg.mode
+    if mode in ("cdm", "slog") and not have_fio:
+        print(ERR.paint("ERROR:", BOLD, RED)
+              + " --fio/--slog require fio (apt install fio / pacman -S fio).",
+              file=sys.stderr)
+        return 1
+    if mode is None:
+        if have_fio:
+            mode = "cdm"
+        else:
+            print(OUT.paint("\nfio is not installed -- falling back to basic dd test.",
+                            YELLOW))
+            print(OUT.paint("Install fio for the full CrystalDiskMark-style benchmark.",
+                            DIM))
+            mode = "dd"
+
+    confirm(cfg)
+
+    {"cdm": cdm_profile, "slog": slog_profile, "dd": dd_profile}[mode](cfg)
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        sys.exit(main())
+    except KeyboardInterrupt:
+        print(ERR.paint("\nInterrupted.", YELLOW), file=sys.stderr)
+        sys.exit(130)

speedtest-hd.sh

@@ -1,8 +1,11 @@
 #!/usr/bin/env bash
 
-# Robust HD/SDD/NVMe performance CLI utility
+# Robust HD/SSD/NVMe performance CLI utility
-# Utilizing FIO for sequential/random writes/writes
+# A CrystalDiskMark-style storage benchmark for Linux, built on fio.
-# Dependencies: fio (apt install fio)
+# Runs the exact same four tests CrystalDiskMark does (SEQ1M Q8T1, SEQ1M Q1T1,
+# RND4K Q32T1, RND4K Q32T16), each measured for both Read and Write, and prints
+# the results in a CrystalDiskMark-style table.
+# Dependencies: fio (apt install fio / pacman -S fio). Falls back to dd if missing.
 # See: https://cloud.google.com/compute/docs/disks/benchmarking-pd-performance
 # See: https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/
 # mReschke 2024-01-18
@@ -10,20 +13,23 @@
 # CLI Parameters
 path="$1"
 option=""
-simple=false
 engine=""        # IO engine; empty = auto-detect (io_uring > libaio > posixaio > sync)
 direct=""        # O_DIRECT; empty = auto-detect; 1 = bypass page cache, 0 = buffered
-runtime=30       # seconds per test (longer = more accurate, exposes SLC cache exhaustion)
+runtime=5        # seconds per measurement (CrystalDiskMark default is 5s)
+size="1g"        # test file size (CrystalDiskMark default is 1GiB)
+verbose=0        # 1 = also print the full fio output for every run
 for arg in "${@:2}"; do
     case "$arg" in
-        --simple) simple=true ;;
         --dd) option="--dd" ;;
         --fio) option="--fio" ;;
+        --slog) option="--slog" ;;
         --buffered) direct=0 ;;
         --direct) direct=1 ;;
+        --verbose) verbose=1 ;;
         --engine=*) engine="${arg#*=}" ;;
         --direct=*) direct="${arg#*=}" ;;
         --runtime=*) runtime="${arg#*=}" ;;
+        --size=*) size="${arg#*=}" ;;
     esac
 done
 
@@ -47,275 +53,35 @@ function main {
     fi
 
     # Must type y or n THEN press enter (which I like better)
-    echo "NOTICE: 1GB free space on '$path' is required to perform the benchmark."
+    echo "NOTICE: ${size^^} free space on '$path' is required to perform the benchmark."
-    echo -n "Are you ready to start a robust IO benchmark against '$path' ?"; read answer
+    echo -n "Are you ready to start a storage benchmark against '$path' ? "; read answer
     if [ "$answer" != "${answer#[Yy]}" ]; then
         echo "Great! Starting benchmark now!"
-        echo "------------------------------"
     else
         echo "Ok, cancelled!"
         exit 0
     fi
 
-    # Use dd of fio based on param or defaults
+    # Use dd or fio based on param or defaults
     if [ "$option" == "--dd" ]; then
         dd_speedtest
     elif [ "$option" == "--fio" ]; then
-        fio_speedtest
+        cdm_speedtest
+    elif [ "$option" == "--slog" ]; then
+        slog_speedtest
     elif [ "$option" == "" ]; then
         # If fio is installed, use it, else use dd
-        [ "$simple" != true ] && echo ""
         if ! command -v fio &> /dev/null; then
+            echo ""
+            echo "fio is not installed -- falling back to basic dd test."
+            echo "Install fio for the full CrystalDiskMark-style benchmark."
             dd_speedtest
         else
-            fio_speedtest
+            cdm_speedtest
         fi
     fi
 }
 
-function print_result {
-    # Print a single fio test result, either as the full "Run status group"
-    # line or, when --simple is given, as a compact aligned "Label  value" row.
-    # The simple value is taken from the parenthetical (MB/s) figure that
-    # follows the first bw= value in fio's output.
-    local label="$1"
-    local output="$2"
-    local status
-    status=$(echo "  - $output " | /usr/bin/grep -A1 'Run status group' | tail -n1)
-    if [ "$simple" == true ]; then
-        local value
-        # Grab the parenthetical (MB/s) figure after the first bw=, then put a
-        # space between the number and the unit, eg "98.4MB/s" -> "98.4 MB/s".
-        value=$(echo "$status" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | sed -E 's/([0-9.]+)([A-Za-z])/\1 \2/')
-        printf "%-28s%s\n" "$label" "$value"
-    else
-        echo "$status"
-    fi
-}
-
-function fio_write_single_random_4k {
-    # 4K Random Writes (parallel, deep queue)
-
-    # Random 4K writes are the worst thing you can ask a disk to do. Where this
-    # happens most in real life: copying home directories and dotfiles, email
-    # stores, some database operations, source code trees.
-
-    # IMPORTANT: the original version of this test used numjobs=1 + iodepth=1.
-    # That measures single-op LATENCY (the absolute worst case): the OS issues
-    # one 4K write, waits for it to be acknowledged, then issues the next. On
-    # NVMe that lands around ~3000 IOPS / ~12MB/sec -- which looks alarmingly
-    # slow but is NOT a bug. NVMe's entire advantage is deep command queues and
-    # parallelism, so a queue depth of 1 deliberately measures the one scenario
-    # NVMe is bad at. To measure the drive's actual capability we now drive it
-    # with multiple jobs and a deep queue (4 jobs x iodepth=64), keeping
-    # hundreds of operations in flight at once. Expect this to jump into the
-    # hundreds of MB/sec (often 1GB/sec+) on a healthy NVMe pool.
-
-    # --name= is a required argument, but it's basically human-friendly fluff—fio will create files based on that name to test with, inside the directory you point it at.
-    # --directory= where the test files are created; this is the path/mount you are benchmarking.
-    # --ioengine=$engine the engine fio uses to talk to the kernel, auto-detected by detect_io_settings(). On modern Linux this is io_uring (lowest overhead); it falls back to libaio, then posixaio, then sync. NOTE: libaio is only truly asynchronous with --direct=1; with buffered IO it silently degrades to synchronous, which is exactly why io_uring is preferred.
-    # --rw=randwrite random write operations. Other options include read, write (sequential), randread, and randrw.
-    # --bs=4k blocksize 4K. These are very small individual operations. This is where the pain lives; it's hard on the disk and adds a ton of command-channel overhead, since a separate operation has to be commanded for each 4K of data.
-    # --size=256m each job's test file is 256MB. With --numjobs=4 that's 4 files totaling ~1GB.
-    # --numjobs=4 run 4 parallel processes, each with its own test file, to feed the drive from multiple threads at once (single-threaded cannot saturate NVMe).
-    # --iodepth=64 how many commands each job keeps stacked in the queue at once. Deep queues are how NVMe reaches its real IOPS; 4 jobs x 64 = up to 256 ops in flight.
-    # --direct=$direct O_DIRECT, auto-detected. 1 bypasses the OS page cache so we measure the device instead of RAM; 0 (buffered) is the fallback for filesystems/mounts that reject O_DIRECT (eg older OpenZFS, some NFS).
-    # --ramp_time=2s discard the first 2 seconds of results so we measure steady state, not the initial warm-up burst.
-    # --runtime=$runtime --time_based run for this many seconds, looping over the file(s) if we finish early.
-    # --group_reporting=1 aggregate all jobs into a single combined result line instead of one line per job.
-    # --end_fsync=1 after the timed run, flush everything to stable storage and count that time, so cached writes can't inflate the number.
-    [ "$simple" != true ] && { echo ""; echo "4K Random Writes (bs=4k, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; }
-    x=`sudo fio \
-        --name=fio-write-random-4k \
-        --directory=$path \
-        --ioengine=$engine \
-        --rw=randwrite \
-        --bs=4k \
-        --size=256m \
-        --numjobs=4 \
-        --iodepth=64 \
-        --direct=$direct \
-        --ramp_time=2s \
-        --time_based --runtime=$runtime \
-        --group_reporting=1 \
-        --end_fsync=1`
-    print_result "4K Random Writes" "$x"
-
-    # Cleanup my test files
-    rm -rf $path/fio-write-random-4k*
-}
-
-function fio_write_parallel_random_64k {
-    # Parallel 64k Random Writes
-
-    # This time, we're creating 16 separate 64MB files (still totaling 1GB, when
-    # all put together) and we're issuing 64KB blocksized random write operations.
-    # We're doing it with sixteen separate processes running in parallel, and
-    # we're queuing up to 16 simultaneous asynchronous ops before we pause and wait
-    # for the OS to start acknowledging their receipt.
-
-    # This is a pretty decent approximation of a significantly busy system. It's
-    # not doing any one particularly nasty thing—like running a database engine or
-    # copying tons of dotfiles from a user's home directory—but it is coping with
-    # a bunch of applications doing moderately demanding stuff all at once.
-
-    # This is also a pretty good, slightly pessimistic approximation of a busy,
-    # multi-user system like a NAS, which needs to handle multiple 1MB operations
-    # simultaneously for different users. If several people or processes are trying
-    # to read or write big files (photos, movies, whatever) at once, the OS tries
-    # to feed them all data simultaneously. This pretty quickly devolves down to a
-    # pattern of multiple random small block access. So in addition to "busy desktop
-    # with lots of apps," think "busy fileserver with several people actively using it."
-
-    # You will see a lot more variation in speed as you watch this operation play
-    # out on the console. Unlike the steady trickle you'd get from a queue-depth-1
-    # single-op latency test, this 16-process job can fluctuate wildly—eg between
-    # about 10MiB/sec and 300MiB/sec during the run—as the OS and SSD firmware
-    # catch good and bad luck aggregating writes.
-
-    # Most of the variation you're seeing here is due to the operating system and
-    # SSD firmware sometimes being able to aggregate multiple writes. When it
-    # manages to aggregate them helpfully, it can write them in a way that allows
-    # parallel writes to all the individual physical media stripes inside the SSD.
-    # Sometimes, it still ends up having to give up and write to only a single
-    # physical media stripe at a time—or a garbage collection or other maintenance
-    # operation at the SSD firmware level needs to run briefly in the background,
-    # slowing things down.
-    [ "$simple" != true ] && { echo ""; echo "Parallel 64K Random Writes (bs=64k, jobs=16, iodepth=16, engine=$engine, direct=$direct, ${runtime}s)"; }
-    x=`sudo fio \
-        --name=fio-write-random-64k \
-        --directory=$path \
-        --ioengine=$engine \
-        --rw=randwrite \
-        --bs=64k \
-        --size=64m \
-        --numjobs=16 \
-        --iodepth=16 \
-        --direct=$direct \
-        --ramp_time=2s \
-        --time_based --runtime=$runtime \
-        --group_reporting=1 \
-        --end_fsync=1`
-    print_result "Parallel 64K Random Writes" "$x"
-
-    # Cleanup my test files
-    rm -rf $path/fio-write-random-64k*
-}
-
-function fio_write_single_sequential_1m {
-    # Single 1M Sequential Writes
-
-    # This is pretty close to the best-case scenario for a real-world system
-    # doing real-world things. No, it's not quite as fast as a single, truly
-    # contiguous write... but the 1MiB blocksize is large enough that it's quite
-    # close. Besides, if literally any other disk activity is requested simultaneously
-    # with a contiguous write, the "contiguous" write devolves to this level of
-    # performance pretty much instantly, so this is a much more realistic test of
-    # the upper end of storage performance on a typical system.
-
-    # You'll see some kooky fluctuations on SSDs when doing this test. This is largely
-    # due to the SSD's firmware having better luck or worse luck at any given time,
-    # when it's trying to queue operations so that it can write across all physical
-    # media stripes cleanly at once. Rust disks will tend to provide a much more
-    # consistent, though typically lower, throughput across the run.
-
-    # You can also see SSD performance fall off a cliff here if you exhaust an
-    # onboard write cache—TLC and QLC drives tend to have small write cache areas
-    # made of much faster MLC or SLC media. Once those get exhausted, the disk has
-    # to drop to writing directly to the much slower TLC/QLC media where the data
-    # eventually lands. This is the major difference between, for example, Samsung
-    # EVO and Pro SSDs—the EVOs have slow TLC media with a fast MLC cache, where
-    # the Pros use the higher-performance, higher-longevity MLC media throughout
-    # the entire SSD.
-
-    # If you have any doubt at all about a TLC or QLC disk's ability to sustain
-    # heavy writes, you may want to experimentally extend your time duration here.
-    # If you watch the throughput live as the job progresses, you'll see the impact
-    # immediately when you run out of cache—what had been a fairly steady,
-    # several-hundred-MiB/sec throughput will suddenly plummet to half the speed
-    # or less and get considerably less stable as well.
-
-    # However, you might choose to take the opposite position—you might not
-    # expect to do sustained heavy writes very frequently, in which case you
-    # actually are more interested in the on-cache behavior. What's important
-    # here is that you understand both what you want to test, and how to test
-    # it accurately.
-
-    [ "$simple" != true ] && { echo ""; echo "Single 1M Sequential Writes (bs=1m, jobs=1, iodepth=16, engine=$engine, direct=$direct, ${runtime}s)"; }
-    x=`sudo fio \
-        --name=fio-write-seq-1m \
-        --directory=$path \
-        --ioengine=$engine \
-        --rw=write \
-        --bs=1m \
-        --size=1g \
-        --numjobs=1 \
-        --iodepth=16 \
-        --direct=$direct \
-        --ramp_time=2s \
-        --time_based --runtime=$runtime \
-        --group_reporting=1 \
-        --end_fsync=1`
-    print_result "Single 1M Sequential Writes" "$x"
-
-    # Cleanup my test files
-    rm -rf $path/fio-write-seq-1m*
-}
-
-function fio_read_sequential_1m {
-    # Sequential Parallel Reads
-
-    [ "$simple" != true ] && { echo ""; echo "Sequential 4x 1M Reads (bs=1m, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; }
-    x=`sudo fio \
-        --name=fio-read-sequential-1m \
-        --directory=$path \
-        --ioengine=$engine \
-        --bs=1M \
-        --numjobs=4 \
-        --size=256M \
-        --time_based --runtime=$runtime \
-        --ramp_time=2s \
-        --direct=$direct \
-        --verify=0 \
-        --iodepth=64 \
-        --rw=read \
-        --group_reporting=1 \
-        --iodepth_batch_submit=64 \
-        --iodepth_batch_complete_max=64`
-    print_result "Sequential 4x 1M Reads" "$x"
-    rm -rf $path/fio-read-sequential-1m*
-}
-
-function fio_read_random_4k {
-    # Random 4k Reads (parallel, deep queue)
-
-    # 4K random reads are the highest-IOPS thing a drive does, but a single
-    # submitting thread goes CPU-bound long before an NVMe runs out of capacity,
-    # so it under-reports peak IOPS. We spread the work over 4 jobs x iodepth=64
-    # (256 ops in flight, like the 4K write test) to saturate fast drives, while
-    # still reporting the honest (low) number on a spinning HDD. size=256m per
-    # job keeps the total footprint at ~1GB across the 4 jobs.
-    [ "$simple" != true ] && { echo ""; echo "Random 4k Reads (bs=4k, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; }
-    x=`sudo fio \
-        --name=fio-read-random-4k \
-        --directory=$path \
-        --ioengine=$engine \
-        --rw=randread \
-        --bs=4k \
-        --size=256m \
-        --numjobs=4 \
-        --time_based --runtime=$runtime \
-        --ramp_time=2s \
-        --direct=$direct \
-        --verify=0 \
-        --iodepth=64 \
-        --group_reporting=1 \
-        --iodepth_batch_submit=64 \
-        --iodepth_batch_complete_max=64`
-    print_result "Random 4k Reads" "$x"
-    rm -rf $path/fio-read-random-4k*
-}
-
 function fio_probe {
     # Run a tiny throwaway fio job to see if a given (engine, direct) combo
     # actually works on this path/filesystem. Returns 0 on success. This is how
@@ -353,57 +119,291 @@ function detect_io_settings {
     if [ -z "$direct" ]; then
         if fio_probe "$engine" 1; then direct=1; else direct=0; fi
     fi
-
-    if [ "$simple" != true ]; then
-        echo ""
-        echo "IO engine : $engine"
-        if [ "$direct" == 1 ]; then
-            echo "O_DIRECT  : enabled (bypassing page cache -- measuring the device)"
-        else
-            echo "O_DIRECT  : DISABLED (buffered) -- this filesystem/mount rejected"
-            echo "            O_DIRECT, so results (especially reads) may reflect the"
-            echo "            RAM cache rather than the underlying device. Pass"
-            echo "            --direct to force it if you believe it should work."
-        fi
-    fi
 }
 
-function fio_speedtest {
+function fio_bw_mbps {
+    # Extract bandwidth in decimal MB/s (matching CrystalDiskMark) from a fio
+    # run. fio prints eg "  READ: bw=3357MiB/s (3521MB/s), ..." -- we take the
+    # parenthetical SI figure and normalise kB/MB/GB into MB/s.
+    local out="$1"
+    local raw num unit
+    raw=$(echo "$out" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | head -n1)
+    [ -z "$raw" ] && { echo "0.00"; return; }
+    num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1)
+    unit=$(echo "$raw" | /usr/bin/grep -oP '[A-Za-z]+/s' | head -n1)
+    case "$unit" in
+        B/s)        awk -v n="$num" 'BEGIN{printf "%.2f", n/1000000}' ;;
+        kB/s|KB/s)  awk -v n="$num" 'BEGIN{printf "%.2f", n/1000}' ;;
+        MB/s)       awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;;
+        GB/s)       awk -v n="$num" 'BEGIN{printf "%.2f", n*1000}' ;;
+        *)          awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;;
+    esac
+}
+
+function fio_iops {
+    # Extract IOPS from a fio run. fio prints eg "  read: IOPS=12.3k, BW=..." --
+    # the value may carry a k/M suffix, which we expand to a whole number.
+    local out="$1"
+    local raw num unit
+    raw=$(echo "$out" | /usr/bin/grep -oP 'IOPS=\K[0-9.]+[kKmM]?' | head -n1)
+    [ -z "$raw" ] && { echo "0"; return; }
+    num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1)
+    unit=$(echo "$raw" | /usr/bin/grep -oP '[kKmM]' | head -n1)
+    case "$unit" in
+        k|K) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000}' ;;
+        m|M) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000000}' ;;
+        *)   awk -v n="$num" 'BEGIN{printf "%.0f", n}' ;;
+    esac
+}
+
+function run_fio {
+    # run_fio <rw> <bs> <iodepth> <numjobs>  ->  echoes "MB/s  IOPS" (space-sep)
+    # A single shared 1GB test file is reused across every run (laid out once),
+    # which keeps the footprint at one file and avoids re-creating it each time.
+    local rw="$1" bs="$2" qd="$3" jobs="$4"
+    local args=(
+        --name=speedtest
+        --filename="$benchfile"
+        --ioengine="$engine"
+        --direct="$direct"
+        --rw="$rw"
+        --bs="$bs"
+        --size="$size"
+        --numjobs="$jobs"
+        --iodepth="$qd"
+        --time_based --runtime="$runtime"
+        --group_reporting=1
+    )
+    # Flush device cache at the end of write tests so cached writes can't lie.
+    case "$rw" in
+        *write) args+=(--end_fsync=1) ;;
+    esac
+    # Capture the full fio output. In verbose mode we also keep fio's stderr
+    # (2>&1); otherwise we discard it. Either way only the parsed bandwidth is
+    # echoed to stdout, so the captured value stays clean.
+    local out
+    if [ "$verbose" = 1 ]; then
+        out=$(sudo fio "${args[@]}" 2>&1)
+    else
+        out=$(sudo fio "${args[@]}" 2>/dev/null)
+    fi
+    # Dump the raw fio output (to stderr, so it never pollutes the captured
+    # values on stdout) when running verbose.
+    if [ "$verbose" = 1 ]; then
+        echo "  --- fio: $rw bs=$bs iodepth=$qd numjobs=$jobs ---" >&2
+        echo "$out" >&2
+        echo "" >&2
+    fi
+    echo "$(fio_bw_mbps "$out") $(fio_iops "$out")"
+}
+
+# Table drawing -------------------------------------------------------------
+# Label cell is "| %-16s |" (18 dashes); the four numeric cells are "| %14s |"
+# (16 dashes each).
+TBL_SEG_LBL="------------------"   # 18 dashes -> matches "| %-16s |"
+TBL_SEG_NUM="----------------"     # 16 dashes -> matches "| %14s |"
+function tbl_rule {
+    printf "+%s+%s+%s+%s+%s+\n" "$TBL_SEG_LBL" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM"
+}
+function tbl_row {
+    # tbl_row <test> <read-mb/s> <write-mb/s> <read-iops> <write-iops>
+    # (col1 left-aligned, the four numeric cols right-aligned)
+    printf "| %-16s | %14s | %14s | %14s | %14s |\n" "$1" "$2" "$3" "$4" "$5"
+}
+
+# Five-column table for the --slog latency profile.
+# "| %-16s |" -> 18 dashes ; "| %12s |" -> 14 dashes
+SLOG_SEG16="------------------"   # 18 dashes
+SLOG_SEG12="--------------"       # 14 dashes
+function slog_rule {
+    printf "+%s+%s+%s+%s+%s+\n" "$SLOG_SEG16" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12"
+}
+function slog_row {
+    # slog_row <test> <iops> <mb/s> <p50> <p99>  (col1 left, rest right-aligned)
+    printf "| %-16s | %12s | %12s | %12s | %12s |\n" "$1" "$2" "$3" "$4" "$5"
+}
+
+function run_fio_sync {
+    # run_fio_sync <bs> <numjobs>
+    #   -> echoes "IOPS  MB/s  p50_us  p99_us  mean_us"  (one space-separated line)
+    # Forces synchronous IO (--sync=1 => O_SYNC) with the portable psync engine so
+    # every write is a ZIL commit -- i.e. it exercises the SLOG exactly the way a
+    # sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own
+    # sync property. We deliberately DON'T use --group_reporting here: each job is
+    # reported independently in the JSON and we aggregate ourselves (sum IOPS/bw,
+    # average p50, take the worst p99), which sidesteps fio's group-merge quirks.
+    local bs="$1" jobs="$2"
+    local args=(
+        --name=slog
+        --filename="$benchfile"
+        --ioengine=psync
+        --rw=randwrite
+        --bs="$bs"
+        --size="$size"
+        --numjobs="$jobs"
+        --sync=1
+        --time_based --runtime="$runtime"
+        --output-format=json
+    )
+    local out
+    out=$(sudo fio "${args[@]}" 2>/dev/null)
+    if [ "$verbose" = 1 ]; then
+        echo "  --- fio json: randwrite bs=$bs numjobs=$jobs sync=1 ---" >&2
+        echo "$out" >&2
+        echo "" >&2
+    fi
+    # Parse the JSON: sum IOPS and bandwidth across jobs, aggregate completion
+    # latency percentiles (clat is in ns -> convert to us). bw_bytes is bytes/s.
+    echo "$out" | python3 -c '
+import json, sys
+try:
+    d = json.load(sys.stdin)
+    jobs = d["jobs"]
+except Exception:
+    print("0 0.00 0.0 0.0 0.0"); sys.exit(0)
+iops = sum(j["write"]["iops"] for j in jobs)
+mbps = sum(j["write"]["bw_bytes"] for j in jobs) / 1e6
+def pctl(j, p):
+    return j["write"]["clat_ns"]["percentile"].get(p)
+p50 = [pctl(j, "50.000000") for j in jobs if pctl(j, "50.000000") is not None]
+p99 = [pctl(j, "99.000000") for j in jobs if pctl(j, "99.000000") is not None]
+mean = [j["write"]["clat_ns"]["mean"] for j in jobs]
+p50_us  = (sum(p50)/len(p50))/1000.0 if p50 else 0.0          # average of per-job medians
+p99_us  = (max(p99))/1000.0          if p99 else 0.0          # worst-case tail across jobs
+mean_us = (sum(mean)/len(mean))/1000.0 if mean else 0.0
+print("%.0f %.2f %.1f %.1f %.1f" % (iops, mbps, p50_us, p99_us, mean_us))
+'
+}
+
+function slog_speedtest {
+    # SLOG / sync-write latency profile. Detect engine/direct only for the banner;
+    # the actual runs always force psync + O_SYNC (see run_fio_sync).
+    detect_io_settings
+    benchfile="$path/speedtest-hd.bench"
+
+    if ! command -v fio &> /dev/null; then
+        echo "ERROR: --slog requires fio (apt install fio / pacman -S fio)." >&2
+        exit 1
+    fi
+    if ! command -v python3 &> /dev/null; then
+        echo "ERROR: --slog requires python3 (used to parse fio JSON latency percentiles)." >&2
+        exit 1
+    fi
+
+    echo ""
+    echo "  speedtest-hd : SLOG / sync-write latency profile"
+    echo "  Target  : $path"
+    echo "  Method  : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine"
+    echo "  Profile : runtime=${runtime}s/run  size=${size^^}"
+    echo "  Note    : every write is a synchronous ZIL commit -- this is the load"
+    echo "            your SLOG actually sees. Watch it live in another shell with:"
+    echo "              zpool iostat -vl <pool> 1"
+    echo ""
+
+    # T1 is the headline single-stream latency; the sweep shows how the SLOG
+    # scales as concurrent sync writers (NFS/iSCSI/VM threads) pile on. Run every
+    # measurement FIRST (progress goes to stderr), collect the results, then draw
+    # the whole table in one block so the "measuring..." lines can't interleave
+    # into the middle of the table.
+    local jobs_list="1 4 8 16"
+    local -a rows
+    local j res iops bw p50 p99 mean
+    for j in $jobs_list; do
+        echo "  measuring 4K sync randwrite T$j ..." >&2
+        res=$(run_fio_sync 4k "$j")
+        read -r iops bw p50 p99 mean <<< "$res"
+        rows+=("$(slog_row "4K sync T$j" "$iops" "$bw" "$p50" "$p99")")
+    done
+
+    rm -f "$benchfile"
+
+    # Render the assembled table
+    echo ""
+    slog_rule
+    slog_row "Test" "IOPS" "MB/s" "p50 lat(us)" "p99 lat(us)"
+    slog_rule
+    printf '%s\n' "${rows[@]}"
+    slog_rule
+    echo ""
+    echo "  Healthy Optane SLOG (eg P1600X) single-stream (T1) target:"
+    echo "    ~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means"
+    echo "    CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling."
+    echo ""
+}
+
+function cdm_speedtest {
     # Auto-detect the best engine and whether O_DIRECT works on this path.
     detect_io_settings
+    benchfile="$path/speedtest-hd.bench"
 
-    # Write tests
+    local direct_label="enabled (device)"
-    fio_write_single_random_4k
+    [ "$direct" != 1 ] && direct_label="DISABLED (buffered -- may reflect RAM cache!)"
-    fio_write_parallel_random_64k
-    fio_write_single_sequential_1m
 
-    # Read Tests
+    # Banner
-    fio_read_sequential_1m
+    echo ""
-    fio_read_random_4k
+    echo "  speedtest-hd : CrystalDiskMark-style storage benchmark"
+    echo "  Target  : $path"
+    echo "  Engine  : $engine    O_DIRECT: $direct_label"
+    echo "  Profile : size=${size^^}  runtime=${runtime}s/run  (8 runs)"
+    echo ""
+
+    # The four CrystalDiskMark default tests. Q = queue depth (--iodepth),
+    # T = threads (--numjobs). The first three are T1 (single thread); the last
+    # is T16. A single shared test file is reused -- with numjobs>1 every job
+    # issues IO against the same file, which is fine.
+    # Each run now returns both MB/s and IOPS ("bw iops"), so we capture the pair.
+    #          label          read-rw    write-rw    bs   Q   T
+    echo "  measuring SEQ1M Q8T1  ..." >&2
+    local s8r_bw s8r_io;   read -r s8r_bw  s8r_io  <<< "$(run_fio read     1m 8  1)"
+    local s8w_bw s8w_io;   read -r s8w_bw  s8w_io  <<< "$(run_fio write    1m 8  1)"
+    echo "  measuring SEQ1M Q1T1  ..." >&2
+    local s1r_bw s1r_io;   read -r s1r_bw  s1r_io  <<< "$(run_fio read     1m 1  1)"
+    local s1w_bw s1w_io;   read -r s1w_bw  s1w_io  <<< "$(run_fio write    1m 1  1)"
+    echo "  measuring RND4K Q32T1 ..." >&2
+    local r32r_bw r32r_io; read -r r32r_bw r32r_io <<< "$(run_fio randread  4k 32 1)"
+    local r32w_bw r32w_io; read -r r32w_bw r32w_io <<< "$(run_fio randwrite 4k 32 1)"
+    echo "  measuring RND4K Q32T16 ..." >&2
+    local r1r_bw r1r_io;   read -r r1r_bw  r1r_io  <<< "$(run_fio randread  4k 32 16)"
+    local r1w_bw r1w_io;   read -r r1w_bw  r1w_io  <<< "$(run_fio randwrite 4k 32 16)"
+
+    # Cleanup the shared test file
+    rm -f "$benchfile"
+
+    # Render the CrystalDiskMark-style table
+    echo ""
+    tbl_rule
+    tbl_row "Test" "Read (MB/s)" "Write (MB/s)" "Read (IOPS)" "Write (IOPS)"
+    tbl_rule
+    tbl_row "SEQ1M  Q8T1"   "$s8r_bw"  "$s8w_bw"  "$s8r_io"  "$s8w_io"
+    tbl_row "SEQ1M  Q1T1"   "$s1r_bw"  "$s1w_bw"  "$s1r_io"  "$s1w_io"
+    tbl_row "RND4K  Q32T1"  "$r32r_bw" "$r32w_bw" "$r32r_io" "$r32w_io"
+    tbl_row "RND4K  Q32T16" "$r1r_bw"  "$r1w_bw"  "$r1r_io"  "$r1w_io"
+    tbl_rule
+    echo ""
 }
 
 function dd_speedtest {
-    # Basic HD speed test using DD
+    # Basic HD speed test using dd (fallback when fio is not installed)
     # mReschke 2017-07-11
-    
+    local file ddsize
     file=$path/bigfile
-    size=1024
+    ddsize=1024
 
     echo "Running dd based HD/SSD/NVMe Benchmarks"
     echo "---------------------------------------"
 
     printf "Cached write speed...\n"
-    dd if=/dev/zero of=$file bs=1M count=$size
+    dd if=/dev/zero of=$file bs=1M count=$ddsize
 
     printf "\nUncached write speed...\n"
-    dd if=/dev/zero of=$file bs=1M count=$size conv=fdatasync,notrunc
+    dd if=/dev/zero of=$file bs=1M count=$ddsize conv=fdatasync,notrunc
 
     printf "\nUncached read speed...\n"
     echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
-    dd if=$file of=/dev/null bs=1M count=$size
+    dd if=$file of=/dev/null bs=1M count=$ddsize
 
     printf "\nCached read speed...\n"
-    dd if=$file of=/dev/null bs=1M count=$size
+    dd if=$file of=/dev/null bs=1M count=$ddsize
 
     rm $file
     printf "\nDone\n"
@@ -411,40 +411,45 @@ function dd_speedtest {
 
 # Show help and usage information
 function usage {
-    echo "Robust Flexible Input/Output HD Speedtest"
+    echo "speedtest-hd : CrystalDiskMark-style storage benchmark"
-    echo "  If FIO is installed, we use FIO for more detailed performance analysis."
+    echo "  Runs the same four tests as CrystalDiskMark (each Read + Write):"
-    echo "  If FIO is not installed, we use basic DD analysis."
+    echo "    SEQ1M  Q8T1   Sequential 1MiB, queue depth 8,  1 thread"
-    echo "  You should apt install fio (pacman -S fio) for detailed analysis."
+    echo "    SEQ1M  Q1T1   Sequential 1MiB, queue depth 1,  1 thread"
+    echo "    RND4K  Q32T1  Random 4KiB,     queue depth 32, 1 thread"
+    echo "    RND4K  Q32T16 Random 4KiB,     queue depth 32, 16 threads"
+    echo "  Uses fio if installed (recommended), else basic dd."
     echo "mReschke 2024-01-18"
     echo ""
-    echo "NOTICE, this creates a 1GB file on the desired destination disk."
+    echo "NOTICE: creates one test file (default 1GB) on the destination disk."
-    echo "Please ensure you have write access with 1GB free space on destination."
+    echo "Please ensure you have write access with enough free space."
     echo ""
     echo "Usage:"
-    echo "  This will use FIO if installed, else DD"
+    echo "  Auto (fio if installed, else dd):"
     echo "    ./speedtest-hd /mnt/somedisk"
     echo "    ./speedtest-hd ."
     echo ""
-    echo "  This will force FIO"
+    echo "  Force fio / force dd:"
     echo "    ./speedtest-hd /mnt/somedisk --fio"
-    echo "    ./speedtest-hd . --fio"
-    echo ""
-    echo "  This will force DD"
     echo "    ./speedtest-hd /mnt/somedisk --dd"
-    echo "    ./speedtest-hd . --dd"
     echo ""
-    echo "  Add --simple (FIO only) for a compact, aligned summary of MB/s values"
+    echo "  SLOG / sync-write latency profile (ZFS ZIL, NFS/iSCSI/VM sync IO):"
-    echo "    ./speedtest-hd . --simple"
+    echo "    ./speedtest-hd /mnt/zpool/dataset --slog"
-    echo "    ./speedtest-hd . --fio --simple"
+    echo "    4K synchronous writes at T1/T4/T8/T16; reports IOPS, MB/s, p50/p99"
+    echo "    commit latency. Requires fio + python3. (Tip: watch the SLOG with"
+    echo "    'zpool iostat -vl <pool> 1' in another shell.)"
     echo ""
-    echo "  FIO tuning flags (all auto-detected by default, override as needed):"
+    echo "  Tuning flags (auto-detected by default, override as needed):"
     echo "    --engine=io_uring|libaio|posixaio|sync   IO engine (default: auto)"
     echo "    --direct        Force O_DIRECT (bypass page cache)"
     echo "    --buffered      Force buffered IO (eg if O_DIRECT is unsupported)"
-    echo "    --runtime=SEC   Seconds per test (default: 30)"
+    echo "    --runtime=SEC   Seconds per run (default: 5, like CrystalDiskMark)"
+    echo "    --size=SIZE     Test file size (default: 1g)"
+    echo "    --verbose       Print full fio output for every run (summary table unchanged)"
     echo "  Examples:"
-    echo "    ./speedtest-hd /mnt/nvmepool --engine=io_uring --runtime=60"
+    echo "    ./speedtest-hd /mnt/nvmepool --runtime=10 --size=4g"
     echo "    ./speedtest-hd /mnt/nfsshare --buffered"
+    echo "    ./speedtest-hd /mnt/nvmepool --verbose"
+    echo "    ./speedtest-hd /mnt/nvme-ultra-r10/vm-root --slog --runtime=30"
     exit 0
 }