diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..377139a --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,65 @@ +# speedtest-hd + +Single-file Bash CLI (`speedtest-hd.sh`) that benchmarks any mounted path (HDD/SSD/NVMe, +local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII table. Falls back +to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount" disk tester — not +a ZFS/NVMe-specific tool. + +## Usage +``` +./speedtest-hd.sh [--fio|--dd|--slog] [flags] +``` +- Default: fio if installed, else dd. `--fio` forces CDM mode, `--slog` forces sync-latency mode. +- Flags: `--engine=io_uring|libaio|posixaio|sync` (default auto), `--direct`/`--buffered` + (default auto-probe), `--runtime=SEC` (default 5), `--size=SIZE` (default 1g), `--verbose` + (also dumps raw fio output to stderr). + +## Modes & layout +- **CDM mode** (`cdm_speedtest`): runs CrystalDiskMark's default tests, each read+write, + prints a 5-col table (Read/Write MB/s, Read/Write IOPS). Tests: `SEQ1M Q8T1`, `SEQ1M Q1T1`, + `RND4K Q32T1`, `RND4K Q32T16`. Q = `--iodepth`, T = `--numjobs`. +- **SLOG mode** (`slog_speedtest`): 4K synchronous randwrite sweep at T1/T4/T8/T16 to profile + ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 commit + latency. Requires `fio` + `python3` (JSON percentile parsing). +- **dd mode** (`dd_speedtest`): legacy cached/uncached read/write fallback. + +## Key implementation notes +- `detect_io_settings` auto-picks the engine (io_uring → libaio → posixaio → sync) and probes + whether O_DIRECT works, via tiny throwaway `fio_probe` jobs. Falls back to buffered with a + warning if O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly async + with `--direct=1`** — that's why io_uring is preferred. O_DIRECT bypasses the page cache so we + measure the device, not RAM (buffered results, esp. reads, can reflect ARC/page cache). +- One shared test file `/speedtest-hd.bench` is reused across all runs (laid out once), + removed at the end. Footprint = `--size` (default 1G), matching the startup notice. +- `run_fio` (CDM) returns `"MB/s IOPS"`; `run_fio_sync` (SLOG) forces `--ioengine=psync --sync=1` + (O_SYNC) so every write is a ZIL commit regardless of dataset sync property, and parses fio + **JSON** to aggregate across jobs (sum IOPS/bw, avg p50, worst-case p99). CDM mode parses + fio's text output: `fio_bw_mbps` takes the parenthetical **SI MB/s** (matches CDM's decimal + MB/s, normalizes kB/MB/GB); `fio_iops` expands k/M suffixes. +- Write tests append `--end_fsync=1` so cached writes can't inflate numbers. +- ASCII tables: cell formats and dash-segment widths must stay in sync (`tbl_*` = 18/16 dashes, + `slog_*` = 18/14). Verify alignment after editing. + +## History / decisions +- Originated from the Ars Technica fio guide. Original 4K test used `numjobs=1 iodepth=1`, which + measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1, not + a bug. Refactored toward parallel/deep-queue tests to show real device capability, then fully + reshaped into the CrystalDiskMark profile above. `--simple` flag was removed; the table is now + the only fio output. +- Note: the current `RND4K` rows are `Q32T1` + `Q32T16`. CrystalDiskMark's actual latest default + profile is `Q32T16` + `Q1T1` (single-queue random is the meaningful low-end number). If aligning + strictly to CDM, the `Q32T1` row should become `Q1T1` (`iodepth=1 numjobs=1`). + +## SLOG performance context (why `--slog` exists) +Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10` += 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). Poor sync +writes were **CPU power management**, not the SLOG: +- Fix (biggest last): Dell BIOS profile DAPC → **Performance** (~2×); cstate kernel args; and the + big one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, which parked cores at + 1.2 GHz because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post + Init: `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`. +- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor). +- Diagnostic: `zpool iostat -vl 1` during fio showed the Optane `logs` vdev at ~90µs + disk_wait — proving the SLOG was fine and latency was upstream (CPU). +- Healthy Optane SLOG single-stream (T1) target: ~15–25k IOPS, p50 ~40–65µs. Much higher usually + = C-states / PCIe ASPM / BIOS power profile throttling. diff --git a/README.md b/README.md new file mode 100644 index 0000000..8192c30 --- /dev/null +++ b/README.md @@ -0,0 +1,291 @@ +# speedtest-hd + +A robust, CrystalDiskMark‑style storage benchmark for Linux, built on [`fio`](https://fio.readthedocs.io/) by mReschke and my buddy Claude Opus. + +It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync‑write latency profile** for diagnosing ZFS ZIL performance (NFS / iSCSI / VM sync workloads). It auto‑detects the best IO engine and whether `O_DIRECT` works on the target, and falls back to a basic `dd` test when `fio` isn't installed. + +--- + +## Table of contents + +- [Features](#features) +- [Requirements](#requirements) +- [Usage](#usage) +- [Output](#output) +- [Understanding the tests](#understanding-the-tests) +- [Case study: diagnosing a "slow" Optane SLOG on TrueNAS](#case-study-diagnosing-a-slow-optane-slog-on-truenas) +- [Notes & caveats](#notes--caveats) + +--- + +## Features + +- **CrystalDiskMark‑style profile** — `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T1`, `RND4K Q32T16`, each measured for Read **and** Write, reported in both **MB/s** and **IOPS**. +- **SLOG / sync‑write latency profile** (`--slog`) — synchronous 4K writes at T1/T4/T8/T16 reporting **IOPS, MB/s, and p50/p99 commit latency**. This is the load a ZFS SLOG actually sees. +- **Auto‑detection** — picks the fastest available IO engine (`io_uring` → `libaio` → `posixaio` → `sync`) and probes whether `O_DIRECT` works on the filesystem, falling back to buffered IO when it doesn't (e.g. older OpenZFS, some NFS mounts). +- **`dd` fallback** — if `fio` isn't present, runs a basic write/read test so you still get a number. +- **Verbose mode** — `--verbose` dumps the full raw `fio` output for every run while keeping the summary table intact. + +--- + +## Requirements + +- `bash` +- [`fio`](https://fio.readthedocs.io/) — recommended (`apt install fio` / `pacman -S fio`). Without it, the tool falls back to `dd`. +- `python3` — **only** required for `--slog` (used to parse `fio`'s JSON output for latency percentiles). The normal profile needs only `fio` + `grep`/`awk`. +- `sudo` — `fio` is invoked via `sudo` so it can use `O_DIRECT` and flush device caches. + +--- + +## Usage + +```bash +./speedtest-hd.sh [options] +``` + +`` is the directory (or mount) to benchmark. Use `.` for the current directory. The tool creates a single test file (default 1 GiB) on the target and removes it afterward, so ensure enough free space. + +### Modes + +| Invocation | What it does | +|---|---| +| `./speedtest-hd.sh /mnt/disk` | Auto: uses `fio` if installed, else `dd` | +| `./speedtest-hd.sh /mnt/disk --fio` | Force the `fio` CrystalDiskMark‑style profile | +| `./speedtest-hd.sh /mnt/disk --dd` | Force the basic `dd` test | +| `./speedtest-hd.sh /mnt/disk --slog` | SLOG / sync‑write latency profile | + +### Tuning flags + +| Flag | Effect | +|---|---| +| `--engine=io_uring\|libaio\|posixaio\|sync` | Force a specific IO engine (default: auto) | +| `--direct` | Force `O_DIRECT` (bypass page cache) | +| `--buffered` | Force buffered IO (e.g. when `O_DIRECT` is unsupported) | +| `--runtime=SEC` | Seconds per run (default: 5, like CrystalDiskMark) | +| `--size=SIZE` | Test file size (default: `1g`) | +| `--verbose` | Also print the full `fio` output for every run (summary table unchanged) | + +### Examples + +```bash +# CrystalDiskMark-style test of the current directory +./speedtest-hd.sh . + +# Larger file, longer runs, on an NVMe pool +./speedtest-hd.sh /mnt/nvmepool --runtime=10 --size=4g + +# Buffered (e.g. an NFS share that doesn't support O_DIRECT) +./speedtest-hd.sh /mnt/nfsshare --buffered + +# SLOG / sync latency profile, 30s per run +./speedtest-hd.sh /mnt/nvme-ultra-r10/vm-root --slog --runtime=30 +``` + +> **Tip:** when running `--slog` against a ZFS dataset, watch the SLOG live in another shell: +> ```bash +> zpool iostat -vl 1 +> ``` + +--- + +## Output + +### CrystalDiskMark‑style profile + +``` ++------------------+----------------+----------------+----------------+----------------+ +| Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) | ++------------------+----------------+----------------+----------------+----------------+ +| SEQ1M Q8T1 | 6873.00 | 9.30 | 6873 | 9 | +| SEQ1M Q1T1 | 1608.00 | 20.00 | 1608 | 20 | +| RND4K Q32T1 | 538.00 | 10.80 | 137728 | 2764 | +| RND4K Q32T16 | 689.00 | 261.00 | 176384 | 66816 | ++------------------+----------------+----------------+----------------+----------------+ +``` + +### SLOG / sync‑write latency profile (`--slog`) + +``` ++------------------+--------------+--------------+--------------+--------------+ +| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) | ++------------------+--------------+--------------+--------------+--------------+ +| 4K sync T1 | 10687 | 43.77 | 85.5 | 185.3 | +| 4K sync T4 | 29873 | 122.36 | 117.8 | 317.4 | +| 4K sync T8 | 52612 | 215.50 | 136.2 | 391.2 | +| 4K sync T16 | 77939 | 319.24 | 180.0 | 505.9 | ++------------------+--------------+--------------+--------------+--------------+ +``` + +--- + +## Understanding the tests + +### The CrystalDiskMark profile + +| Test | Pattern | Queue depth | Threads | +|---|---|---|---| +| `SEQ1M Q8T1` | Sequential 1 MiB | 8 | 1 | +| `SEQ1M Q1T1` | Sequential 1 MiB | 1 | 1 | +| `RND4K Q32T1` | Random 4 KiB | 32 | 1 | +| `RND4K Q32T16` | Random 4 KiB | 32 | 16 | + +`Q` = queue depth (`--iodepth`), `T` = threads (`--numjobs`). Note that `--iodepth` only produces real queue depth when the IO is truly asynchronous (async engine **and** `O_DIRECT`). On filesystems where that isn't available, queue depth effectively collapses toward 1 and concurrency comes only from threads (`T`). + +### The SLOG profile + +`--slog` forces **synchronous** 4K random writes (`--sync=1` → `O_SYNC`) via the portable `psync` engine. Every write becomes a durable commit, so it traverses the ZFS ZIL / SLOG commit path exactly the way a `sync=always` dataset (NFS, iSCSI, VM storage) does — regardless of the dataset's own `sync` property. It sweeps thread counts (T1 → T4 → T8 → T16): + +- **T1** is the headline single‑stream latency (e.g. one database committing in a tight loop). +- **The sweep** shows how the SLOG scales as concurrent sync writers pile on (multiple VMs / NFS clients) — usually the more important number for a virtualization host. + +A healthy Optane SLOG (e.g. P1600X) single‑stream target is roughly **15–25k IOPS, p50 ~40–65 µs**. Much higher latency usually points at **CPU C‑states / PCIe ASPM / a BIOS power profile** throttling the host — see the case study below. + +--- + +## Case study: diagnosing a "slow" Optane SLOG on TrueNAS + +A real investigation that this tool's `--slog` mode was built to support. **Spoiler: the Optane SSD was healthy the entire time. The bottleneck was CPU power management.** + +### The setup + +| Component | Detail | +|---|---| +| Server | Dell PowerEdge R630 | +| CPU | Intel Xeon E5‑2680 v3 (Haswell‑EP, 12C/24T, 2.5 GHz base, 3.3 GHz turbo) | +| OS | TrueNAS SCALE 25.10 (OpenZFS 2.3) | +| Pool | `nvme-ultra-r10` — 6× 4 TB KingSpec XG7000 NVMe in RAID10 (3 mirror vdevs) | +| SLOG | Intel Optane **P1600X** | +| Dataset | `/mnt/nvme-ultra-r10/vm-root`, `sync=always` | + +### The symptom + +The standard benchmark looked alarming — huge reads, tiny writes: + +``` ++------------------+------------------+------------------+ +| Test | Read (MB/s) | Write (MB/s) | ++------------------+------------------+------------------+ +| SEQ1M Q8T1 | 6873.00 | 9.30 | +| SEQ1M Q1T1 | 1608.00 | 20.00 | +| RND4K Q32T1 | 538.00 | 10.80 | +| RND4K Q32T16 | 689.00 | 261.00 | ++------------------+------------------+------------------+ +``` + +9.3 MB/s sequential write on an Optane‑backed NVMe pool looks broken. + +### Investigation + +**1. The reads are RAM, not disk.** With a 1 GiB test file on ZFS, reads come straight from ARC (RAM cache). The huge read/write asymmetry is the tell — ignore the read column for judging the disks. + +**2. `sync=always` makes writes latency‑bound.** Every write must be durably committed to the ZIL before it's acknowledged, so throughput ≈ (block size) ÷ (per‑commit latency). Anything running at low effective concurrency looks slow regardless of raw device speed. + +**3. The `Q8`/`Q32` labels were misleading.** On this ZFS setup `--iodepth` didn't produce real queue depth, so most rows effectively ran at QD1. Proof: `RND4K Q32T1` (10.8 MB/s) vs `RND4K Q32T16` (261 MB/s) — the 24× jump came entirely from threads (`numjobs=16`), not queue depth. + +**4. Large sequential sync writes bypass the SLOG.** With ZFS's default `logbias=latency`, writes larger than `zfs_immediate_write_sz` (32 KB) use an *indirect* ZIL record — the data goes straight to the main pool and only a pointer hits the Optane. So the `SEQ1M` write test was measuring the **consumer KingSpec pool's** forced‑sync performance, not the SLOG. Only small (4K) sync writes exercise the Optane. + +**5. Confirm with `zpool iostat -vl 1` during a 4K sync test.** This was decisive: + +- The `logs` (Optane) vdev took **all** the sync writes (~2.3–2.6k ops, ~18–20 MB/s); the data vdevs were idle between txg flushes. → **SLOG configured correctly, `logbias` fine, not bypassed.** +- But the Optane's own `disk_wait` was **~90 µs** (a P1600X should be ~10–15 µs), and the fio‑level commit latency was **~328 µs** — meaning ~238 µs was being spent *above* the device, in the host/ZFS/CPU path. + +That "faster when busy, slow when idle" device latency plus huge host overhead is the classic signature of **power‑saving idle states** on a latency‑bound, QD1 workload. + +**6. Find the throttle.** Checking the CPU revealed the cores pinned at **1.2 GHz** — the E5‑2680 v3's *minimum* P‑state — on a chip rated for 2.5–3.3 GHz: + +``` +$ grep MHz /proc/cpuinfo | sort -u +cpu MHz : 1200.069 +... + +$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver +intel_cpufreq # = intel_pstate in passive mode +$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor +schedutil # picks frequency from CPU utilization +``` + +The vicious loop: a QD1 sync workload spends each commit **blocked** waiting on the SLOG → the `schedutil` governor sees near‑zero utilization → parks the cores at 1.2 GHz → the ZFS commit code path runs ~2–3× slower → latency climbs. + +### Root cause + +**CPU power management, in two layers — not the SSD, pool, or PCIe link:** + +1. **BIOS System Profile = "Performance Per Watt Optimized (DAPC)"** — Dell Active Power Controller manages C‑states/P‑states in firmware and largely ignores the OS, keeping cores idling deep and clocked low. +2. **OS `schedutil` governor** (TrueNAS SCALE default) — pinned cores at the 1.2 GHz floor for this bursty, IO‑blocked workload. + +### The fixes + +Applied in order, biggest impact last: + +**1. BIOS System Profile → Performance** (disables C‑states/C1E, raises P‑states): +``` +# In BIOS (F2): System BIOS → System Profile Settings → System Profile → Performance +# Or via iDRAC: +racadm set BIOS.SysProfileSettings.SysProfile PerfOptimized +racadm jobqueue create BIOS.Setup.1-1 +# reboot to apply +``` + +**2. Kernel parameters** (target PCIe/NVMe link power saving + residual C‑states). On TrueNAS: *System → Advanced Settings → Kernel Arguments*, then reboot: +``` +intel_idle.max_cstate=1 processor.max_cstate=1 pcie_aspm=off nvme_core.default_ps_max_latency_us=0 +``` + +**3. CPU governor → `performance`** *(the single biggest win)*: +```bash +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +``` +Make it persistent on TrueNAS — *System → Advanced Settings → Init/Shutdown Scripts*, add a **Post Init** *Command*: +``` +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +``` +> ⚠️ Without the Post Init script the governor reverts to `schedutil` on every reboot, silently dropping you back to slow numbers. + +### Results — before & after + +**4K synchronous random writes** (`--slog`), per stage of the fix: + +| Stage | T1 IOPS | T1 p50 | T1 MB/s | T16 IOPS | T16 MB/s | T16 p99 | +|---|---|---|---|---|---|---| +| **DAPC** (start) | ~3,050 | ~328 µs | ~12.5 | — | 38 *(regressing)* | thrashing | +| BIOS → Performance | 5,849 | 160.8 µs | 23.96 | 46,009 | 188.45 | 1057 µs | +| + kernel parameters | 6,217 | 150.5 µs | 25.46 | 47,166 | 193.19 | 684 µs | +| **+ `performance` governor** | **10,687** | **85.5 µs** | **43.77** | **77,939** | **319.24** | **506 µs** | + +**Full final result:** + +``` ++------------------+--------------+--------------+--------------+--------------+ +| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) | ++------------------+--------------+--------------+--------------+--------------+ +| 4K sync T1 | 10687 | 43.77 | 85.5 | 185.3 | +| 4K sync T4 | 29873 | 122.36 | 117.8 | 317.4 | +| 4K sync T8 | 52612 | 215.50 | 136.2 | 391.2 | +| 4K sync T16 | 77939 | 319.24 | 180.0 | 505.9 | ++------------------+--------------+--------------+--------------+--------------+ +``` + +**Net improvement:** ~**3.5× IOPS**, ~**3.8× lower latency** at T1, and ~**8.4× aggregate throughput** at T16 — with the scaling regression eliminated entirely. + +### Lessons learned + +- **An Optane SLOG showing high latency is usually a host‑side power‑management problem, not the device.** Confirm where the time goes before blaming hardware. +- **`zpool iostat -vl 1` is the key diagnostic** — it shows whether the `logs` vdev is actually taking the writes and splits device latency (`disk_wait`) from host/ZFS overhead (`total_wait`). +- **Latency‑bound QD1 sync workloads are the worst case for power saving.** The CPU looks idle (blocked on IO), so governors and firmware clock it down — which directly inflates the latency you're trying to measure. +- **On TrueNAS SCALE, the default `schedutil` governor cripples sync‑write latency.** Set `performance` (and persist it). +- **Reads from a small test file measure ARC (RAM), not the disk.** Watch the read/write asymmetry. +- **Large sync writes bypass the SLOG** (indirect ZIL) — to actually test a SLOG, use small (4K) sync writes, which is exactly what `--slog` does. + +### `~85 µs` is roughly the floor here + +The residual gap from raw Optane (~15 µs) is ZFS ZIL‑commit overhead plus the per‑op cost of a 2014‑era Haswell core. Closing it further would need a newer/faster CPU for sharply diminishing returns. For a virtualization host the aggregate (78k IOPS / 319 MB/s) is what the workload feels, and it's healthy. + +--- + +## Notes & caveats + +- **`sudo`** is used for `fio` so it can apply `O_DIRECT` and flush device write caches at the end of write runs (`--end_fsync=1`), so cached writes can't inflate results. +- **`O_DIRECT` is auto‑detected.** If the banner shows `O_DIRECT: DISABLED (buffered ...)`, results may reflect the page cache (RAM) rather than the device. +- **A single shared test file** is reused across runs to keep the footprint to one file. +- **`--slog` requires `python3`** (for JSON latency‑percentile parsing); the standard profile does not. +- The `--slog` profile forces synchronous IO and is intended for ZFS ZIL / SLOG and other sync‑write (NFS/iSCSI/VM) investigations. diff --git a/__pycache__/speedtest-hd.cpython-314.pyc b/__pycache__/speedtest-hd.cpython-314.pyc new file mode 100644 index 0000000..0e52c8e Binary files /dev/null and b/__pycache__/speedtest-hd.cpython-314.pyc differ diff --git a/speedtest-hd.py b/speedtest-hd.py new file mode 100755 index 0000000..b2f6251 --- /dev/null +++ b/speedtest-hd.py @@ -0,0 +1,727 @@ +#!/usr/bin/env python3 +"""speedtest-hd — a CrystalDiskMark-style storage benchmark for Linux, on fio. + +This is the Python successor to ``speedtest-hd.sh``. It measures storage the way +CrystalDiskMark does, and adds a dedicated **SLOG / sync-write latency profile** +for diagnosing ZFS ZIL performance (NFS / iSCSI / VM sync workloads). + +Three profiles +-------------- +* **cdm** — the four CrystalDiskMark default tests (``SEQ1M Q8T1``, ``SEQ1M + Q1T1``, ``RND4K Q32T1``, ``RND4K Q32T16``), each measured for read *and* + write, reported in MB/s and IOPS. +* **slog** — synchronous 4K random writes swept across thread counts, reporting + IOPS, MB/s and p50/p99 *commit latency*. This is the load a ZFS SLOG sees. +* **dd** — a dependency-free fallback when ``fio`` isn't installed. + +Why fio JSON? +------------- +Every measurement asks fio for ``--output-format=json`` and parses it with the +standard library. That's the whole reason this is Python and not bash: robust, +unit-safe parsing of bandwidth / IOPS / latency percentiles with no fragile +text scraping. + +See README.md for the full case study (diagnosing a "slow" Optane SLOG that +turned out to be CPU power management, not the disk). +""" + +from __future__ import annotations + +import argparse +import glob +import json +import os +import shutil +import statistics +import subprocess +import sys +import time +from dataclasses import dataclass, field +from typing import Callable, Optional, Sequence + +# --------------------------------------------------------------------------- # +# Color / styling (standard library only -- raw ANSI escape codes) +# --------------------------------------------------------------------------- # + +# SGR escape sequences. We stick to the basic 16-color set so output respects +# the user's terminal theme instead of hard-coding RGB. +RESET = "\033[0m" +BOLD = "\033[1m" +DIM = "\033[2m" +ITALIC = "\033[3m" +RED = "\033[31m" +GREEN = "\033[32m" +YELLOW = "\033[33m" +BLUE = "\033[34m" +MAGENTA = "\033[35m" +CYAN = "\033[36m" +WHITE = "\033[37m" +BRIGHT_CYAN = "\033[96m" + + +def supports_color(stream) -> bool: + """Decide per-stream whether to emit ANSI codes. + + Honors the de-facto ``NO_COLOR`` / ``FORCE_COLOR`` conventions and otherwise + colors only when the stream is an interactive terminal. + """ + if os.environ.get("NO_COLOR"): + return False + if os.environ.get("FORCE_COLOR"): + return True + if os.environ.get("TERM") == "dumb": + return False + return bool(getattr(stream, "isatty", None)) and stream.isatty() + + +class Painter: + """Wraps text in SGR codes, or returns it untouched when color is off.""" + + def __init__(self, enabled: bool): + self.enabled = enabled + + def paint(self, text: str, *codes: str) -> str: + if not self.enabled or not codes: + return text + return "".join(codes) + text + RESET + + +# stdout carries results (banner + tables); stderr carries progress + verbose +# fio dumps. Each gets its own enable flag so piping one but not the other works. +OUT = Painter(supports_color(sys.stdout)) +ERR = Painter(supports_color(sys.stderr)) + + +# --------------------------------------------------------------------------- # +# Constants +# --------------------------------------------------------------------------- # + +# Preference order for the IO engine. io_uring is the modern, lowest-overhead +# Linux engine; libaio is older (and only truly async with O_DIRECT); posixaio +# and sync are the portable fallbacks (e.g. some NFS mounts). +ENGINE_CANDIDATES = ("io_uring", "libaio", "posixaio", "sync") + +# Binary unit multipliers (fio treats "1g" as 1 GiB, so we match that). +_SIZE_UNITS = {"k": 1024, "m": 1024**2, "g": 1024**3, "t": 1024**4} + + +@dataclass(frozen=True) +class CdmTest: + """One CrystalDiskMark test. ``seq`` picks sequential vs random patterns.""" + + label: str + bs: str + iodepth: int + numjobs: int + seq: bool + + @property + def read_rw(self) -> str: + return "read" if self.seq else "randread" + + @property + def write_rw(self) -> str: + return "write" if self.seq else "randwrite" + + +# The four CrystalDiskMark default tests, as data. Q = queue depth (iodepth), +# T = threads (numjobs). +CDM_TESTS: tuple[CdmTest, ...] = ( + CdmTest("SEQ1M Q8T1", bs="1m", iodepth=8, numjobs=1, seq=True), + CdmTest("SEQ1M Q1T1", bs="1m", iodepth=1, numjobs=1, seq=True), + CdmTest("RND4K Q32T1", bs="4k", iodepth=32, numjobs=1, seq=False), + CdmTest("RND4K Q32T16", bs="4k", iodepth=32, numjobs=16, seq=False), +) + +# Thread counts for the SLOG sweep. T1 is the headline single-stream latency; +# the higher counts show how the SLOG scales as concurrent sync writers pile on. +SLOG_THREADS: tuple[int, ...] = (1, 4, 8, 16) + + +# --------------------------------------------------------------------------- # +# Configuration +# --------------------------------------------------------------------------- # + + +@dataclass +class Config: + """Resolved run settings. ``engine`` and ``direct`` may start unset and get + filled in by :func:`detect_io_settings`.""" + + path: str + mode: Optional[str] # "cdm" | "slog" | "dd" | None (auto) + engine: Optional[str] # None => auto-detect + direct: Optional[bool] # None => auto-detect + runtime: int + size: str + verbose: bool + assume_yes: bool + benchfile: str = field(default="") + + @property + def direct_flag(self) -> int: + """fio's --direct takes 0/1; default to 0 until detection runs.""" + return 1 if self.direct else 0 + + +# --------------------------------------------------------------------------- # +# Small helpers +# --------------------------------------------------------------------------- # + + +def log(message: str) -> None: + """Progress/diagnostic line -> stderr, so stdout stays pure results.""" + print(message, file=sys.stderr, flush=True) + + +def step(message: str) -> None: + """A single colored progress line ('measuring ...') -> stderr.""" + log(f" {ERR.paint('▶', BOLD, CYAN)} {ERR.paint(message, DIM)}") + + +def vsection(title: str) -> None: + """A clear, ruled header for one --verbose fio section -> stderr.""" + width = 78 + head = f"─── {title} " + head += "─" * max(3, width - len(head)) + log("") + log(ERR.paint(head, BOLD, CYAN)) + + +def parse_size_bytes(size: str) -> int: + """Turn an fio-style size ("4k", "1g", "512m", "2048") into bytes.""" + s = size.strip().lower() + if s and s[-1] in _SIZE_UNITS: + return int(float(s[:-1]) * _SIZE_UNITS[s[-1]]) + return int(s) + + +def render_table( + headers: Sequence[str], + rows: Sequence[Sequence[str]], + aligns: Optional[Sequence[str]] = None, + col_styles: Optional[Sequence[Optional[str]]] = None, +) -> str: + """Render a bordered ASCII table whose columns auto-size to their content. + + ``aligns`` is a per-column "<" (left) or ">" (right); by default the first + column is left-aligned (the label) and the rest are right-aligned (numbers). + ``col_styles`` is an optional per-column ANSI style applied to body cells + (header is always bold, borders dim). Widths are computed on the *plain* + text and color is applied after padding, so escape codes never skew layout. + """ + ncols = len(headers) + if aligns is None: + aligns = ["<"] + [">"] * (ncols - 1) + if col_styles is None: + col_styles = [None] * ncols + + # Widest cell (header or any row) sets each column's width. + widths = [ + max([len(str(headers[c]))] + [len(str(r[c])) for r in rows]) + for c in range(ncols) + ] + + bar = OUT.paint("|", DIM) + sep = OUT.paint(" | ", DIM) + + def rule() -> str: + return OUT.paint("+" + "+".join("-" * (w + 2) for w in widths) + "+", DIM) + + def line(cells: Sequence[str], *, header: bool = False) -> str: + parts = [] + for c in range(ncols): + padded = f"{str(cells[c]):{aligns[c]}{widths[c]}}" + if header: + parts.append(OUT.paint(padded, BOLD)) + elif col_styles[c]: + parts.append(OUT.paint(padded, col_styles[c])) + else: + parts.append(padded) + return f"{bar} " + sep.join(parts) + f" {bar}" + + out = [rule(), line(headers, header=True), rule()] + out += [line(r) for r in rows] + out.append(rule()) + return "\n".join(out) + + +# --------------------------------------------------------------------------- # +# fio: results, invocation, parsing +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class FioResult: + """Parsed metrics for one direction (read or write) of an fio run. + + Latencies are completion latency (clat) in microseconds; for synchronous + writes that is effectively the durable-commit latency. + """ + + bw_mbps: float # decimal MB/s, matching CrystalDiskMark's SI figure + iops: float + p50_us: float + p99_us: float + mean_us: float + + @classmethod + def zero(cls) -> "FioResult": + return cls(0.0, 0.0, 0.0, 0.0, 0.0) + + +def _direction_of(rw: str) -> str: + """fio reports 'read' and 'write' sub-objects; pick the active one.""" + return "read" if "read" in rw else "write" + + +def _aggregate(jobs: list[dict], direction: str) -> FioResult: + """Combine per-job fio JSON stats into a single :class:`FioResult`. + + We deliberately run fio *without* --group_reporting and aggregate ourselves, + which sidesteps fio's group-merge quirks: sum throughput, average the median + latency, and take the worst-case tail (p99) across jobs. + """ + + def section(job: dict) -> dict: + return job.get(direction, {}) + + bw_mbps = sum(section(j).get("bw_bytes", 0) for j in jobs) / 1e6 + iops = sum(section(j).get("iops", 0.0) for j in jobs) + + def percentiles(key: str) -> list[float]: + vals = [] + for j in jobs: + pct = section(j).get("clat_ns", {}).get("percentile", {}) + v = pct.get(key) + if v is not None: + vals.append(v) + return vals + + p50_ns = percentiles("50.000000") + p99_ns = percentiles("99.000000") + mean_ns = [section(j).get("clat_ns", {}).get("mean", 0.0) for j in jobs] + + return FioResult( + bw_mbps=bw_mbps, + iops=iops, + p50_us=(statistics.mean(p50_ns) / 1000.0) if p50_ns else 0.0, # avg of medians + p99_us=(max(p99_ns) / 1000.0) if p99_ns else 0.0, # worst tail + mean_us=(statistics.mean(mean_ns) / 1000.0) if mean_ns else 0.0, + ) + + +def run_fio( + cfg: Config, + *, + rw: str, + bs: str, + iodepth: int, + numjobs: int, + engine: Optional[str] = None, + direct: Optional[bool] = None, + sync: bool = False, + end_fsync: bool = False, +) -> FioResult: + """Run a single fio job against the shared bench file and parse its JSON. + + ``engine``/``direct`` default to the detected config values; the SLOG path + overrides them (psync + buffered + O_SYNC). ``end_fsync`` flushes the device + cache at the end of a write run so cached writes can't inflate the result. + """ + engine = engine if engine is not None else cfg.engine + direct = direct if direct is not None else cfg.direct + + cmd = [ + "sudo", "fio", + "--name=speedtest", + f"--filename={cfg.benchfile}", + f"--ioengine={engine}", + f"--direct={1 if direct else 0}", + f"--rw={rw}", + f"--bs={bs}", + f"--size={cfg.size}", + f"--numjobs={numjobs}", + f"--iodepth={iodepth}", + "--time_based", + f"--runtime={cfg.runtime}", + "--output-format=json", + ] + if sync: + cmd.append("--sync=1") # O_SYNC: every write is a durable commit + if end_fsync: + cmd.append("--end_fsync=1") # flush device cache once at the end + + proc = subprocess.run(cmd, capture_output=True, text=True) + + if cfg.verbose: + sync_tag = " sync=1" if sync else "" + vsection(f"fio · {rw} bs={bs} iodepth={iodepth} numjobs={numjobs}{sync_tag}") + log(" " + ERR.paint("$ " + " ".join(cmd[1:]), DIM, ITALIC)) + log(ERR.paint(proc.stdout.strip() or "(no stdout)", DIM)) + if proc.stderr.strip(): + log(ERR.paint(proc.stderr.strip(), YELLOW)) + + try: + data = json.loads(proc.stdout) + return _aggregate(data["jobs"], _direction_of(rw)) + except (json.JSONDecodeError, KeyError, ValueError): + # fio failed or produced no parseable JSON; report zeros rather than + # crashing the whole run (verbose mode above shows what went wrong). + return FioResult.zero() + + +def fio_probe(path: str, engine: str, direct: bool) -> bool: + """Throwaway 1s fio job to see if an (engine, O_DIRECT) combo works here. + + This is how we stay accurate *and* portable across ext4/xfs/btrfs/ZFS/NFS + without the caller needing to know what each filesystem supports. + """ + cmd = [ + "sudo", "fio", "--name=probe", + f"--directory={path}", + f"--ioengine={engine}", + "--rw=write", "--bs=4k", "--size=1m", + f"--direct={1 if direct else 0}", + "--time_based", "--runtime=1", + ] + proc = subprocess.run(cmd, capture_output=True, text=True) + for leftover in glob.glob(os.path.join(path, "probe*")): + try: + os.remove(leftover) + except OSError: + pass + return proc.returncode == 0 + + +def detect_io_settings(cfg: Config) -> None: + """Fill in ``cfg.engine`` and ``cfg.direct`` if the user didn't force them. + + O_DIRECT bypasses the OS page cache so we measure the device, not RAM. Not + every filesystem supports it (older OpenZFS, some NFS), so we probe and fall + back to buffered rather than erroring out. + """ + if cfg.engine is None: + for engine in ENGINE_CANDIDATES: + if fio_probe(cfg.path, engine, direct=False): + cfg.engine = engine + break + else: + cfg.engine = "sync" + + if cfg.direct is None: + cfg.direct = fio_probe(cfg.path, cfg.engine, direct=True) + + +# --------------------------------------------------------------------------- # +# Profiles +# --------------------------------------------------------------------------- # + + +def _meta_line(text: str) -> str: + """Color the 'Label :' prefix of a banner metadata line, leave value as-is. + + The value may already contain its own ANSI codes (e.g. a red O_DIRECT + warning); since these lines aren't width-aligned that's harmless. + """ + key, sep, val = text.partition(":") + if not sep: + return " " + text + return " " + OUT.paint(key + ":", BOLD, BRIGHT_CYAN) + val + + +def banner(title: str, cfg: Config, extra: Sequence[str] = ()) -> None: + full = f"speedtest-hd · {title}" + inner = len(full) + 4 + + print() + print(OUT.paint("╭" + "─" * inner + "╮", BOLD, CYAN)) + print( + OUT.paint("│", BOLD, CYAN) + + " " + OUT.paint(full, BOLD, WHITE) + " " + + OUT.paint("│", BOLD, CYAN) + ) + print(OUT.paint("╰" + "─" * inner + "╯", BOLD, CYAN)) + print(_meta_line(f"Target : {OUT.paint(cfg.path, CYAN)}")) + for line_ in extra: + print(_meta_line(line_)) + print() + + +def cdm_profile(cfg: Config) -> None: + """The CrystalDiskMark-style profile: 4 tests x (read, write), MB/s + IOPS.""" + detect_io_settings(cfg) + + if cfg.direct: + direct_label = OUT.paint("enabled (device)", GREEN) + else: + direct_label = OUT.paint( + "DISABLED (buffered -- may reflect RAM cache!)", BOLD, RED + ) + + banner( + "CrystalDiskMark-style storage benchmark", + cfg, + extra=[ + f"Engine : {cfg.engine} O_DIRECT: {direct_label}", + f"Profile : size={cfg.size.upper()} runtime={cfg.runtime}s/run (8 runs)", + ], + ) + + rows: list[list[str]] = [] + for test in CDM_TESTS: + step(f"measuring {test.label.strip()} ...") + r = run_fio(cfg, rw=test.read_rw, bs=test.bs, + iodepth=test.iodepth, numjobs=test.numjobs) + w = run_fio(cfg, rw=test.write_rw, bs=test.bs, + iodepth=test.iodepth, numjobs=test.numjobs, end_fsync=True) + rows.append([ + test.label, + f"{r.bw_mbps:.2f}", f"{w.bw_mbps:.2f}", + f"{r.iops:.0f}", f"{w.iops:.0f}", + ]) + + _cleanup(cfg) + + print() + print(render_table( + ["Test", "Read (MB/s)", "Write (MB/s)", "Read (IOPS)", "Write (IOPS)"], + rows, + col_styles=[CYAN, GREEN, YELLOW, GREEN, YELLOW], + )) + print() + + +def slog_profile(cfg: Config) -> None: + """SLOG / sync-write latency profile. + + Forces synchronous 4K random writes (O_SYNC) via the portable psync engine, + so every write is a ZIL commit -- exercising a ZFS SLOG exactly the way a + sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own + sync property. Engine/direct detection is only for the banner here. + """ + detect_io_settings(cfg) + + banner( + "SLOG / sync-write latency profile", + cfg, + extra=[ + "Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine", + f"Profile : runtime={cfg.runtime}s/run size={cfg.size.upper()}", + "Note : every write is a synchronous ZIL commit -- the load your", + " SLOG actually sees. Watch it live in another shell with:", + " zpool iostat -vl 1", + ], + ) + + # Measure everything first (progress -> stderr), then draw the whole table, + # so the "measuring..." lines can't interleave into the middle of it. + rows: list[list[str]] = [] + for threads in SLOG_THREADS: + step(f"measuring 4K sync randwrite T{threads} ...") + res = run_fio( + cfg, rw="randwrite", bs="4k", iodepth=1, numjobs=threads, + engine="psync", direct=False, sync=True, + ) + rows.append([ + f"4K sync T{threads}", + f"{res.iops:.0f}", f"{res.bw_mbps:.2f}", + f"{res.p50_us:.1f}", f"{res.p99_us:.1f}", + ]) + + _cleanup(cfg) + + print() + print(render_table( + ["Test", "IOPS", "MB/s", "p50 lat(us)", "p99 lat(us)"], + rows, + col_styles=[CYAN, GREEN, GREEN, YELLOW, MAGENTA], + )) + print() + print(OUT.paint(" Healthy Optane SLOG (eg P1600X) single-stream (T1) target:", BOLD)) + print(OUT.paint(" ~15-25k IOPS, p50 latency ~40-65us.", GREEN) + + OUT.paint(" Much higher latency usually means", DIM)) + print(OUT.paint(" CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling.", DIM)) + print() + + +def dd_profile(cfg: Config) -> None: + """Dependency-free fallback when fio isn't installed. + + Far cruder than fio: a single sequential stream, and the cached-read figure + will reflect RAM. Use it only as a rough sanity check. + """ + count_mib = max(1, parse_size_bytes(cfg.size) // (1024**2)) + nbytes = count_mib * 1024**2 + + banner( + "basic dd benchmark (fio not installed)", + cfg, + extra=[f"Profile : {count_mib} MiB sequential stream"], + ) + + def timed_dd(args: list[str]) -> float: + start = time.monotonic() + subprocess.run(args, capture_output=True, text=True) + elapsed = time.monotonic() - start + return (nbytes / elapsed / 1e6) if elapsed > 0 else 0.0 # decimal MB/s + + step("measuring sequential write ...") + write_mbps = timed_dd([ + "dd", "if=/dev/zero", f"of={cfg.benchfile}", + "bs=1M", f"count={count_mib}", "conv=fdatasync,notrunc", + ]) + + step("dropping caches for uncached read ...") + subprocess.run(["sudo", "sh", "-c", "echo 3 > /proc/sys/vm/drop_caches"], + capture_output=True, text=True) + + step("measuring uncached read ...") + uncached_mbps = timed_dd([ + "dd", f"if={cfg.benchfile}", "of=/dev/null", "bs=1M", f"count={count_mib}", + ]) + + step("measuring cached read ...") + cached_mbps = timed_dd([ + "dd", f"if={cfg.benchfile}", "of=/dev/null", "bs=1M", f"count={count_mib}", + ]) + + _cleanup(cfg) + + print() + print(render_table( + ["Test", "MB/s"], + [ + ["Sequential write", f"{write_mbps:.2f}"], + ["Uncached read", f"{uncached_mbps:.2f}"], + ["Cached read (RAM)", f"{cached_mbps:.2f}"], + ], + col_styles=[CYAN, GREEN], + )) + print() + + +def _cleanup(cfg: Config) -> None: + """Remove the shared bench file.""" + try: + os.remove(cfg.benchfile) + except OSError: + pass + + +# --------------------------------------------------------------------------- # +# CLI +# --------------------------------------------------------------------------- # + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser( + prog="speedtest-hd.py", + description="CrystalDiskMark-style storage benchmark built on fio.", + epilog=( + "Examples:\n" + " speedtest-hd.py .\n" + " speedtest-hd.py /mnt/nvmepool --runtime=10 --size=4g\n" + " speedtest-hd.py /mnt/nfsshare --buffered\n" + " speedtest-hd.py /mnt/nvme-ultra-r10/vm-root --slog --runtime=30\n" + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p.add_argument("path", help="directory/mount to benchmark ('.' for cwd)") + + mode = p.add_mutually_exclusive_group() + mode.add_argument("--fio", action="store_const", const="cdm", dest="mode", + help="force the fio CrystalDiskMark-style profile") + mode.add_argument("--dd", action="store_const", const="dd", dest="mode", + help="force the basic dd fallback test") + mode.add_argument("--slog", action="store_const", const="slog", dest="mode", + help="SLOG / sync-write latency profile (ZFS ZIL)") + + direct = p.add_mutually_exclusive_group() + direct.add_argument("--direct", action="store_const", const=True, dest="direct", + help="force O_DIRECT (bypass page cache)") + direct.add_argument("--buffered", action="store_const", const=False, dest="direct", + help="force buffered IO (e.g. if O_DIRECT unsupported)") + + p.add_argument("--engine", choices=ENGINE_CANDIDATES, + help="force a specific IO engine (default: auto-detect)") + p.add_argument("--runtime", type=int, default=5, metavar="SEC", + help="seconds per run (default: 5, like CrystalDiskMark)") + p.add_argument("--size", default="1g", metavar="SIZE", + help="test file size (default: 1g)") + p.add_argument("--verbose", action="store_true", + help="also print the full fio output for every run") + p.add_argument("-y", "--yes", action="store_true", dest="assume_yes", + help="skip the confirmation prompt") + + p.set_defaults(mode=None, direct=None) + return p + + +def confirm(cfg: Config) -> None: + """Guard prompt -- we're about to write a multi-GB file to the target.""" + if cfg.assume_yes: + return + print(OUT.paint("NOTICE:", BOLD, YELLOW) + + f" {cfg.size.upper()} free space on " + + OUT.paint(f"'{cfg.path}'", CYAN) + + " is required to perform the benchmark.") + answer = input(f"Are you ready to start a storage benchmark against " + f"'{cfg.path}' ? ") + if not answer.strip().lower().startswith("y"): + print(OUT.paint("Ok, cancelled!", YELLOW)) + sys.exit(0) + print(OUT.paint("Great! Starting benchmark now!", BOLD, GREEN)) + + +def main(argv: Optional[Sequence[str]] = None) -> int: + args = build_parser().parse_args(argv) + + path = os.getcwd() if args.path == "." else args.path + if not os.path.exists(path): + print(f"Path {path} does not exist", file=sys.stderr) + return 1 + + cfg = Config( + path=path, + mode=args.mode, + engine=args.engine, + direct=args.direct, + runtime=args.runtime, + size=args.size, + verbose=args.verbose, + assume_yes=args.assume_yes, + benchfile=os.path.join(path, "speedtest-hd.bench"), + ) + + have_fio = shutil.which("fio") is not None + + # Resolve the mode: explicit flag wins; otherwise fio if available, else dd. + mode = cfg.mode + if mode in ("cdm", "slog") and not have_fio: + print(ERR.paint("ERROR:", BOLD, RED) + + " --fio/--slog require fio (apt install fio / pacman -S fio).", + file=sys.stderr) + return 1 + if mode is None: + if have_fio: + mode = "cdm" + else: + print(OUT.paint("\nfio is not installed -- falling back to basic dd test.", + YELLOW)) + print(OUT.paint("Install fio for the full CrystalDiskMark-style benchmark.", + DIM)) + mode = "dd" + + confirm(cfg) + + {"cdm": cdm_profile, "slog": slog_profile, "dd": dd_profile}[mode](cfg) + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except KeyboardInterrupt: + print(ERR.paint("\nInterrupted.", YELLOW), file=sys.stderr) + sys.exit(130) diff --git a/speedtest-hd.sh b/speedtest-hd.sh index fa85f83..205fa0c 100755 --- a/speedtest-hd.sh +++ b/speedtest-hd.sh @@ -1,8 +1,11 @@ #!/usr/bin/env bash -# Robust HD/SDD/NVMe performance CLI utility -# Utilizing FIO for sequential/random writes/writes -# Dependencies: fio (apt install fio) +# Robust HD/SSD/NVMe performance CLI utility +# A CrystalDiskMark-style storage benchmark for Linux, built on fio. +# Runs the exact same four tests CrystalDiskMark does (SEQ1M Q8T1, SEQ1M Q1T1, +# RND4K Q32T1, RND4K Q32T16), each measured for both Read and Write, and prints +# the results in a CrystalDiskMark-style table. +# Dependencies: fio (apt install fio / pacman -S fio). Falls back to dd if missing. # See: https://cloud.google.com/compute/docs/disks/benchmarking-pd-performance # See: https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/ # mReschke 2024-01-18 @@ -10,20 +13,23 @@ # CLI Parameters path="$1" option="" -simple=false engine="" # IO engine; empty = auto-detect (io_uring > libaio > posixaio > sync) direct="" # O_DIRECT; empty = auto-detect; 1 = bypass page cache, 0 = buffered -runtime=30 # seconds per test (longer = more accurate, exposes SLC cache exhaustion) +runtime=5 # seconds per measurement (CrystalDiskMark default is 5s) +size="1g" # test file size (CrystalDiskMark default is 1GiB) +verbose=0 # 1 = also print the full fio output for every run for arg in "${@:2}"; do case "$arg" in - --simple) simple=true ;; --dd) option="--dd" ;; --fio) option="--fio" ;; + --slog) option="--slog" ;; --buffered) direct=0 ;; --direct) direct=1 ;; + --verbose) verbose=1 ;; --engine=*) engine="${arg#*=}" ;; --direct=*) direct="${arg#*=}" ;; --runtime=*) runtime="${arg#*=}" ;; + --size=*) size="${arg#*=}" ;; esac done @@ -47,275 +53,35 @@ function main { fi # Must type y or n THEN press enter (which I like better) - echo "NOTICE: 1GB free space on '$path' is required to perform the benchmark." - echo -n "Are you ready to start a robust IO benchmark against '$path' ?"; read answer + echo "NOTICE: ${size^^} free space on '$path' is required to perform the benchmark." + echo -n "Are you ready to start a storage benchmark against '$path' ? "; read answer if [ "$answer" != "${answer#[Yy]}" ]; then echo "Great! Starting benchmark now!" - echo "------------------------------" else echo "Ok, cancelled!" exit 0 fi - # Use dd of fio based on param or defaults + # Use dd or fio based on param or defaults if [ "$option" == "--dd" ]; then dd_speedtest elif [ "$option" == "--fio" ]; then - fio_speedtest + cdm_speedtest + elif [ "$option" == "--slog" ]; then + slog_speedtest elif [ "$option" == "" ]; then # If fio is installed, use it, else use dd - [ "$simple" != true ] && echo "" if ! command -v fio &> /dev/null; then + echo "" + echo "fio is not installed -- falling back to basic dd test." + echo "Install fio for the full CrystalDiskMark-style benchmark." dd_speedtest else - fio_speedtest + cdm_speedtest fi fi } -function print_result { - # Print a single fio test result, either as the full "Run status group" - # line or, when --simple is given, as a compact aligned "Label value" row. - # The simple value is taken from the parenthetical (MB/s) figure that - # follows the first bw= value in fio's output. - local label="$1" - local output="$2" - local status - status=$(echo " - $output " | /usr/bin/grep -A1 'Run status group' | tail -n1) - if [ "$simple" == true ]; then - local value - # Grab the parenthetical (MB/s) figure after the first bw=, then put a - # space between the number and the unit, eg "98.4MB/s" -> "98.4 MB/s". - value=$(echo "$status" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | sed -E 's/([0-9.]+)([A-Za-z])/\1 \2/') - printf "%-28s%s\n" "$label" "$value" - else - echo "$status" - fi -} - -function fio_write_single_random_4k { - # 4K Random Writes (parallel, deep queue) - - # Random 4K writes are the worst thing you can ask a disk to do. Where this - # happens most in real life: copying home directories and dotfiles, email - # stores, some database operations, source code trees. - - # IMPORTANT: the original version of this test used numjobs=1 + iodepth=1. - # That measures single-op LATENCY (the absolute worst case): the OS issues - # one 4K write, waits for it to be acknowledged, then issues the next. On - # NVMe that lands around ~3000 IOPS / ~12MB/sec -- which looks alarmingly - # slow but is NOT a bug. NVMe's entire advantage is deep command queues and - # parallelism, so a queue depth of 1 deliberately measures the one scenario - # NVMe is bad at. To measure the drive's actual capability we now drive it - # with multiple jobs and a deep queue (4 jobs x iodepth=64), keeping - # hundreds of operations in flight at once. Expect this to jump into the - # hundreds of MB/sec (often 1GB/sec+) on a healthy NVMe pool. - - # --name= is a required argument, but it's basically human-friendly fluff—fio will create files based on that name to test with, inside the directory you point it at. - # --directory= where the test files are created; this is the path/mount you are benchmarking. - # --ioengine=$engine the engine fio uses to talk to the kernel, auto-detected by detect_io_settings(). On modern Linux this is io_uring (lowest overhead); it falls back to libaio, then posixaio, then sync. NOTE: libaio is only truly asynchronous with --direct=1; with buffered IO it silently degrades to synchronous, which is exactly why io_uring is preferred. - # --rw=randwrite random write operations. Other options include read, write (sequential), randread, and randrw. - # --bs=4k blocksize 4K. These are very small individual operations. This is where the pain lives; it's hard on the disk and adds a ton of command-channel overhead, since a separate operation has to be commanded for each 4K of data. - # --size=256m each job's test file is 256MB. With --numjobs=4 that's 4 files totaling ~1GB. - # --numjobs=4 run 4 parallel processes, each with its own test file, to feed the drive from multiple threads at once (single-threaded cannot saturate NVMe). - # --iodepth=64 how many commands each job keeps stacked in the queue at once. Deep queues are how NVMe reaches its real IOPS; 4 jobs x 64 = up to 256 ops in flight. - # --direct=$direct O_DIRECT, auto-detected. 1 bypasses the OS page cache so we measure the device instead of RAM; 0 (buffered) is the fallback for filesystems/mounts that reject O_DIRECT (eg older OpenZFS, some NFS). - # --ramp_time=2s discard the first 2 seconds of results so we measure steady state, not the initial warm-up burst. - # --runtime=$runtime --time_based run for this many seconds, looping over the file(s) if we finish early. - # --group_reporting=1 aggregate all jobs into a single combined result line instead of one line per job. - # --end_fsync=1 after the timed run, flush everything to stable storage and count that time, so cached writes can't inflate the number. - [ "$simple" != true ] && { echo ""; echo "4K Random Writes (bs=4k, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; } - x=`sudo fio \ - --name=fio-write-random-4k \ - --directory=$path \ - --ioengine=$engine \ - --rw=randwrite \ - --bs=4k \ - --size=256m \ - --numjobs=4 \ - --iodepth=64 \ - --direct=$direct \ - --ramp_time=2s \ - --time_based --runtime=$runtime \ - --group_reporting=1 \ - --end_fsync=1` - print_result "4K Random Writes" "$x" - - # Cleanup my test files - rm -rf $path/fio-write-random-4k* -} - -function fio_write_parallel_random_64k { - # Parallel 64k Random Writes - - # This time, we're creating 16 separate 64MB files (still totaling 1GB, when - # all put together) and we're issuing 64KB blocksized random write operations. - # We're doing it with sixteen separate processes running in parallel, and - # we're queuing up to 16 simultaneous asynchronous ops before we pause and wait - # for the OS to start acknowledging their receipt. - - # This is a pretty decent approximation of a significantly busy system. It's - # not doing any one particularly nasty thing—like running a database engine or - # copying tons of dotfiles from a user's home directory—but it is coping with - # a bunch of applications doing moderately demanding stuff all at once. - - # This is also a pretty good, slightly pessimistic approximation of a busy, - # multi-user system like a NAS, which needs to handle multiple 1MB operations - # simultaneously for different users. If several people or processes are trying - # to read or write big files (photos, movies, whatever) at once, the OS tries - # to feed them all data simultaneously. This pretty quickly devolves down to a - # pattern of multiple random small block access. So in addition to "busy desktop - # with lots of apps," think "busy fileserver with several people actively using it." - - # You will see a lot more variation in speed as you watch this operation play - # out on the console. Unlike the steady trickle you'd get from a queue-depth-1 - # single-op latency test, this 16-process job can fluctuate wildly—eg between - # about 10MiB/sec and 300MiB/sec during the run—as the OS and SSD firmware - # catch good and bad luck aggregating writes. - - # Most of the variation you're seeing here is due to the operating system and - # SSD firmware sometimes being able to aggregate multiple writes. When it - # manages to aggregate them helpfully, it can write them in a way that allows - # parallel writes to all the individual physical media stripes inside the SSD. - # Sometimes, it still ends up having to give up and write to only a single - # physical media stripe at a time—or a garbage collection or other maintenance - # operation at the SSD firmware level needs to run briefly in the background, - # slowing things down. - [ "$simple" != true ] && { echo ""; echo "Parallel 64K Random Writes (bs=64k, jobs=16, iodepth=16, engine=$engine, direct=$direct, ${runtime}s)"; } - x=`sudo fio \ - --name=fio-write-random-64k \ - --directory=$path \ - --ioengine=$engine \ - --rw=randwrite \ - --bs=64k \ - --size=64m \ - --numjobs=16 \ - --iodepth=16 \ - --direct=$direct \ - --ramp_time=2s \ - --time_based --runtime=$runtime \ - --group_reporting=1 \ - --end_fsync=1` - print_result "Parallel 64K Random Writes" "$x" - - # Cleanup my test files - rm -rf $path/fio-write-random-64k* -} - -function fio_write_single_sequential_1m { - # Single 1M Sequential Writes - - # This is pretty close to the best-case scenario for a real-world system - # doing real-world things. No, it's not quite as fast as a single, truly - # contiguous write... but the 1MiB blocksize is large enough that it's quite - # close. Besides, if literally any other disk activity is requested simultaneously - # with a contiguous write, the "contiguous" write devolves to this level of - # performance pretty much instantly, so this is a much more realistic test of - # the upper end of storage performance on a typical system. - - # You'll see some kooky fluctuations on SSDs when doing this test. This is largely - # due to the SSD's firmware having better luck or worse luck at any given time, - # when it's trying to queue operations so that it can write across all physical - # media stripes cleanly at once. Rust disks will tend to provide a much more - # consistent, though typically lower, throughput across the run. - - # You can also see SSD performance fall off a cliff here if you exhaust an - # onboard write cache—TLC and QLC drives tend to have small write cache areas - # made of much faster MLC or SLC media. Once those get exhausted, the disk has - # to drop to writing directly to the much slower TLC/QLC media where the data - # eventually lands. This is the major difference between, for example, Samsung - # EVO and Pro SSDs—the EVOs have slow TLC media with a fast MLC cache, where - # the Pros use the higher-performance, higher-longevity MLC media throughout - # the entire SSD. - - # If you have any doubt at all about a TLC or QLC disk's ability to sustain - # heavy writes, you may want to experimentally extend your time duration here. - # If you watch the throughput live as the job progresses, you'll see the impact - # immediately when you run out of cache—what had been a fairly steady, - # several-hundred-MiB/sec throughput will suddenly plummet to half the speed - # or less and get considerably less stable as well. - - # However, you might choose to take the opposite position—you might not - # expect to do sustained heavy writes very frequently, in which case you - # actually are more interested in the on-cache behavior. What's important - # here is that you understand both what you want to test, and how to test - # it accurately. - - [ "$simple" != true ] && { echo ""; echo "Single 1M Sequential Writes (bs=1m, jobs=1, iodepth=16, engine=$engine, direct=$direct, ${runtime}s)"; } - x=`sudo fio \ - --name=fio-write-seq-1m \ - --directory=$path \ - --ioengine=$engine \ - --rw=write \ - --bs=1m \ - --size=1g \ - --numjobs=1 \ - --iodepth=16 \ - --direct=$direct \ - --ramp_time=2s \ - --time_based --runtime=$runtime \ - --group_reporting=1 \ - --end_fsync=1` - print_result "Single 1M Sequential Writes" "$x" - - # Cleanup my test files - rm -rf $path/fio-write-seq-1m* -} - -function fio_read_sequential_1m { - # Sequential Parallel Reads - - [ "$simple" != true ] && { echo ""; echo "Sequential 4x 1M Reads (bs=1m, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; } - x=`sudo fio \ - --name=fio-read-sequential-1m \ - --directory=$path \ - --ioengine=$engine \ - --bs=1M \ - --numjobs=4 \ - --size=256M \ - --time_based --runtime=$runtime \ - --ramp_time=2s \ - --direct=$direct \ - --verify=0 \ - --iodepth=64 \ - --rw=read \ - --group_reporting=1 \ - --iodepth_batch_submit=64 \ - --iodepth_batch_complete_max=64` - print_result "Sequential 4x 1M Reads" "$x" - rm -rf $path/fio-read-sequential-1m* -} - -function fio_read_random_4k { - # Random 4k Reads (parallel, deep queue) - - # 4K random reads are the highest-IOPS thing a drive does, but a single - # submitting thread goes CPU-bound long before an NVMe runs out of capacity, - # so it under-reports peak IOPS. We spread the work over 4 jobs x iodepth=64 - # (256 ops in flight, like the 4K write test) to saturate fast drives, while - # still reporting the honest (low) number on a spinning HDD. size=256m per - # job keeps the total footprint at ~1GB across the 4 jobs. - [ "$simple" != true ] && { echo ""; echo "Random 4k Reads (bs=4k, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; } - x=`sudo fio \ - --name=fio-read-random-4k \ - --directory=$path \ - --ioengine=$engine \ - --rw=randread \ - --bs=4k \ - --size=256m \ - --numjobs=4 \ - --time_based --runtime=$runtime \ - --ramp_time=2s \ - --direct=$direct \ - --verify=0 \ - --iodepth=64 \ - --group_reporting=1 \ - --iodepth_batch_submit=64 \ - --iodepth_batch_complete_max=64` - print_result "Random 4k Reads" "$x" - rm -rf $path/fio-read-random-4k* -} - function fio_probe { # Run a tiny throwaway fio job to see if a given (engine, direct) combo # actually works on this path/filesystem. Returns 0 on success. This is how @@ -353,57 +119,291 @@ function detect_io_settings { if [ -z "$direct" ]; then if fio_probe "$engine" 1; then direct=1; else direct=0; fi fi - - if [ "$simple" != true ]; then - echo "" - echo "IO engine : $engine" - if [ "$direct" == 1 ]; then - echo "O_DIRECT : enabled (bypassing page cache -- measuring the device)" - else - echo "O_DIRECT : DISABLED (buffered) -- this filesystem/mount rejected" - echo " O_DIRECT, so results (especially reads) may reflect the" - echo " RAM cache rather than the underlying device. Pass" - echo " --direct to force it if you believe it should work." - fi - fi } -function fio_speedtest { +function fio_bw_mbps { + # Extract bandwidth in decimal MB/s (matching CrystalDiskMark) from a fio + # run. fio prints eg " READ: bw=3357MiB/s (3521MB/s), ..." -- we take the + # parenthetical SI figure and normalise kB/MB/GB into MB/s. + local out="$1" + local raw num unit + raw=$(echo "$out" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | head -n1) + [ -z "$raw" ] && { echo "0.00"; return; } + num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1) + unit=$(echo "$raw" | /usr/bin/grep -oP '[A-Za-z]+/s' | head -n1) + case "$unit" in + B/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000000}' ;; + kB/s|KB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000}' ;; + MB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;; + GB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n*1000}' ;; + *) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;; + esac +} + +function fio_iops { + # Extract IOPS from a fio run. fio prints eg " read: IOPS=12.3k, BW=..." -- + # the value may carry a k/M suffix, which we expand to a whole number. + local out="$1" + local raw num unit + raw=$(echo "$out" | /usr/bin/grep -oP 'IOPS=\K[0-9.]+[kKmM]?' | head -n1) + [ -z "$raw" ] && { echo "0"; return; } + num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1) + unit=$(echo "$raw" | /usr/bin/grep -oP '[kKmM]' | head -n1) + case "$unit" in + k|K) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000}' ;; + m|M) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000000}' ;; + *) awk -v n="$num" 'BEGIN{printf "%.0f", n}' ;; + esac +} + +function run_fio { + # run_fio -> echoes "MB/s IOPS" (space-sep) + # A single shared 1GB test file is reused across every run (laid out once), + # which keeps the footprint at one file and avoids re-creating it each time. + local rw="$1" bs="$2" qd="$3" jobs="$4" + local args=( + --name=speedtest + --filename="$benchfile" + --ioengine="$engine" + --direct="$direct" + --rw="$rw" + --bs="$bs" + --size="$size" + --numjobs="$jobs" + --iodepth="$qd" + --time_based --runtime="$runtime" + --group_reporting=1 + ) + # Flush device cache at the end of write tests so cached writes can't lie. + case "$rw" in + *write) args+=(--end_fsync=1) ;; + esac + # Capture the full fio output. In verbose mode we also keep fio's stderr + # (2>&1); otherwise we discard it. Either way only the parsed bandwidth is + # echoed to stdout, so the captured value stays clean. + local out + if [ "$verbose" = 1 ]; then + out=$(sudo fio "${args[@]}" 2>&1) + else + out=$(sudo fio "${args[@]}" 2>/dev/null) + fi + # Dump the raw fio output (to stderr, so it never pollutes the captured + # values on stdout) when running verbose. + if [ "$verbose" = 1 ]; then + echo " --- fio: $rw bs=$bs iodepth=$qd numjobs=$jobs ---" >&2 + echo "$out" >&2 + echo "" >&2 + fi + echo "$(fio_bw_mbps "$out") $(fio_iops "$out")" +} + +# Table drawing ------------------------------------------------------------- +# Label cell is "| %-16s |" (18 dashes); the four numeric cells are "| %14s |" +# (16 dashes each). +TBL_SEG_LBL="------------------" # 18 dashes -> matches "| %-16s |" +TBL_SEG_NUM="----------------" # 16 dashes -> matches "| %14s |" +function tbl_rule { + printf "+%s+%s+%s+%s+%s+\n" "$TBL_SEG_LBL" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM" +} +function tbl_row { + # tbl_row + # (col1 left-aligned, the four numeric cols right-aligned) + printf "| %-16s | %14s | %14s | %14s | %14s |\n" "$1" "$2" "$3" "$4" "$5" +} + +# Five-column table for the --slog latency profile. +# "| %-16s |" -> 18 dashes ; "| %12s |" -> 14 dashes +SLOG_SEG16="------------------" # 18 dashes +SLOG_SEG12="--------------" # 14 dashes +function slog_rule { + printf "+%s+%s+%s+%s+%s+\n" "$SLOG_SEG16" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12" +} +function slog_row { + # slog_row (col1 left, rest right-aligned) + printf "| %-16s | %12s | %12s | %12s | %12s |\n" "$1" "$2" "$3" "$4" "$5" +} + +function run_fio_sync { + # run_fio_sync + # -> echoes "IOPS MB/s p50_us p99_us mean_us" (one space-separated line) + # Forces synchronous IO (--sync=1 => O_SYNC) with the portable psync engine so + # every write is a ZIL commit -- i.e. it exercises the SLOG exactly the way a + # sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own + # sync property. We deliberately DON'T use --group_reporting here: each job is + # reported independently in the JSON and we aggregate ourselves (sum IOPS/bw, + # average p50, take the worst p99), which sidesteps fio's group-merge quirks. + local bs="$1" jobs="$2" + local args=( + --name=slog + --filename="$benchfile" + --ioengine=psync + --rw=randwrite + --bs="$bs" + --size="$size" + --numjobs="$jobs" + --sync=1 + --time_based --runtime="$runtime" + --output-format=json + ) + local out + out=$(sudo fio "${args[@]}" 2>/dev/null) + if [ "$verbose" = 1 ]; then + echo " --- fio json: randwrite bs=$bs numjobs=$jobs sync=1 ---" >&2 + echo "$out" >&2 + echo "" >&2 + fi + # Parse the JSON: sum IOPS and bandwidth across jobs, aggregate completion + # latency percentiles (clat is in ns -> convert to us). bw_bytes is bytes/s. + echo "$out" | python3 -c ' +import json, sys +try: + d = json.load(sys.stdin) + jobs = d["jobs"] +except Exception: + print("0 0.00 0.0 0.0 0.0"); sys.exit(0) +iops = sum(j["write"]["iops"] for j in jobs) +mbps = sum(j["write"]["bw_bytes"] for j in jobs) / 1e6 +def pctl(j, p): + return j["write"]["clat_ns"]["percentile"].get(p) +p50 = [pctl(j, "50.000000") for j in jobs if pctl(j, "50.000000") is not None] +p99 = [pctl(j, "99.000000") for j in jobs if pctl(j, "99.000000") is not None] +mean = [j["write"]["clat_ns"]["mean"] for j in jobs] +p50_us = (sum(p50)/len(p50))/1000.0 if p50 else 0.0 # average of per-job medians +p99_us = (max(p99))/1000.0 if p99 else 0.0 # worst-case tail across jobs +mean_us = (sum(mean)/len(mean))/1000.0 if mean else 0.0 +print("%.0f %.2f %.1f %.1f %.1f" % (iops, mbps, p50_us, p99_us, mean_us)) +' +} + +function slog_speedtest { + # SLOG / sync-write latency profile. Detect engine/direct only for the banner; + # the actual runs always force psync + O_SYNC (see run_fio_sync). + detect_io_settings + benchfile="$path/speedtest-hd.bench" + + if ! command -v fio &> /dev/null; then + echo "ERROR: --slog requires fio (apt install fio / pacman -S fio)." >&2 + exit 1 + fi + if ! command -v python3 &> /dev/null; then + echo "ERROR: --slog requires python3 (used to parse fio JSON latency percentiles)." >&2 + exit 1 + fi + + echo "" + echo " speedtest-hd : SLOG / sync-write latency profile" + echo " Target : $path" + echo " Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine" + echo " Profile : runtime=${runtime}s/run size=${size^^}" + echo " Note : every write is a synchronous ZIL commit -- this is the load" + echo " your SLOG actually sees. Watch it live in another shell with:" + echo " zpool iostat -vl 1" + echo "" + + # T1 is the headline single-stream latency; the sweep shows how the SLOG + # scales as concurrent sync writers (NFS/iSCSI/VM threads) pile on. Run every + # measurement FIRST (progress goes to stderr), collect the results, then draw + # the whole table in one block so the "measuring..." lines can't interleave + # into the middle of the table. + local jobs_list="1 4 8 16" + local -a rows + local j res iops bw p50 p99 mean + for j in $jobs_list; do + echo " measuring 4K sync randwrite T$j ..." >&2 + res=$(run_fio_sync 4k "$j") + read -r iops bw p50 p99 mean <<< "$res" + rows+=("$(slog_row "4K sync T$j" "$iops" "$bw" "$p50" "$p99")") + done + + rm -f "$benchfile" + + # Render the assembled table + echo "" + slog_rule + slog_row "Test" "IOPS" "MB/s" "p50 lat(us)" "p99 lat(us)" + slog_rule + printf '%s\n' "${rows[@]}" + slog_rule + echo "" + echo " Healthy Optane SLOG (eg P1600X) single-stream (T1) target:" + echo " ~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means" + echo " CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling." + echo "" +} + +function cdm_speedtest { # Auto-detect the best engine and whether O_DIRECT works on this path. detect_io_settings + benchfile="$path/speedtest-hd.bench" - # Write tests - fio_write_single_random_4k - fio_write_parallel_random_64k - fio_write_single_sequential_1m + local direct_label="enabled (device)" + [ "$direct" != 1 ] && direct_label="DISABLED (buffered -- may reflect RAM cache!)" - # Read Tests - fio_read_sequential_1m - fio_read_random_4k + # Banner + echo "" + echo " speedtest-hd : CrystalDiskMark-style storage benchmark" + echo " Target : $path" + echo " Engine : $engine O_DIRECT: $direct_label" + echo " Profile : size=${size^^} runtime=${runtime}s/run (8 runs)" + echo "" + + # The four CrystalDiskMark default tests. Q = queue depth (--iodepth), + # T = threads (--numjobs). The first three are T1 (single thread); the last + # is T16. A single shared test file is reused -- with numjobs>1 every job + # issues IO against the same file, which is fine. + # Each run now returns both MB/s and IOPS ("bw iops"), so we capture the pair. + # label read-rw write-rw bs Q T + echo " measuring SEQ1M Q8T1 ..." >&2 + local s8r_bw s8r_io; read -r s8r_bw s8r_io <<< "$(run_fio read 1m 8 1)" + local s8w_bw s8w_io; read -r s8w_bw s8w_io <<< "$(run_fio write 1m 8 1)" + echo " measuring SEQ1M Q1T1 ..." >&2 + local s1r_bw s1r_io; read -r s1r_bw s1r_io <<< "$(run_fio read 1m 1 1)" + local s1w_bw s1w_io; read -r s1w_bw s1w_io <<< "$(run_fio write 1m 1 1)" + echo " measuring RND4K Q32T1 ..." >&2 + local r32r_bw r32r_io; read -r r32r_bw r32r_io <<< "$(run_fio randread 4k 32 1)" + local r32w_bw r32w_io; read -r r32w_bw r32w_io <<< "$(run_fio randwrite 4k 32 1)" + echo " measuring RND4K Q32T16 ..." >&2 + local r1r_bw r1r_io; read -r r1r_bw r1r_io <<< "$(run_fio randread 4k 32 16)" + local r1w_bw r1w_io; read -r r1w_bw r1w_io <<< "$(run_fio randwrite 4k 32 16)" + + # Cleanup the shared test file + rm -f "$benchfile" + + # Render the CrystalDiskMark-style table + echo "" + tbl_rule + tbl_row "Test" "Read (MB/s)" "Write (MB/s)" "Read (IOPS)" "Write (IOPS)" + tbl_rule + tbl_row "SEQ1M Q8T1" "$s8r_bw" "$s8w_bw" "$s8r_io" "$s8w_io" + tbl_row "SEQ1M Q1T1" "$s1r_bw" "$s1w_bw" "$s1r_io" "$s1w_io" + tbl_row "RND4K Q32T1" "$r32r_bw" "$r32w_bw" "$r32r_io" "$r32w_io" + tbl_row "RND4K Q32T16" "$r1r_bw" "$r1w_bw" "$r1r_io" "$r1w_io" + tbl_rule + echo "" } function dd_speedtest { - # Basic HD speed test using DD + # Basic HD speed test using dd (fallback when fio is not installed) # mReschke 2017-07-11 - + local file ddsize file=$path/bigfile - size=1024 + ddsize=1024 echo "Running dd based HD/SSD/NVMe Benchmarks" echo "---------------------------------------" printf "Cached write speed...\n" - dd if=/dev/zero of=$file bs=1M count=$size + dd if=/dev/zero of=$file bs=1M count=$ddsize printf "\nUncached write speed...\n" - dd if=/dev/zero of=$file bs=1M count=$size conv=fdatasync,notrunc + dd if=/dev/zero of=$file bs=1M count=$ddsize conv=fdatasync,notrunc printf "\nUncached read speed...\n" echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null - dd if=$file of=/dev/null bs=1M count=$size + dd if=$file of=/dev/null bs=1M count=$ddsize printf "\nCached read speed...\n" - dd if=$file of=/dev/null bs=1M count=$size + dd if=$file of=/dev/null bs=1M count=$ddsize rm $file printf "\nDone\n" @@ -411,40 +411,45 @@ function dd_speedtest { # Show help and usage information function usage { - echo "Robust Flexible Input/Output HD Speedtest" - echo " If FIO is installed, we use FIO for more detailed performance analysis." - echo " If FIO is not installed, we use basic DD analysis." - echo " You should apt install fio (pacman -S fio) for detailed analysis." + echo "speedtest-hd : CrystalDiskMark-style storage benchmark" + echo " Runs the same four tests as CrystalDiskMark (each Read + Write):" + echo " SEQ1M Q8T1 Sequential 1MiB, queue depth 8, 1 thread" + echo " SEQ1M Q1T1 Sequential 1MiB, queue depth 1, 1 thread" + echo " RND4K Q32T1 Random 4KiB, queue depth 32, 1 thread" + echo " RND4K Q32T16 Random 4KiB, queue depth 32, 16 threads" + echo " Uses fio if installed (recommended), else basic dd." echo "mReschke 2024-01-18" echo "" - echo "NOTICE, this creates a 1GB file on the desired destination disk." - echo "Please ensure you have write access with 1GB free space on destination." + echo "NOTICE: creates one test file (default 1GB) on the destination disk." + echo "Please ensure you have write access with enough free space." echo "" echo "Usage:" - echo " This will use FIO if installed, else DD" + echo " Auto (fio if installed, else dd):" echo " ./speedtest-hd /mnt/somedisk" echo " ./speedtest-hd ." echo "" - echo " This will force FIO" + echo " Force fio / force dd:" echo " ./speedtest-hd /mnt/somedisk --fio" - echo " ./speedtest-hd . --fio" - echo "" - echo " This will force DD" echo " ./speedtest-hd /mnt/somedisk --dd" - echo " ./speedtest-hd . --dd" echo "" - echo " Add --simple (FIO only) for a compact, aligned summary of MB/s values" - echo " ./speedtest-hd . --simple" - echo " ./speedtest-hd . --fio --simple" + echo " SLOG / sync-write latency profile (ZFS ZIL, NFS/iSCSI/VM sync IO):" + echo " ./speedtest-hd /mnt/zpool/dataset --slog" + echo " 4K synchronous writes at T1/T4/T8/T16; reports IOPS, MB/s, p50/p99" + echo " commit latency. Requires fio + python3. (Tip: watch the SLOG with" + echo " 'zpool iostat -vl 1' in another shell.)" echo "" - echo " FIO tuning flags (all auto-detected by default, override as needed):" + echo " Tuning flags (auto-detected by default, override as needed):" echo " --engine=io_uring|libaio|posixaio|sync IO engine (default: auto)" echo " --direct Force O_DIRECT (bypass page cache)" echo " --buffered Force buffered IO (eg if O_DIRECT is unsupported)" - echo " --runtime=SEC Seconds per test (default: 30)" + echo " --runtime=SEC Seconds per run (default: 5, like CrystalDiskMark)" + echo " --size=SIZE Test file size (default: 1g)" + echo " --verbose Print full fio output for every run (summary table unchanged)" echo " Examples:" - echo " ./speedtest-hd /mnt/nvmepool --engine=io_uring --runtime=60" + echo " ./speedtest-hd /mnt/nvmepool --runtime=10 --size=4g" echo " ./speedtest-hd /mnt/nfsshare --buffered" + echo " ./speedtest-hd /mnt/nvmepool --verbose" + echo " ./speedtest-hd /mnt/nvme-ultra-r10/vm-root --slog --runtime=30" exit 0 }