New python speedtest to replace bash. New CLAUDE.md
This commit is contained in:
@@ -0,0 +1,65 @@
|
||||
# speedtest-hd
|
||||
|
||||
Single-file Bash CLI (`speedtest-hd.sh`) that benchmarks any mounted path (HDD/SSD/NVMe,
|
||||
local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII table. Falls back
|
||||
to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount" disk tester — not
|
||||
a ZFS/NVMe-specific tool.
|
||||
|
||||
## Usage
|
||||
```
|
||||
./speedtest-hd.sh <path> [--fio|--dd|--slog] [flags]
|
||||
```
|
||||
- Default: fio if installed, else dd. `--fio` forces CDM mode, `--slog` forces sync-latency mode.
|
||||
- Flags: `--engine=io_uring|libaio|posixaio|sync` (default auto), `--direct`/`--buffered`
|
||||
(default auto-probe), `--runtime=SEC` (default 5), `--size=SIZE` (default 1g), `--verbose`
|
||||
(also dumps raw fio output to stderr).
|
||||
|
||||
## Modes & layout
|
||||
- **CDM mode** (`cdm_speedtest`): runs CrystalDiskMark's default tests, each read+write,
|
||||
prints a 5-col table (Read/Write MB/s, Read/Write IOPS). Tests: `SEQ1M Q8T1`, `SEQ1M Q1T1`,
|
||||
`RND4K Q32T1`, `RND4K Q32T16`. Q = `--iodepth`, T = `--numjobs`.
|
||||
- **SLOG mode** (`slog_speedtest`): 4K synchronous randwrite sweep at T1/T4/T8/T16 to profile
|
||||
ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 commit
|
||||
latency. Requires `fio` + `python3` (JSON percentile parsing).
|
||||
- **dd mode** (`dd_speedtest`): legacy cached/uncached read/write fallback.
|
||||
|
||||
## Key implementation notes
|
||||
- `detect_io_settings` auto-picks the engine (io_uring → libaio → posixaio → sync) and probes
|
||||
whether O_DIRECT works, via tiny throwaway `fio_probe` jobs. Falls back to buffered with a
|
||||
warning if O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly async
|
||||
with `--direct=1`** — that's why io_uring is preferred. O_DIRECT bypasses the page cache so we
|
||||
measure the device, not RAM (buffered results, esp. reads, can reflect ARC/page cache).
|
||||
- One shared test file `<path>/speedtest-hd.bench` is reused across all runs (laid out once),
|
||||
removed at the end. Footprint = `--size` (default 1G), matching the startup notice.
|
||||
- `run_fio` (CDM) returns `"MB/s IOPS"`; `run_fio_sync` (SLOG) forces `--ioengine=psync --sync=1`
|
||||
(O_SYNC) so every write is a ZIL commit regardless of dataset sync property, and parses fio
|
||||
**JSON** to aggregate across jobs (sum IOPS/bw, avg p50, worst-case p99). CDM mode parses
|
||||
fio's text output: `fio_bw_mbps` takes the parenthetical **SI MB/s** (matches CDM's decimal
|
||||
MB/s, normalizes kB/MB/GB); `fio_iops` expands k/M suffixes.
|
||||
- Write tests append `--end_fsync=1` so cached writes can't inflate numbers.
|
||||
- ASCII tables: cell formats and dash-segment widths must stay in sync (`tbl_*` = 18/16 dashes,
|
||||
`slog_*` = 18/14). Verify alignment after editing.
|
||||
|
||||
## History / decisions
|
||||
- Originated from the Ars Technica fio guide. Original 4K test used `numjobs=1 iodepth=1`, which
|
||||
measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1, not
|
||||
a bug. Refactored toward parallel/deep-queue tests to show real device capability, then fully
|
||||
reshaped into the CrystalDiskMark profile above. `--simple` flag was removed; the table is now
|
||||
the only fio output.
|
||||
- Note: the current `RND4K` rows are `Q32T1` + `Q32T16`. CrystalDiskMark's actual latest default
|
||||
profile is `Q32T16` + `Q1T1` (single-queue random is the meaningful low-end number). If aligning
|
||||
strictly to CDM, the `Q32T1` row should become `Q1T1` (`iodepth=1 numjobs=1`).
|
||||
|
||||
## SLOG performance context (why `--slog` exists)
|
||||
Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10`
|
||||
= 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). Poor sync
|
||||
writes were **CPU power management**, not the SLOG:
|
||||
- Fix (biggest last): Dell BIOS profile DAPC → **Performance** (~2×); cstate kernel args; and the
|
||||
big one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, which parked cores at
|
||||
1.2 GHz because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post
|
||||
Init: `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`.
|
||||
- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor).
|
||||
- Diagnostic: `zpool iostat -vl <pool> 1` during fio showed the Optane `logs` vdev at ~90µs
|
||||
disk_wait — proving the SLOG was fine and latency was upstream (CPU).
|
||||
- Healthy Optane SLOG single-stream (T1) target: ~15–25k IOPS, p50 ~40–65µs. Much higher usually
|
||||
= C-states / PCIe ASPM / BIOS power profile throttling.
|
||||
@@ -0,0 +1,291 @@
|
||||
# speedtest-hd
|
||||
|
||||
A robust, CrystalDiskMark‑style storage benchmark for Linux, built on [`fio`](https://fio.readthedocs.io/) by mReschke and my buddy Claude Opus.
|
||||
|
||||
It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync‑write latency profile** for diagnosing ZFS ZIL performance (NFS / iSCSI / VM sync workloads). It auto‑detects the best IO engine and whether `O_DIRECT` works on the target, and falls back to a basic `dd` test when `fio` isn't installed.
|
||||
|
||||
---
|
||||
|
||||
## Table of contents
|
||||
|
||||
- [Features](#features)
|
||||
- [Requirements](#requirements)
|
||||
- [Usage](#usage)
|
||||
- [Output](#output)
|
||||
- [Understanding the tests](#understanding-the-tests)
|
||||
- [Case study: diagnosing a "slow" Optane SLOG on TrueNAS](#case-study-diagnosing-a-slow-optane-slog-on-truenas)
|
||||
- [Notes & caveats](#notes--caveats)
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
- **CrystalDiskMark‑style profile** — `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T1`, `RND4K Q32T16`, each measured for Read **and** Write, reported in both **MB/s** and **IOPS**.
|
||||
- **SLOG / sync‑write latency profile** (`--slog`) — synchronous 4K writes at T1/T4/T8/T16 reporting **IOPS, MB/s, and p50/p99 commit latency**. This is the load a ZFS SLOG actually sees.
|
||||
- **Auto‑detection** — picks the fastest available IO engine (`io_uring` → `libaio` → `posixaio` → `sync`) and probes whether `O_DIRECT` works on the filesystem, falling back to buffered IO when it doesn't (e.g. older OpenZFS, some NFS mounts).
|
||||
- **`dd` fallback** — if `fio` isn't present, runs a basic write/read test so you still get a number.
|
||||
- **Verbose mode** — `--verbose` dumps the full raw `fio` output for every run while keeping the summary table intact.
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
|
||||
- `bash`
|
||||
- [`fio`](https://fio.readthedocs.io/) — recommended (`apt install fio` / `pacman -S fio`). Without it, the tool falls back to `dd`.
|
||||
- `python3` — **only** required for `--slog` (used to parse `fio`'s JSON output for latency percentiles). The normal profile needs only `fio` + `grep`/`awk`.
|
||||
- `sudo` — `fio` is invoked via `sudo` so it can use `O_DIRECT` and flush device caches.
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
./speedtest-hd.sh <path> [options]
|
||||
```
|
||||
|
||||
`<path>` is the directory (or mount) to benchmark. Use `.` for the current directory. The tool creates a single test file (default 1 GiB) on the target and removes it afterward, so ensure enough free space.
|
||||
|
||||
### Modes
|
||||
|
||||
| Invocation | What it does |
|
||||
|---|---|
|
||||
| `./speedtest-hd.sh /mnt/disk` | Auto: uses `fio` if installed, else `dd` |
|
||||
| `./speedtest-hd.sh /mnt/disk --fio` | Force the `fio` CrystalDiskMark‑style profile |
|
||||
| `./speedtest-hd.sh /mnt/disk --dd` | Force the basic `dd` test |
|
||||
| `./speedtest-hd.sh /mnt/disk --slog` | SLOG / sync‑write latency profile |
|
||||
|
||||
### Tuning flags
|
||||
|
||||
| Flag | Effect |
|
||||
|---|---|
|
||||
| `--engine=io_uring\|libaio\|posixaio\|sync` | Force a specific IO engine (default: auto) |
|
||||
| `--direct` | Force `O_DIRECT` (bypass page cache) |
|
||||
| `--buffered` | Force buffered IO (e.g. when `O_DIRECT` is unsupported) |
|
||||
| `--runtime=SEC` | Seconds per run (default: 5, like CrystalDiskMark) |
|
||||
| `--size=SIZE` | Test file size (default: `1g`) |
|
||||
| `--verbose` | Also print the full `fio` output for every run (summary table unchanged) |
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# CrystalDiskMark-style test of the current directory
|
||||
./speedtest-hd.sh .
|
||||
|
||||
# Larger file, longer runs, on an NVMe pool
|
||||
./speedtest-hd.sh /mnt/nvmepool --runtime=10 --size=4g
|
||||
|
||||
# Buffered (e.g. an NFS share that doesn't support O_DIRECT)
|
||||
./speedtest-hd.sh /mnt/nfsshare --buffered
|
||||
|
||||
# SLOG / sync latency profile, 30s per run
|
||||
./speedtest-hd.sh /mnt/nvme-ultra-r10/vm-root --slog --runtime=30
|
||||
```
|
||||
|
||||
> **Tip:** when running `--slog` against a ZFS dataset, watch the SLOG live in another shell:
|
||||
> ```bash
|
||||
> zpool iostat -vl <pool> 1
|
||||
> ```
|
||||
|
||||
---
|
||||
|
||||
## Output
|
||||
|
||||
### CrystalDiskMark‑style profile
|
||||
|
||||
```
|
||||
+------------------+----------------+----------------+----------------+----------------+
|
||||
| Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) |
|
||||
+------------------+----------------+----------------+----------------+----------------+
|
||||
| SEQ1M Q8T1 | 6873.00 | 9.30 | 6873 | 9 |
|
||||
| SEQ1M Q1T1 | 1608.00 | 20.00 | 1608 | 20 |
|
||||
| RND4K Q32T1 | 538.00 | 10.80 | 137728 | 2764 |
|
||||
| RND4K Q32T16 | 689.00 | 261.00 | 176384 | 66816 |
|
||||
+------------------+----------------+----------------+----------------+----------------+
|
||||
```
|
||||
|
||||
### SLOG / sync‑write latency profile (`--slog`)
|
||||
|
||||
```
|
||||
+------------------+--------------+--------------+--------------+--------------+
|
||||
| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) |
|
||||
+------------------+--------------+--------------+--------------+--------------+
|
||||
| 4K sync T1 | 10687 | 43.77 | 85.5 | 185.3 |
|
||||
| 4K sync T4 | 29873 | 122.36 | 117.8 | 317.4 |
|
||||
| 4K sync T8 | 52612 | 215.50 | 136.2 | 391.2 |
|
||||
| 4K sync T16 | 77939 | 319.24 | 180.0 | 505.9 |
|
||||
+------------------+--------------+--------------+--------------+--------------+
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Understanding the tests
|
||||
|
||||
### The CrystalDiskMark profile
|
||||
|
||||
| Test | Pattern | Queue depth | Threads |
|
||||
|---|---|---|---|
|
||||
| `SEQ1M Q8T1` | Sequential 1 MiB | 8 | 1 |
|
||||
| `SEQ1M Q1T1` | Sequential 1 MiB | 1 | 1 |
|
||||
| `RND4K Q32T1` | Random 4 KiB | 32 | 1 |
|
||||
| `RND4K Q32T16` | Random 4 KiB | 32 | 16 |
|
||||
|
||||
`Q` = queue depth (`--iodepth`), `T` = threads (`--numjobs`). Note that `--iodepth` only produces real queue depth when the IO is truly asynchronous (async engine **and** `O_DIRECT`). On filesystems where that isn't available, queue depth effectively collapses toward 1 and concurrency comes only from threads (`T`).
|
||||
|
||||
### The SLOG profile
|
||||
|
||||
`--slog` forces **synchronous** 4K random writes (`--sync=1` → `O_SYNC`) via the portable `psync` engine. Every write becomes a durable commit, so it traverses the ZFS ZIL / SLOG commit path exactly the way a `sync=always` dataset (NFS, iSCSI, VM storage) does — regardless of the dataset's own `sync` property. It sweeps thread counts (T1 → T4 → T8 → T16):
|
||||
|
||||
- **T1** is the headline single‑stream latency (e.g. one database committing in a tight loop).
|
||||
- **The sweep** shows how the SLOG scales as concurrent sync writers pile on (multiple VMs / NFS clients) — usually the more important number for a virtualization host.
|
||||
|
||||
A healthy Optane SLOG (e.g. P1600X) single‑stream target is roughly **15–25k IOPS, p50 ~40–65 µs**. Much higher latency usually points at **CPU C‑states / PCIe ASPM / a BIOS power profile** throttling the host — see the case study below.
|
||||
|
||||
---
|
||||
|
||||
## Case study: diagnosing a "slow" Optane SLOG on TrueNAS
|
||||
|
||||
A real investigation that this tool's `--slog` mode was built to support. **Spoiler: the Optane SSD was healthy the entire time. The bottleneck was CPU power management.**
|
||||
|
||||
### The setup
|
||||
|
||||
| Component | Detail |
|
||||
|---|---|
|
||||
| Server | Dell PowerEdge R630 |
|
||||
| CPU | Intel Xeon E5‑2680 v3 (Haswell‑EP, 12C/24T, 2.5 GHz base, 3.3 GHz turbo) |
|
||||
| OS | TrueNAS SCALE 25.10 (OpenZFS 2.3) |
|
||||
| Pool | `nvme-ultra-r10` — 6× 4 TB KingSpec XG7000 NVMe in RAID10 (3 mirror vdevs) |
|
||||
| SLOG | Intel Optane **P1600X** |
|
||||
| Dataset | `/mnt/nvme-ultra-r10/vm-root`, `sync=always` |
|
||||
|
||||
### The symptom
|
||||
|
||||
The standard benchmark looked alarming — huge reads, tiny writes:
|
||||
|
||||
```
|
||||
+------------------+------------------+------------------+
|
||||
| Test | Read (MB/s) | Write (MB/s) |
|
||||
+------------------+------------------+------------------+
|
||||
| SEQ1M Q8T1 | 6873.00 | 9.30 |
|
||||
| SEQ1M Q1T1 | 1608.00 | 20.00 |
|
||||
| RND4K Q32T1 | 538.00 | 10.80 |
|
||||
| RND4K Q32T16 | 689.00 | 261.00 |
|
||||
+------------------+------------------+------------------+
|
||||
```
|
||||
|
||||
9.3 MB/s sequential write on an Optane‑backed NVMe pool looks broken.
|
||||
|
||||
### Investigation
|
||||
|
||||
**1. The reads are RAM, not disk.** With a 1 GiB test file on ZFS, reads come straight from ARC (RAM cache). The huge read/write asymmetry is the tell — ignore the read column for judging the disks.
|
||||
|
||||
**2. `sync=always` makes writes latency‑bound.** Every write must be durably committed to the ZIL before it's acknowledged, so throughput ≈ (block size) ÷ (per‑commit latency). Anything running at low effective concurrency looks slow regardless of raw device speed.
|
||||
|
||||
**3. The `Q8`/`Q32` labels were misleading.** On this ZFS setup `--iodepth` didn't produce real queue depth, so most rows effectively ran at QD1. Proof: `RND4K Q32T1` (10.8 MB/s) vs `RND4K Q32T16` (261 MB/s) — the 24× jump came entirely from threads (`numjobs=16`), not queue depth.
|
||||
|
||||
**4. Large sequential sync writes bypass the SLOG.** With ZFS's default `logbias=latency`, writes larger than `zfs_immediate_write_sz` (32 KB) use an *indirect* ZIL record — the data goes straight to the main pool and only a pointer hits the Optane. So the `SEQ1M` write test was measuring the **consumer KingSpec pool's** forced‑sync performance, not the SLOG. Only small (4K) sync writes exercise the Optane.
|
||||
|
||||
**5. Confirm with `zpool iostat -vl <pool> 1` during a 4K sync test.** This was decisive:
|
||||
|
||||
- The `logs` (Optane) vdev took **all** the sync writes (~2.3–2.6k ops, ~18–20 MB/s); the data vdevs were idle between txg flushes. → **SLOG configured correctly, `logbias` fine, not bypassed.**
|
||||
- But the Optane's own `disk_wait` was **~90 µs** (a P1600X should be ~10–15 µs), and the fio‑level commit latency was **~328 µs** — meaning ~238 µs was being spent *above* the device, in the host/ZFS/CPU path.
|
||||
|
||||
That "faster when busy, slow when idle" device latency plus huge host overhead is the classic signature of **power‑saving idle states** on a latency‑bound, QD1 workload.
|
||||
|
||||
**6. Find the throttle.** Checking the CPU revealed the cores pinned at **1.2 GHz** — the E5‑2680 v3's *minimum* P‑state — on a chip rated for 2.5–3.3 GHz:
|
||||
|
||||
```
|
||||
$ grep MHz /proc/cpuinfo | sort -u
|
||||
cpu MHz : 1200.069
|
||||
...
|
||||
|
||||
$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
|
||||
intel_cpufreq # = intel_pstate in passive mode
|
||||
$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
|
||||
schedutil # picks frequency from CPU utilization
|
||||
```
|
||||
|
||||
The vicious loop: a QD1 sync workload spends each commit **blocked** waiting on the SLOG → the `schedutil` governor sees near‑zero utilization → parks the cores at 1.2 GHz → the ZFS commit code path runs ~2–3× slower → latency climbs.
|
||||
|
||||
### Root cause
|
||||
|
||||
**CPU power management, in two layers — not the SSD, pool, or PCIe link:**
|
||||
|
||||
1. **BIOS System Profile = "Performance Per Watt Optimized (DAPC)"** — Dell Active Power Controller manages C‑states/P‑states in firmware and largely ignores the OS, keeping cores idling deep and clocked low.
|
||||
2. **OS `schedutil` governor** (TrueNAS SCALE default) — pinned cores at the 1.2 GHz floor for this bursty, IO‑blocked workload.
|
||||
|
||||
### The fixes
|
||||
|
||||
Applied in order, biggest impact last:
|
||||
|
||||
**1. BIOS System Profile → Performance** (disables C‑states/C1E, raises P‑states):
|
||||
```
|
||||
# In BIOS (F2): System BIOS → System Profile Settings → System Profile → Performance
|
||||
# Or via iDRAC:
|
||||
racadm set BIOS.SysProfileSettings.SysProfile PerfOptimized
|
||||
racadm jobqueue create BIOS.Setup.1-1
|
||||
# reboot to apply
|
||||
```
|
||||
|
||||
**2. Kernel parameters** (target PCIe/NVMe link power saving + residual C‑states). On TrueNAS: *System → Advanced Settings → Kernel Arguments*, then reboot:
|
||||
```
|
||||
intel_idle.max_cstate=1 processor.max_cstate=1 pcie_aspm=off nvme_core.default_ps_max_latency_us=0
|
||||
```
|
||||
|
||||
**3. CPU governor → `performance`** *(the single biggest win)*:
|
||||
```bash
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
```
|
||||
Make it persistent on TrueNAS — *System → Advanced Settings → Init/Shutdown Scripts*, add a **Post Init** *Command*:
|
||||
```
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
```
|
||||
> ⚠️ Without the Post Init script the governor reverts to `schedutil` on every reboot, silently dropping you back to slow numbers.
|
||||
|
||||
### Results — before & after
|
||||
|
||||
**4K synchronous random writes** (`--slog`), per stage of the fix:
|
||||
|
||||
| Stage | T1 IOPS | T1 p50 | T1 MB/s | T16 IOPS | T16 MB/s | T16 p99 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **DAPC** (start) | ~3,050 | ~328 µs | ~12.5 | — | 38 *(regressing)* | thrashing |
|
||||
| BIOS → Performance | 5,849 | 160.8 µs | 23.96 | 46,009 | 188.45 | 1057 µs |
|
||||
| + kernel parameters | 6,217 | 150.5 µs | 25.46 | 47,166 | 193.19 | 684 µs |
|
||||
| **+ `performance` governor** | **10,687** | **85.5 µs** | **43.77** | **77,939** | **319.24** | **506 µs** |
|
||||
|
||||
**Full final result:**
|
||||
|
||||
```
|
||||
+------------------+--------------+--------------+--------------+--------------+
|
||||
| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) |
|
||||
+------------------+--------------+--------------+--------------+--------------+
|
||||
| 4K sync T1 | 10687 | 43.77 | 85.5 | 185.3 |
|
||||
| 4K sync T4 | 29873 | 122.36 | 117.8 | 317.4 |
|
||||
| 4K sync T8 | 52612 | 215.50 | 136.2 | 391.2 |
|
||||
| 4K sync T16 | 77939 | 319.24 | 180.0 | 505.9 |
|
||||
+------------------+--------------+--------------+--------------+--------------+
|
||||
```
|
||||
|
||||
**Net improvement:** ~**3.5× IOPS**, ~**3.8× lower latency** at T1, and ~**8.4× aggregate throughput** at T16 — with the scaling regression eliminated entirely.
|
||||
|
||||
### Lessons learned
|
||||
|
||||
- **An Optane SLOG showing high latency is usually a host‑side power‑management problem, not the device.** Confirm where the time goes before blaming hardware.
|
||||
- **`zpool iostat -vl <pool> 1` is the key diagnostic** — it shows whether the `logs` vdev is actually taking the writes and splits device latency (`disk_wait`) from host/ZFS overhead (`total_wait`).
|
||||
- **Latency‑bound QD1 sync workloads are the worst case for power saving.** The CPU looks idle (blocked on IO), so governors and firmware clock it down — which directly inflates the latency you're trying to measure.
|
||||
- **On TrueNAS SCALE, the default `schedutil` governor cripples sync‑write latency.** Set `performance` (and persist it).
|
||||
- **Reads from a small test file measure ARC (RAM), not the disk.** Watch the read/write asymmetry.
|
||||
- **Large sync writes bypass the SLOG** (indirect ZIL) — to actually test a SLOG, use small (4K) sync writes, which is exactly what `--slog` does.
|
||||
|
||||
### `~85 µs` is roughly the floor here
|
||||
|
||||
The residual gap from raw Optane (~15 µs) is ZFS ZIL‑commit overhead plus the per‑op cost of a 2014‑era Haswell core. Closing it further would need a newer/faster CPU for sharply diminishing returns. For a virtualization host the aggregate (78k IOPS / 319 MB/s) is what the workload feels, and it's healthy.
|
||||
|
||||
---
|
||||
|
||||
## Notes & caveats
|
||||
|
||||
- **`sudo`** is used for `fio` so it can apply `O_DIRECT` and flush device write caches at the end of write runs (`--end_fsync=1`), so cached writes can't inflate results.
|
||||
- **`O_DIRECT` is auto‑detected.** If the banner shows `O_DIRECT: DISABLED (buffered ...)`, results may reflect the page cache (RAM) rather than the device.
|
||||
- **A single shared test file** is reused across runs to keep the footprint to one file.
|
||||
- **`--slog` requires `python3`** (for JSON latency‑percentile parsing); the standard profile does not.
|
||||
- The `--slog` profile forces synchronous IO and is intended for ZFS ZIL / SLOG and other sync‑write (NFS/iSCSI/VM) investigations.
|
||||
Binary file not shown.
Executable
+727
@@ -0,0 +1,727 @@
|
||||
#!/usr/bin/env python3
|
||||
"""speedtest-hd — a CrystalDiskMark-style storage benchmark for Linux, on fio.
|
||||
|
||||
This is the Python successor to ``speedtest-hd.sh``. It measures storage the way
|
||||
CrystalDiskMark does, and adds a dedicated **SLOG / sync-write latency profile**
|
||||
for diagnosing ZFS ZIL performance (NFS / iSCSI / VM sync workloads).
|
||||
|
||||
Three profiles
|
||||
--------------
|
||||
* **cdm** — the four CrystalDiskMark default tests (``SEQ1M Q8T1``, ``SEQ1M
|
||||
Q1T1``, ``RND4K Q32T1``, ``RND4K Q32T16``), each measured for read *and*
|
||||
write, reported in MB/s and IOPS.
|
||||
* **slog** — synchronous 4K random writes swept across thread counts, reporting
|
||||
IOPS, MB/s and p50/p99 *commit latency*. This is the load a ZFS SLOG sees.
|
||||
* **dd** — a dependency-free fallback when ``fio`` isn't installed.
|
||||
|
||||
Why fio JSON?
|
||||
-------------
|
||||
Every measurement asks fio for ``--output-format=json`` and parses it with the
|
||||
standard library. That's the whole reason this is Python and not bash: robust,
|
||||
unit-safe parsing of bandwidth / IOPS / latency percentiles with no fragile
|
||||
text scraping.
|
||||
|
||||
See README.md for the full case study (diagnosing a "slow" Optane SLOG that
|
||||
turned out to be CPU power management, not the disk).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable, Optional, Sequence
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Color / styling (standard library only -- raw ANSI escape codes)
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
# SGR escape sequences. We stick to the basic 16-color set so output respects
|
||||
# the user's terminal theme instead of hard-coding RGB.
|
||||
RESET = "\033[0m"
|
||||
BOLD = "\033[1m"
|
||||
DIM = "\033[2m"
|
||||
ITALIC = "\033[3m"
|
||||
RED = "\033[31m"
|
||||
GREEN = "\033[32m"
|
||||
YELLOW = "\033[33m"
|
||||
BLUE = "\033[34m"
|
||||
MAGENTA = "\033[35m"
|
||||
CYAN = "\033[36m"
|
||||
WHITE = "\033[37m"
|
||||
BRIGHT_CYAN = "\033[96m"
|
||||
|
||||
|
||||
def supports_color(stream) -> bool:
|
||||
"""Decide per-stream whether to emit ANSI codes.
|
||||
|
||||
Honors the de-facto ``NO_COLOR`` / ``FORCE_COLOR`` conventions and otherwise
|
||||
colors only when the stream is an interactive terminal.
|
||||
"""
|
||||
if os.environ.get("NO_COLOR"):
|
||||
return False
|
||||
if os.environ.get("FORCE_COLOR"):
|
||||
return True
|
||||
if os.environ.get("TERM") == "dumb":
|
||||
return False
|
||||
return bool(getattr(stream, "isatty", None)) and stream.isatty()
|
||||
|
||||
|
||||
class Painter:
|
||||
"""Wraps text in SGR codes, or returns it untouched when color is off."""
|
||||
|
||||
def __init__(self, enabled: bool):
|
||||
self.enabled = enabled
|
||||
|
||||
def paint(self, text: str, *codes: str) -> str:
|
||||
if not self.enabled or not codes:
|
||||
return text
|
||||
return "".join(codes) + text + RESET
|
||||
|
||||
|
||||
# stdout carries results (banner + tables); stderr carries progress + verbose
|
||||
# fio dumps. Each gets its own enable flag so piping one but not the other works.
|
||||
OUT = Painter(supports_color(sys.stdout))
|
||||
ERR = Painter(supports_color(sys.stderr))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Constants
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
# Preference order for the IO engine. io_uring is the modern, lowest-overhead
|
||||
# Linux engine; libaio is older (and only truly async with O_DIRECT); posixaio
|
||||
# and sync are the portable fallbacks (e.g. some NFS mounts).
|
||||
ENGINE_CANDIDATES = ("io_uring", "libaio", "posixaio", "sync")
|
||||
|
||||
# Binary unit multipliers (fio treats "1g" as 1 GiB, so we match that).
|
||||
_SIZE_UNITS = {"k": 1024, "m": 1024**2, "g": 1024**3, "t": 1024**4}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CdmTest:
|
||||
"""One CrystalDiskMark test. ``seq`` picks sequential vs random patterns."""
|
||||
|
||||
label: str
|
||||
bs: str
|
||||
iodepth: int
|
||||
numjobs: int
|
||||
seq: bool
|
||||
|
||||
@property
|
||||
def read_rw(self) -> str:
|
||||
return "read" if self.seq else "randread"
|
||||
|
||||
@property
|
||||
def write_rw(self) -> str:
|
||||
return "write" if self.seq else "randwrite"
|
||||
|
||||
|
||||
# The four CrystalDiskMark default tests, as data. Q = queue depth (iodepth),
|
||||
# T = threads (numjobs).
|
||||
CDM_TESTS: tuple[CdmTest, ...] = (
|
||||
CdmTest("SEQ1M Q8T1", bs="1m", iodepth=8, numjobs=1, seq=True),
|
||||
CdmTest("SEQ1M Q1T1", bs="1m", iodepth=1, numjobs=1, seq=True),
|
||||
CdmTest("RND4K Q32T1", bs="4k", iodepth=32, numjobs=1, seq=False),
|
||||
CdmTest("RND4K Q32T16", bs="4k", iodepth=32, numjobs=16, seq=False),
|
||||
)
|
||||
|
||||
# Thread counts for the SLOG sweep. T1 is the headline single-stream latency;
|
||||
# the higher counts show how the SLOG scales as concurrent sync writers pile on.
|
||||
SLOG_THREADS: tuple[int, ...] = (1, 4, 8, 16)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Configuration
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@dataclass
|
||||
class Config:
|
||||
"""Resolved run settings. ``engine`` and ``direct`` may start unset and get
|
||||
filled in by :func:`detect_io_settings`."""
|
||||
|
||||
path: str
|
||||
mode: Optional[str] # "cdm" | "slog" | "dd" | None (auto)
|
||||
engine: Optional[str] # None => auto-detect
|
||||
direct: Optional[bool] # None => auto-detect
|
||||
runtime: int
|
||||
size: str
|
||||
verbose: bool
|
||||
assume_yes: bool
|
||||
benchfile: str = field(default="")
|
||||
|
||||
@property
|
||||
def direct_flag(self) -> int:
|
||||
"""fio's --direct takes 0/1; default to 0 until detection runs."""
|
||||
return 1 if self.direct else 0
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Small helpers
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def log(message: str) -> None:
|
||||
"""Progress/diagnostic line -> stderr, so stdout stays pure results."""
|
||||
print(message, file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def step(message: str) -> None:
|
||||
"""A single colored progress line ('measuring ...') -> stderr."""
|
||||
log(f" {ERR.paint('▶', BOLD, CYAN)} {ERR.paint(message, DIM)}")
|
||||
|
||||
|
||||
def vsection(title: str) -> None:
|
||||
"""A clear, ruled header for one --verbose fio section -> stderr."""
|
||||
width = 78
|
||||
head = f"─── {title} "
|
||||
head += "─" * max(3, width - len(head))
|
||||
log("")
|
||||
log(ERR.paint(head, BOLD, CYAN))
|
||||
|
||||
|
||||
def parse_size_bytes(size: str) -> int:
|
||||
"""Turn an fio-style size ("4k", "1g", "512m", "2048") into bytes."""
|
||||
s = size.strip().lower()
|
||||
if s and s[-1] in _SIZE_UNITS:
|
||||
return int(float(s[:-1]) * _SIZE_UNITS[s[-1]])
|
||||
return int(s)
|
||||
|
||||
|
||||
def render_table(
|
||||
headers: Sequence[str],
|
||||
rows: Sequence[Sequence[str]],
|
||||
aligns: Optional[Sequence[str]] = None,
|
||||
col_styles: Optional[Sequence[Optional[str]]] = None,
|
||||
) -> str:
|
||||
"""Render a bordered ASCII table whose columns auto-size to their content.
|
||||
|
||||
``aligns`` is a per-column "<" (left) or ">" (right); by default the first
|
||||
column is left-aligned (the label) and the rest are right-aligned (numbers).
|
||||
``col_styles`` is an optional per-column ANSI style applied to body cells
|
||||
(header is always bold, borders dim). Widths are computed on the *plain*
|
||||
text and color is applied after padding, so escape codes never skew layout.
|
||||
"""
|
||||
ncols = len(headers)
|
||||
if aligns is None:
|
||||
aligns = ["<"] + [">"] * (ncols - 1)
|
||||
if col_styles is None:
|
||||
col_styles = [None] * ncols
|
||||
|
||||
# Widest cell (header or any row) sets each column's width.
|
||||
widths = [
|
||||
max([len(str(headers[c]))] + [len(str(r[c])) for r in rows])
|
||||
for c in range(ncols)
|
||||
]
|
||||
|
||||
bar = OUT.paint("|", DIM)
|
||||
sep = OUT.paint(" | ", DIM)
|
||||
|
||||
def rule() -> str:
|
||||
return OUT.paint("+" + "+".join("-" * (w + 2) for w in widths) + "+", DIM)
|
||||
|
||||
def line(cells: Sequence[str], *, header: bool = False) -> str:
|
||||
parts = []
|
||||
for c in range(ncols):
|
||||
padded = f"{str(cells[c]):{aligns[c]}{widths[c]}}"
|
||||
if header:
|
||||
parts.append(OUT.paint(padded, BOLD))
|
||||
elif col_styles[c]:
|
||||
parts.append(OUT.paint(padded, col_styles[c]))
|
||||
else:
|
||||
parts.append(padded)
|
||||
return f"{bar} " + sep.join(parts) + f" {bar}"
|
||||
|
||||
out = [rule(), line(headers, header=True), rule()]
|
||||
out += [line(r) for r in rows]
|
||||
out.append(rule())
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# fio: results, invocation, parsing
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FioResult:
|
||||
"""Parsed metrics for one direction (read or write) of an fio run.
|
||||
|
||||
Latencies are completion latency (clat) in microseconds; for synchronous
|
||||
writes that is effectively the durable-commit latency.
|
||||
"""
|
||||
|
||||
bw_mbps: float # decimal MB/s, matching CrystalDiskMark's SI figure
|
||||
iops: float
|
||||
p50_us: float
|
||||
p99_us: float
|
||||
mean_us: float
|
||||
|
||||
@classmethod
|
||||
def zero(cls) -> "FioResult":
|
||||
return cls(0.0, 0.0, 0.0, 0.0, 0.0)
|
||||
|
||||
|
||||
def _direction_of(rw: str) -> str:
|
||||
"""fio reports 'read' and 'write' sub-objects; pick the active one."""
|
||||
return "read" if "read" in rw else "write"
|
||||
|
||||
|
||||
def _aggregate(jobs: list[dict], direction: str) -> FioResult:
|
||||
"""Combine per-job fio JSON stats into a single :class:`FioResult`.
|
||||
|
||||
We deliberately run fio *without* --group_reporting and aggregate ourselves,
|
||||
which sidesteps fio's group-merge quirks: sum throughput, average the median
|
||||
latency, and take the worst-case tail (p99) across jobs.
|
||||
"""
|
||||
|
||||
def section(job: dict) -> dict:
|
||||
return job.get(direction, {})
|
||||
|
||||
bw_mbps = sum(section(j).get("bw_bytes", 0) for j in jobs) / 1e6
|
||||
iops = sum(section(j).get("iops", 0.0) for j in jobs)
|
||||
|
||||
def percentiles(key: str) -> list[float]:
|
||||
vals = []
|
||||
for j in jobs:
|
||||
pct = section(j).get("clat_ns", {}).get("percentile", {})
|
||||
v = pct.get(key)
|
||||
if v is not None:
|
||||
vals.append(v)
|
||||
return vals
|
||||
|
||||
p50_ns = percentiles("50.000000")
|
||||
p99_ns = percentiles("99.000000")
|
||||
mean_ns = [section(j).get("clat_ns", {}).get("mean", 0.0) for j in jobs]
|
||||
|
||||
return FioResult(
|
||||
bw_mbps=bw_mbps,
|
||||
iops=iops,
|
||||
p50_us=(statistics.mean(p50_ns) / 1000.0) if p50_ns else 0.0, # avg of medians
|
||||
p99_us=(max(p99_ns) / 1000.0) if p99_ns else 0.0, # worst tail
|
||||
mean_us=(statistics.mean(mean_ns) / 1000.0) if mean_ns else 0.0,
|
||||
)
|
||||
|
||||
|
||||
def run_fio(
|
||||
cfg: Config,
|
||||
*,
|
||||
rw: str,
|
||||
bs: str,
|
||||
iodepth: int,
|
||||
numjobs: int,
|
||||
engine: Optional[str] = None,
|
||||
direct: Optional[bool] = None,
|
||||
sync: bool = False,
|
||||
end_fsync: bool = False,
|
||||
) -> FioResult:
|
||||
"""Run a single fio job against the shared bench file and parse its JSON.
|
||||
|
||||
``engine``/``direct`` default to the detected config values; the SLOG path
|
||||
overrides them (psync + buffered + O_SYNC). ``end_fsync`` flushes the device
|
||||
cache at the end of a write run so cached writes can't inflate the result.
|
||||
"""
|
||||
engine = engine if engine is not None else cfg.engine
|
||||
direct = direct if direct is not None else cfg.direct
|
||||
|
||||
cmd = [
|
||||
"sudo", "fio",
|
||||
"--name=speedtest",
|
||||
f"--filename={cfg.benchfile}",
|
||||
f"--ioengine={engine}",
|
||||
f"--direct={1 if direct else 0}",
|
||||
f"--rw={rw}",
|
||||
f"--bs={bs}",
|
||||
f"--size={cfg.size}",
|
||||
f"--numjobs={numjobs}",
|
||||
f"--iodepth={iodepth}",
|
||||
"--time_based",
|
||||
f"--runtime={cfg.runtime}",
|
||||
"--output-format=json",
|
||||
]
|
||||
if sync:
|
||||
cmd.append("--sync=1") # O_SYNC: every write is a durable commit
|
||||
if end_fsync:
|
||||
cmd.append("--end_fsync=1") # flush device cache once at the end
|
||||
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True)
|
||||
|
||||
if cfg.verbose:
|
||||
sync_tag = " sync=1" if sync else ""
|
||||
vsection(f"fio · {rw} bs={bs} iodepth={iodepth} numjobs={numjobs}{sync_tag}")
|
||||
log(" " + ERR.paint("$ " + " ".join(cmd[1:]), DIM, ITALIC))
|
||||
log(ERR.paint(proc.stdout.strip() or "(no stdout)", DIM))
|
||||
if proc.stderr.strip():
|
||||
log(ERR.paint(proc.stderr.strip(), YELLOW))
|
||||
|
||||
try:
|
||||
data = json.loads(proc.stdout)
|
||||
return _aggregate(data["jobs"], _direction_of(rw))
|
||||
except (json.JSONDecodeError, KeyError, ValueError):
|
||||
# fio failed or produced no parseable JSON; report zeros rather than
|
||||
# crashing the whole run (verbose mode above shows what went wrong).
|
||||
return FioResult.zero()
|
||||
|
||||
|
||||
def fio_probe(path: str, engine: str, direct: bool) -> bool:
|
||||
"""Throwaway 1s fio job to see if an (engine, O_DIRECT) combo works here.
|
||||
|
||||
This is how we stay accurate *and* portable across ext4/xfs/btrfs/ZFS/NFS
|
||||
without the caller needing to know what each filesystem supports.
|
||||
"""
|
||||
cmd = [
|
||||
"sudo", "fio", "--name=probe",
|
||||
f"--directory={path}",
|
||||
f"--ioengine={engine}",
|
||||
"--rw=write", "--bs=4k", "--size=1m",
|
||||
f"--direct={1 if direct else 0}",
|
||||
"--time_based", "--runtime=1",
|
||||
]
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True)
|
||||
for leftover in glob.glob(os.path.join(path, "probe*")):
|
||||
try:
|
||||
os.remove(leftover)
|
||||
except OSError:
|
||||
pass
|
||||
return proc.returncode == 0
|
||||
|
||||
|
||||
def detect_io_settings(cfg: Config) -> None:
|
||||
"""Fill in ``cfg.engine`` and ``cfg.direct`` if the user didn't force them.
|
||||
|
||||
O_DIRECT bypasses the OS page cache so we measure the device, not RAM. Not
|
||||
every filesystem supports it (older OpenZFS, some NFS), so we probe and fall
|
||||
back to buffered rather than erroring out.
|
||||
"""
|
||||
if cfg.engine is None:
|
||||
for engine in ENGINE_CANDIDATES:
|
||||
if fio_probe(cfg.path, engine, direct=False):
|
||||
cfg.engine = engine
|
||||
break
|
||||
else:
|
||||
cfg.engine = "sync"
|
||||
|
||||
if cfg.direct is None:
|
||||
cfg.direct = fio_probe(cfg.path, cfg.engine, direct=True)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Profiles
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _meta_line(text: str) -> str:
|
||||
"""Color the 'Label :' prefix of a banner metadata line, leave value as-is.
|
||||
|
||||
The value may already contain its own ANSI codes (e.g. a red O_DIRECT
|
||||
warning); since these lines aren't width-aligned that's harmless.
|
||||
"""
|
||||
key, sep, val = text.partition(":")
|
||||
if not sep:
|
||||
return " " + text
|
||||
return " " + OUT.paint(key + ":", BOLD, BRIGHT_CYAN) + val
|
||||
|
||||
|
||||
def banner(title: str, cfg: Config, extra: Sequence[str] = ()) -> None:
|
||||
full = f"speedtest-hd · {title}"
|
||||
inner = len(full) + 4
|
||||
|
||||
print()
|
||||
print(OUT.paint("╭" + "─" * inner + "╮", BOLD, CYAN))
|
||||
print(
|
||||
OUT.paint("│", BOLD, CYAN)
|
||||
+ " " + OUT.paint(full, BOLD, WHITE) + " "
|
||||
+ OUT.paint("│", BOLD, CYAN)
|
||||
)
|
||||
print(OUT.paint("╰" + "─" * inner + "╯", BOLD, CYAN))
|
||||
print(_meta_line(f"Target : {OUT.paint(cfg.path, CYAN)}"))
|
||||
for line_ in extra:
|
||||
print(_meta_line(line_))
|
||||
print()
|
||||
|
||||
|
||||
def cdm_profile(cfg: Config) -> None:
|
||||
"""The CrystalDiskMark-style profile: 4 tests x (read, write), MB/s + IOPS."""
|
||||
detect_io_settings(cfg)
|
||||
|
||||
if cfg.direct:
|
||||
direct_label = OUT.paint("enabled (device)", GREEN)
|
||||
else:
|
||||
direct_label = OUT.paint(
|
||||
"DISABLED (buffered -- may reflect RAM cache!)", BOLD, RED
|
||||
)
|
||||
|
||||
banner(
|
||||
"CrystalDiskMark-style storage benchmark",
|
||||
cfg,
|
||||
extra=[
|
||||
f"Engine : {cfg.engine} O_DIRECT: {direct_label}",
|
||||
f"Profile : size={cfg.size.upper()} runtime={cfg.runtime}s/run (8 runs)",
|
||||
],
|
||||
)
|
||||
|
||||
rows: list[list[str]] = []
|
||||
for test in CDM_TESTS:
|
||||
step(f"measuring {test.label.strip()} ...")
|
||||
r = run_fio(cfg, rw=test.read_rw, bs=test.bs,
|
||||
iodepth=test.iodepth, numjobs=test.numjobs)
|
||||
w = run_fio(cfg, rw=test.write_rw, bs=test.bs,
|
||||
iodepth=test.iodepth, numjobs=test.numjobs, end_fsync=True)
|
||||
rows.append([
|
||||
test.label,
|
||||
f"{r.bw_mbps:.2f}", f"{w.bw_mbps:.2f}",
|
||||
f"{r.iops:.0f}", f"{w.iops:.0f}",
|
||||
])
|
||||
|
||||
_cleanup(cfg)
|
||||
|
||||
print()
|
||||
print(render_table(
|
||||
["Test", "Read (MB/s)", "Write (MB/s)", "Read (IOPS)", "Write (IOPS)"],
|
||||
rows,
|
||||
col_styles=[CYAN, GREEN, YELLOW, GREEN, YELLOW],
|
||||
))
|
||||
print()
|
||||
|
||||
|
||||
def slog_profile(cfg: Config) -> None:
|
||||
"""SLOG / sync-write latency profile.
|
||||
|
||||
Forces synchronous 4K random writes (O_SYNC) via the portable psync engine,
|
||||
so every write is a ZIL commit -- exercising a ZFS SLOG exactly the way a
|
||||
sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own
|
||||
sync property. Engine/direct detection is only for the banner here.
|
||||
"""
|
||||
detect_io_settings(cfg)
|
||||
|
||||
banner(
|
||||
"SLOG / sync-write latency profile",
|
||||
cfg,
|
||||
extra=[
|
||||
"Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine",
|
||||
f"Profile : runtime={cfg.runtime}s/run size={cfg.size.upper()}",
|
||||
"Note : every write is a synchronous ZIL commit -- the load your",
|
||||
" SLOG actually sees. Watch it live in another shell with:",
|
||||
" zpool iostat -vl <pool> 1",
|
||||
],
|
||||
)
|
||||
|
||||
# Measure everything first (progress -> stderr), then draw the whole table,
|
||||
# so the "measuring..." lines can't interleave into the middle of it.
|
||||
rows: list[list[str]] = []
|
||||
for threads in SLOG_THREADS:
|
||||
step(f"measuring 4K sync randwrite T{threads} ...")
|
||||
res = run_fio(
|
||||
cfg, rw="randwrite", bs="4k", iodepth=1, numjobs=threads,
|
||||
engine="psync", direct=False, sync=True,
|
||||
)
|
||||
rows.append([
|
||||
f"4K sync T{threads}",
|
||||
f"{res.iops:.0f}", f"{res.bw_mbps:.2f}",
|
||||
f"{res.p50_us:.1f}", f"{res.p99_us:.1f}",
|
||||
])
|
||||
|
||||
_cleanup(cfg)
|
||||
|
||||
print()
|
||||
print(render_table(
|
||||
["Test", "IOPS", "MB/s", "p50 lat(us)", "p99 lat(us)"],
|
||||
rows,
|
||||
col_styles=[CYAN, GREEN, GREEN, YELLOW, MAGENTA],
|
||||
))
|
||||
print()
|
||||
print(OUT.paint(" Healthy Optane SLOG (eg P1600X) single-stream (T1) target:", BOLD))
|
||||
print(OUT.paint(" ~15-25k IOPS, p50 latency ~40-65us.", GREEN)
|
||||
+ OUT.paint(" Much higher latency usually means", DIM))
|
||||
print(OUT.paint(" CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling.", DIM))
|
||||
print()
|
||||
|
||||
|
||||
def dd_profile(cfg: Config) -> None:
|
||||
"""Dependency-free fallback when fio isn't installed.
|
||||
|
||||
Far cruder than fio: a single sequential stream, and the cached-read figure
|
||||
will reflect RAM. Use it only as a rough sanity check.
|
||||
"""
|
||||
count_mib = max(1, parse_size_bytes(cfg.size) // (1024**2))
|
||||
nbytes = count_mib * 1024**2
|
||||
|
||||
banner(
|
||||
"basic dd benchmark (fio not installed)",
|
||||
cfg,
|
||||
extra=[f"Profile : {count_mib} MiB sequential stream"],
|
||||
)
|
||||
|
||||
def timed_dd(args: list[str]) -> float:
|
||||
start = time.monotonic()
|
||||
subprocess.run(args, capture_output=True, text=True)
|
||||
elapsed = time.monotonic() - start
|
||||
return (nbytes / elapsed / 1e6) if elapsed > 0 else 0.0 # decimal MB/s
|
||||
|
||||
step("measuring sequential write ...")
|
||||
write_mbps = timed_dd([
|
||||
"dd", "if=/dev/zero", f"of={cfg.benchfile}",
|
||||
"bs=1M", f"count={count_mib}", "conv=fdatasync,notrunc",
|
||||
])
|
||||
|
||||
step("dropping caches for uncached read ...")
|
||||
subprocess.run(["sudo", "sh", "-c", "echo 3 > /proc/sys/vm/drop_caches"],
|
||||
capture_output=True, text=True)
|
||||
|
||||
step("measuring uncached read ...")
|
||||
uncached_mbps = timed_dd([
|
||||
"dd", f"if={cfg.benchfile}", "of=/dev/null", "bs=1M", f"count={count_mib}",
|
||||
])
|
||||
|
||||
step("measuring cached read ...")
|
||||
cached_mbps = timed_dd([
|
||||
"dd", f"if={cfg.benchfile}", "of=/dev/null", "bs=1M", f"count={count_mib}",
|
||||
])
|
||||
|
||||
_cleanup(cfg)
|
||||
|
||||
print()
|
||||
print(render_table(
|
||||
["Test", "MB/s"],
|
||||
[
|
||||
["Sequential write", f"{write_mbps:.2f}"],
|
||||
["Uncached read", f"{uncached_mbps:.2f}"],
|
||||
["Cached read (RAM)", f"{cached_mbps:.2f}"],
|
||||
],
|
||||
col_styles=[CYAN, GREEN],
|
||||
))
|
||||
print()
|
||||
|
||||
|
||||
def _cleanup(cfg: Config) -> None:
|
||||
"""Remove the shared bench file."""
|
||||
try:
|
||||
os.remove(cfg.benchfile)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# CLI
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(
|
||||
prog="speedtest-hd.py",
|
||||
description="CrystalDiskMark-style storage benchmark built on fio.",
|
||||
epilog=(
|
||||
"Examples:\n"
|
||||
" speedtest-hd.py .\n"
|
||||
" speedtest-hd.py /mnt/nvmepool --runtime=10 --size=4g\n"
|
||||
" speedtest-hd.py /mnt/nfsshare --buffered\n"
|
||||
" speedtest-hd.py /mnt/nvme-ultra-r10/vm-root --slog --runtime=30\n"
|
||||
),
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
p.add_argument("path", help="directory/mount to benchmark ('.' for cwd)")
|
||||
|
||||
mode = p.add_mutually_exclusive_group()
|
||||
mode.add_argument("--fio", action="store_const", const="cdm", dest="mode",
|
||||
help="force the fio CrystalDiskMark-style profile")
|
||||
mode.add_argument("--dd", action="store_const", const="dd", dest="mode",
|
||||
help="force the basic dd fallback test")
|
||||
mode.add_argument("--slog", action="store_const", const="slog", dest="mode",
|
||||
help="SLOG / sync-write latency profile (ZFS ZIL)")
|
||||
|
||||
direct = p.add_mutually_exclusive_group()
|
||||
direct.add_argument("--direct", action="store_const", const=True, dest="direct",
|
||||
help="force O_DIRECT (bypass page cache)")
|
||||
direct.add_argument("--buffered", action="store_const", const=False, dest="direct",
|
||||
help="force buffered IO (e.g. if O_DIRECT unsupported)")
|
||||
|
||||
p.add_argument("--engine", choices=ENGINE_CANDIDATES,
|
||||
help="force a specific IO engine (default: auto-detect)")
|
||||
p.add_argument("--runtime", type=int, default=5, metavar="SEC",
|
||||
help="seconds per run (default: 5, like CrystalDiskMark)")
|
||||
p.add_argument("--size", default="1g", metavar="SIZE",
|
||||
help="test file size (default: 1g)")
|
||||
p.add_argument("--verbose", action="store_true",
|
||||
help="also print the full fio output for every run")
|
||||
p.add_argument("-y", "--yes", action="store_true", dest="assume_yes",
|
||||
help="skip the confirmation prompt")
|
||||
|
||||
p.set_defaults(mode=None, direct=None)
|
||||
return p
|
||||
|
||||
|
||||
def confirm(cfg: Config) -> None:
|
||||
"""Guard prompt -- we're about to write a multi-GB file to the target."""
|
||||
if cfg.assume_yes:
|
||||
return
|
||||
print(OUT.paint("NOTICE:", BOLD, YELLOW)
|
||||
+ f" {cfg.size.upper()} free space on "
|
||||
+ OUT.paint(f"'{cfg.path}'", CYAN)
|
||||
+ " is required to perform the benchmark.")
|
||||
answer = input(f"Are you ready to start a storage benchmark against "
|
||||
f"'{cfg.path}' ? ")
|
||||
if not answer.strip().lower().startswith("y"):
|
||||
print(OUT.paint("Ok, cancelled!", YELLOW))
|
||||
sys.exit(0)
|
||||
print(OUT.paint("Great! Starting benchmark now!", BOLD, GREEN))
|
||||
|
||||
|
||||
def main(argv: Optional[Sequence[str]] = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
|
||||
path = os.getcwd() if args.path == "." else args.path
|
||||
if not os.path.exists(path):
|
||||
print(f"Path {path} does not exist", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
cfg = Config(
|
||||
path=path,
|
||||
mode=args.mode,
|
||||
engine=args.engine,
|
||||
direct=args.direct,
|
||||
runtime=args.runtime,
|
||||
size=args.size,
|
||||
verbose=args.verbose,
|
||||
assume_yes=args.assume_yes,
|
||||
benchfile=os.path.join(path, "speedtest-hd.bench"),
|
||||
)
|
||||
|
||||
have_fio = shutil.which("fio") is not None
|
||||
|
||||
# Resolve the mode: explicit flag wins; otherwise fio if available, else dd.
|
||||
mode = cfg.mode
|
||||
if mode in ("cdm", "slog") and not have_fio:
|
||||
print(ERR.paint("ERROR:", BOLD, RED)
|
||||
+ " --fio/--slog require fio (apt install fio / pacman -S fio).",
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
if mode is None:
|
||||
if have_fio:
|
||||
mode = "cdm"
|
||||
else:
|
||||
print(OUT.paint("\nfio is not installed -- falling back to basic dd test.",
|
||||
YELLOW))
|
||||
print(OUT.paint("Install fio for the full CrystalDiskMark-style benchmark.",
|
||||
DIM))
|
||||
mode = "dd"
|
||||
|
||||
confirm(cfg)
|
||||
|
||||
{"cdm": cdm_profile, "slog": slog_profile, "dd": dd_profile}[mode](cfg)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
print(ERR.paint("\nInterrupted.", YELLOW), file=sys.stderr)
|
||||
sys.exit(130)
|
||||
+307
-302
@@ -1,8 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Robust HD/SDD/NVMe performance CLI utility
|
||||
# Utilizing FIO for sequential/random writes/writes
|
||||
# Dependencies: fio (apt install fio)
|
||||
# Robust HD/SSD/NVMe performance CLI utility
|
||||
# A CrystalDiskMark-style storage benchmark for Linux, built on fio.
|
||||
# Runs the exact same four tests CrystalDiskMark does (SEQ1M Q8T1, SEQ1M Q1T1,
|
||||
# RND4K Q32T1, RND4K Q32T16), each measured for both Read and Write, and prints
|
||||
# the results in a CrystalDiskMark-style table.
|
||||
# Dependencies: fio (apt install fio / pacman -S fio). Falls back to dd if missing.
|
||||
# See: https://cloud.google.com/compute/docs/disks/benchmarking-pd-performance
|
||||
# See: https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/
|
||||
# mReschke 2024-01-18
|
||||
@@ -10,20 +13,23 @@
|
||||
# CLI Parameters
|
||||
path="$1"
|
||||
option=""
|
||||
simple=false
|
||||
engine="" # IO engine; empty = auto-detect (io_uring > libaio > posixaio > sync)
|
||||
direct="" # O_DIRECT; empty = auto-detect; 1 = bypass page cache, 0 = buffered
|
||||
runtime=30 # seconds per test (longer = more accurate, exposes SLC cache exhaustion)
|
||||
runtime=5 # seconds per measurement (CrystalDiskMark default is 5s)
|
||||
size="1g" # test file size (CrystalDiskMark default is 1GiB)
|
||||
verbose=0 # 1 = also print the full fio output for every run
|
||||
for arg in "${@:2}"; do
|
||||
case "$arg" in
|
||||
--simple) simple=true ;;
|
||||
--dd) option="--dd" ;;
|
||||
--fio) option="--fio" ;;
|
||||
--slog) option="--slog" ;;
|
||||
--buffered) direct=0 ;;
|
||||
--direct) direct=1 ;;
|
||||
--verbose) verbose=1 ;;
|
||||
--engine=*) engine="${arg#*=}" ;;
|
||||
--direct=*) direct="${arg#*=}" ;;
|
||||
--runtime=*) runtime="${arg#*=}" ;;
|
||||
--size=*) size="${arg#*=}" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
@@ -47,275 +53,35 @@ function main {
|
||||
fi
|
||||
|
||||
# Must type y or n THEN press enter (which I like better)
|
||||
echo "NOTICE: 1GB free space on '$path' is required to perform the benchmark."
|
||||
echo -n "Are you ready to start a robust IO benchmark against '$path' ?"; read answer
|
||||
echo "NOTICE: ${size^^} free space on '$path' is required to perform the benchmark."
|
||||
echo -n "Are you ready to start a storage benchmark against '$path' ? "; read answer
|
||||
if [ "$answer" != "${answer#[Yy]}" ]; then
|
||||
echo "Great! Starting benchmark now!"
|
||||
echo "------------------------------"
|
||||
else
|
||||
echo "Ok, cancelled!"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Use dd of fio based on param or defaults
|
||||
# Use dd or fio based on param or defaults
|
||||
if [ "$option" == "--dd" ]; then
|
||||
dd_speedtest
|
||||
elif [ "$option" == "--fio" ]; then
|
||||
fio_speedtest
|
||||
cdm_speedtest
|
||||
elif [ "$option" == "--slog" ]; then
|
||||
slog_speedtest
|
||||
elif [ "$option" == "" ]; then
|
||||
# If fio is installed, use it, else use dd
|
||||
[ "$simple" != true ] && echo ""
|
||||
if ! command -v fio &> /dev/null; then
|
||||
echo ""
|
||||
echo "fio is not installed -- falling back to basic dd test."
|
||||
echo "Install fio for the full CrystalDiskMark-style benchmark."
|
||||
dd_speedtest
|
||||
else
|
||||
fio_speedtest
|
||||
cdm_speedtest
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
function print_result {
|
||||
# Print a single fio test result, either as the full "Run status group"
|
||||
# line or, when --simple is given, as a compact aligned "Label value" row.
|
||||
# The simple value is taken from the parenthetical (MB/s) figure that
|
||||
# follows the first bw= value in fio's output.
|
||||
local label="$1"
|
||||
local output="$2"
|
||||
local status
|
||||
status=$(echo " - $output " | /usr/bin/grep -A1 'Run status group' | tail -n1)
|
||||
if [ "$simple" == true ]; then
|
||||
local value
|
||||
# Grab the parenthetical (MB/s) figure after the first bw=, then put a
|
||||
# space between the number and the unit, eg "98.4MB/s" -> "98.4 MB/s".
|
||||
value=$(echo "$status" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | sed -E 's/([0-9.]+)([A-Za-z])/\1 \2/')
|
||||
printf "%-28s%s\n" "$label" "$value"
|
||||
else
|
||||
echo "$status"
|
||||
fi
|
||||
}
|
||||
|
||||
function fio_write_single_random_4k {
|
||||
# 4K Random Writes (parallel, deep queue)
|
||||
|
||||
# Random 4K writes are the worst thing you can ask a disk to do. Where this
|
||||
# happens most in real life: copying home directories and dotfiles, email
|
||||
# stores, some database operations, source code trees.
|
||||
|
||||
# IMPORTANT: the original version of this test used numjobs=1 + iodepth=1.
|
||||
# That measures single-op LATENCY (the absolute worst case): the OS issues
|
||||
# one 4K write, waits for it to be acknowledged, then issues the next. On
|
||||
# NVMe that lands around ~3000 IOPS / ~12MB/sec -- which looks alarmingly
|
||||
# slow but is NOT a bug. NVMe's entire advantage is deep command queues and
|
||||
# parallelism, so a queue depth of 1 deliberately measures the one scenario
|
||||
# NVMe is bad at. To measure the drive's actual capability we now drive it
|
||||
# with multiple jobs and a deep queue (4 jobs x iodepth=64), keeping
|
||||
# hundreds of operations in flight at once. Expect this to jump into the
|
||||
# hundreds of MB/sec (often 1GB/sec+) on a healthy NVMe pool.
|
||||
|
||||
# --name= is a required argument, but it's basically human-friendly fluff—fio will create files based on that name to test with, inside the directory you point it at.
|
||||
# --directory= where the test files are created; this is the path/mount you are benchmarking.
|
||||
# --ioengine=$engine the engine fio uses to talk to the kernel, auto-detected by detect_io_settings(). On modern Linux this is io_uring (lowest overhead); it falls back to libaio, then posixaio, then sync. NOTE: libaio is only truly asynchronous with --direct=1; with buffered IO it silently degrades to synchronous, which is exactly why io_uring is preferred.
|
||||
# --rw=randwrite random write operations. Other options include read, write (sequential), randread, and randrw.
|
||||
# --bs=4k blocksize 4K. These are very small individual operations. This is where the pain lives; it's hard on the disk and adds a ton of command-channel overhead, since a separate operation has to be commanded for each 4K of data.
|
||||
# --size=256m each job's test file is 256MB. With --numjobs=4 that's 4 files totaling ~1GB.
|
||||
# --numjobs=4 run 4 parallel processes, each with its own test file, to feed the drive from multiple threads at once (single-threaded cannot saturate NVMe).
|
||||
# --iodepth=64 how many commands each job keeps stacked in the queue at once. Deep queues are how NVMe reaches its real IOPS; 4 jobs x 64 = up to 256 ops in flight.
|
||||
# --direct=$direct O_DIRECT, auto-detected. 1 bypasses the OS page cache so we measure the device instead of RAM; 0 (buffered) is the fallback for filesystems/mounts that reject O_DIRECT (eg older OpenZFS, some NFS).
|
||||
# --ramp_time=2s discard the first 2 seconds of results so we measure steady state, not the initial warm-up burst.
|
||||
# --runtime=$runtime --time_based run for this many seconds, looping over the file(s) if we finish early.
|
||||
# --group_reporting=1 aggregate all jobs into a single combined result line instead of one line per job.
|
||||
# --end_fsync=1 after the timed run, flush everything to stable storage and count that time, so cached writes can't inflate the number.
|
||||
[ "$simple" != true ] && { echo ""; echo "4K Random Writes (bs=4k, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; }
|
||||
x=`sudo fio \
|
||||
--name=fio-write-random-4k \
|
||||
--directory=$path \
|
||||
--ioengine=$engine \
|
||||
--rw=randwrite \
|
||||
--bs=4k \
|
||||
--size=256m \
|
||||
--numjobs=4 \
|
||||
--iodepth=64 \
|
||||
--direct=$direct \
|
||||
--ramp_time=2s \
|
||||
--time_based --runtime=$runtime \
|
||||
--group_reporting=1 \
|
||||
--end_fsync=1`
|
||||
print_result "4K Random Writes" "$x"
|
||||
|
||||
# Cleanup my test files
|
||||
rm -rf $path/fio-write-random-4k*
|
||||
}
|
||||
|
||||
function fio_write_parallel_random_64k {
|
||||
# Parallel 64k Random Writes
|
||||
|
||||
# This time, we're creating 16 separate 64MB files (still totaling 1GB, when
|
||||
# all put together) and we're issuing 64KB blocksized random write operations.
|
||||
# We're doing it with sixteen separate processes running in parallel, and
|
||||
# we're queuing up to 16 simultaneous asynchronous ops before we pause and wait
|
||||
# for the OS to start acknowledging their receipt.
|
||||
|
||||
# This is a pretty decent approximation of a significantly busy system. It's
|
||||
# not doing any one particularly nasty thing—like running a database engine or
|
||||
# copying tons of dotfiles from a user's home directory—but it is coping with
|
||||
# a bunch of applications doing moderately demanding stuff all at once.
|
||||
|
||||
# This is also a pretty good, slightly pessimistic approximation of a busy,
|
||||
# multi-user system like a NAS, which needs to handle multiple 1MB operations
|
||||
# simultaneously for different users. If several people or processes are trying
|
||||
# to read or write big files (photos, movies, whatever) at once, the OS tries
|
||||
# to feed them all data simultaneously. This pretty quickly devolves down to a
|
||||
# pattern of multiple random small block access. So in addition to "busy desktop
|
||||
# with lots of apps," think "busy fileserver with several people actively using it."
|
||||
|
||||
# You will see a lot more variation in speed as you watch this operation play
|
||||
# out on the console. Unlike the steady trickle you'd get from a queue-depth-1
|
||||
# single-op latency test, this 16-process job can fluctuate wildly—eg between
|
||||
# about 10MiB/sec and 300MiB/sec during the run—as the OS and SSD firmware
|
||||
# catch good and bad luck aggregating writes.
|
||||
|
||||
# Most of the variation you're seeing here is due to the operating system and
|
||||
# SSD firmware sometimes being able to aggregate multiple writes. When it
|
||||
# manages to aggregate them helpfully, it can write them in a way that allows
|
||||
# parallel writes to all the individual physical media stripes inside the SSD.
|
||||
# Sometimes, it still ends up having to give up and write to only a single
|
||||
# physical media stripe at a time—or a garbage collection or other maintenance
|
||||
# operation at the SSD firmware level needs to run briefly in the background,
|
||||
# slowing things down.
|
||||
[ "$simple" != true ] && { echo ""; echo "Parallel 64K Random Writes (bs=64k, jobs=16, iodepth=16, engine=$engine, direct=$direct, ${runtime}s)"; }
|
||||
x=`sudo fio \
|
||||
--name=fio-write-random-64k \
|
||||
--directory=$path \
|
||||
--ioengine=$engine \
|
||||
--rw=randwrite \
|
||||
--bs=64k \
|
||||
--size=64m \
|
||||
--numjobs=16 \
|
||||
--iodepth=16 \
|
||||
--direct=$direct \
|
||||
--ramp_time=2s \
|
||||
--time_based --runtime=$runtime \
|
||||
--group_reporting=1 \
|
||||
--end_fsync=1`
|
||||
print_result "Parallel 64K Random Writes" "$x"
|
||||
|
||||
# Cleanup my test files
|
||||
rm -rf $path/fio-write-random-64k*
|
||||
}
|
||||
|
||||
function fio_write_single_sequential_1m {
|
||||
# Single 1M Sequential Writes
|
||||
|
||||
# This is pretty close to the best-case scenario for a real-world system
|
||||
# doing real-world things. No, it's not quite as fast as a single, truly
|
||||
# contiguous write... but the 1MiB blocksize is large enough that it's quite
|
||||
# close. Besides, if literally any other disk activity is requested simultaneously
|
||||
# with a contiguous write, the "contiguous" write devolves to this level of
|
||||
# performance pretty much instantly, so this is a much more realistic test of
|
||||
# the upper end of storage performance on a typical system.
|
||||
|
||||
# You'll see some kooky fluctuations on SSDs when doing this test. This is largely
|
||||
# due to the SSD's firmware having better luck or worse luck at any given time,
|
||||
# when it's trying to queue operations so that it can write across all physical
|
||||
# media stripes cleanly at once. Rust disks will tend to provide a much more
|
||||
# consistent, though typically lower, throughput across the run.
|
||||
|
||||
# You can also see SSD performance fall off a cliff here if you exhaust an
|
||||
# onboard write cache—TLC and QLC drives tend to have small write cache areas
|
||||
# made of much faster MLC or SLC media. Once those get exhausted, the disk has
|
||||
# to drop to writing directly to the much slower TLC/QLC media where the data
|
||||
# eventually lands. This is the major difference between, for example, Samsung
|
||||
# EVO and Pro SSDs—the EVOs have slow TLC media with a fast MLC cache, where
|
||||
# the Pros use the higher-performance, higher-longevity MLC media throughout
|
||||
# the entire SSD.
|
||||
|
||||
# If you have any doubt at all about a TLC or QLC disk's ability to sustain
|
||||
# heavy writes, you may want to experimentally extend your time duration here.
|
||||
# If you watch the throughput live as the job progresses, you'll see the impact
|
||||
# immediately when you run out of cache—what had been a fairly steady,
|
||||
# several-hundred-MiB/sec throughput will suddenly plummet to half the speed
|
||||
# or less and get considerably less stable as well.
|
||||
|
||||
# However, you might choose to take the opposite position—you might not
|
||||
# expect to do sustained heavy writes very frequently, in which case you
|
||||
# actually are more interested in the on-cache behavior. What's important
|
||||
# here is that you understand both what you want to test, and how to test
|
||||
# it accurately.
|
||||
|
||||
[ "$simple" != true ] && { echo ""; echo "Single 1M Sequential Writes (bs=1m, jobs=1, iodepth=16, engine=$engine, direct=$direct, ${runtime}s)"; }
|
||||
x=`sudo fio \
|
||||
--name=fio-write-seq-1m \
|
||||
--directory=$path \
|
||||
--ioengine=$engine \
|
||||
--rw=write \
|
||||
--bs=1m \
|
||||
--size=1g \
|
||||
--numjobs=1 \
|
||||
--iodepth=16 \
|
||||
--direct=$direct \
|
||||
--ramp_time=2s \
|
||||
--time_based --runtime=$runtime \
|
||||
--group_reporting=1 \
|
||||
--end_fsync=1`
|
||||
print_result "Single 1M Sequential Writes" "$x"
|
||||
|
||||
# Cleanup my test files
|
||||
rm -rf $path/fio-write-seq-1m*
|
||||
}
|
||||
|
||||
function fio_read_sequential_1m {
|
||||
# Sequential Parallel Reads
|
||||
|
||||
[ "$simple" != true ] && { echo ""; echo "Sequential 4x 1M Reads (bs=1m, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; }
|
||||
x=`sudo fio \
|
||||
--name=fio-read-sequential-1m \
|
||||
--directory=$path \
|
||||
--ioengine=$engine \
|
||||
--bs=1M \
|
||||
--numjobs=4 \
|
||||
--size=256M \
|
||||
--time_based --runtime=$runtime \
|
||||
--ramp_time=2s \
|
||||
--direct=$direct \
|
||||
--verify=0 \
|
||||
--iodepth=64 \
|
||||
--rw=read \
|
||||
--group_reporting=1 \
|
||||
--iodepth_batch_submit=64 \
|
||||
--iodepth_batch_complete_max=64`
|
||||
print_result "Sequential 4x 1M Reads" "$x"
|
||||
rm -rf $path/fio-read-sequential-1m*
|
||||
}
|
||||
|
||||
function fio_read_random_4k {
|
||||
# Random 4k Reads (parallel, deep queue)
|
||||
|
||||
# 4K random reads are the highest-IOPS thing a drive does, but a single
|
||||
# submitting thread goes CPU-bound long before an NVMe runs out of capacity,
|
||||
# so it under-reports peak IOPS. We spread the work over 4 jobs x iodepth=64
|
||||
# (256 ops in flight, like the 4K write test) to saturate fast drives, while
|
||||
# still reporting the honest (low) number on a spinning HDD. size=256m per
|
||||
# job keeps the total footprint at ~1GB across the 4 jobs.
|
||||
[ "$simple" != true ] && { echo ""; echo "Random 4k Reads (bs=4k, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; }
|
||||
x=`sudo fio \
|
||||
--name=fio-read-random-4k \
|
||||
--directory=$path \
|
||||
--ioengine=$engine \
|
||||
--rw=randread \
|
||||
--bs=4k \
|
||||
--size=256m \
|
||||
--numjobs=4 \
|
||||
--time_based --runtime=$runtime \
|
||||
--ramp_time=2s \
|
||||
--direct=$direct \
|
||||
--verify=0 \
|
||||
--iodepth=64 \
|
||||
--group_reporting=1 \
|
||||
--iodepth_batch_submit=64 \
|
||||
--iodepth_batch_complete_max=64`
|
||||
print_result "Random 4k Reads" "$x"
|
||||
rm -rf $path/fio-read-random-4k*
|
||||
}
|
||||
|
||||
function fio_probe {
|
||||
# Run a tiny throwaway fio job to see if a given (engine, direct) combo
|
||||
# actually works on this path/filesystem. Returns 0 on success. This is how
|
||||
@@ -353,57 +119,291 @@ function detect_io_settings {
|
||||
if [ -z "$direct" ]; then
|
||||
if fio_probe "$engine" 1; then direct=1; else direct=0; fi
|
||||
fi
|
||||
|
||||
if [ "$simple" != true ]; then
|
||||
echo ""
|
||||
echo "IO engine : $engine"
|
||||
if [ "$direct" == 1 ]; then
|
||||
echo "O_DIRECT : enabled (bypassing page cache -- measuring the device)"
|
||||
else
|
||||
echo "O_DIRECT : DISABLED (buffered) -- this filesystem/mount rejected"
|
||||
echo " O_DIRECT, so results (especially reads) may reflect the"
|
||||
echo " RAM cache rather than the underlying device. Pass"
|
||||
echo " --direct to force it if you believe it should work."
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
function fio_speedtest {
|
||||
function fio_bw_mbps {
|
||||
# Extract bandwidth in decimal MB/s (matching CrystalDiskMark) from a fio
|
||||
# run. fio prints eg " READ: bw=3357MiB/s (3521MB/s), ..." -- we take the
|
||||
# parenthetical SI figure and normalise kB/MB/GB into MB/s.
|
||||
local out="$1"
|
||||
local raw num unit
|
||||
raw=$(echo "$out" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | head -n1)
|
||||
[ -z "$raw" ] && { echo "0.00"; return; }
|
||||
num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1)
|
||||
unit=$(echo "$raw" | /usr/bin/grep -oP '[A-Za-z]+/s' | head -n1)
|
||||
case "$unit" in
|
||||
B/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000000}' ;;
|
||||
kB/s|KB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000}' ;;
|
||||
MB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;;
|
||||
GB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n*1000}' ;;
|
||||
*) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;;
|
||||
esac
|
||||
}
|
||||
|
||||
function fio_iops {
|
||||
# Extract IOPS from a fio run. fio prints eg " read: IOPS=12.3k, BW=..." --
|
||||
# the value may carry a k/M suffix, which we expand to a whole number.
|
||||
local out="$1"
|
||||
local raw num unit
|
||||
raw=$(echo "$out" | /usr/bin/grep -oP 'IOPS=\K[0-9.]+[kKmM]?' | head -n1)
|
||||
[ -z "$raw" ] && { echo "0"; return; }
|
||||
num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1)
|
||||
unit=$(echo "$raw" | /usr/bin/grep -oP '[kKmM]' | head -n1)
|
||||
case "$unit" in
|
||||
k|K) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000}' ;;
|
||||
m|M) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000000}' ;;
|
||||
*) awk -v n="$num" 'BEGIN{printf "%.0f", n}' ;;
|
||||
esac
|
||||
}
|
||||
|
||||
function run_fio {
|
||||
# run_fio <rw> <bs> <iodepth> <numjobs> -> echoes "MB/s IOPS" (space-sep)
|
||||
# A single shared 1GB test file is reused across every run (laid out once),
|
||||
# which keeps the footprint at one file and avoids re-creating it each time.
|
||||
local rw="$1" bs="$2" qd="$3" jobs="$4"
|
||||
local args=(
|
||||
--name=speedtest
|
||||
--filename="$benchfile"
|
||||
--ioengine="$engine"
|
||||
--direct="$direct"
|
||||
--rw="$rw"
|
||||
--bs="$bs"
|
||||
--size="$size"
|
||||
--numjobs="$jobs"
|
||||
--iodepth="$qd"
|
||||
--time_based --runtime="$runtime"
|
||||
--group_reporting=1
|
||||
)
|
||||
# Flush device cache at the end of write tests so cached writes can't lie.
|
||||
case "$rw" in
|
||||
*write) args+=(--end_fsync=1) ;;
|
||||
esac
|
||||
# Capture the full fio output. In verbose mode we also keep fio's stderr
|
||||
# (2>&1); otherwise we discard it. Either way only the parsed bandwidth is
|
||||
# echoed to stdout, so the captured value stays clean.
|
||||
local out
|
||||
if [ "$verbose" = 1 ]; then
|
||||
out=$(sudo fio "${args[@]}" 2>&1)
|
||||
else
|
||||
out=$(sudo fio "${args[@]}" 2>/dev/null)
|
||||
fi
|
||||
# Dump the raw fio output (to stderr, so it never pollutes the captured
|
||||
# values on stdout) when running verbose.
|
||||
if [ "$verbose" = 1 ]; then
|
||||
echo " --- fio: $rw bs=$bs iodepth=$qd numjobs=$jobs ---" >&2
|
||||
echo "$out" >&2
|
||||
echo "" >&2
|
||||
fi
|
||||
echo "$(fio_bw_mbps "$out") $(fio_iops "$out")"
|
||||
}
|
||||
|
||||
# Table drawing -------------------------------------------------------------
|
||||
# Label cell is "| %-16s |" (18 dashes); the four numeric cells are "| %14s |"
|
||||
# (16 dashes each).
|
||||
TBL_SEG_LBL="------------------" # 18 dashes -> matches "| %-16s |"
|
||||
TBL_SEG_NUM="----------------" # 16 dashes -> matches "| %14s |"
|
||||
function tbl_rule {
|
||||
printf "+%s+%s+%s+%s+%s+\n" "$TBL_SEG_LBL" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM"
|
||||
}
|
||||
function tbl_row {
|
||||
# tbl_row <test> <read-mb/s> <write-mb/s> <read-iops> <write-iops>
|
||||
# (col1 left-aligned, the four numeric cols right-aligned)
|
||||
printf "| %-16s | %14s | %14s | %14s | %14s |\n" "$1" "$2" "$3" "$4" "$5"
|
||||
}
|
||||
|
||||
# Five-column table for the --slog latency profile.
|
||||
# "| %-16s |" -> 18 dashes ; "| %12s |" -> 14 dashes
|
||||
SLOG_SEG16="------------------" # 18 dashes
|
||||
SLOG_SEG12="--------------" # 14 dashes
|
||||
function slog_rule {
|
||||
printf "+%s+%s+%s+%s+%s+\n" "$SLOG_SEG16" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12"
|
||||
}
|
||||
function slog_row {
|
||||
# slog_row <test> <iops> <mb/s> <p50> <p99> (col1 left, rest right-aligned)
|
||||
printf "| %-16s | %12s | %12s | %12s | %12s |\n" "$1" "$2" "$3" "$4" "$5"
|
||||
}
|
||||
|
||||
function run_fio_sync {
|
||||
# run_fio_sync <bs> <numjobs>
|
||||
# -> echoes "IOPS MB/s p50_us p99_us mean_us" (one space-separated line)
|
||||
# Forces synchronous IO (--sync=1 => O_SYNC) with the portable psync engine so
|
||||
# every write is a ZIL commit -- i.e. it exercises the SLOG exactly the way a
|
||||
# sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own
|
||||
# sync property. We deliberately DON'T use --group_reporting here: each job is
|
||||
# reported independently in the JSON and we aggregate ourselves (sum IOPS/bw,
|
||||
# average p50, take the worst p99), which sidesteps fio's group-merge quirks.
|
||||
local bs="$1" jobs="$2"
|
||||
local args=(
|
||||
--name=slog
|
||||
--filename="$benchfile"
|
||||
--ioengine=psync
|
||||
--rw=randwrite
|
||||
--bs="$bs"
|
||||
--size="$size"
|
||||
--numjobs="$jobs"
|
||||
--sync=1
|
||||
--time_based --runtime="$runtime"
|
||||
--output-format=json
|
||||
)
|
||||
local out
|
||||
out=$(sudo fio "${args[@]}" 2>/dev/null)
|
||||
if [ "$verbose" = 1 ]; then
|
||||
echo " --- fio json: randwrite bs=$bs numjobs=$jobs sync=1 ---" >&2
|
||||
echo "$out" >&2
|
||||
echo "" >&2
|
||||
fi
|
||||
# Parse the JSON: sum IOPS and bandwidth across jobs, aggregate completion
|
||||
# latency percentiles (clat is in ns -> convert to us). bw_bytes is bytes/s.
|
||||
echo "$out" | python3 -c '
|
||||
import json, sys
|
||||
try:
|
||||
d = json.load(sys.stdin)
|
||||
jobs = d["jobs"]
|
||||
except Exception:
|
||||
print("0 0.00 0.0 0.0 0.0"); sys.exit(0)
|
||||
iops = sum(j["write"]["iops"] for j in jobs)
|
||||
mbps = sum(j["write"]["bw_bytes"] for j in jobs) / 1e6
|
||||
def pctl(j, p):
|
||||
return j["write"]["clat_ns"]["percentile"].get(p)
|
||||
p50 = [pctl(j, "50.000000") for j in jobs if pctl(j, "50.000000") is not None]
|
||||
p99 = [pctl(j, "99.000000") for j in jobs if pctl(j, "99.000000") is not None]
|
||||
mean = [j["write"]["clat_ns"]["mean"] for j in jobs]
|
||||
p50_us = (sum(p50)/len(p50))/1000.0 if p50 else 0.0 # average of per-job medians
|
||||
p99_us = (max(p99))/1000.0 if p99 else 0.0 # worst-case tail across jobs
|
||||
mean_us = (sum(mean)/len(mean))/1000.0 if mean else 0.0
|
||||
print("%.0f %.2f %.1f %.1f %.1f" % (iops, mbps, p50_us, p99_us, mean_us))
|
||||
'
|
||||
}
|
||||
|
||||
function slog_speedtest {
|
||||
# SLOG / sync-write latency profile. Detect engine/direct only for the banner;
|
||||
# the actual runs always force psync + O_SYNC (see run_fio_sync).
|
||||
detect_io_settings
|
||||
benchfile="$path/speedtest-hd.bench"
|
||||
|
||||
if ! command -v fio &> /dev/null; then
|
||||
echo "ERROR: --slog requires fio (apt install fio / pacman -S fio)." >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! command -v python3 &> /dev/null; then
|
||||
echo "ERROR: --slog requires python3 (used to parse fio JSON latency percentiles)." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo " speedtest-hd : SLOG / sync-write latency profile"
|
||||
echo " Target : $path"
|
||||
echo " Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine"
|
||||
echo " Profile : runtime=${runtime}s/run size=${size^^}"
|
||||
echo " Note : every write is a synchronous ZIL commit -- this is the load"
|
||||
echo " your SLOG actually sees. Watch it live in another shell with:"
|
||||
echo " zpool iostat -vl <pool> 1"
|
||||
echo ""
|
||||
|
||||
# T1 is the headline single-stream latency; the sweep shows how the SLOG
|
||||
# scales as concurrent sync writers (NFS/iSCSI/VM threads) pile on. Run every
|
||||
# measurement FIRST (progress goes to stderr), collect the results, then draw
|
||||
# the whole table in one block so the "measuring..." lines can't interleave
|
||||
# into the middle of the table.
|
||||
local jobs_list="1 4 8 16"
|
||||
local -a rows
|
||||
local j res iops bw p50 p99 mean
|
||||
for j in $jobs_list; do
|
||||
echo " measuring 4K sync randwrite T$j ..." >&2
|
||||
res=$(run_fio_sync 4k "$j")
|
||||
read -r iops bw p50 p99 mean <<< "$res"
|
||||
rows+=("$(slog_row "4K sync T$j" "$iops" "$bw" "$p50" "$p99")")
|
||||
done
|
||||
|
||||
rm -f "$benchfile"
|
||||
|
||||
# Render the assembled table
|
||||
echo ""
|
||||
slog_rule
|
||||
slog_row "Test" "IOPS" "MB/s" "p50 lat(us)" "p99 lat(us)"
|
||||
slog_rule
|
||||
printf '%s\n' "${rows[@]}"
|
||||
slog_rule
|
||||
echo ""
|
||||
echo " Healthy Optane SLOG (eg P1600X) single-stream (T1) target:"
|
||||
echo " ~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means"
|
||||
echo " CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling."
|
||||
echo ""
|
||||
}
|
||||
|
||||
function cdm_speedtest {
|
||||
# Auto-detect the best engine and whether O_DIRECT works on this path.
|
||||
detect_io_settings
|
||||
benchfile="$path/speedtest-hd.bench"
|
||||
|
||||
# Write tests
|
||||
fio_write_single_random_4k
|
||||
fio_write_parallel_random_64k
|
||||
fio_write_single_sequential_1m
|
||||
local direct_label="enabled (device)"
|
||||
[ "$direct" != 1 ] && direct_label="DISABLED (buffered -- may reflect RAM cache!)"
|
||||
|
||||
# Read Tests
|
||||
fio_read_sequential_1m
|
||||
fio_read_random_4k
|
||||
# Banner
|
||||
echo ""
|
||||
echo " speedtest-hd : CrystalDiskMark-style storage benchmark"
|
||||
echo " Target : $path"
|
||||
echo " Engine : $engine O_DIRECT: $direct_label"
|
||||
echo " Profile : size=${size^^} runtime=${runtime}s/run (8 runs)"
|
||||
echo ""
|
||||
|
||||
# The four CrystalDiskMark default tests. Q = queue depth (--iodepth),
|
||||
# T = threads (--numjobs). The first three are T1 (single thread); the last
|
||||
# is T16. A single shared test file is reused -- with numjobs>1 every job
|
||||
# issues IO against the same file, which is fine.
|
||||
# Each run now returns both MB/s and IOPS ("bw iops"), so we capture the pair.
|
||||
# label read-rw write-rw bs Q T
|
||||
echo " measuring SEQ1M Q8T1 ..." >&2
|
||||
local s8r_bw s8r_io; read -r s8r_bw s8r_io <<< "$(run_fio read 1m 8 1)"
|
||||
local s8w_bw s8w_io; read -r s8w_bw s8w_io <<< "$(run_fio write 1m 8 1)"
|
||||
echo " measuring SEQ1M Q1T1 ..." >&2
|
||||
local s1r_bw s1r_io; read -r s1r_bw s1r_io <<< "$(run_fio read 1m 1 1)"
|
||||
local s1w_bw s1w_io; read -r s1w_bw s1w_io <<< "$(run_fio write 1m 1 1)"
|
||||
echo " measuring RND4K Q32T1 ..." >&2
|
||||
local r32r_bw r32r_io; read -r r32r_bw r32r_io <<< "$(run_fio randread 4k 32 1)"
|
||||
local r32w_bw r32w_io; read -r r32w_bw r32w_io <<< "$(run_fio randwrite 4k 32 1)"
|
||||
echo " measuring RND4K Q32T16 ..." >&2
|
||||
local r1r_bw r1r_io; read -r r1r_bw r1r_io <<< "$(run_fio randread 4k 32 16)"
|
||||
local r1w_bw r1w_io; read -r r1w_bw r1w_io <<< "$(run_fio randwrite 4k 32 16)"
|
||||
|
||||
# Cleanup the shared test file
|
||||
rm -f "$benchfile"
|
||||
|
||||
# Render the CrystalDiskMark-style table
|
||||
echo ""
|
||||
tbl_rule
|
||||
tbl_row "Test" "Read (MB/s)" "Write (MB/s)" "Read (IOPS)" "Write (IOPS)"
|
||||
tbl_rule
|
||||
tbl_row "SEQ1M Q8T1" "$s8r_bw" "$s8w_bw" "$s8r_io" "$s8w_io"
|
||||
tbl_row "SEQ1M Q1T1" "$s1r_bw" "$s1w_bw" "$s1r_io" "$s1w_io"
|
||||
tbl_row "RND4K Q32T1" "$r32r_bw" "$r32w_bw" "$r32r_io" "$r32w_io"
|
||||
tbl_row "RND4K Q32T16" "$r1r_bw" "$r1w_bw" "$r1r_io" "$r1w_io"
|
||||
tbl_rule
|
||||
echo ""
|
||||
}
|
||||
|
||||
function dd_speedtest {
|
||||
# Basic HD speed test using DD
|
||||
# Basic HD speed test using dd (fallback when fio is not installed)
|
||||
# mReschke 2017-07-11
|
||||
|
||||
local file ddsize
|
||||
file=$path/bigfile
|
||||
size=1024
|
||||
ddsize=1024
|
||||
|
||||
echo "Running dd based HD/SSD/NVMe Benchmarks"
|
||||
echo "---------------------------------------"
|
||||
|
||||
printf "Cached write speed...\n"
|
||||
dd if=/dev/zero of=$file bs=1M count=$size
|
||||
dd if=/dev/zero of=$file bs=1M count=$ddsize
|
||||
|
||||
printf "\nUncached write speed...\n"
|
||||
dd if=/dev/zero of=$file bs=1M count=$size conv=fdatasync,notrunc
|
||||
dd if=/dev/zero of=$file bs=1M count=$ddsize conv=fdatasync,notrunc
|
||||
|
||||
printf "\nUncached read speed...\n"
|
||||
echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
|
||||
dd if=$file of=/dev/null bs=1M count=$size
|
||||
dd if=$file of=/dev/null bs=1M count=$ddsize
|
||||
|
||||
printf "\nCached read speed...\n"
|
||||
dd if=$file of=/dev/null bs=1M count=$size
|
||||
dd if=$file of=/dev/null bs=1M count=$ddsize
|
||||
|
||||
rm $file
|
||||
printf "\nDone\n"
|
||||
@@ -411,40 +411,45 @@ function dd_speedtest {
|
||||
|
||||
# Show help and usage information
|
||||
function usage {
|
||||
echo "Robust Flexible Input/Output HD Speedtest"
|
||||
echo " If FIO is installed, we use FIO for more detailed performance analysis."
|
||||
echo " If FIO is not installed, we use basic DD analysis."
|
||||
echo " You should apt install fio (pacman -S fio) for detailed analysis."
|
||||
echo "speedtest-hd : CrystalDiskMark-style storage benchmark"
|
||||
echo " Runs the same four tests as CrystalDiskMark (each Read + Write):"
|
||||
echo " SEQ1M Q8T1 Sequential 1MiB, queue depth 8, 1 thread"
|
||||
echo " SEQ1M Q1T1 Sequential 1MiB, queue depth 1, 1 thread"
|
||||
echo " RND4K Q32T1 Random 4KiB, queue depth 32, 1 thread"
|
||||
echo " RND4K Q32T16 Random 4KiB, queue depth 32, 16 threads"
|
||||
echo " Uses fio if installed (recommended), else basic dd."
|
||||
echo "mReschke 2024-01-18"
|
||||
echo ""
|
||||
echo "NOTICE, this creates a 1GB file on the desired destination disk."
|
||||
echo "Please ensure you have write access with 1GB free space on destination."
|
||||
echo "NOTICE: creates one test file (default 1GB) on the destination disk."
|
||||
echo "Please ensure you have write access with enough free space."
|
||||
echo ""
|
||||
echo "Usage:"
|
||||
echo " This will use FIO if installed, else DD"
|
||||
echo " Auto (fio if installed, else dd):"
|
||||
echo " ./speedtest-hd /mnt/somedisk"
|
||||
echo " ./speedtest-hd ."
|
||||
echo ""
|
||||
echo " This will force FIO"
|
||||
echo " Force fio / force dd:"
|
||||
echo " ./speedtest-hd /mnt/somedisk --fio"
|
||||
echo " ./speedtest-hd . --fio"
|
||||
echo ""
|
||||
echo " This will force DD"
|
||||
echo " ./speedtest-hd /mnt/somedisk --dd"
|
||||
echo " ./speedtest-hd . --dd"
|
||||
echo ""
|
||||
echo " Add --simple (FIO only) for a compact, aligned summary of MB/s values"
|
||||
echo " ./speedtest-hd . --simple"
|
||||
echo " ./speedtest-hd . --fio --simple"
|
||||
echo " SLOG / sync-write latency profile (ZFS ZIL, NFS/iSCSI/VM sync IO):"
|
||||
echo " ./speedtest-hd /mnt/zpool/dataset --slog"
|
||||
echo " 4K synchronous writes at T1/T4/T8/T16; reports IOPS, MB/s, p50/p99"
|
||||
echo " commit latency. Requires fio + python3. (Tip: watch the SLOG with"
|
||||
echo " 'zpool iostat -vl <pool> 1' in another shell.)"
|
||||
echo ""
|
||||
echo " FIO tuning flags (all auto-detected by default, override as needed):"
|
||||
echo " Tuning flags (auto-detected by default, override as needed):"
|
||||
echo " --engine=io_uring|libaio|posixaio|sync IO engine (default: auto)"
|
||||
echo " --direct Force O_DIRECT (bypass page cache)"
|
||||
echo " --buffered Force buffered IO (eg if O_DIRECT is unsupported)"
|
||||
echo " --runtime=SEC Seconds per test (default: 30)"
|
||||
echo " --runtime=SEC Seconds per run (default: 5, like CrystalDiskMark)"
|
||||
echo " --size=SIZE Test file size (default: 1g)"
|
||||
echo " --verbose Print full fio output for every run (summary table unchanged)"
|
||||
echo " Examples:"
|
||||
echo " ./speedtest-hd /mnt/nvmepool --engine=io_uring --runtime=60"
|
||||
echo " ./speedtest-hd /mnt/nvmepool --runtime=10 --size=4g"
|
||||
echo " ./speedtest-hd /mnt/nfsshare --buffered"
|
||||
echo " ./speedtest-hd /mnt/nvmepool --verbose"
|
||||
echo " ./speedtest-hd /mnt/nvme-ultra-r10/vm-root --slog --runtime=30"
|
||||
exit 0
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user