From 5561a95a947e58a6d01681ddbb4655cd01d0b058 Mon Sep 17 00:00:00 2001 From: Matthew Reschke Date: Sun, 21 Jun 2026 21:05:21 -0600 Subject: [PATCH] New python speedtest to replace bash. New CLAUDE.md --- CLAUDE.md | 65 ++ README.md | 291 +++++++++ __pycache__/speedtest-hd.cpython-314.pyc | Bin 0 -> 39465 bytes speedtest-hd.py | 727 +++++++++++++++++++++++ speedtest-hd.sh | 609 +++++++++---------- 5 files changed, 1390 insertions(+), 302 deletions(-) create mode 100644 CLAUDE.md create mode 100644 README.md create mode 100644 __pycache__/speedtest-hd.cpython-314.pyc create mode 100755 speedtest-hd.py diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..377139a --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,65 @@ +# speedtest-hd + +Single-file Bash CLI (`speedtest-hd.sh`) that benchmarks any mounted path (HDD/SSD/NVMe, +local or NFS/ZFS) using `fio`, presented as a CrystalDiskMark-style ASCII table. Falls back +to `dd` when fio is absent. Goal: a quick, generic, "point it at a mount" disk tester — not +a ZFS/NVMe-specific tool. + +## Usage +``` +./speedtest-hd.sh [--fio|--dd|--slog] [flags] +``` +- Default: fio if installed, else dd. `--fio` forces CDM mode, `--slog` forces sync-latency mode. +- Flags: `--engine=io_uring|libaio|posixaio|sync` (default auto), `--direct`/`--buffered` + (default auto-probe), `--runtime=SEC` (default 5), `--size=SIZE` (default 1g), `--verbose` + (also dumps raw fio output to stderr). + +## Modes & layout +- **CDM mode** (`cdm_speedtest`): runs CrystalDiskMark's default tests, each read+write, + prints a 5-col table (Read/Write MB/s, Read/Write IOPS). Tests: `SEQ1M Q8T1`, `SEQ1M Q1T1`, + `RND4K Q32T1`, `RND4K Q32T16`. Q = `--iodepth`, T = `--numjobs`. +- **SLOG mode** (`slog_speedtest`): 4K synchronous randwrite sweep at T1/T4/T8/T16 to profile + ZFS ZIL / SLOG (and any NFS/iSCSI/VM sync workload). Reports IOPS, MB/s, p50/p99 commit + latency. Requires `fio` + `python3` (JSON percentile parsing). +- **dd mode** (`dd_speedtest`): legacy cached/uncached read/write fallback. + +## Key implementation notes +- `detect_io_settings` auto-picks the engine (io_uring → libaio → posixaio → sync) and probes + whether O_DIRECT works, via tiny throwaway `fio_probe` jobs. Falls back to buffered with a + warning if O_DIRECT is rejected (older OpenZFS <2.3, some NFS). **libaio is only truly async + with `--direct=1`** — that's why io_uring is preferred. O_DIRECT bypasses the page cache so we + measure the device, not RAM (buffered results, esp. reads, can reflect ARC/page cache). +- One shared test file `/speedtest-hd.bench` is reused across all runs (laid out once), + removed at the end. Footprint = `--size` (default 1G), matching the startup notice. +- `run_fio` (CDM) returns `"MB/s IOPS"`; `run_fio_sync` (SLOG) forces `--ioengine=psync --sync=1` + (O_SYNC) so every write is a ZIL commit regardless of dataset sync property, and parses fio + **JSON** to aggregate across jobs (sum IOPS/bw, avg p50, worst-case p99). CDM mode parses + fio's text output: `fio_bw_mbps` takes the parenthetical **SI MB/s** (matches CDM's decimal + MB/s, normalizes kB/MB/GB); `fio_iops` expands k/M suffixes. +- Write tests append `--end_fsync=1` so cached writes can't inflate numbers. +- ASCII tables: cell formats and dash-segment widths must stay in sync (`tbl_*` = 18/16 dashes, + `slog_*` = 18/14). Verify alignment after editing. + +## History / decisions +- Originated from the Ars Technica fio guide. Original 4K test used `numjobs=1 iodepth=1`, which + measures single-op **latency**, not throughput — ~12 MB/s on fast NVMe is correct for QD1, not + a bug. Refactored toward parallel/deep-queue tests to show real device capability, then fully + reshaped into the CrystalDiskMark profile above. `--simple` flag was removed; the table is now + the only fio output. +- Note: the current `RND4K` rows are `Q32T1` + `Q32T16`. CrystalDiskMark's actual latest default + profile is `Q32T16` + `Q1T1` (single-queue random is the meaningful low-end number). If aligning + strictly to CDM, the `Q32T1` row should become `Q1T1` (`iodepth=1 numjobs=1`). + +## SLOG performance context (why `--slog` exists) +Built to diagnose TrueNAS SCALE box `linvault1` (Dell R630, Xeon E5-2680 v3; pool `nvme-ultra-r10` += 6× KingSpec XG7000 RAID10 + Intel Optane P1600X SLOG; dataset `vm-root` sync=always). Poor sync +writes were **CPU power management**, not the SLOG: +- Fix (biggest last): Dell BIOS profile DAPC → **Performance** (~2×); cstate kernel args; and the + big one — CPU **governor `performance`** (was `intel_cpufreq`+`schedutil`, which parked cores at + 1.2 GHz because QD1 sync load blocks on the SLOG and reads as "idle"). Persist via TrueNAS Post + Init: `echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`. +- Result: 4K sync T1 ~3,050 → 10,687 IOPS, p50 ~328 → 85µs (≈ Haswell ZIL-commit floor). +- Diagnostic: `zpool iostat -vl 1` during fio showed the Optane `logs` vdev at ~90µs + disk_wait — proving the SLOG was fine and latency was upstream (CPU). +- Healthy Optane SLOG single-stream (T1) target: ~15–25k IOPS, p50 ~40–65µs. Much higher usually + = C-states / PCIe ASPM / BIOS power profile throttling. diff --git a/README.md b/README.md new file mode 100644 index 0000000..8192c30 --- /dev/null +++ b/README.md @@ -0,0 +1,291 @@ +# speedtest-hd + +A robust, CrystalDiskMark‑style storage benchmark for Linux, built on [`fio`](https://fio.readthedocs.io/) by mReschke and my buddy Claude Opus. + +It runs the same four tests CrystalDiskMark does, plus a dedicated **SLOG / sync‑write latency profile** for diagnosing ZFS ZIL performance (NFS / iSCSI / VM sync workloads). It auto‑detects the best IO engine and whether `O_DIRECT` works on the target, and falls back to a basic `dd` test when `fio` isn't installed. + +--- + +## Table of contents + +- [Features](#features) +- [Requirements](#requirements) +- [Usage](#usage) +- [Output](#output) +- [Understanding the tests](#understanding-the-tests) +- [Case study: diagnosing a "slow" Optane SLOG on TrueNAS](#case-study-diagnosing-a-slow-optane-slog-on-truenas) +- [Notes & caveats](#notes--caveats) + +--- + +## Features + +- **CrystalDiskMark‑style profile** — `SEQ1M Q8T1`, `SEQ1M Q1T1`, `RND4K Q32T1`, `RND4K Q32T16`, each measured for Read **and** Write, reported in both **MB/s** and **IOPS**. +- **SLOG / sync‑write latency profile** (`--slog`) — synchronous 4K writes at T1/T4/T8/T16 reporting **IOPS, MB/s, and p50/p99 commit latency**. This is the load a ZFS SLOG actually sees. +- **Auto‑detection** — picks the fastest available IO engine (`io_uring` → `libaio` → `posixaio` → `sync`) and probes whether `O_DIRECT` works on the filesystem, falling back to buffered IO when it doesn't (e.g. older OpenZFS, some NFS mounts). +- **`dd` fallback** — if `fio` isn't present, runs a basic write/read test so you still get a number. +- **Verbose mode** — `--verbose` dumps the full raw `fio` output for every run while keeping the summary table intact. + +--- + +## Requirements + +- `bash` +- [`fio`](https://fio.readthedocs.io/) — recommended (`apt install fio` / `pacman -S fio`). Without it, the tool falls back to `dd`. +- `python3` — **only** required for `--slog` (used to parse `fio`'s JSON output for latency percentiles). The normal profile needs only `fio` + `grep`/`awk`. +- `sudo` — `fio` is invoked via `sudo` so it can use `O_DIRECT` and flush device caches. + +--- + +## Usage + +```bash +./speedtest-hd.sh [options] +``` + +`` is the directory (or mount) to benchmark. Use `.` for the current directory. The tool creates a single test file (default 1 GiB) on the target and removes it afterward, so ensure enough free space. + +### Modes + +| Invocation | What it does | +|---|---| +| `./speedtest-hd.sh /mnt/disk` | Auto: uses `fio` if installed, else `dd` | +| `./speedtest-hd.sh /mnt/disk --fio` | Force the `fio` CrystalDiskMark‑style profile | +| `./speedtest-hd.sh /mnt/disk --dd` | Force the basic `dd` test | +| `./speedtest-hd.sh /mnt/disk --slog` | SLOG / sync‑write latency profile | + +### Tuning flags + +| Flag | Effect | +|---|---| +| `--engine=io_uring\|libaio\|posixaio\|sync` | Force a specific IO engine (default: auto) | +| `--direct` | Force `O_DIRECT` (bypass page cache) | +| `--buffered` | Force buffered IO (e.g. when `O_DIRECT` is unsupported) | +| `--runtime=SEC` | Seconds per run (default: 5, like CrystalDiskMark) | +| `--size=SIZE` | Test file size (default: `1g`) | +| `--verbose` | Also print the full `fio` output for every run (summary table unchanged) | + +### Examples + +```bash +# CrystalDiskMark-style test of the current directory +./speedtest-hd.sh . + +# Larger file, longer runs, on an NVMe pool +./speedtest-hd.sh /mnt/nvmepool --runtime=10 --size=4g + +# Buffered (e.g. an NFS share that doesn't support O_DIRECT) +./speedtest-hd.sh /mnt/nfsshare --buffered + +# SLOG / sync latency profile, 30s per run +./speedtest-hd.sh /mnt/nvme-ultra-r10/vm-root --slog --runtime=30 +``` + +> **Tip:** when running `--slog` against a ZFS dataset, watch the SLOG live in another shell: +> ```bash +> zpool iostat -vl 1 +> ``` + +--- + +## Output + +### CrystalDiskMark‑style profile + +``` ++------------------+----------------+----------------+----------------+----------------+ +| Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) | ++------------------+----------------+----------------+----------------+----------------+ +| SEQ1M Q8T1 | 6873.00 | 9.30 | 6873 | 9 | +| SEQ1M Q1T1 | 1608.00 | 20.00 | 1608 | 20 | +| RND4K Q32T1 | 538.00 | 10.80 | 137728 | 2764 | +| RND4K Q32T16 | 689.00 | 261.00 | 176384 | 66816 | ++------------------+----------------+----------------+----------------+----------------+ +``` + +### SLOG / sync‑write latency profile (`--slog`) + +``` ++------------------+--------------+--------------+--------------+--------------+ +| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) | ++------------------+--------------+--------------+--------------+--------------+ +| 4K sync T1 | 10687 | 43.77 | 85.5 | 185.3 | +| 4K sync T4 | 29873 | 122.36 | 117.8 | 317.4 | +| 4K sync T8 | 52612 | 215.50 | 136.2 | 391.2 | +| 4K sync T16 | 77939 | 319.24 | 180.0 | 505.9 | ++------------------+--------------+--------------+--------------+--------------+ +``` + +--- + +## Understanding the tests + +### The CrystalDiskMark profile + +| Test | Pattern | Queue depth | Threads | +|---|---|---|---| +| `SEQ1M Q8T1` | Sequential 1 MiB | 8 | 1 | +| `SEQ1M Q1T1` | Sequential 1 MiB | 1 | 1 | +| `RND4K Q32T1` | Random 4 KiB | 32 | 1 | +| `RND4K Q32T16` | Random 4 KiB | 32 | 16 | + +`Q` = queue depth (`--iodepth`), `T` = threads (`--numjobs`). Note that `--iodepth` only produces real queue depth when the IO is truly asynchronous (async engine **and** `O_DIRECT`). On filesystems where that isn't available, queue depth effectively collapses toward 1 and concurrency comes only from threads (`T`). + +### The SLOG profile + +`--slog` forces **synchronous** 4K random writes (`--sync=1` → `O_SYNC`) via the portable `psync` engine. Every write becomes a durable commit, so it traverses the ZFS ZIL / SLOG commit path exactly the way a `sync=always` dataset (NFS, iSCSI, VM storage) does — regardless of the dataset's own `sync` property. It sweeps thread counts (T1 → T4 → T8 → T16): + +- **T1** is the headline single‑stream latency (e.g. one database committing in a tight loop). +- **The sweep** shows how the SLOG scales as concurrent sync writers pile on (multiple VMs / NFS clients) — usually the more important number for a virtualization host. + +A healthy Optane SLOG (e.g. P1600X) single‑stream target is roughly **15–25k IOPS, p50 ~40–65 µs**. Much higher latency usually points at **CPU C‑states / PCIe ASPM / a BIOS power profile** throttling the host — see the case study below. + +--- + +## Case study: diagnosing a "slow" Optane SLOG on TrueNAS + +A real investigation that this tool's `--slog` mode was built to support. **Spoiler: the Optane SSD was healthy the entire time. The bottleneck was CPU power management.** + +### The setup + +| Component | Detail | +|---|---| +| Server | Dell PowerEdge R630 | +| CPU | Intel Xeon E5‑2680 v3 (Haswell‑EP, 12C/24T, 2.5 GHz base, 3.3 GHz turbo) | +| OS | TrueNAS SCALE 25.10 (OpenZFS 2.3) | +| Pool | `nvme-ultra-r10` — 6× 4 TB KingSpec XG7000 NVMe in RAID10 (3 mirror vdevs) | +| SLOG | Intel Optane **P1600X** | +| Dataset | `/mnt/nvme-ultra-r10/vm-root`, `sync=always` | + +### The symptom + +The standard benchmark looked alarming — huge reads, tiny writes: + +``` ++------------------+------------------+------------------+ +| Test | Read (MB/s) | Write (MB/s) | ++------------------+------------------+------------------+ +| SEQ1M Q8T1 | 6873.00 | 9.30 | +| SEQ1M Q1T1 | 1608.00 | 20.00 | +| RND4K Q32T1 | 538.00 | 10.80 | +| RND4K Q32T16 | 689.00 | 261.00 | ++------------------+------------------+------------------+ +``` + +9.3 MB/s sequential write on an Optane‑backed NVMe pool looks broken. + +### Investigation + +**1. The reads are RAM, not disk.** With a 1 GiB test file on ZFS, reads come straight from ARC (RAM cache). The huge read/write asymmetry is the tell — ignore the read column for judging the disks. + +**2. `sync=always` makes writes latency‑bound.** Every write must be durably committed to the ZIL before it's acknowledged, so throughput ≈ (block size) ÷ (per‑commit latency). Anything running at low effective concurrency looks slow regardless of raw device speed. + +**3. The `Q8`/`Q32` labels were misleading.** On this ZFS setup `--iodepth` didn't produce real queue depth, so most rows effectively ran at QD1. Proof: `RND4K Q32T1` (10.8 MB/s) vs `RND4K Q32T16` (261 MB/s) — the 24× jump came entirely from threads (`numjobs=16`), not queue depth. + +**4. Large sequential sync writes bypass the SLOG.** With ZFS's default `logbias=latency`, writes larger than `zfs_immediate_write_sz` (32 KB) use an *indirect* ZIL record — the data goes straight to the main pool and only a pointer hits the Optane. So the `SEQ1M` write test was measuring the **consumer KingSpec pool's** forced‑sync performance, not the SLOG. Only small (4K) sync writes exercise the Optane. + +**5. Confirm with `zpool iostat -vl 1` during a 4K sync test.** This was decisive: + +- The `logs` (Optane) vdev took **all** the sync writes (~2.3–2.6k ops, ~18–20 MB/s); the data vdevs were idle between txg flushes. → **SLOG configured correctly, `logbias` fine, not bypassed.** +- But the Optane's own `disk_wait` was **~90 µs** (a P1600X should be ~10–15 µs), and the fio‑level commit latency was **~328 µs** — meaning ~238 µs was being spent *above* the device, in the host/ZFS/CPU path. + +That "faster when busy, slow when idle" device latency plus huge host overhead is the classic signature of **power‑saving idle states** on a latency‑bound, QD1 workload. + +**6. Find the throttle.** Checking the CPU revealed the cores pinned at **1.2 GHz** — the E5‑2680 v3's *minimum* P‑state — on a chip rated for 2.5–3.3 GHz: + +``` +$ grep MHz /proc/cpuinfo | sort -u +cpu MHz : 1200.069 +... + +$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver +intel_cpufreq # = intel_pstate in passive mode +$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor +schedutil # picks frequency from CPU utilization +``` + +The vicious loop: a QD1 sync workload spends each commit **blocked** waiting on the SLOG → the `schedutil` governor sees near‑zero utilization → parks the cores at 1.2 GHz → the ZFS commit code path runs ~2–3× slower → latency climbs. + +### Root cause + +**CPU power management, in two layers — not the SSD, pool, or PCIe link:** + +1. **BIOS System Profile = "Performance Per Watt Optimized (DAPC)"** — Dell Active Power Controller manages C‑states/P‑states in firmware and largely ignores the OS, keeping cores idling deep and clocked low. +2. **OS `schedutil` governor** (TrueNAS SCALE default) — pinned cores at the 1.2 GHz floor for this bursty, IO‑blocked workload. + +### The fixes + +Applied in order, biggest impact last: + +**1. BIOS System Profile → Performance** (disables C‑states/C1E, raises P‑states): +``` +# In BIOS (F2): System BIOS → System Profile Settings → System Profile → Performance +# Or via iDRAC: +racadm set BIOS.SysProfileSettings.SysProfile PerfOptimized +racadm jobqueue create BIOS.Setup.1-1 +# reboot to apply +``` + +**2. Kernel parameters** (target PCIe/NVMe link power saving + residual C‑states). On TrueNAS: *System → Advanced Settings → Kernel Arguments*, then reboot: +``` +intel_idle.max_cstate=1 processor.max_cstate=1 pcie_aspm=off nvme_core.default_ps_max_latency_us=0 +``` + +**3. CPU governor → `performance`** *(the single biggest win)*: +```bash +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +``` +Make it persistent on TrueNAS — *System → Advanced Settings → Init/Shutdown Scripts*, add a **Post Init** *Command*: +``` +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +``` +> ⚠️ Without the Post Init script the governor reverts to `schedutil` on every reboot, silently dropping you back to slow numbers. + +### Results — before & after + +**4K synchronous random writes** (`--slog`), per stage of the fix: + +| Stage | T1 IOPS | T1 p50 | T1 MB/s | T16 IOPS | T16 MB/s | T16 p99 | +|---|---|---|---|---|---|---| +| **DAPC** (start) | ~3,050 | ~328 µs | ~12.5 | — | 38 *(regressing)* | thrashing | +| BIOS → Performance | 5,849 | 160.8 µs | 23.96 | 46,009 | 188.45 | 1057 µs | +| + kernel parameters | 6,217 | 150.5 µs | 25.46 | 47,166 | 193.19 | 684 µs | +| **+ `performance` governor** | **10,687** | **85.5 µs** | **43.77** | **77,939** | **319.24** | **506 µs** | + +**Full final result:** + +``` ++------------------+--------------+--------------+--------------+--------------+ +| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) | ++------------------+--------------+--------------+--------------+--------------+ +| 4K sync T1 | 10687 | 43.77 | 85.5 | 185.3 | +| 4K sync T4 | 29873 | 122.36 | 117.8 | 317.4 | +| 4K sync T8 | 52612 | 215.50 | 136.2 | 391.2 | +| 4K sync T16 | 77939 | 319.24 | 180.0 | 505.9 | ++------------------+--------------+--------------+--------------+--------------+ +``` + +**Net improvement:** ~**3.5× IOPS**, ~**3.8× lower latency** at T1, and ~**8.4× aggregate throughput** at T16 — with the scaling regression eliminated entirely. + +### Lessons learned + +- **An Optane SLOG showing high latency is usually a host‑side power‑management problem, not the device.** Confirm where the time goes before blaming hardware. +- **`zpool iostat -vl 1` is the key diagnostic** — it shows whether the `logs` vdev is actually taking the writes and splits device latency (`disk_wait`) from host/ZFS overhead (`total_wait`). +- **Latency‑bound QD1 sync workloads are the worst case for power saving.** The CPU looks idle (blocked on IO), so governors and firmware clock it down — which directly inflates the latency you're trying to measure. +- **On TrueNAS SCALE, the default `schedutil` governor cripples sync‑write latency.** Set `performance` (and persist it). +- **Reads from a small test file measure ARC (RAM), not the disk.** Watch the read/write asymmetry. +- **Large sync writes bypass the SLOG** (indirect ZIL) — to actually test a SLOG, use small (4K) sync writes, which is exactly what `--slog` does. + +### `~85 µs` is roughly the floor here + +The residual gap from raw Optane (~15 µs) is ZFS ZIL‑commit overhead plus the per‑op cost of a 2014‑era Haswell core. Closing it further would need a newer/faster CPU for sharply diminishing returns. For a virtualization host the aggregate (78k IOPS / 319 MB/s) is what the workload feels, and it's healthy. + +--- + +## Notes & caveats + +- **`sudo`** is used for `fio` so it can apply `O_DIRECT` and flush device write caches at the end of write runs (`--end_fsync=1`), so cached writes can't inflate results. +- **`O_DIRECT` is auto‑detected.** If the banner shows `O_DIRECT: DISABLED (buffered ...)`, results may reflect the page cache (RAM) rather than the device. +- **A single shared test file** is reused across runs to keep the footprint to one file. +- **`--slog` requires `python3`** (for JSON latency‑percentile parsing); the standard profile does not. +- The `--slog` profile forces synchronous IO and is intended for ZFS ZIL / SLOG and other sync‑write (NFS/iSCSI/VM) investigations. diff --git a/__pycache__/speedtest-hd.cpython-314.pyc b/__pycache__/speedtest-hd.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..0e52c8e41256a2a82f6461ad13c3254f1c220e4d GIT binary patch literal 39465 zcmch=3shTInkITA9la&Q8;r3z*w`Qp!Wh4w!GLXIgUNy5N`j4rBtS7BawKe{RAn=I6W1{xDNq1-N+`4zxy0~rM zRAtTWx!=FfOJK!zWqS1i?X%B5`|S7r|9}7gzyG}<&+XuFZTnZ-pB)(DxWA(Z^>WG{ z!<|B&o+HJr%B3@13z&}lqmikYf7_>G->jAwUq%*^hVn1$V~F)O>^En2tYjNdvWELg z)*v<_+{aqCQEY;LBlG*jX81QTf0Nh(|7PajD4rCz;Aty8jTvIiIa^^HVr;)JuTA0( zgzseOw1~Ul-_87+#XX`?+>2ic>a*TjpD;cY9+!x-Bl?#ELbi@Q~9hrfgQw}}Vf zKgj&sMZb6mPo3-=Lu`lmIGzr(cRR%+@OLr)F0mW_qs+fs?1BFY=HDZV@b@zRUa=4U zCz*erI3)Jt=@>nszKDVY?y)WNWSWFis`PmBmp zALtdH?mQw)h9tZh_fG^uLR~ikn}l$0TW=@s{aq|Y;Y>t2H5Ty)qh3}~DHKwZjXD~a zT#kAaCJ-F2uNPRGQe}oCQ_}LLMd642Q)4lKIv^_4ojlpw@kDc%@Wl4M=94EI&|2)- zAqY=2t07`{`{u*K6D^x^1Z+Kd(%T?}{DD!mL4&N9q>vxYgGQ@IQmkPb1SvEbk*L|i z6GGSiCNw45wDV|BuaA{JIu;p0@o7X*bB;=piO5t`K-Ll>M#kl2&|hallQF>`kRnm) zb(T{gGBpuHFUeV;m60o@+@NPPxn*P1E5rV=u_1rplyGJ=G=WCM&^&n(qjI7)hT78D9t#E0 zhsQ>zY1j&n_a5!ux3mi#9j8OmwAzZ}p$XJ6dMe7I(+E0wvavBT6`P!jHL_t7+w*KR zGJ#yE6({{t6hj%+IunkK(p+#L(}|#83JPQ4A;~XIQ$78$T6v%0%!?IEWmZSaaj14gZi7 zKpijxqH>ugBEm4HGXfm3(Ak&}4M_e;q=Q!JMGJ}@t?gYMzHxc@u;Dv3HYNo8Q33>0 z!D*o`YYzE^H5e{u)(A%@QO%ItS25Jj5u1`GFvQSOH2H@@LR-(1!er!3ND?sB08Xf7 z8(0M?lORA2CYRTcar!4vYkw>pnTU3KjTw8;AM*#s{LyH}JRA;<1-+(>tqmRN9~ui~ zY{(cf{bLzhZ|J!xjD%2}U-`_&adMlPyNS6i<8kxKXIc#K&W0p_Z)#;6CUqW!ZXA5b0>`3feL~MtK^}?Jew8+CIN=dHCc$PrL$?1%1a8PN1(BPotL8QI-MN8q#a9?mPLx@d-w;*MG+Sgs&pvagYv=_^v> zzgB8yH8s(~a58IpRx+wLy9Vs4ub&gGqHQzc(4~I4%b?Y(K*q-XT#rIE%Q0nF#pUQu zd;Hz@P#_${ybDP{jF_{4grgx0UWVR61j|~xfi?n>Ai*lfA;3*Zp60<&&x(c9b#984^E8_Wh~*SKNg$r_F6K= zNHk*&O`Hw`>}O0Pp_oMNlCi>vD3V0ukOr}wHygz3r96s~7oD0U#1b83g_UTAu%Y`b zJTu%~7iV+48hI*Nvka+f{wfZA#^(UU%bab#CY|4uc5b}sY)?7cuk`=?_|K02yTKISF6AS;0Djt~40^+zO$eId zj$;`laxvqHu9!(Q#CXvNQqRN)KeNcgZ)ScAmMd1#imtb1tUZ3#yz#bU0G8-e0-V5} zy$40PyFmv@0OK^%8T1Dhj7$Ya0RR~yVhuvXQDk`7_jTckM1z|8h5%oXb&KRekk=r& zag*T*>k0-nGQ~=vec1q|hhi4NAy{HCSKTIOj%PQy<+4Ep`7w3ucM91T$MPp|moJf9 z%e`&re%l}k2mqYsqoJ{3_7J7d)+?jJHaHjt${HNZa`c<_#2*@ zx2A4*8dA0f);bb(S~q47hQNe>9M~-57#tjr1gFNx4?OqWlz&Wl;~5+r4olJ4Sa>2d z5y5Ng;9xKk00yMhth5nMretss#8fOC7$is(9-0E}j0iLbH7YNyH{P+SaLP`MeysW6Ae%oxTqhLMaR*2gfl6p4eRKrHQ2I>&G)j-V`H zG8Qv|!@|JHakFS*D{>=zJoB01vuHjm^Vtw)C*N_4=m6Mp64+5OPRxo>7ed`jLT%uY z*hLRII6q@;3y$|;^$hJSolF9#lrqE;dE4`<|omZ zA%HEPM17nwgRTmVWsF17jETmAH4Jn%85_-5C#J@qjSNL)d|*OaUk{MIZDLbYo{*H~ zgJxx>SgGU^Aa({{WdX5G1idyTi`Y$`m;^-Bi~Q&{`yC%VM6x|_Vj^ZGCLtywCL@70 zTGwQ3wDch#;9Uoj!HwooHu|&D>p|&E#-kQl2{=s=OIY5YtyuMD`J3g*^4ZfVTk~8T zj$U51%nHzbIEg78M4}IBJ&9Nc&r|kAEPI0Zo2ivHC)cu@o;lvL>wm7X%xs03u~Ta? zuXF^dN^B-l$|_D^@2IWv)wWVYXjZ9>Rk8Zbx;N{Rb+czuww8G(9C_B#gzxq`Btqv> z3pq4vq%GuZg_8j=$y!7JnApGs0NECsmbxiYA358|Sq3}yQ5ZR6_(j{{yvqFz-}9L> z@8!oAY|fVtF1YhwJ|a7I_sgB1nca7d7<+e2oF#9Oe2W(NC~PweV?Mdmlx*B7ukzV4 zKVxhj_wrIJ!lgEHC_gs3^k`W7$6fNeUtaiUjGIp#h`Y0cw#JKMQFzlHcH?xG%*5<0tiT}>60geVJQ@d z;c?tQ4f5G9#e}H|#AmNT;yS=80&zvO-O#kKb9ibZu=8Xv6vIb@;mDwx@<|0)@&KNr zU%J&b=rf=xz3ldtV8#7=z~B>rzb62Py@vR1uxh|CX>63+PKfzWft#_hsd<;qMvDj= z(UstHAc(pPYbY8FQ6E4xjqv+`kW)@&Eqmvn(8PiqA}-r6HYM!$$kNKvAMY0+ae90Yrb+ys_A zbWhgE&w5!9m0biuSYOa+&F%})kkuEt*h|I%>PL1B*i@g&q;a8l9{fG}nwE3yQUxV0 zZWFJrk(89^v;NJ<^`x+n#Fe&Y0ReyCg=uP){jz4j+w$CZ$&pz}mPO z9_nN3?3&km&p&nUsl?P9!MEDeZg0v}KVSV$+fO^+?z}Sci)gx`Ew!ONW$XBuKcF}J z*BS!qg4i4j$Hubes1!wE9y&DsJtApS)M8k7n}d^RTclgEojvL0LtJ)g!y<*p8SX^) zB$%T89J(oY21G{V;T!hurVji zAPyUI(u?GK2@Wv$w{Xjt8Q(v{Qz_)dydcPndE03vbLl&X$hwkNJ~P~BuDqF!1$SA} zoHr<}q}`vPy9?YP*MG$gmY)iM|Rmpg8(Y)BV3-f%VE;2US!Z(+!$Pe zbM{2d8~hu=v~BIje2qQ;)K2A*3o7(zXha#HQV1XED1?zm4;`Mr#5daTG#d(Ub8wO^ zi*%c7VNZJL5SLxx(a2Ei|AdDv2i0=R@0qJE!*lUs)OVAHew~fnCi8DZV*_C{<vT zF^DNuunVG7s9Ur7)S3oiP4fu3TbehG(|yy%&D+;_iKz_X%`}(^zV6@r4_sbz#tbSq zJPAet33D>e!QRfNI|iTZ?(FN8iCQ4Sh(G2vvM$XSqB1*B8I<`|ov(k5-ym;!myZV-=@Zy$aL9!*eKUw;{ zJ#X!rf9^`*Po{3H-EyO7OWLzFW!rkonK%2~Yg=c!^zK(XnPwjwoDJ|ih{h4Iq#@aX zXcrw#H1nMb16H2z(J$Sqz)2zdrO26Ryi5z!JhUhP-R~a@k4!|rEYsEl%OVhqAN(x{ z+@ta8`y*)~EV<*4#y(+m)~`{~=uTGKFSsT{#cgsH+!c!^+&^M6xlh46f?6;lp$<0XfTD|h&K{J}~{3ifsztL~<+x-r|)9=cr1NyZQ9dYy;hcPq2 zS~2V^Jc;m~qnDk9E5W3HznAL`ug3lep7n&dsbqCtjS|=N|+xl%iBm-S`Wv8#UAP{t3Pr zt5^?&Wgly4#gBdob6$`CeGCzilnNxlFAPN_c!1iuON$?h(QK;@}xZN!Eqod9g`i(9|~)BuMz6TLc=jH$&S|S#gh~s z8I5^&5$i!y4#VvG}bKz@`AebmU(MZxMw5XgA8UdF_rM=Vaj(V$fdC8$RQ2X`nW zkrKNnn&k2&nHxLm8x5`Pvj!r$9F75xMLqokmvaZQwj(#;}34o|-P< z0I*yQclN`x1}o_qu8%ta?ly>g$gfTqvy;q*)n~aGa}@{OJT}jXxxU2&RWxMK*ew%! z#*ERoF=IHhgkG@k9tll^&Q3~uGmW`hn}~3|?X~Ft1?6M3;crmHRPBNHPG7B_H-S`W z{>ju&-T&JCXQzJ`rOLdts>2uS`qGa^VvzO0aqhh3M52ULO+*ono!=f82j{N69+9RfHA{z)9 zi#(YQEE%x0RAwWh2A+peEKCS$l_?c7o1xNWd?sBXha~xrNI3jUME$S$Y0^-KdE0Fc zPNHj(Zu3EQy9#M2o^&z`pD~Y39f|34E5bKK)q>-@|_*&$3<%7Tw$?W3Rm@3#ZEEq;%98WzSiWqpvnC zna2PG2DVPc%G37B%O*={IchWmZGHZECGYHi3zok$bcVLKt(H_kh0vQWIaVRhR z5e1uu{F3xT7GVNF9|FfH*d;$fK$J%6`WmSD&9FLgs;Nis*ns^pKE>2q$S<5V z-zq3gO#Ps8w)Ix|%4Ek65B-sQ?)1F>kEiEOU0Hv%;7`3*R;JeNPL=JMJ@BcgWFf!g zlIw!&jovq(y8P7K>33r3RhvJ~-?C7!GTC}*)x}lI-p|LcR{!J=(yR7-oWJ)o`O{?o z$N4q4JzU}H#eA-)G*Oc9&rT;>-rRn9`&`E#?YrTr2mC3zVxE8M23w}EX#+vC+)4(A z*DieqteI(4tY1GojD{A3$8gCF?aGF3(g%q2Yy6@cF@66(DDqTt?)-%HwIg66 zSlutZuuxc%C`qnOmdvf2ul{52jTH?y3L0iD3)X^#UP3cRy+-v5)} zpLqY(#CySO>)LL(+CH|me^RjO?n`nEZDD7A^({{_@X3rgOw4@zX zGi_ghdERSSWH@)x0!Ve`$u9_CG3RzU=gNO=>xIop>pXwGsPXOCpPWuPw$8LI@YZBs z+PUU(U(zt!o+$XkgO~fRIoI6a*RXf5KK|n4=VGs&zRs5}I6dcgoZFG;eQj@wU-7%Y zZ{gOr8-91kgyQ|~?>BO;wT!X)fwQio!tz1O+K!UKUzHZX{}(0s9fIXAOU&>G(6-uT zl6{6dL3py{Z~u@({-&!JkWZGxkx!Pyk&h`FK#{-(DVlwW6wQjw7dFv}neEEh4}>ES zpn;zq|DcByScA}h#Tq{VrB{WiCCf!frcy^zZK7D2#1^#cu)NGEAi6QBG&!2V+(7hUL*f~BTZ&d`WLV`g+V5a3&1eA# zPqn|z3LQw`nm{4Dlrm*ZfianmQF+slH%2mfdZFmqMTL=775>Ikm}p6wlD%_l=b~xP zhLmjsD-KJq8<1)flFuc5XA-ZBlWFW?7>!1P(*H(Lw~^x_hj<4J&`5tyK63s6zbGvy zN#4bqZHq20uju9O1&8P5!wW7FY%#}O_;UAW=6sT>H;^=|fcfMEDVZhx$s>UErvT3b z=q4jcvml5k&Tz&koyp=tQgH#E2~08?aE)AST97iV zJ>B1j$~ZIMDMvChU~BYqj6I055|aa(CsZC&{IOvusLP7Hz9g zweVlr3#HTx)rd&~tZIYhHaUr>7wI++VFhkiI$5Ax0qOwtqE1J{o9|R0)X(`1emivu zI3EUZM4%Z;91-wU)CzO`c7&6*s4ds>O5_`ebWyHsikE#ynx#3vn=~;l45e<=_%M7w zkLj&Z%9CdHY*(qaNeh+WNvpz{+5jrr+G_`2#j3{0#k~eIQxmA@=6OBs7Xg`85BbrE6q%YF z)V9#DN$($l*m4BlU`GWoG<02}4URLT;lL=e0~Z8>Mz%!-Y7*@8#_^DZEnCk`g{4!` zoq!eNfDTA^WE9L7?4kn8gtl}MCLPB%D*uE-!O!vzE7GuFFR}9p8^i*BFuY=ZC|A@0 zgM!f{Z~CD*>~q`)blJL3K<7Uy&@~|;XrcoQ$g9%_fZdERs5OPY>euwWpjf(^pn(dL z4d(TPN!KU(hz49wTPASR;g~u?d3ac{>);`-3$z5aDX&JtqR*gDp0%8WVMm&PD{zCj zBZHfC6OPQlQ1}H_MF^K!nXBN+OIGG$d`b%!@C-<&c6x42+OsZYTX%nH3D36>rP2q_ zZiB{#A?CE9*NDlciGFg`LWEw8I>i;XjKZAGV$0Y-K`{YMxAeaw7iewHSQ8j<`rKn8)67#*Z*|wvAab<0n2I+yvvwUCh0Gclk}hA$hDXL4&M9g zEd3V*uUT5Nyd6JlpMMVh_uOmo^DmrxAvru3xap}+dFuaN`|qUx10Q^?`NC9gRAIj9 zRA^d9l5r=)qD<;`{DH{w7ILr2o*q^+^lacHp>>1X+)DP8d%&^j@gI=FZX@I77<$d% z-SlF}8G9iM(76e@hnzlfa=y=+Sd*rHQ?59=GzmN&e%M4X|L}hBF$}rz*b;uolQI?~ z{d#a5t+WCeoQP%|faA*MILu#~#%yuzwMzdN{2Bf<*tyc@0^q+CV&;b z^|WlBQ6{=mPb`|L$~76XBCuXKh9ua8FPP>3x}apX^^?LC+Jsm!zdv2rbhEJKT4BqT zBk!+C7k13HE#wznvR<$zP2Y3Pwk~+`U;hIrd|h&0a3))Skaye86_!F~mtQ>VWL+4y zZ`tVE$ga43#}36S{Wq!*Pb(lc!bkAPagUvU45RBW;FHJxUy<+EC7Cyl1o?&7v6HrMP>`mwGeZS~NUi-{Jh=H%ud`#!< zNIQ4Tw12^?1O9d)m$xFhGVNYBbKnzqx#~N3IGuOogV_7kS@SnMUzqkZ-1Ka`=Gl0q z^lH(^o;@GLW)7zEj(qMcUdS(l-rc8Lc3pH_@_g4bXG!PR&aeMCf8+Z_pL&}vni2=T z=S&WRD|Dsz(`|b%R$Qw3Zq-~_y1+Ys;6}lwyH1PC0%aPr1>M$RIAqWRe)sog&eLUx zT2R#wDx2GFyyS=H5pDBHg0Iv9LG_ra?N|1pzCT7NkdsG?4D{i`cKCx!=t0mMr3!<$c1? z1uGuxoweVb1FK^P`Z`BCwLaKGY zJ*Obrqmt-V5Vbzi7y0@Zxm+ZcaK-YGNQPyVQI;3_-c9df8d4 ziA7jw6i+$_fX3uYETPsDOR4R|GHMmEoZ6`0i5AcnSykKs@k7`pt{5m){XHvGpMHmp zkd-|v^=H+GI#lKoX*8%2a382t9|o|TlCRAuIbGTDvRY0n*Mm#d@8}U&GE1DEwM!n^ z!)7H&-yJJ&OM_s2m0muTcG63N9BS@Q?Lj%Hy%pTe7H*GTlixF+f%R&+?l1L9ZlF$m z+f%Rl^gG60&X!Z_J?R?2;#j`=UE=EO9v*-YNO_}C;%W)jazb{_8`YTlePEO7A5go5 zYcauc2pC3WIb>j~`o3p}>eKJ)CsW2%(4N<9epLB40H}&sZu->)Dy=(d73&{?03I3x zv>1$zf^8d=O?_Zt3T+1x$up%bST%?;g9%KpLETuvGLHgvgJ6zt0-BOy^d$)+K&3jk|tDw(X= zjh)kcbSfChn9xe`{KiJwr?^Mks~>kZHd5=b&@b^2j>rwN=S#b~O<_;m+}J3cVYi{^ zp18GndzBMSutp64E*ZnLfk9}R7RNV zP>Vx;wV2Wih#EK6$l9tqe}j?4b$7>|bucC&>&AN-<>y_cQz`O(hY(^)L2~{S4#>u- zA*f&npmLuvp?cCVz4{J0&yzDl4owLr;9_F7& zn-My#f^x-Pu0oTs3V{UG6FUR%vD6)c)z|zxoDLjQ?s<+ z2m4$jQZZVP5jp=2o*8bjhAUtFQJME#|9teP@wekM2Nud!EqNxQ$CEHG*eVtrC5gbL z@eAXLK`;mCX*ltXlx_8bqv*}j3$56jQdFC=)q!=MFwVwiU0I%j#M(=B7wQrfDO=@& zqd0La(aAp4e6~^w?!ENPg=Z37DchO_+v>%Yc}@!iX{$M#3tQkwH#4#Gx^4BR)_nDG z-*wyC&z&Xb_kMeCa_xNZYVmdF-kJ8#td3WofA#qnpZ~Oa-G!a=n`UjdZ25_8$xX?c zWN2>x+~&FXmD(#SuGYVQ?ETIUM?ZWzRq&1Lwx@2{oU`p0TQ2Rmup`+w*Y@K>KRPtu z3;oVx>6$G!R&9AV`ileUB9a7Nx3weg>uu*d&vhn(mrh+cb+fegT4`;%)O)jZXR37P z{PAD3rCpC*x3!X#Zr8b8H=UK&oRv47t3PtCzUA^vomGe`P*Br z{F^S}noD@A?w!X|&TV(GVO+qXB!3050MeeSk8D+oR)l>S1$Dyxx^CB@8t%iI&7CVv z|FOaVcL^O&M45@`c*0K)pyO%RJ1MD}n`pO#tn?wBz&A3Lt`Av8f-~y-VTAW-49BRP z?H61z5#De};UBj-FbQxY%@5qJ#MnETWJYPVi!#Hr`Oed9XzQddBP`7d$*QQ<>O0_N z10z|)EmkEyC-g$JgnM$3(m>KCtX)X+P$>hnJ$dz$T*P(tLpasBNL=VP=BemgMgQP^ zfXO&%1{tA|+&$F1%Py2q({qI82*^CrbL{7H5wRz@?T0uZZ4d|=F$}a1h8LmzRovlS zoQ}{v@#}ry=$`RI=}u^t?Gq6;hk722B?)0Bm#verQUg$Ed#BjZ*5@VtyP*iPXcfiQ z1t|nP01 zvm6Q+n;}vt!(qn>)bqe!#{L?V>(m5gKmuK6sEQCM_DIEm^;Od$kl+3T8c_NlaN@v+ z48H=x4u$ARmJmgx={*uD88H-jW)H*Ugk6EaHlKxo+=qX9cmz#z+BszgjS>3)Kbfe7*#m`}X0;3!SR62mFm zsw@a^D^7&xRxSthw$h|~(QPFteK((Tl`ezBw~EUzb$++=jiEP3E|0u9c6n@m^Of@S z>YX1K@0ztO8cp_sTdutGo!{pN&hZ4X<%VL#L?$xR+uWcN7$D|%kKz(>NWK;3+YM+%JXPoc}MVaz*7 zr8M}1x$h-byDkySz3`^p8=^^tOzNUXUQ``k%W>KERI0cxJ3$6w3QTDE=U$=@W4VIz z04~VaSLB}4&;c-pRwCr2!Ad50HUN)U{G^g_0Q;axM+1t&!y`VG*k_9YIY^$1C!yCo z%x3#k6mvZo4iY&Ht&spDrpEzXRLBFZ=1J^mR_OJky;xL003{RbAj<)F6ia|CIhmf9 z?SqM}T|S{Z5_6EEFWGIo4{VUZwK}LBqX@#N&$gkv=zD{&5H zCXRHO)pZ7-9kb;$iajzRq8ABjf#8HX@d0TZEzL;3KCnssc*qoVbRX>OhV7=-?)J|1 zR_wHAlUarb1PZ`XX5=KrUZT@lc>jiSsTyt01~Glf;DF6ZW|5#m^=WDC?15XJ@?_DQ z<(JFnwq99x)3g1CXZymcwHFt7NsTb~<15(~7q% zuC`w`~`xwz`CDM}`*9@?3q`H2q^_s7+J5;W128audCa z>`vRq>G}R?N+=e9#C+#n);t*SRP<$+Znf+T;!1@OBpC?ll~>#6w2RJHSewtf`Az;f^kei z6eMHV$(DxL6=o+L&oF6vW$7!vh&ISL_lune5A_YwK3S;Q0CI9nzQXhnGbR$U(Ad+_ zqhpYoV2}zT)uEr+sA)$O%nKLo#P7b)J=c;hu6?m9& zUh-b>CfnxL{;>0{*gMap3wPXb?zq}^!?`cT@6*SYI)WaR+?I)L>}M#CMtXY;uG<_O+Juc8;W6Ci_psah*pZE=`$=y_ znZM~H!Lo~p{C+-L z-9I~RamifJ=S`N1wUUoow%;RX{z2)9OSW1)xDNfMtdt0lu`Acy{(Vx#HZ2-o?7F|c z*{$?7>TY>pt6Q^Ee^4oM=B#bH5^WD@H$$$aOjcbLvWZ5bnv;5m#J~t*3IsPc)#+nR z%UfSM>0cZM=bZ>8J-$9(MLQs^V~W<;8mo@18xxC5+06g_pGdQKE${6A{=bp$7hhsv zObNWA+%Rn64kex9fJJi|N!5|hR02}llZNkf z&2+$I1Z~W=zrI#hd`@=Eb^NgFMzMFk<3@4wyPJPeoObNJ&hJAMuuItH?R9SyzY$B@ zY8D)M_dv>fZWR{Ll zUz;6|knQXBiTGPI{ra=Xl5}3x z$2I{ZN?{pl_N{~WgKHk`fuZtfYYK@pYaY6K_;WtZr-0#|)8wkU5(t7D06Ly(#IBy4pjDr`Di31P&{g%2uO4X904o=5 ziisw0H%JhfL#PIkn9Q$%v=}jHS}nVJG|`RzLY`*bT1)As^=^4dDT)RI721nR?c!e*KOd zYV;@7htTz^U%w-?PJdE;qD$2!1VYWujcDX$*FQpTklAlj2vP{1jXjOavcMy9$%pZ4jFoS1=vEq4PIBC>Ao;^Kssqv0+$VkGYa>e4FXcPQ` zP`91JqF(4V63&V{bPG9O@?<0jB>YEcW~r5&HgekG(6Ts3FjM^a(&XDT3}@po5~clK z-B>U1%ygwwq}?QSWQE=vcO7Gf`;{l<6Aej&A8i z@|(ntc7{UPT3xn8VTO5Ptm}jH!PDftLe694Fp*L*sw;KVLO zYmi2lv=3=9+;kQmvK&;Jh+gMcvL*hj@tOFovWiR77p7CyE$=>=F57j^^|>6d;M(V=|UIW?gq}oY}cxYlgVyvpnJ=R9vn|y5_pC?7relxmp+URwV;(j$R&pbK>$u zdbKaLstH@$*8XzCdmFB+ba6>3U_+^VXX_D}OyKonwg&5~%O{&M}z ziu!96_34U+)QZOWzLa~@bz93VS5czv%_EnO%=i8D*xSbzDg}DDQuFhMpEcYy8QhN< zW(}XSI9->!=7T>Sd3$6@oDXX#Ru5u*28$rhbIt{@Bp*NbIJDm6Ew!_ex#l15_|cB} z-nk|UJbmtTA}~8PXZW$@N0zy0@|%lRzQpnWa=hJn+szeJ+%4zK?%Cc~x-^QZ0miWx zY%VZJ-&prn={qIwT2e*Z?pk@L9`(ssD38G$E76ZE4ROrgE zxj|n70GorQiQNz+7G;-cXKFW&CD>V`Nl;E`q@4hiSG3adw<(J#uyhQ$Y&?hsvI-Qi zT@ETmECh}+^%SeQ>vyq;y~#nNh{f!UMv-KFgEvd?+N6TlG+Jc|UV~D$e5q{)yrzW> zXw(bv+DhOx!+@a1AJ8b2EMJZGR-;MK@+E4^XW=z1kDU2xv`fzhHAVd{mJVoZN0sKx zm2#E>O|DeR^m41IJmib=0gWnAzt9ST;(d)WthhhLT&d`7Bvt_TSNacPLO%3N8F1nb z5UU%Ol_CEB%7aqs6#W(9blA_B+nO*<)^$;2bPlkoWezu!Xvqdc=ztn!izCQ`&{-%f zhuNtgS!bh!&SKM-vMrK?fqn;53sKB*Fnt89Vd-d_CU&q*Q-7CNIZcQ5lt@8h<`D~$ zSd&V0u=&73WC@YYJRr^(PA-v&3m8izE2@fq4{Zv?{!Df<0c{thL=;Y@EMAOPKeQp^ zd%D<8K_Xj7BSo_xOiC%6Phn4ttmq=BP1zu1H(k6?r))HW#-$K6GO4p4@`Dhe*`4>d zadzh|ZEvazJBbEnMWpzC`d+?RjnL{-kR-s6EFDfG9P`Jp8;4G`p(7Z;hxOwe6Ot%I zM`0<29lo+NzD`Yn?(s>OgoVNl=~4@gr^kfd6tGulmRd0U;;VGh3y1GeDYF`;Poi-w z^6`4{hcIBVh778bQ8GUBSJg1d3$S58;XUQ86(T`BR)``(sooZ!xJU>B~OHB z+^H0i;?L@rp~mB9-pD6WE5mEJu>p}FDdLV$oVUBeofA9 zdb^B9+(BVPC}yl(tp_{0`&ySXlwQWCAhd~!l<7t*DgjcNw5Bjw34iPSau=Q0q!EpW zkS-%9M&QA2{d~?`#>mFcdB=in15tsl!wY#(=lJo;AFZ5o&kumut9!5Rm%jIW>79?K zb{tN*jx4x(*ni8S(?YbL1GHbIO#2m8&00bGr92Ju$L2e)fZT&R(5_o~B@19n9iD5y zQhoK2wk#p!iiMnn++K~4D|yHAt~FJ(eHkH_#Ux7=i`VV`<#I%RN19P?-Ao(`$H-K=I* zxKQHiqXz7QAdBQ^H@fhZo>qaLmQuRH2b8TyFP%;?f-|J2SNxBVo_U}^E-AP@*)&p- z8mdz_y83&TT6>9D3SJyk;Kg!KwnlM9Kd<;I`pvS>XVR6-2Q-=(aaXFXq}M~wL(f;N z1XWcP|9SfZ&ub&pfB#u+2mG+8CQXrW9|B0WU3RD7SCyEf`k5A3SmRJ8WdkcPx2)+4 zvr%9%K>rxpu+UEeR>)$o_=G1R@&wLi-oYpjk^P)n*jAeptrd2WbjI0P|qjZ9U!CLtD`)*!3Um%%D;rYC8=OtqE?N?$qY`bq2mR|CG*Ox9_Gxy|;!unb3 z!iG&(o_J?h!jdezkuS`(+{j;_vhBOt28VTEe6>zuW(`z%{Q+KI#uyC7UE$$9P2gKL zk%f@BDKZR=c97-GU2zLL?r~4N6q*O8_YAY818C3&vVr%=ju?!h_yc$_u5dhCL2;t>dRgtf>8G~0 zZRxV+D9Zp1%whU$ z8Q%~)5fh~;Tlr@Y%E}~a%C%*|T{*W3^s3Op9)!7Pt`wzewxpm-ZsqM{{icC=F{4Y>X zbRe8mDUPVaUG6%()Gp$eveM1ryNBaq7?FYsPgeI**Dt%EH>!kAl3Y8NteWwHRU5!e z`fXp92p)C34_ zBBWJ}qsvQ|q?J2v8RC{B?usRGbGKlH&NXMp%%czKVr`kLtybwPniFPmH61sf)n>W# zs210VIA&g4r}pPproLXR73)N=PIrApzt@W!5EIABi@qhfYaD&K7UYUe_}iF0MzAty zhWIh-G;-hMbbQMrN<5*z8^$r(U_yc%_r-tJan?VM!?2<|A>3S+rr^sF)-*m5YnnJc zjx({ri_6+3u#vcqw2@#l9b&yaDfzSEC<9H1sS0twGAeRtgoQoH-zYV21i{cKMX+>0 z7U11vW!tjRLG&dki;;{In<4`+wNFZwI8SOajHnq8c&y{>$UaCQp-;_JjpFs%)`+HY zI)a-SX~xNg@^S5TwZ2+rUoUVb=!IEZ92PM)NxKu#WEm$3kwZAM6{{F*X$y`^C(%vu z4Vn@+`gIOeH;yPswck!O+5Ag9mFlTmDTO?G0RTuU;rVyvXxz(Ez%6MfJ_{-Pg)@ z-z{<4EiM^8^5~c z{nHC84=tKFb1Bg?j~hPYZOZH6x#L%B-tS*nxkr1wPk%c)_Xk%Gd|0~lV{AOl^W1cA zxaQuFb~j>UD(|?-mtEt_l8*U3@ADsG8$v53R!(`fAusi{YySBA>pmP^Sano;{e%Ij zDOK5dy>|OT`C)ojM4xuyQ!}gNuD5!@Lw<6h{D}6lTYI^EuJ6j$h4QW1tL@rnbKhJj zKcIyl(mvUoESs;qdSuB*S~-&ZTpzV){esgqb4VwF(Ld}Wiigor0z41ci^+B{J({-R z_lc5IIV^5NJ2))lCQ29Coz7LFXT2<}AaE3#&V1)vjHUxFR^MdZYZbq|pOT1bT>~JC z&ytIlFlK2|Y1~#-0h;LzXxvjh!(68`s1zRQ6l_|m{H;)sICef)3CyTdzMi(~LptV7 zrU6Z|uX3Op)CjnXmh2~e$&tn9ByGgpsoQit1mbGnHi6JgkcGLSfi8v01O^Z2pL0dd_75-C2&(Ad}fDG0M7yIl3haxoK;%K z7vJ29+##=J{3X~5CJP00&L?m)2p!sUrbX5)U)09=T45iunU3cjJ=Fkq752jrN4Gk@ z`k)l@$Et;1N`zQh`q$XaTJ0@h$V-;+y@SS*trxsVK9cas{2wMUJV9YE!66NBvdM6P z0=`QDWK9be(rNdsbe016(AjWIA~~V5iwp$TL=@*OE!jwBRS>203L-MF@;CSxhFh@* z?1fifn0et7Tb?H1N_4&vd#gX~XjsVeButlV7i^dEE)X9v<*A;lO1YXz)V1kzS^R~A zz}CK5ceyTEG1r`kycJ8k8ld>?Yy{kNZ@FhRIk(Uj_pRWK@R_sV{O)tRU)y`rS%s~nX=n9KXTvpT!+h^eXY)tSX43i&oDZK1 zr^@TzS)a~pzHZw@YU;Lc+Y$w@JGH{6iZ{O-{6$CFvHxSfO^1xAHoIsBJu;Ne=AfUJ z-oU3a@0>9ELzm%yOu(C_IkCszpP*SYM)pBcV z;T~;nDqrBH%W@b@qB<_SL~i>xz{`gA65xr(RLN!3N#}^eD3u|X{ui{f^b~UBm2+Rb zwxL6VZEILrbUiQVZ~=N70lhVOTP_$3-U1qyt}2cWe${12IDyEJOLGgMIkdJAs!6Bh z^vFeMh5n@a5UN2>HH^j00Z9>8(Br7ShqTyQHJ*NlMjNj}anZa@;f_FBS)<0urE3l; zs3zGIt&baE|1hUE8+5?UqGtz^3O-K#(9diAg29b3qV>x|^spSd{#Ta%lrly|XEvt& z{ra`vJlwamWI>EIY|OMxn!65Vm2acg9=SBjT)K%QwKa9T^vJ2Wb@NL1ry}NIOOmE_ zu5X)z^xCHDo{M%}=UmJi*sYej3HFh*4MLHf3cZBvSmXSR!5268VAC-)vza;;6D9)i zM`O^FoGpS5Jwt=6Sv^_ z(xc)|X!x+@f0M$2P!KCyxz)jRkSf$<0Zmvm2gV_KZS19IuP?q02W6A2cB%Z6us^Nm z8JhHyL2_)bRu}rT+VsVDbjrd~mM+k@zGhsum{k<*HEio+ECgUftt#Fsww{6&Pg-2j zB2H$8Fvf7k0!<{aJHR)g293%Hns6&K9VA;cJV}p!a=x++NgolGheoGxvNX=xB!kP+ z41GZZljUcEwmks{3ITfyYRmwX2v#(?Y+z<&`CVr zrw`7LV@I!)z&8x@P#0p|=qw|QT64`=Glz33>S1iBV)dKe%ig)@^@{q}H=W<{?H#Z0 zy2Ah5{xdtai#grrx1HN|e($-x$r|9{kDOINy9sk5h?6-WG1>6$+Mn0|tp0{=&nKSz z&mH+NXo-J1pBCF{&p&_edG!X#BhE)r-ziT;Wta9{*mtY2_)_hK+Di==8j`Wpx*h4l zop*~I%p^rQmtT7Ro8SIsvLo$TGhdVP_+}1)i~yan{oHn}U9Ha9e|Fuv`p0KAS|DkA ztL7b3I%TzcNNHcep8kG zw<_1%F&T<#u%vXa%X)F_R%I2vsHPWdJ}WAlZU3Bi&bGaBhAl{vp@WaQ&y;fvi~4;!{A`t?dZhOZjv-Li4)9x(m8Yom@YP$^Oz!Y0T~NS zd;7suvrR^#;j{3`=Aw=v9?izb#uOOJaXcg#W(5BR@Q?cqE$x0oYZI@ryBksr7rfG1 zdOiTd(Zrk!vc1**u|aGSkC5$E@aqN#rLBmQaSRR)gJ}US-k?NiREi;Csf-*a^HCUC z7m)6ekBJ}IR5*$-uod0Y2SlUvd&q-N2L|hqp#?m~K54Ly- zzDxn^g$+fpy3sma1|u~0(Md_fVWOtT6lwnM@kkJ-gYA|61>WGFi;kc>ehY%wV7SYd z8~A&hIm6n&<2--OIoZGcuQ~T$a|P_*fsjwRx=*>LYh2S}K4&kSF)vv2XZQt+XU4RE zN6UgGA3htsuovOcT7b8X;u)*#V=wZ`XPoyeE<=ami`8}m#6#p%a@M?a6Up8;pT7Kb zIc6U?yX&)s-Gg!k*N^M~mDQTDus zJulkLYm77YlymhR$R&-~ZBp%h#zaVq5_s$xM;rhs)%)O0duHuVw?8d)WgI%KczG1*ljeE};po@2J z(g4}&y$Xou5KJ7MF9a+O7GV~P$IhASrWK1u-dsvX)UevQQ2f2pk8S0*%^Y9xTLO=( z?s7(R=_fY#thzp$r}fcQ(=Q$GId0h6l bool: + """Decide per-stream whether to emit ANSI codes. + + Honors the de-facto ``NO_COLOR`` / ``FORCE_COLOR`` conventions and otherwise + colors only when the stream is an interactive terminal. + """ + if os.environ.get("NO_COLOR"): + return False + if os.environ.get("FORCE_COLOR"): + return True + if os.environ.get("TERM") == "dumb": + return False + return bool(getattr(stream, "isatty", None)) and stream.isatty() + + +class Painter: + """Wraps text in SGR codes, or returns it untouched when color is off.""" + + def __init__(self, enabled: bool): + self.enabled = enabled + + def paint(self, text: str, *codes: str) -> str: + if not self.enabled or not codes: + return text + return "".join(codes) + text + RESET + + +# stdout carries results (banner + tables); stderr carries progress + verbose +# fio dumps. Each gets its own enable flag so piping one but not the other works. +OUT = Painter(supports_color(sys.stdout)) +ERR = Painter(supports_color(sys.stderr)) + + +# --------------------------------------------------------------------------- # +# Constants +# --------------------------------------------------------------------------- # + +# Preference order for the IO engine. io_uring is the modern, lowest-overhead +# Linux engine; libaio is older (and only truly async with O_DIRECT); posixaio +# and sync are the portable fallbacks (e.g. some NFS mounts). +ENGINE_CANDIDATES = ("io_uring", "libaio", "posixaio", "sync") + +# Binary unit multipliers (fio treats "1g" as 1 GiB, so we match that). +_SIZE_UNITS = {"k": 1024, "m": 1024**2, "g": 1024**3, "t": 1024**4} + + +@dataclass(frozen=True) +class CdmTest: + """One CrystalDiskMark test. ``seq`` picks sequential vs random patterns.""" + + label: str + bs: str + iodepth: int + numjobs: int + seq: bool + + @property + def read_rw(self) -> str: + return "read" if self.seq else "randread" + + @property + def write_rw(self) -> str: + return "write" if self.seq else "randwrite" + + +# The four CrystalDiskMark default tests, as data. Q = queue depth (iodepth), +# T = threads (numjobs). +CDM_TESTS: tuple[CdmTest, ...] = ( + CdmTest("SEQ1M Q8T1", bs="1m", iodepth=8, numjobs=1, seq=True), + CdmTest("SEQ1M Q1T1", bs="1m", iodepth=1, numjobs=1, seq=True), + CdmTest("RND4K Q32T1", bs="4k", iodepth=32, numjobs=1, seq=False), + CdmTest("RND4K Q32T16", bs="4k", iodepth=32, numjobs=16, seq=False), +) + +# Thread counts for the SLOG sweep. T1 is the headline single-stream latency; +# the higher counts show how the SLOG scales as concurrent sync writers pile on. +SLOG_THREADS: tuple[int, ...] = (1, 4, 8, 16) + + +# --------------------------------------------------------------------------- # +# Configuration +# --------------------------------------------------------------------------- # + + +@dataclass +class Config: + """Resolved run settings. ``engine`` and ``direct`` may start unset and get + filled in by :func:`detect_io_settings`.""" + + path: str + mode: Optional[str] # "cdm" | "slog" | "dd" | None (auto) + engine: Optional[str] # None => auto-detect + direct: Optional[bool] # None => auto-detect + runtime: int + size: str + verbose: bool + assume_yes: bool + benchfile: str = field(default="") + + @property + def direct_flag(self) -> int: + """fio's --direct takes 0/1; default to 0 until detection runs.""" + return 1 if self.direct else 0 + + +# --------------------------------------------------------------------------- # +# Small helpers +# --------------------------------------------------------------------------- # + + +def log(message: str) -> None: + """Progress/diagnostic line -> stderr, so stdout stays pure results.""" + print(message, file=sys.stderr, flush=True) + + +def step(message: str) -> None: + """A single colored progress line ('measuring ...') -> stderr.""" + log(f" {ERR.paint('▶', BOLD, CYAN)} {ERR.paint(message, DIM)}") + + +def vsection(title: str) -> None: + """A clear, ruled header for one --verbose fio section -> stderr.""" + width = 78 + head = f"─── {title} " + head += "─" * max(3, width - len(head)) + log("") + log(ERR.paint(head, BOLD, CYAN)) + + +def parse_size_bytes(size: str) -> int: + """Turn an fio-style size ("4k", "1g", "512m", "2048") into bytes.""" + s = size.strip().lower() + if s and s[-1] in _SIZE_UNITS: + return int(float(s[:-1]) * _SIZE_UNITS[s[-1]]) + return int(s) + + +def render_table( + headers: Sequence[str], + rows: Sequence[Sequence[str]], + aligns: Optional[Sequence[str]] = None, + col_styles: Optional[Sequence[Optional[str]]] = None, +) -> str: + """Render a bordered ASCII table whose columns auto-size to their content. + + ``aligns`` is a per-column "<" (left) or ">" (right); by default the first + column is left-aligned (the label) and the rest are right-aligned (numbers). + ``col_styles`` is an optional per-column ANSI style applied to body cells + (header is always bold, borders dim). Widths are computed on the *plain* + text and color is applied after padding, so escape codes never skew layout. + """ + ncols = len(headers) + if aligns is None: + aligns = ["<"] + [">"] * (ncols - 1) + if col_styles is None: + col_styles = [None] * ncols + + # Widest cell (header or any row) sets each column's width. + widths = [ + max([len(str(headers[c]))] + [len(str(r[c])) for r in rows]) + for c in range(ncols) + ] + + bar = OUT.paint("|", DIM) + sep = OUT.paint(" | ", DIM) + + def rule() -> str: + return OUT.paint("+" + "+".join("-" * (w + 2) for w in widths) + "+", DIM) + + def line(cells: Sequence[str], *, header: bool = False) -> str: + parts = [] + for c in range(ncols): + padded = f"{str(cells[c]):{aligns[c]}{widths[c]}}" + if header: + parts.append(OUT.paint(padded, BOLD)) + elif col_styles[c]: + parts.append(OUT.paint(padded, col_styles[c])) + else: + parts.append(padded) + return f"{bar} " + sep.join(parts) + f" {bar}" + + out = [rule(), line(headers, header=True), rule()] + out += [line(r) for r in rows] + out.append(rule()) + return "\n".join(out) + + +# --------------------------------------------------------------------------- # +# fio: results, invocation, parsing +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class FioResult: + """Parsed metrics for one direction (read or write) of an fio run. + + Latencies are completion latency (clat) in microseconds; for synchronous + writes that is effectively the durable-commit latency. + """ + + bw_mbps: float # decimal MB/s, matching CrystalDiskMark's SI figure + iops: float + p50_us: float + p99_us: float + mean_us: float + + @classmethod + def zero(cls) -> "FioResult": + return cls(0.0, 0.0, 0.0, 0.0, 0.0) + + +def _direction_of(rw: str) -> str: + """fio reports 'read' and 'write' sub-objects; pick the active one.""" + return "read" if "read" in rw else "write" + + +def _aggregate(jobs: list[dict], direction: str) -> FioResult: + """Combine per-job fio JSON stats into a single :class:`FioResult`. + + We deliberately run fio *without* --group_reporting and aggregate ourselves, + which sidesteps fio's group-merge quirks: sum throughput, average the median + latency, and take the worst-case tail (p99) across jobs. + """ + + def section(job: dict) -> dict: + return job.get(direction, {}) + + bw_mbps = sum(section(j).get("bw_bytes", 0) for j in jobs) / 1e6 + iops = sum(section(j).get("iops", 0.0) for j in jobs) + + def percentiles(key: str) -> list[float]: + vals = [] + for j in jobs: + pct = section(j).get("clat_ns", {}).get("percentile", {}) + v = pct.get(key) + if v is not None: + vals.append(v) + return vals + + p50_ns = percentiles("50.000000") + p99_ns = percentiles("99.000000") + mean_ns = [section(j).get("clat_ns", {}).get("mean", 0.0) for j in jobs] + + return FioResult( + bw_mbps=bw_mbps, + iops=iops, + p50_us=(statistics.mean(p50_ns) / 1000.0) if p50_ns else 0.0, # avg of medians + p99_us=(max(p99_ns) / 1000.0) if p99_ns else 0.0, # worst tail + mean_us=(statistics.mean(mean_ns) / 1000.0) if mean_ns else 0.0, + ) + + +def run_fio( + cfg: Config, + *, + rw: str, + bs: str, + iodepth: int, + numjobs: int, + engine: Optional[str] = None, + direct: Optional[bool] = None, + sync: bool = False, + end_fsync: bool = False, +) -> FioResult: + """Run a single fio job against the shared bench file and parse its JSON. + + ``engine``/``direct`` default to the detected config values; the SLOG path + overrides them (psync + buffered + O_SYNC). ``end_fsync`` flushes the device + cache at the end of a write run so cached writes can't inflate the result. + """ + engine = engine if engine is not None else cfg.engine + direct = direct if direct is not None else cfg.direct + + cmd = [ + "sudo", "fio", + "--name=speedtest", + f"--filename={cfg.benchfile}", + f"--ioengine={engine}", + f"--direct={1 if direct else 0}", + f"--rw={rw}", + f"--bs={bs}", + f"--size={cfg.size}", + f"--numjobs={numjobs}", + f"--iodepth={iodepth}", + "--time_based", + f"--runtime={cfg.runtime}", + "--output-format=json", + ] + if sync: + cmd.append("--sync=1") # O_SYNC: every write is a durable commit + if end_fsync: + cmd.append("--end_fsync=1") # flush device cache once at the end + + proc = subprocess.run(cmd, capture_output=True, text=True) + + if cfg.verbose: + sync_tag = " sync=1" if sync else "" + vsection(f"fio · {rw} bs={bs} iodepth={iodepth} numjobs={numjobs}{sync_tag}") + log(" " + ERR.paint("$ " + " ".join(cmd[1:]), DIM, ITALIC)) + log(ERR.paint(proc.stdout.strip() or "(no stdout)", DIM)) + if proc.stderr.strip(): + log(ERR.paint(proc.stderr.strip(), YELLOW)) + + try: + data = json.loads(proc.stdout) + return _aggregate(data["jobs"], _direction_of(rw)) + except (json.JSONDecodeError, KeyError, ValueError): + # fio failed or produced no parseable JSON; report zeros rather than + # crashing the whole run (verbose mode above shows what went wrong). + return FioResult.zero() + + +def fio_probe(path: str, engine: str, direct: bool) -> bool: + """Throwaway 1s fio job to see if an (engine, O_DIRECT) combo works here. + + This is how we stay accurate *and* portable across ext4/xfs/btrfs/ZFS/NFS + without the caller needing to know what each filesystem supports. + """ + cmd = [ + "sudo", "fio", "--name=probe", + f"--directory={path}", + f"--ioengine={engine}", + "--rw=write", "--bs=4k", "--size=1m", + f"--direct={1 if direct else 0}", + "--time_based", "--runtime=1", + ] + proc = subprocess.run(cmd, capture_output=True, text=True) + for leftover in glob.glob(os.path.join(path, "probe*")): + try: + os.remove(leftover) + except OSError: + pass + return proc.returncode == 0 + + +def detect_io_settings(cfg: Config) -> None: + """Fill in ``cfg.engine`` and ``cfg.direct`` if the user didn't force them. + + O_DIRECT bypasses the OS page cache so we measure the device, not RAM. Not + every filesystem supports it (older OpenZFS, some NFS), so we probe and fall + back to buffered rather than erroring out. + """ + if cfg.engine is None: + for engine in ENGINE_CANDIDATES: + if fio_probe(cfg.path, engine, direct=False): + cfg.engine = engine + break + else: + cfg.engine = "sync" + + if cfg.direct is None: + cfg.direct = fio_probe(cfg.path, cfg.engine, direct=True) + + +# --------------------------------------------------------------------------- # +# Profiles +# --------------------------------------------------------------------------- # + + +def _meta_line(text: str) -> str: + """Color the 'Label :' prefix of a banner metadata line, leave value as-is. + + The value may already contain its own ANSI codes (e.g. a red O_DIRECT + warning); since these lines aren't width-aligned that's harmless. + """ + key, sep, val = text.partition(":") + if not sep: + return " " + text + return " " + OUT.paint(key + ":", BOLD, BRIGHT_CYAN) + val + + +def banner(title: str, cfg: Config, extra: Sequence[str] = ()) -> None: + full = f"speedtest-hd · {title}" + inner = len(full) + 4 + + print() + print(OUT.paint("╭" + "─" * inner + "╮", BOLD, CYAN)) + print( + OUT.paint("│", BOLD, CYAN) + + " " + OUT.paint(full, BOLD, WHITE) + " " + + OUT.paint("│", BOLD, CYAN) + ) + print(OUT.paint("╰" + "─" * inner + "╯", BOLD, CYAN)) + print(_meta_line(f"Target : {OUT.paint(cfg.path, CYAN)}")) + for line_ in extra: + print(_meta_line(line_)) + print() + + +def cdm_profile(cfg: Config) -> None: + """The CrystalDiskMark-style profile: 4 tests x (read, write), MB/s + IOPS.""" + detect_io_settings(cfg) + + if cfg.direct: + direct_label = OUT.paint("enabled (device)", GREEN) + else: + direct_label = OUT.paint( + "DISABLED (buffered -- may reflect RAM cache!)", BOLD, RED + ) + + banner( + "CrystalDiskMark-style storage benchmark", + cfg, + extra=[ + f"Engine : {cfg.engine} O_DIRECT: {direct_label}", + f"Profile : size={cfg.size.upper()} runtime={cfg.runtime}s/run (8 runs)", + ], + ) + + rows: list[list[str]] = [] + for test in CDM_TESTS: + step(f"measuring {test.label.strip()} ...") + r = run_fio(cfg, rw=test.read_rw, bs=test.bs, + iodepth=test.iodepth, numjobs=test.numjobs) + w = run_fio(cfg, rw=test.write_rw, bs=test.bs, + iodepth=test.iodepth, numjobs=test.numjobs, end_fsync=True) + rows.append([ + test.label, + f"{r.bw_mbps:.2f}", f"{w.bw_mbps:.2f}", + f"{r.iops:.0f}", f"{w.iops:.0f}", + ]) + + _cleanup(cfg) + + print() + print(render_table( + ["Test", "Read (MB/s)", "Write (MB/s)", "Read (IOPS)", "Write (IOPS)"], + rows, + col_styles=[CYAN, GREEN, YELLOW, GREEN, YELLOW], + )) + print() + + +def slog_profile(cfg: Config) -> None: + """SLOG / sync-write latency profile. + + Forces synchronous 4K random writes (O_SYNC) via the portable psync engine, + so every write is a ZIL commit -- exercising a ZFS SLOG exactly the way a + sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own + sync property. Engine/direct detection is only for the banner here. + """ + detect_io_settings(cfg) + + banner( + "SLOG / sync-write latency profile", + cfg, + extra=[ + "Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine", + f"Profile : runtime={cfg.runtime}s/run size={cfg.size.upper()}", + "Note : every write is a synchronous ZIL commit -- the load your", + " SLOG actually sees. Watch it live in another shell with:", + " zpool iostat -vl 1", + ], + ) + + # Measure everything first (progress -> stderr), then draw the whole table, + # so the "measuring..." lines can't interleave into the middle of it. + rows: list[list[str]] = [] + for threads in SLOG_THREADS: + step(f"measuring 4K sync randwrite T{threads} ...") + res = run_fio( + cfg, rw="randwrite", bs="4k", iodepth=1, numjobs=threads, + engine="psync", direct=False, sync=True, + ) + rows.append([ + f"4K sync T{threads}", + f"{res.iops:.0f}", f"{res.bw_mbps:.2f}", + f"{res.p50_us:.1f}", f"{res.p99_us:.1f}", + ]) + + _cleanup(cfg) + + print() + print(render_table( + ["Test", "IOPS", "MB/s", "p50 lat(us)", "p99 lat(us)"], + rows, + col_styles=[CYAN, GREEN, GREEN, YELLOW, MAGENTA], + )) + print() + print(OUT.paint(" Healthy Optane SLOG (eg P1600X) single-stream (T1) target:", BOLD)) + print(OUT.paint(" ~15-25k IOPS, p50 latency ~40-65us.", GREEN) + + OUT.paint(" Much higher latency usually means", DIM)) + print(OUT.paint(" CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling.", DIM)) + print() + + +def dd_profile(cfg: Config) -> None: + """Dependency-free fallback when fio isn't installed. + + Far cruder than fio: a single sequential stream, and the cached-read figure + will reflect RAM. Use it only as a rough sanity check. + """ + count_mib = max(1, parse_size_bytes(cfg.size) // (1024**2)) + nbytes = count_mib * 1024**2 + + banner( + "basic dd benchmark (fio not installed)", + cfg, + extra=[f"Profile : {count_mib} MiB sequential stream"], + ) + + def timed_dd(args: list[str]) -> float: + start = time.monotonic() + subprocess.run(args, capture_output=True, text=True) + elapsed = time.monotonic() - start + return (nbytes / elapsed / 1e6) if elapsed > 0 else 0.0 # decimal MB/s + + step("measuring sequential write ...") + write_mbps = timed_dd([ + "dd", "if=/dev/zero", f"of={cfg.benchfile}", + "bs=1M", f"count={count_mib}", "conv=fdatasync,notrunc", + ]) + + step("dropping caches for uncached read ...") + subprocess.run(["sudo", "sh", "-c", "echo 3 > /proc/sys/vm/drop_caches"], + capture_output=True, text=True) + + step("measuring uncached read ...") + uncached_mbps = timed_dd([ + "dd", f"if={cfg.benchfile}", "of=/dev/null", "bs=1M", f"count={count_mib}", + ]) + + step("measuring cached read ...") + cached_mbps = timed_dd([ + "dd", f"if={cfg.benchfile}", "of=/dev/null", "bs=1M", f"count={count_mib}", + ]) + + _cleanup(cfg) + + print() + print(render_table( + ["Test", "MB/s"], + [ + ["Sequential write", f"{write_mbps:.2f}"], + ["Uncached read", f"{uncached_mbps:.2f}"], + ["Cached read (RAM)", f"{cached_mbps:.2f}"], + ], + col_styles=[CYAN, GREEN], + )) + print() + + +def _cleanup(cfg: Config) -> None: + """Remove the shared bench file.""" + try: + os.remove(cfg.benchfile) + except OSError: + pass + + +# --------------------------------------------------------------------------- # +# CLI +# --------------------------------------------------------------------------- # + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser( + prog="speedtest-hd.py", + description="CrystalDiskMark-style storage benchmark built on fio.", + epilog=( + "Examples:\n" + " speedtest-hd.py .\n" + " speedtest-hd.py /mnt/nvmepool --runtime=10 --size=4g\n" + " speedtest-hd.py /mnt/nfsshare --buffered\n" + " speedtest-hd.py /mnt/nvme-ultra-r10/vm-root --slog --runtime=30\n" + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p.add_argument("path", help="directory/mount to benchmark ('.' for cwd)") + + mode = p.add_mutually_exclusive_group() + mode.add_argument("--fio", action="store_const", const="cdm", dest="mode", + help="force the fio CrystalDiskMark-style profile") + mode.add_argument("--dd", action="store_const", const="dd", dest="mode", + help="force the basic dd fallback test") + mode.add_argument("--slog", action="store_const", const="slog", dest="mode", + help="SLOG / sync-write latency profile (ZFS ZIL)") + + direct = p.add_mutually_exclusive_group() + direct.add_argument("--direct", action="store_const", const=True, dest="direct", + help="force O_DIRECT (bypass page cache)") + direct.add_argument("--buffered", action="store_const", const=False, dest="direct", + help="force buffered IO (e.g. if O_DIRECT unsupported)") + + p.add_argument("--engine", choices=ENGINE_CANDIDATES, + help="force a specific IO engine (default: auto-detect)") + p.add_argument("--runtime", type=int, default=5, metavar="SEC", + help="seconds per run (default: 5, like CrystalDiskMark)") + p.add_argument("--size", default="1g", metavar="SIZE", + help="test file size (default: 1g)") + p.add_argument("--verbose", action="store_true", + help="also print the full fio output for every run") + p.add_argument("-y", "--yes", action="store_true", dest="assume_yes", + help="skip the confirmation prompt") + + p.set_defaults(mode=None, direct=None) + return p + + +def confirm(cfg: Config) -> None: + """Guard prompt -- we're about to write a multi-GB file to the target.""" + if cfg.assume_yes: + return + print(OUT.paint("NOTICE:", BOLD, YELLOW) + + f" {cfg.size.upper()} free space on " + + OUT.paint(f"'{cfg.path}'", CYAN) + + " is required to perform the benchmark.") + answer = input(f"Are you ready to start a storage benchmark against " + f"'{cfg.path}' ? ") + if not answer.strip().lower().startswith("y"): + print(OUT.paint("Ok, cancelled!", YELLOW)) + sys.exit(0) + print(OUT.paint("Great! Starting benchmark now!", BOLD, GREEN)) + + +def main(argv: Optional[Sequence[str]] = None) -> int: + args = build_parser().parse_args(argv) + + path = os.getcwd() if args.path == "." else args.path + if not os.path.exists(path): + print(f"Path {path} does not exist", file=sys.stderr) + return 1 + + cfg = Config( + path=path, + mode=args.mode, + engine=args.engine, + direct=args.direct, + runtime=args.runtime, + size=args.size, + verbose=args.verbose, + assume_yes=args.assume_yes, + benchfile=os.path.join(path, "speedtest-hd.bench"), + ) + + have_fio = shutil.which("fio") is not None + + # Resolve the mode: explicit flag wins; otherwise fio if available, else dd. + mode = cfg.mode + if mode in ("cdm", "slog") and not have_fio: + print(ERR.paint("ERROR:", BOLD, RED) + + " --fio/--slog require fio (apt install fio / pacman -S fio).", + file=sys.stderr) + return 1 + if mode is None: + if have_fio: + mode = "cdm" + else: + print(OUT.paint("\nfio is not installed -- falling back to basic dd test.", + YELLOW)) + print(OUT.paint("Install fio for the full CrystalDiskMark-style benchmark.", + DIM)) + mode = "dd" + + confirm(cfg) + + {"cdm": cdm_profile, "slog": slog_profile, "dd": dd_profile}[mode](cfg) + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except KeyboardInterrupt: + print(ERR.paint("\nInterrupted.", YELLOW), file=sys.stderr) + sys.exit(130) diff --git a/speedtest-hd.sh b/speedtest-hd.sh index fa85f83..205fa0c 100755 --- a/speedtest-hd.sh +++ b/speedtest-hd.sh @@ -1,8 +1,11 @@ #!/usr/bin/env bash -# Robust HD/SDD/NVMe performance CLI utility -# Utilizing FIO for sequential/random writes/writes -# Dependencies: fio (apt install fio) +# Robust HD/SSD/NVMe performance CLI utility +# A CrystalDiskMark-style storage benchmark for Linux, built on fio. +# Runs the exact same four tests CrystalDiskMark does (SEQ1M Q8T1, SEQ1M Q1T1, +# RND4K Q32T1, RND4K Q32T16), each measured for both Read and Write, and prints +# the results in a CrystalDiskMark-style table. +# Dependencies: fio (apt install fio / pacman -S fio). Falls back to dd if missing. # See: https://cloud.google.com/compute/docs/disks/benchmarking-pd-performance # See: https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/ # mReschke 2024-01-18 @@ -10,20 +13,23 @@ # CLI Parameters path="$1" option="" -simple=false engine="" # IO engine; empty = auto-detect (io_uring > libaio > posixaio > sync) direct="" # O_DIRECT; empty = auto-detect; 1 = bypass page cache, 0 = buffered -runtime=30 # seconds per test (longer = more accurate, exposes SLC cache exhaustion) +runtime=5 # seconds per measurement (CrystalDiskMark default is 5s) +size="1g" # test file size (CrystalDiskMark default is 1GiB) +verbose=0 # 1 = also print the full fio output for every run for arg in "${@:2}"; do case "$arg" in - --simple) simple=true ;; --dd) option="--dd" ;; --fio) option="--fio" ;; + --slog) option="--slog" ;; --buffered) direct=0 ;; --direct) direct=1 ;; + --verbose) verbose=1 ;; --engine=*) engine="${arg#*=}" ;; --direct=*) direct="${arg#*=}" ;; --runtime=*) runtime="${arg#*=}" ;; + --size=*) size="${arg#*=}" ;; esac done @@ -47,275 +53,35 @@ function main { fi # Must type y or n THEN press enter (which I like better) - echo "NOTICE: 1GB free space on '$path' is required to perform the benchmark." - echo -n "Are you ready to start a robust IO benchmark against '$path' ?"; read answer + echo "NOTICE: ${size^^} free space on '$path' is required to perform the benchmark." + echo -n "Are you ready to start a storage benchmark against '$path' ? "; read answer if [ "$answer" != "${answer#[Yy]}" ]; then echo "Great! Starting benchmark now!" - echo "------------------------------" else echo "Ok, cancelled!" exit 0 fi - # Use dd of fio based on param or defaults + # Use dd or fio based on param or defaults if [ "$option" == "--dd" ]; then dd_speedtest elif [ "$option" == "--fio" ]; then - fio_speedtest + cdm_speedtest + elif [ "$option" == "--slog" ]; then + slog_speedtest elif [ "$option" == "" ]; then # If fio is installed, use it, else use dd - [ "$simple" != true ] && echo "" if ! command -v fio &> /dev/null; then + echo "" + echo "fio is not installed -- falling back to basic dd test." + echo "Install fio for the full CrystalDiskMark-style benchmark." dd_speedtest else - fio_speedtest + cdm_speedtest fi fi } -function print_result { - # Print a single fio test result, either as the full "Run status group" - # line or, when --simple is given, as a compact aligned "Label value" row. - # The simple value is taken from the parenthetical (MB/s) figure that - # follows the first bw= value in fio's output. - local label="$1" - local output="$2" - local status - status=$(echo " - $output " | /usr/bin/grep -A1 'Run status group' | tail -n1) - if [ "$simple" == true ]; then - local value - # Grab the parenthetical (MB/s) figure after the first bw=, then put a - # space between the number and the unit, eg "98.4MB/s" -> "98.4 MB/s". - value=$(echo "$status" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | sed -E 's/([0-9.]+)([A-Za-z])/\1 \2/') - printf "%-28s%s\n" "$label" "$value" - else - echo "$status" - fi -} - -function fio_write_single_random_4k { - # 4K Random Writes (parallel, deep queue) - - # Random 4K writes are the worst thing you can ask a disk to do. Where this - # happens most in real life: copying home directories and dotfiles, email - # stores, some database operations, source code trees. - - # IMPORTANT: the original version of this test used numjobs=1 + iodepth=1. - # That measures single-op LATENCY (the absolute worst case): the OS issues - # one 4K write, waits for it to be acknowledged, then issues the next. On - # NVMe that lands around ~3000 IOPS / ~12MB/sec -- which looks alarmingly - # slow but is NOT a bug. NVMe's entire advantage is deep command queues and - # parallelism, so a queue depth of 1 deliberately measures the one scenario - # NVMe is bad at. To measure the drive's actual capability we now drive it - # with multiple jobs and a deep queue (4 jobs x iodepth=64), keeping - # hundreds of operations in flight at once. Expect this to jump into the - # hundreds of MB/sec (often 1GB/sec+) on a healthy NVMe pool. - - # --name= is a required argument, but it's basically human-friendly fluff—fio will create files based on that name to test with, inside the directory you point it at. - # --directory= where the test files are created; this is the path/mount you are benchmarking. - # --ioengine=$engine the engine fio uses to talk to the kernel, auto-detected by detect_io_settings(). On modern Linux this is io_uring (lowest overhead); it falls back to libaio, then posixaio, then sync. NOTE: libaio is only truly asynchronous with --direct=1; with buffered IO it silently degrades to synchronous, which is exactly why io_uring is preferred. - # --rw=randwrite random write operations. Other options include read, write (sequential), randread, and randrw. - # --bs=4k blocksize 4K. These are very small individual operations. This is where the pain lives; it's hard on the disk and adds a ton of command-channel overhead, since a separate operation has to be commanded for each 4K of data. - # --size=256m each job's test file is 256MB. With --numjobs=4 that's 4 files totaling ~1GB. - # --numjobs=4 run 4 parallel processes, each with its own test file, to feed the drive from multiple threads at once (single-threaded cannot saturate NVMe). - # --iodepth=64 how many commands each job keeps stacked in the queue at once. Deep queues are how NVMe reaches its real IOPS; 4 jobs x 64 = up to 256 ops in flight. - # --direct=$direct O_DIRECT, auto-detected. 1 bypasses the OS page cache so we measure the device instead of RAM; 0 (buffered) is the fallback for filesystems/mounts that reject O_DIRECT (eg older OpenZFS, some NFS). - # --ramp_time=2s discard the first 2 seconds of results so we measure steady state, not the initial warm-up burst. - # --runtime=$runtime --time_based run for this many seconds, looping over the file(s) if we finish early. - # --group_reporting=1 aggregate all jobs into a single combined result line instead of one line per job. - # --end_fsync=1 after the timed run, flush everything to stable storage and count that time, so cached writes can't inflate the number. - [ "$simple" != true ] && { echo ""; echo "4K Random Writes (bs=4k, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; } - x=`sudo fio \ - --name=fio-write-random-4k \ - --directory=$path \ - --ioengine=$engine \ - --rw=randwrite \ - --bs=4k \ - --size=256m \ - --numjobs=4 \ - --iodepth=64 \ - --direct=$direct \ - --ramp_time=2s \ - --time_based --runtime=$runtime \ - --group_reporting=1 \ - --end_fsync=1` - print_result "4K Random Writes" "$x" - - # Cleanup my test files - rm -rf $path/fio-write-random-4k* -} - -function fio_write_parallel_random_64k { - # Parallel 64k Random Writes - - # This time, we're creating 16 separate 64MB files (still totaling 1GB, when - # all put together) and we're issuing 64KB blocksized random write operations. - # We're doing it with sixteen separate processes running in parallel, and - # we're queuing up to 16 simultaneous asynchronous ops before we pause and wait - # for the OS to start acknowledging their receipt. - - # This is a pretty decent approximation of a significantly busy system. It's - # not doing any one particularly nasty thing—like running a database engine or - # copying tons of dotfiles from a user's home directory—but it is coping with - # a bunch of applications doing moderately demanding stuff all at once. - - # This is also a pretty good, slightly pessimistic approximation of a busy, - # multi-user system like a NAS, which needs to handle multiple 1MB operations - # simultaneously for different users. If several people or processes are trying - # to read or write big files (photos, movies, whatever) at once, the OS tries - # to feed them all data simultaneously. This pretty quickly devolves down to a - # pattern of multiple random small block access. So in addition to "busy desktop - # with lots of apps," think "busy fileserver with several people actively using it." - - # You will see a lot more variation in speed as you watch this operation play - # out on the console. Unlike the steady trickle you'd get from a queue-depth-1 - # single-op latency test, this 16-process job can fluctuate wildly—eg between - # about 10MiB/sec and 300MiB/sec during the run—as the OS and SSD firmware - # catch good and bad luck aggregating writes. - - # Most of the variation you're seeing here is due to the operating system and - # SSD firmware sometimes being able to aggregate multiple writes. When it - # manages to aggregate them helpfully, it can write them in a way that allows - # parallel writes to all the individual physical media stripes inside the SSD. - # Sometimes, it still ends up having to give up and write to only a single - # physical media stripe at a time—or a garbage collection or other maintenance - # operation at the SSD firmware level needs to run briefly in the background, - # slowing things down. - [ "$simple" != true ] && { echo ""; echo "Parallel 64K Random Writes (bs=64k, jobs=16, iodepth=16, engine=$engine, direct=$direct, ${runtime}s)"; } - x=`sudo fio \ - --name=fio-write-random-64k \ - --directory=$path \ - --ioengine=$engine \ - --rw=randwrite \ - --bs=64k \ - --size=64m \ - --numjobs=16 \ - --iodepth=16 \ - --direct=$direct \ - --ramp_time=2s \ - --time_based --runtime=$runtime \ - --group_reporting=1 \ - --end_fsync=1` - print_result "Parallel 64K Random Writes" "$x" - - # Cleanup my test files - rm -rf $path/fio-write-random-64k* -} - -function fio_write_single_sequential_1m { - # Single 1M Sequential Writes - - # This is pretty close to the best-case scenario for a real-world system - # doing real-world things. No, it's not quite as fast as a single, truly - # contiguous write... but the 1MiB blocksize is large enough that it's quite - # close. Besides, if literally any other disk activity is requested simultaneously - # with a contiguous write, the "contiguous" write devolves to this level of - # performance pretty much instantly, so this is a much more realistic test of - # the upper end of storage performance on a typical system. - - # You'll see some kooky fluctuations on SSDs when doing this test. This is largely - # due to the SSD's firmware having better luck or worse luck at any given time, - # when it's trying to queue operations so that it can write across all physical - # media stripes cleanly at once. Rust disks will tend to provide a much more - # consistent, though typically lower, throughput across the run. - - # You can also see SSD performance fall off a cliff here if you exhaust an - # onboard write cache—TLC and QLC drives tend to have small write cache areas - # made of much faster MLC or SLC media. Once those get exhausted, the disk has - # to drop to writing directly to the much slower TLC/QLC media where the data - # eventually lands. This is the major difference between, for example, Samsung - # EVO and Pro SSDs—the EVOs have slow TLC media with a fast MLC cache, where - # the Pros use the higher-performance, higher-longevity MLC media throughout - # the entire SSD. - - # If you have any doubt at all about a TLC or QLC disk's ability to sustain - # heavy writes, you may want to experimentally extend your time duration here. - # If you watch the throughput live as the job progresses, you'll see the impact - # immediately when you run out of cache—what had been a fairly steady, - # several-hundred-MiB/sec throughput will suddenly plummet to half the speed - # or less and get considerably less stable as well. - - # However, you might choose to take the opposite position—you might not - # expect to do sustained heavy writes very frequently, in which case you - # actually are more interested in the on-cache behavior. What's important - # here is that you understand both what you want to test, and how to test - # it accurately. - - [ "$simple" != true ] && { echo ""; echo "Single 1M Sequential Writes (bs=1m, jobs=1, iodepth=16, engine=$engine, direct=$direct, ${runtime}s)"; } - x=`sudo fio \ - --name=fio-write-seq-1m \ - --directory=$path \ - --ioengine=$engine \ - --rw=write \ - --bs=1m \ - --size=1g \ - --numjobs=1 \ - --iodepth=16 \ - --direct=$direct \ - --ramp_time=2s \ - --time_based --runtime=$runtime \ - --group_reporting=1 \ - --end_fsync=1` - print_result "Single 1M Sequential Writes" "$x" - - # Cleanup my test files - rm -rf $path/fio-write-seq-1m* -} - -function fio_read_sequential_1m { - # Sequential Parallel Reads - - [ "$simple" != true ] && { echo ""; echo "Sequential 4x 1M Reads (bs=1m, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; } - x=`sudo fio \ - --name=fio-read-sequential-1m \ - --directory=$path \ - --ioengine=$engine \ - --bs=1M \ - --numjobs=4 \ - --size=256M \ - --time_based --runtime=$runtime \ - --ramp_time=2s \ - --direct=$direct \ - --verify=0 \ - --iodepth=64 \ - --rw=read \ - --group_reporting=1 \ - --iodepth_batch_submit=64 \ - --iodepth_batch_complete_max=64` - print_result "Sequential 4x 1M Reads" "$x" - rm -rf $path/fio-read-sequential-1m* -} - -function fio_read_random_4k { - # Random 4k Reads (parallel, deep queue) - - # 4K random reads are the highest-IOPS thing a drive does, but a single - # submitting thread goes CPU-bound long before an NVMe runs out of capacity, - # so it under-reports peak IOPS. We spread the work over 4 jobs x iodepth=64 - # (256 ops in flight, like the 4K write test) to saturate fast drives, while - # still reporting the honest (low) number on a spinning HDD. size=256m per - # job keeps the total footprint at ~1GB across the 4 jobs. - [ "$simple" != true ] && { echo ""; echo "Random 4k Reads (bs=4k, jobs=4, iodepth=64, engine=$engine, direct=$direct, ${runtime}s)"; } - x=`sudo fio \ - --name=fio-read-random-4k \ - --directory=$path \ - --ioengine=$engine \ - --rw=randread \ - --bs=4k \ - --size=256m \ - --numjobs=4 \ - --time_based --runtime=$runtime \ - --ramp_time=2s \ - --direct=$direct \ - --verify=0 \ - --iodepth=64 \ - --group_reporting=1 \ - --iodepth_batch_submit=64 \ - --iodepth_batch_complete_max=64` - print_result "Random 4k Reads" "$x" - rm -rf $path/fio-read-random-4k* -} - function fio_probe { # Run a tiny throwaway fio job to see if a given (engine, direct) combo # actually works on this path/filesystem. Returns 0 on success. This is how @@ -353,57 +119,291 @@ function detect_io_settings { if [ -z "$direct" ]; then if fio_probe "$engine" 1; then direct=1; else direct=0; fi fi - - if [ "$simple" != true ]; then - echo "" - echo "IO engine : $engine" - if [ "$direct" == 1 ]; then - echo "O_DIRECT : enabled (bypassing page cache -- measuring the device)" - else - echo "O_DIRECT : DISABLED (buffered) -- this filesystem/mount rejected" - echo " O_DIRECT, so results (especially reads) may reflect the" - echo " RAM cache rather than the underlying device. Pass" - echo " --direct to force it if you believe it should work." - fi - fi } -function fio_speedtest { +function fio_bw_mbps { + # Extract bandwidth in decimal MB/s (matching CrystalDiskMark) from a fio + # run. fio prints eg " READ: bw=3357MiB/s (3521MB/s), ..." -- we take the + # parenthetical SI figure and normalise kB/MB/GB into MB/s. + local out="$1" + local raw num unit + raw=$(echo "$out" | /usr/bin/grep -oP 'bw=\S+\s+\(\K[^)]+' | head -n1) + [ -z "$raw" ] && { echo "0.00"; return; } + num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1) + unit=$(echo "$raw" | /usr/bin/grep -oP '[A-Za-z]+/s' | head -n1) + case "$unit" in + B/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000000}' ;; + kB/s|KB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n/1000}' ;; + MB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;; + GB/s) awk -v n="$num" 'BEGIN{printf "%.2f", n*1000}' ;; + *) awk -v n="$num" 'BEGIN{printf "%.2f", n}' ;; + esac +} + +function fio_iops { + # Extract IOPS from a fio run. fio prints eg " read: IOPS=12.3k, BW=..." -- + # the value may carry a k/M suffix, which we expand to a whole number. + local out="$1" + local raw num unit + raw=$(echo "$out" | /usr/bin/grep -oP 'IOPS=\K[0-9.]+[kKmM]?' | head -n1) + [ -z "$raw" ] && { echo "0"; return; } + num=$(echo "$raw" | /usr/bin/grep -oP '[0-9.]+' | head -n1) + unit=$(echo "$raw" | /usr/bin/grep -oP '[kKmM]' | head -n1) + case "$unit" in + k|K) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000}' ;; + m|M) awk -v n="$num" 'BEGIN{printf "%.0f", n*1000000}' ;; + *) awk -v n="$num" 'BEGIN{printf "%.0f", n}' ;; + esac +} + +function run_fio { + # run_fio -> echoes "MB/s IOPS" (space-sep) + # A single shared 1GB test file is reused across every run (laid out once), + # which keeps the footprint at one file and avoids re-creating it each time. + local rw="$1" bs="$2" qd="$3" jobs="$4" + local args=( + --name=speedtest + --filename="$benchfile" + --ioengine="$engine" + --direct="$direct" + --rw="$rw" + --bs="$bs" + --size="$size" + --numjobs="$jobs" + --iodepth="$qd" + --time_based --runtime="$runtime" + --group_reporting=1 + ) + # Flush device cache at the end of write tests so cached writes can't lie. + case "$rw" in + *write) args+=(--end_fsync=1) ;; + esac + # Capture the full fio output. In verbose mode we also keep fio's stderr + # (2>&1); otherwise we discard it. Either way only the parsed bandwidth is + # echoed to stdout, so the captured value stays clean. + local out + if [ "$verbose" = 1 ]; then + out=$(sudo fio "${args[@]}" 2>&1) + else + out=$(sudo fio "${args[@]}" 2>/dev/null) + fi + # Dump the raw fio output (to stderr, so it never pollutes the captured + # values on stdout) when running verbose. + if [ "$verbose" = 1 ]; then + echo " --- fio: $rw bs=$bs iodepth=$qd numjobs=$jobs ---" >&2 + echo "$out" >&2 + echo "" >&2 + fi + echo "$(fio_bw_mbps "$out") $(fio_iops "$out")" +} + +# Table drawing ------------------------------------------------------------- +# Label cell is "| %-16s |" (18 dashes); the four numeric cells are "| %14s |" +# (16 dashes each). +TBL_SEG_LBL="------------------" # 18 dashes -> matches "| %-16s |" +TBL_SEG_NUM="----------------" # 16 dashes -> matches "| %14s |" +function tbl_rule { + printf "+%s+%s+%s+%s+%s+\n" "$TBL_SEG_LBL" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM" "$TBL_SEG_NUM" +} +function tbl_row { + # tbl_row + # (col1 left-aligned, the four numeric cols right-aligned) + printf "| %-16s | %14s | %14s | %14s | %14s |\n" "$1" "$2" "$3" "$4" "$5" +} + +# Five-column table for the --slog latency profile. +# "| %-16s |" -> 18 dashes ; "| %12s |" -> 14 dashes +SLOG_SEG16="------------------" # 18 dashes +SLOG_SEG12="--------------" # 14 dashes +function slog_rule { + printf "+%s+%s+%s+%s+%s+\n" "$SLOG_SEG16" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12" "$SLOG_SEG12" +} +function slog_row { + # slog_row (col1 left, rest right-aligned) + printf "| %-16s | %12s | %12s | %12s | %12s |\n" "$1" "$2" "$3" "$4" "$5" +} + +function run_fio_sync { + # run_fio_sync + # -> echoes "IOPS MB/s p50_us p99_us mean_us" (one space-separated line) + # Forces synchronous IO (--sync=1 => O_SYNC) with the portable psync engine so + # every write is a ZIL commit -- i.e. it exercises the SLOG exactly the way a + # sync=always dataset (NFS/iSCSI/VM) does, regardless of the dataset's own + # sync property. We deliberately DON'T use --group_reporting here: each job is + # reported independently in the JSON and we aggregate ourselves (sum IOPS/bw, + # average p50, take the worst p99), which sidesteps fio's group-merge quirks. + local bs="$1" jobs="$2" + local args=( + --name=slog + --filename="$benchfile" + --ioengine=psync + --rw=randwrite + --bs="$bs" + --size="$size" + --numjobs="$jobs" + --sync=1 + --time_based --runtime="$runtime" + --output-format=json + ) + local out + out=$(sudo fio "${args[@]}" 2>/dev/null) + if [ "$verbose" = 1 ]; then + echo " --- fio json: randwrite bs=$bs numjobs=$jobs sync=1 ---" >&2 + echo "$out" >&2 + echo "" >&2 + fi + # Parse the JSON: sum IOPS and bandwidth across jobs, aggregate completion + # latency percentiles (clat is in ns -> convert to us). bw_bytes is bytes/s. + echo "$out" | python3 -c ' +import json, sys +try: + d = json.load(sys.stdin) + jobs = d["jobs"] +except Exception: + print("0 0.00 0.0 0.0 0.0"); sys.exit(0) +iops = sum(j["write"]["iops"] for j in jobs) +mbps = sum(j["write"]["bw_bytes"] for j in jobs) / 1e6 +def pctl(j, p): + return j["write"]["clat_ns"]["percentile"].get(p) +p50 = [pctl(j, "50.000000") for j in jobs if pctl(j, "50.000000") is not None] +p99 = [pctl(j, "99.000000") for j in jobs if pctl(j, "99.000000") is not None] +mean = [j["write"]["clat_ns"]["mean"] for j in jobs] +p50_us = (sum(p50)/len(p50))/1000.0 if p50 else 0.0 # average of per-job medians +p99_us = (max(p99))/1000.0 if p99 else 0.0 # worst-case tail across jobs +mean_us = (sum(mean)/len(mean))/1000.0 if mean else 0.0 +print("%.0f %.2f %.1f %.1f %.1f" % (iops, mbps, p50_us, p99_us, mean_us)) +' +} + +function slog_speedtest { + # SLOG / sync-write latency profile. Detect engine/direct only for the banner; + # the actual runs always force psync + O_SYNC (see run_fio_sync). + detect_io_settings + benchfile="$path/speedtest-hd.bench" + + if ! command -v fio &> /dev/null; then + echo "ERROR: --slog requires fio (apt install fio / pacman -S fio)." >&2 + exit 1 + fi + if ! command -v python3 &> /dev/null; then + echo "ERROR: --slog requires python3 (used to parse fio JSON latency percentiles)." >&2 + exit 1 + fi + + echo "" + echo " speedtest-hd : SLOG / sync-write latency profile" + echo " Target : $path" + echo " Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine" + echo " Profile : runtime=${runtime}s/run size=${size^^}" + echo " Note : every write is a synchronous ZIL commit -- this is the load" + echo " your SLOG actually sees. Watch it live in another shell with:" + echo " zpool iostat -vl 1" + echo "" + + # T1 is the headline single-stream latency; the sweep shows how the SLOG + # scales as concurrent sync writers (NFS/iSCSI/VM threads) pile on. Run every + # measurement FIRST (progress goes to stderr), collect the results, then draw + # the whole table in one block so the "measuring..." lines can't interleave + # into the middle of the table. + local jobs_list="1 4 8 16" + local -a rows + local j res iops bw p50 p99 mean + for j in $jobs_list; do + echo " measuring 4K sync randwrite T$j ..." >&2 + res=$(run_fio_sync 4k "$j") + read -r iops bw p50 p99 mean <<< "$res" + rows+=("$(slog_row "4K sync T$j" "$iops" "$bw" "$p50" "$p99")") + done + + rm -f "$benchfile" + + # Render the assembled table + echo "" + slog_rule + slog_row "Test" "IOPS" "MB/s" "p50 lat(us)" "p99 lat(us)" + slog_rule + printf '%s\n' "${rows[@]}" + slog_rule + echo "" + echo " Healthy Optane SLOG (eg P1600X) single-stream (T1) target:" + echo " ~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means" + echo " CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling." + echo "" +} + +function cdm_speedtest { # Auto-detect the best engine and whether O_DIRECT works on this path. detect_io_settings + benchfile="$path/speedtest-hd.bench" - # Write tests - fio_write_single_random_4k - fio_write_parallel_random_64k - fio_write_single_sequential_1m + local direct_label="enabled (device)" + [ "$direct" != 1 ] && direct_label="DISABLED (buffered -- may reflect RAM cache!)" - # Read Tests - fio_read_sequential_1m - fio_read_random_4k + # Banner + echo "" + echo " speedtest-hd : CrystalDiskMark-style storage benchmark" + echo " Target : $path" + echo " Engine : $engine O_DIRECT: $direct_label" + echo " Profile : size=${size^^} runtime=${runtime}s/run (8 runs)" + echo "" + + # The four CrystalDiskMark default tests. Q = queue depth (--iodepth), + # T = threads (--numjobs). The first three are T1 (single thread); the last + # is T16. A single shared test file is reused -- with numjobs>1 every job + # issues IO against the same file, which is fine. + # Each run now returns both MB/s and IOPS ("bw iops"), so we capture the pair. + # label read-rw write-rw bs Q T + echo " measuring SEQ1M Q8T1 ..." >&2 + local s8r_bw s8r_io; read -r s8r_bw s8r_io <<< "$(run_fio read 1m 8 1)" + local s8w_bw s8w_io; read -r s8w_bw s8w_io <<< "$(run_fio write 1m 8 1)" + echo " measuring SEQ1M Q1T1 ..." >&2 + local s1r_bw s1r_io; read -r s1r_bw s1r_io <<< "$(run_fio read 1m 1 1)" + local s1w_bw s1w_io; read -r s1w_bw s1w_io <<< "$(run_fio write 1m 1 1)" + echo " measuring RND4K Q32T1 ..." >&2 + local r32r_bw r32r_io; read -r r32r_bw r32r_io <<< "$(run_fio randread 4k 32 1)" + local r32w_bw r32w_io; read -r r32w_bw r32w_io <<< "$(run_fio randwrite 4k 32 1)" + echo " measuring RND4K Q32T16 ..." >&2 + local r1r_bw r1r_io; read -r r1r_bw r1r_io <<< "$(run_fio randread 4k 32 16)" + local r1w_bw r1w_io; read -r r1w_bw r1w_io <<< "$(run_fio randwrite 4k 32 16)" + + # Cleanup the shared test file + rm -f "$benchfile" + + # Render the CrystalDiskMark-style table + echo "" + tbl_rule + tbl_row "Test" "Read (MB/s)" "Write (MB/s)" "Read (IOPS)" "Write (IOPS)" + tbl_rule + tbl_row "SEQ1M Q8T1" "$s8r_bw" "$s8w_bw" "$s8r_io" "$s8w_io" + tbl_row "SEQ1M Q1T1" "$s1r_bw" "$s1w_bw" "$s1r_io" "$s1w_io" + tbl_row "RND4K Q32T1" "$r32r_bw" "$r32w_bw" "$r32r_io" "$r32w_io" + tbl_row "RND4K Q32T16" "$r1r_bw" "$r1w_bw" "$r1r_io" "$r1w_io" + tbl_rule + echo "" } function dd_speedtest { - # Basic HD speed test using DD + # Basic HD speed test using dd (fallback when fio is not installed) # mReschke 2017-07-11 - + local file ddsize file=$path/bigfile - size=1024 + ddsize=1024 echo "Running dd based HD/SSD/NVMe Benchmarks" echo "---------------------------------------" printf "Cached write speed...\n" - dd if=/dev/zero of=$file bs=1M count=$size + dd if=/dev/zero of=$file bs=1M count=$ddsize printf "\nUncached write speed...\n" - dd if=/dev/zero of=$file bs=1M count=$size conv=fdatasync,notrunc + dd if=/dev/zero of=$file bs=1M count=$ddsize conv=fdatasync,notrunc printf "\nUncached read speed...\n" echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null - dd if=$file of=/dev/null bs=1M count=$size + dd if=$file of=/dev/null bs=1M count=$ddsize printf "\nCached read speed...\n" - dd if=$file of=/dev/null bs=1M count=$size + dd if=$file of=/dev/null bs=1M count=$ddsize rm $file printf "\nDone\n" @@ -411,40 +411,45 @@ function dd_speedtest { # Show help and usage information function usage { - echo "Robust Flexible Input/Output HD Speedtest" - echo " If FIO is installed, we use FIO for more detailed performance analysis." - echo " If FIO is not installed, we use basic DD analysis." - echo " You should apt install fio (pacman -S fio) for detailed analysis." + echo "speedtest-hd : CrystalDiskMark-style storage benchmark" + echo " Runs the same four tests as CrystalDiskMark (each Read + Write):" + echo " SEQ1M Q8T1 Sequential 1MiB, queue depth 8, 1 thread" + echo " SEQ1M Q1T1 Sequential 1MiB, queue depth 1, 1 thread" + echo " RND4K Q32T1 Random 4KiB, queue depth 32, 1 thread" + echo " RND4K Q32T16 Random 4KiB, queue depth 32, 16 threads" + echo " Uses fio if installed (recommended), else basic dd." echo "mReschke 2024-01-18" echo "" - echo "NOTICE, this creates a 1GB file on the desired destination disk." - echo "Please ensure you have write access with 1GB free space on destination." + echo "NOTICE: creates one test file (default 1GB) on the destination disk." + echo "Please ensure you have write access with enough free space." echo "" echo "Usage:" - echo " This will use FIO if installed, else DD" + echo " Auto (fio if installed, else dd):" echo " ./speedtest-hd /mnt/somedisk" echo " ./speedtest-hd ." echo "" - echo " This will force FIO" + echo " Force fio / force dd:" echo " ./speedtest-hd /mnt/somedisk --fio" - echo " ./speedtest-hd . --fio" - echo "" - echo " This will force DD" echo " ./speedtest-hd /mnt/somedisk --dd" - echo " ./speedtest-hd . --dd" echo "" - echo " Add --simple (FIO only) for a compact, aligned summary of MB/s values" - echo " ./speedtest-hd . --simple" - echo " ./speedtest-hd . --fio --simple" + echo " SLOG / sync-write latency profile (ZFS ZIL, NFS/iSCSI/VM sync IO):" + echo " ./speedtest-hd /mnt/zpool/dataset --slog" + echo " 4K synchronous writes at T1/T4/T8/T16; reports IOPS, MB/s, p50/p99" + echo " commit latency. Requires fio + python3. (Tip: watch the SLOG with" + echo " 'zpool iostat -vl 1' in another shell.)" echo "" - echo " FIO tuning flags (all auto-detected by default, override as needed):" + echo " Tuning flags (auto-detected by default, override as needed):" echo " --engine=io_uring|libaio|posixaio|sync IO engine (default: auto)" echo " --direct Force O_DIRECT (bypass page cache)" echo " --buffered Force buffered IO (eg if O_DIRECT is unsupported)" - echo " --runtime=SEC Seconds per test (default: 30)" + echo " --runtime=SEC Seconds per run (default: 5, like CrystalDiskMark)" + echo " --size=SIZE Test file size (default: 1g)" + echo " --verbose Print full fio output for every run (summary table unchanged)" echo " Examples:" - echo " ./speedtest-hd /mnt/nvmepool --engine=io_uring --runtime=60" + echo " ./speedtest-hd /mnt/nvmepool --runtime=10 --size=4g" echo " ./speedtest-hd /mnt/nfsshare --buffered" + echo " ./speedtest-hd /mnt/nvmepool --verbose" + echo " ./speedtest-hd /mnt/nvme-ultra-r10/vm-root --slog --runtime=30" exit 0 }