12 KiB
12 KiB
Notes
latest full nvme tests
always
+---------------+-------------+--------------+-------------+--------------+
| Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) |
+---------------+-------------+--------------+-------------+--------------+
| SEQ1M Q8T1 | 10635.20 | 62.69 | 10143 | 60 |
| SEQ1M Q1T1 | 2505.60 | 100.53 | 2390 | 96 |
| RND4K Q32T1 | 542.36 | 21.57 | 132411 | 5267 |
| RND4K Q32T16 | 877.08 | 337.07 | 214132 | 82292 |
+---------------+-------------+--------------+-------------+--------------+
Sync=Disabled
+---------------+-------------+--------------+-------------+--------------+
| Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) |
+---------------+-------------+--------------+-------------+--------------+
| SEQ1M Q8T1 | 11173.56 | 13.46 | 10656 | 13 |
| SEQ1M Q1T1 | 2472.89 | 45.27 | 2358 | 43 |
| RND4K Q32T1 | 548.09 | 234.05 | 133810 | 57142 |
| RND4K Q32T16 | 1003.89 | 1498.63 | 245090 | 365877 |
+---------------+-------------+--------------+-------------+--------------+
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# cat /proc/cmdline
BOOT_IMAGE=/ROOT/25.10.4@/boot/vmlinuz-6.12.91-production+truenas root=ZFS=boot-pool/ROOT/25.10.4 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N intel_idle.max_cstate=1 processor.max_cstate=1 pcie_aspm=off nvme_core.default_ps_max_latency_us=0
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# grep MHz /proc/cpuinfo | sort -u
cpu MHz : 1199.947
cpu MHz : 1199.997
cpu MHz : 1200.000
cpu MHz : 1200.055
cpu MHz : 1200.060
cpu MHz : 1200.063
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# lspci | grep -i optane
03:00.0 Non-Volatile memory controller: Intel Corporation Optane NVME SSD P1600X Series
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# lspci -vv -s 03:00.0 | grep -iE 'LnkSta\|ASPM'
NO output
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# nvme get-feature /dev/nvme6n1 -f 0x0c -H
get-feature:0x0c (Autonomous Power State Transition), Current value:00000000
Autonomous Power State Transition Enable (APSTE): Disabled
Auto PST Entries .................
Entry[ 0]
.................
Idle Time Prior to Transition (ITPT): 500 ms
Idle Transition Power State (ITPS): 3
.................
Entry[ 1]
.................
Idle Time Prior to Transition (ITPT): 500 ms
Idle Transition Power State (ITPS): 3
.................
---
Original prompt that started it all
I have a TrueNAS Server using an Intel Optane P1600X SLOG on top of 6x4TB KingSpec XG7000 NVMe drives in a RAID 10, and when SSH into the TrueNAS server itself, and going to /mnt/nvme-ultra-r10/vm-root which is set to sync=always (using the SLOG), the speedtest-hd.sh output looks like this
+------------------+------------------+------------------+
| Test | Read (MB/s) | Write (MB/s) |
+------------------+------------------+------------------+
| SEQ1M Q8T1 | 6873.00 | 9.30 |
| SEQ1M Q1T1 | 1608.00 | 20.00 |
| RND4K Q32T1 | 538.00 | 10.80 |
| RND4K Q32T16 | 689.00 | 261.00 |
+------------------+------------------+------------------+
Which seems really slow for an Optane.
---
simple
fio --name=fio_test --filename=testfile.dat --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --iodepth=16 --numjobs=2 --size=1G --runtime=30 --time_based --direct=1 --group_reporting
CrystalDiskMark
Seq Q32T1 (Sequential Read/Write): Large blocks (usually 1 MB) transferred sequentially across 1 thread and a queue depth of 32. This mimics copying large contiguous files (like movies or ISOs) and typically yields the highest speed ratings.
4K Q32T1 (Random 4K Read/Write): 4 KB random blocks across 1 thread and a queue depth of 32. This heavily tests the drive’s IOPS and controller capability for handling simultaneous, small, multi-threaded requests.
Seq Q1T1 (Sequential Single-Queue): Large 1 MB blocks transferred with a queue depth of 1 and 1 thread. Represents a basic, single-task file transfer.
4K Q1T1 (Random 4K Single-Thread): 4 KB random blocks using 1 thread and a queue depth of 1. This represents typical day-to-day OS performance and random single-file access.
Understood — and good news: this change is fully in the spirit of CrystalDiskMark, which itself uses parallel deep-queue tests (its RND4K test is Q32×T16 = 512 in flight). 4 jobs × QD64 is generic and correct for any drive — on an HDD it'll just report the honestly-low number, which is what you want. Re-applying:
What types of tests does CrystalDiskMark run? Does it do Seq Q32T1 (Sequential Read/Write), 4K Q32T1 (Random 4K Read/Write), Seq Q1T1 (Sequential Single-Queue) 4K Q1T1 (Random 4K Single-Thread), or something else? The UI shows SEQ1M Q8T1, SEQ1M Q1T1, RND4k Q32T1 and RND4k Q1T1. Can we refactor this entire speedtest-hd.sh to run the exact same test CrystalDiskMark runs. Can we also layout the output with those tests on the left the colums for Read (MB/s) and Write (MB/s) just like their UI does? At the same time I think you can REMOVE the --simple option and just output the script in simple CrystalDiskMark table style always. If you can make the table pretty in ASCII using dashes, underscores, pipes etc...
fix...
So Q = queue depth → fio --iodepth, and T = threads → fio --numjobs. The older "Seq Q32T1 / 4K Q8T8" layout you mentioned was CDM 5/6; they replaced it with the SEQ1M / RND4K scheme above. (One note: some CDM builds/profiles use RND4K Q32T16 — 16 threads — for the NVMe "peak" profile, but your UI shows Q32T1, so I'll match
---
when I disabled sync and ran numjobs=1
WRITE: bw=56.0MiB/s (58.7MB/s), 56.0MiB/s-56.0MiB/s (58.7MB/s-58.7MB/s), io=1680MiB (1761MB), run=30001-30001msec
back to sync
before power tweaks
WRITE: bw=9630KiB/s (9862kB/s), 9630KiB/s-9630KiB/s (9862kB/s-9862kB/s), io=282MiB (296MB), run=30001-30001msec
set to performance mode
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
for s in /sys/devices/system/cpu/cpu*/cpuidle/state[2-9]/disable; do echo 1 | tee $s; done
WRITE: bw=13.8MiB/s (14.5MB/s), 13.8MiB/s-13.8MiB/s (14.5MB/s-14.5MB/s), io=415MiB (435MB), run=30001-30001msec
Bottom line: your Optane and pool are fine and the SLOG is correctly carrying the sync load. The disappointing numbers are CPU C-states and PCIe/NVMe power management adding ~300 µs of wake-up latency to every QD1 sync commit. Disable deep idle states and ASPM, set the performance governor, and re-test — single-thread 4K sync should climb from ~3K IOPS toward 15–25K. Want me to add a --slog/sync-latency test mode to speedtest-hd.sh so you can re-measure this properly (with latency percentiles, not just bandwidth) after each change?
Kernal tweaks
midclt call system.advanced.config
midclt call system.advanced.update '{"kernel_extra_options": "intel_idle.max_cstate=1 processor.max_cstate=1 pcie_aspm=off nvme_core.default_ps_max_latency_us=0"}'
*_cstate=1 keeps the CPU responsive (kills the ~238 µs host overhead).
pcie_aspm=off + nvme_core.default_ps_max_latency_us=0 keep the Optane link awake (targets the ~90 µs device latency).
after bios=performance, but before kernel params
+------------------+--------------+--------------+--------------+--------------+
| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) |
+------------------+--------------+--------------+--------------+--------------+
| 4K sync T1 | 5849 | 23.96 | 160.8 | 244.7 |
| 4K sync T4 | 17204 | 70.47 | 208.4 | 419.8 |
| 4K sync T8 | 29358 | 120.25 | 235.8 | 634.9 |
| 4K sync T16 | 46009 | 188.45 | 296.2 | 1056.8 |
+------------------+--------------+--------------+--------------+--------------+
Healthy Optane SLOG (eg P1600X) single-stream (T1) target:
~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means
CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling.
after kernel boot params
speedtest-hd : SLOG / sync-write latency profile
Target : /mnt/nvme-ultra-r10/vm-root
Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine
Profile : runtime=5s/run size=1G
Note : every write is a synchronous ZIL commit -- this is the load
your SLOG actually sees. Watch it live in another shell with:
zpool iostat -vl <pool> 1
+------------------+--------------+--------------+--------------+--------------+
| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) |
+------------------+--------------+--------------+--------------+--------------+
| 4K sync T1 | 6217 | 25.46 | 150.5 | 272.4 |
| 4K sync T4 | 17018 | 69.71 | 208.4 | 493.6 |
| 4K sync T8 | 30193 | 123.67 | 231.9 | 536.6 |
| 4K sync T16 | 47166 | 193.19 | 315.1 | 684.0 |
+------------------+--------------+--------------+--------------+--------------+
Healthy Optane SLOG (eg P1600X) single-stream (T1) target:
~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means
CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling.
again
+------------------+--------------+--------------+--------------+--------------+
| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) |
+------------------+--------------+--------------+--------------+--------------+
| 4K sync T1 | 5786 | 23.70 | 162.8 | 250.9 |
| 4K sync T4 | 17106 | 70.07 | 209.4 | 391.2 |
| 4K sync T8 | 29694 | 121.62 | 234.8 | 528.4 |
| 4K sync T16 | 46697 | 191.27 | 289.5 | 888.8 |
+------------------+--------------+--------------+--------------+--------------+
NOW this fix
Claude says CPU is locked low
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# grep MHz /proc/cpuinfo | sort -u
cpu MHz : 1199.947
cpu MHz : 1199.997
cpu MHz : 1200.000
cpu MHz : 1200.055
cpu MHz : 1200.060
cpu MHz : 1200.063
FIX to performance node
# Mine was intel_cpufreq and schedutil
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver # expect: intel_pstate
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # expect: powersave
the fix
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
grep MHz /proc/cpuinfo | sort -u # should now read ~3000-3300, not 1200
Now its
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# grep MHz /proc/cpuinfo | sort -u
cpu MHz : 2882.804
cpu MHz : 2894.792
cpu MHz : 2900.123
cpu MHz : 3300.000
+------------------+--------------+--------------+--------------+--------------+
| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) |
+------------------+--------------+--------------+--------------+--------------+
| 4K sync T1 | 10687 | 43.77 | 85.5 | 185.3 |
| 4K sync T4 | 29873 | 122.36 | 117.8 | 317.4 |
| 4K sync T8 | 52612 | 215.50 | 136.2 | 391.2 |
| 4K sync T16 | 77939 | 319.24 | 180.0 | 505.9 |
+------------------+--------------+--------------+--------------+--------------+
Make it stick
Make it persistent — System → Advanced Settings → Init/Shutdown Scripts, Type Command, When Post Init:
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor