243 lines
12 KiB
Markdown
243 lines
12 KiB
Markdown
# Notes
|
||
|
||
```
|
||
latest full nvme tests
|
||
|
||
always
|
||
+---------------+-------------+--------------+-------------+--------------+
|
||
| Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) |
|
||
+---------------+-------------+--------------+-------------+--------------+
|
||
| SEQ1M Q8T1 | 10635.20 | 62.69 | 10143 | 60 |
|
||
| SEQ1M Q1T1 | 2505.60 | 100.53 | 2390 | 96 |
|
||
| RND4K Q32T1 | 542.36 | 21.57 | 132411 | 5267 |
|
||
| RND4K Q32T16 | 877.08 | 337.07 | 214132 | 82292 |
|
||
+---------------+-------------+--------------+-------------+--------------+
|
||
|
||
Sync=Disabled
|
||
+---------------+-------------+--------------+-------------+--------------+
|
||
| Test | Read (MB/s) | Write (MB/s) | Read (IOPS) | Write (IOPS) |
|
||
+---------------+-------------+--------------+-------------+--------------+
|
||
| SEQ1M Q8T1 | 11173.56 | 13.46 | 10656 | 13 |
|
||
| SEQ1M Q1T1 | 2472.89 | 45.27 | 2358 | 43 |
|
||
| RND4K Q32T1 | 548.09 | 234.05 | 133810 | 57142 |
|
||
| RND4K Q32T16 | 1003.89 | 1498.63 | 245090 | 365877 |
|
||
+---------------+-------------+--------------+-------------+--------------+
|
||
|
||
|
||
|
||
|
||
|
||
|
||
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# cat /proc/cmdline
|
||
BOOT_IMAGE=/ROOT/25.10.4@/boot/vmlinuz-6.12.91-production+truenas root=ZFS=boot-pool/ROOT/25.10.4 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N intel_idle.max_cstate=1 processor.max_cstate=1 pcie_aspm=off nvme_core.default_ps_max_latency_us=0
|
||
|
||
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# grep MHz /proc/cpuinfo | sort -u
|
||
cpu MHz : 1199.947
|
||
cpu MHz : 1199.997
|
||
cpu MHz : 1200.000
|
||
cpu MHz : 1200.055
|
||
cpu MHz : 1200.060
|
||
cpu MHz : 1200.063
|
||
|
||
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# lspci | grep -i optane
|
||
03:00.0 Non-Volatile memory controller: Intel Corporation Optane NVME SSD P1600X Series
|
||
|
||
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# lspci -vv -s 03:00.0 | grep -iE 'LnkSta\|ASPM'
|
||
NO output
|
||
|
||
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# nvme get-feature /dev/nvme6n1 -f 0x0c -H
|
||
get-feature:0x0c (Autonomous Power State Transition), Current value:00000000
|
||
Autonomous Power State Transition Enable (APSTE): Disabled
|
||
Auto PST Entries .................
|
||
Entry[ 0]
|
||
.................
|
||
Idle Time Prior to Transition (ITPT): 500 ms
|
||
Idle Transition Power State (ITPS): 3
|
||
.................
|
||
Entry[ 1]
|
||
.................
|
||
Idle Time Prior to Transition (ITPT): 500 ms
|
||
Idle Transition Power State (ITPS): 3
|
||
.................
|
||
|
||
|
||
---
|
||
|
||
Original prompt that started it all
|
||
|
||
I have a TrueNAS Server using an Intel Optane P1600X SLOG on top of 6x4TB KingSpec XG7000 NVMe drives in a RAID 10, and when SSH into the TrueNAS server itself, and going to /mnt/nvme-ultra-r10/vm-root which is set to sync=always (using the SLOG), the speedtest-hd.sh output looks like this
|
||
|
||
+------------------+------------------+------------------+
|
||
| Test | Read (MB/s) | Write (MB/s) |
|
||
+------------------+------------------+------------------+
|
||
| SEQ1M Q8T1 | 6873.00 | 9.30 |
|
||
| SEQ1M Q1T1 | 1608.00 | 20.00 |
|
||
| RND4K Q32T1 | 538.00 | 10.80 |
|
||
| RND4K Q32T16 | 689.00 | 261.00 |
|
||
+------------------+------------------+------------------+
|
||
|
||
Which seems really slow for an Optane.
|
||
|
||
|
||
---
|
||
|
||
simple
|
||
|
||
fio --name=fio_test --filename=testfile.dat --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --iodepth=16 --numjobs=2 --size=1G --runtime=30 --time_based --direct=1 --group_reporting
|
||
|
||
|
||
CrystalDiskMark
|
||
|
||
Seq Q32T1 (Sequential Read/Write): Large blocks (usually 1 MB) transferred sequentially across 1 thread and a queue depth of 32. This mimics copying large contiguous files (like movies or ISOs) and typically yields the highest speed ratings.
|
||
|
||
4K Q32T1 (Random 4K Read/Write): 4 KB random blocks across 1 thread and a queue depth of 32. This heavily tests the drive’s IOPS and controller capability for handling simultaneous, small, multi-threaded requests.
|
||
|
||
Seq Q1T1 (Sequential Single-Queue): Large 1 MB blocks transferred with a queue depth of 1 and 1 thread. Represents a basic, single-task file transfer.
|
||
|
||
4K Q1T1 (Random 4K Single-Thread): 4 KB random blocks using 1 thread and a queue depth of 1. This represents typical day-to-day OS performance and random single-file access.
|
||
|
||
|
||
|
||
|
||
Understood — and good news: this change is fully in the spirit of CrystalDiskMark, which itself uses parallel deep-queue tests (its RND4K test is Q32×T16 = 512 in flight). 4 jobs × QD64 is generic and correct for any drive — on an HDD it'll just report the honestly-low number, which is what you want. Re-applying:
|
||
|
||
|
||
What types of tests does CrystalDiskMark run? Does it do Seq Q32T1 (Sequential Read/Write), 4K Q32T1 (Random 4K Read/Write), Seq Q1T1 (Sequential Single-Queue) 4K Q1T1 (Random 4K Single-Thread), or something else? The UI shows SEQ1M Q8T1, SEQ1M Q1T1, RND4k Q32T1 and RND4k Q1T1. Can we refactor this entire speedtest-hd.sh to run the exact same test CrystalDiskMark runs. Can we also layout the output with those tests on the left the colums for Read (MB/s) and Write (MB/s) just like their UI does? At the same time I think you can REMOVE the --simple option and just output the script in simple CrystalDiskMark table style always. If you can make the table pretty in ASCII using dashes, underscores, pipes etc...
|
||
|
||
|
||
fix...
|
||
|
||
So Q = queue depth → fio --iodepth, and T = threads → fio --numjobs. The older "Seq Q32T1 / 4K Q8T8" layout you mentioned was CDM 5/6; they replaced it with the SEQ1M / RND4K scheme above. (One note: some CDM builds/profiles use RND4K Q32T16 — 16 threads — for the NVMe "peak" profile, but your UI shows Q32T1, so I'll match
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
when I disabled sync and ran numjobs=1
|
||
WRITE: bw=56.0MiB/s (58.7MB/s), 56.0MiB/s-56.0MiB/s (58.7MB/s-58.7MB/s), io=1680MiB (1761MB), run=30001-30001msec
|
||
|
||
back to sync
|
||
before power tweaks
|
||
WRITE: bw=9630KiB/s (9862kB/s), 9630KiB/s-9630KiB/s (9862kB/s-9862kB/s), io=282MiB (296MB), run=30001-30001msec
|
||
|
||
set to performance mode
|
||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||
for s in /sys/devices/system/cpu/cpu*/cpuidle/state[2-9]/disable; do echo 1 | tee $s; done
|
||
WRITE: bw=13.8MiB/s (14.5MB/s), 13.8MiB/s-13.8MiB/s (14.5MB/s-14.5MB/s), io=415MiB (435MB), run=30001-30001msec
|
||
|
||
|
||
Bottom line: your Optane and pool are fine and the SLOG is correctly carrying the sync load. The disappointing numbers are CPU C-states and PCIe/NVMe power management adding ~300 µs of wake-up latency to every QD1 sync commit. Disable deep idle states and ASPM, set the performance governor, and re-test — single-thread 4K sync should climb from ~3K IOPS toward 15–25K. Want me to add a --slog/sync-latency test mode to speedtest-hd.sh so you can re-measure this properly (with latency percentiles, not just bandwidth) after each change?
|
||
|
||
|
||
Kernal tweaks
|
||
|
||
In System → Advanced Settings → Kernel Arguments, add (then reboot):
|
||
|
||
intel_idle.max_cstate=1 processor.max_cstate=1 pcie_aspm=off nvme_core.default_ps_max_latency_us=0
|
||
|
||
|
||
*_cstate=1 keeps the CPU responsive (kills the ~238 µs host overhead).
|
||
pcie_aspm=off + nvme_core.default_ps_max_latency_us=0 keep the Optane link awake (targets the ~90 µs device latency).
|
||
|
||
|
||
after bios=performance, but before kernel params
|
||
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) |
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
| 4K sync T1 | 5849 | 23.96 | 160.8 | 244.7 |
|
||
| 4K sync T4 | 17204 | 70.47 | 208.4 | 419.8 |
|
||
| 4K sync T8 | 29358 | 120.25 | 235.8 | 634.9 |
|
||
| 4K sync T16 | 46009 | 188.45 | 296.2 | 1056.8 |
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
|
||
Healthy Optane SLOG (eg P1600X) single-stream (T1) target:
|
||
~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means
|
||
CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling.
|
||
|
||
|
||
after kernel boot params
|
||
|
||
|
||
speedtest-hd : SLOG / sync-write latency profile
|
||
Target : /mnt/nvme-ultra-r10/vm-root
|
||
Method : fio randwrite bs=4k --sync=1 (O_SYNC), psync engine
|
||
Profile : runtime=5s/run size=1G
|
||
Note : every write is a synchronous ZIL commit -- this is the load
|
||
your SLOG actually sees. Watch it live in another shell with:
|
||
zpool iostat -vl <pool> 1
|
||
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) |
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
| 4K sync T1 | 6217 | 25.46 | 150.5 | 272.4 |
|
||
| 4K sync T4 | 17018 | 69.71 | 208.4 | 493.6 |
|
||
| 4K sync T8 | 30193 | 123.67 | 231.9 | 536.6 |
|
||
| 4K sync T16 | 47166 | 193.19 | 315.1 | 684.0 |
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
|
||
Healthy Optane SLOG (eg P1600X) single-stream (T1) target:
|
||
~15-25k IOPS, p50 latency ~40-65us. Much higher latency usually means
|
||
CPU C-states / PCIe ASPM / BIOS power profile (eg Dell DAPC) throttling.
|
||
|
||
again
|
||
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) |
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
| 4K sync T1 | 5786 | 23.70 | 162.8 | 250.9 |
|
||
| 4K sync T4 | 17106 | 70.07 | 209.4 | 391.2 |
|
||
| 4K sync T8 | 29694 | 121.62 | 234.8 | 528.4 |
|
||
| 4K sync T16 | 46697 | 191.27 | 289.5 | 888.8 |
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
|
||
|
||
NOW this fix
|
||
|
||
Claude says CPU is locked low
|
||
|
||
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# grep MHz /proc/cpuinfo | sort -u
|
||
cpu MHz : 1199.947
|
||
cpu MHz : 1199.997
|
||
cpu MHz : 1200.000
|
||
cpu MHz : 1200.055
|
||
cpu MHz : 1200.060
|
||
cpu MHz : 1200.063
|
||
|
||
|
||
FIX to performance node
|
||
|
||
# Mine was intel_cpufreq and schedutil
|
||
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver # expect: intel_pstate
|
||
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # expect: powersave
|
||
|
||
the fix
|
||
|
||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||
|
||
grep MHz /proc/cpuinfo | sort -u # should now read ~3000-3300, not 1200
|
||
|
||
Now its
|
||
|
||
root@linvault1[/mnt/nvme-ultra-r10/vm-root]# grep MHz /proc/cpuinfo | sort -u
|
||
cpu MHz : 2882.804
|
||
cpu MHz : 2894.792
|
||
cpu MHz : 2900.123
|
||
cpu MHz : 3300.000
|
||
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
| Test | IOPS | MB/s | p50 lat(us) | p99 lat(us) |
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
| 4K sync T1 | 10687 | 43.77 | 85.5 | 185.3 |
|
||
| 4K sync T4 | 29873 | 122.36 | 117.8 | 317.4 |
|
||
| 4K sync T8 | 52612 | 215.50 | 136.2 | 391.2 |
|
||
| 4K sync T16 | 77939 | 319.24 | 180.0 | 505.9 |
|
||
+------------------+--------------+--------------+--------------+--------------+
|
||
|
||
Make it stick
|
||
|
||
Make it persistent — System → Advanced Settings → Init/Shutdown Scripts, Type Command, When Post Init:
|
||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||
```
|