Booting from NVMe USB enclosure

[rych]

Does Porteus 5.0 (with the stock or updated kernel, or PorteuX) recognize and boot from an NVMe SSD in a USB 3.1 enclosure, with one of the following interface chipsets (10Gbps):

* Realtek RTL9210B or RTL9210
* JMicron JMS583 A3
* Asmedia ASM2362 or even Asmedia ASM2364 (20Gbps)

Does anyone have any experience with those? Which chipset would be better?

The enclosure will be carrying a Samsung 980 Samsung 970 EVO Plus 1TB M.2 PCIe 3.0 NVMe Internal SSD. It's on sale but still not cheap. So need some advice first. I hope this works and I get as much speed out of my USB 3.1 as possible.

The motherboard/BIOS clearly support booting the Windows off the internal system disk which is also an NVMe PCIe disk. So I guess it's a matter of grub2 or a porteus native boot loader seeing and capable of booting off a similar disk through the USB interface?

[Ed_P]

Sounds reasonable to me. What are you using to boot the USB drive, grub2 or some app?

[rych]

What are you using to boot the USB drive

Yes, I use grub2 instead of the native porteus boot loader. The question is which chip will work guaranteed?

(I'm leaning toward Realtek RTL9210B. The newer Asmedia ASM2364 will support up to 20Gbps though. Not that I have such USB ports yet. Besides, all that speed doesn't matter, if it won't boot reliably for example)

[Ed_P]

I've never tried a USB drive with the options you are considering so I can't suggest one over another. They sound impressive though and will be interested in your experience.

If you partition the drive to have a FAT32 partition you have the option for UEFI booting otherwise you're limited to BIOS booting. These are the two grub2 menus I use for booting my USB drives, which are all less then 32GB and formated as FAT32.

Code: Select all

menuentry " Porteus 5.0 USB - EFI" --class slackware   --class icon-porteus  {

     set bootmgr=/EFI/boot/bootx64.efi          #grubx64.efi    # bootx64.efi

     set bootdrv=$root
     search -f $bootmgr --set=root
     echo bootmgr: $bootmgr
     sleep -v -i 3
     if [ $root != $bootdrv ]; then
        chainloader $bootmgr
     else
        echo "----------------------------------------"
        echo USB drive NOT found.
        echo
        sleep -v -i 10
     fi
     set root=$bootdrv
     }

menuentry " Porteus 5.0 USB - AF'" --class slackware   --class icon-porteus  {

     set porteus_parms="volume=33 reboot=cold extramod=/Modules"  #;cinnamon" # changes=EXIT:/changes/porteussave.dat"

     set bootdrv=$root
     search -f /boot/syslinux/vmlinuz --set=root
     if [ $root != $bootdrv ]; then
        linux  /boot/syslinux/vmlinuz $porteus_parms
        initrd /boot/syslinux/initrd.xz
     else
        echo "----------------------------------------"
        echo USB drive NOT found.
        echo
        sleep -v -i 10
     fi
     set root=$bootdrv
     }

[rych]

OK, I've ordered the above SSD and will be choosing my enclosure next. So it'd better work. What I'm after is the 10Gbps class speed. You see below how my best-in-class (SanDisk Extreme Pro 120GB) USB disk only claims up to 5Gbps out of the available on my system USB 3.1 gen 2 hub throughput of 10Gbps?

Code: Select all

lsusb -t | egrep -i "storage" -B1
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 10000M
    |__ Port 1: Dev 2, If 0, Class=, Driver=usb-storage, 5000M

[rych]

My Realtek RTL9210B enclosure carrying a Samsung 970 EVO Plus 1TB M.2 PCIe 3.0 NVMe Internal SSD when plugged into the USB-C port (!) reports 10000M bandwidth:

Code: Select all

root@porteus:~# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 10000M
    |__ Port 1: Dev 2, If 0, Class=, Driver=usb-storage, 5000M
    |__ Port 2: Dev 8, If 0, Class=, Driver=uas, 10000M

That's a new line in that output with "Driver=uas", the preceding "usb-storage" is my ordinary USB 3.0 SanDisk Extreme Pro 120GB disk which currently runs Porteus and is only up to 5000M

Next I'm going to test some actual speeds of the USB 3, the new USB-C, and the system SSD.

[rych]

I know disk benchmarking is a science in itself but using hdparm -t to check Read speed of these 3 disks does show that the new disk (/dev/sda) outperforms tboth the old USB (fastest pendrive as it were) and the system SSD!!

1. Samsung 970 EVO Plus 1TB M.2 PCIe 3.0 NVMe (rated 3,500MB/s) in Realtek RTL9210B USB 3.1 enclosure (10Gbs)

2. SanDisk Extreme Pro (5Gbs, upto 420MB/s)

3. System's SSD, NVMe SK hynix (upto 2200 MB/s)

Code: Select all

root@porteus:~# hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads: 2502 MB in  3.00 seconds = 833.99 MB/sec
root@porteus:~# hdparm -t /dev/sdb

/dev/sdb:
 Timing buffered disk reads: 1136 MB in  3.00 seconds = 378.36 MB/sec
root@porteus:~# hdparm -t /dev/nvme0n1p3

/dev/nvme0n1p3:
 Timing buffered disk reads: 1798 MB in  3.00 seconds = 598.74 MB/sec

See also: https://www.everythingusb.com/speed.html

[rych]

Code: Select all

root@porteus:~# hdparm -t --direct /dev/sda

/dev/sda:
 Timing O_DIRECT disk reads: 2214 MB in  3.00 seconds = 737.83 MB/sec
root@porteus:~# hdparm -t --direct /dev/sdb

/dev/sdb:
 Timing O_DIRECT disk reads: 1038 MB in  3.00 seconds = 345.96 MB/sec
root@porteus:~# hdparm -t --direct /dev/nvme0n1p3

/dev/nvme0n1p3:
 Timing O_DIRECT disk reads: 1916 MB in  3.00 seconds = 638.55 MB/sec

[rych]

High speed may cause critical temperatures. I didn't apply a thermal pad to the NVMe before inserting it into the enclosure relying on the thin air. So l'm going to be checking the temperature:

Code: Select all

root@porteus:~# smartctl -x -d sntrealtek /dev/disk/by-label/PorteuX | grep Temp*
Warning  Comp. Temp. Threshold:     85 Celsius
Critical Comp. Temp. Threshold:     85 Celsius
Temperature:                        30 Celsius
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               30 Celsius
Temperature Sensor 2:               33 Celsius

[rych]

This setup is now stable (and fast!). I had to update the controller firmware with one important setting to completely disable the sleep/standby/poweroff of the idle NVMe SSD disk. Usually a desirable feature in hope to save some (small) power and to keep temperature low (it's already low); but for our purposes of running Porteus off this enclosure we actually want to prevent it from going to sleep in order to avoid stability issues: on some machines, when it goes into sleep, it often can't wake up properly leading to the system freeze with the only terrible remedy being the button-restarting the computer. Now it's fine and I can recommend this setup as the fastest/largest/affordable USB Porteus system.

[rych]

I spoke too soon. The mundane USB cable that came with the enclosure quickly wore out and started only allowing USB 2 speeds suddenly. I had to invest in a good thick and short 20Gbps USB-C 3.2 cable at about the same price as the enclosure itself, so no wonder. Now it's back to its max speed and reliability. One of the cable ends is L-shape plugging nicely along the enclosure side instead of wildly sticking out of it perpendicularly.

[rych]

Other ways to test read-write speeds: https://www.baeldung.com/linux/disk-performance-test

Code: Select all

dd if=/dev/zero of=/tmp/tempfile bs=1M count=1024 conv=fdatasync
sudo sh -c "/usr/bin/echo 3 > /proc/sys/vm/drop_caches"
dd if=/tmp/tempfile of=/dev/null bs=1M count=1024

[Rapha_]

Other ways to test read-write speeds: https://www.baeldung.com/linux/disk-performance-test

Code: Select all

dd if=/dev/zero of=/tmp/tempfile bs=1M count=1024 conv=fdatasync
sudo sh -c "/usr/bin/echo 3 > /proc/sys/vm/drop_caches"
dd if=/tmp/tempfile of=/dev/null bs=1M count=1024

Very interesting !

[rych]

My own tests give this

Here are my tests on this machine. Both the host system disk and the USB enclosed disk are NVMe SSD. The partitions are:
sda3: ext4 on USB
sda4: NTFS on USB
nvme0n1p3: NTFS host system

The NTFS driver is the FUSE ntfs-3g, hence perhaps slower than expected speeds. Unless that's the limitation of my system disk, see the bottom of this post. Anyways, will be interesting to compare when I learn how to remount my NTFS partitions with the kernel driver ntfs3.

Writing:

Code: Select all

dd if=/dev/zero of=/tmp/tempfile bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.299371 s, 3.6 GB/s
dd if=/dev/zero of=/mnt/sda3/tempfile bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.87841 s, 572 MB/s
dd if=/dev/zero of=/mnt/sda4/tempfile bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.21219 s, 485 MB/s
dd if=/dev/zero of=/mnt/nvme0n1p3/tempfile bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.51177 s, 427 MB/s

Reading:

Code: Select all

sudo sh -c "/usr/bin/echo 3 > /proc/sys/vm/drop_caches"
dd if=/tmp/tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.126275 s, 8.5 GB/s
dd if=/mnt/sda3/tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.35096 s, 795 MB/s
dd if=/mnt/sda4/tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.7413 s, 226 MB/s
dd if=/mnt/nvme0n1p3/tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.11127 s, 966 MB/s

Reading with hdparm (reads off the device, not a partition, either --direct or "buffered")

Code: Select all

hdparm -t --direct /dev/sda

/dev/sda:
 Timing O_DIRECT disk reads: 2182 MB in  3.00 seconds = 726.90 MB/sec
hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads: 2300 MB in  3.05 seconds = 755.00 MB/sec
hdparm -t --direct /dev/nvme0n1

/dev/nvme0n1:
 Timing O_DIRECT disk reads: 1998 MB in  3.00 seconds = 665.79 MB/sec
hdparm -t /dev/nvme0n1

/dev/nvme0n1:
 Timing buffered disk reads: 1856 MB in  3.00 seconds = 618.38 MB/sec

My system disk is BC511 NVMe SK hynix 256GB, with the advertised speeds "up to 2000 MB/s read and 1000 MB/s write" or according to https://www.harddrivebenchmark.net/hdd. ... B&id=24728 :
Sequential Read 1,382 MBytes/Sec
Sequential Write 579 MBytes/Sec

[rych]

I'm on a slightly different machine today. The system disk is a SATA SSD with an NTFS partition on sda3. The USB enclosed NVMe SSD is on sdb3 for ext4 partition, and sdb4 -- NTFS partition. The USB port is 5Gbit/s this time, so twice slower throughput than the 10Gbit/s USB.

We'll pipeline the output to the last line only (or, `2> >(grep copied)`)

Writing:

Code: Select all

dd if=/dev/zero of=/tmp/tempfile bs=1M count=1024 conv=fdatasync |& tail -1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.33387 s, 3.2 GB/s
dd if=/dev/zero of=/mnt/sda3/tempfile bs=1M count=1024 conv=fdatasync |& tail -1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.8858 s, 67.6 MB/s
dd if=/dev/zero of=/mnt/sdb3/tempfile bs=1M count=1024 conv=fdatasync |& tail -1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.49062 s, 431 MB/s
dd if=/dev/zero of=/mnt/sdb4/tempfile bs=1M count=1024 conv=fdatasync |& tail -1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.60756 s, 412 MB/s

Reading after the mandatory

Code: Select all

sudo sh -c "/usr/bin/echo 3 > /proc/sys/vm/drop_caches"

:

Code: Select all

dd if=/tmp/tempfile of=/dev/null bs=1M count=1024 |& tail -1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.169581 s, 6.3 GB/s
dd if=/mnt/sda3/tempfile of=/dev/null bs=1M count=1024 |& tail -1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.09434 s, 513 MB/s
dd if=/mnt/sdb3/tempfile of=/dev/null bs=1M count=1024 |& tail -1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.40742 s, 446 MB/s
dd if=/mnt/sdb4/tempfile of=/dev/null bs=1M count=1024 |& tail -1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.42216 s, 127 MB/s

Code: Select all

hdparm -t /dev/sda3
/dev/sda3:
 Timing buffered disk reads: 1352 MB in  3.00 seconds = 450.25 MB/sec
hdparm -t /dev/sdb3

/dev/sdb3:
 Timing buffered disk reads: 1196 MB in  3.00 seconds = 398.64 MB/sec

Looks like

1. writing on the system SSD NTFS is much slower than expected

2. reading from an NTFS partition on the USB is much slower than reading from the adjacent ext4 partition


(By the way, dd has `oflag=direct` and `iflag=direct` options to bypass caching?)