NTFS3 Kernel Module

[rych]

OK, I've made those replacements, and built the new initrd with

Code: Select all

root@porteus:/P/initrd-mods/initrd501# /opt/porteus-scripts/makeinitrd -z

... renamed and copied to /boot, modified grub.cfg and rebooted: the system disk is mounted with ntfs3 now according to cat /proc/mounts and findmnt (/etc/fstab still lists them as ntfs-3g hot plugged mounts -- will fix /etc/fstab later)

Testing the speed with dd.

Writing:

Code: Select all

dd if=/dev/zero of=/mnt/nvme0n1p3/tempfile bs=1M count=1024 conv=fdatasync |& tail -1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.84128 s, 583 MB/s

was: 427 MB/s

Reading:

Code: Select all

sudo sh -c "/usr/bin/echo 3 > /proc/sys/vm/drop_caches"
root@porteus:~# dd if=/mnt/nvme0n1p3/tempfile of=/dev/null bs=1M count=1024 |& tail -1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.01625 s, 1.1 GB/s

was: 966 MB/s

A very moderate improvement so far

The old KDiskMark results for this disk were posted in Booting from NVMe USB enclosure (Post by rych #96889)

New results:

KDiskMark (3.1.3): https://github.com/JonMagon/KDiskMark
Flexible I/O Tester (fio-3.34): https://github.com/axboe/fio
--------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1 MiB (Q= 8, T= 1): 1984.937 MB/s [ 1938.4 IOPS] < 4106.15 us>
Sequential 128 KiB (Q= 32, T= 1): 2010.358 MB/s [ 15705.9 IOPS] < 2031.21 us>
Random 4 KiB (Q= 32, T=16): 370.517 MB/s [ 92630.2 IOPS] < 1380.91 us>
Random 4 KiB (Q= 1, T= 1): 42.088 MB/s [ 10522.1 IOPS] < 93.45 us>

[Write]
Sequential 1 MiB (Q= 8, T= 1): 701.939 MB/s [ 685.5 IOPS] < 11550.89 us>
Sequential 128 KiB (Q= 32, T= 1): 828.291 MB/s [ 6471.0 IOPS] < 4914.83 us>
Random 4 KiB (Q= 32, T=16): 148.121 MB/s [ 37031.1 IOPS] < 3483.61 us>
Random 4 KiB (Q= 1, T= 1): 122.710 MB/s [ 30677.7 IOPS] < 30.89 us>

Profile: Default
Test: 1 GiB (x5) [Measure: 5 sec / Interval: 5 sec]
Date: 2023-11-07 22:35:17
OS: slackware 5.01 [linux 6.5.5-porteus]

The improvement in almost all speeds is OUTSTANDING!

[rych]

Unfortunately, possibly due to unclean unmount, a dirty bit is set and while booting into Windows straight after it starts with Scanning and Repairing disk C: !! Even if there are no errors found and Windows boot eventually proceeds this could be a show-stopper for the ntfs3 driver at this stage.

Could Porteus do a more thorough job at cleanly unmounting ntfs3 mounted partitions? What worked with ntfs-3g may not be enough for ntfs3. If anything maybe issue a ntfsfix --clear-dirty after unmount, or wait longer for it to actually unmount?

[ncmprhnsbl]

thanks for your efforts with this
you seem to be on the right track
this:

(/etc/fstab still lists them as ntfs-3g hot plugged mounts -- will fix /etc/fstab later)

is handled by /sbin/udev-update-fstab, you'll see where..
for the unmounting, take a look at initrd>cleanup script, i see some references to ntfs-3g in there..
if we can get this working, we can drop ntfs-3g from initrd, though since ntfs3 has no userspace utils for now, we probly need to keep those parts of ntfs-3g, at least ntfsfix, it seems..

[rych]

ncmprhnsbl, Thanks for checking in, I was beginning to feel a bit lonely on this.

The issue has fixed itself for now: rebooting into Windows there's no more "Scanning and Repairing disk C:"

Still, I've replaced the 2 occurrences of of ntfs-3g with ntfs3 in the /sbin/udev-update-fstab. The /etc/fstab now has "ntfs3" -- which only matters for hot-plugged disks from now on I guess.

I've left the initrd>cleanup untouched for now. However, if the issue re-appear, I'm going to try to add the block below at the end of initrd>cleanup. Although I wonder whether the echo commands would even display in the terminal at this final stage.

Code: Select all

# Everything should be unmounted by now, in particular
echo "confirming unmount"
cat /proc/mounts | egrep ntfs
# If we have access to ntfsfix here we should be able to: 
echo "clearing the dirty-bit"
ntfsfix --clear-dirty /dev/nvme0n1p3
# Let's have a 10 seconds to read the error message, for example:
# "Refusing to operate on read-write mounted device /dev/nvme0n1p3"
sleep 10

# Launch debugging shell if requested:

Anyway, let's wait for more events of "Scanning and Repairing disk C:". I'm going to boot Porteus on different machines and I always reboot into Windows afterwards, so I will know immediately. Until then I'll enjoy the ntfs3 driver.

[rych]

wait for more events of "Scanning and Repairing disk C:". I'm going to boot Porteus on different machines and I always reboot into Windows afterwards, so I will know immediately.

It seems that "Scanning and Repairing disk C:" happens at most once when booting and mounting with ntfs3 on a machine that was previously mounted with ntfs-3g. I've read somewhere that for example symbolic (or hard?) links created with ntfs-3g won't work with ntfs3. I don't think I use links on NTFS ever, but that suggests some incompatibility between ntfs-3g and ntfs3 in general. I've also read that ntfs3 is stricter.

So perhaps when ntfs3 first mounts a partition it may find some left-over from ntfs-3g, which even Windows itself didn't mind, but ntfs3 sets the dirty bit, forcing Windows to check on the next reboot. I don't think there is any actual "corruption", just the pedantic dirty bit.

Otherwise, using ntfs3 has been fine -- no issues whatsoever. I would just like to avoid mixing ntfs3 and ntfs-3g mounting. Now that I've patched Porteus, I'm going to similarly patch PorteuX which I occasionally use to also use ntfs3 only.

[rych]

using ntfs3 has been fine -- no issues whatsoever

Besides the initial dirty bit on some partitions which were previously mounted with ntfs-3g, which was fixed without finding any errors by booting into Windows once... There has been 1 issue during these months. Once porteus/ntfs3 failed to finalize writing or deleting files to an NTFS partition before shutting down. Again, booting into Windows fixed that, but those files were corrupted. Perhaps, I unplugged the USB too soon or some delay somewhere wasn't enough. I decided to update the kernel too to 6.6.3, because ntfs3 kernel module is in active development and there were some bug fixed in 6.6 I've read. No issues since then.

Also, I didn't replace all occurrences of ntfs-3g --> ntfs3 in the initrd yet. I feel I need an advice on changing cleanup script. I'll list all the changes I've done so far in the following post.

[rych]

@ncmprhnsbl and @fulalas,
I've made these changes:

A. initrd/finit

1.

Code: Select all

ntfs-3g /dev/$x /mnt/$x -o $MOPT,big_writes 2>/dev/null || { sed -i "/$x /d" /etc/fstab; rmdir /mnt/$x; }

--->

Code: Select all

mount -n -t ntfs3 /dev/$x /mnt/$x -o $MOPT 2>/dev/null || { sed -i "/$x /d" /etc/fstab; rmdir /mnt/$x; }

2.

Code: Select all

ntfs-3g /dev/$1 /mnt/$1 -o $MOPT,big_writes 2>/dev/null || rmdir /mnt/$1

--->

Code: Select all

mount -n -t ntfs3 /dev/$1 /mnt/$1 -o $MOPT 2>/dev/null || rmdir /mnt/$1

B. initrd/linuxrc

Code: Select all

sed -i 's/ ntfs / ntfs-3g /g' /etc/fstab

--->

Code: Select all

sed -i 's/ ntfs / ntfs3 /g' /etc/fstab

C. initrd/cleanup No changes yet so it still has this:

Code: Select all

# Send TERM signal to processes which can be killed:
PID=`ps | sed -n '/PID/,/{cleanup}/p' | egrep -v '\[.*\]|PID|cleanup|ntfs-3g' | sed -r "s/^ *([0-9]+).*/\\1/" | tr "\n" " "`
kill -15 `echo $PID` >/dev/null 2>&1

--->??? How/Why to kill a kernel module?? ntfs-3g was a process, ntfs3 is part of the kernel, right?

D. Also I simply replaced 2 occurences of "ntfs-3g" with "ntfs3" in the /sbin/udev-fstab-update, so now there are 2 lines like this

Code: Select all

elif [ "$FS" = ntfs ]; then sed -i '\?^'$DEVNAME'?d' $FSTAB; echo "$DEVNAME $MNTPT ntfs3 $NMOPT 0 0" >> $FSTAB; chmod 777 $MNTPT

[tome]

NTFS partitions mounted with NTFS3 driver keep permissions and owners - then they can be used fully for rootcopy= or changes= cheadcodes.
Remounting NTFS partition once with ntfs3 another time with ntfs3-g or vice versa cause some(many) reads/writes to disk.

https://wiki.archlinux.org/title/NTFS

[rych]

I've had another incident of ntfs3 leaving an NTFS partition dirty, with this error message: https://wiki.archlinux.org/title/NTFS#U ... rked_dirty

Windows dealt with it on reboot, scanned the C:, found no errors (apparently), and booted successfully. Rebooting into Porteus, ntfs3 was still unhappy about the partition and refused to mount it. We can issue the ntfsfix command below and only then it mounts successfully. So, sometimes both Windows and ntfsfix have to clean up after the ntfs3 driver.

ntfsfix --clear-dirty /dev/disk/by-label/Windows

I wish ntfsfix was part of the busybox, otherwise we sometimes won't even be able to mount an NTFS partition at the time of initrd.

I'm currently on kernel 6.6.3. I'm going to check any ntfs3-related bug fixes in newer kernel -- maybe it's worth upgrading. Indeed there are some minor bugfixes on https://lore.kernel.org/ntfs3/ for 6.7 and even into 6.8. I'll wait for those kernels to mature somewhat before upgrading.

There is a recent discussion about ntfsfix potentially hiding issues and only clearing the dirty bit, and that a better utility exists in principle, chkntfs; initird is also mentioned as if any of this can be made accessible to initrd?!

[jjr]

ntfs3 has no userspace utils for now

FWIW, Paragon (author the ntfs3 driver) has 'discreetly published' the proprietary NTFS utils for Linux for free. I made the discovery a few months back and posted it in this thread as a comment:
https://www.reddit.com/r/archlinux/comm ... t/k6nq7g6/

The OP csdvrx has been trying to get word out to the Linux community now, as Linux has lacked a decent NTFS fsck for so long. The macOS version is a commercial product for many years and it seems mature. Several Linux testimonials are in the thread now.

I'm guessing it's problematic to include in Porteus due to licensing issues, if these binaries weren't meant to be extracted and run outside of the [discontinued] UFSD Root Mounter for Android.

But Porteus users can use it personally, at least.

I wish ntfsfix was part of the busybox, otherwise we sometimes won't even be able to mount an NTFS partition at the time of initrd.

ntfsfix is so dangerous, though. IME, GParted refuses to use it at all -- telling you to repair in Windows only.

Corroborating your link, the manpage warns it is not a valid substitute for chkdisk.

It may do some things, but clearing the dirty bit without a thorough repair (e.g. Windows or Paragon utils) is still like taking a hammer and knocking out the engine warning light in your car. It will hide the symptom, but allows the filesystem to build its operations on an inconsistent data structure, which will propagate that inconsistency and eventually lead to an irreparable volume.

[jjr]

Sorry for all the edits.

[rych]

Porteus users can use it personally

-- I'd like to use to check my NTFS partitions: do you have a binary already for chkntfs (?) etc., please?

[jjr]

Porteus users can use it personally

-- I'd like to use to check my NTFS partitions: do you have a binary already for chkntfs (?) etc., please?

In that Reddit thread, someone else uploaded them to the Internet Archive for preservation:
https://archive.org/details/tools_202401

But that could be viewed as sketchy, and I would recommend people to download and unzip the official APK from a trusted source. I know APKPure has it.

Either way, get the chkufsd x86 static binary, then either create a symlink chkntfs -> chkufsd, or else run chkufsd with an explicit filesystem option -fs:ntfs. (It's a BusyBox-style program, supporting Apple HFS+ and exFAT as well.)

[jjr]

x86 static binary

To clarify, it runs on non-Android x64 because it's statically linked.

[ncmprhnsbl]

@jjr thanks for your insights
i'm inclined to stick with ntfs-3g for porteus for now, as it seems to be less troublesome(maybe?) than the current kernel implementation, if less performant.
inclusion of chkntfs in any official porteus initrd (for on the fly fix ups) also seem problematic for the reasons you stated (plus general binary blob issue and that it's 1.6MB)
also i'd leave ntfsfix out too for those reasons you gave.
of course, anyone is free to do what they want, and porteus has all the tools to rebuild the initrd.

the impression i'm left with is: for windows filesystems(ntfs) use windows tools
that said, the only time i had an external ntfs drive wiped(just the partition table corrupted), it was by windows :p