Finding the best filesystem (for USB flash drive installs)
Posted: 12 Jul 2011, 17:57
Ok, now that I've got the power saving article into draft form, I'm finally ready to pursue my next pet project: FIlesystems!
Yes, yes, I know that which filesystem you use is largely a matter of preference. I know that most folks are using FAT32 because that's how their flash drives come formatted, and I've heard that you should use ext2 if you want a POSIX-compatible system, since it's non-journaling.
However, I remain curious (I know, my curiousity will be the death of me!) and I'd like to put some of the myths to rest (or, to test!).
My plan is to set up some test partitions on my flash drives, format them to various filesystems, and generate a script to run them through their paces and catalogue the results. In my first round of testing, I'll try many filesystems, and test them when mounted from Porteus (with Porteus being installed to a different partition). From that testing, I'll select some of the most appealing systems, and test them again with Porteus installed to the test partition.
In round one, the testing will consist of various activities that simulate use under normal conditions, such as:
-copy the Porteus ISO and linux kernel tarball ( single large files) to the test partition
-mloop the ISO and copy the contents (a few large files) to the test partition
-extract the kernel tarball (many small files/directories) to the test partition
-extract an xzm (write many small files) to the partition
-create an xzm from the files on the partition (read many small files)
-copy the above files and/or their extracted components to a directory on the live filesystem (test reading instead of writing)
-perhaps do some writing of 0's to the drive, creation and deletion of thousands of directories, etc --this kind of benchmarking has been done in the past, but is less reflective of actual use, IMHO.
Round two will consist of the same tests as above, as well as timing how quickly Porteus boots into Always Fresh mode, with saved changes (first and second boots, to measure writing (first boot) and reading (second boot)), and copy2ram. I'll also consider the ease of booting the filesystem with the default porteus kernel/initrd, as well as the ability to boot it with syslinux/extlinux for a graphical boot menu (not that important to power users, but I want my suggestions to apply broadly to our userbase).
Filesystems I'd like to include in this test are (so far):
FAT32 (to have a baseline for what most people are using)
NTFS (I know some windows users boot Porteus off NTFS, and it may outperform FAT in some respects)
EXT2
EXT3
EXT4 (journaling and non-journaling)
XFS
ReiserFS
NILFS2
JFS
BTRFS
For the record, I used a 4GB flash drive (Lexar TwistTurn JumpDrive) formatted entirely with FAT32 for a long time, and then switched to an 8GB Kingston DataTraveler 101 G2 with a FAT32 partition in front (for windows compatibility) and an EXT2 partition in the back (for all my Porteus installs and data that needs POSIX compatibility). These two flash drives will be my test subjects. If I can successfully write a script to run my tests, I'll share it here and maybe others can test their drives. I'd love to see a more thorough review on this. I know that the filesystem of your flashdrive probably matters very little (other than windows/POSIX compatibility) because most of the actions in Porteus take place in aufs/RAM, but I feel a strong desire to test the hell out of it anway
I've nearly ruled out JFS (preliminary testing has been very bad for writing many small files), and I have many reservations about BTRFS and NILFS2 because they are still experimental and lack good fsck and other tools/support. So, they'll have to perform substantially better than the others for me to use them. Reiser4 is not supported in the Porteus kernel, and as I understand it, JFFS2, YAFFS and UBIFS don't work for flashdrives, and LogFS has only basic, slow support for them (source: LogFS).
If you have any suggestions (i.e. additional tests to run, filesystems to try), please let me know!
Yes, yes, I know that which filesystem you use is largely a matter of preference. I know that most folks are using FAT32 because that's how their flash drives come formatted, and I've heard that you should use ext2 if you want a POSIX-compatible system, since it's non-journaling.
However, I remain curious (I know, my curiousity will be the death of me!) and I'd like to put some of the myths to rest (or, to test!).
My plan is to set up some test partitions on my flash drives, format them to various filesystems, and generate a script to run them through their paces and catalogue the results. In my first round of testing, I'll try many filesystems, and test them when mounted from Porteus (with Porteus being installed to a different partition). From that testing, I'll select some of the most appealing systems, and test them again with Porteus installed to the test partition.
In round one, the testing will consist of various activities that simulate use under normal conditions, such as:
-copy the Porteus ISO and linux kernel tarball ( single large files) to the test partition
-mloop the ISO and copy the contents (a few large files) to the test partition
-extract the kernel tarball (many small files/directories) to the test partition
-extract an xzm (write many small files) to the partition
-create an xzm from the files on the partition (read many small files)
-copy the above files and/or their extracted components to a directory on the live filesystem (test reading instead of writing)
-perhaps do some writing of 0's to the drive, creation and deletion of thousands of directories, etc --this kind of benchmarking has been done in the past, but is less reflective of actual use, IMHO.
Round two will consist of the same tests as above, as well as timing how quickly Porteus boots into Always Fresh mode, with saved changes (first and second boots, to measure writing (first boot) and reading (second boot)), and copy2ram. I'll also consider the ease of booting the filesystem with the default porteus kernel/initrd, as well as the ability to boot it with syslinux/extlinux for a graphical boot menu (not that important to power users, but I want my suggestions to apply broadly to our userbase).
Filesystems I'd like to include in this test are (so far):
FAT32 (to have a baseline for what most people are using)
NTFS (I know some windows users boot Porteus off NTFS, and it may outperform FAT in some respects)
EXT2
EXT3
EXT4 (journaling and non-journaling)
XFS
ReiserFS
NILFS2
JFS
BTRFS
For the record, I used a 4GB flash drive (Lexar TwistTurn JumpDrive) formatted entirely with FAT32 for a long time, and then switched to an 8GB Kingston DataTraveler 101 G2 with a FAT32 partition in front (for windows compatibility) and an EXT2 partition in the back (for all my Porteus installs and data that needs POSIX compatibility). These two flash drives will be my test subjects. If I can successfully write a script to run my tests, I'll share it here and maybe others can test their drives. I'd love to see a more thorough review on this. I know that the filesystem of your flashdrive probably matters very little (other than windows/POSIX compatibility) because most of the actions in Porteus take place in aufs/RAM, but I feel a strong desire to test the hell out of it anway
I've nearly ruled out JFS (preliminary testing has been very bad for writing many small files), and I have many reservations about BTRFS and NILFS2 because they are still experimental and lack good fsck and other tools/support. So, they'll have to perform substantially better than the others for me to use them. Reiser4 is not supported in the Porteus kernel, and as I understand it, JFFS2, YAFFS and UBIFS don't work for flashdrives, and LogFS has only basic, slow support for them (source: LogFS).
If you have any suggestions (i.e. additional tests to run, filesystems to try), please let me know!