Finding the best filesystem (for USB flash drive installs)

Non release banter
User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Finding the best filesystem (for USB flash drive installs)

Post#1 by Ahau » 12 Jul 2011, 17:57

Ok, now that I've got the power saving article into draft form, I'm finally ready to pursue my next pet project: FIlesystems!

Yes, yes, I know that which filesystem you use is largely a matter of preference. I know that most folks are using FAT32 because that's how their flash drives come formatted, and I've heard that you should use ext2 if you want a POSIX-compatible system, since it's non-journaling.

However, I remain curious (I know, my curiousity will be the death of me!) and I'd like to put some of the myths to rest (or, to test!).

My plan is to set up some test partitions on my flash drives, format them to various filesystems, and generate a script to run them through their paces and catalogue the results. In my first round of testing, I'll try many filesystems, and test them when mounted from Porteus (with Porteus being installed to a different partition). From that testing, I'll select some of the most appealing systems, and test them again with Porteus installed to the test partition.

In round one, the testing will consist of various activities that simulate use under normal conditions, such as:

-copy the Porteus ISO and linux kernel tarball ( single large files) to the test partition
-mloop the ISO and copy the contents (a few large files) to the test partition
-extract the kernel tarball (many small files/directories) to the test partition
-extract an xzm (write many small files) to the partition
-create an xzm from the files on the partition (read many small files)
-copy the above files and/or their extracted components to a directory on the live filesystem (test reading instead of writing)
-perhaps do some writing of 0's to the drive, creation and deletion of thousands of directories, etc --this kind of benchmarking has been done in the past, but is less reflective of actual use, IMHO.

Round two will consist of the same tests as above, as well as timing how quickly Porteus boots into Always Fresh mode, with saved changes (first and second boots, to measure writing (first boot) and reading (second boot)), and copy2ram. I'll also consider the ease of booting the filesystem with the default porteus kernel/initrd, as well as the ability to boot it with syslinux/extlinux for a graphical boot menu (not that important to power users, but I want my suggestions to apply broadly to our userbase).

Filesystems I'd like to include in this test are (so far):

FAT32 (to have a baseline for what most people are using)
NTFS (I know some windows users boot Porteus off NTFS, and it may outperform FAT in some respects)
EXT2
EXT3
EXT4 (journaling and non-journaling)
XFS
ReiserFS
NILFS2
JFS
BTRFS

For the record, I used a 4GB flash drive (Lexar TwistTurn JumpDrive) formatted entirely with FAT32 for a long time, and then switched to an 8GB Kingston DataTraveler 101 G2 with a FAT32 partition in front (for windows compatibility) and an EXT2 partition in the back (for all my Porteus installs and data that needs POSIX compatibility). These two flash drives will be my test subjects. If I can successfully write a script to run my tests, I'll share it here and maybe others can test their drives. I'd love to see a more thorough review on this. I know that the filesystem of your flashdrive probably matters very little (other than windows/POSIX compatibility) because most of the actions in Porteus take place in aufs/RAM, but I feel a strong desire to test the hell out of it anway ;)

I've nearly ruled out JFS (preliminary testing has been very bad for writing many small files), and I have many reservations about BTRFS and NILFS2 because they are still experimental and lack good fsck and other tools/support. So, they'll have to perform substantially better than the others for me to use them. Reiser4 is not supported in the Porteus kernel, and as I understand it, JFFS2, YAFFS and UBIFS don't work for flashdrives, and LogFS has only basic, slow support for them (source: LogFS).

If you have any suggestions (i.e. additional tests to run, filesystems to try), please let me know!
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: Finding the best filesystem (for USB flash drive install

Post#2 by brokenman » 13 Jul 2011, 18:27

Excellent idea !!

We could use some empirical data about filesystems to put to rest some myths. Be sure to post your results ... i will keenly follow this thread.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#3 by Ahau » 13 Jul 2011, 18:35

just by embarking on this endeavor, I'm already learning very much. First, I'm not good with scripting, so writing a script is putting me on the steep edge of a learning curve. Second, I'm learning more commands and what they do. Third, I'm learning more about filesystems and how they work.

It's interesting to see how linux will show that a transfer of a file is complete, when it is, in fact, not. It will move the file to its' new location and tell the system that it is done moving, when in fact most of the file could still be in RAM, waiting to be written. Thus, my script makes generous usage of the 'sync' command, and I'm timing syncs as well as transfers, so that I can calculate the actual transfer time, as well as the apparent transfer time (and then I can compare how different filesystems might handle this differently). I also didn't know that linux keeps pagefiles cached in memory. This can produce misleading results if you, for example, copy a large file to a device and then try to copy it back to a location in your live filesystem. It will appear to read a 70MB file instantaneously -- so my script will also be making good use of 'sync; echo 1 > /proc/sys/vm/drop_caches'

I could probably do this faster by moving things manually with a stopwatch...but I think having a script will make the process more uniform, and the comparisons will be better. It will also allow other users to run the same benchmarks on their systems, to gain even more data.
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
Hamza
Warlord
Warlord
Posts: 1908
Joined: 28 Dec 2010, 07:41
Distribution: Porteus
Location: France

Re: Finding the best filesystem (for USB flash drive install

Post#4 by Hamza » 14 Jul 2011, 09:57

I think, the better filesystem is ext4, for me, it is the more stable and reliable.

If there is a better, post it!
NjVFQzY2Rg==

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#5 by Ahau » 14 Jul 2011, 19:24

@Hamza -- rest assured, I'll be presenting my data and an analysis of it. I'll put my script out with directions for others to generate data if they are so inclined. However, this is a significant time commitment (it takes over 30 minutes for the script to run on my machine, and it would need to be run multiple times to test multiple filesystems), so I'll understand if no one volunteers!

Truth be told, I'm probaly generating WAY more data than I really need for this...but I'd rather have too much than too little ;)

Here's my script, which is currently in alpha testing (hopefully I'll get it through beta later today or into tomorrow):

http://pastebin.com/U3fDkQFe

Some people start off scripting with "hello world"... I don't know what possessed me to tackle this. If anyone sees errors (or just stupid stuff, like doing something in a less efficient way than it could otherwise be done), feel free to let me know.

Here's an example of what the output looks like (there were some errors with the script that are apparent in the results, which I think I've fixed):

http://pastebin.com/WHpxcFFZ

I'm starting off with NTFS because it's a pain in the butt for me to format my flash drive to NTFS and I wanted to get it over with (I'm also not sure how well kde partition manager will resizing it to accomodate another partition). I don't think I'd ever use NTFS...but I know some do, and I'm *trying* to remain impartial :)

More to come!
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: Finding the best filesystem (for USB flash drive install

Post#6 by brokenman » 15 Jul 2011, 20:58

'best' is a very subjective term .... this is why i am happy (read ecstatic) that someone has taken on this monster and is willing to document the results.

Until you have empirical evidence ... everything else is just opinion .

Thanks Ahau ... i even learned some stuff from what you have posted so far!! Keep it up.

PS: Will this script always be run from a non FAT32 partition?
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#7 by Ahau » 18 Jul 2011, 22:55

I know what you're getting at -- loss of links, etc,. when saving files to a Fat32. I intend on including FAT in my testing, as it's the most used filesystem. I'll have to perform some kind of corection or throw out any results with substantial links. You'll notice the script keeps track of how many links are present in an extracted dataset.

Cheers!

Posted after 3 days 40 minutes 29 seconds:
Re: Finding the best filesystem (for USB flash drive installs)
Just a little update today -- I'm in the data collection phase. I have completed testing NTFS, FAT32, and EXT2, and starting on EXT3. The results have already yielded some things of interest:

Average write speed for single, large files:
NTFS: 5.18 MB/s
FAT32: 5.69 MB/s
EXT2: 4.29 MB/s

However, as I mentioned above, Linux will load large files into RAM, and leave them there to transfer, while releasing the shell or process for other tasks. So, you might see the transfer a 100 MB file in 2 seconds (even though the transfer will continue for 18 more seconds in the background). Using this 'Apparent' transfer speed, you get these results (in 64-bit; 32-bit apparent transfer speeds are a bit slower) :

NTFS: 61.73 MB/s
FAT32: 196 MB/s
EXT2: 345 MB/s

EXT2 and NTFS are nearly twice as fast as FAT32 at writing many small files. EXT2 is far and away faster when it comes to deleting many small files. It is also faster at extracting modules.

I am absolutely swimming in data; my big problem now is trying to consolidate all of it down into manageable bits and pieces without skewing my results. I have 1 main spreadsheet to summarize data from three filesystem-specific spreadsheets, each of which contain data from 6 tests of that filesystem, and each test has 32 individual timed events that report the real and apparent rates, as well as processor load.

I'm excited to get through the boring old FS's and get to something fun, like BTRFS or NILFS2 :D

(much!) more to come!
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
BlackRider
Black ninja
Black ninja
Posts: 70
Joined: 13 Jul 2011, 11:04
Location: Nowhere
Contact:

Re: Finding the best filesystem (for USB flash drive install

Post#8 by BlackRider » 18 Jul 2011, 23:22

Are you performing any of the following tests?:
----------------------------------------------------------------------------
Degradation of the fille system after filling it with files and then deleting them. Some file systems are not able to regain all of the room.

Fragmentation rate after certain period or repetitive activity.

CPU usage, memory usage.

Frequency of read-write access. This one is important for usb-flash units. (Anyone knows how this can be calculated?)

Time taken for defragmentation.

Room the empty filesystem wastes.

An analisys about recovery capabilities after a crash would be nice. Maybe you should crash each working filesystem 700 times and take note of how often each one breaks, and how hard it breaks.
---------------------------------------------------------------------------
These are as relevant as speed.

Note that the results can be heavily affected by the size of the blocks of the file system and other factors.

My choices, based in the results of this tests (found them on the WWW, I can't remember where):

XFS is my /home . It has a good speed (no the fastest, but is enough), no bad resource usage and good recovery capabilities. It degradates with the time and is prone to fragmentation. I would have chosen Riserfs but it is error prone and when it crashes, it crashes badly.

Ext2 is my /tmp. I don't mind if it has no journal. If it breaks, only temporal files can get lost. Lacking a journal, it is faster.

FAT32 for portable devices, because of compatibility. I run Porteus in RAM, so read-write access is not a problem.

Ext4 for everything else, because of recovering capabilities. I don't mind my /usr being slower in moving files, because I don't use to move files inside /usr, if you get my point.

Falcony
Full of knowledge
Full of knowledge
Posts: 237
Joined: 01 Jan 2011, 12:44
Location: Russia

Re: Finding the best filesystem (for USB flash drive install

Post#9 by Falcony » 19 Jul 2011, 04:56

One more comment regarding booting Porteus from partiotion which formatted with some FS

Not every boot loader allow to do this(of course if yo have only one partition on USB on which catalogs boot and porteus reside)

Table which I used and it is works is follow:
FAT16 - GRUB4DOS & SYSLINUX loaders
FAT32 - GRUB4DOS & SYSLINUX loaders
NTFS - GRUB4DOS
EXT2 - GRUB4DOS, EXTLINUX
EXT3 - GRUB4DOS, EXTLINUX
EXT4 - EXTLINUX
BTRFS - EXTLINUX

Basically I may say in point of view of convinience selection ReiserFS, XFS, JFS, etc is not good choice.

And as I remember I have some problem to boot with LILO(using liloinst.sh) on some FS but now I do no remember details.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#10 by Ahau » 19 Jul 2011, 06:09

@falcony-- thanks for the comment! The availability of extlinux or syslinux is a benefit for porteus users to be sure. LILO should theoretically be able to boot any of them, but the vanilla kernel/initrd do not support some (btrfs, jfs, nilfs2). I do want to keep my recommendation geared towards the average user, so ease of formatting and installation is important.

@BlackRider -- thank you for your post as well -- so much in there to consider!
To your questions:
1)my tests do not specifically look at degredation of the filesystem, but I think it would be good to start checking available space after deleting all files (before and after testing). If you have any links that could shed more light on this, please post them (though I'll research this as well).

2)My tests also do not test fragmentation rate or the speed of defragging;it has been my understanding that both of these issues are not very relevant for flash drives with wear leveling -- I could be wrong on that.

3)I am keeping track of CPU usage, in a way (but not memory usage--it may be a good idea to check TOP while it's unpacking the kernel tarball !). The 'time' command prints the amount of time that a process uses in the kernel. I am tracking that and keeping on eye on it, especially for writing and reading large numbers of small files.

4) frequency of read-write access -- I'm not sure that I understand exactly what this means or how to test it (again, I'll do more research). My tests do compare the rate of copying from RAM vs from one spot on the device to another, if that is what you mean.

5)I'll start keeping track of room the empty filesystem wastes. I know it was substantial when I formatted to ext3 earlier today--but I didn't write it down!

6) I was planning on researching more about recovery -- I know NILFS2 and BTRFS are short here, so they would have to perform damn well in other regards to garner my recommendation for daily use.

7) If you have a reproducible way for me to 'crash' a filesystem and quantify the results, I might be willing to test it--however, my resources are limited, and I don't want to break anything lol

I agree that these are as relevant as speed. For Porteus, that is even moreso the case, as files are predominantly written once, and then used in RAM. I'd like to have as holistic of an approach as possible, and I think your contribution is helping me get there!
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5666
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

Re: Finding the best filesystem (for USB flash drive install

Post#11 by fanthom » 19 Jul 2011, 07:28

hi Ahau,
i have a simple benchmark request - could you measure performance of XFS compared to (XFS formatted) *.dat container placed also on XFS?
i'm just wonder if XFS mounted on a loop device does some performance impact.
in short words: i have a impression that porteus with 'changes=' saved on xfs partition works little faster that with 'changes=' saved in *.dat container.

Cheers
Please add [Solved] to your thread title if the solution was found.

User avatar
BlackRider
Black ninja
Black ninja
Posts: 70
Joined: 13 Jul 2011, 11:04
Location: Nowhere
Contact:

Re: Finding the best filesystem (for USB flash drive install

Post#12 by BlackRider » 19 Jul 2011, 10:36

1)my tests do not specifically look at degredation of the filesystem, but I think it would be good to start checking available space after deleting all files (before and after testing). If you have any links that could shed more light on this, please post them (though I'll research this as well).
I found this test performed in a very old article I found somewhere. No idea oF where to find it again. You can make a script that creates many random generated files (use /dev/frandom, for example) until filling the drive and then delete all of them.
2)My tests also do not test fragmentation rate or the speed of defragging;it has been my understanding that both of these issues are not very relevant for flash drives with wear leveling -- I could be wrong on that.
Fragmentation is not a significative problem for solid state drives and flash units, thats right. However, it is important for internal drives that you don't want to format/defrag often. Some guy out there could install Porteus in a compact drive instead of a flash unit, so then it is a problem for him too.
4) frequency of read-write access -- I'm not sure that I understand exactly what this means or how to test it (again, I'll do more research). My tests do compare the rate of copying from RAM vs from one spot on the device to another, if that is what you mean.
With "frequency of read-write access" I mean exactly that: how much times the device is accessed during common operations. When you copy a file, it remains in the RAM for a while until it is copied completly. If you modify the file before it is completly moved, the Operating System should copy only the modified version, not the first one and then the second one. A filesystem with a hight apparent transfer rate and a low real transfer rate is supposed to reduce read-writte access frequency. I have no idea of how to quantify it.

You can have a look at the true writting speed of the drive if you disable the writting cache with hdparm (this forces all to be written in order in most drives). The cache can have serious security implications when wipping drives with pseudo-random data, as it prevents all the overwrittings to reach the surface of the disk.

7) If you have a reproducible way for me to 'crash' a filesystem and quantify the results, I might be willing to test it--however, my resources are limited, and I don't want to break anything lol

I think the guys that made this test had an army of testers. No idea of how they tested this feature exactly. Crashing the unit is easy: just have a script perform a bussy file operation and then unplug it out without warning. Damages can be quantified, I am not sure about how. The test I read provided information about inode level damage and the like...

Hint: Ext2 has horrible recovering capabilities.
i'm just wonder if XFS mounted on a loop device does some performance impact.
A filesystem hosted in a regular file is usually a worse option than a filesystem in a device. Your .dat file can get much fragmented with the use, and the contents of the .dat file can become fragmented inside the guest filesystem. The impact of this is obvious for traditional drives. For flash and solid state drives, it is to determinate.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#13 by Ahau » 19 Jul 2011, 13:28

Arrg! I just wrote a long response to this thread, and lost it because my filesystem test script has an option at the end of it to halt my machine (so I can set it running and then go to bed, and it will shut down). We'll see how much I can reproduce :) I guess EXT4 is faster than I thought?...

@fanthom --sure, no problem to include testing a loopmounted .dat container. I'll just create one and point my script to /mnt/loop instead of /mnt/sdb2. I also wonder if it will perform better or worse with the .dat on different filesystems -- I do know that loopmounted modules can be read faster from some filesystems than others.

@Blackrider,

1) I have started keeping track of how much filespace is unuseable on my test partition after creation, one test, and my final test. Since I'm doing writes with several GB of data multiple times, this should pick up any degradation.

2) My tests are specific to USB Flash Drives. There's tons of information out there for anyone who wants to read up on benchmarks, fragmentation, etc., for internal drives. In fact, I don't have any frugal installs. If anyone would like to run my tests on internal drives, that would be great, and I would help all I could, but it's not something I can take on myself. I'd also love to see how flashcards (compact flash, SD, memorystick, etc) perform, but I don't have a card reader for my laptop. I'll leave fragmentation issues out of my testing for now.

4) I am closely tracking the real vs. apparent transfer rates, hopefully this will give an approximation on the read/write access concern, especially if your assumption about high apparent transfer rates is correct. Also, how would I disable the writing cache on a USB flash drive with hdparm? I read the man page and tried `hdparm -W /dev/sdb2` to retrieve information, and received an error stating that it "write-caching=not supported".

7) I may do some crash testing -- both with pulling the drive out while mounted (while idle and while transferring), and with hitting the power switch while Porteus is running. The latter test would be more harmful for my system, but it would more accurately test what would happen if Porteus froze and needed to be manually shut down. I'll do this after I complete the rest of my baseline testing, so that I only do it on the filesystems I favor for use in an every day system (no point in torturing my system on JFS, for example -- which in preliminary testing was absolutely wretched for writing many small files on a flash drive).


Thanks again for your suggestions!!
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
BlackRider
Black ninja
Black ninja
Posts: 70
Joined: 13 Jul 2011, 11:04
Location: Nowhere
Contact:

Re: Finding the best filesystem (for USB flash drive install

Post#14 by BlackRider » 19 Jul 2011, 14:38

I read the man page and tried `hdparm -W /dev/sdb2` to retrieve information, and received an error stating that it "write-caching=not supported".
Yeah, sometimes that happens. Nice try, anyway.

Some devices have not a buffer and force every write to pass in an instant, while others won't allow you to turn the cache off. An article I read some time ago about secure deletion methods claims that many drives report themselves to have turned the cache off, while they keep it on without telling us, because then they can perform better in the benchmarks.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#15 by Ahau » 19 Jul 2011, 14:45

hehe. Good to know. I'm glad I found out about buffer caching before I started my testing, or else I would have some very skewed results (things seem to be pretty consistent with the liberal use of the sync command and echo 1 > /proc/sys/vm/drop_caches.

I'm researching ext4 now -- it appears that I can disable the journal simply by adding 'noload' to /etc/fstab (or, I can make it permanent with tune2fs -O ^has_journal). Or, I can put the journal in 'writeback' mode -- it seems 'ordered' is the default mode when creating an ext4 partition w/kde partition manager in 64-bit porteus. any comments here?

references:
http://www.mjmwired.net/kernel/Document ... s/ext4.txt
http://unix.stackexchange.com/questions ... -ssd-drive
Please take a look at our online documentation, here. Suggestions are welcome!

Post Reply