Finding the best filesystem (for USB flash drive installs)

Non release banter
User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: Finding the best filesystem (for USB flash drive install

Post#16 by brokenman » 19 Jul 2011, 16:47

writeback mode is more efficient in terms of speed but doesn't really do any real data journaling. Your sacrificing safety for speed. Ordered mode (the mode your root filesystem is most probably set to by default) is a little more stable for it at least uses metadata journaling. journal mode is the slowest but most comprehensive mode to use. It does real data journaling aswell as metadata but is much slower as you're essentially writing twice as much data to your disk.

You can set your peffered ext3 mode by adding a line to the options section of your /etc/fstab. data=journal, data=writeback or data=oredered. You can also specify the -o data=journal command-line option when mounting directly. You can put your entie root filesystem into full data journaling mode by adding rootflags=data=journal to your kernel boot options.

Check /proc/mounts to see what it is now set to. Prob date=ordered and errors=continue
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#17 by Ahau » 20 Jul 2011, 13:13

Thanks, brokenman!

/proc/mounts was showing data=ordered, until I removed has_journal with tune2fs. Now it is showing data=writeback...even when mounted with the noload option in fstab. There is a slight performance increase (but it is quite slight thus far--I've only finished testing in 32-bit and haven't fully analyzed the results). I guess I would be concerned about the amount of writes to the flashdrive when using a journal (especially data=journal), but as it's been said, this is probably not so much of a concern with a wear-leveling device (besides, the assload of writing I'm doing to this flashdrive is probably far and away more damaging than keeping a journal while using a primarily read-only system like Porteus!).

Posted after 3 hours 5 minutes 19 seconds:
Re: Finding the best filesystem (for USB flash drive installs)
I sure hope that .dat containers perform better on XFS and/or FAT than they do on EXT4 lol. I set one up to test it out before deleting my EXT4 partition, and it's taking nearly three times as long to copy large numbers of small files... more incentive to use a POSIX compatible filesystem, perhaps? :D

Posted after 17 hours 2 minutes 19 seconds:
Re: Finding the best filesystem (for USB flash drive installs)
@fanthom --

an XFS formatted .dat container performs a little worse than native XFS in some regards -- it is especially (much) worse at writing many small files -- I feel bad for anyone who extracts a kernel tarball with saved changes on a .dat lol. The .dat container did better on native XFS than it did on native EXT4. Of course, all of this doesn't matter as much as how the .dat performs on FAT and/or NTFS (which I'll test after I finish up testing native filesystems). I'm also going to

Native XFS did not perform very well in general for writing/reading/deleting many small files, and it must have some peculiar way of dealing with overwriting files -- somehow it managed to run out of room on a 2GB partition when all the partition had was the kernel tarball and the extracted kernel sources (should be about 500MB). So, overwriting many small files is the worst for XFS, as it took 5 times longer than other file systems, and didn't even finish the job.

That said, XFS has had some recent development, and has changed the way its' log works. It now uses a 'delayed log' similar to EXT, and that is supposed to make it as fast as BTRFS and EXT4 when dealing with many small files. This can be tested in linux 2.6.35+ by mounting with '-o delaylog', and has become the default mount option in 2.6.39+ (sources: http://xfs.org/index.php/Improving_Meta ... l_Overhead , XFS: Disadvantages). I ran one test with the delaylog mount option and saw some improvements, but it still didn't compare well with EXT4 for handling many small files on my flash drive. Still, with all of the ongoing development, XFS will need to stay on the list of filesystems to watch. Between ease of installation (i.e. works with extlinux), speed, and recovery handling, my money right now is still on EXT4 with a journal in ordered mode (default).
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
BlackRider
Black ninja
Black ninja
Posts: 70
Joined: 13 Jul 2011, 11:04
Location: Nowhere
Contact:

Re: Finding the best filesystem (for USB flash drive install

Post#18 by BlackRider » 20 Jul 2011, 15:50

You have to test XFS with bigger files (several Gb). The issue with small files is well known.

I think only Reiserfs moves bigger files faster than XFS, but it is more fragile.

Sitwon
White ninja
White ninja
Posts: 16
Joined: 27 Jun 2011, 21:24
Location: Silver Spring, MD USA

Re: Finding the best filesystem (for USB flash drive install

Post#19 by Sitwon » 21 Jul 2011, 12:03

JFS is pretty efficient with large files.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#20 by Ahau » 21 Jul 2011, 17:56

JFS appears to be horribly inneficient with small files, especially on USB flash drives (I'm guessing this may have to do with the block sizes not lining up, or some regressions with older filesystems in the newer kernels). I threw out JFS halfway through my first test, when it took 6+ minutes to extract the 000-kernel.xzm module. This takes 40 seconds on EXT2, and 24 seconds on EXT4. The largest single file I transferred was the 300MB porteus ISO (I'm guessing multi GB transfers to a flash drive are rare for most users), and while JFS was somewhat faster with this file than EXT2, it was narrowly beaten by EXT4.

Here's some early results to whet your appetites (results from 64-bit)
Sorry, I can't seem to get these to line up better...

I have some naming conventions here that are a bit confusing-- EXT4-J is EXT4 with a Journal. EXT4-NJ is 'no journal'. XFS-DL is XFS mounted with the delaylog option (NOTE: I only ran one test with this option -- all other figures are the average result of 3 tests). BTRFS-AM is BTRFS mounted with the following options: rw,noatime,nodatasum,nodatacow,ssd_spread (standard BTRFS mount is only rw,noatime)

Sustained Write Speed (copy 300MB ISO), given in Megabytes per second (MB/s):

Code: Select all

NTFS	FAT	EXT2	EXT3	EXT4-J	EXT4-NJ	XFS	XFS-DL	REIS	BTRFS	BTRFS-AM
5.09	5.53	4.65	4.20	5.72	 5.81	 5.87	5.89	4.80	5.92	5.91
Sustained Read Speed (copy 300MB ISO back to RAM), MB/s:

Code: Select all

NTFS	FAT	EXT2	EXT3	EXT4-J	EXT4-NJ	XFS	XFS-DL	REIS	BTRFS	BTRFS-AM
17.65	17.81	17.64	17.82	17.05	17.39	17.84	17.83	17.66	18.56	18.61
Module Extraction Speed (extract 000-kernel.xzm module onto disk), MB/s:

Code: Select all

NTFS	FAT	EXT2	EXT3	EXT4-J	EXT4-NJ	XFS	XFS-DL	REIS	BTRFS	BTRFS-AM
1.48	1.59	1.85	1.94	3.07	3.17	3.20	3.54	3.55	4.33	4.63
Copy2RAM Speed (loop mount 000-kernel.xzm on disk and copy contents to RAM), MB/s:

Code: Select all

NTFS	FAT	EXT2	EXT3	EXT4-J	EXT4-NJ	XFS	XFS-DL	REIS	BTRFS	BTRFS-AM
22.96	13.28	20.11	23.20	23.19	23.00	23.39	23.54	22.94	18.01*	23.30
Module Compression Speed (compress contents of 000-kernel.xzm back into a module), MB/s:

Code: Select all

NTFS	FAT	EXT2	EXT3	EXT4-J	EXT4-NJ	XFS	XFS-DL	REIS	BTRFS	BTRFS-AM
1.22	0.66	1.02	1.24	1.16	1.14	1.22	1.18	1.14	0.95*	1.21
*NOTE: these two figures for BTRFS appear to be too low, I need to re-run one of my tests. The actual rates are closer to what is shown for BTRFS-AM. About one in six of my tests on the 64-bit platform come back with unusually low speeds. When this occurrs I throw out that data and run the test again, but I didn't see that I had one of these faulty tests for BTRFS until I put in in my spreadsheet and saw an unusually high standard deviation. I have yet to identify whether this is an issue with the OS or my hardware, but additional testing with another USB drive will help narrow this down.

Write many small files (Extract kernel source tarball located in RAM onto device), MB/s:

Code: Select all

NTFS	FAT	EXT2	EXT3	EXT4-J	EXT4-NJ	XFS	XFS-DL	REIS	BTRFS	BTRFS-AM
2.51	1.25	2.35	1.58	3.84	4.06	1.21	1.47	1.56	4.09	4.09
Read many small files (Copy the extracted kernel source files from the disk back to RAM), MB/s:

Code: Select all

NTFS	FAT	EXT2	EXT3	EXT4-J	EXT4-NJ	XFS	XFS-DL	REIS	BTRFS	BTRFS-AM
2.72	9.98	8.95	8.99	8.81	8.57	9.38	8.75	11.12	11.22	12.26
Please take a look at our online documentation, here. Suggestions are welcome!

pizzar0
White ninja
White ninja
Posts: 28
Joined: 08 May 2011, 22:26
Location: Chritmas Island

Re: Finding the best filesystem (for USB flash drive install

Post#21 by pizzar0 » 02 Aug 2011, 21:26

the again, XFS with an external log device and an ag-count thats the proper (2n) multiple of the usb drive's channel count (w barriers) will smoke just about all of the above results.... by a mile.

Corsair Voyager USB 3; 32 GB
mkfs.xfs -f -l logdev=/dev/loop(X-1),size=128m,lazy-count=1 -d agcount=32
mount -t xfs -o noatime,nodiratime,logdev=/dev/loop9,logbufs=8 /dev/loop10 /VoyTest

1) 740 MB iso: Read 94 MB/s; Write 82 MB/s
2) 800 x 521k: - " - 86 MB/s; Write 79 MB/s

the same as above but with:
mkfs.ext4 /dev/sdX; mount -o noatime /dev/sdX /VoyTest1

1) 740 MB iso: Read 75 MB/s; Write 71 MB/s
2) 800 x 521k: - " - 72 MB/s; Write 62 MB/s

Similar ratios hold true for USB 2.0, as well.

p.s. There SHOULD NOT be a question whether a file-backed loop device is better or worse than a partition backed one. File-backed loop devices perform 0 read and 0 write after they crap, as they always will. Its not an IFF but When - unless no journal! (Weren't designed to handle journaling!!)

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#22 by Ahau » 03 Aug 2011, 14:27

Thanks, pizzar0! I'll try those options out on my end. However, I won't be testing with an external log device. I'm not performing this exercise to learn how to tweak one particular system, I'm looking for the best overall filesystem for USB flash drives running Porteus. Unless I'm missing something, using an external log device would not be practical for running Porteus in always fresh mode on multiple machines (unless it makes sense to flush your log every time you reboot), or for use as an XFS formatted save.dat.

Would you mind timing the results of extracting a kernel tarball to XFS vs EXT4 on that drive? That's where I really see a stumbling block with XFS (lots of tiny files).

I don't know how many channels my flashdrive has. I know that it can handle 12 open allocation units at a time, so I'm going with agcount=24. If there's a good way to find the number of channels, please let me know :) I know I'm horribly underqualified in terms of experience, but I'm doing what I can to learn.

Posted after 16 hours 42 minutes 14 seconds:
Re: Finding the best filesystem (for USB flash drive installs)
Well, I clearly did something wrong... using the mkfs.xfs and mount options as shown below, large files moved at the same rate, but it took 25 minutes to extract the kernel tarball to an XFS formatted partition on my flash drive, over 5 minutes to delete it, and nearly an hour to overwrite it. I know I'm testing the worst case scenario for XFS here, but I do perform kernel recompiles from within Porteus, with the sources saved on my flashdrive. Here's how I made and mounted this partition (sorry, the line returns didn't format correctly):

Code: Select all

bash-4.1# mkfs.xfs -f -d agcount=24 /dev/sdb2
meta-data=/dev/sdb2              isize=256    agcount=24, agsize=60032 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=1440768, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
bash-4.1# 
bash-4.1# mount -t xfs -o noatime,nodiratime,logbufs=8 /dev/sdb2 /mnt/sdb2
bash-4.1#
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
francois
Contributor
Contributor
Posts: 6434
Joined: 28 Dec 2010, 14:25
Distribution: xfce plank porteus nemesis
Location: Le printemps, le printemps, le printemps... ... l'hiver s'essoufle.

Re: Finding the best filesystem (for USB flash drive install

Post#23 by francois » 03 Aug 2011, 16:31

Hello Ahau. Very interesting topic. May I ask a really thight summary of your findings up to date? :oops:
Prendre son temps, profiter de celui qui passe.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#24 by Ahau » 03 Aug 2011, 18:55

The briefest of brief summaries:

EXT4 seems to outperform all other filesystems, except for BTRFS, which is still experimental.

The alignment of your flashdrive seems to be important -- if fdisk -l shows that your partitions start on an odd sector (e.g. 63), your performance will suffer a little; your drive's lifespan may suffer much more. Aligning your partitions to a page size (typically 8 or 16KB) is of primary importance; aligning to an eraseblock boundary (typically 1MB, 2MB, 4MB, etc) is of secondary importance.

With default creation/mount options, XFS performs very poorly for writing, reading, and deleting many small files (e.g. extracting a kernel tarball). This suggests we could improve performance of save.dat containers by switching to another filesystem. As of this morning, I've tested save.dat's formatted with XFS, EXT4 (with and without journaling), NILFS2 and BTRFS. BTRFS .dats seem to be very unstable (or I am missing something in creating or mounting them). Of the other options, XFS is the slowest (particularly with writing many small files at once--again), EXT4 without a journal option is the fastest. NILFS is just slightly slower and actually beats EXT4 with a journal in most regards. NILFS2 also has some cool options for creating checkpoints and snapshots -- however, it's more work to create one, and it is still experimental. I need to perform additional research/testing in order to determine just how bad it is to run EXT4 without a journal ;)
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
francois
Contributor
Contributor
Posts: 6434
Joined: 28 Dec 2010, 14:25
Distribution: xfce plank porteus nemesis
Location: Le printemps, le printemps, le printemps... ... l'hiver s'essoufle.

Re: Finding the best filesystem (for USB flash drive install

Post#25 by francois » 03 Aug 2011, 18:59

Succinct is the word. Thanks a lot :), definition:
http://www.merriam-webster.com/dictionary/succinct
"marked by compact precise expression without wasted words" 8)
Prendre son temps, profiter de celui qui passe.

User avatar
wread
Module Guard
Module Guard
Posts: 1255
Joined: 09 Jan 2011, 18:48
Distribution: Porteus v5.0-kde-64 bits
Location: Santo Domingo
Contact:

Re: Finding the best filesystem (for USB flash drive install

Post#26 by wread » 09 Aug 2011, 02:46

I once read a post from Thomas; he wrote he prefered xfs, but he didn't explain why. I would like to see a comparison of this filesystem with all others.

I can imagine that the members of our community who deal with kernel versions and know the details of the different filesystems supported, can understand slight differences among them. Maybe we should let some of our experts explain filesystems that can be used for Porteus.

I usually partition my usb's, first partition windows bootable, second partition xfs or ext3.

Regards!
Porteus is proud of the FASTEST KDE ever made.....(take akonadi, nepomuk and soprano out and you will have a decent OS).
The Porteus Community never sleeps!

User avatar
BlackRider
Black ninja
Black ninja
Posts: 70
Joined: 13 Jul 2011, 11:04
Location: Nowhere
Contact:

Re: Finding the best filesystem (for USB flash drive install

Post#27 by BlackRider » 09 Aug 2011, 11:43

I'm not sure if this is in-topic, but it is interesting anyway.

I have discovered recently that XFS has some recovery issues, because an unclean shutdown caused me to loose twenty files or so. Futher investigation revealed this:
Does XFS still overwrite existing files with zeros, when those
files were open for write at the time of unclean shutdown? This
would violate a basic requirement of Postfix (don't lose data after
fsync). Postfix updates existing files all the time: it updates
queue files as it marks recipients as done, and it updates mailbox
files as it appends mail.
So that's it: the filesystem consistence is guaranteed, but XFS has trouble to preserve individual files from getting doomed.

Glad I have two external backup drives, just in case this kind of issue strikes my internal drive.

User avatar
Hamza
Warlord
Warlord
Posts: 1908
Joined: 28 Dec 2010, 07:41
Distribution: Porteus
Location: France

Re: Finding the best filesystem (for USB flash drive install

Post#28 by Hamza » 09 Aug 2011, 12:01

@BlackRider,

Don't worry, You are in-topic ;)

Cheers!
NjVFQzY2Rg==

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Finding the best filesystem (for USB flash drive install

Post#29 by Ahau » 09 Aug 2011, 14:10

Blackrider, do you know what mount options were in use, and the options passed to mkfs.xfs when the partition was created? Were you using an internal or external log?
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
BlackRider
Black ninja
Black ninja
Posts: 70
Joined: 13 Jul 2011, 11:04
Location: Nowhere
Contact:

Re: Finding the best filesystem (for USB flash drive install

Post#30 by BlackRider » 09 Aug 2011, 23:28

Blackrider, do you know what mount options were in use, and the options passed to mkfs.xfs when the partition was created? Were you using an internal or external log?
Creation options: Defaults provided by Slackware's mkfs.xfs. I just passed the -L option to name the filesystem.

Mount options: noexec, nosuid, nodev, nothing else.

It seems that XFS zeroes the files it is writing when an unclear shutdown happens, because this way they prevent sensitive data from being taken by an attacker exploiting the filesystem in certain ways. They prevent the data from being stolen because they destroy it. Good move!

I have reached a conclusion: XFS is great if you can ensure there are not going to be troubles (you mount the filesystem read only, use an electricity generator to avoid power failures, etc). If there are reasons to suspect that a failure is likely, then stick with any other thing. Maybe ext4 with data=journal and barrier=1.

I have been told that Reiserfs is prone to directory tree destruction under some circumstances. I have to investigate this. Can someone confirm it?

Posted after 48 minutes 51 second:
Re: Finding the best filesystem (for USB flash drive installs)
Those who want to read a good benchmark can click here: http://linuxgazette.net/122/TWDT.html#piszcz

Post Reply