container file system advice

Technical issues/questions of an intermediate or advanced nature.
Shadow0815
Black ninja
Black ninja
Posts: 33
Joined: 16 Jul 2015, 03:50
Distribution: Porteus,Xubuntu,Arch
Location: DE

container file system advice

Post#16 by Shadow0815 » 08 Aug 2022, 17:39

I boot to RAM and i can't use rootcopy folder because i use fat file system. creating a changes.xzm and changing config files with: copyxzmToDesktop+extract+modify+deleteOldxzm+pack+overwriteToStick is too error prone and work intensive for me.


So i now have a 200MiB:
myChanges.ext4.img as a staging area for my "make persistent" config files.
And bash functions to mount and pack to xzm etc. like:
myChanges-mount will mount to ~/myChanges
myChanges-update will pack to xzm putting it in the modules folder ...


i like to use:
cp -avr --parents <full_path_to_dir> /home/guest/myChanges
cp -av --parents <full_path_to_file> /home/guest/myChanges


my ~/.mozilla firefox default profile comes from an xzm file but on laptop i start laptop.sh script and it moves that profile to /tmp and creates a /home/guest/.mozilla symlink to laptop hdd ...
(laptop.sh does also create more symlinks somewhere inside /home/guest to stuff on my laptop hdd ...)

so i have porteus universal usable but still use custom profile/config persistent stuff from laptop HDD when on laptop ...

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#17 by rych » 22 Nov 2022, 08:11

rych wrote:
03 Aug 2022, 13:26
Over the years I've come to the conclusion that the best container file in Porteus is xzm module
Furthermore, another option for stand-alone programs is just to make a squashfs image and mount it (read-only) on demand. Instead of xzm, that is, and therefore avoiding the aufs layering or keeping track of changes. It's an option for the (portable folder) programs that don't need to write into its installation folder. If occasionally a self-update is required, there is still an option to extract rather than mount, to modify, and then to make a new .sqfs module. See my helper little scripts and context menu desktop files in my other thread: why modules (Post by rych #91537)

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

container file system advice

Post#18 by Rava » 22 Dec 2022, 02:18

rych wrote:
10 Jan 2018, 00:16
How do I go about creating my own container image file
I do it like so, for that purpose I create ext2 container files.

Code: Select all

dd if=/dev/zero of=YOURNAMEHERE.ext2 bs=1M count=60 # create 60 MB container
mkfs.ext2 -F YOURNAMEHERE.ext2 -L LABEL # format
tune2fs -r 50 YOURNAMEHERE.ext2          # tune
Replace YOURNAMEHERE with the name of your ext2 container.
Replace LABEL with the label your container should have. You could leave the label part out but I recommend using it, it doesn't slow the mkfs.ext2 command and it gives you a hint, maybe many many moons later for what a certain ext2 container was initially created for, e.g. when you left it lying around unused for a while.
The tune2fs reduces the often unneeded high standard reserved_blocks_counts to a mere 50.

And of course adjust the "60" in the dd count= part to whatever MB your container should have.
E.g. you want 1 GB, use this:

Code: Select all

dd if=/dev/zero of=lsfind.ext2 bs=1M count=1024
Cave! The tweak of reducing the standard reserved_blocks_counts to 50 is okay for a small container, like 20 MB or 80 MB, but it should definitely be higher for a much larger container file, e.g. a mere 50 standard reserved_blocks_counts is too few for an ext2 1 GB container file.

Please feel free to either have a quick glance at all the other options (and what the few options I use as quoted above are for) by doing these

Code: Select all

dd --help
mkfs.ext2   # there is no mkfs.ext2 --help nor is there a mkfs.ext2 -h
tune2fs # there is no tune2fs --help nor is there a tune2fs -h
though I personally recommend the man pages:

Code: Select all

man dd
man mkfs.ext2
man tune2fs 
(man mkfs.ext2 is actually man mke2fs)

When you use reserved-blocks-percentage instead of standard reserved_blocks_counts you can give that parameter to mkfs.ext2 directly and thus skip the invocation of tune2fs entirely. And yes, a half percent of reserved-blocks-percentage would work:

Code: Select all

root@porteus:/tmp# df -Tm .
Filesystem     Type 1M-blocks  Used Available Use% Mounted on
aufs           aufs       388    62       326  16% /
root@porteus:/tmp# dd if=/dev/zero of=TEST.ext2 bs=1M count=100 # create 100 MB container
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.0661513 s, 1.6 GB/s
root@porteus:/tmp# mkfs.ext2 -F TEST.ext2 -L 100MB_TEST -m 0.5
mke2fs 1.46.5 (30-Dec-2021)
Discarding device blocks: done                            
Creating filesystem with 102400 1k blocks and 25584 inodes
Filesystem UUID: 94c3f80f-bf87-4241-b7e4-cac88b65570a
Superblock backups stored on blocks: 
	8193, 24577, 40961, 57345, 73729

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done 

root@porteus:/tmp# fsck.ext2 TEST.ext2
e2fsck 1.46.5 (30-Dec-2021)
100MB_TEST: clean, 11/25584 files, 7985/102400 blocks
root@porteus:/tmp# df -Tm .
Filesystem     Type 1M-blocks  Used Available Use% Mounted on
aufs           aufs       388    63       326  17% /
root@porteus:/tmp# ls -o TEST.ext2
-rw-r--r-- 1 root 104857600 2022-12-22 03:16 TEST.ext2
root@porteus:/tmp# mkdir /100MB_TEST
root@porteus:/tmp# mount TEST.ext2 /100MB_TEST
root@porteus:/tmp# df -Tm /100MB_TEST
Filesystem     Type 1M-blocks  Used Available Use% Mounted on
/dev/loop20    ext2        93     1        92   1% /100MB_TEST
HTH! :)

Added in 22 hours 22 minutes 46 seconds:
Not only would a half percent work when invoking

Code: Select all

-m 0.5
but also less than that, e.g.

Code: Select all

-m 0.2
or even

Code: Select all

-m 0.1
But do not go too small, while your created container file gets a bit more usable space, its safety can be compromised when you reduce the reserved-blocks-percentage or reserved_blocks_count by too much.

For my lsfind suite I use 60 MB container files for years now, and these all have 50 reserved blocks which is much lower than what mkfs.ext2 initially assigns - and I never ran into any issues. But a 60 MB container file is not a very large one, and 50 reserved blocks is to be seen in relations to the 60 MB - therefore I would not use a mere 50 reserved blocks for a container file of 500 MB.
(Calculating this relatively, using 50 reserved blocks for a container file of 60 MB translates into 0.8333 reserved blocks / MB . Meaning a 500 MB container file would have the same relative amount of reserved blocks / MB when giving it 417 reserved blocks.
Or for a 1 GB container file (1024 MB) : assigning 853 reserved blocks)
Last edited by Rava on 23 Dec 2022, 00:59, edited 5 times in total.
Reason: Coloured the Cave! part to make it stand out more. // Added the missing --help for mkfs.ext2 and tune2fs
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#19 by rych » 23 Dec 2022, 15:42

Thanks. How do you mount the
Rava wrote:
23 Dec 2022, 00:41
YOURNAMEHERE.ext2
after that, with what parameters?

Also, why not use the compressed filesystem -- a SquashFS file instead?

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

container file system advice

Post#20 by Rava » 23 Dec 2022, 16:01

rych wrote:
23 Dec 2022, 15:42
Also, why not use the compressed filesystem -- a SquashFS file instead?
A xzm module is read-only.
I am not familiar in how to set up a compressed file system, sorry.
I keep it uncompressed, that is the more secure way.

How you mount it? easy as pie:

Code: Select all

root@porteus:/# mount  /welt/lsfind/lsfind.ext2 /Lsfind
root@porteus:/# /bin/df -Tm  /Lsfind
Filesystem     Type 1M-blocks  Used Available Use% Mounted on
/dev/loop18    ext2        68    55        14  80% /Lsfind
Simply

Code: Select all

mount /path/to/container.ext2 /existing-folder
it is.
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#21 by rych » 24 Dec 2022, 06:23

Rava wrote:
23 Dec 2022, 16:01
xzm module is read-only.
I am not familiar in how to set up a compressed file system, sorry.
I keep it uncompressed, that is the more secure way
Ah, yes. This is great as it's not read-only. So it's an ext2 container file living on a (host's) NTFS for example. And if we want it to be compressed inside we could instead "format" it as btrfs:
BTRFS is newer and is Linux-only. I've been running it on Ubuntu 14.04 and found it stable. It supports multiple compression methods. I've found LZO compresses fairly well and is really fast; I don't see any performance slowdown from the compression operation.
https://superuser.com/questions/884682/ ... m-on-linux

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

container file system advice

Post#22 by Rava » 24 Dec 2022, 07:19

rych wrote:
24 Dec 2022, 06:23
Rava wrote:
23 Dec 2022, 16:01
xzm module is read-only.
I am not familiar in how to set up a compressed file system, sorry.
I keep it uncompressed, that is the more secure way
Ah, yes. This is great as it's not read-only.
when it's a xzm module like I wrote then it's read-only.

You can test that easily for yourself:
I chose a small xzm module of only 72 KB in size:

Code: Select all

root@porteus:/tmp# file isomaster-1.3.16-x86_64-rava.xzm 
isomaster-1.3.16-x86_64-rava.xzm: Squashfs filesystem, little endian, version 4.0, xz compressed, 73404 bytes, 42 inodes, blocksize: 262144 bytes, created: Sun Dec 18 21:43:56 2022
root@porteus:/tmp# mkdir isomaster-1.3.16-x86_64-rava
root@porteus:/tmp# mount isomaster-1.3.16-x86_64-rava.xzm isomaster-1.3.16-x86_64-rava
root@porteus:/tmp# cd isomaster-1.3.16-x86_64-rava
root@porteus:/tmp/isomaster-1.3.16-x86_64-rava# ls -o
total 0
drwxr-xr-x 5 root 50 2022-12-18 22:41 usr
drwxr-xr-x 4 root 37 2022-12-18 01:54 var
root@porteus:/tmp/isomaster-1.3.16-x86_64-rava# cd var/log/packages
root@porteus:/tmp/isomaster-1.3.16-x86_64-rava/var/log/packages# ls
isomaster-1.3.16-x86_64-1gv
root@porteus:/tmp/isomaster-1.3.16-x86_64-rava/var/log/packages# echo test >>isomaster-1.3.16-x86_64-1gv 
bash: isomaster-1.3.16-x86_64-1gv: Read-only file system
root@porteus:/tmp/isomaster-1.3.16-x86_64-rava/var/log/packages# 
The
bash: isomaster-1.3.16-x86_64-1gv: Read-only file system
is the main point in my mounting an xzm module demonstration.

Please show me how you can mount a xzm module and that where the module is mounted, you can write to any of its files.

You can surely use several methods of compressing an ext2 (or ext[23]) container file.
Though I recommend when using compression on an extfs container file to use ext3 because of the added complexity, a journaling FS is the better choice.
I have no clue about BTRFS - but when it's a rather new FS I presume it is able to use journaling as well.

Added in 4 minutes 25 seconds:
wikipedia entry on Btrfs
Transparent compression via zlib, LZO and (since 4.14) ZSTD, configurable per file or volume
and it also says this:
Btrfs#In-place conversion from ext2/3/4 and ReiserFS
In-place conversion from ext2/3/4 and ReiserFS

As the result of having very little metadata anchored in fixed locations, Btrfs can warp to fit unusual spatial layouts of the backend storage devices. The btrfs-convert tool exploits this ability to do an in-place conversion of an ext2/3/4 or ReiserFS file system, by nesting the equivalent Btrfs metadata in its unallocated space—while preserving an unmodified copy of the original file system.[71]

The conversion involves creating a copy of the whole ext2/3/4 metadata, while the Btrfs files simply point to the same blocks used by the ext2/3/4 files. This makes the bulk of the blocks shared between the two filesystems before the conversion becomes permanent. Thanks to the copy-on-write nature of Btrfs, the original versions of the file data blocks are preserved during all file modifications. Until the conversion becomes permanent, only the blocks that were marked as free in ext2/3/4 are used to hold new Btrfs modifications, meaning that the conversion can be undone at any time (although doing so will erase any changes made after the conversion to Btrfs).[71]

All converted files are available and writable in the default subvolume of the Btrfs. A sparse file holding all of the references to the original ext2/3/4 filesystem is created in a separate subvolume, which is mountable on its own as a read-only disk image, allowing both original and converted file systems to be accessed at the same time. Deleting this sparse file frees up the space and makes the conversion permanent.[71]

In 4.x versions of the mainline Linux kernel, the in-place ext3/4 conversion was considered untested and rarely used.[71] However, the feature was rewritten from scratch in 2016 for btrfs-progs 4.6.[45] and has been considered stable since then.

In-place conversion from ReiserFS was introduced in September 2017 with kernel 4.13.
Added in 2 minutes 47 seconds:
The only time "journal" is to be found on that wiki entry is this:
Log tree
An fsync request commits modified data immediately to stable storage. fsync-heavy workloads (like a database or a virtual machine whose running OS fsyncs frequently) could potentially generate a great deal of redundant write I/O by forcing the file system to repeatedly copy-on-write and flush frequently modified parts of trees to storage. To avoid this, a temporary per-subvolume log tree is created to journal fsync-triggered copies on write. Log trees are self-contained, tracking their own extents and keeping their own checksum items. Their items are replayed and deleted at the next full tree commit or (if there was a system crash) at the next remount.
(coloured highlighting by me)

Added in 3 minutes 43 seconds:
And then I found these:
https://itsfoss.com/btrfs/
If you are one of them who want to switch to Btrfs, let me help you by listing the Pros and Cons of Btrfs as a file system. This will help you to decide whether to switch or not.
[…]

Compression

Btrfs also supports data compression, reducing the file size so you can store more data in less space. This also increases the lifespan of flash-based media storage devices (e.g., SSD, eMMC) as less data is written to disk.

It also increases performance for a process which do not load many files from your HDD. However, if a process needs to access more files from the HDD it can cause some slow down as decompression is done to access files.
https://btrfs.wiki.kernel.org/index.php/Main_Page
no hit for "journal" on the main btrfs.wiki.kernel.org page.

Added in 2 minutes 37 seconds:
But there is an extra entry on the kernel wiki for btrfs compression:
https://btrfs.wiki.kernel.org/index.php/Compression
What are the differences between compression methods?

There's a speed/ratio trade-off:

ZLIB -- slower, higher compression ratio (uses zlib level 3 setting, you can see the zlib level difference between 1 and 6 in zlib sources).
LZO -- faster compression and decompression than zlib, worse compression ratio, designed to be fast
ZSTD -- (since v4.14) compression comparable to zlib with higher compression/decompression speeds and different ratio levels (details)

The differences depend on the actual data set and cannot be expressed by a single number or recommendation. Do your own benchmarks. LZO seems to give satisfying results for general use.

Are there other compression methods supported?
Currently no, and with ZSTD, there are no further plans to add more. […]
:)
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#23 by rych » 25 Dec 2022, 10:18

Rava wrote:
24 Dec 2022, 07:33
BTRFS
Turns out we don't have it:

Code: Select all

dd if=/dev/zero of=container.btrfs bs=1M count=60
mkfs.btrfs container.btrfs 
bash: mkfs.btrfs: command not found

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

container file system advice

Post#24 by Rava » 25 Dec 2022, 10:26

^
Seems like so:

Code: Select all

root@porteus:~# getmod -m mkfs.btrfs
Reading Package Lists...Done
No such package: mkfs.btrfs
root@porteus:~# getmod -m btrfs
Reading Package Lists...Done
No such package: btrfs
Could also be that the package itself has a different name?

Added in 2 minutes 31 seconds:
Seems package is called btrfs-progs
https://slackware.pkgs.org/15.0/slackwa ... 1.txz.html
Btrfs is a new copy on write filesystem for Linux aimed at implementing
advanced features while focusing on fault tolerance, repair and easy
administration. Initially developed by Oracle, Btrfs is licensed under
the GPL and open for contribution from anyone.
Homepage: http://btrfs.wiki.kernel.org
URL for Slackware 15 x86-64:
https://slackware.uk/slackware/slackwar ... 6_64-1.txz
According to https://slackware.pkgs.org package size s a mere 776.00 KB
Cheers!
Yours Rava

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

container file system advice

Post#25 by Rava » 09 Jan 2023, 01:21

My system:

Code: Select all

guest@porteus:~$ cat /etc/porteus/00*
001-core.xzm:20221211
002-xorg.xzm:20221211
002-xtra.xzm:20221211
003-xfce.xzm:20220925
I downloaded https://slackware.uk/slackware/slackwar ... 6_64-1.txz and created a local btrfs-progs-5.16-x86_64-1.xzm module (size quite larger than the package - 2.06 MB

Activated btrfs-progs-5.16-x86_64-1.xzm

Creating a 60 MB container file failed:

Code: Select all

root@porteus:/tmp/TEST# dd if=/dev/zero of=container.btrfs bs=1M count=60
60+0 records in
60+0 records out
62914560 bytes (63 MB, 60 MiB) copied, 0.0357233 s, 1.8 GB/s
root@porteus:/tmp/TEST# mkfs.btrfs container.btrfs 
btrfs-progs v5.16 
See http://btrfs.wiki.kernel.org for more information.

ERROR: 'container.btrfs' is too small to make a usable filesystem
ERROR: minimum size for each btrfs device is 114294784
Since 114294784 byte translates into 109 MB I tried again:

Code: Select all

root@porteus:/tmp/TEST# dd if=/dev/zero of=container.btrfs bs=1M count=109
109+0 records in
109+0 records out
114294784 bytes (114 MB, 109 MiB) copied, 0.0624429 s, 1.8 GB/s
root@porteus:/tmp/TEST# mkfs.btrfs container.btrfs 
btrfs-progs v5.16 
See http://btrfs.wiki.kernel.org for more information.

NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               9513bd4a-9152-4170-b046-b123acd454b8
Node size:          16384
Sector size:        4096
Filesystem size:    109.00MiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              32.00MiB
  System:           DUP               8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1   109.00MiB  container.btrfs

root@porteus:/tmp/TEST# file container.btrfs 
container.btrfs: BTRFS Filesystem sectorsize 4096, nodesize 16384, leafsize 16384, UUID=9513bd4a-9152-4170-b046-b123acd454b8, 147456/114294784 bytes used, 1 devices
and it works.
seems when you want a container file of the minimum size of 109 MB, then you can use mkfs.btrfs
If you want a smaller container (e.g. by test size of 60 MB) you have to go with e.g. ext2.

Added in 11 minutes 27 seconds:
I cannot say that I like btrfs as a container file system:

Code: Select all

root@porteus:/tmp/TEST# mkdir /BTRFS-TEST
root@porteus:/tmp/TEST# mount container.btrfs /BTRFS-TEST/
root@porteus:/tmp/TEST# ls -ao /BTRFS-TEST
total 16root@porteus:/tmp/TEST# mkdir /BTRFS-TEST
root@porteus:/tmp/TEST# mount container.btrfs /BTRFS-TEST/
root@porteus:/tmp/TEST# ls -ao /BTRFS-TEST
total 16
drwxr-xr-x  1 root   0 2023-01-09 02:15 .
drwxrwxrwt 94 root 400 2023-01-09 02:23 ..
root@porteus:/tmp/TEST# df -Tm /BTRFS-TEST
Filesystem     Type  1M-blocks  Used Available Use% Mounted on
/dev/loop21    btrfs       109     4        28  12% /BTRFS-TEST

drwxr-xr-x  1 root   0 2023-01-09 02:15 .
drwxrwxrwt 94 root 400 2023-01-09 02:23 ..
root@porteus:/tmp/TEST# df -Tm /BTRFS-TEST
Filesystem     Type  1M-blocks  Used Available Use% Mounted on
/dev/loop21    btrfs       109     4        28  12% /BTRFS-TEST
From 109 MB, when 4 MB are used, why are only 28 MB available?
109 minus 4 should state that 105 MB is available, not a mere 28. :confused:

Let's compare that to my auto-mounted at boot /Lsfind ext2 container file:

Code: Select all

root@porteus:~# df -Tm /Lsfind/
Filesystem     Type 1M-blocks  Used Available Use% Mounted on
/dev/loop18    ext2        68    54        15  80% /Lsfind
68 minus 54 would be 14, the error that it's 15 MB free and not 14 MB free is a rounding error.

Maybe this is explained in detail in some btrfs readme or in man mkfs.btrfs.8 but I am too lazy to browse through all that. (And maybe the info about that is in one of the other man pages - there are 26 man pages included in btrfs-progs-5.16-x86_64-1.xzm after all.)

Code: Select all

root@porteus:/# man mkfs.btrfs.8|wc
    583    3231   31839
583 lines, 3231 words and 31839 bytes is on the larger size when it comes to man pages.

Added in 1 minute 55 seconds:
When it comes to a container file system, I will stick with either ext2 or ext3 and skip on using btrfs for now. :)
But as you can see above, it can easily implemented in Porteus 5.0. :D

Added in 13 minutes 3 seconds:
Just as additional heads-up:
I umounted /BTRFS-TEST
I deactivated btrfs-progs-5.16-x86_64-1.xzm
and then mounted again /tmp/TEST/container.btrfs on /BTRFS-TEST/ to see if that works without the need of the btrfs module, and it does.

So, creating a btrfs file system or a btrfs container file, you need the btrfs tools module activated.
But only mounting and umounting an already created btrfs container file you not need the btrfs tools module - unless you want a btrfsck.
Cheers!
Yours Rava

Post Reply