container file system advice

Technical issues/questions of an intermediate or advanced nature.
rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#1 by rych » 10 Jan 2018, 00:16

How do I go about creating my own container image file, similar in spirit to porteus save.dat I guess, that I can mount and unmount and work with like a file system. I'd appreciate a guidance into the search results I'm getting: Docker, overlay fs, overlay2, aufs, etc. or simply doing what's described here: https://unix.stackexchange.com/question ... -container

The image file could be quite large and will be living permanently on a host Windows NTFS disk. As I boot into porteus, I'd mount it and use a normal linux file system to save arbitrary data, not the porteus "changes".

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

container file system advice

Post#2 by Ed_P » 10 Jan 2018, 00:48

Why? :%) I save files I download and backup directly to my NTFS file systems.
Ed

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#3 by rych » 10 Jan 2018, 06:05

same reasons that justify save.dat I guess: symbolic links, linux permissions -- all features of an ext4 file system encapsulated in an NTFS file. Plus, I'm going to leave it on the host between sessions: content hidden from Windows search, perhaps even encrypted.

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

container file system advice

Post#4 by Ed_P » 10 Jan 2018, 06:14

Oh. Ok. :happy62:
Ed

User avatar
wread
Module Guard
Module Guard
Posts: 1255
Joined: 09 Jan 2011, 18:48
Distribution: Porteus v5.0-kde-64 bits
Location: Santo Domingo
Contact:

container file system advice

Post#5 by wread » 10 Jan 2018, 22:39

@rych
I do not have a changes file. I make a module of my changes every time I build or install a new application.

Then I put them all together in a folder named Settings, contained in modules. Then I have:
myDolphin.xzm
myDTSettings.xzm (desktop settings)
myKBSettings.xzm (keyboard settings)
myKDEConnect.xzm (cellphone control)
myKingSoftSettings.xzm (office programs settings)
myKPatSettings.xzm (solitarie game settings)
myLocalTime.xzm (local time and zone settings)
myLoginSound.xzm (login sound setting)
myMCSettings.xm(midnight commander settings)
myPDFEditorSettings.xzm
myRcLocalSettings.xzm
myRVSettings.xzm (network access settings)
myWiFiSettings.xzm

It is not straight forward to build such modules. You must hack a little bit, but they load faster than the changes or the rootcopy folder. And ... I use porteus as root....

Geschmack Sache, sagte der Affe! (It is a matter of taste, said the monkey!)

Cheers!
Porteus is proud of the FASTEST KDE ever made.....(take akonadi, nepomuk and soprano out and you will have a decent OS).
The Porteus Community never sleeps!

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

container file system advice

Post#6 by brokenman » 13 Jan 2018, 12:10

You can create an image file like so:

Code: Select all

dd if=/dev/zero of=/tmp/myimage.dat bs=1024 count=10000
mkfs.ext4 /tmp/myimage.dat
This will create a 10MB image.
How do i become super user?
Wear your underpants on the outside and put on a cape.

tome
Contributor
Contributor
Posts: 675
Joined: 26 Jun 2013, 14:03
Distribution: x64 Openbox
Location: Russia is causing the immense damage to humanity
Contact:

container file system advice

Post#7 by tome » 13 Jan 2018, 19:03

Some advices to create quickly large file, maybe on NTFS also: https://stackoverflow.com/questions/257 ... nux-system
You have mind and feelings. Be wise and clever.

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#8 by rych » 03 Aug 2022, 13:26

Over the years I've come to the conclusion that the best container file in Porteus is xzm module :)

wread, I've seen you mentioning your method in posts, could you share your scripts please or help me write my own?
wread wrote:
10 Jan 2018, 22:39
I do not have a changes file. I make a module of my changes every time I build or install a new application.

...

It is not straight forward to build such modules. You must hack a little bit, but they load faster than the changes or the rootcopy folder. And ... I use porteus as root....
For example, I want to apply your method to my Firefox Profile. Let's say my Firefox has been set up to load a profile located at /tmp/FirefoxProfile. The idea is that that folder doesn't exist until I activate my module FirefoxProfile.xzm. Running Firefox is going to read and write (changes) to that folder. On quitting Firefox a script is to be run that creates a new FirefoxProfile.xzm from the modified folder /tmp/FirefoxProfile, deactivates and deletes the old FirefoxProfile.xzm, deletes the folder /tmp/FirefoxProfile, preserves the new FirefoxProfile.xzm.

In particular, how do you make an xzm module from /tmp/FirefoxProfile preserving the directory tree like that, so that it expands onto the same path when activated? If I use

Code: Select all

mksquashfs -keep-as-directory
it still doesn't store the /tmp/ part? Or do you use something like

Code: Select all

mksquashfs -no-strip
mksquashfs -tarstyle
( https://manpages.debian.org/unstable/sq ... .1.en.html )

Also, what would be a difference, if any, between /tmp/FirefoxProfile and /mnt/live/tmp/FirefoxProfile?

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

container file system advice

Post#9 by Ed_P » 03 Aug 2022, 14:42

This is how I make a module rych.

Code: Select all

#!/bin/sh
if [ `whoami` != "root" ]; then
   echo -e "Enter root's password\033[1;31m"
   su -c "sh $0 $1"
   exit
fi
echo -e "\033[0m"; echo -en "\033]0;Make Mod\a"

if [ -d /tmp/mod/ ]; then
   rm -rf /tmp/mod
fi
mkdir -p /tmp/mod
cp -a --parents /lib/firmware/iwlwifi-QuZ-a0-hr-b0-71.ucode  /tmp/mod/
cp -a --parents /lib/firmware/iwlwifi-QuZ-a0-jf-b0-71.ucode  /tmp/mod/
dir2xzm   /tmp/mod/                   /tmp/DellInspiron14wifi.xzm
rm    -rf /tmp/mod/ && echo && ls -sh /tmp/*.xzm
read
Added in 10 hours 28 minutes 54 seconds:
Another option I use, create a save.dat file, make all your changes to the system, exit Porteus (I use the cheatcode changes=EXIT for this), rename the save.dat file to be save.xzm and put it in your modules folder. :happy62:
Ed

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#10 by rych » 04 Aug 2022, 10:25

Ed_P wrote:
04 Aug 2022, 01:11
This is how I make a module rych.
-- can't we use symbolic links, we actually have to copy stuff to the mod folder?
Ed_P wrote:
04 Aug 2022, 01:11
Another option I use, create a save.dat file, make all your changes to the system, exit Porteus (I use the cheatcode changes=EXIT for this), rename the save.dat file to be save.xzm and put it in your modules folder.
-- when do you get the opportunity to turn save.dat into a module and move it when the system already shut down? Your changes are written at the very end. So I guess, you have an early startup script that quickly finds save.dat, renames and moves it to modules folder before porteus starts loading modules?

User avatar
wread
Module Guard
Module Guard
Posts: 1255
Joined: 09 Jan 2011, 18:48
Distribution: Porteus v5.0-kde-64 bits
Location: Santo Domingo
Contact:

container file system advice

Post#11 by wread » 04 Aug 2022, 12:59

@rych
I will explain how I store my changes "per application" in Porteus.

Normally, settings are stored in a hidden file, a file starting with a point that does not show in the home screen of the user. The trick is to find your selected settings in one of those pointed files and make a xzm module with it.

The next time you start as always fresh, mySettings module will be also installed, voila!

That was the theory, now a practice example:
A couple of weeks ago, I installed the application "Ookla Speed Test". After entering the required settings and before login out I went to "root>(switched to view hidden files)>.config>(and bingo!)ookla>speedtest-cli.json". In this last file were my settings stored.

Now let's make the xzm module to it: Folder>New(rename it "myOoklaSettings")>Folder>New(rename it "root")>Folder>New>(rename it ".config")>Folder>New(rename it "ookla", then open that folder and copy speedtest-cli.json into it. Now convert dir2xzm on myOoklaSettings and you are done!

Fine bussinnes!!
Porteus is proud of the FASTEST KDE ever made.....(take akonadi, nepomuk and soprano out and you will have a decent OS).
The Porteus Community never sleeps!

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#12 by rych » 06 Aug 2022, 10:33

wread, Ed_P, you and the official /opt/porteus-scripts/update-module copy stuff into a temporary folder before turning it into a module. However, the current version of mksquashfs has the -no-strip parameter which preserves the full path. So we can run it on the actual mounted union directory we're trying to preserve in the module being updated/rewritten. For example in my case, after closing Firefox I simply run:

Code: Select all

COMP_ZSTD="-comp zstd -b 256K -Xcompression-level 22"
mksquashfs /root/.mozilla/firefox/FirefoxProfile $PORTDIR/optional/FirefoxProfile.xzm $COMP_ZSTD -noappend -no-strip
then remove the /root/.mozilla/firefox/FirefoxProfile (or not save it on EXIT:)

Next time I activate the optional/FirefoxProfile.xzm and it expands to /root/.mozilla/firefox/FirefoxProfile

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

container file system advice

Post#13 by Ed_P » 06 Aug 2022, 22:35

rych wrote:
06 Aug 2022, 10:33
the current version of mksquashfs has the -no-strip parameter which preserves the full path. So we can run it on the actual mounted union directory we're trying to preserve in the module
And do the current Porteus 5.0 ISOs have this version of mksquashfs? :%)
Ed

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#14 by rych » 07 Aug 2022, 03:18

Ed_P wrote:
06 Aug 2022, 22:35
do the current Porteus 5.0 ISOs have this version of mksquashfs
Apparently so, as I'm using it and didn't have to install anything extra. Reportedly it was introduced upstream in squashfs-tools a year ago: https://lore.kernel.org/lkml/CAB3woddJs ... gmail.com/ -- search on page for the word "strip" :)

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

container file system advice

Post#15 by rych » 07 Aug 2022, 08:48

I've noticed that dropping -Xcompression-level 22 from

Code: Select all

COMP_ZSTD="-comp zstd -b 256K -Xcompression-level 22"
, thereby using the unknown default value results in a faster compression and only 0.5% increase in size. Actually even using -Xcompression-level 1 the compression is very fast and only 10% larger file. Does anyone have any info on a recommended compression-level for zstd? Otherwise, will be using the unknown default value

Post Reply