why modules

Technical issues/questions of an intermediate or advanced nature.
nanZor
Shogun
Shogun
Posts: 381
Joined: 09 Apr 2019, 03:27
Distribution: Porteus 5.01 x86-64 LXQT

why modules

Post#16 by nanZor » 22 Sep 2022, 01:33

What I love about modules is the reusability.

Once I have a module I want, I copy it somewhere else for safekeeping so I don't have to download it again.

Then if I make a new Porteus stick, I simply transfer the saved modules to the new stick without having to go online and get them all again etc.

Being able to toggle the modules on / off at will blew my mind for those older ram-limited machines. I dig modules!
That's a UNIX book - cool. -Garth

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#17 by rych » 12 Nov 2022, 12:15

Thank you. I'm now considering to use even my largest software as xzm modules (for example, one of my huge 7GB program installation folder came down to a 2GB xzm module with default compression ratio, so didn't even take long). The xzm module is to reside and be activated from the host NTFS SSD disk.

My next question is, is it better to mount this xzm rather than to "activate" it?! The normal (loop?) mount is going to be read-only, while "activating" makes it part of AUFS union mount with changes layer. Suppose the program files 1) don't need to mix with anything in the target tree and 2) don't change, would we avoid an overhead, if any, of aufs union mount e.g. keeping track of potential changes which we know won't happen?

In fact ALL xzm modules in porteus are currently "activated" into writeable aufs unions. Shouldn't some xzm modules be simply mounted read only with the mount -o loop command?

For example, take the Firefox module with disabled self-update or ncmprhnsbl's blender module: 220MB, activates into /opt/blender I think so a part of aufs union, but it's rather standalone and there won't be any changes overlaying on top of it. So, we could just mount (read-only) that blender module into an empty folder outside aufs, say, "/C/Blender". Would we be gaining a better performance?

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#18 by Rava » 12 Nov 2022, 18:44

rych wrote:
12 Nov 2022, 12:15
My next question is, is it better to mount this xzm rather than to "activate" it?! The normal (loop?) mount is going to be read-only, while "activating" makes it part of AUFS union mount with changes layer. Suppose the program files 1) don't need to mix with anything in the target tree and 2) don't change, would we avoid an overhead, if any, of aufs union mount e.g. keeping track of potential changes which we know won't happen?
I think most programs are coded that way that they are in a normal hard disc installation, therefore the program "thinks" it can read and write its settings files, or it has a certain folder that is part of its installation path but also uses that as cache .

I think most programs would misbehave when they cannot write to such files or folders, therefore activating is the only safe way handling it.

When you do not use the changes= cheatcode (like some people prefer doing it, including yours truly) then nothing what is done or written by the activated module is saved by default.

Therefore I use 3 extra modules to be loaded at the end of all modules (therefore I rename all my modles into the nnn- naming scheme, from 001- to 992- for now.

Code: Select all

985-palemoon-settings--RECENT.xzm
991-usr_local_bin_RECENT.xzm
992-rootcopy_5.0-RECENT.xzm
They hold:
my browser settings incl. all add-ons
my scripts and local programs (991-usr_local_bin) - in an extra module since 991- is meant to be shared with all possible DEs
and the extra 992-rootcopy_5.0 module, which is meant one version for each DE.

And only when I manually copy files into certain folder hierarchies what will then become one of these three modules then that script or settings file will be remembered, by default my system has digital dementia by design.
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#19 by rych » 13 Nov 2022, 06:21

Rava wrote:
12 Nov 2022, 18:44
I think most programs are coded that way that they are in a normal hard disc installation, therefore the program "thinks" it can read and write its settings files, or it has a certain folder that is part of its installation path but also uses that as cache .

I think most programs would misbehave when they cannot write to such files or folders, therefore activating is the only safe way handling it.
I've just turned a 7GB Matlab installation monolith into a 3GB module and instead of "activating" it, "mounted" it. It mounts under /mnt/loop/ as a read-only filesystem. Went to that path and ran the matlab binary from there, and it worked. The question is whether I gained anything or not, or how much overhead aufs union introduce really.

So I'd respectfully disagree and say that many programs actually don't write to their installation folder. Unless they're self-updating. So, one could run Firefox from a read-only FS, and it should run just the same except an attempted self-update in-place. Typically programs keep all the settings, cache, etc. in the profile folder in the user home directory.

Which programs do modify their installation folders we can see from examining the porteus changes folder.

In a sense, I'm considering of bringing some of my modules back to the LiveCD, read-only era. I just wonder whether they'd run faster that way.
Last edited by rych on 14 Nov 2022, 08:14, edited 1 time in total.

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#20 by Rava » 13 Nov 2022, 20:54

rych wrote:
13 Nov 2022, 06:21
Which programs do modify their installation folders we can see from examining the porteus changes folder.
That only works when you use the changes cheatcode, which I do not.

But I know of a program that gets into issues when it cannot write to its settings files:
audacity - at least that's true for audacity-linux-3.2.0-x86_64.xzm and audacity-linux-3.2.1-x86_64.xzm .
Cheers!
Yours Rava

beny
Full of knowledge
Full of knowledge
Posts: 2086
Joined: 02 Jan 2011, 11:33
Location: italy

why modules

Post#21 by beny » 13 Nov 2022, 21:12

hi Rava,when you modify something in the aufs system the /mnt/live/memory/changes store the changes locally, and you can use the save-changes script to make a packages or you can make a package yourself isn't so simple but if you don't have the changes folder this can help sometime.

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#22 by Rava » 13 Nov 2022, 21:20

Thanks beny, usually I use this one liner :

Code: Select all

find . -mtime 0 2>/dev/null | xargs ls -oad --color=auto --time-style=long-iso 2>&1 |grep -vE "cache| .$| ..$"|grep "2021-01-24 08:00"
You have to adjust the date and time at the very end to what is appropriate right now.
That is just the version as displayed by my help.find script.


E.g. as test run:

Code: Select all

guest@porteus:~$ find . -mtime 0 2>/dev/null | xargs ls -oad --color=auto --time-style=long-iso 2>&1 |grep -vE "cache| .$| ..$"|grep "2022-11-13 22:1"
drwxr-xr-x  2 guest    60 2022-11-13 22:14 ./.config/dconf
-rw-r--r--  1 guest  3601 2022-11-13 22:14 ./.config/dconf/user
drwxr-xr-x 13 guest   180 2022-11-13 22:14 ./.local/share
drwx------  2 guest    60 2022-11-13 22:14 ./.local/share/Mousepad
-rw-r--r--  1 guest   903 2022-11-13 22:12 ./.local/share/Mousepad/autosave-10
-rw-r--r--  1 guest  1029 2022-11-13 22:14 ./.local/share/clipit/history
drwxr-xr-x  2 guest   300 2022-11-13 22:15 ./.local/share/gvfs-metadata
-rw-------  1 guest   396 2022-11-13 22:15 ./.local/share/gvfs-metadata/home
-rw-r--r--  1 guest 32768 2022-11-13 22:15 ./.local/share/gvfs-metadata/home-b5cbabb1.log
-rw-------  1 guest 13614 2022-11-13 22:14 ./.local/share/recently-used.xbel
Added in 54 seconds:
"2022-11-13 22:1" of course means all times from 22:10 to 22:19
Cheers!
Yours Rava

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

why modules

Post#23 by Ed_P » 13 Nov 2022, 22:22

Rava wrote:
13 Nov 2022, 21:21
"2022-11-13 22:1" of course means all times from 22:10 to 22:19
Shouldn't that be "2022-11-13 22:1?" to mean all times from 22:10 to 22:19? ;)
Ed

beny
Full of knowledge
Full of knowledge
Posts: 2086
Joined: 02 Jan 2011, 11:33
Location: italy

why modules

Post#24 by beny » 13 Nov 2022, 22:37

if i remember well default time is 3 minutes,but you can change it

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#25 by Rava » 14 Nov 2022, 02:58

Ed_P wrote:
13 Nov 2022, 22:22
Rava wrote:
13 Nov 2022, 21:21
"2022-11-13 22:1" of course means all times from 22:10 to 22:19
Shouldn't that be "2022-11-13 22:1?" to mean all times from 22:10 to 22:19? ;)
No, you best try it out yourself.
Now in my system only one file remains in guest's ~ - with the time stamp of "2022-11-13 22:14" - all other files like quoted above have been changed in the meantime to a newer time stamp.

You best use a search pattern that is older because the most recent one can change - as seen when you ask your system about the current time via

Code: Select all

date +%d.%m.%Y\ %H:%M:%S
because that's the time format used by --time-style=long-iso
The older ones can change as well of course, but still a changing result is less likely than using a current time stamp.

Code: Select all

guest@porteus:~$ find . -mtime 0 2>/dev/null | xargs ls -oad --color=auto --time-style=long-iso 2>&1 |grep -vE "cache| .$| ..$"|grep "2022-11-13 22:1"
-rw-r--r--  1 guest  1029 2022-11-13 22:14 ./.local/share/clipit/history
guest@porteus:~$ find . -mtime 0 2>/dev/null | xargs ls -oad --color=auto --time-style=long-iso 2>&1 |grep -vE "cache| .$| ..$"|grep "2022-11-13 22:1?"
guest@porteus:~$ find . -mtime 0 2>/dev/null | xargs ls -oad --color=auto --time-style=long-iso 2>&1 |grep -vE "cache| .$| ..$"|grep "2022-11-13 22:1[0-9]"
-rw-r--r--  1 guest  1029 2022-11-13 22:14 ./.local/share/clipit/history
guest@porteus:~$ find . -mtime 0 2>/dev/null | xargs ls -oad --color=auto --time-style=long-iso 2>&1 |grep -vE "cache| .$| ..$"|grep "2022-11-13 22:14"
-rw-r--r--  1 guest  1029 2022-11-13 22:14 ./.local/share/clipit/history
You see, the last of the pipes using grep "2022-11-13 22:1?" gives no results.
grep "2022-11-13 22:1[0-9]" here is the same as grep "2022-11-13 22:1" - but that's only the case because only numbers can be at that part of the searched for info returned by ls using the parameter --time-style=long-iso and no other kind of characters can be at that position.
And of course using the literal "4" gives the same results in my example since only an entry with "2022-11-13 22:14" remains.

HTH.
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#26 by rych » 14 Nov 2022, 09:13

Bringing us back on topic, let me share another experiment of mine.

I've mounted my host's SSD disk (carrying NTFS) to /C, and let's say I have a 7GB /C/HugeSoftware folder installation. Converted it to xzm using two methods: "Build zstd module" GUI context menu command which took about 7 minutes because under the hood it's trying the highest -Xcompression-level 22 ... and my own 1-liner:

Code: Select all

mksquashfs HugeSoftware HugeSoftware.xzm -noappend -comp zstd -b 256K
... which took 3 minutes. Both resulted in an xzm module of 2.2GB anyway.

Moving on. We're going to mount (instead of activating) it now like this:

Code: Select all

mv HugeSoftware HugeSoftwareDEL
mkdir HugeSoftware
mount HugeSoftware.xzm /G/HugeSoftware
Notice the simplicity of the mount command: no "-o loop -t squashfs" seem to be needed.

Going into the new mounted path and running the program binary works nicely.

I'm going to pay attention to how it feels running it off a mounted xzm module rather that from a fully expanded folder. For now I don't seem to notice any impact, maybe only in the start-up time from cold, a couple seconds longer start.

Question: why shouldn't all software run from squashfs (xzm) modules?

beny
Full of knowledge
Full of knowledge
Posts: 2086
Joined: 02 Jan 2011, 11:33
Location: italy

why modules

Post#27 by beny » 14 Nov 2022, 13:33

hi list them please

Question: why shouldn't all software run from squashfs (xzm) modules?

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#28 by Rava » 14 Nov 2022, 23:03

rych wrote:
14 Nov 2022, 09:13
Question: why shouldn't all software run from squashfs (xzm) modules?
You mean, using e.g.

Code: Select all

mount HugeSoftware.xzm /G/HugeSoftware
instead of

Code: Select all

activate HugeSoftware.xzm
?
because many programs need to think that they could write to their settings file.

If you have the chance, can you please try your

Code: Select all

mount HugeSoftware.xzm /G/HugeSoftware
approach with either audacity-linux-3.2.0-x86_64.xzm or audacity-linux-3.2.1-x86_64.xzm and report how well that went?
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#29 by rych » 15 Nov 2022, 11:55

Rava wrote:
14 Nov 2022, 23:03
You mean
No, mount xzm vs. fully expanded, decompressed into a folder. Although mount vs. activate is also an interesting question for programs that don't need write access to its installation folder -- essentially we avoid aufs layer by simply mounting

gnintilgyes
Black ninja
Black ninja
Posts: 73
Joined: 14 Sep 2022, 17:52
Distribution: Debian

why modules

Post#30 by gnintilgyes » 16 Nov 2022, 05:23

I'm going to contradict myself soon.

I've been trying to get up and running with Nemesis, for somebody who doesn't really like rolling-release because he/she is struggling currently with a slow Internet connection.
rych wrote:
14 Nov 2022, 09:13
Question: why shouldn't all software run from squashfs (xzm) modules?
Because it could greatly increase the loading times, either during starting before desktop, or on demand which would peak off enough users.

After my first message box which tells me, "Well done!" I have to wait like 10 seconds or longer before the "Porteus Modules" dialog updates from red button to green button. It doesn't matter what is the size of the module, of course placed in "optional" folder.

Post Reply