why modules

Technical issues/questions of an intermediate or advanced nature.
rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#1 by rych » 09 Feb 2022, 09:48

What are RAM/CPU advantages of turning all packages into modules, why can't I just install (large) programs as normal in Linux, have them automatically mapped to the /changes folder on an ext4 partition on my porteus USB, and let them simply live there without building the modules?

How is the /changes ext4 folder merged into the live system? Is there CPU/RAM overhead involved (as compared to modules)?

(My /changes folder is currently 15GB)

User avatar
babam
Warlord
Warlord
Posts: 526
Joined: 16 Nov 2016, 10:30
Distribution: Porteus 5.0rc3 Xfce K6.1.1
Location: Rainy city

why modules

Post#2 by babam » 09 Feb 2022, 13:16

rych wrote:
09 Feb 2022, 09:48
What are RAM/CPU advantages of turning all packages into modules
There is no advantage for RAM and CPU, in fact a slight disadvantage.
Sorry, my English is bad.

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

why modules

Post#3 by Ed_P » 09 Feb 2022, 16:57

Modules support brokenman's goal of a smaller system with ease for users to add, upgrade and remove apps.

BTW My changes file is 512MB and I boot ISOs.
Ed

beny
Full of knowledge
Full of knowledge
Posts: 2086
Joined: 02 Jan 2011, 11:33
Location: italy

why modules

Post#4 by beny » 09 Feb 2022, 17:33

rich if i remember well you can install into porteus the slackware packages without change into xzm but you have the software only in changes directory,the speed of the aufs filesystem is the read only you have a safe way to run your system,if something went wrong in your changes directory well good luck and backup if you have any.

fulalas
DEV Team
DEV Team
Posts: 2050
Joined: 26 Oct 2016, 15:34
Distribution: Porteus
Location: Brazil

why modules

Post#5 by fulalas » 11 Feb 2022, 01:02

@rych, either using rootcopy or save.dat these files will be merged into filesystem (RAM) during the boot process -- after all xzm modules are loaded. There's a performance impact, sure, and it will depend on the amount and size of files and your CPU/RAM and storage speeds.

As beny pointed, xzm modules are optimized to read mode, which means creating a module can be really slow but decompressing it is really fast, therefore desirable when compared to rootcopy or even save.dat if you worry about maximum performance. If you take a look at how live sessions work on Linux, you'll see that they all (Debian, Arch, Fedora, etc) use the same idea: compressed modules that are partially extracted to RAM during boot time.

You can install programs the old fashion way, including Slackware official way (installpkg package.txz) or simply manually extracting files to the filesystem. The only caveat I would say is how to keep this after rebooting, which leads us back to rootcopy/save.dat or xzm module again -- remember: Porteus is a live distro, so everything you change is in RAM, which is volatile.

Last but not least, turning all packages into modules might be appealing from the perspective of being more easily updatable, yes. But there are some drawbacks. For instance: the boot process would be slower because decompressing one big xzm module is much faster than decompressing a bunch of small ones. Also, for us developers it would be a nightmare managing all this. Finally, in many cases new packages are incompatible with other packages, and figuring out what to do in this case would be anything but trivial. But if you have endless patience you can try yourself :D

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#6 by rych » 15 Feb 2022, 06:36

Thank you everyone for your explanation. I've decided that packages which are awkward to make into modules should be whenever possible installed portable and frugal on the porteus USB, for example, into a folder portable. This way I also avoid the (small) overheads of Aufs (no contaminating of the changes folder merged into a live tree) and Squashfs (no modules). The next problem is to mount the portable folder under an invariant name. I accomplish this by using bind mount as in Porteus USB always mounted as /mnt/porteus

raja
Shogun
Shogun
Posts: 434
Joined: 02 May 2017, 09:51
Distribution: v3.2.2-32 and Porteus-Artix-64
Location: Chennai,India

why modules

Post#7 by raja » 15 Feb 2022, 09:56

fulalas... You may consider producing a fully Hard disk installable version of Porteus by clubbing all modules to a single squashfs file which can be installed into a hdd with all filesystem in place along with boot folder as "future" project. Just like any other mainstream Linux offering.

A new initrd file has to be developed to boot into an user Id. This would be the extra work apart from an installer.

Current module based, compact portable Porteus OS can exist for ever.

Users will have a choice to choose. fully installable or portable
Linux Kernel-4.4.272 -32 bit; Linux Kernel-5.4.185 - 64 bit

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#8 by Rava » 16 Feb 2022, 03:18

When one wants no modules, why not install just Slackware current instead as regular hard disk based installation?

Like fulalas wrote
fulalas wrote:
11 Feb 2022, 01:02
If you take a look at how live sessions work on Linux, you'll see that they all (Debian, Arch, Fedora, etc) use the same idea: compressed modules that are partially extracted to RAM during boot time.
and we use a heavily altered Linux live script to optimise speed and RAM usage when it comes to the modules…

And the live systems that aim at older hardware - puppy and DSL - use very similar approaches.
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#9 by rych » 16 Feb 2022, 09:05

raja wrote:
15 Feb 2022, 09:56
You may consider producing a fully Hard disk installable version of Porteus
Rava wrote:
16 Feb 2022, 03:18
why not install just Slackware current instead as regular hard disk based installation?

I only have my USB which I carry and boot into different computers. Porteus is the best for this, I think? The combination of modules and frugal-USB-portable installation of frequently updated (e.g. TeXLive) and/or huge software environments (Conda SageMath) is perfect.

I was just asking about the CPU/RAM performance comparison of frugal (always expanded) installation vs module. For example, why would one want to make a 10GB SageMath (and cleverly remake with each frequent update) installation into a module? I chose not to, partly because:
babam wrote:
09 Feb 2022, 13:16
There is no advantage for RAM and CPU, in fact a slight disadvantage.

User avatar
babam
Warlord
Warlord
Posts: 526
Joined: 16 Nov 2016, 10:30
Distribution: Porteus 5.0rc3 Xfce K6.1.1
Location: Rainy city

why modules

Post#10 by babam » 16 Feb 2022, 16:09

Because decompressing modules (xzm) requires CPU and RAM, and the decompression speed depends on the compressor that has been used (xz, zstd, gzip).
Sorry, my English is bad.

User avatar
wread
Module Guard
Module Guard
Posts: 1255
Joined: 09 Jan 2011, 18:48
Distribution: Porteus v5.0-kde-64 bits
Location: Santo Domingo
Contact:

why modules

Post#11 by wread » 26 Jul 2022, 16:09

I have no changes folder at all...
Porteus is proud of the FASTEST KDE ever made.....(take akonadi, nepomuk and soprano out and you will have a decent OS).
The Porteus Community never sleeps!

Kulle
Warlord
Warlord
Posts: 594
Joined: 28 Jan 2017, 10:39
Distribution: v4.0 64bit Xfce
Location: Berlin

why modules

Post#12 by Kulle » 27 Jul 2022, 17:01

Hi all,
fulalas wrote: ↑ decompressing one big xzm module is much faster than decompressing a bunch of small ones.
Would it be worthwhile (saving time when booting) to combine the modules in the folder base (kernel, core, ...) into one ?
Because these base modules are only updated occasionally (very rarely).

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#13 by Rava » 18 Aug 2022, 23:12

Kulle wrote:
27 Jul 2022, 17:01
Would it be worthwhile (saving time when booting) to combine the modules in the folder base (kernel, core, ...) into one ?
Because these base modules are only updated occasionally (very rarely).
If you have your system as modules (instead of as ISO) you can try that for yourself.
Create one large module out of your current system, then start both, and then look into the time stamps of /var/log/messages, probably that will tell you if that made any worthwhile speed-up.

You have to figure out which moment to use to realize the system is up, and that it's the same time for both attempts.
If you use local $PORTDIR/rootcopy you can add some logger into rc.local that would be executed after all modules are loaded, and when both attempts use the same rc.local that logger would be written into /var/log/messages at the same moment of your startup.
Then just calculate the time from the first recorded time stamp to the one from that rc.local logger.
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#14 by rych » 21 Sep 2022, 11:23

OK, modules are faster than portable frugal installs and deployments onto the USB stick -- and I've bought the fastest USB. The stuff on USB is all better be compressed modules, packages etc. expandable onto RAM I guess at run time. I'm convinced.

My next question to you (fulalas, babam, and others) is whether just before activation of xzm modules (or running AppImage etc.) it's worth copying them first to my host system (NVMe) SSD disk which is so much faster than my USB disk? Activating modules from a folder on the host system disk, even if its NTFS, should be faster right? As compared to accessing them from the USB at run time?

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#15 by Rava » 21 Sep 2022, 14:00

rych wrote:
21 Sep 2022, 11:23
My next question to you (fulalas, babam, and others) is whether just before activation of xzm modules (or running AppImage etc.) it's worth copying them first to my host system (NVMe) SSD disk which is so much faster than my USB disk? Activating modules from a folder on the host system disk, even if its NTFS, should be faster right? As compared to accessing them from the USB at run time?
You have to consider that you have to copy it first, and that also takes time.
If you disregard that and you have the free space on your fastest host SSD disk then have all modules you ever use on there. Since they are compressed they are not that large anyway.
But activating a module in itself always only takes a few seconds thatmost.
What takes time is when you actually execute the program that is in the module. Only then the real data that the module is will the accessed. So, testing that would only work if you time the time a more complex program takes to fully load, e.g. GIMP, or most full browsers, or such.

I think it comes down to convenience. Where I in your shoes, I would have the most used modules on the host SSD - and all the other rarely used ones, I would keep on some USB-3 or eSATA external harddrive, and even when these then take a few more seconds to fully load, I would not bother since that would be a program I only start once every blue moon.
Of course with that approach it can happen that I realize over time that one of the previous thought of "rarely used" modules - I indeed use more often that previously realized, then I would copy that one module to the host SSD as well. Image
Cheers!
Yours Rava

Post Reply