why modules

[rych]

What are RAM/CPU advantages of turning all packages into modules, why can't I just install (large) programs as normal in Linux, have them automatically mapped to the /changes folder on an ext4 partition on my porteus USB, and let them simply live there without building the modules?

How is the /changes ext4 folder merged into the live system? Is there CPU/RAM overhead involved (as compared to modules)?

(My /changes folder is currently 15GB)

[babam]

What are RAM/CPU advantages of turning all packages into modules

There is no advantage for RAM and CPU, in fact a slight disadvantage.

[Ed_P]

Modules support brokenman's goal of a smaller system with ease for users to add, upgrade and remove apps.

BTW My changes file is 512MB and I boot ISOs.

[beny]

rich if i remember well you can install into porteus the slackware packages without change into xzm but you have the software only in changes directory,the speed of the aufs filesystem is the read only you have a safe way to run your system,if something went wrong in your changes directory well good luck and backup if you have any.

[fulalas]

@rych, either using rootcopy or save.dat these files will be merged into filesystem (RAM) during the boot process -- after all xzm modules are loaded. There's a performance impact, sure, and it will depend on the amount and size of files and your CPU/RAM and storage speeds.

As beny pointed, xzm modules are optimized to read mode, which means creating a module can be really slow but decompressing it is really fast, therefore desirable when compared to rootcopy or even save.dat if you worry about maximum performance. If you take a look at how live sessions work on Linux, you'll see that they all (Debian, Arch, Fedora, etc) use the same idea: compressed modules that are partially extracted to RAM during boot time.

You can install programs the old fashion way, including Slackware official way (installpkg package.txz) or simply manually extracting files to the filesystem. The only caveat I would say is how to keep this after rebooting, which leads us back to rootcopy/save.dat or xzm module again -- remember: Porteus is a live distro, so everything you change is in RAM, which is volatile.

Last but not least, turning all packages into modules might be appealing from the perspective of being more easily updatable, yes. But there are some drawbacks. For instance: the boot process would be slower because decompressing one big xzm module is much faster than decompressing a bunch of small ones. Also, for us developers it would be a nightmare managing all this. Finally, in many cases new packages are incompatible with other packages, and figuring out what to do in this case would be anything but trivial. But if you have endless patience you can try yourself

[rych]

Thank you everyone for your explanation. I've decided that packages which are awkward to make into modules should be whenever possible installed portable and frugal on the porteus USB, for example, into a folder portable. This way I also avoid the (small) overheads of Aufs (no contaminating of the changes folder merged into a live tree) and Squashfs (no modules). The next problem is to mount the portable folder under an invariant name. I accomplish this by using bind mount as in Porteus USB always mounted as /mnt/porteus

[raja]

fulalas... You may consider producing a fully Hard disk installable version of Porteus by clubbing all modules to a single squashfs file which can be installed into a hdd with all filesystem in place along with boot folder as "future" project. Just like any other mainstream Linux offering.

A new initrd file has to be developed to boot into an user Id. This would be the extra work apart from an installer.

Current module based, compact portable Porteus OS can exist for ever.

Users will have a choice to choose. fully installable or portable

[Rava]

When one wants no modules, why not install just Slackware current instead as regular hard disk based installation?

Like fulalas wrote

If you take a look at how live sessions work on Linux, you'll see that they all (Debian, Arch, Fedora, etc) use the same idea: compressed modules that are partially extracted to RAM during boot time.

and we use a heavily altered Linux live script to optimise speed and RAM usage when it comes to the modules…

And the live systems that aim at older hardware - puppy and DSL - use very similar approaches.

[rych]

You may consider producing a fully Hard disk installable version of Porteus

why not install just Slackware current instead as regular hard disk based installation?

I only have my USB which I carry and boot into different computers. Porteus is the best for this, I think? The combination of modules and frugal-USB-portable installation of frequently updated (e.g. TeXLive) and/or huge software environments (Conda SageMath) is perfect.

I was just asking about the CPU/RAM performance comparison of frugal (always expanded) installation vs module. For example, why would one want to make a 10GB SageMath (and cleverly remake with each frequent update) installation into a module? I chose not to, partly because:

There is no advantage for RAM and CPU, in fact a slight disadvantage.

[babam]

Because decompressing modules (xzm) requires CPU and RAM, and the decompression speed depends on the compressor that has been used (xz, zstd, gzip).

[wread]

I have no changes folder at all...

[Kulle]

Hi all,

fulalas wrote: ↑ decompressing one big xzm module is much faster than decompressing a bunch of small ones.

Would it be worthwhile (saving time when booting) to combine the modules in the folder base (kernel, core, ...) into one ?
Because these base modules are only updated occasionally (very rarely).

[Rava]

Would it be worthwhile (saving time when booting) to combine the modules in the folder base (kernel, core, ...) into one ?
Because these base modules are only updated occasionally (very rarely).

If you have your system as modules (instead of as ISO) you can try that for yourself.
Create one large module out of your current system, then start both, and then look into the time stamps of /var/log/messages, probably that will tell you if that made any worthwhile speed-up.

You have to figure out which moment to use to realize the system is up, and that it's the same time for both attempts.
If you use local $PORTDIR/rootcopy you can add some logger into rc.local that would be executed after all modules are loaded, and when both attempts use the same rc.local that logger would be written into /var/log/messages at the same moment of your startup.
Then just calculate the time from the first recorded time stamp to the one from that rc.local logger.

[rych]

OK, modules are faster than portable frugal installs and deployments onto the USB stick -- and I've bought the fastest USB. The stuff on USB is all better be compressed modules, packages etc. expandable onto RAM I guess at run time. I'm convinced.

My next question to you (fulalas, babam, and others) is whether just before activation of xzm modules (or running AppImage etc.) it's worth copying them first to my host system (NVMe) SSD disk which is so much faster than my USB disk? Activating modules from a folder on the host system disk, even if its NTFS, should be faster right? As compared to accessing them from the USB at run time?

[Rava]

My next question to you (fulalas, babam, and others) is whether just before activation of xzm modules (or running AppImage etc.) it's worth copying them first to my host system (NVMe) SSD disk which is so much faster than my USB disk? Activating modules from a folder on the host system disk, even if its NTFS, should be faster right? As compared to accessing them from the USB at run time?

You have to consider that you have to copy it first, and that also takes time.
If you disregard that and you have the free space on your fastest host SSD disk then have all modules you ever use on there. Since they are compressed they are not that large anyway.
But activating a module in itself always only takes a few seconds thatmost.
What takes time is when you actually execute the program that is in the module. Only then the real data that the module is will the accessed. So, testing that would only work if you time the time a more complex program takes to fully load, e.g. GIMP, or most full browsers, or such.

I think it comes down to convenience. Where I in your shoes, I would have the most used modules on the host SSD - and all the other rarely used ones, I would keep on some USB-3 or eSATA external harddrive, and even when these then take a few more seconds to fully load, I would not bother since that would be a program I only start once every blue moon.
Of course with that approach it can happen that I realize over time that one of the previous thought of "rarely used" modules - I indeed use more often that previously realized, then I would copy that one module to the host SSD as well.