why modules
-
- Warlord
- Posts: 622
- Joined: 04 Jan 2014, 04:27
- Distribution: Porteus 5.0 x64 OpenBox
- Location: NZ
- Contact:
why modules
What are RAM/CPU advantages of turning all packages into modules, why can't I just install (large) programs as normal in Linux, have them automatically mapped to the /changes folder on an ext4 partition on my porteus USB, and let them simply live there without building the modules?
How is the /changes ext4 folder merged into the live system? Is there CPU/RAM overhead involved (as compared to modules)?
(My /changes folder is currently 15GB)
How is the /changes ext4 folder merged into the live system? Is there CPU/RAM overhead involved (as compared to modules)?
(My /changes folder is currently 15GB)
- babam
- Warlord
- Posts: 528
- Joined: 16 Nov 2016, 10:30
- Distribution: Porteus 5.0rc3 Xfce K6.1.1
- Location: Rainy city
why modules
There is no advantage for RAM and CPU, in fact a slight disadvantage.
Sorry, my English is bad.
- Ed_P
- Contributor
- Posts: 8369
- Joined: 06 Feb 2013, 22:12
- Distribution: Cinnamon 5.01 ISO
- Location: Western NY, USA
why modules
Modules support brokenman's goal of a smaller system with ease for users to add, upgrade and remove apps.
BTW My changes file is 512MB and I boot ISOs.
BTW My changes file is 512MB and I boot ISOs.
Ed
why modules
rich if i remember well you can install into porteus the slackware packages without change into xzm but you have the software only in changes directory,the speed of the aufs filesystem is the read only you have a safe way to run your system,if something went wrong in your changes directory well good luck and backup if you have any.
why modules
@rych, either using rootcopy or save.dat these files will be merged into filesystem (RAM) during the boot process -- after all xzm modules are loaded. There's a performance impact, sure, and it will depend on the amount and size of files and your CPU/RAM and storage speeds.
As beny pointed, xzm modules are optimized to read mode, which means creating a module can be really slow but decompressing it is really fast, therefore desirable when compared to rootcopy or even save.dat if you worry about maximum performance. If you take a look at how live sessions work on Linux, you'll see that they all (Debian, Arch, Fedora, etc) use the same idea: compressed modules that are partially extracted to RAM during boot time.
You can install programs the old fashion way, including Slackware official way (installpkg package.txz) or simply manually extracting files to the filesystem. The only caveat I would say is how to keep this after rebooting, which leads us back to rootcopy/save.dat or xzm module again -- remember: Porteus is a live distro, so everything you change is in RAM, which is volatile.
Last but not least, turning all packages into modules might be appealing from the perspective of being more easily updatable, yes. But there are some drawbacks. For instance: the boot process would be slower because decompressing one big xzm module is much faster than decompressing a bunch of small ones. Also, for us developers it would be a nightmare managing all this. Finally, in many cases new packages are incompatible with other packages, and figuring out what to do in this case would be anything but trivial. But if you have endless patience you can try yourself
As beny pointed, xzm modules are optimized to read mode, which means creating a module can be really slow but decompressing it is really fast, therefore desirable when compared to rootcopy or even save.dat if you worry about maximum performance. If you take a look at how live sessions work on Linux, you'll see that they all (Debian, Arch, Fedora, etc) use the same idea: compressed modules that are partially extracted to RAM during boot time.
You can install programs the old fashion way, including Slackware official way (installpkg package.txz) or simply manually extracting files to the filesystem. The only caveat I would say is how to keep this after rebooting, which leads us back to rootcopy/save.dat or xzm module again -- remember: Porteus is a live distro, so everything you change is in RAM, which is volatile.
Last but not least, turning all packages into modules might be appealing from the perspective of being more easily updatable, yes. But there are some drawbacks. For instance: the boot process would be slower because decompressing one big xzm module is much faster than decompressing a bunch of small ones. Also, for us developers it would be a nightmare managing all this. Finally, in many cases new packages are incompatible with other packages, and figuring out what to do in this case would be anything but trivial. But if you have endless patience you can try yourself
-
- Warlord
- Posts: 622
- Joined: 04 Jan 2014, 04:27
- Distribution: Porteus 5.0 x64 OpenBox
- Location: NZ
- Contact:
why modules
Thank you everyone for your explanation. I've decided that packages which are awkward to make into modules should be whenever possible installed portable and frugal on the porteus USB, for example, into a folder portable. This way I also avoid the (small) overheads of Aufs (no contaminating of the changes folder merged into a live tree) and Squashfs (no modules). The next problem is to mount the portable folder under an invariant name. I accomplish this by using bind mount as in Porteus USB always mounted as /mnt/porteus
-
- Shogun
- Posts: 434
- Joined: 02 May 2017, 09:51
- Distribution: v3.2.2-32 and Porteus-Artix-64
- Location: Chennai,India
why modules
fulalas... You may consider producing a fully Hard disk installable version of Porteus by clubbing all modules to a single squashfs file which can be installed into a hdd with all filesystem in place along with boot folder as "future" project. Just like any other mainstream Linux offering.
A new initrd file has to be developed to boot into an user Id. This would be the extra work apart from an installer.
Current module based, compact portable Porteus OS can exist for ever.
Users will have a choice to choose. fully installable or portable
A new initrd file has to be developed to boot into an user Id. This would be the extra work apart from an installer.
Current module based, compact portable Porteus OS can exist for ever.
Users will have a choice to choose. fully installable or portable
Linux Kernel-4.4.272 -32 bit; Linux Kernel-5.4.185 - 64 bit
- Rava
- Contributor
- Posts: 5401
- Joined: 11 Jan 2011, 02:46
- Distribution: XFCE 5.01 x86_64 + 4.0 i586
- Location: Forests of Germany
why modules
When one wants no modules, why not install just Slackware current instead as regular hard disk based installation?
Like fulalas wrote
And the live systems that aim at older hardware - puppy and DSL - use very similar approaches.
Like fulalas wrote
and we use a heavily altered Linux live script to optimise speed and RAM usage when it comes to the modules…
And the live systems that aim at older hardware - puppy and DSL - use very similar approaches.
Cheers!
Yours Rava
Yours Rava
-
- Warlord
- Posts: 622
- Joined: 04 Jan 2014, 04:27
- Distribution: Porteus 5.0 x64 OpenBox
- Location: NZ
- Contact:
why modules
I only have my USB which I carry and boot into different computers. Porteus is the best for this, I think? The combination of modules and frugal-USB-portable installation of frequently updated (e.g. TeXLive) and/or huge software environments (Conda SageMath) is perfect.
I was just asking about the CPU/RAM performance comparison of frugal (always expanded) installation vs module. For example, why would one want to make a 10GB SageMath (and cleverly remake with each frequent update) installation into a module? I chose not to, partly because:
- babam
- Warlord
- Posts: 528
- Joined: 16 Nov 2016, 10:30
- Distribution: Porteus 5.0rc3 Xfce K6.1.1
- Location: Rainy city
why modules
Because decompressing modules (xzm) requires CPU and RAM, and the decompression speed depends on the compressor that has been used (xz, zstd, gzip).
Sorry, my English is bad.
- wread
- Module Guard
- Posts: 1255
- Joined: 09 Jan 2011, 18:48
- Distribution: Porteus v5.0-kde-64 bits
- Location: Santo Domingo
- Contact:
why modules
I have no changes folder at all...
Porteus is proud of the FASTEST KDE ever made.....(take akonadi, nepomuk and soprano out and you will have a decent OS).
The Porteus Community never sleeps!
The Porteus Community never sleeps!
why modules
Hi all,
Because these base modules are only updated occasionally (very rarely).
Would it be worthwhile (saving time when booting) to combine the modules in the folder base (kernel, core, ...) into one ?fulalas wrote: ↑ decompressing one big xzm module is much faster than decompressing a bunch of small ones.
Because these base modules are only updated occasionally (very rarely).
- Rava
- Contributor
- Posts: 5401
- Joined: 11 Jan 2011, 02:46
- Distribution: XFCE 5.01 x86_64 + 4.0 i586
- Location: Forests of Germany
why modules
If you have your system as modules (instead of as ISO) you can try that for yourself.
Create one large module out of your current system, then start both, and then look into the time stamps of /var/log/messages, probably that will tell you if that made any worthwhile speed-up.
You have to figure out which moment to use to realize the system is up, and that it's the same time for both attempts.
If you use local $PORTDIR/rootcopy you can add some logger into rc.local that would be executed after all modules are loaded, and when both attempts use the same rc.local that logger would be written into /var/log/messages at the same moment of your startup.
Then just calculate the time from the first recorded time stamp to the one from that rc.local logger.
Cheers!
Yours Rava
Yours Rava
-
- Warlord
- Posts: 622
- Joined: 04 Jan 2014, 04:27
- Distribution: Porteus 5.0 x64 OpenBox
- Location: NZ
- Contact:
why modules
OK, modules are faster than portable frugal installs and deployments onto the USB stick -- and I've bought the fastest USB. The stuff on USB is all better be compressed modules, packages etc. expandable onto RAM I guess at run time. I'm convinced.
My next question to you (fulalas, babam, and others) is whether just before activation of xzm modules (or running AppImage etc.) it's worth copying them first to my host system (NVMe) SSD disk which is so much faster than my USB disk? Activating modules from a folder on the host system disk, even if its NTFS, should be faster right? As compared to accessing them from the USB at run time?
My next question to you (fulalas, babam, and others) is whether just before activation of xzm modules (or running AppImage etc.) it's worth copying them first to my host system (NVMe) SSD disk which is so much faster than my USB disk? Activating modules from a folder on the host system disk, even if its NTFS, should be faster right? As compared to accessing them from the USB at run time?
- Rava
- Contributor
- Posts: 5401
- Joined: 11 Jan 2011, 02:46
- Distribution: XFCE 5.01 x86_64 + 4.0 i586
- Location: Forests of Germany
why modules
You have to consider that you have to copy it first, and that also takes time.rych wrote: ↑21 Sep 2022, 11:23My next question to you (fulalas, babam, and others) is whether just before activation of xzm modules (or running AppImage etc.) it's worth copying them first to my host system (NVMe) SSD disk which is so much faster than my USB disk? Activating modules from a folder on the host system disk, even if its NTFS, should be faster right? As compared to accessing them from the USB at run time?
If you disregard that and you have the free space on your fastest host SSD disk then have all modules you ever use on there. Since they are compressed they are not that large anyway.
But activating a module in itself always only takes a few seconds thatmost.
What takes time is when you actually execute the program that is in the module. Only then the real data that the module is will the accessed. So, testing that would only work if you time the time a more complex program takes to fully load, e.g. GIMP, or most full browsers, or such.
I think it comes down to convenience. Where I in your shoes, I would have the most used modules on the host SSD - and all the other rarely used ones, I would keep on some USB-3 or eSATA external harddrive, and even when these then take a few more seconds to fully load, I would not bother since that would be a program I only start once every blue moon.
Of course with that approach it can happen that I realize over time that one of the previous thought of "rarely used" modules - I indeed use more often that previously realized, then I would copy that one module to the host SSD as well.
Cheers!
Yours Rava
Yours Rava