Page 1 of 1

[Solved] How do I generate modules after modifying system?

Posted: 09 Jan 2013, 02:41
by claude
Coming from slax 6.1.2, I used to do this the easy way...meaning, I would first uncompress the various
base slax mo's to an ext2 filesystem on hard drive, then boot that system on that partition and modify it
at will, and then just recompress the filesystem thus modified with a simple 'dir2mo' and it would result in a single big module that worked just fine.
But I noticed that if I do this same procedure in porteus v1.2, the resulting large single xzm file becomes
much slower to boot when compared to the original porteus smaller modules, and there seem to be
permission issues for user guest when booting into xwindows.....

So my question is this: how do I do this properly, that is, starting with an uncompressed, modified
porteus filesystem on a hard drive ext4 partition, how do I get to make several base modules like in the
original? Are there any scripts (similar to the live-linux ones from Tomas M?) or any other procedure
that I can use to do this?

Re: How do I generate modules after modifying the system?

Posted: 09 Jan 2013, 16:13
by brokenman
In porteus v1.2, the resulting large single xzm file becomes much slower to boot when compared to the original porteus smaller modules.
Have you tried the latest V2.0rc1 Porteus? Inserting the large module into aufs could be faster in this version although i would expect a slight slow down when inserting an overly large module.

how do I get to make several base modules like in the original?
By unpacking everything to a file system you have just killed all modularity, and the idea of Porteus live with it. Creating the base modules again from this scenario, while not impossible, is certainly not going to be a simple task. If your end goal is to unpack, modify and repack might i suggest using a changes module instead? Just run porteus and setup everything how you want, then save the session as a module. In v2.0 this is done through the Porteus Settings Centre.

When creating porteus we keep all packages that belong in 001 in a folder called '001'. There is also a 'changes' tree for each module (files that are unique or have been altered). Then all packages are installed into a tmp folder, the changes added and finally a module is created from the tmp folder - 001-core.xzm. This method is clean, fast and probably the easiest way.

Re: How do I generate modules after modifying the system?

Posted: 09 Jan 2013, 17:58
by claude
I use Porteus 1.2 modularized, as is, on my latest machine, and it works flawlessly :) - thank you for doing such a great job!
But I have also made two base modules for two smaller PCs I own, older ones, and this with the purpose of keeping a lower overhead on resources on the older equipment...one module is 160 MB (700 MB filesystem) while the other is 105 MB (450 MB filesystem). After chowning the guest directory with the proper ownership, the smaller porteuses now work flawlessly too, they just take a long time to load at boot up. I think this will slow down the boot process in v2.0 too as far as older computers are concerned.
I understand that as developers the system you are using to develop Porteus is very clean and foolproof. However, for an end user such as myself, it would be beyond my scope to employ that type of approach, it would instead be much easier and practical to have a way to uncompress the system on a HD partition, use pkgtool to remove all unneeded programs, prune the system as needed, and then build the base modules back again and, of course, use standard Porteus modularization for everything else. But, as we have seen, if I build the basic filesystem back into a single xzm file with dir2xzm, the resulting large xzm module is very slow to boot and heavy on the limited resources of an older machine.

Therefore my question is, why don't you maintain an updated, porteus-specific version of the linux-live scripts, which could very easily achieve what an user like me needs very simply and quickly? I think many Porteurs users would love such a thing. It makes the task of uncompressing the filesystem, modifying it and rebuilding the base modules a breeze. So, seriously, why not update and make the linux-live scripts or something similar available to Porteus users?
I am sure it would be something that Porteus developers would also very much like and use.

Re: How do I generate modules after modifying the system?

Posted: 09 Jan 2013, 20:07
by fanthom
it would instead be much easier and practical to have a way to uncompress the system on a HD partition, use pkgtool to remove all unneeded programs, prune the system as needed, and then build the base modules back again
all this is still possible in stock Porteus. please use:
a) xzm2dir or unsquashfs tools - to unpack the module
b) ROOT=/path/to_uppacked_module removepkg some_package - to remove unnecessary stuff
c) dir2xzm to pack stuff back to xzm
as we have seen, if I build the basic filesystem back into a single xzm file with dir2xzm, the resulting large xzm module is very slow to boot and heavy on the limited resource of an older machine.
Porteus modules from /base folder contain updated caches (gtk, pango, mime, etc..) so updating of these can be skipped resulting in a faster boot.
please have a look on /opt/porteus-scripts/xorg/update-cache script which is run when other than /base modules are discovered by booting scripts.
if you are able to add all caches to your custom modules then you could place them in /base folder and name as 009x- so it would be treated as native porteus module and not external one.
this action should make booting nearly as fast as it's with stock porteus modules only.
why don't you maintain an updated, porteus-specific version of the linux-live scripts, which could very easily achieve what an user like me needs very simply and quickly?
i see no point for this for several reasons:
a) Tomas M is maintaining one on http://www.linux-live.org/
b) there is similar slackware-live project on http://slackware-live.tuxfamily.org/
c) it's easy enough to use Porteus ISO as a template for your own project as per this instruction:
link
I am sure it would be something that Porteus developers would also very much like and use.
@brokenman/Ahau
do you think that our process could be improved by the live-kit?

Re: How do I generate modules after modifying the system?

Posted: 09 Jan 2013, 21:27
by Ahau
I don't think my process would be improved, because I build Xfce ISO's directly over the Standard edition modules rather than creating them from scratch. I do make some tweaks, which I apply by extracting the xzm to a folder (xzm2dir), running a script to make the tweak or performing it manually (a script is better so I don't forget to do it the next time!) then recompress from the folder back to a module (dir2xzm).

If I understand claude's post correctly, I think where our current process could be improved is in the removal of packages from base modules to lighten the system. It's easy to add packages (installpkg -root /path/to/extracted/module pkg.txz) and rebuild, but it's more difficult to remove them -- if I recall correctly, removekpkg doesn't support the -root switch. This might be accomplished with chroot or some similar method (I haven't tested to sort it out), or by the use of brokenman's 'removepkg' and 'extractpkg' scripts which were around for 1.1 but I think we dropped them or they got wrapped into ppm. Maybe we could update those scripts, make p_cache suitable for publishing so users can update 002-xorg.xzm with their modifications, post it all online and publish a HOWTO.

I've previously written a script to build a module for Xfce from start to finish. It downloaded all required txz's plus slackbuilds and source files, tweaked all of the slackbuilds, compiled everything and output a module. After using it for a while, however, I realized that maintaining that script (600+ lines to sort through and debug every time I made a minor change to one package) was more work than compiling each package through a slackbuild individually.

claude, if we had a HOWTO and scripts to remove packages from individual base modules and then update the cache files so the system would boot faster, would that meet your customization needs?

Re: How do I generate modules after modifying the system?

Posted: 10 Jan 2013, 00:18
by Ahau
fanthom pointed something out to me (thanks, fanthom!) -- you can remove packages that are installed to a fake root directory with:

Code: Select all

ROOT=/path/to/fakeroot removepkg packagename
So, you could strip packages from a module like so:

Code: Select all

cp /mnt/sdb2/porteus/base/0090-xfce.xzm /tmp
cd /tmp
mkdir xfce
xzm2dir 0090-xfce.xzm xfce
ROOT=/tmp/xfce removepkg exo thunar thunar-volman thunar-archive-plugin
dir2xzm xfce 0090-xfce.xzm
now you can use 'mloop' to mount the module and check out the contents, then copy it back into place in the /porteus/base folder, and those packages will be gone from your install.

Re: How do I generate modules after modifying the system?

Posted: 10 Jan 2013, 00:29
by brokenman
I don't see the point in decompressing everything in order to edit files. When testing i usually boot to my desktop (freshmode) and then make any file edits i need. The edited files are placed in rootcopy and when i am happy they work as intended i merge them back into the main modules. A little like modifying a car (which is modular). We don't dismantle the entire car just to tweak the injection system. You'll learn more for sure, but it's just not economical.

As fanthom suggested, if you are looking for a single script to create a live version after your edits, that's what the linux live scripts do. No point re-inventing that wheel.

If you're looking for something to remove packages where removepkg doesn't work for you then something like this may work:

Code: Select all

grep -A`wc -l </var/log/packages/my-package-1.0-i486-1` LIST: /var/log/packages/my-package-1.0-i486-1|xargs rm -rf
EDIT: Ahau's post above is a better solution. If i new of it before creating removepkg script it would have saved me some time.

Re: How do I generate modules after modifying the system?

Posted: 10 Jan 2013, 01:26
by claude
@fanthom
Thank you so much for your comments and suggestions. I will have to try them out and, if workable, it looks like you have addressed all my questions very thoroughly. Very honestly, it is a blessing to be able to talk to someone with your knowledge and amazing eye for solutions....no surprise Porteus is as good as it is.

@Ahau
Yes, that was exactly my post's perspective, to be able to generate a much lighter base system for my older hardware. But I do not think in the least that removing packages from base modules in order to lighten the basic system would make Porteus better. I tried out Porteus v1.2 KDE only very recently (had used slax 6.1.2 up to then...I know, it was quite a bit outdated) and I can tell you Porteus is a beauty, very stable and truly excellent. I was very, very much impressed by it and wouldn't want it to be anything less when it comes to relatively modern hardware. It would be a mistake to make the base modules lighter with fewer packages. Don't forget that most users downloading versions of Porteus will want something basically complete and ready to go from the very start. Modularization is a perfectly fine concept, but the first, lasting impression will be made when they see basic Porteus boot up the first time just as it was downloaded.
So, I would not advise to make the basic system lighter, but rather to provide clear instructions (HOWTO), and perhaps scripts for those who don't like to get their hands "dirty", that will make it practical for newbie users intending to produce a stripped down version of Porteus for their older hardware to remove packages from and prune the basic modules and then easily rebuild them as efficient as they were originally.
Personally speaking, yes, that would definitely meet my current customization needs. But I'll try and follow fanthom's advice and I might have my answers there already.

EDIT: thanks Ahau for your second post in which you give the specific code...very, very helpful.

@brokenman
Thanks for your clarifying comments.

Thanks guys, it is all very much appreciated.

Re: How do I generate modules after modifying the system?

Posted: 10 Jan 2013, 15:22
by claude
fanthom wrote:
Porteus modules from /base folder contain updated caches (gtk, pango, mime, etc..) so updating of these can be skipped resulting in a faster boot.
please have a look on /opt/porteus-scripts/xorg/update-cache script which is run when other than /base modules are discovered by booting scripts.
if you are able to add all caches to your custom modules then you could place them in /base folder and name as 009x- so it would be treated as native porteus module and not external one.
this action should make booting nearly as fast as it's with stock porteus modules only.
@fanthom
I've looked at that script as you suggested, and by making use of this single piece of information you provided, I have been able to make my single, large xzm base module comprising the entire Porteus base system as fast as the original base modules, and perhaps even faster, by simply renaming it to something the update-cache script understands to be a normal base module already cached.
You see, not being aware of this mechanism, I had named my single large xzm module something different than 00*-*.xzm and, therefore, the script would update the system cache for the entire Porteus system each time at boot up. On slower machines this would take a long, long time and the boot process would hang at:

"Updating kernel module dependencies
Additional xzm modules have been found - updating system cache"

tremendously slowing down booting.
I simply renamed my single base xzm module to "000-kernel.xzm" (probably any 000-*.xzm name would have done the trick) and now that single module boots quicker than the original Porteus base modules did. It was misleading on my part to assume that the slow down was due to having a single large module as opposed to several smaller ones.

So I have resolved the whole issue by simply renaming it to something the system understands as already cached - which, in fact, it is....since I had copied the whole Porteus filesystem booted in aufs to an ext4 hard drive partition and then booted it there as a traditional slackware install in order to take out programs and make changes before making an xzm module out of it with dir2xzm..

In the future, when customizing a system again from scratch, I'll go through the procedure you have outlined and end up with several base modules as in the original Porteus base folder, but right now there is no point in doing all that work....the single base module is lightning fast even in my older hardware.

Thank you for being here and for pointing out this very simple solution to what seemed to be a much more complex and demanding issue.

Re: SOLVED How do I generate modules after modifying the sys

Posted: 10 Jan 2013, 16:15
by Ahau
glad it's working for you, claude!

To clarify my previous post, it was not my intention to suggest that we should remove more packages from the standard Porteus ISO's. We've done our best to remove everything we can while preserving functionality, so removing more would definitly start to break things or limit what the system could do. My comments were in the context of an individual user wanting to remove packages from the base modules for their own use, or perhaps for a Porteus-based distro.

I'm working on a doc now that will bring a lot of these concepts together, and I'll post it in the HOWTO's section when I have it more complete.

Thanks!