Docs and Porteus info. don`t give clear picture.

Post here if you are a new Porteus member and you're looking for some help.
SunBurnt
White ninja
White ninja
Posts: 13
Joined: 05 Aug 2012, 20:24
Location: Arizona, U.S.A.

Docs and Porteus info. don`t give clear picture.

Post#1 by SunBurnt » 08 Aug 2012, 05:18

I assume that as Porteus uses aufs and Squash files, then it like Puppy makes union layers of the app. modules?

Ahau:
One pic. being worth Ks of words, a graphic depicting this would be great! Also one showing the module layout.


A thought:
If this is the case, then Porteus like Puppy has the potential for lib. conflicts, and white-out file problems as well...
I rather dislike union file systems as they`re an unnecessary complexity and induce new problems.
A fully functional Linux O.S. simply doesn`t need it! All that needs to be done can be done without a union FS.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Docs and Porteus info. don`t give clear picture.

Post#2 by Ahau » 08 Aug 2012, 07:08

Welcome, SunBurnt!

Yes, Porteus uses aufs to combine multiple xz compressed squashfs filesystems (plus other odds and ends) into a 'live' filesystem. Each module is mounted as a layer in aufs, correct.

RE: a picture... the best I'm able to offer at present is this article, which is text only: http://porteus.org/info/docs/57-general ... cture.html another good one (but presently in need of some updating) is this: http://porteus.org/info/docs/57-general ... order.html

I'm afraid I'm awful at graphic design -- maybe I will take a stab at hand-drawing something :D The problem, though is that the way aufs works to build Porteus, as I see it in my head, is more of a flash animation with transparency effects, rather than a picture (it is a 'live' system after all). As bad as I am at drawing pictures, I'm afraid I'm even worse at animations lol.

I do suppose there is a potential for problems with library conflicts and white-out files, but these are generally rare -- I've only seen reports involving white out problems once or twice, and our library conflicts are pretty minimal as well because we stick pretty close to the libraries in standard slackware, and fanthom and brokenman do a good job of putting well tested modules into the repo that are built on the same core libraries.

A fully functional Linux OS doesn't need union, you are correct. However, it does come in quite handy for a 'live' distribution, and (in my opinion) we're able to work some real magic because of the union FS. Porteus would lose all kinds of flexibility and functionality if we restricted ourselves to running without aufs-- this would be a "standard install" linux OS, which is not what we're doing here. This functionality offsets the complexities that are introduced (when those complexities are handled well by the developers), and everything should flow smoothly for the end user. Feel free to ask us questions as you go along and we can dig deeper.
Please take a look at our online documentation, here. Suggestions are welcome!

SunBurnt
White ninja
White ninja
Posts: 13
Joined: 05 Aug 2012, 20:24
Location: Arizona, U.S.A.

Re: Docs and Porteus info. don`t give clear picture.

Post#3 by SunBurnt » 08 Aug 2012, 08:19

Knoppix originally used a union for merging the legacy loose file system with "live" compressed files ( Cram, Cloop, Sq.).
A legacy full install of loose files needs no union, nor does a "all Squash file" O. S., and even it can have loose files in dirs.

In reading on Porteus ( I`ve read the pages you linked ) I see no need for the union, all of it can be done without one.
If ( like Puppy Linux ) you`re trying to / "root" merge loose files and Sq. files, then a union is needed.
But then libs. will over shadow duplicate libs. in lower union layers, and of course other files do this as well...

But in the idea of apps. in separate dirs. so the files are not scattered in /usr, no merging of the dir. tree is needed.
Each app. dir. is simply added and used as is. So there`s no intermixing of files, and no file tracking or union is needed.
The legacy Linux dirs. including /usr would be only for the base O. S., /opt or other dirs. hold the self-contained app. dirs.

Linux dirs. should be divided into R/O and R/W :
R/O dirs. ( /bin, /lib, /sbin, /usr ) go in Sq. files.
R/W dirs. ( /etc, /root, /var, homes, /usr/etc, /usr/local/etc, "others" ) go in the save file or on a partition.
Some of the R/W dirs. ( /dev, /mnt, /proc, /sys, /tmp ) stay in initramfs.

If many of the / dirs. are links they can point to any available file system, making the O. S. flexible and dynamic.
This design allows a highly modifiable O. S. with great simplicity. Porteus has some of it, Tiny Core Linux does too.


I don`t expect Portus to change any more than Puppy Linux has changed ( not at all...), but ideas are the foundation.
I hope others will join this discussion and add their thoughts and ideas to build upon that foundation... Terry B.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Docs and Porteus info. don`t give clear picture.

Post#4 by Ahau » 08 Aug 2012, 12:52

So, now I think it is me who needs a picture --

So, in your method, you would have a base installation inside one squashfs module, which would contain all of the RO files, and then a separate location for all RW files -- is this one location on disk, for example, you have one xzm module on your flashdrive, and then one folder, and the two are mounted into different directories at boot-up? Or, you would have dozens (or more) of directories on your flashdrive, each one being mounted into / at boot-up (or symlinked?) with a separate folder for every singly application inside /opt/usr/bin/$programname (or /opt/$programname)? If all of your RW files must exist 'natively' on disk (not compressed into separate squashfs modules), how would you handle installations on FAT or NTFS formatted flash devices?

Thus far, it is not really sounding more simple, so please help me to understand a little better.
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
Hamza
Warlord
Warlord
Posts: 1847
Joined: 28 Dec 2010, 07:41
Distribution: Porteus
Location: France

Re: Docs and Porteus info. don`t give clear picture.

Post#5 by Hamza » 08 Aug 2012, 13:01

That remember me Gentoo ... I don't know why (stage?) :)
NjVFQzY2Rg==

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Docs and Porteus info. don`t give clear picture.

Post#6 by Ahau » 08 Aug 2012, 16:42

@SunBurnt,

Here's a flow chart I put together: http://porteus-xfce.googlecode.com/files/flowchart.pdf
This is conceptual only, the actual process is not as linear (aufs gets created first and branches are merged in at the same time they are found and mounted on loops). Is this more or less what you were looking for in your OP? (excusing my lack of pretty pictures and my horrid penmanship).

As I re-read your last post, I think I may be understanding things a bit better -- in your setup, you would use an initramfs (which can be resized on demand) rather than an initrd, and I'm assuming the entire OS would run from inside the initramfs? Or would you use the switch_root function to somehow mount a new root in RAM? if you're constructing a new root file heirarchy inside ram, it seems like you'd either have a ton of folders mounted into it or you'd have to copy the entire OS into RAM; With the symlink approach, I'm trying to understand how this would be an improvement over aufs in any sense other than "it doesn't use aufs" -- if everything is arranged with symlinks, then any libraries that would overwrite each other in aufs would destroy each other's symlinks when inserted into the filesystem, and you'd have the same problems. You would also be unable to safely peel back the layers, as any links that were overwritten by a second symlink would be entirely deleted from the system if you tried to roll back the second insertion.

In my opinion, conflicting libraries/library versions are a problem associated with good or bad package management, rather than the use (or non-us) of aufs. If you're overwriting working libraries with broken ones with an aufs branch, then those same libraries would be inserted via symlinks or normal installation on a standard linux. If you are installing a package that relies on a different version of libraries than what is present in the OS, then the solutions are to recompile that package against the proper libraries, set everything for it in /opt, or upgrade everything in the OS to the newer version... perhaps I am missing something -- I'm quite prone to that :)
Please take a look at our online documentation, here. Suggestions are welcome!

SunBurnt
White ninja
White ninja
Posts: 13
Joined: 05 Aug 2012, 20:24
Location: Arizona, U.S.A.

Re: Docs and Porteus info. don`t give clear picture.

Post#7 by SunBurnt » 08 Aug 2012, 21:55

I think you`re still thinking in a layered mind set... Yes / is in /rootfs ( initramfs ).
The base O. S. is X desktop with a VT, editor, filer, and few other utils. ( like Tiny Core ).
Apps. should be replaceable, right? There`s always a new Firefox or Chrome coming out.

The only dirs. that apps. and libs. are added to is: /opt/AppPkg and: /opt/lib
AppPkg use shared libs., and can contain shared libs. and/or included libs. in them.
A run script in the AppPkg sets up it`s environment and paths to it`s libs and deps.
The apps. are then relocatable, same as compiling the apps. with -prefix does.

# I know a lot of the explanation is unfilled, but I think this gives an idea. Ask Qs!

Here`s a description page I wrote quite awhile back. I`ve rewritten it to update and clarify.
There`s no description of the AppPkg format, I`ll have to add it, but it`s much like AppDir.
##### A new design for a Linux distro. #####

# No union file system, it`s unnecessary and uses ram and cpu resources.
# Divide the main Linux directories into R/W and R/O uses ( shown below ).
# The R/W directories are on the Save media in the distro`s directory and in /
The R/O directories are in the boot Sq. file on a partition, LAN, or web.
# The boot Squash file can be copied to /rootfs, or to a separate ram disk.
# The Save can be a ext4 compressed image file, or a partition, or LAN, or web.
# All / directories and / links are in: /rootfs

### Future plans:
# No initrd.gz file, boot directly to the Squash file, run init and create / dir. tree.


##### All directories and links in / and links in /usr directory.

Link, r-w: /etc =========> /mnt/(save)/(distro)/r-w/etc ( SAVE )
Link, r-w: /lost+found ====> /mnt/(save)/(distro)/r-w/lost+found ( SAVE )
Link, r-w: /root ========> /mnt/(save)/(distro)/r-w/root ( SAVE )
Link, r-o: /bin =========> /(Sq.Mnt.)/bin ( SQ. )
Link, r-o: /dev ========> /(Sq.Mnt.)/dev ( SQ. )
Link, r-o: /lib =========> /(Sq.Mnt.)/lib ( SQ. )
Link, r-o: /sbin ========> /(Sq.Mnt.)/sbin ( SQ. )
Link, r-o: /opt ========> /(partition or network)/AppPkg ( Media )
Dir., r-w: /mnt ( / )
Dir., r-w: /proc ( / )
Dir., r-w: /sys ( / )
Dir., r-w: /tmp ( / )
Dir., r-w: /var ( / )
Dir., r-w: /usr ( / )
Link, r-o: /usr/bin ======> /(Sq.Mnt.)/usr/bin ( SQ. )
Link, r-o: /usr/include ===> /(Sq.Mnt.)/usr/include ( SQ. )
Link, r-o: /usr/lib ======> /(Sq.Mnt.)/usr/lib ( SQ. )
Link, r-o: /usr/sbin =====> /(Sq.Mnt.)/usr/sbin ( SQ. )
Link, r-o: /usr/X11R7 ===> /(Sq.Mnt.)/usr/X11R7 ( SQ. )
Link, r-w: /usr/etc =====> /etc ( SAVE )
Dir., r-w: /usr/local ( Contains a mix of links to SAVE and SQ. )
Dir., r-w: /usr/share ( Contains a mix of links to SAVE and SQ. )

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Docs and Porteus info. don`t give clear picture.

Post#8 by Ahau » 08 Aug 2012, 22:30

Do you have an estimate of how much aufs uses in terms of ram and cpu? How much efficiency are you really gaining?

Looking down your list, it sounds like it would be quite a bit of hassle to maintain, as you will have discrete packages that are split across modules -- if you want to remove a package, you have to pull the pieces from each module individually or maintain a tree with your whole RO system and rebuild every module every time you add or remove any package. That said, that could probably be managed with some scripts, and with the minimalist nature of it, you probably aren't looking at repacking 300MB of modules each time.

As you continue to develop your plan, I suggest you continue to focus on your problem statement and resolution; that is to say, you need to demonstrate how this setup will be better, by detailing the actual efficiencies (how much RAM/cpu are you saving, in a direct comparison) and problems that you are solving, while demonstrating that you aren't introducing new complexities and problems in their place. For example, I'd need to see reproducible white file/lib issues cropping up with Porteus -- if these problems aren't a factor for most users, then you will be hard pressed to develop a following for your distro.

Very interesting concept :)
Please take a look at our online documentation, here. Suggestions are welcome!

SunBurnt
White ninja
White ninja
Posts: 13
Joined: 05 Aug 2012, 20:24
Location: Arizona, U.S.A.

Re: Docs and Porteus info. don`t give clear picture.

Post#9 by SunBurnt » 09 Aug 2012, 07:17

Over the years unions have gotten much better ( aufs as to unionfs ).
The savings are not much considering newer cpus and large ram.
Many folks like aufs, they want control over the base O. S. file tree.
But it`s not necessary to micro manage it. Just rebuild it occationally.
Porteus has a better tool chain than Puppy it looks like, so few problem it seems.

I`,m not sure what you mean by this:
you will have discrete packages that are split across modules
The complete base O. S. is self-contained in the base Squash file.
The apps. are independant and self-contained in Squash files in AppPkg dirs.
The only other parts are added shared libs. in /opt/lib

I intended the AppPkg to be built from binary files, not compled.
So I`m not sure what exactly the problems are that you`re talking about.

beny
Full of knowledge
Full of knowledge
Posts: 724
Joined: 02 Jan 2011, 11:33
Location: italy

Re: Docs and Porteus info. don`t give clear picture.

Post#10 by beny » 09 Aug 2012, 12:24

well, apologize me, sunburnt how many time you have used porteus to make a compare with puppy,xz compression you can change kernel and can use the same init? i don't think,you can use xzm packages and txz packages at same time?,you can run a whole current into one usb key?if you like a quiet system copy2ram and no problem,if you want more power use a save dat or a folder where you save data,imho.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Docs and Porteus info. don`t give clear picture.

Post#11 by Ahau » 09 Aug 2012, 12:50

SunBurnt wrote: The complete base O. S. is self-contained in the base Squash file.
The apps. are independant and self-contained in Squash files in AppPkg dirs.
The only other parts are added shared libs. in /opt/lib

I intended the AppPkg to be built from binary files, not compled.
So I`m not sure what exactly the problems are that you`re talking about.
Ahh, I see. I misunderstood-- based on your directory layout, I thought you were going to have separate squashfs files for /bin, /dev, /lib, usr/bin, etc., rather than a single squashfs for all of your RO stuff. I've seen distros that package things that way, and it seems like a pain to administrate.
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
brokenman
Site Admin
Site Admin
Posts: 5461
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

Re: Docs and Porteus info. don`t give clear picture.

Post#12 by brokenman » 10 Aug 2012, 13:39

While fanthom had a few earlier teething problems (upstream) with aufs they seem to be ironed out now. Personally i see advantages in the layering nature of aufs and use it frequently to my advantage. As long as one manages packages correctly then breaking libs should not be an issue. The end user just sees a working system and whether he is working from a real fs or a layered fs is not important in most cases.

As far as building goes ... with the help of a few scripts we can start from scratch and rebuild Porteus in about 15 minutes (providing DM is already compiled). It's just a matter of grabbing the relevent slackware packages, unpacking them into a universal folder and squashing. Maintenance must be simple

I find your alternative layout interesting. Particularly the separation of R/O R/W areas. We are always open to improving Porteus with the help of community input, so your experience and input is highly appreciated. I see you have contributed quite a bit over at the puppy shelter.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
fanthom
Site Admin
Site Admin
Posts: 4566
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland, currently - Cork, IE
Contact:

Re: Docs and Porteus info. don`t give clear picture.

Post#13 by fanthom » 11 Aug 2012, 09:09

@SunBurnt

dividing the system into r/o and r/w areas is a bit of showstopper for me:
a) as a developer i love to mess with everything around and would hate to find modifying a script in /bin not possible because it's mounted as read-only.
yes - i can copy the script to any r/w location from the PATH (like /usr/local/bin) and modify it there.
unfortunately it takes an extra effort.
b) explaining to the users that some dirs are writable and some not is a real challenge

other things:
a) layered approach gives you advantages, for example i can modify a config in /etc, test it and when something goes wrong revert to the original by copying from /mnt/live/memory/images/module.xzm
b) layered approach allows you for tricks not possible in traditional system, example:
/etc/rc.d/rc.inet1 from 001-core.xzm has executable bit set thus is started by the init scripts.
Porteus has Network Manager placed in 002-xorg (NM-applet needs GUI) so we could put second copy of /etc/rc.d/rc.inet1 in 002-xorg module and make remove exec bit.
this way rc.inet1 wont be started during boot as is not necessary (NM takes over the job).
in short words: if 002-xorg is present then NM manages the network, if 002-xorg is absent (Porteus in text mode) then rc.inet1 does the job.
same applies to xinitrc symlink which can be "overriden" by extra desktop (Gnome, E17) modules added to the system. all what you need to do is to place correct symlink in your desktop module - no other action is required.

if we drop aufs in favor of your solution - will activating/deactivating of new modules be possible?
if yest then please explain how do you see it.
Please add [Solved] to your thread title if the solution was found.

User avatar
wread
Module Guard
Module Guard
Posts: 1070
Joined: 09 Jan 2011, 18:48
Distribution: Porteus v3.2.5-kde5-64 bits
Location: Santo Domingo
Contact:

Re: Docs and Porteus info. don`t give clear picture.

Post#14 by wread » 12 Aug 2012, 18:44

@SunBurnt:
There is a section of this forum named "Community Effort", where you can experiment the praxis of your ideas.
Porteus is a community married to the live system because we recognize its advantages and are aware of its weaknesses; that doesn't mean we are closed to any better way of doing things.
I have six or seven pendrives -all different- that could be used to write the story of Porteus -> evolution is the name of the game. Keep trying...

Regards!
Porteus is proud of the FASTEST KDE ever made.....(take akonadi, nepomuk and soprano out and you will have a decent OS).
The Porteus Community never sleeps!

User avatar
Hamza
Warlord
Warlord
Posts: 1847
Joined: 28 Dec 2010, 07:41
Distribution: Porteus
Location: France

Re: Docs and Porteus info. don`t give clear picture.

Post#15 by Hamza » 12 Aug 2012, 20:57

evolution is the name of the game. Keep trying...
+1 :good:
NjVFQzY2Rg==

Post Reply