Neater Package Management

[edge226]

At the moment it does not seem as if the boot up scripts can handle this, However it would be a very cool addition to Porteus and keeping a tidy system without package dissarray.

Okay so say you are on a clean porteus system freshly installed. I install Gparted which has a few dependencies.

This is what is required to be installed:
gparted-0.12.1-x86_64-1sl --> porteus repo
atkmm-2.22.4-x86_64-1sl --> porteus repo
cairomm-1.9.8-x86_64-1sl --> porteus repo
gtkmm-2.24.0-x86_64-1sl --> porteus repo
glibmm-2.27.99.2-x86_64-1sl --> porteus repo
pangomm-2.28.1-x86_64-1sl --> porteus repo
libsigc++-2.2.9-x86_64-1sl --> porteus repo

All of these modules are placed into the root of the modules folder, This eventually leads to a very messy modules folder. I have thought up a few ideas that could tidy this up, two in particular I like best for different usage applications so I will discuss them.

The first one would be for a desktop system where porteus would be stored upon a usb drive or a hard drive.

This also has two branching ways of doing things, one being more static and standard the latter being more dynamic and flexible. The boot up would need to ignore symlinks in order for this to work correctly.
A) Have all packages be within their own folders and organized via preflagged categories, I purpose that dependencies are symbolically linked to where they are located within the required programs folder. Thus dependencies are simple to track on the system.

1) CLI - More Splits in Categories and then folders with program names, dependencies are symlinked for simple CLI or file browser tracking.
2)GUI - More Splits in Categories and then folders with program names, dependencies are symlinked for simple CLI or file browser tracking.

This is probably the simplest way of doing the sorting and organization logically. Upon download of a package you create a directory within the modules folder that has the package name, within this directory contains The program you downloaded and directories for the dependencies that were downloaded with that program, if dependencies were already resolved, then a symlink is created to the location of that package and created within the package directory to all other requiring packages. Creating a web of dependency symlinks allowing simple CLI or file browser tracking.

in the case of gparted the structure would look like this:

$MODULES/gparted/gparted*.xzm
$MODULES/gparted/atkmm/atkmm*.xzm
$MODULES/gparted/atkmm/depends/symlinks
$MODULES/gparted/atkmm/required/symlinks
$MODULES/gparted/cairomm/cairomm*.xzm
$MODULES/gparted/cairomm/depends/symlinks
$MODULES/gparted/cairomm/required/symlinks
$MODULES/gparted/gtkmm/gtkmm*.xzm
$MODULES/gparted/gtkmm/depends/symlinks
$MODULES/gparted/gtkmm/required/symlinks
$MODULES/gparted/glibmm/glibmm*.xzm
$MODULES/gparted/glibmm/depends/symlinks
$MODULES/gparted/glibmm/required/symlinks
$MODULES/gparted/pangomm/pangomm*.xzm
$MODULES/gparted/pangomm/depends/symlinks
$MODULES/gparted/pangomm/required/symlinks
$MODULES/gparted/libsigc++/libsigc++*.xzm
$MODULES/gparted/libsigc++/depends/symlinks
$MODULES/gparted/libsigc++/required/symlinks

Next you install a program that depends upon both gtkmm and glibmm.

$MODULES/proggy/proggy*.xzm
$MODULES/proggy/depends/symlinks >> $MODULES/gparted/gtkmm && $MODULES/gparted/glibmm
$MODULES/gparted/gtkmm/required/symlink && $MODULES/gparted/glibmm/required/symlink >> $MODULES/proggy


The second suggestion would be more for a server setup using NFS and this is modules within modules. Doing so you can have a centralized module location in one place and have many different machines connect to those modules and load them via network to the decentralized location. I could see this being useful for a business or school network. modify packages in one location, reload the machines and the changes are reflected upon all of them.

[brokenman]

Interesting ideas. Before we created the package management we had a long thread discussing the best way to do things. We arrived at the current system in this way. I wish you present at this time.

I think the majority of people that us the PPM for package management don't really look (and dont want to) inside the modules folder. They activate a main module, and PPM checks if the dependencies exist within the storage folder. Storage doesn't have to be 'modules' folder, it can be a universal folder on a network drive.

I like the idea of symlinks, it has some very useful advantages. If you want gparted you activate it, and then activate everything symlinked in it's folder .... but i don't see how having a folder for each module is any neater. At the moment PPM pulls apart the module and does a direct 'ldd' for required libraries, then searches for modules containing those libraries. I have left open the possibility to add a 'dependencies file' within the module which could store a list of dependencies.

With your current system (as i understand) ... when first creating a folder heirarchy ... you still need to analyze the module, find the dependencies, find which package contains those dependencies (at this point PPM just activates them) and then create the main folder with symlinked deps. It is doable, but again i don't see how it is cleaner. What i really like about this system is that all 'non-main' applications (things people will never want to activate as a main app - like libgtk-something) could be stored in a 'library' folder and would not appear in any menus.

[edge226]

well in the current system when I run ls /mnt/live/porteus/modules I get a listing of everything installed. If I wanted to only remove gparted and its dependencies, finding whether everything is orphaned or not. Doing this via command line does not seem like a very neat solution since I currently have 129 modules installed. Without a bunch of research into what was required this is not a simple task, which it should be. It is in the majority of distribution and something alone these lines would resolve that issue.

[brokenman]

Ok i see. The 'deactivate' function of the PPM works like this. If you want to deactivate gparted it will search for all dependencies. If deps are found it will deactivate them ONLY if they are not required by some other application that is currently activated. It still needs improving so thanks for your suggestions.

[edge226]

I see, that works well for deactivation, possibly a rmxzm script that will search the dependencies and remove them, or a xzmdeps that will simply print the deps out in a way that you could do

Code: Select all

rm `xzmdeps name.xzm`

I personally cannot seem to get ldd to work, the man does not specify the type of file required to check, if I check on a binary it just says
"ldd /usr/sbin/gparted
not a dynamic executable"

[bigbass]

Hey edg226

I personally cannot seem to get ldd to work, the man does not specify the type of file required to check, if I check on a binary it just says
"ldd /usr/sbin/gparted
not a dynamic executable"

try this instead

Code: Select all

ldd /usr/sbin/gpartedbin 

however the correct way is to check every bin in the package there may be multiple bins
I wrote a dependency checker in my light package tools
it keeps a log in /var/lib/pkgtool/dependencies
but this doesnt resolve dependencies just warns you to whats missing
or what is required I spent a lot of time with packages and package management

I have to take off my hat to brokenman one very smart guy
he is doing a excellent job if you knew how complex dependency
resolving is and even more complex when you have custom packages
thrown in the mix

the only perfect way to have dependency resolving 100%
is to limit things to only a limited repo
and that would just cramp the freedom of linux

so what we got is is the best of both worlds
organized and flexible that takes care of almost all
of hard work automatically

Joe

[edge226]

@bigbass

I am working on writing a bash script that will tell you of the dependencies of packages, not libraries.
This is what I have thus far, not too much but I just started it.

Code: Select all

#!/bin/bash
#gain storage location from /etc/ppm/porteus.conf
storage=$(grep -A1 "STORAGE:" /etc/ppm/porteus.conf | tail -1);

#input of module name to find dependencies
echo "Please input module name: ";
read name;

#search for module name at storage location, overwriting module name with entire path to module.
name=$(ls $storage/$name*);

[bigbass]

edg226

I am working on writing a bash script that will tell you of the dependencies of packages, not libraries.

This may give you a nice head start ( I didnt have that backend tool when I started writing my tools in bash )

Code: Select all

slackyd -help 

that will save you a lot of time brokenman wrote a GUI for it here and I will write one too but for another language * it will make for a much cleaner way to code system calls from BaCon

just scratching the surface here
I didn't mention glibc and GTK matching
the tool chain used to compile or headers used to compile the kernel
which all are very important (slackware leaves this up to the system administrator because it is a complex topic)
that the system admin would have to carefully resolve

but the good news is we have many nice tools to aid the tedious job
of maintaining a system that doesnt break

*slackware doesnt like symlinks they get converted to scripts to make the links during the install
by makepkg ( but using modules you can do whatever works correctly and is standardized)


Anyway have fun its a good thing to make some more tools
I would like to see more good tools so have a go at it
there is always room to improve
but maintaining a standard is much more important

Joe

[SunBurnt]

Perhaps a package setup like AppDir ?

The apps. files are in a dir. and stay there and are used from there, so no need to install or uninstall.
Very few links or no links are needed, and the standard desktop menu can list the apps.
So they`re truly portable and separate packages, and they can be totally independent of other packages.

The package can be binary files that`re ready to run, or source files that`re compiled when the package is run.
In the case of Porteus I don`t think the source package type would be useful.


# I made a modified AppDir package type called AppPkg that can have many apps. in one package.
It can use shared libs, or include some or all of the libs in the package, and can also be a statically compiled app.
This eliminates the problem of lib. conflicts as each app. can carry it`s own libs. and other needed deps.
It can also contain the apps. config. files inside the package so that they`re carried with the package.
Good for many types of apps. like games that you want the game saves to go with the game files in the package.

I`ve looked at package systems for awhile now and this seems to be the best all around setup.
Everyone has their own favorite setup for everything, but this setup solves problems and simplifies packaging.
I`ve built a number of AppPkg packages from Ubuntu Lucid binary files. All work on Puppy528 ( and probably Ubuntu ).
It`s nice not having to compile the apps., just pick a solid distro. ( Debian, Ubu., Slack., etc.) and use their files...

[brokenman]

I also have an app here for collecting ALL required files for running an app. Can't remember the name of it right now, but you run the app with the target app as an example: packit gparted it will then package everything that gparted uses into one directory (fakeroot) and you can run the main executable. I have used it to run 64bit apps on 32bit. Very useful.

[bigbass]

There are other major concerns and thats when you have to build the entire system from Packages (not add an app to an already working system)
this is when theory is tested and only a working system is the winner


I had done this part before installing over 500 packages to build an iso (another project that gave me some practical experience to see what was needed)
this is when everything needs to be sorted out and there is also a proper install order
it is very tedious to set up for the first time but much easier if you start on a 100% slackware base
but it can be modified easily since everything is accounted for in an organized manner

As it stands we have thousands of well written build scripts for official slackware that just changing the source version number we can recreate the package on any system
we have thousands of well written build scripts by slackbuilds.org http://slackbuilds.org/
we have hundreds of modules built for Porteus by the Porteus team and tested
http://ponce.cc/porteus/i486/modules/
http://ponce.cc/porteus/x86_64/modules/
we have numerous well built packages from people that have a good reputation of building packages

Slacky.eu in English


We can install our own custom package of our choice whenever we want and remove them easily
As it stands We got everything very neat

The good news this will be fun when we switch over to slackware 14 !



The reason third party packages dont get into official slackware is
http://www.slackwiki.com/Third_Party_Package_Managers
here is a quote

Third Party (Unofficial) Packages

This subject can elicit opinions just as strong as the topic of third-party package managers, and there are knowledgeable people on both sides of the issue. Third party packages can obviously present a security risk, as there's no way to know what's really in them, but for the average user, there are (amazingly enough) bigger concerns. On one hand, the user wants a package of some application that's not included with Slackware, but he doesn't want it to create other problems. Oftentimes, third party packages are not built on "clean" systems -- that is, the computer on which they're built has quite a few other non-official packages installed -- so the resulting package needs those other things to work properly. If the packager doesn't make note of all the different packages that this one depends on, the user installs it only to find that it still doesn't work. Even worse, those other dependencies are often not even necessary if the package is built on a clean system - if the other package had not been present, the new one obviously wouldn't have linked against it. Installing third party packages from two different packagers can cause problems. Package A might depend on Package B, which depends on Package C and optionally Package D. If the person who built Package B only linked it against Package C (not D), but the Package A expects Package B to have both Package C and Package D dependencies, then Package A may not work, even though you (think you) have everything it needs installed. Another frequent problem with third party packages is encountered by replacing stock (official) packages with unofficial ones. In addition to the library version issues mentioned in the swaret discussion, newer package versions often require newer versions of other packages, which might require newer versions of still other packages, and before you know it, you have replaced a sizable portion of the official Slackware packages. If/when something breaks, you will find it very difficult to find someone willing to help you, as there's no way to know how much effect the unofficial packages are having.

[brokenman]

The good news this will be fun when we switch over to slackware 14 !

When is that again? I think i have holidays in a remote part of the Amazon on that day. No internet in the amazon (yet).

[edge226]

I have hit a snag on a portion of parsing, I know how to do this action in C++ however that is not the preferred method, I want to learn more about how to do things in linux rather than the stuff I learned in College. I am trying to use sed and I have managed to get the output to become equal to running ldd on a specific binary file.

At the moment I just want the script to pop out the package name and its dependencies, in the future I may add more options or flags to the script allowing it to pop out whether any of the dependencies are orphaned or not. I am attempting to do my parse with sed and having some trouble with the syntax of the command. Here a current sample output and what I would like to end up with:

#current
linux-vdso.so.1 => (0x00007fffe60ed000)
libparted.so.0 => /usr/lib64/libparted.so.0 (0x00007fdcae8bc000)
#parsed
/usr/lib64/libparted.so.0

I know there are multiple ways to do such and this may not be the most efficient way of doing things, however it is the way that will teach me the most about how to do linux parsing.

Posted after 11 minutes 48 seconds:
@bigbass I agree with your post that the package management is done in a neat manner, I suppose the word I am looking for is organization. A new user of any distribution will not mess around with any of the internals and the modules directory thus becomes quite disorganized. I have used porteus as a general user would use it. Leaving all my modules in the modules directory. As being a command line junky to me it has become quite a mess. A tool that gave a little background on the organization such as dependencies is what is missing IMHO. Hense my attempt to create a tool for this purpose. I want to create it using bash or other preinstalled packages so it does not create any extra dependencies required to use the script and thus would be possible to be added to Porteus during a later release if others like it as well.

[bigbass]

Hey edge226
http://forum.porteus.org/viewtopic.php? ... a1053e02b4

in /tmp is a bash script Light-Package-Tool if you look at lines starting at 555
there is a dependency checker that filters the output

select dependency from the menu
then lets say acl for an example does this looks like what you want to filter?


Joe

[brokenman]

libparted.so.0 => /usr/lib64/libparted.so.0 (0x00007fdcae8bc000)

First step is remove lines like libc and linux-gate.
sed -e '/gate/d' -e '/libs/d'

Then grab the 3rd column.
awk '{print$3}'

You can use a delimiter (other than the default 'space') with:
awk -F/ '{print$3}'