Official Package Manager

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Official Package Manager

Post#1 by brokenman » 20 May 2011, 23:37

I'm starting this thread so i can get advice and make announcements as to the progress of the package manager. The package manager will be a well commented shell script so that anyone else can make changes/improvements. It will use the Xdialog library as a front end.

At the moment all modules will need to be created/converted by someone in the XZM team. This will prevent the 'wild west' package repo that some other places have, where anyone can upload anything if it works or not. It means more work but this will ensure that all modules on the official repo 'just work.'

I have created a script that will convert a standard slackware package into a 'porteus compliant' module that contains two extra lines in the /var/log/packages file. People in the XZM team will be sent this script and a tutorial on how to build modules for the server.

PACKAGE CATEGORY: internet
LIBS REQUIRED: lib1.so,lib2.so.1.2,lib3.so.4


The converter script will show the user a description of the package and ask which category it should belong to. The second line is generated by an 'object dump' of all executable files found in the package.

A third file will be created by the person building the module and will contain a list of required packages.
/var/log/required/porteus-required

-------------------
The Server

On the server will reside files similar to the slackware standard.

CONTAINS.TXT
PACKAGE NAME: pidgin-1.2.3-i486-1
./usr/bin/pidgin
./usr/bin/pidgin-poo
/usr/lib/nest.so.1.3

PACKAGES.TXT
PACKAGE NAME: pidgin-1.2.3-i486-1
SIZE COMPRESSED: 234 K
SIZE UNCOMPRESSED: 1200 K
PACKAGE CATEGORY: internet
PACKAGE DESCRIPTION:
pidgin: description goes here
pidgin:
pidgin:
pidgin:

RESOLVED:
package1-1.0-i486-1.xzm (these are other modules that are required for pidgin to run smoothly)
package2-required-1.1-i486-1.xzm
package3-noarch-1.xzm


FILELIST.TXT
a list of files on the server and their download paths.
--------------------------------

The package manager will download these three files and use them to resolve a package and it's dependencies.

So this is the basic structure i have in my head. If anyone has any constructive ideas, or things i may have overlooked then this is place to air your concerns. I am pushing hard to have this completed and testing finished for the v1.0 release. I will need help creating modules for the server. This is not a complicated task but requires time (to find deps). Please post in if you want to help with this.

Please remember: Second to the OS itself, the next most important thing are the modules that run on it.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Official Package Manager

Post#2 by Ahau » 21 May 2011, 00:42

Pfft. Everyone knows the documentation is more important than the modules or the distro. If folks don't know how to install it, how are they going to use it?

Ok, all joking aside, I'm really excited by this announcement, and would love to help out as much as I can (though I'll have to give priority to documentation).

Are you intending to have a 1:1 ratio between slack packages and porteus modules, or will multiple packages be combined into a module? I'm a tad concerned about keeping my /porteus/modules clean and tidy ;)
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: Official Package Manager

Post#3 by brokenman » 21 May 2011, 01:10

1:1 ratio. Having all the required files inside one large package would be ideal, the package would just work ... however as we know from experience (rnport most recently), overlapping files can break porteus. For this reason all files will be separate on the server. The package manager will grab them all.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Official Package Manager

Post#4 by Ahau » 21 May 2011, 01:51

works for me. Here's a wild-ass idea that may be of no merit: what if a package and all it's deps were in a subdirectory of /porteus/modules, to help know which libs belong to which apps? If a lib was required for more than one app, a symlink could be made in the new apps' directory, or maybe the lib could get moved to a common-lib directory.
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5660
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

Re: Official Package Manager

Post#5 by fanthom » 21 May 2011, 09:09

"Here's a wild-ass idea that may be of no merit: what if a package and all it's deps were in a subdirectory of /porteus/modules, to help know which libs belong to which apps?"
linuxrc supports sub-folders so this could be done.

package manager could create folder: /home/guest/opera and put all deps into it. in case when user gets bored with opera he could just delete /porteus/modules/opera and get rid of all unnecessary deps.

in case of deps shared between modules i could extend porteusmodman with feature which checks if deps from opera folder are needed by other modules and move them to first one found in /porteus/modules/some_app folder.
in this case other deps could be still resolved.

there is one disadvantage:
removal of folders in /porteus/modules must have been done through porteusmodman only (to make sure that all deps are resolved correctly).
Please add [Solved] to your thread title if the solution was found.

User avatar
wread
Module Guard
Module Guard
Posts: 1255
Joined: 09 Jan 2011, 18:48
Distribution: Porteus v5.0-kde-64 bits
Location: Santo Domingo
Contact:

Re: Official Package Manager

Post#6 by wread » 21 May 2011, 12:19

Meanwhile, until we decide how to organize our official package manager, I am posting the set of modules you will need to get Gimp-2.6.11 up and running in Porteus V1.0 32 bits. I made extra a directory in mediafire for it. Give it a try!......

http://www.mediafire.com/?bajz5tl6nlf7j

I have put them in an extra folder /modules/Gimp to keep overview of dependencies.

Enjoy!
Last edited by wread on 15 Dec 2011, 13:21, edited 1 time in total.
Porteus is proud of the FASTEST KDE ever made.....(take akonadi, nepomuk and soprano out and you will have a decent OS).
The Porteus Community never sleeps!

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: Official Package Manager

Post#7 by brokenman » 21 May 2011, 18:39

Not a bad idea Ahau, although it would mean that every package must have it's own folder on the server. I already have a libraries-common folder the same as the slacky directory where popular libraries will reside.

At the moment i am trying to get the package manager (ppm) to work like so:
User chooses to look for a package by either:

1. Search by category
2. Search by package
3. Search by string

Once they choose the package they will get a description of it, and see the compressed size. If they decide to download then all dependencies will be gathered into a list and downloading will commence. After downloading user will have option to merge or not. The idea is to get the user to do as little as possible, and have the script take care of deps. I think all deps will reside on the server because modules will be checked before uploading.

Later will come an option to use the slackware repos to generate modules aswell.

Wread .. thanks. i was not looking forward to tackling that one.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5660
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

Re: Official Package Manager

Post#8 by fanthom » 21 May 2011, 19:04

@brokenman
thanks for skipping my post silently :)

i think i get it now: modules would be spitted on the server only and merged locally on the user request. makes sense for me.
i would still expect libs conflicts after some time. my solution is more solid but in cost of loosing flexibility (modules would have to be managed by ppc only. no removing under windows, other linuxes, cli, etc...)

Cheers
Please add [Solved] to your thread title if the solution was found.

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: Official Package Manager

Post#9 by brokenman » 22 May 2011, 17:04

Sorry Fanthom ... didn't mean to hurt your feelings. I just needed some time to construct this new concept in my head.

i think i get it now: modules would be spitted on the server only and merged locally on the user request. makes sense for me.
Yes the user would have the option .

i would still expect libs conflicts after some time.
That is true, and yours/Ahaus version may solve this.

there is one disadvantage:
removal of folders in /porteus/modules must have been done through porteusmodman only (to make sure that all deps are resolved correctly).

It means user must use PPM to download packages, but then use porteusmodman to activate/deactivate or remove packages. This is not a bad idea but what about people who like to keep modules on a dedicated partition and not modules folder? Perhaps they run a couple of different versions of porteus and want a central local repo.

I think user could choose where the 'module folder' will be when they first start PPM. This info could be passed onto porteusmodman.
I think i can continue with building PPM and get as far as downloading the module and required deps, and then i will need some guidance about how the porteusmodman will interact. Sound like a good plan?
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: Official Package Manager

Post#10 by Ahau » 22 May 2011, 17:31

works for me. I'm a "learn by doing" kind of guy, so it will help me see the vision better once I can start playing with it. I like the idea of merging modules locally at the user's request, that makes managing the /modules and /optional folders easier, but I see the concerns that fanthom has as well. You don't want to build a module for midori that includes webkit+all its deps, then build a module for another webkit browser that you wind up liking better, only to have it go tits up when you pull the midori module.

We'll sort it out :) I'm continually impressed by our development team's ability to problem solve.
Please take a look at our online documentation, here. Suggestions are welcome!

Falcony
Full of knowledge
Full of knowledge
Posts: 237
Joined: 01 Jan 2011, 12:44
Location: Russia

Re: Official Package Manager

Post#11 by Falcony » 23 May 2011, 12:03

@brokenman,

interesting spec. for package manager

Q1 - Will be versions control & update imlemented? If so how;

Q2 - What about conflict of modules?

@fantom,

I liked you idea basically.

But same Q1 and Q2 to you, could you explain how to avoid such situation which lead to mess of slax modules database on slax.org modules repo which was created by different repo's provider?

Or it will be done via one cenrillized repo?

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: Official Package Manager

Post#12 by brokenman » 23 May 2011, 14:20

Or it will be done via one cenrillized repo?
Yes there will be one centralized repo, with all modules being 'verified' before uploading.

Q1 - Will be versions control & update imlemented? If so how;
You mean for just the package manager? A simple version of yymmdd (110529) variable into the package manager script will signify the version. A menu item in the package manager menu will be 'Check for an update'. When chosen it will download and check the current version online to the version in the variable.


Q2 - What about conflict of modules?

This is something that is almost unavoidable, but should not happen very often. Slacky handles this with a conflicts file, but i very rarely see this being used. The porteus package manager will look for the exact version library it requires instead of looking for version 1.3+. It uses 'ldconfig -p` to list all local existing libraries, but has no way of knowing if a new chosen library will conflict or not ... until it happens.

--------------------------------
An alternative solution
--------------------------------
What about having a folder structure locally that mirrors the online repo. For instance ... in a chosen folder (perhaps the optional folder) the user would have subfolders network, office, desktop, graphic etc. If pidgin is chosen (for example) it will be downloaded into the 'network' folder and its deps will be put in there associated folders. Porteusmodman could then take care of the modules by first "sniffing" the included porteus-required file which contains a list of the required modules.

In this way a user can delete whatever module they don't want without killing all deps too. Modules can be deleted in whatever way the user prefers because porteusmodman only needs to see the 'porteus-required' file in order to activate all the deps. This method also keeps things tidy for the user when they begin to accumulate alot of modules.

Using this method ... in the future i also see it supporting 3rd party repos such as puppy, slacky and perhaps fidoslax by converting the modules using a script. With what i have here at the moment, it looks like it will all be automated. You can just run a module through the 'Porteus modwash' and it comes out the other side ready to upload onto the server!
How do i become super user?
Wear your underpants on the outside and put on a cape.

Falcony
Full of knowledge
Full of knowledge
Posts: 237
Joined: 01 Jan 2011, 12:44
Location: Russia

Re: Official Package Manager

Post#13 by Falcony » 24 May 2011, 04:51

Thank you for detailed explanation.
Using this method ... in the future i also see it supporting 3rd party repos such as puppy, slacky and perhaps fidoslax by converting the modules using a script.
Regarding fidoslax - if official repo will be good enough I will drop supporting fidoslax repo.

This is reasonable - be close to Porteus mainline and save time.

But idea of @fantom liked me much - very interesting solution which useful when you need local repo or mirror - jusr rsync official repo and use additional modules from usb flash when needed

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

XZM team listen in ...

Post#14 by brokenman » 25 May 2011, 02:44

Ok i have everything laid out and working well now. Here is the description of the process i am using.

I have a local repo on my drive sda6 that has subfolders for all categories of modules (office, development, libraries etc).
I have three scripts which i use to produce the module and it's dependencies.

1. Convert slackware package to Porteus
- supports using local folder of packages
- supports using a slackware ISO
- supports using a single local package
- support using an online repo


2. Libfinder (does a better job than slacky for finding which package a lib belongs to)
- it greps all slackware repos (including extra) from 12.2 to current
- after finding the correct package you can choose to download


3. Package manager (which also acts as a libfinder for me locally)

Step 1.
Convert the slackware package (e.g pidgin) that you want to use as the main module. The script will add the necessary extra lines to the /var/log/packages file including a full list of all required libraries. During conversion you are shown a description and asked which category you want to put the module in. After conversion, drop the module into the local repo.

Step 2.
Run the package manager and select the pidgin package. The package manager will read all the required libs, and then remove from the list anything that exists already on the users system or in the module itself. It then resolves all of the required packages for those libraries if they are already in the repo, or produces a file of 'required libraries' that i need to resolve.

Step 3.
Take the list of required libs and run them through the 'libfinder' script which will download the slackware package. These are then converted and dumped into the repo. When i run the 'package manager' again it reports that all deps are resolved.

That is the basic procedure. All up it takes me around 3 minutes to decide on a package, and have all libs resolved and working in the repo. This method means that users won't ever have non working packages because they are tested before uploading.

The XZM team will each have the conversion script, and after converting a slackware package, they can upload to temporary storage. They can either resolve the deps themselves or myself or Fanthom can do it by running the script. Nice and easy. Once more modules are uploaded, i just run a fourth script from the server that generates the 'INDEX' files that the package manager will use. On the users side ... it all happens very quickly because the script is just processing text files. The only other processes involved on the users side is one ldconfig, and some object dumps. The downloading will be the longest part, but as the users local repo grows, less will be needed to download.

Now that i have that part done, we need to sort out how the packages will reside on the users machine. I vote for the 'own personal repo' in a folder of your choice. This way the package manager doesn't care if you delete a module, a library, or if you like to wear womens underwear at night. It just scans the local repo each time you request a new module, and sees if you have the deps there or not. I am also playing with the idea of having a folder on the repo with JUST the libraries that are very common. This way a user won't have to download an entire module just to use a couple of common library that it contains.

Please keep the ideas coming fast because i am about to start stage 2 .... the users side of things. Let me knwo what you want.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Hamza
Warlord
Warlord
Posts: 1908
Joined: 28 Dec 2010, 07:41
Distribution: Porteus
Location: France

Re: Official Package Manager

Post#15 by Hamza » 25 May 2011, 04:22

Thanks a lot!

We will have some news feature on v1.0!
-Windows Installer
-New Install guide
-Full Network Installation
...
NjVFQzY2Rg==

Locked