Porteus-AUR: Building Porteus modules w/ Arch build scripts!

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
User avatar
ViktorNova
Contributor
Contributor
Posts: 33
Joined: 19 Jun 2013, 19:38
Distribution: Porteus 3.0rc1 64-bit
Location: Portland, Oregon, USA

Porteus-AUR: Building Porteus modules w/ Arch build scripts!

Post#1 by ViktorNova » 23 Jan 2014, 07:46

I have been doing this for awhile on my own, but finally put it together and am excited to finally share =D

After some trial and error, I've come up with a very easy and clean way to build true Porteus packages from the vast build script repositories of Arch Linux and the AUR. I have put everything up on GitHub, including 64 bit xzm modules and detailed instructions on how to use it
https://github.com/ViktorNova/porteus-aur

This will build real Porteus packages, without weird compatibility issues, and the available software list is massive.

In my experience, this has been a very pleasant experience in bringing new software to Porteus, and have had surprisingly more success doing it this way than using Slackbuilds. Like other things, there is no dependency tracking at the moment, but if you are missing a required library or build dependency, you will be told about it so you can go install the dependency from Porteus/Slackware/Slackbuilds/or Arch/AUR using this method, and then try again.

Please test and tell me how it goes! Something along these lines would be really great to see integrated into a future version of PPM ;)

User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5660
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#2 by fanthom » 23 Jan 2014, 09:49

sounds promising and could be added to usm rather than PPM :)
will ask brokenman.

btw: bumped your rank to Contributor.

EDIT://
just realized that it may kill porteus if user compile some lib which is in higher version than in slackware world. what's your experience? do you compile apps only or libs as well?
Please add [Solved] to your thread title if the solution was found.

User avatar
francois
Contributor
Contributor
Posts: 6433
Joined: 28 Dec 2010, 14:25
Distribution: xfce plank porteus nemesis
Location: Le printemps, le printemps, le printemps... ... l'hiver s'essoufle.

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#3 by francois » 23 Jan 2014, 11:51

Thanks VickorNova, I will surely try this one this coming weekend. :D
Prendre son temps, profiter de celui qui passe.

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#4 by brokenman » 23 Jan 2014, 14:07

Wow this looks like a real asset to Porteus. Looks like you spent a lot of time getting this to work. Bravo!

I have some concerns/questions regarding compatibility with PPM and USM.

1) Non slackware compliant names such as abs-2.4.4-1-x86_64.xzm will upset slackware scripts as they check for slackware names. The slackware package name standard is also used to do version checks and this will completely break. In the Unified Slackware Manager i have written in a function to skip non slackware compliant names.

2) I am not familiar with Archlinux at all so i am not sure how the AUR system affects users. How sure can one be that a user repository is trusted? For example could I start an AUR and upload malicious packages?

3) Are there any database files that would allow a program to search for and report available packages from within the program? It would be nice to do all searching and downloading without having to switch to a browser.

4) Dependency resolution is the deal breaker for USM. It was written to address this problem exactly.

5) The duplication of the file makepkg is a problem. While you can specify an absolute path to find arch makepkg many programs use 'which' to locate binary executables. I guess the arch makepkg could be renamed for porteus.

With some tweaking this could probably be integrated into Porteus without upsetting other package managers. It certainly opens up MANY more packages to users which is exactly what we need. I apologize for not taking an indepth look but time is a little scarce at the moment. I will have a further look on Sunday.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
ViktorNova
Contributor
Contributor
Posts: 33
Joined: 19 Jun 2013, 19:38
Distribution: Porteus 3.0rc1 64-bit
Location: Portland, Oregon, USA

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#5 by ViktorNova » 24 Jan 2014, 02:48

I'm glad this has stirred some interest! =D
These are good concerns and good things to be talking about, I'll share a little more of my experience.

The binaries and instructions I posted are just the first way I tried that has worked consistently, but I'm hoping for this just to be a starting point - there is lots of room to improve, and I think integration into PPM or UPM with full dependency resolution is very realistic!

The reason I chose these specific tools is they are for one simple, and are also able to do the process of repo syncing/downloading/patching/compiling/packaging while keeping the process totally separate from actual package management, and it works great doing it by hand this way, it just takes a little interaction. That said, I'm pretty sure there are existing tools in Arch that would suit our needs better then this and make integrating it into one of our package managers more realistic. I'm testing different ones, and will report back with progress. PBfetch is looking very promising! (It's also 100% bash so very easy to tweak)
https://github.com/dalingrin/pbfetch/bl ... er/pbfetch

@fathom (thanks for the bump!)
I've build quite a few libs as well as apps, but ONLY when that lib doesn't exist in Porteus/Slackware (I check first, in that order) If it's something important like python3 or jack, I will check Alien and SBo too. I've had zero issues with libs cause I'm careful, but could definitely see how a user could accidentally bork their system. Ideally if this was wrapped into a gui, it would search Porteus/Slackware first and warn the user if the lib was available from there first

@brokenman
1) Good point. This is an easy one, I'll make sure xzm modules are named properly once I find a way to automatically create them after compiling a package

2) The AUR is one giant build script repository that is a similar idea to SBo, except maintained very actively by the community and completely massive. There is a big amount of community involvement, for instance people vote packages up, flag packages out of date, etc., and every package has a comments section beneath it and most have semi active discussions going on. Although the community does an amazing job, you are strongly encouraged to examine the PKGBUILD (Arch's version of a .Slackbuild) before building, and many of the helper apps make you look at it before proceeding. If you uploaded malicious code, it would be discovered QUICK

3) Yes! All the Arch tools I have used search the online database rather than a local one (probably because a there's a new package every few minutes) I don't think you can do this yet with the modules I made, that is, searching Arch proper as well as the AUR for a package and returning a list of matching packages with descriptions, but pbfetch does this beautifully, I will try to put an xzm of it up tonight

4) I think this will be possible with a small amount of effort! PKGBUILDS list their dependencies like this:

Code: Select all

depends=('ncurses' 'sh')
or with required versions:

Code: Select all

depends=('foobar>=1.8.0')
or

Code: Select all

depends=('foobar>=1.8.0' 'foobar<2.0.0')
There is also a separate section for build dependencies and optional ones too.
Is that enough info for PPM/USM to figure the rest out?

5) Agreed, makepkg will need to be renamed and any helper scripts modified to use the new name before this thing gets any kind of major use

Have a quick look at a these pages and a sample PKGBUILD to get an idea of how they do it
https://wiki.archlinux.org/index.php/PKGBUILD
https://projects.archlinux.org/svntogit ... kages/nano
https://wiki.archlinux.org/index.php/AU ... Guidelines

roelof
Samurai
Samurai
Posts: 112
Joined: 06 Aug 2013, 15:32
Distribution: Porteus 2.1 Gnome
Location: Netherlands

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#6 by roelof » 24 Jan 2014, 12:28

If I understand it well this tool does not have dependency tracking ?

Still I think it can be a good tool to port my favourite DE (Cinnamon) to Porteus.
How does this tool deal with the fact that Arch uses Systemd and Porteus does not ?

Roelof

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#7 by brokenman » 24 Jan 2014, 13:40

Thanks VictorNova, you have clearly explained all questions I had and with this information it appears it may be a viable option for inclusion into Porteus.
How do i become super user?
Wear your underpants on the outside and put on a cape.

roelof
Samurai
Samurai
Posts: 112
Joined: 06 Aug 2013, 15:32
Distribution: Porteus 2.1 Gnome
Location: Netherlands

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#8 by roelof » 25 Jan 2014, 08:19

Thanks.

The only thing that I wonder is how the tool deals with systemd.

If I try to port Cinnamon this way Im sure systemd is pulled in.
And I do not know if systemd and sysVinit that Porteus is using now can work both on a system.

Maybe I can try it out ?


Roelof

User avatar
ViktorNova
Contributor
Contributor
Posts: 33
Joined: 19 Jun 2013, 19:38
Distribution: Porteus 3.0rc1 64-bit
Location: Portland, Oregon, USA

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#9 by ViktorNova » 25 Jan 2014, 08:44

@phhpro
I just tried libarchive myself and got the same result. Not super surprising though, since libarchive is kind of a core library that depends on a few other core libraries..my guess is the version of libarchive that Arch uses depends on some libs or minimum versions of libs that don't exist on your system (or Slackware proper). Slackware does have it's own libarchive though, so your best bet is to install the Slack version and try again. It's been my experience that most apps (but not all!) don't care what version of a lib you have, as long as the lib/headers are found.

If/when building Arch source packages gets build into a dependency resolving package manager, this type of situation would be a cool thing to automatically resolve--installing deps from Porteus/Slackware first before checking ABS/AUR

@roelof
Right now all this does is download the source code (I believe according to Arch philosophy that it always downloads thse vanilla source), applies any Arch-specific patches, if any (which I've found is about a 50/50 chance that the patches do/don't work for Slackware/Porteus), compiles it, and installs it into a temporary directory, so no dependency tracking at the moment.

As far as systemd goes, at least right now I think 99% of Linux source code doesn't depend on it's existence to compile, (with the exception of the latest Gnome and of course systemd-related tools, to my knowledge..not sure where Cinnamon falls though). Since 'the Arch way' is to download vanilla source code and then apply Arch-specific patches, I'd assume any systemd requirements would be applied with those patches 99% of the time. So if this happens in a package you're building, I would edit the PKGBUILD and comment out any lines that begin with 'patch' or any extra 'install' files that look systemd related, and then try again. Depending on the PKGBUILD, you might have to run makepkg with "makepkg -d --skipinteg" too

@brokenman
That is encouraging to hear! I am working on a way to make the process more seamless and will keep updating with progress. I'm not really a programmer, but have done tons of packaging and scripting and I'll be able to at least produce something that is easy to wrap into a 'mother program'. Do you have any other specifics I should keep in mind to make the process more PPM/USM friendly? The next wave of this is starting to look something like
1: Query the online database for existing source packages in both Arch proper & AUR
2: Receiving a list of the package names, with versions and descriptions
3: User selects the package
4: PKGBUILD is downloaded, and optionally edited
5: Source is compiled, patched, and packaged into an xzm

roelof
Samurai
Samurai
Posts: 112
Joined: 06 Aug 2013, 15:32
Distribution: Porteus 2.1 Gnome
Location: Netherlands

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#10 by roelof » 25 Jan 2014, 09:08

Now im confused.

First you said it does have dependency tracking if I do not use the d switch.
Now you saying it does not have.

On Cinnamon systemd is enabled by configure parameters. I can change them so disable but that means I have to use consolekit and friends to make it work. So I have to rewrite the Pkgbuild for a moment and make sure that the right services are started at boot time.
At my knowlegde systemd makes this a lot easier.

Roelof

User avatar
ViktorNova
Contributor
Contributor
Posts: 33
Joined: 19 Jun 2013, 19:38
Distribution: Porteus 3.0rc1 64-bit
Location: Portland, Oregon, USA

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#11 by ViktorNova » 25 Jan 2014, 09:26

Sorry for the confusion. This is in the very beginning stage and right now the only way it works is by completely disabling dependency resolution (as this currently can only check for binary dependencies installed from Arch's package manager, which we do not use).

Just as with Slackbuilds, you have to take care of any dependencies on your own, we're only starting to discuss talking about how proper dependency resolution might be possible in the future

roelof
Samurai
Samurai
Posts: 112
Joined: 06 Aug 2013, 15:32
Distribution: Porteus 2.1 Gnome
Location: Netherlands

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#12 by roelof » 25 Jan 2014, 09:30

oke,

No problem.
Do I get a message when something is missing then ?

Roelof

roelof
Samurai
Samurai
Posts: 112
Joined: 06 Aug 2013, 15:32
Distribution: Porteus 2.1 Gnome
Location: Netherlands

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#13 by roelof » 25 Jan 2014, 14:08

I tried to build systemd as a try as a normal user.
It worked very well but at the end I see this output:

Making install in docs/libudev
make[3]: Nothing to be done for `install-exec-am'.
/usr/bin/ginstall -c -m 644 ./html/api-index-full.html
/usr/bin/ginstall -c -m 644 ./html/ch01.html
/usr/bin/ginstall -c -m 644 ./html/home.png
/usr/bin/ginstall -c -m 644 ./html/index.html
/usr/bin/ginstall -c -m 644 ./html/index.sgml
/usr/bin/ginstall -c -m 644 ./html/left.png
/usr/bin/ginstall -c -m 644 ./html/libudev-udev-device.html
/usr/bin/ginstall -c -m 644 ./html/libudev-udev-enumerate.html
/usr/bin/ginstall -c -m 644 ./html/libudev-udev-hwdb.html
/usr/bin/ginstall -c -m 644 ./html/libudev-udev-list.html
/usr/bin/ginstall -c -m 644 ./html/libudev-udev-monitor.html
/usr/bin/ginstall -c -m 644 ./html/libudev-udev-queue.html
/usr/bin/ginstall -c -m 644 ./html/libudev-udev-util.html
/usr/bin/ginstall -c -m 644 ./html/libudev-udev.html
/usr/bin/ginstall -c -m 644 ./html/libudev.devhelp2
/usr/bin/ginstall -c -m 644 ./html/right.png
/usr/bin/ginstall -c -m 644 ./html/style.css
/usr/bin/ginstall -c -m 644 ./html/up.png
Cannot open /usr/share/gtk-doc/html: No such file or directory
Making install in docs/gudev
make[3]: Nothing to be done for `install-exec-am'.
/usr/bin/ginstall -c -m 644 ./html/GUdevClient.html
/usr/bin/ginstall -c -m 644 ./html/GUdevDevice.html
/usr/bin/ginstall -c -m 644 ./html/GUdevEnumerator.html
/usr/bin/ginstall -c -m 644 ./html/annotation-glossary.html
/usr/bin/ginstall -c -m 644 ./html/api-index-deprecated.html
/usr/bin/ginstall -c -m 644 ./html/api-index-full.html
/usr/bin/ginstall -c -m 644 ./html/gudev-hierarchy.html
/usr/bin/ginstall -c -m 644 ./html/gudev.devhelp2
/usr/bin/ginstall -c -m 644 ./html/home.png
/usr/bin/ginstall -c -m 644 ./html/index.html
/usr/bin/ginstall -c -m 644 ./html/index.sgml
/usr/bin/ginstall -c -m 644 ./html/ix02.html
/usr/bin/ginstall -c -m 644 ./html/left.png
/usr/bin/ginstall -c -m 644 ./html/ref-API.html
/usr/bin/ginstall -c -m 644 ./html/right.png
/usr/bin/ginstall -c -m 644 ./html/style.css
/usr/bin/ginstall -c -m 644 ./html/up.png
Cannot open /usr/share/gtk-doc/html: No such file or directory
make: Leaving directory `/var/abs/core/systemd/src/systemd-208'
chown: invalid group: ‘root:systemd-journal

Roelof

roelof
Samurai
Samurai
Posts: 112
Joined: 06 Aug 2013, 15:32
Distribution: Porteus 2.1 Gnome
Location: Netherlands

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#14 by roelof » 25 Jan 2014, 19:29

Python fails with this message :

[371/371/2] test_zlib
340 tests OK.
2 tests failed:
test_strptime test_time
2 tests altered the execution environment:
test_site test_urllib2_localnet
27 tests skipped:
test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp
test_codecmaps_kr test_codecmaps_tw test_curses test_dbm_ndbm
test_devpoll test_gdb test_idle test_kqueue test_msilib
test_ossaudiodev test

Roelof

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: Porteus-AUR: Building Porteus modules w/ Arch build scri

Post#15 by brokenman » 25 Jan 2014, 19:34

@roelof
Somewhere in the build script you will need to comment out the chown line since there is no group called systemd-journal in porteus. If this breaks anything you will need to add the group (groupadd) before trying to build.

As root you can type: groups to see what groups exist. The other errors are minor and caused because porteus strips most documentation.

@ViktorNova
don't care what version of a lib you have, as long as the lib/headers are found.
That's a recipe for disaster.

There is not much more I can offer you at the moment. The plan is to have USM replace PPM so that nobody has to maintain a porteus repository. No need to double handle packages this way. PPM has many requirements when building a package and infact won't deal with any packages that it didn't create/download itself.

I would need to see the database file for the repository to offer anything useful. The process (for dependency resolution) will be:

1) Download pkg
2) build pkg
3) Run ldd against all libraries and executables in the package and see what is "not found"
4) Search slackware for the mother package of these libraries (and download)
5) If not found in slackware then search slackbuilds, then arch
6) Download these dependencies and repeat this loop until everything is resolved.

This is why precompiled packages are very useful. The difficult part will be during the build. If the SLKBLD fails because it missing a build time dependency then automating anything will be difficult if not impossible. Human intervention will be needed to read the error, make a decision on what is needed and then act. Same thing applies for slackbuilds.
How do i become super user?
Wear your underpants on the outside and put on a cape.

Post Reply