build xzm from AUR

[vinnie]

Okay, I already did, I couldn't resist....
I tried trizen and I was able to install the packages, although I don't really know how I did it
In these two messages I had received insights: 1 2
In fact I didn't exactly follow them but
As usual I want to keep track of what I've done, so that it's easier for me to make mental order.

1)I downloaded and activated the 05-devel.xzm module.
2)I created a workdir
$ mkdir -p /mnt/sdb4/aur/sdrpp
$ cd /mnt/sdb4/aur/sdrpp
3)I believe this downloaded the necessary PKBUILDs from AUR in the workdir
$ trizen -Gd sdrpp-git
4)and this compiled the PKBUILDs, installed the dependencies from the repository, and finally installed in the system the program I tried to build
$ trizen -Sl sdrpp-git
5)start the program
$ MESA_GL_VERSION_OVERRIDE=3.0 sdrpp
or also in the /usr/share/applications/sdrpp.desktop edit "Exec=env MESA_GL_VERSION_OVERRIDE=3.0 /usr/bin/sdrpp" (for the notes of those who have problems with opengl version like me)

During step 4 I think (I'm not sure) the arch packages were created:
find -iname "*.zst*"
./libsdrplay/libsdrplay-3.15.2-2-x86_64.pkg.tar.zst
./sdrpp-git/sdrpp-git-1.0.4.r714.e81db5d8-1-x86_64.pkg.tar.zst
./airspyhf-git/airspyhf-git-r124.39f06a4-1-x86_64.pkg.tar.zst

and the packages that pacman downloaded should be in /mnt/live/memory/changes/var/cache/pacman/pkg/ (I think it's less than the ones actually needed, so I'll have to repeat the procedure without any pre-activated modules).
I don't really like this behavior because since I wasn't using any "changes" directory, I think these packages were all installed in ram, so I could have run out of ram without being able to finish the procedure (if you know a command to do everything in the workdir, it would be helpful to me).

Based on the advice I've been given untill now, I shouldn't unpack all the repository packages into a dir (or maybe convert them with "pkg2xzm" and unpack them with "xzm2dir") and then have the package created with "dir2xzm" (maybe after removing the documentation).
This is currently the only procedure I would know how to do.
Whether it is possible to merge the xzm packages all together (it seems to me that the script is not there on nemesis), or install everything in a separate root and then just create the package for this root, I don't know.
However, it should work, so I'm getting close

[beny]

hi i use yay for all the arch systems you can search the packages that the aur helper build on porteus/changes/home/guest/.cache you have the directory of the software and the packages in tar.zst the helper install the packages like a real packages manager so you have it on changes directory,when you use the helper in terminal,you can search software with yay, name of sotfware, or maybe trizen can do this task too,try if it give you an index list

[vinnie]

So it's as I assumed, once I've managed to collect all the necessary tar.zst packages, I just need to merge them and convert them to xzm.
Since I assume pmod does the same (obviously not including the compilation of the aur packages), isn't it possible to pass the packages to be merged to pmod or some other script to do everything automatically?
I want to see how pmod (pman) does it, maybe I can use that system.
Anyway for now for me to use yay or trizen makes no difference since I don't know either of them, although trizen being already provided seems more straightforward.

[ncmprhnsbl]

isn't it possible to pass the packages to be merged to pmod

havn't tried it, but i would guess that by putting all the packages in /var/cache/pacman/pkg and running pmod "the_princple_package" might work .. but it might just give database errors..
the manual way is:
assume pwd is $wrkdir and your packages are in PKGS there
# mkdir -p fakeroot
# cp -a --parents /var/lib/pacman/local/ALPM_DB_VERSION fakeroot
# pacman -Uddr fakeroot PKGS/*.pkg.tar.zst ## U = upgrade, dd = no dep test, r = path to alternative root (you'll see complaints about missing database files and post install operations fail < safe to ignore)
# rm fakeroot/var/lib/pacman/local/ALPM_DB_VERSION ##don't need this in a module and manually remove or configure how you want eg. removing headers and locales etc..
# dir2xzm fakeroot new_name_of_module.xzm

more or less..

[vinnie]

I was just checking the pman script and that pacman command.
Now I want to try it, first if it works directly using pmod (that would be great, the stripmodule function would do the dirty work for me).
If it doesn't then maybe I'll try the second procedure you gave me.
I will make updates after the attempts

P.s. although I don't know bash so well to understand all the commands, pman is very well commented and I can understand the script too, great job!

[vinnie]

Okay, I wanted to make a recap message today, I had assembled everything into one script and was trying to restart it to build the package one last time.
However, I ran into a problem, it seems that packages are missing in the repository (of aur/artix).
Until a few days ago these packages were there since I was able to download them, so I thought it was a repository update issue and tried this command which seems to be successful:

Code: Select all

[porteus ~]# pmod -y
:: Synchronizing package databases...
 system is up to date
 world is up to date
 galaxy is up to date
 extra is up to date

But it seems that the problem remains, here the final part of the output.

Code: Select all

==> Installing missing dependencies...
[sudo] password for root: 
resolving dependencies...
looking for conflicting packages...

Package (45)                 New Version       Net Change  Download Size

world/double-conversion      3.3.0-1             0.18 MiB       0.06 MiB
world/fltk                   1.3.9-1             6.38 MiB       1.55 MiB
world/gd                     2.3.3-7             0.67 MiB       0.15 MiB
world/gnuplot                6.0.1-1             3.53 MiB       1.27 MiB
world/gssdp                  1.6.3-1             0.24 MiB       0.06 MiB
world/gst-plugins-bad-libs   1.24.3-1           11.71 MiB       2.69 MiB
world/gst-plugins-base-libs  1.24.3-1           12.02 MiB       2.24 MiB
world/gstreamer              1.24.3-1           11.14 MiB       2.00 MiB
world/gupnp                  1:1.6.6-1           0.70 MiB       0.12 MiB
world/gupnp-igd              1.6.0-1             0.11 MiB       0.03 MiB
world/libavif                1.0.4-3             0.39 MiB       0.14 MiB
world/libb2                  0.98.1-2            0.12 MiB       0.04 MiB
world/libcaca                0.99.beta20-4       2.53 MiB       0.41 MiB
world/libcerf                1:2.4-1             0.12 MiB       0.05 MiB
world/libde265               1.0.15-1            0.76 MiB       0.26 MiB
world/libheif                1.17.6-5            1.17 MiB       0.36 MiB
world/libmspack              1:1.11-1            0.16 MiB       0.05 MiB
world/libnice                0.1.22-1            1.84 MiB       0.31 MiB
extra/libserialport          0.1.1-5             0.12 MiB       0.03 MiB
world/libyuv                 r2426+464c51a0-1    1.85 MiB       0.30 MiB
world/md4c                   0.5.2-1             0.20 MiB       0.07 MiB
world/qt6-5compat            6.7.1-1             1.64 MiB       0.44 MiB
world/qt6-base               6.7.1-4            56.39 MiB               
world/qt6-shadertools        6.7.1-1             4.46 MiB       1.46 MiB
world/qt6-svg                6.7.1-1             0.79 MiB       0.22 MiB
world/qt6-translations       6.7.1-1.4          13.65 MiB               
world/tslib                  1.23-1              0.47 MiB       0.10 MiB
world/vulkan-headers         1:1.3.285-1        26.13 MiB       1.21 MiB
world/wxwidgets-common       3.2.5-1            15.33 MiB               
world/wxwidgets-gtk3         3.2.5-1            17.07 MiB               
world/xcb-util-cursor        0.1.5-1             0.03 MiB       0.01 MiB
world/xcb-util-keysyms       0.4.1-4             0.02 MiB       0.01 MiB
world/xcb-util-renderutil    0.3.10-1            0.03 MiB       0.01 MiB
world/xcb-util-wm            0.4.2-1             0.21 MiB       0.03 MiB
extra/airspy                 1.0.10-1            0.18 MiB       0.04 MiB
extra/bladerf                2023.02-2          23.76 MiB               
extra/codec2                 1:1.2.0-1           1.84 MiB       0.69 MiB
extra/hackrf                 2024.02.1-1         0.58 MiB       0.13 MiB
extra/libad9361              0.3-1               0.19 MiB       0.08 MiB
extra/libiio                 0.25-3              0.51 MiB       0.15 MiB
extra/limesuite              23.11.0-3           6.84 MiB               
world/portaudio              1:19.7.0-2          0.33 MiB       0.10 MiB
world/rtaudio                6.0.1-1             0.18 MiB       0.06 MiB
world/rtl-sdr                1:2.0.1-3           0.29 MiB       0.09 MiB
extra/soapysdr               0.8.1-5             5.22 MiB       0.85 MiB

Total Download Size:    17.86 MiB
Total Installed Size:  232.06 MiB

:: Proceed with installation? [Y/n] Y
:: Retrieving packages...
 gst-plugins-bad-libs-1.24.3-1-x86_64.pkg.tar.zst failed to download
 gst-plugins-base-libs-1.24.3-1-x86_64.pkg.tar.zst failed to download
 gstreamer-1.24.3-1-x86_64.pkg.tar.zst failed to download
 qt6-shadertools-...  1498.2 KiB   910 KiB/s 00:02 [######################] 100%
 fltk-1.3.9-1-x86_64  1588.7 KiB   687 KiB/s 00:02 [######################] 100%
 Total ( 5/39)          17.9 MiB  7.51 MiB/s 00:02 [######################] 100%
error: failed retrieving file 'gst-plugins-bad-libs-1.24.3-1-x86_64.pkg.tar.zst' from archive.artixlinux.org : The requested URL returned error: 404
error: failed retrieving file 'gst-plugins-base-libs-1.24.3-1-x86_64.pkg.tar.zst' from archive.artixlinux.org : The requested URL returned error: 404
error: failed retrieving file 'gstreamer-1.24.3-1-x86_64.pkg.tar.zst' from archive.artixlinux.org : The requested URL returned error: 404
warning: too many errors from archive.artixlinux.org, skipping for the remainder of this transaction
warning: failed to retrieve some files
error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.
==> ERROR: 'pacman' failed to install missing dependencies.
==> Missing dependencies:
  -> airspy
  -> bladerf
  -> codec2
  -> hackrf
  -> libad9361
  -> libiio
  -> limesuite
  -> rtaudio
  -> rtl-sdr
  -> portaudio
  -> soapysdr
==> ERROR: Could not resolve all dependencies.
:: Unable to build sdrpp-git - makepkg exited with code: 8

Coincidentally, today I had not even been able to install audacity with pmod, and it seems to me that common packages are missing.

How can this be solved?


update
I tried to do an "always fresh" boot and restart "setup-pman".
Then I tried "pmod audacity" but those packages are still missing, so it's not a repository update problem:

Code: Select all

pmod audacity
resolving dependencies...

Package (21)                 New Version  Net Change  Download Size

world/gssdp                  1.6.3-1        0.24 MiB       0.06 MiB
world/gst-plugins-bad-libs   1.24.3-1      11.71 MiB       2.69 MiB
world/gst-plugins-base-libs  1.24.3-1      12.02 MiB       2.24 MiB
world/gstreamer              1.24.3-1      11.14 MiB       2.00 MiB
world/gupnp                  1:1.6.6-1      0.70 MiB       0.12 MiB
world/gupnp-igd              1.6.0-1        0.11 MiB       0.03 MiB
world/libmad                 0.15.1b-10     0.16 MiB       0.06 MiB
world/libmspack              1:1.11-1       0.16 MiB       0.05 MiB
world/libnice                0.1.22-1       1.84 MiB       0.31 MiB
world/libsbsms               2.3.0-5        0.22 MiB       0.10 MiB
world/opusfile               0.12-3         0.20 MiB       0.06 MiB
world/portaudio              1:19.7.0-2     0.33 MiB       0.10 MiB
world/portmidi               1:2.0.4-1      0.12 MiB       0.04 MiB
world/portsmf                234-2          0.20 MiB       0.07 MiB
world/rapidjson              1.1.0-6        0.62 MiB       0.10 MiB
world/soundtouch             2.3.3-1        0.21 MiB       0.07 MiB
world/suil                   0.10.20-1      0.11 MiB       0.02 MiB
world/vamp-plugin-sdk        2.10.0-2       0.85 MiB       0.24 MiB
world/wxwidgets-common       3.2.5-1       15.33 MiB       2.83 MiB
world/wxwidgets-gtk3         3.2.5-1       17.07 MiB       4.62 MiB
world/audacity               1:3.5.1-3     36.90 MiB       8.49 MiB

Total Download Size:  24.30 MiB

:: Proceed with download? [Y/n] Y
:: Retrieving packages...
 wxwidgets-gtk3-3...     4.6 MiB  1201 KiB/s 00:04 [######################] 100%
 wxwidgets-common...     2.8 MiB   542 KiB/s 00:05 [######################] 100%
 audacity-1:3.5.1...     8.5 MiB  1074 KiB/s 00:08 [######################] 100%
 gst-plugins-bad-libs-1.24.3-1-x86_64.pkg.tar.zst failed to download
 gst-plugins-base-libs-1.24.3-1-x86_64.pkg.tar.zst failed to download
 Total ( 5/21)          24.3 MiB  2.98 MiB/s 00:08 [######################] 100%
error: failed retrieving file 'gst-plugins-base-libs-1.24.3-1-x86_64.pkg.tar.zst' from archive.artixlinux.org : The requested URL returned error: 404
error: failed retrieving file 'gst-plugins-bad-libs-1.24.3-1-x86_64.pkg.tar.zst' from archive.artixlinux.org : The requested URL returned error: 404
warning: failed to retrieve some files
error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.

gst-plugins-base and -bad I remembered correctly, these are the same packages that give problems with sdrpp.
looking on the aur and artix repositories though, they don't seem to have been updated these days, so I don't understand what the problem is:
https://archlinux.org/packages/extra/x8 ... -bad-libs/
https://packages.artixlinux.org/details ... s-bad-libs
https://archlinux.org/packages/extra/x8 ... base-libs/
https://packages.artixlinux.org/details ... -base-libs

[ncmprhnsbl]

looks like there's a problem with archive.artixlinux.org/ for it seems at least the gstreamer packages..
fortunately, the recently made available north american mirror doesn't seem to have this problem: archive2.artixlinux.org
so.. if you edit /etc/pacman.d/mirrorlist:

Code: Select all

Server = https://archive.artixlinux.org/repos/2024/06/17/$repo/os/$arch

to

Code: Select all

Server = https://archive2.artixlinux.org/repos/2024/06/17/$repo/os/$arch

then resync (pman -Sy (or even -Syy) (or rerun setup-pman if you need a new pacman-settings.xzm)
you should be good to go.

[vinnie]

ok, thanks, now to do the test quickly I tried to build the audacity module and it went without any problems.
At next reboot I will try to rebuild sdr++ (I want to do it in "always fresh" mode but now I can't reboot) .
Anyway this problem with the two modules is a fairly recent thing because precisely the first time I had tried to build the sdr++ package the download had worked, so who knows what happened

[ncmprhnsbl]

"--cachedir <dir> set an alternate package cache location" might be useful too, to avoid using /var/cache/pacman/pkg, as well as keeping packages for future rebuilds or update.
the other thing to mention for aur builds is to differentiate between 'make' deps and 'runtime' deps, (can seen in the PKGBUILD) because you'll not want the make deps in your module.
some future script could grep the PKGBUILDs for this..

[vinnie]

okay, i need to understand something though, i saw that --cachedir is a parameter of trizen that refers to pacman
So I am thinking that trizen download the arch repository packages in the indicated dir, but what it does with the ones built by aur?
Ok, experments are worth a thousand words, I'll try now
(p.s. i changed the connection to usb tethering, it works out off the box)

[ncmprhnsbl]

but what it does with the ones built by aur?

they're usually dumped in the directory where the PKGBUILD is.. this can be defined in ~./config/trizen/trizen.conf file , i believe. param: clone_dir (and other things)
i see there now:
movepkg => 0, # bool -- Move built packages in the directory `movepkg_dir`.
movepkg_dir => "/var/cache/pacman/pkg", # str -- Absolute path to the directory where to move built packages (with `movepkg`).
so:
movepkg ==> 1,
movepkg_dir ==>"same/dir/as/--cachedir=/someplace ##not literally of course :p
should get all the packages in one place.

[vinnie]

"--cachedir <dir> set an alternate package cache location" might be useful too, to avoid using /var/cache/pacman/pkg, as well as keeping packages for future rebuilds or update.

I tried "--cachedir" specified as a trizen parameter, but it didn't work, it didn't return any error but pacman packages ended up in the usual directory in var

the other thing to mention for aur builds is to differentiate between 'make' deps and 'runtime' deps, (can seen in the PKGBUILD) because you'll not want the make deps in your module.
some future script could grep the PKGBUILDs for this..

right now I tried to reuse your pman code to strip the module, it took out a lot of things but of course you're absolutely right, along with the package travels a lot of things that should be removed. With sdrpp there's even an fltk interface editor and some other software that I don't even understand what it's for.
However it is also understandable, they are packages converted from another distro and among other things from aur, it could be very nice to grep the PKGBUILD, anyway porteus modules without any removal are already one fifth the size of the program installed on arch!

meanwhile I did several attempts until my list of commands didn't work and in the end I went long, however now it works.
the things you said in your last message however I still have to process them

I'm inserting the list of commands (I don't even feel like calling it a script XD), it's for the record, so if anyone wants to know what operations to do can consult it as a reminder.
The script has to be launched from user because it seems that trizen wants to be launched from user.

Code: Select all

#!/bin/bash

#I capture the password to some reuse
read -s -p 'Password: ' password
echo ${password} | su -l -c "printf ''" &>/dev/null || exit 0
echo

#package name, module name and work subdir (sorry for the italian words)
export programma="sdrpp-git"
export pacchetto="${programma}.xzm"
export workdir="/mnt/sdb4/aur/${programma}"
#cachedir="${workdir}/pkgcache" #cachedir that not seems to work in trizen

#we need pman module settings
if ! $(ls /mnt/live/memory/images | grep pacman-settings -q); then
  echo ${password} | su -l -c "activate -q /mnt/sdb3/porteus/modules/pacman-settings.xzm"
fi

#we need devel module
if ! [ -x "$(command -v trizen)" ]; then
  echo ${password} | su -l -c "activate -q /mnt/sdb3/porteus/optional/05-devel.xzm"
fi


#create workdir
sudo rm -rf "${workdir}" && sleep 1
mkdir -p "${workdir}" #"${cachedir}" 
cd "${workdir}"

#install package from aur to the system
echo ${password} | su -l -c "rm /mnt/live/memory/changes/var/cache/pacman/pkg/*" && sleep 1
trizen  -Gd  "${programma}" && sleep 1 #--cachedir="${cachedir}"
trizen -Sl "${programma}" && sleep 1 #--cachedir="${cachedir}" 


#module creation (I included the functions taken from pman script to strip the module)
echo "${password}" | sudo -s --preserve-env  <<'EOF'
## Source some useful functions
. gettext.sh
. /usr/share/porteus/porteus-functions
get_colors


##############
# strip_module
# report a fatal error and exit
##############
strip_module(){
echo
bold "`eval_gettext "Size before strip:"`"; echo " `du -sh $1 | awk '{print$1}'`"
cfg=/etc/porteus.d/modules

## Remove package data files
#rm $UNP/{.MTREE,.PKGINFO,.INSTALL} 2>/dev/null

## Remove APLM_DB_VERSION and sync dbs
rm -f $1/var/lib/pacman/local/ALPM_DB_VERSION
rm -f $1/var/lib/pacman/sync

## Strip locales
if [ -d $1/usr/share/locale ]; then
PLOC=$(awk -F= '/^LocalesP/{print$NF}' $cfg | tr ':' '|')
find $1/usr/share/locale/* -maxdepth 0 -type d | grep -Ev "$PLOC" | xargs -i rm -r {}
fi
if [ -d $1/usr/share/i18n/locales ];  then
find $1/usr/share/i18n/locales/* -maxdepth 0 -type d | grep -Ev "$PLOC" | xargs -i rm -r {}
fi

## Strip man files
if [ "$ManPreserve" != "yes" ]; then
	rm -rf $1/usr/share/man/*
fi

## Strip headers
[ -d $1/usr/include ] && rm -rf $1/usr/include/* 2>/dev/null

## Strip docs
for a in $1/usr/share/doc $1/usr/share/gtk-doc; do
	[ -d $a ] && rm -rf $a/*
done

## Strip info dir
[ $1/usr/share/info ] && rm -rf $1/usr/share/info/*

## Fix weird var permissions
chmod -R 755 $1/var

bold "`gettext "Size after strip:"`"; echo " `du -sh $1 | awk '{print$1}'`"

}


#here the creation, all this command need to be runned by root
#rm -rf ./pkg ./fakeroot "${pacchetto}"  && sleep 1
cp -a /mnt/live/memory/changes/var/cache/pacman/pkg/ ./ && sleep 1
mkdir -p ./fakeroot
cp -a --parents /var/lib/pacman/local/ALPM_DB_VERSION ./fakeroot
pacman -Uddr ./fakeroot $(find -iname "*.pkg.tar.zst" | tr  "\n" " ") --noconfirm
strip_module ./fakeroot
dir2xzm ./fakeroot ${pacchetto}
</dev/tty
EOF

[ncmprhnsbl]

I tried "--cachedir" specified as a trizen parameter, but it didn't work, it didn't return any error but pacman packages ended up in the usual directory in var

another thing to look at is /etc/pacman.conf , among other things, you can set the cachedir there too:

Code: Select all

#CacheDir    = /var/cache/pacman/pkg/

a bit clunky for a script, but might be something to add the /config/modules file
come to think of it a lot of the stripping could be handled this way too, with the NoExtract=/some/file/you/do/not/want flag.
and while i'm mentioning configs, check out /etc/makepkg.conf where you can set all sorts more technical stuff like build flags ..etc..

..and, just to throw yet another slightly related thing: the chaotic repository, of prebuilt AUR packages : https://wiki.archlinux.org/title/Unoffi ... haotic-aur
requiring adding to /etc/pacman.conf, receiving/updating the keys..
warning: these packages are built with stock arch which may have dep issues because:
1.systemd
2.may have rolled too far away from nemesis build point (lessened if used promptly at or close to the update point)
same goes for any of the other unofficial repos on that page.

[M. Eerie]

but what it does with the ones built by aur

I believe they are stored in ~/.cache/trizen since trizen (as well as yay) is not invoked as root user

[ncmprhnsbl]

I believe they are stored in ~/.cache/trizen

yeah that's the default, but can be set to anywhere in ~/.config/trizen/trizen.conf (clone_dir)