wget.sh for downloading the separate Porteus modules

Here is a place for your projects which are not officially supported by the Porteus Team. For example: your own kernel patched with extra features; desktops not included in the standard ISO like Gnome; base modules that are different than the standard ISO, etc...
User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

wget.sh for downloading the separate Porteus modules

Post#1 by Rava » 19 Aug 2022, 05:01

This would be the skeleton script

Code: Select all

#!/bin/bash
#	for downloading all wanted files one after the other for maximum download speed
# use wget -c URL -O outfile if needed.
#wget -c "" ;notify-send -i info -t 0 "Downloading 001-core.xzm" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 002-xorg.xzm" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 002-xtra.xzm" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 003-gnome" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 003-kde5" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 003-lxde" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 003-lxqt.xzm" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 003-mate" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 003-openbox" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 003-xfce" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 05-devel.xzm" "finished"
#wget -c "" ;notify-send -i info -t 0 "Downloading 06-man.xzm" "finished"
echo press enter
read
As you can see, all relevant entries are disabled by default.

As example, I use https://linux.rz.rub.de/ as mirror, and only include the updated 001-core.xzm - that would then be made into this:

Code: Select all

#!/bin/bash
#	for downloading all wanted files one after the other for maximum download speed
# use wget -c URL -O outfile if needed.
wget -c "https://linux.rz.rub.de/porteus/x86_64/Porteus-v5.0/updates/core/20220723/001-core.xzm" ;notify-send -i info -t 0 "Downloading 001-core.xzm" "finished"
echo press enter
read
Now, it would be nice if one someone manages to create a script that downloads the text version of the mirror list - http://porteus.org/porteus-mirrors.txt - and like update-browser - e.g. make a download-porteus script then create a menu out of the online version of porteus-mirrors.txt like so:

Code: Select all

Choose your mirror - type the number you want to use and press enter

1 http://ftp.vim.org/ftp/os/Linux/distr/porteus/
2 http://ftp.nluug.nl/os/Linux/distr/porteus/
3 http://mirrors.dotsrc.org/porteus/
4 https://mirrors.dotsrc.org/porteus/
5 http://ftp.cc.uoc.gr/pub/linux/porteus/
6 http://www6.frugalware.org/mirrors/linux/porteus/
7 http://ftp.riken.jp/Linux/porteus/
8 http://www.mirrorservice.org/sites/dl.porteus.org/
9 http://linux.rz.rub.de/porteus/
10 http://ftpmirror1.infania.net/mirror/porteus/
11 https://mirror.yandex.ru/mirrors/porteus/

type the number associated with your chosen mirror, or press ctrl+c to abort.
?
Of course the script could use a TAB instead of a simple whitespace after the number and prior the mirror URL.

Then the script would need to ask more questions - or have the info already given via parameters - so that one can use the same parameters in the future and only use the script for choosing the mirror.

Like:
only one DE and only the main modules (= 001-core + 002-xorg + 002-xtra )
- without the kernel and without the 000-kernel module
- meaning the user wants to use her/his own kernel and 000-kernel module

or:
two DEs
the main modules
the default kernel and the default 000-kernel module
05-devel and 06-man

or
three DEs
the main modules
and 06-man

etcetera.

Since it does not make much sense in creating a download without the main modules - (= 001-core + 002-xorg + 002-xtra ) I see no reason to include a parameter for these, these should be always included.

Now, as possible parameters for the other stuff, I go with longer ones that are more memorable:

for the DEs:
-gnome -kde5 -lxde -lxqt -mate -openbox -xfce
e.g. when you want -xfce and -lxde use
download-porteus -xfce -lxde

for adding the default kernel ( vmlinuz and 000-kernel.xzm ):
-kernel

for adding 05-devel
-devel

and last but not least for adding 06-man
-man

Now back to the above examples:

Only one DE and only the main modules
could be
download-porteus -xfce (since the main modules are included by default)

Two DEs
the main modules
the default kernel and the default 000-kernel module
05-devel and 06-man
could be
download-porteus -kernel -xfce -lxde -devel -man

and finally
three DEs
the main modules
and 06-man
could be
download-porteus -xfce -lxde -kde5 -man
________________________________________________________

All the above is not that hard to code, but I struggle with this part: The issue with how could the inquiry about possible updates be handled in a failsafe way?

At the time of me writing this, two updated modules are available, the relative paths are

Code: Select all

porteus/x86_64/Porteus-v5.0/updates/core/20220723/001-core.xzm
porteus/x86_64/Porteus-v5.0/updates/core/20220723/002-xorg.xzm
So, I would need a way to test possible xzm modules with the known names in
porteus/x86_64/Porteus-v5.0/updates/core/ - without downloading them, since the script should be clever enough to only download the updates and not the original (older) version(s)

or a way to determine, when the non-core DE or 06-man module got updated - would that then be in
porteus/x86_64/Porteus-v5.0/modules
or also in
porteus/x86_64/Porteus-v5.0/updates/core/

The script needs to figure out if there is a valid folder in
porteus/x86_64/Porteus-v5.0/updates/core/

and if there is more than one, to choose the newest one. But there is the possibility that more than one updated modules exist, and in an older folder is the newest variant of one module, while one or more modules exist in the newest folder… and this is when it gets hairy. The more updates there are, the more complex it gets.
_____________________________________________________________

Example, say there also exists a (theoretically) folder by the name of
20220718
that has only one module
001-core.xzm

then the script needs to know by itself to only download
porteus/x86_64/Porteus-v5.0/updates/core/20220723/001-core.xzm
and not
porteus/x86_64/Porteus-v5.0/updates/core/20220718/001-core.xzm
and also not
porteus/x86_64/Porteus-v5.0/modules/001-core.xzm
________________________________________________________

but, say,
20220718
instead has these two modules
001-core.xzm
002-xtra.xzm


and 20220723 has these
001-core.xzm
002-xorg.xzm


then the script needs to figure out by itself that the main modules to download (to automatically choose the newest ones) are:
porteus/x86_64/Porteus-v5.0/updates/core/20220723/001-core.xzm
porteus/x86_64/Porteus-v5.0/updates/core/20220723/002-xorg.xzm
porteus/x86_64/Porteus-v5.0/updates/core/20220718/002-xtra.xzm


All that might sound trivial, but it is not so trivial when you want to code it as error-resistant as possible…

And the above example is only with 2 folders with updated modules, there could be a large number of folders with a complex assortment. Sure, since the folder name is YYYYMMDD the result could be searched for "only containing 001-core.xzm" and via sorting, choose the last entry, and that will be always the newest one.


When creating the list, first add the original files, then the updates, sort the list, but only sort the updates (else it could be due to some weird coincidence that the original files get put at the end of the list due to how the paths on the server got made, maybe by mistake…)

Since the script knows
where the originals are
and how possible update folders are named

it is possible to first add the originals, then create the list with the updates, only sort the updates list, then append the updates list at the end of the original files list.

When you search for one file name, and the server has the correct files available, there will always be a hit. If the created list with the originals and updates and all possible files (only the main modules for now) is searched for one file name - "only containing 001-core.xzm" - there will be at least one hit, and when we choose the last entry in a sorted like above described list, that entry will always be the newest file.


Your thoughts?
Cheers!
Yours Rava

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

wget.sh for downloading the separate Porteus modules

Post#2 by Ed_P » 19 Aug 2022, 06:20

Rava wrote:
19 Aug 2022, 05:01
Your thoughts?
:lol: With the amount of time you spent writing your posting you could have created the scripts. :lol:

Welcome back Rava. :)
Ed

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

wget.sh for downloading the separate Porteus modules

Post#3 by Rava » 19 Aug 2022, 07:25

Ed_P wrote:
19 Aug 2022, 06:20
:lol: With the amount of time you spent writing your posting you could have created the scripts. :lol:
No. The time it took me to write the article was the time it took me to figure out the main crucial idea how to handle the updates.

The best (as in: less errors, more efficient code) is done when you have lots of thinking and re-thinking at the beginning, and only then start coding.

Most people do it the other way around: think as little as possible, start coding right from the start as soon as they have a small idea how something could be done, and the code gets messier and messier (worst example: SM-DOS and SM-Witless) the more they code, because they had not the best ideas to begin with.

Like in the above example.

I first had two different approaches at the end, and both were prone to failure.

Only then I started thinking of another way of doing things and realized when I combine the original paths of the modules at the beginning and the sorted list of all existing updates at the end, and then search the one list for one file name at a time and only pick the last entry, all problems have been solved.

All other approaches I came up with first got more complicated the more possible updates will exist in the future, but that is not the case with my final idea about how to solve it.

Again the slower, yet more thoughtful approach turned out to be the better one in the end. Usually when it comes to coding it is like that more than not. Image

And to write it out like so could hopefully be an inspiration for others to follow a similar approach. Me just being silent about it and telling the "think first, think a lot, re-think, and only then start coding" would be less convincing otherwise, at least in my book. :)

But, to come back to your quote above, my approach saves much time and headache in the overall time span of a larger project. And I consider the above script as one of my larger ones. (Not necessary in terms of lines of code or overall bytes of the script, but complexity of what is to be done.)
Cheers!
Yours Rava

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 3924
Joined: 20 Mar 2012, 03:42
Distribution: v5.0-64bit
Location: australia
Contact:

wget.sh for downloading the separate Porteus modules

Post#4 by ncmprhnsbl » 20 Aug 2022, 00:36

you appear to be trying to reinvent the wheel.
most of what you mention is already covered various existing scripts in porteus.
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

wget.sh for downloading the separate Porteus modules

Post#5 by Rava » 20 Aug 2022, 15:53

ncmprhnsbl wrote:
20 Aug 2022, 00:36
most of what you mention is already covered various existing scripts in porteus.
Can you point them out to me?
Cheers!
Yours Rava

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

wget.sh for downloading the separate Porteus modules

Post#6 by Ed_P » 20 Aug 2022, 17:56

The Porteus Settings Center supports the Porteus Updater which updates "your modules folder". Whether it supports updating what's in the /porteus/base, /boot/syslinux and /EFI/boot folders I don't know, but it "will check for Porteus updates on the server" so it might do what you want, with a minor tweak or 2 that you could do Rava.
Ed

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

wget.sh for downloading the separate Porteus modules

Post#7 by Rava » 21 Aug 2022, 16:18

^
When it is does via a script language I am familiar with. When it's a compiled program (e.g. via C or C++ or such) then no chance.
I look into it.
Cheers!
Yours Rava

Post Reply