Updates Folders Organisation (suggestion)

[robang74]

This is one mirror among many others, and all are presenting the same folder structure (as expected being mirrors)

http://www6.frugalware.org/mirrors/linu ... ates/core/

From a human point of having a component update 20250130/001-core.xzm is nicer than 001-core-20250130.xzm

Code: Select all

Index of /mirrors/linux/porteus/x86_64/current/updates/core/
../
20230930/                                          03-Oct-2023 10:52       -
20231228/                                          30-Dec-2023 06:36       -
20240229/                                          03-Mar-2024 06:16       -
20240428/                                          04-May-2024 08:01       -
20240510/                                          11-May-2024 07:36       -
20240801/                                          04-Aug-2024 03:35       -
20240923/                                          24-Sep-2024 12:31       -
20241211/                                          17-Dec-2024 16:00       -
20250130/                                          01-Feb-2025 06:48       -
updates.txt                                        01-Feb-2025 06:48    2564

Fortunately the updates.txt mitigate the "scripting" effort (aka the BOFH's WTF cast streaming). Unfortunately, it still not address at which point stop searching for updates because from name files or even the entire path, it is not known the ISO date. Which can be retrieved by other information. So, to take a decision the script needs to compare more source of information.

Is it a problem? Yes it is. How this problem has been resolved for the ISO/distro files? Adding versioning in the path *AND* creating main 3 levels: archive (old), current (latest stable), testing (on the edge).

Code: Select all

Index of /mirrors/linux/porteus/x86_64/
../
Porteus-v3.2/                                      28-Apr-2018 21:29       -
Porteus-v3.2.2/                                    28-Apr-2018 21:29       -
Porteus-v4.0/                                      16-Jun-2018 21:07       -
Porteus-v5.0/                                      05-Jul-2022 07:58       -
Porteus-v5.01/                                     20-May-2024 07:56       -
Porteus-v5.1/                                      03-Jan-2025 09:48       -
archive/                                           06-Jul-2021 18:50       -
current/                                           20-May-2024 07:56       -
kernel/                                            13-May-2018 14:17       -
testing/                                           11-Dec-2017 02:52       -

So, my suggestion is to add a folder named "current" also in every place where updates.txt exists. In that folder all the components (eg. 001-core-.xzm, 002-xorg.xzm) are linked. Obviously, this extra folder do not increase the size of the repository because the files are NOT duplicated but linked (also in Posix-compliant filesystem, links exists at filesystem-internals level).

Advantages? We can wget-ting the whole folder without have to takes decision because ALL latest updates are in THAT place. While others places are fine for humans, keep tracks, etc. Moreover, it is a simple straightforward change that change everything!

[robang74]

Code: Select all

WARNING

Checking here

- https://ftp.riken.jp/Linux/porteus/x86_ ... .0/kernel/

We can find the initrd.xz file which is not the one contained into 5.01 version and that file is not into mirrors anymore but just into the ISO. AFAIK

Instead, it should be part of the mirror as file. IMHO. The same happens for all the others files and in particular for these two:

190 /tmp/d/boot/Porteus-installer-for-Windows.exe
376 /tmp/d/boot/Porteus-installer-for-Linux.com

[Ed_P]

Porteus users simply use the Menu's Porteus Settings Centre app to Update their Porteus system modules. It checks what modules the system is running, what updates are available and allows the user to determine if they want to do the update. No searching necessary. And the Centre's app includes a Porteus Installer function.

Similar to Porteus users using the Menu's Browser Selection and Update Tool app. Easy for the Porteus user to see what's available and to choose.

[robang74]

Porteus users simply use the Menu's Porteus Settings Centre app to Update their Porteus system modules

Please consider the following:

• 1. The end-users do that, after having download an entire ISO 400Mb+ then the download 400MB+ again.

• 2. A correct organisation of a download repository has nothing to do with apps or users or whatever.

About #1, if you were paying the bill for such download - even a little amount multiplies many users would be noticeable. Free resources are not made for being wasted (cfr. The tragedy of Commons). Probably the number of your "end-users" -- hence downloads -- is negligible, anyway. Which is the reason because nobody felt the need to provide a feedback about the download structure. However, since I started to write on this forum, I have seen some mirrors suspend the service for Porteus or cutting down the bandwidth [1]... eh già...

About #2, if the software included into the package is GPLv2 (or alike) then also the sources should be available (or a text file which redirect on some specific repository or project from which is possible to download the exact version that creates the binaries and related artifacts). However, this an aspect that I did not investigated yet. From this perspective - source availability and reproducibility - at the first sight I would say that Porteus is a derived from ProteuX and not the viceversa.

Ed_P, do you have a name? Please (your name), avoid to reply my posts/comments on this forum. If something puzzle you about what I am writing, please ask in PVT to a DEV. Thanks, R.

Note

[1] The slower ones, by 20%-25% and the faster by 40%-50%, as per rule of thumb. Which is the reason because I wrote the script that find the fastest available. Which add just +0.1% (1/1000) overload on a full download but can shorten the download time by 4x in best scenario, usually by 2x on a random pickup.

Code: Select all

02-wget-log: 701 KB/s
09-wget-log: 554 KB/s
03-wget-log: 489 KB/s
11-wget-log: 474 KB/s
04-wget-log: 462 KB/s
10-wget-log: 453 KB/s
08-wget-log: 412 KB/s
05-wget-log: 405 KB/s
07-wget-log: 316 KB/s
06-wget-log: 293 KB/s

Above said, when the faster were working around 1.2MB/s. Now the ratio on a random pickup, dropped because the difference in bandwidth has been sensitivity reduced. Obviously, the faster are hit more frequently once it is possible to identify them in a reliable manner.

[Ed_P]

Please consider the following:
1. The end-users do that, after having download an entire ISO 400Mb+ then the download 400MB+ again.

Pretty sure the updates don't match every module, but I could be wrong. The ISO systems work and the user only has to download modules they need. ISOs are released by the Porteus developers and when the updates are sufficiently significant they release new, tested, ISOs.

2. A correct organisation of a download repository has nothing to do with apps or users or whatever.

Then the current organization is fine.

From this perspective - source availability and reproducibility - at the first sight I would say that Porteus is a derived from ProteuX and not the viceversa.

Nope. Wrong.

Ed_P, do you have a name? Please (your name), avoid to reply my posts/comments on this forum. If something puzzle you about what I am writing, please ask in PVT to a DEV. Thanks, R.

Forums, and this one in particular, are for sharing ideas and improving on them via discussions with other members/users. Everybody wins.

I like your new kmap script. Any chance you could make one for formating USB drives. I know you have the function built into the install script but a standalone version for users wanting to try something new it could be useful.

[robang74]

Pretty sure the updates don't match every module, but I could be wrong.

You ARE wrong:

Code: Select all

# For everybody
001-core.xzm   124M
002-xorg.xzm   112M
002-xtra.xzm    21M

# For some people, only
003-cinnamon.xzm 24M
003-kde5.xzm 83M
003-lxde.xzm  9M

• http://www6.frugalware.org/mirrors/linu ... pdates.txt

Plus in one way or another - not necessarily for all the user all the stuff - after 2024.05.20 starting from here

• http://www6.frugalware.org/mirrors/linu ... 4/current/

It is clear for me that you did not even check the mirror, because "users" uses applications hence they do not care.

Please, I insist - refer to a DEV, as far as someone of them would care - but not to me because "sharing ideas" works when people are on the same technical level.

Thanks for your comprehension, I hope the best for your quest in seeking attentions. Best regards, R-

---

POST SCRIPTUM

So a folder populated by links (or just an HTML page with the links) that refers to the latest stable ISO version

Code: Select all

mirrors/linux/porteus/x86_64/current/updates/core/current

that contains all the core's stuff that is useful to download to create an up-to-date ISO image will avoid multiple downloads from the end-users as well.

How? To install or create an ISO, it is not need to have 400+ MB of stuff but 23 MB is enough (wireless stuff apart, possibly).

• http://www.tinycorelinux.net/downloads.html

Or in a more reasonable way, using another Linux live [1] that people might have around and use a network-only installation (cfr. Debian, for example).

So, if someone was shy to ask and @Ed_P is the front-end who writes critics just to receive more information... useless. All information will be written here:

• Code: Select all

  https://github.com/robang74/porteus-usb-installer#readme

In case of doubts, Google Gemini 2, X Grok3 or ChatGPT half of the times are able to get the point. For the other half, humans still have a hope!

Note

Despite Windows cannot be considered a "Linux live" and the Porteus ISO comes with a couple of executables (.com and .exe, 1MB), those executable can be VERY useful ONLY if they can downloaded core's components separately at their latest version and able to create an USB bootable without the need to upgrade the system. As extreme ratio a minimal system with Dillo

• https://dillo-browser.github.io

can be enough to let the user choose the essential parameters (Kiosk mode, and by the way I have used Dillo in 2006 for providing transportation info to people on the bus-stations of Bolzano Province). Such a system can also work with X on framebuffer (for those few minutes that it is needed) and apart the wifi cards that are a pain because proprietary firmware and settings, everything else is quite common in the wild.

Guess what? A universally Linux compatible Wi-Fi USB dongle can be bought for $2

• https://robang74.github.io/chatbots-for ... o.html#bom

and a good-enough USB for around $12. A postage stamp to round the number to $15 and everyone is able to provide themselves a USB bootable maker Kiosk.

Under this point of view this line

Code: Select all

wget -O- $url | sudo dd bs=1M of=/dev/sdx

means exactly what is supposed to say: a remote server that creates on-demand and up-to-date ISO and the client should just have a network connection to reach the server and the permission to write a block device. Because even a 13yo laptop has enough power to create it on-the-fly, just the core components should be available separately and the client have a list of them plus the recipe to put all together.

• https://github.com/robang74/porteus-usb ... provements

Which is exactly what I am doing here: avoid that users need to cope with "technical stuff" directly but just a HTML form simple enough to be useful from a bare simple web-browser like Lynx or Dillo.

[Ed_P]

Thanks for your comprehension, I hope the best for your quest in seeking attentions. Best regards, R-

Write's the poster above.

Added in 9 minutes 12 seconds:

Under this point of view this line

Code: Select all

wget -O- $url | sudo dd bs=1M of=/dev/sdx

means exactly what is supposed to say: a remote server that creates on-demand and up-to-date ISO and the client should just have a network connection to reach the server and the permission to write a block device.

I interpret that code to mean you download an .iso file and write it directly to the /dev/sdx but does it extract the files from the .iso in the process?

[robang74]

I interpret that code to mean you download an .iso file and write it directly to the /dev/sdx but does it extract the files from the .iso in the process?

Which is EXACTLY the reason for which I wrote this:

Please, I insist - refer to a DEV

and this:

because "sharing ideas" works when people are on the same technical level.

and this sentence here:

• https://github.com/robang74/porteus-usb ... ick--dirty

If – for you – that script above did nothing, then it is ok in that way: do not use it. ;-)

and finally, the reason because I wrote my own BOFH manifesto!

[Ed_P]

I interpret that code to mean you download an .iso file and write it directly to the /dev/sdx but does it extract the files from the .iso in the process?

Which is EXACTLY the reason for which I wrote this:

robang74 wrote: ↑March 22nd, 2025, 9:10 pm
Please, I insist - refer to a DEV

I thought I had Rob. People who post code they have written are thought of that way. When I've used dd in the past to copy drives I don't recall it expanding .iso files, or .zip or .gz files.

[robang74]

I thought I had Rob. People who post code they have written are thought of that way. When I've used dd in the past to copy drives I don't recall it expanding .iso files, or .zip or .gz files.

Some people think that they switched to "Linux" because they managed to burn a ISO file on a USB stick and boot a PC with it.

Some people think that Linux is nice because Ubuntu looks eye-candy and it is also user-friendly because they can use the mouse.

Nothing of this, is my fault!

[Ed_P]

FYI Rob, one of my old dd clone drive Porteus scripts:

Code: Select all

#!/bin/sh

ID=sda    # Input device
OD=sdb    # Output device

if [ `whoami` != "root" ]; then
   echo -e "Enter root's password\033[1;31m"
   su -c "sh $0 $1"
   exit
fi
echo -e "\033[0m"; echo -en "\033]0;Clone Drive\a"

echo && echo "Input device"
lsblk -f --output NAME,FSTYPE,LABEL,SIZE | grep $ID
echo && echo "Output device"#fdisk -l /dev/$OD 
lsblk -f --output NAME,FSTYPE,LABEL,SIZE | grep $OD

echo
#set -x;
read -p "Enter Drive number? " DRV1
if [ ! "$DRV1" == "" ]; then
   ID=$ID$DRV1
   read -p "Enter Drive number 2? " DRV2
   if [ ! "$DRV2" == "" ]; then
      OD=$OD$DRV2
   else
      OD=$OD$DRV1
   fi
fi
echo 
read -p "Copy /dev/$ID to /dev/$OD? " YN
if [ ! "$YN" == "y" ]; then
   exit
fi

echo && date && echo
dd bs=1M if=/dev/$ID  of=/dev/$OD status=progress
echo && date && echo

lsblk -f --output NAME,FSTYPE,LABEL,SIZE,FSUSED | grep $ID
echo
lsblk -f --output NAME,FSTYPE,LABEL,SIZE,FSUSED | grep $OD
read

The ID drive had many .iso files on it.

Added in 7 hours 8 minutes 40 seconds:
Interesting.

So tweaking my script to support your wget -O- $url | sudo dd bs=1M of=/dev/sdx command,

ID=http://mirrors.dotsrc.org/porteus/x86_6 ... x86_64.iso # Input device
wget -O- $ID | sudo dd bs=1M of=/dev/$OD

which yields:

Code: Select all

guest@porteus:~$ ./iso2usb.sh
Enter root's password
Password: 

Input 
http://mirrors.dotsrc.org/porteus/x86_64/current/Porteus-CINNAMON-v5.01-x86_64.iso

Output device
sda                               28.9G
└─sda1      vfat     PORTEUS501   28.9G

Enter Drive number 1? 
Enter Drive number 2? 1

Copy /dev/http://mirrors.dotsrc.org/porteus/x86_64/current/Porteus-CINNAMON-v5.01-x86_64.iso to /dev/sda1? y

Mon Mar 24 01:10:39 EDT 2025

+ wget -O- http://mirrors.dotsrc.org/porteus/x86_64/current/Porteus-CINNAMON-v5.01-x86_64.iso
+ sudo dd bs=1M of=/dev/sda1
--2025-03-24 01:10:39--  http://mirrors.dotsrc.org/porteus/x86_64/current/Porteus-CINNAMON-v5.01-x86_64.iso
Resolving mirrors.dotsrc.org (mirrors.dotsrc.org)... 2001:878:346::116, 130.225.254.116
Connecting to mirrors.dotsrc.org (mirrors.dotsrc.org)|2001:878:346::116|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 457179136 (436M) [application/octet-stream]
Saving to: ‘STDOUT’

-                           100%[==========================================>] 436.00M  1.87MB/s    in 3m 25s  

2025-03-24 01:14:05 (2.12 MB/s) - written to stdout [457179136/457179136]

0+26407 records in
0+26407 records out
457179136 bytes (457 MB, 436 MiB) copied, 206.014 s, 2.2 MB/s
+ set +x

Mon Mar 24 01:14:05 EDT 2025

└─sda1      iso9660  Porteus      28.9G   3.9G

ls: cannot access 'sda1': No such file or directory

guest@porteus:~$ ls /dev/sda1/
/bin/ls: cannot access '/dev/sda1/': Not a directory
guest@porteus:~$ ls /dev/sda1
/dev/sda1
guest@porteus:~$ ls -nom /dev/sda1
/dev/sda1
guest@porteus:~$ 

The readable FAT32 USB drive is now a iso9660 USB drive that I can't open in any file manager to access the .iso files I downloaded.

[robang74]

In the meantime, I reach a version that start to be "nice" for others people testing, and soon I will tag the v0.3.2 as well (faster, much faster)

Code: Select all

#!/bin/bash
#
# (C) 2025, Roberto A. Foglietta <[email protected]> - 3-clause BSD
#
# Support for expert users wishing to test on-demand installation starts from
#
# URL: https://raw.githubusercontent.com/robang74/porteus-usb-installer
# VER:                         /refs/tags/v0.3.1/porteus-net-install.sh
# MAN:                           ?tab=readme-ov-file#usage-quick--dirty
#
# It is suggest testing on a spare machine that does NOT hold valuable data,
# possibly using a Porteus MATE v5.0.1 with the network support available.
#
################################################################################

[robang74]

UPDATE - V0.3.2- WITH SUPPORT FOR XPU AND DEV

Check the following link to this forum post:

• Porteus v5.0.1 network install v0.3.2

[robang74]

UPDATE - HOW/WHY TO PROVIDE SECURITY WITH SHA256

Check the following link to this forum post:

• viewtopic.php?p=102470#p102470