[Solved] The generic Linux installer is incompatible with Ubuntu + fixing LILO/exFAT

Please reproduce your error on a second machine before posting, and check the error by running without saved changes or extra modules (See FAQ No. 13, "How to report a bug"). For unstable Porteus versions (alpha, beta, rc) please use the relevant thread in our "Development" section.
jjr
Black ninja
Black ninja
Posts: 46
Joined: 18 Nov 2023, 17:10
Distribution: 5.0

[Solved] The generic Linux installer is incompatible with Ubuntu + fixing LILO/exFAT

Post#1 by jjr » 17 Feb 2024, 22:34

3 fatal issues on Ubuntu (tested 22.04).

According to USB_INSTALLATION.txt, it's intended to work "From any Linux Distribution other than Porteus".

Issues 2-3 are caused by Ubuntu's default handling of ownership and permissions on FAT filesystems.

SUMMARY
  1. The installer tries to parse the target device from $PWD, but Ubuntu mounts storage devices at /media/USER_NAME/VOLUME_LABEL (rather than the expected /mnt/DEV_NAME), so that is not possible on Ubuntu.

    Installer result: "Installing Porteus to /dev/josh" (which doesn't exist). There's a universal POSIX way to do this instead (see below).
    .
  2. Ubuntu does not silently ignore chown on a FAT filesystem, causing MakeSelf's default targetdir="." to fail UnTAR(), and the installer thus terminates with "Extraction failed." (tar's default behavior as root is --same-owner, which performs chown.)
    .
  3. Ubuntu uses an opaque heuristic to determine executable permissions on a FAT filesystem, by default. Ultimately, the extlinux binary cannot be run when extracted to the target device.
#2-3 are problems because USB_INSTALLATION.txt tells the user to navigate to the 'boot' directory before running the installer -- so that it can detect the target device -- but this is typically a FAT filesystem.

FIX FOR ISSUE #1

The POSIX-guaranteed way to acquire the device node name from a directory is by parsing the output of df -P <dirname>.

The code you currently have is this:

Code: Select all

PRT=/dev/`echo $PWD | cut -d/ -f3`
MPT=/mnt/`echo $PWD | cut -d/ -f3`
DEV=/dev/`lsblk -ndo pkname $PRT`
PRTN=`echo ${PRT#"$DEV"} | sed 's/[^0-9]*//g'`
As a result, on Ubuntu the Porteus installer thinks the partition node name is USER_NAME (3rd field of "/media/USER_NAME/VOLUME_LABEL"),
and so the installer says "Installing Porteus to /dev/USER_NAME" (which obviously doesn't exist).

These replacement definitions should be portable for all Linux, and work fine for Ubuntu:

Code: Select all

PRT="$(df -P "$PWD" | grep -o '^/dev/[^ ]\+')"
MPT="$(grep "^$PRT" /proc/mounts | cut -d' ' -f2)"
The existing DEV and PRTN definitions seem fine.

FIX FOR ISSUE #2

If you try to workaround issues #2-3 by invoking the installer with "--target /tmp" (or some other Unix-compatible filesystem), then the inner installer.com's $PWD is no longer on the target device, and so it fails to detect the target device even if issue #1 was fixed.

The only simple workaround I found for issue #2 was adding --no-same-owner to the tar command:

Code: Select all

UnTAR()
{
    if test x"$quiet" = xn; then
		tar $1vf - --no-same-owner 2>&1 || { echo " ... Extraction failed." >&2; kill -15 $$; }
    else
		tar $1f - --no-same-owner  2>&1 || { echo Extraction failed. >&2; kill -15 $$; }
    fi
}
But that is patching the MakeSelf output, so probably not something you want to do.

FIX FOR ISSUE #3

This is where it gets uglier...

The installer pack extracted to a FAT filesystem on Ubuntu (tested 22.04) had the following permissions:

Code: Select all

-rw-r--r-- 1 root root 208480 Feb 17 16:36 extlinux.x32
-rw-r--r-- 1 root root 209424 Feb 17 16:36 extlinux.x64
-rwxr-xr-x 1 root root   5589 Feb 17 16:36 installer.com
-rwxr-xr-x 1 root root  95392 Feb 17 16:36 lilo.com
-rw-r--r-- 1 root root    440 Feb 17 16:36 mbr.bin
From some experiments, Ubuntu's heuristic for eXec seems to depend on both the filename extension and content (??)

Anyway, LILO and the entry script are OK, but AFAIK the only way to run extlinux from FAT on Ubuntu is using the ld interpreter trick. The interpreter paths are hardcoded into the binaries (check them by running "file <binary>"), so it should be safe.

That was the only way I was able to get the installer to work on Ubuntu.

Basically, you'd need to change this line in installer.com:

Code: Select all

	$bin/$extlinux -i "$IPT"/syslinux >/dev/null 2>&1
To be effectively this for extlinux.x64:

Code: Select all

	/lib64/ld-linux-x86-64.so.2 $bin/$extlinux -i "$IPT"/syslinux >/dev/null 2>&1
...or this for extlinux.x32:

Code: Select all

	/lib/ld-linux.so.2 $bin/$extlinux -i "$IPT"/syslinux >/dev/null 2>&1
(Of course, you'd do it with a variable...)

RETROSPECT

These are ugly approaches for #2-3, IMO. I think it would be cleaner and safer for the MakeSelf script to extract its payload to "/tmp" by default, and then pass its own launch location to the inner script as a "--target-dir" param. Then the inner installer.com would use that param to determine the target device.

Just a suggestion.
Last edited by jjr on 24 Feb 2024, 02:35, edited 4 times in total.

jjr
Black ninja
Black ninja
Posts: 46
Joined: 18 Nov 2023, 17:10
Distribution: 5.0

The generic Linux installer is incompatible with Ubuntu

Post#2 by jjr » 18 Feb 2024, 18:48

For anyone who wants to rebuild the installer with the fixes to support Ubuntu--

Linux text editors have trouble opening the outer com, because of the binary data at the end.

I used Geany with the special "No Encoding" open option, and it will roughly-truncate the script. Tar/gzip the new payload, update the MakeSelf metadata vars (size, md5, crc=cksum), delete the binary residue at EOF, and re-append (>>) the new payload.

jjr
Black ninja
Black ninja
Posts: 46
Joined: 18 Nov 2023, 17:10
Distribution: 5.0

The generic Linux installer is incompatible with Ubuntu

Post#3 by jjr » 18 Feb 2024, 19:35

A 4th issue on Ubuntu (minor)--

Because Ubuntu mountpoints include the volume label (/media/USER_NAME/VOLUME_LABEL), if the user has a space in the volume label, then the lack of quotes in the inner installer.com's su command will fail...

i.e. this:

Code: Select all

	su -c "$0 $PWD"
Should be this instead:

Code: Select all

	su -c "'$0' '$PWD'"
Edit: All the unquoted $bin uses are referencing the same path, so they need to be fixed too.

User avatar
Ed_P
Contributor
Contributor
Posts: 8374
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

The generic Linux installer is incompatible with Ubuntu

Post#4 by Ed_P » 18 Feb 2024, 20:20

jjr wrote:
17 Feb 2024, 22:34
According to USB_INSTALLATION.txt, it's intended to work "From any Linux Distribution other than Porteus".
So the fix for this Ubuntu problem is to update the USB_INSTALLATION.txt to state that it works "From any other Linux Distributions, other than Ubuntu:". Sounds simple enough to do. :D
Ed

jjr
Black ninja
Black ninja
Posts: 46
Joined: 18 Nov 2023, 17:10
Distribution: 5.0

The generic Linux installer is incompatible with Ubuntu

Post#5 by jjr » 18 Feb 2024, 21:03

Ed_P wrote:
18 Feb 2024, 20:20
jjr wrote:
17 Feb 2024, 22:34
According to USB_INSTALLATION.txt, it's intended to work "From any Linux Distribution other than Porteus".
So the fix for this Ubuntu problem is to update the USB_INSTALLATION.txt to state that it works "From any other Linux Distributions, other than Ubuntu:". Sounds simple enough to do. :D
lol :D
Honestly, I would understand that. Ubuntu is being weird/non-conventional, and dev time is a limited resource.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 3941
Joined: 20 Mar 2012, 03:42
Distribution: v5.0-64bit
Location: australia
Contact:

The generic Linux installer is incompatible with Ubuntu

Post#6 by ncmprhnsbl » 19 Feb 2024, 05:44

suffice to say, the docs are lacking..
there's also /boot/docs/install.txt which goes into a bit more detail (but has nothing about EFI :p)
the whole mounting_in_media_blah is fairly generic across distros afaia so your fixes for that (and probly what you say about the launching of the makeself) would make life easier, thanks for that. :)

i'm curious though, would the workaround(to be fair, it's not mentioned as such) of manually mounting the drive first work with ubuntu? ie. as per /boot/docs/install.txt:

Code: Select all

If the device is not mounted automatically, then you can open a 
console and type:

fdisk -l

to get the path of your flash drive (e.g., /dev/sdb1), and then:

mkdir /mnt/sdb1
mount /dev/sdb1 /mnt/sdb1
then carrying on from there

one day, when non efi hardware becomes extinct, this will be moot.
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

jjr
Black ninja
Black ninja
Posts: 46
Joined: 18 Nov 2023, 17:10
Distribution: 5.0

The generic Linux installer is incompatible with Ubuntu

Post#7 by jjr » 19 Feb 2024, 07:35

ncmprhnsbl wrote:
19 Feb 2024, 05:44
i'm curious though, would the workaround(to be fair, it's not mentioned as such) of manually mounting the drive first work with ubuntu? ie. as per /boot/docs/install.txt
OK, this is nice.

A user can workaround all 4 issues on Ubuntu (and same-behaving Linuxes) without patching anything, just by doing what you wrote:

Code: Select all

# kick Ubuntu's automount
sudo umount /dev/sdb1
sudo mkdir /mnt/sdb1
sudo mount /dev/sdb1 /mnt/sdb1
#1,4: As you noted, the old-fashioned mountpoint avoids issues #1 and #4(separate post)
#3: Surprisingly "-o exec" is Not needed here to bless extlinux: just a plain manual mount restored the conventional all-exec for FAT.
#2: Mounting as root (not Ubuntu's auto uid=,gid=) avoided issue #2: tar sees that chown'ing to root is unnecessary and doesn't invoke it -> doesn't die.

And the installation works! :)

So it could be document something and wait for BIOS to go extinct, or patch the com scripts, up to you guys...

jjr
Black ninja
Black ninja
Posts: 46
Joined: 18 Nov 2023, 17:10
Distribution: 5.0

The generic Linux installer is incompatible with Ubuntu

Post#8 by jjr » 19 Feb 2024, 18:47

ncmprhnsbl wrote:
19 Feb 2024, 05:44
Update: it looks like issue #1 was reported for Kali Linux and got misdiagnosed here:
Porteus installer chosing the wrong drive

Code: Select all

partition: /dev/kali
partition mount point: /mnt/kali
installation path: /media/kali/d292297d-1b43-42b7-bc7d-b5e2acbd3d43/boot
See how the "cut -d/ -f3" behaved on the mountpoint:
/media/kali/d292297d-1b43-42b7-bc7d-b5e2acbd3d43/boot
...leading to this...

Code: Select all

Installing Porteus to /dev/kali
(which doesn't exist)

Edit: The logged "partition mount point" /mnt/kali was of course also bogus, for the same reason.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 3941
Joined: 20 Mar 2012, 03:42
Distribution: v5.0-64bit
Location: australia
Contact:

The generic Linux installer is incompatible with Ubuntu

Post#9 by ncmprhnsbl » 20 Feb 2024, 02:47

jjr wrote:
19 Feb 2024, 18:47
Update: it looks like issue #1 was reported for Kali Linux and got misdiagnosed here:
yeah, thanks
i can think quite a few instances over the last ten years or so, before that i imagine people more often booted from cds and installed from within porteus.
another workaround i've seen, is to boot porteus in virtualbox, mount the usb and install from the virtual machine.
on the documentation side, i can see some scope to cover some more of the many ways of booting a porteus installation eg. Ed here uses grub to boot isos, ..how to add porteus to an existing bootloader.. and so on

lately i've been using limine(which i'll open another thread for) which might be a possible replacement for syslinux
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 3941
Joined: 20 Mar 2012, 03:42
Distribution: v5.0-64bit
Location: australia
Contact:

The generic Linux installer is incompatible with Ubuntu

Post#10 by ncmprhnsbl » 20 Feb 2024, 07:09

jjr wrote:
17 Feb 2024, 22:34
..
here's an updated :dl-green: Porteus-installer-for-Linux.com :dl-green: md5sum: e6ffbb50f60bdc14331d05f711a6049e
using your fixes and suggestions(as far as i am able)
ie. extracts to /tmp with param "-T $PWD" so should avoid issues #2/3 ,
shouldn't get /different/types/of/mount/points confused
for privilege escalation i went back to old method: none ... just a message telling you to run it as root.
cleaned up the egrep messages (that slackware suppresses)
removed the UUID initrd injection since that isn't being used in porteus at present.
tested on a couple of arch based distros with different mounting methods( one udevil /media/some_devic_name the other udisks2 i think, /run/media/user_name/bunch_o_numbers_letters)
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

jjr
Black ninja
Black ninja
Posts: 46
Joined: 18 Nov 2023, 17:10
Distribution: 5.0

The generic Linux installer is incompatible with Ubuntu

Post#11 by jjr » 20 Feb 2024, 11:38

ncmprhnsbl wrote:
20 Feb 2024, 07:09
here's an updated :dl-green: Porteus-installer-for-Linux.com :dl-green: md5sum: e6ffbb50f60bdc14331d05f711a6049e
Looks good! It works for me unless there's a space in the volume label, per issue #4. Lots of people use spaces nowadays.
The $PWD would need to be single-quoted inside "-T $PWD":

Code: Select all

scriptargs="-T '$PWD'"
The rm -rf $bin (installer.com line 10) without safe quoting is okay, because there are no spaces in "/tmp/.portues_installer".
(User would have to contrive a --target override to produce a disaster in rm -rf, but that won't happen.)

jjr
Black ninja
Black ninja
Posts: 46
Joined: 18 Nov 2023, 17:10
Distribution: 5.0

The generic Linux installer is incompatible with Ubuntu

Post#12 by jjr » 20 Feb 2024, 11:48

note: edited the above post
ncmprhnsbl wrote:
20 Feb 2024, 02:47
limine(which i'll open another thread for) which might be a possible replacement for syslinux
That is attractive...
>i686+: no more porteus on attic dinosaur Pentium post-apocalypse
What I did personally was consider stagnant syslinux + BIOS as 'frozen together', and add grub just for the UEFI side, to stay up-to-date for modern machines.
But maintaining 2 menus would be unwieldy for an official release.

burdi01
Shogun
Shogun
Posts: 206
Joined: 18 Aug 2013, 12:09
Distribution: Slackware PartedMagic Xubuntu
Location: The Netherlands

The generic Linux installer is incompatible with Ubuntu

Post#13 by burdi01 » 20 Feb 2024, 12:39

Why not use Grub for BIOS/MBR booting?
LD

donald
Full of knowledge
Full of knowledge
Posts: 2074
Joined: 17 Jun 2013, 13:17
Distribution: Porteus 3.2.2 XFCE 32bit
Location: Germany

The generic Linux installer is incompatible with Ubuntu

Post#14 by donald » 20 Feb 2024, 15:30

ncmprhnsbl wrote:
20 Feb 2024, 02:47
lately i've been using limine (which i'll open another thread for) which might be a possible replacement for syslinux
Starting with version 6, Limine dropped support for ext2/3/4 file systems
and only supports FAT12, FAT16, FAT32 and ISO9660.

..so anyone who is used to loading the kernel and initrd from a Linux file system,
will have to copy them to an extra FAT boot partition. -- no thanks

jjr
Black ninja
Black ninja
Posts: 46
Joined: 18 Nov 2023, 17:10
Distribution: 5.0

The generic Linux installer is incompatible with Ubuntu

Post#15 by jjr » 20 Feb 2024, 16:23

burdi01 wrote:
20 Feb 2024, 12:39
Why not use Grub for BIOS/MBR booting?
LD
If you were asking me, there's an amalgamation of reasons why I don't use GRUB2 for BIOS.

1. Drives over 2TB with 512-byte sectors can't be partitioned with MBR, and GRUB2 for GPT-BIOS requires inserting a hacky extra partition at the start of your drive.
  • (A) You will no longer be able to simply copy Porteus system files onto any old partition and run an easy installer script. You would have to erase and repartition the drive.
  • (B) Porteus is already outside of UEFI spec by not properly marking the ESP-- hoping the firmware will find it anyway. Inserting the specially-marked Grub BIOS boot partition as the first GPT partition might be pushing luck for some firmwares. I don't know. Yet properly marking it also blocks it from being mountable in Windows and macOS disk utilities - not desirable.
2. GRUB2 is heavyweight and a mess (more things to break) compared to SYSLINUX, which I simply trust more for ancient/legacy machines.

History on how GRUB2 got to be that way (post by Brendan): https://forum.osdev.org/viewtopic.php?p=242859#p242859

3. GRUB2 apparently has multiple bugs when trying to create El-Torito/ISO9660 for BIOS (see comments), which is a requirement for me:
https://unix.stackexchange.com/question ... b-versions

My custom make_iso.sh generates El-Torito for BIOS + UEFI using mkisofs 3.02 (which Porteus includes). It assigns each bootloader to what it does best imo (SYSLINUX for BIOS, GRUB2 for UEFI).

4. GRUB2 resolution-setting has been quirky on some machines for me (Google concurs). And I felt that keeping these 2 bootloaders separate provided a slightly-better guarantee of the appropriate (different) graphics resolution for each hardware.

Specifically, I read in several places that 640x480 may not work with all EFI firmwares and 1024x768 is recommended, while you definitely want 640x480 for old VGA BIOS. The original place I read it was on a syslinux wiki. A Macrium representative also said this: "The minimum and default resolution in UEFI is 1024x768; if 640x480 is set that would represent a firmware bug." (edit: expanded quote to show context)

5. As a bonus point, I like the graphical layout of vesamenu (not obscuring my splash screen) + custom help page, and I didn't want to lose that completely.

-------------------
If none of these reasons apply to you, then ofc you can use Grub for BIOS.
Last edited by jjr on 20 Feb 2024, 23:24, edited 6 times in total.

Post Reply