Windows and Linux installer revisions, including GPT-BIOS support for >2 TiB drives

[jjr]

This was a personal project to polish the installer for use in a cold storage building system.

It has some improvements I think are worthwhile, and it did get some granular testing, but I wouldn't suggest using it in anything permanent. It's not time-tested.

Please just ignore this thread if not interested.

Linux installer changes

• Autodetects and supports GPT and MBR target disks
• Reviewed: incorporates best of fanthom's good code and the regressed newer version
• Prints helpful error messages for all failure conditions
• Reorganized into sections and commented
• Reworked and commented device<->partition logic
• No longer requires bash
• Implemented fanthom's automatic install option (-a) completely (see "minor enhancements")
• Added fallback parent device detection for util-linux <2.23 (pre-2013, lacking lsblk), with warning message
• MBR side should support the oldest util-linux on kernel.org (2.13, 2007)
• Internal script is drop-in compatible with both wrappers (Porteus' very heavyweight, very slightly buggy MakeSelf, and my mini appimagelike wrapper with checksum).
• Appimagelike wrapper only: Installs to the location of the outer installer script instead of to the user's working directory, to prevent user from mistakenly damaging his system drive.
• Command-line [-h]elp, accessible through the wrapper, with options:
  
  • custom boot path target (defaults to the wrapper location)
  • the previously-undocumented LILO MBR option (-f)
  • automatic installation bypassing prompts (-a)
• Prompts before installing LILO to an unknown filesystem's boot sector (potentially corrupting the filesystem), suggesting to use -f option
• Minor enhancements
  • Avoid clobbering the sanity file across instances- use unique ID
  • Use getopts instead of grepping "$*"
  • Automatic mode is now honored for all prompts
  • Automatic mode suppresses 'clear' command to play nicer with parent scripts (e.g. cold storage builders)
  • Always exit through a fail function except for success or the usage-print, for better updating/maintenance of failure codes
  • More debug log fields
  • Check for presence of more required commands
  • Check commands with 'type' instead of 'which' ('which' is not defined in any specification; 'type' is POSIX sh built-in)
  • Refactored unclear use of PRTN to avoid re-regression in future
  • Removed exit delay on failure: this isn't a Windows exe console, and 1 second isn't enough to read an error message
  • Removed exit trap: any cleanup is already the wrapper's responsibility
  • Miscellaneous tweaks

Windows installer changes

• Split into separate MBR and GPT editions.
• Fixed Windows warning "This program might not have installed correctly" by adding the current LZMA SDK compatibility attributes to the sfx header
• Added icon resolutions up to 48 px
• GPT version shows diskpart example to help user manually set Legacy BIOS Bootable flag.

Windows notes

• See this warning if formatting the disk outside of Windows.
• It's possible to automate the Legacy BIOS Bootable attribute on Windows, but challenging because it would need either compiling a special-purpose C program, bundling CYGWIN sfdisk, or scripting diskpart and parsing WMI by invoking powershell, which is harder if supporting Windows 7 (PS 2). WMIC is deprecated.
• Windows utilities insert a hidden Microsoft Reserved partition at the start of any GPT format. For UEFI systems, you may want to remove it to place the ESP first. In diskpart: delete partition override

Reliability Dilemma

Clarification: You can boot BIOS and UEFI with the same GPT table, or with the same MBR table. That's according to both firmware designs but is often misunderstood. See post here.

However, a few BIOSes are 'buggy' and will refuse to boot from GPT unless you put the Active flag in the protective MBR, which in turn breaks booting on a few UEFI, which prohibit the flag.

That's the dilemma this BSD report is referring to:
https://bugs.freebsd.org/bugzilla/show_ ... =194359#c0

Contents

• Linux installer for MBR/GPT with appimagelike wrapper
• Linux installer for MBR/GPT with original MakeSelf wrapper
• Separate Windows installers for MBR and GPT
• Component files and readmes for 'building', but not with MakeSelf
• Optional 'updated' sfx header from LZMA SDK 23 with asm patch/instructions.

Download

always current
https://drive.google.com/file/d/1PC04Yz ... drive_link


Updates
2024/04/04:
* Fixed typos in doc
* Improve error message for outdated (MBR-only) sfdisk, instructing user to upgrade their util-linux package.
* Avoid MakeSelf's broken ARCHIVE_DIR by using its USER_PWD when available, and ORIGIN_DIR from appimagelike wrapper otherwise.

[jjr]

Prompts before installing LILO to an unknown filesystem's boot sector (potentially corrupting the filesystem), suggesting to use -f option

This is related to a fix in the last Porteus update..

use lilo for exfat

It did already use LILO for exFAT, it was just corrupting the FS by writing a partition boot sector..

[Ed_P]

jjr, your installer is a 3 MB download, the current 2 Porteus installers together are 570 KB.

[ncmprhnsbl]

wow, thank you very much
much to digest here.. looks nice at first squint..

jjr, your installer is a 3 MB download, the current 2 Porteus installers together are 570 KB.

have at look at the "Content" section, there are two archive versions of the linux installer(iso,makeself) and sources and documentation for all, so while necessarily larger, not by that much.

[ncmprhnsbl]

first possible bug:
with the makeself version of Porteus-installer-for-linux.com, a path is required,
ie. when running from the <installation>/boot folder without adding that path, the wrong partition was indicated(the / (root partition) of the host OS(booted from a separate EFI boot partition with limine)
it did however correctly identify that (wrong) partition table as GPT (:
so, "(defaults to the wrapper location)" failed in this case. (noting, with the target path, it can be run from anywhere )
EDIT: did a quick test, reverting to $USER_PWD instead of $ARCHIVE_DIR and that worked correctly.
just running `dirname "$0"` (ie. the command for $ARCHIVE_DIR) directly in the terminal returns: /bin (with zsh) and . (with bash) where pwd returns: /media/porteus-beta/boot (which upon reflection means nothing)
EDIT2: ok after looking closer at what dirname does, i see that it's only when running the script in the boot dir without a path that the failure occurs.
but if run using the absolute path (from anywhere) it's fine ie.
pwd is /media/porteus-beta/boot

Code: Select all

./Porteus-installer-for-Linux.com

fails to find the correct partition,
but

Code: Select all

/media/porteus-beta/boot/Porteus-installer-for-Linux.com

is fine (and regardless of what pwd is)
so a path, either prefixed or appended is needed.

the appimagelike wrapper version works as expected.

[jjr]

first possible bug:
with the makeself version of Porteus-installer-for-linux.com, a path is required,
... just running `dirname "$0"`...

the appimagelike wrapper version works as expected.

Ah...
Yes, it's looks like MakeSelf uses a buggy/faulty way to get the script directory.
I really do not like or trust MakeSelf's code. You just showed another reason why.

Correct (mine):

Code: Select all

ARCHIVE_DIR="$( cd -- "$(dirname -- "$0")" > /dev/null 2>&1 ; pwd -P )"

Wrong (MakeSelf):

Code: Select all

ARCHIVE_DIR=`dirname "$0"`

Source: https://stackoverflow.com/a/4774063
As you said, you can switch to USER_PWD for that one. We can't fix their ARCHIVE_DIR.

Edit: I will update the OP to make the change.

[jjr]

jjr, your installer is a 3 MB download, the current 2 Porteus installers together are 570 KB.

have at look at the "Content" section, there are two archive versions of the linux installer(iso,makeself) and sources and documentation for all, so while necessarily larger, not by that much.

Thank you, yes. According to Google Drive it's only 1.3 MB, and that's including all the build resources.
The actual installers are still tiny, of course.

[jjr]

first possible bug:
with the makeself version of Porteus-installer-for-linux.com, a path is required,
... just running `dirname "$0"`...

the appimagelike wrapper version works as expected.

Yes, it's looks like MakeSelf uses a buggy/faulty way to get the script directory.
I really do not like or trust MakeSelf's code. You just showed another reason why.
[...]

Updated OP:
The inner installer is still drop-in compatible with both wrappers, but the order of precedence is now OPT_BOOT > USER_PWD > ORIGIN_DIR.

It's against my better judgment not to just drop MakeSelf altogether. But I understand some people probably want it.

[ncmprhnsbl]

Updated OP:
The inner installer is still drop-in compatible with both wrappers, but the order of precedence is now OPT_BOOT > USER_PWD > ORIGIN_DIR.

confirming both wrapper versions are working for basic tests.

It's against my better judgment not to just drop MakeSelf altogether. But I understand some people probably want it.

speaking for myself, i've no special attachment to makeself, it's just what's been passed down, as it were, off the shelf and finding alternatives wasn't on the radar.
your "appimage-like-self-mounting/executing-iso" is a new concept that is very interesting to say the least. (needs a catchy name :p )
the extra 200ishK from no compression doesn't bother me either.
next step for me is to reproduce the build steps and try to get my head around it some more. (and check out windows version)
and i encourage others to test these installers and give some feedback.

[jjr]

Thanks for your kind reply.

reproduce the build steps

Yeah, it would need an automated build system, which wasn't a priority. (I'm rushing some things before Middle East gets worse...)

the extra 200ishK from no compression

squashfs is an option, but I thought backwards compatibility was more important.

___________________________________
Update:
* Improve error message for outdated (MBR-only) sfdisk, instructing user to upgrade their util-linux package.

[jjr]

A warning for Windows:

If you're partitioning a disk with GPT on Linux or macOS, especially with a small disk-

Certain tools will silently create a hybrid MBR instead of a protective MBR, which defies the Intel GPT spec and will be invalid on Windows. Parted/GParted on Ubuntu 22 is doing this for me. (If you open the disk with fdisk afterward, it will warn you that it found a hybrid.)

The author of GPT fdisk (gdisk) in his article warning against hybrid MBRs mentions this:

Windows — When confronted with a hybrid MBR, Windows tries to treat the disk as an MBR disk, ignoring the GPT partitions completely.

Result: you won't be able to set the Legacy BIOS Bootable GPT attribute in diskpart.

These tools are safe to use AFAIK:
* Linux: fdisk, gdisk
* macOS: gpt (BSD built-in)
* Windows: diskpart

[Ed_P]

This 10 yr old thread may interest you jjr: Uefi and porteus 3.0 (Post by Ed_P #25587)

[jjr]

This 10 yr old thread may interest you jjr: Uefi and porteus 3.0 (Post by Ed_P #25587)

Thank you, I saw that but I missed that brokenman said 2 completely false things..

... UEFI systems, which require a GPT disk to boot from.
... NON-UEFI systems (that require a real MBR) ...

Those are false, which it sounds like you knew.

A. The UEFI spec mandates support for detecting the EFI system partition in an MBR partition table and will boot perfectly fine without a GPT.

UEFI Specification 2.10 Section 5.2.1

See also Microsoft FAQ: https://learn.microsoft.com/en-us/windo ... s-and-esps
"UEFI specifies booting from either GPT or MBR. The ESP on an MBR disk is identified by partition type 0xEF."

B. BIOS systems do not require an MBR table. In fact, they're theoretically partition table agnostic. They're supposed to only read the first 440 bytes of address 0 and execute it. It doesn't (or shouldn't) care what is on the disk after those 440 bytes. It could be unicorns after that.

That bootloader code itself is free to locate second-stage boot code however it wants:

• SYSLINUX, for example, scans the partition table (either MBR or GPT -- depending on which 440-byte code you write, mbr.bin or gptmbr.bin) for a flag indicating that the partition contains second-stage bootloader code at its first address. That second-stage code is able to chain into code on the filesystem which can read the filesystem and find the Linux kernel.
  
  (For MBR tables, that partition flag is the "Active" flag. For GPT, that flag is the "Legacy BIOS Bootable" attribute, which was an original proposal by the SYSLINUX author, which was later standardized.)
  .
• LILO's 440 bytes, in contrast, is able to jump into the filesystem directly, which is why it's used for filesystems like exFAT that don't have space at their beginning for second-stage boot code. (That's why the Porteus' earlier installer was corrupting exFAT filesystems.)

-----------

I didn't study your post thoroughly, so pardon if I misunderstood some part. But...

Bottom-line: You can boot both UEFI and BIOS systems on a single GPT table, or on a single MBR table.
I shouldn't have assumed everyone knew that. I'll edit the OP to clarify.

The exception is a few machines that behave poorly.

The "Reliability Dilemma" section in the OP was referring to that. Basically, a few BIOS's refuse to boot unless they find a partition Active flag in the GPT's protective MBR. But adding that flag breaks booting for a few UEFIs, because UEFI mandates no flag in the PMBR.

[jjr]

Improved second source in previous post (Microsoft's summary of the UEFI spec)

...
See also Microsoft FAQ: https://learn.microsoft.com/en-us/windo ... s-and-esps
"UEFI specifies booting from either GPT or MBR. The ESP on an MBR disk is identified by partition type 0xEF."
...

Side note-
Most UEFIs will detect an ESP without the MBR 0xEF type or correct GPT GUID, a behavior which Porteus' USB_INSTALLATION.txt is relying on.

If you do mark the partition as an ESP, major operating systems (Windows and macOS, various Linuxes) will try to hide the partition and prevent you from mounting it, so the user won't be able to copy or update Porteus files on it.

Ed_P I was curious what you were doing.. I'll study it better later.