5.5.0 & Firefox - file not found?

[LuxZg]

Hi Tomasz,

Here I am descending into my annual Porteus Kiosk dungeon mastering

We've got new PC testing, with Alder Lake (N100), and after some initial issues we got it to PXE boot, and get into single-screen with Chrome. But dual screen with Chrome keeps failing, and using modesetting for GPU didn't work. So we turned to Firefox.

Some initial, mostly manual test confirmed that N100 + Porteus Kiosk 5.5.0 + Firefox + dual screen actually - works!

But now we've created new image for proper use, and when the PC boots there is no browser at all, just wallpaper.

Now the debugging part:
- I tried running firefox via SSH session, and I keep getting

Code: Select all

/usr/bin/apulse: exec: line 14: /usr/lib64/firefox/firefox: not found

- checking the folder confirms suspicions:

Code: Select all

ls -al /usr/lib64/firefox/
total 0
drwxr-xr-x    6 root     root            60 Jan 15 15:51 .
drwxr-xr-x   24 root     root            60 Jan 15 15:51 ..
drwxr-xr-x    6 root     root            96 May 24  2015 browser
drwxr-xr-x    2 root     root            80 Jan 15 15:51 content
drwxr-xr-x    2 root     root            36 Jun 14  2021 distribution

So it looks like it didn't overwrite the files on system with the files from 002firefox.xzm

I tried to manually see if I can run the browser from SSH, by getting and unpacking XZM manually. I am sure this is not the intended way, it's just to debug and get more info hopefully:

Code: Select all

curl https://porteus-kiosk.org/public/5.5/002-firefox.xzm -O
mount -o loop 002-firefox.xzm  /home/something/mnt

Doing that I can access the files

Code: Select all

root@TRYCOOT1:/home/something/mnt# ls -al /home/something/mnt/usr/lib64/firefox/
total 185834
drwxr-xr-x    5 root     root           613 Mar 18  2023 .
drwxr-xr-x    3 root     root           261 Aug 25  2019 ..
-rw-r--r--    1 root     root           902 Jan  1  1970 application.ini
drwxr-xr-x    3 guest    guest           46 Mar 18  2023 browser
-rw-r--r--    1 root     root           174 Jan  1  1970 dependentlibs.list
-rwxr-xr-x    1 root     root         14416 Jan  1  1970 firefox
-rwxr-xr-x    1 root     root        646568 Jan  1  1970 firefox-bin
drwxr-xr-x    2 root     root            41 Mar 18  2023 fonts
drwxr-xr-x    3 root     root            26 Mar 18  2023 gmp-clearkey
-rwxr-xr-x    1 root     root        813432 Jan  1  1970 libfreeblpriv3.so
-rwxr-xr-x    1 root     root        355392 Jan  1  1970 libipcclientcerts.so
-rwxr-xr-x    1 root     root         43200 Jan  1  1970 liblgpllibs.so
-rwxr-xr-x    1 root     root       3841464 Jan  1  1970 libmozavcodec.so
-rwxr-xr-x    1 root     root        228136 Jan  1  1970 libmozavutil.so
-rwxr-xr-x    1 root     root         14056 Jan  1  1970 libmozgtk.so
-rwxr-xr-x    1 root     root        117384 Jan  1  1970 libmozsandbox.so
-rwxr-xr-x    1 root     root       1158728 Jan  1  1970 libmozsqlite3.so
-rwxr-xr-x    1 root     root         18224 Jan  1  1970 libmozwayland.so
-rwxr-xr-x    1 root     root        223024 Jan  1  1970 libnspr4.so
-rwxr-xr-x    1 root     root        707000 Jan  1  1970 libnss3.so
-rwxr-xr-x    1 root     root        437448 Jan  1  1970 libnssckbi.so
-rwxr-xr-x    1 root     root        187416 Jan  1  1970 libnssutil3.so
-rwxr-xr-x    1 root     root         22656 Jan  1  1970 libplc4.so
-rwxr-xr-x    1 root     root         14376 Jan  1  1970 libplds4.so
-rwxr-xr-x    1 root     root        163456 Jan  1  1970 libsmime3.so
-rwxr-xr-x    1 root     root        321992 Jan  1  1970 libsoftokn3.so
-rwxr-xr-x    1 root     root        410232 Jan  1  1970 libssl3.so
-rwxr-xr-x    1 root     root     149340504 Jan  1  1970 libxul.so
-rw-r--r--    1 root     root      30563863 Jan  1  1970 omni.ja
-rw-r--r--    1 root     root           166 Jan  1  1970 platform.ini
-rwxr-xr-x    1 root     root        642392 Jan  1  1970 plugin-container

So I copy them:

Code: Select all

cp -r /home/something/mnt/usr/* /usr/

And now Firefox works fine. But I can't do that manually later, it just proves that something weird happened with Firefox, or unpacking or mounting of .xzm file.

Any suggestions what I could try next? Either to fix Chrome + Alder Lake, or to fix / debug further whatever happened with Firefox booting.
Thanks in advance!

Luka

P.S. Just FYI these low-cost N100 mini-PCs are like made for Porteus, 100$ a piece, if only we can make it work smoothly!

[fanthom]

Luka,

Standard installation using USB stick works (Firefox module is downloaded from our website) so I believe the bug is related to your PXE boot environment.

Thanks

[LuxZg]

Thanks, we will recreate clean ISO both for Chrome and Firefox and see if we missed anything. I'll report here with the status.

[LuxZg]

Luka,

Standard installation using USB stick works (Firefox module is downloaded from our website) so I believe the bug is related to your PXE boot environment.

Thanks

Hi Tomasz,

We had a lot of testing today again, and here is what I can report:

• Yes, installing on local drive works fine. If Firefox is installed, the browser starts correctly and automatically on first screen. If i connect and run second instance via SSH, both browser instances will work, each on its own screen

• Likewise, if installed on local drive Chrome can run with modesetting, and again dual screen will work fine, each browser on its own screen, no graphical issues anymore

• Doing any of that over PXE will cause issues

• PXE boot of Chrome even with modesetting will run with same graphical issues as default gpu option

• PXE boot of Firefox will download 002-firefox.xzm but the files will NOT appear once OS is fully ready (only files in /usr/lib64/firefox are ones that are there even on "Chrome version", so probably packed in one of the other XZM files. As soon as I download and extract firefox.xzm manually browser will run (via gui-app that keeps retrying until files are finally there)

I don't see any issues with PXE itself. Running same procedure on 5.4.0 and previous devices will work fine. Logs show that files, specially 002-firefox.xzm, are indeed downloaded by the client.

What can I do to further debug? I can enable debug mode, but not sure what should I look for.

For Firefox:
Could you maybe point me to script that does that xzm downloading and extraction? If I'm looking correctly it's in initrd, "init" script. Those parts have error catching for not enough memory, or corrupted files, but I don't see any of those errors when booting. I just get wallpaper and firefox not starting.

For Chrome/modesetting:
Is there maybe a way to force modesetting even before OS pulls the config script? Should I add modesetting in the pxelinux/syslinux at the PXE boot?

Thanks in advance,
Luka

[fanthom]

Perhaps your '002-firefox.xzm' has incorrect permissions (or maybe its hidden?) on the server and thats why its not downloaded during PXE boot.
This is the only explanation why its being skipped.

Try to download this module over SSH from your PXE (http) server and not from porteus kiosk domain.

Also - check how the init script from initrd finds available modules on the http server and run relevant commands over SSH on the failing kiosk.
If 002-firefox.xzm is not there then you know where to look for.

Thanks

Added in 4 minutes 16 seconds:
Here is the command which you should run:

Code: Select all

for x in `wget -q -O- $IP/xzm | tr " " "\n" | grep -i "href.*.xzm</" | cut -d">" -f2 | cut -d"<" -f1`; do echo "$x" >>/tmp/modules-pxe; done

1) Replace $IP with the IP address of your http server.

2) run 'cat /tmp/modules-pxe' to find modules which will be downloaded during PXE boot.

Thanks

[LuxZg]

Good morning,

we did download the file from our local server as well, but we've done it today again just to confirm everything is in order after we've put up new files.

I've ran that command from "edit" of your answer, but that didn't produce the file at all. Still, it's probably something with the command or quotes, because that line is also needed for SSH to work, and kiosk is available via SSH so files are downloaded.

This shorter command listed all XZM files in the console output, as expected.

Code: Select all

wget -q -O- https://<our-local-server>/porteus/550ff/xzm | tr " " "\n" | grep -i "href.*.xzm</" | cut -d">" -f2 | cut -d"<" -f1

Output:

Code: Select all

000-kernel.xzm
001-core.xzm
002-firefox.xzm
003-settings.xzm
004-wifi.xzm
08-ssh.xzm

Then we've proceeded to download using these commands:

Code: Select all

wget -q -O 000-kernel.xzm https://<our-local-server>/porteus/550ff/xzm/000-kernel.xzm
wget -O 002-firefox.xzm https://<our-local-server>/porteus/550ff/xzm/002-firefox.xzm

Both files downloaded fine, no errors or warnings. Listed directory shows them on disk.

Code: Select all

-rw-r--r--    1 root     root      79937536 Apr 13  2023 000-kernel.xzm
-rw-r--r--    1 root     root      79208448 Mar 18  2023 002-firefox.xzm

Next we've tested mounting. I did my best to replicate the way that init script does it, though I can't find actual part where it should mount/copy files in case of PXE boot, so I used part of script that copies kernel and core modules.

Code: Select all

mkdir /opt/000
mount -o loop 000-kernel.xzm /opt/000
ls -al /opt/000

Directory listing shows files available on new mount

Code: Select all

total 0
drwxr-xr-x    5 root     root            73 Mar  6  2023 .
drwxr-xr-x    9 root     root           120 Jan 19 09:48 ..
drwxr-xr-x    3 root     root            54 Mar  6  2023 etc
lrwxrwxrwx    1 root     root             6 Apr 13  2023 lib -> lib64/
drwxr-xr-x    4 root     root            58 Jan  7  2018 lib64
drwxr-xr-x    4 root     root            49 Mar  6  2023 usr

So I repeat that for Firefox module:

Code: Select all

mkdir /opt/002
mount -o loop 002-firefox.xzm /opt/002
ls -al /opt/002

And again directory listing:

Code: Select all

total 0
drwxr-xr-x    4 root     root            38 Apr 21  2017 .
drwxr-xr-x    9 root     root           120 Jan 19 09:48 ..
drwxr-xr-x    3 root     root            28 Aug  4  2012 home
drwxr-xr-x    3 root     root            28 Jul  9  2019 usr

.. and finally looking into "firefox" folder to see if executable and other files are there

Code: Select all

ls -al /opt/002/usr/lib64/firefox/

And the output says all is fine

Code: Select all

total 185834
drwxr-xr-x    5 root     root           613 Mar 18  2023 .
drwxr-xr-x    3 root     root           261 Aug 25  2019 ..
-rw-r--r--    1 root     root           902 Jan  1  1970 application.ini
drwxr-xr-x    3 guest    guest           46 Mar 18  2023 browser
-rw-r--r--    1 root     root           174 Jan  1  1970 dependentlibs.list
-rwxr-xr-x    1 root     root         14416 Jan  1  1970 firefox
-rwxr-xr-x    1 root     root        646568 Jan  1  1970 firefox-bin
drwxr-xr-x    2 root     root            41 Mar 18  2023 fonts
drwxr-xr-x    3 root     root            26 Mar 18  2023 gmp-clearkey
-rwxr-xr-x    1 root     root        813432 Jan  1  1970 libfreeblpriv3.so
-rwxr-xr-x    1 root     root        355392 Jan  1  1970 libipcclientcerts.so
-rwxr-xr-x    1 root     root         43200 Jan  1  1970 liblgpllibs.so
-rwxr-xr-x    1 root     root       3841464 Jan  1  1970 libmozavcodec.so
-rwxr-xr-x    1 root     root        228136 Jan  1  1970 libmozavutil.so
-rwxr-xr-x    1 root     root         14056 Jan  1  1970 libmozgtk.so
-rwxr-xr-x    1 root     root        117384 Jan  1  1970 libmozsandbox.so
-rwxr-xr-x    1 root     root       1158728 Jan  1  1970 libmozsqlite3.so
-rwxr-xr-x    1 root     root         18224 Jan  1  1970 libmozwayland.so
-rwxr-xr-x    1 root     root        223024 Jan  1  1970 libnspr4.so
-rwxr-xr-x    1 root     root        707000 Jan  1  1970 libnss3.so
-rwxr-xr-x    1 root     root        437448 Jan  1  1970 libnssckbi.so
-rwxr-xr-x    1 root     root        187416 Jan  1  1970 libnssutil3.so
-rwxr-xr-x    1 root     root         22656 Jan  1  1970 libplc4.so
-rwxr-xr-x    1 root     root         14376 Jan  1  1970 libplds4.so
-rwxr-xr-x    1 root     root        163456 Jan  1  1970 libsmime3.so
-rwxr-xr-x    1 root     root        321992 Jan  1  1970 libsoftokn3.so
-rwxr-xr-x    1 root     root        410232 Jan  1  1970 libssl3.so
-rwxr-xr-x    1 root     root     149340504 Jan  1  1970 libxul.so
-rw-r--r--    1 root     root      30563863 Jan  1  1970 omni.ja
-rw-r--r--    1 root     root           166 Jan  1  1970 platform.ini
-rwxr-xr-x    1 root     root        642392 Jan  1  1970 plugin-container

Now we simply copy all that to

Code: Select all

/usr/lib64/firefox/

and browser starts by gui-app script automatically after that

Code: Select all

cp -r /opt/002/usr/* /usr/

So this should prove that HTTP server and PXE boot as such work fine. This was all done on the test device that was booted to that PXE "image", and I used SSH to connect to it and run these commands. So I know that it downloaded SSH module, it downloaded wallpapers, downloaded config (ssh password was our custom set one). And manual test confirms that permissions are ok, file is available, new logs from web server confirm that device downloaded 0002-firefox.xzm both at boot, and when we did it manually. XZM was extracted (mounted), files copied, and I am looking at browser running for past 15 minutes.

I'll try to fiddle with the command you provided so I can confirm that command can write the

Code: Select all

/tmp/modules-pxe

file, but like I said, I assume if it failed writing that file (or reading it at next steps) then SSH wouldn't work either (?)

EDIT: I must have copied something wrong in earlier tries, the command you gave does indeed work, and file is there.

Code: Select all

for x in `wget -q -O- http://<our server>/porteus/550ff/xzm | tr " " "\n" | grep -i "href.*.xzm</" | cut -d">" -f2 | cut -d"<" -f1`; do echo "$x" >>/tmp/modules-pxe; done

And reading the resulting file:

Code: Select all

cat /tmp/modules-pxe
000-kernel.xzm
001-core.xzm
002-firefox.xzm
003-settings.xzm
004-wifi.xzm
08-ssh.xzm

Firefox file included.

[LuxZg]

Could you please give me info about this?

For Chrome/modesetting:
Is there maybe a way to force modesetting even before OS pulls the config script? Should I add modesetting in the pxelinux/syslinux at the PXE boot?

For example if this is what we use to boot through syslinux:

Code: Select all

LABEL Porteus-unpacked-550-firefox-pxe-single
	MENU LABEL Boot Porteus Kiosk 5.5.0 Firefox single PXE
	MENU DEFAULT
	KERNEL ../porteus/550ff/boot/vmlinuz
	APPEND initrd=../porteus/550ff/boot/initrd.xz,../porteus/550ff/boot/pxelinux.cfg/initrdpxe.xz quiet video=1920x1080 http_server=<local_server_ip>:80/porteus/550ff

Can we add something to APPEND line that would trigger gpu drive modesetting directly there?

[fanthom]

1) Maybe its the SSL certificate fault. Please disable it temporarily and see if that helps.

2) You could do this by modifying the init script in the initrd (update files directly in /union to force the modesetting driver).
This is considered as 'user made customization' which I do not support. Please do not report any bugs from your custom setup.

Thanks

[LuxZg]

1) It is not SSL, because all other files are downloaded and Chrome version boots with browser and additional modules (like SSH, VNC etc). So I still don't know why only 002-firefox.xzm isn't being mounted/copied into filesystem

2) I did not change initrd as that would require too many changes to the system, plus, as you say, it is unsupported. I still don't know why PXE boot ignores the

Code: Select all

gpu_driver=modesetting

but I may have found a workaround.Steps I did just now (manually, so just for test)

Code: Select all

cd /etc/X11/xorg.conf.d/
mcedit 20-intel.conf

Added new line just before EndSection:

Code: Select all

Driver "modesetting"

Next I had to restart X server, don't know nicer way so I just killed it:

Code: Select all

killall -9 X

It will restart by scripts, but the display will be +1 so after few trials I ended up with ":4" but if it's done just one then it will move from ":0" to ":1"
This new number needs to be setup in script here:

Code: Select all

mcedit /etc/profile.d/variables.sh

And in the last line change the display number:

Code: Select all

DISPLAY=:1

Immediately wallpaper will show, and after few moments Chrome will start on both screens, two instances, different pages, no graphical issues observed so far, animations work, text is readable, no shaking and skipping.

Since it's Friday I'll continue on Monday, I will try putting this into a startup script or via

Code: Select all

run_command=

. If you know how to restart X without forcefully killing it, so it keeps the display=:0 it would be even better. Maybe I dig it out next week. But even without that I think I can hack a one-liner with "sed" to add the line in intel.conf, change line in "variables.sh", then kill X and let it restart on its own.

[fanthom]

You can use 'openbox --exit' command to restart the GUI session without creating a new display.

Thanks

[LuxZg]

You can use 'openbox --exit' command to restart the GUI session without creating a new display.

Thanks

Thanks. We've figured out that by ourselves eventually (same day when you posted that actually), with bunch of trial and error and digging through the existing scripts.

This is our current workaround, adding this to config fixes Chrome and dual screen display on Alder Lake N100 PC:

Code: Select all

run_command=test -f mst-file && echo \"skip\" || (touch mst-file ; sleep 1 ; sed -i '4i Driver \"modesetting\" /etc/X11/xorg.conf.d/20-intel.conf && rm -r /tmp/.X* && sleep 2 && openbox --exit ;

Removing the tmp/.X files is probably not necessary but now that it works I'm leaving it like that for one day to see if any quirks pop up (note: we did leave it for 2 days, no issues)

The test for "mst-file" is here because I've noticed it runs command whenever X is killed, so I ended up with bunch of same modesetting lines in same config file. This way it will create "mst-fil" on first run and use as check, and if it exists already it will skip re-adding modesetting line or restarting X & desktop session, so I am sure it only runs once. It's hacky, but works.

I still don't understand why having "gpu-driver=modesetting" in that same config file doesn't get applied, and doing it this way works, I mean, config is the same file, from same source, if it wasn't reading it from HTTP/S or anything similar it wouldn't work now either.

Hopefully 5.6.0 is coming in March and will properly support Alder Lake so we can put this hack behind us soon But new hardware is now on its way, and will probably be here mid February, and we'll just have to survive first month or two until 5.6.0 lands, and deploy them like this.

Cheers!
Luka

P.S. Sorry for not replying in a while, I wrote a reply but it got lost when I tried to submit and I forgot to re-type everything when I had time.
P.P.S. And sorry if I made any typos, I was retyping this from a screenshot when submit form froze :-/ There's some issue when tab is open with this forum, and you just refresh (even Ctrl+F5), as I keep getting "The form token was outdated" and then it just hangs there , I waited like 10 minutes last time it happened, then took screenshot and closed the tab.

[fanthom]

"I still don't understand why having "gpu-driver=modesetting" in that same config file doesn't get applied"
Let me guess - because this is not a correct parameter name?

At least you have learnt something new when debugging a non existing problem

[LuxZg]

"I still don't understand why having "gpu-driver=modesetting" in that same config file doesn't get applied"
Let me guess - because this is not a correct parameter name?

At least you have learnt something new when debugging a non existing problem

You're referring to the dash instead underscore, but that was typo in the post comment, as I didn't copy paste from config when retyping from the screenshot of a failed submit form. When configuring actual file, I always copy from the parameters help page and I did copy "gpu_driver=modesetting" from there multiple times into our config files, I am not stupid, lazy, incompetent, nor do I appreciate it being suggested to be even when you add a smiley at the end.

I wrote previous reports in early posts of this thread, and you can see it was written correctly there (where I did copy/paste directly from config, as that time I actually managed to submit form without the errors)

Also please fix your error prompting a message, and I quote and copy/paste:

Loading
The form token was outdated and has been updated.
Submitting the form again...

Even in separate browser on separate computer, when opening this page 5.5.0 & Firefox - file not found? logging into forum and typing just about anything, trying to review or submit throws this error and I need to spend 20 minutes to post a simple message here, which also prompts retyping and typing mistake ON THE FORUM, NOT IN THE CONFIG FILE.

But yes, thank you for your time. I hope I won't be needing to debug your mistakes anytime soon again.

[fanthom]

Luke,

Please check on a standard installation (not a PXE boot) and I'm 100% sure that 'gpu_driver=modesetting' will work correctly.

Thanks

[LuxZg]

Luke,

Please check on a standard installation (not a PXE boot) and I'm 100% sure that 'gpu_driver=modesetting' will work correctly.

Thanks

Is PXE unsupported? If it does work on HDD, and doesn't work on PXE, what does it mean? Reast of configuration file gets applied correctly. Looking at scripts, I see that Porteus Kiosk enters two different logical branches one if booting from network, other when NOT booting from network. I do not see anything that would cause some of the parameters to be skipped, but I'm not an advanced programmer. It is still logically possible that these two branches of code do not function the same, so "if boot from network" brach could be ignoring GPU modesetting in some way. It is your script, you should know the best. If this parameter is not supported for network boot, then documentation should state so on the "parameters" web page. But since you do have PXE booting enabled, perhaps you should just boot once with this parameter over PXE and see if it works for you. If it does, then fine, it's my fault. But otherwise, I am using unmodified installation, just over PXE, and gpu driver parameter gets ignored for some reason. I have solved my issue with a workaround, I am now writing this in a good faith trying to help the project by pointing to possible issues in boot logic.