why modules

Technical issues/questions of an intermediate or advanced nature.
rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#31 by rych » 17 Nov 2022, 13:19

gnintilgyes wrote:
16 Nov 2022, 05:23
it could greatly increase the loading times, either during starting before desktop, or on demand
Yes, activating modules involves updating system indexes, databases etc. and so take noticeable time. Mounting the same xzm module read-only is instantaneous though.

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#32 by rych » 17 Nov 2022, 13:22

beny wrote:
14 Nov 2022, 13:33
list them
Well, any software released as AppImage for example essentially is a squashfs mounted read-only and seems to run (almost?) just as fast as from unpacked and installed properly into a folder, right?

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#33 by rych » 17 Nov 2022, 13:26

Rava wrote:
14 Nov 2022, 23:03
approach with either audacity-linux-3.2.0-x86_64.xzm or audacity-linux-3.2.1-x86_64.xzm and report how well that went?
"Audacity 3.2.0 is available as an AppImage" https://www.audacityteam.org/download/linux/ -- surelly AppImage does not modify itself when running, and stores settings, cache etc. in some other (user) folders. So I'm guessing audacity-linux-3.2.0-x86_64.xzm does the same when simply mounted read-only

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#34 by Rava » 17 Nov 2022, 16:33

rych wrote:
17 Nov 2022, 13:26
So I'm guessing audacity-linux-3.2.0-x86_64.xzm does the same when simply mounted read-only
Mine did not, I made the mistake when running the activated audacity-linux-3.2.0-x86_64.xzm (or audacity-linux-3.2.1-x86_64.xzm) in its default settings and opening a larger audio file, it filled up all of / due to that also being its path for its cache folder.
And when / is full no programs can write to any of their settings files, nor could audacity, and thus audacity threw around many error messages due to not being able to write to its folder in guest's ~/ folder.


That's the reason I specifically told you to mount-only audacity-linux-3.2.0-x86_64.xzm instead of activating it since in my case without the ability to write to its settings file(s) somewhere in ~/ it went bonkers.
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#35 by rych » 19 Nov 2022, 06:59

So I now treat some of my modules as simply squashfs compressed images (which they are essentially). I've made additional context menu entries for them and a little helper script as follows:

Code: Select all

root@porteus:~# cat /root/.local/share/applications/squashfs-mount.desktop 
[Desktop Entry]
Name=Mount here
Icon=cdr
Exec=squashfs_mount %F
MimeType=application/vnd.squashfs

root@porteus:~# cat /root/.local/share/applications/squashfs-unmount.desktop 
[Desktop Entry]
Name=Unmount
MimeType=application/vnd.squashfs
Icon=cdr
Exec=umount %F

root@porteus:~# cat /P/scripts/squashfs_mount
#!/bin/bash
mnt_name=${1%.*}
umount $mnt_name
mkdir $mnt_name
mount $1 $mnt_name
-- mounting read-only an app stored as an .sqfs file on a fast SSD onto the path minus the extension.

Furthermore, if you'd rather extract it into the same path, just modify slightly the /usr/share/applications/pcmanfm-extract-module.desktop to have

Code: Select all

MimeType=application/x-xzm;application/vnd.squashfs
So that the Extract context menu entry applies to .squashfs images as well. But it might take a while if the app is several GB.

Finally, and the most importantly let's make an entry for turning any folder into a .squashfs image. Firstly we can build an zstd module and then rename the extension, but it can be slow for large folders, as it uses the hardest compression parameter. Also, it's interesting to see how one turns a folder into an AppImage https://docs.appimage.org/packaging-guide/manual.html. Anyway we simple add another context menu command:

Code: Select all

root@porteus:~# cat /usr/share/applications/squashfs-create.desktop 
[Desktop Entry]
Name=Build SquashFS
Icon=cdr
MimeType=inode/directory
NoDisplay=true
Terminal=true
Exec=squashfs_create %F
and the squashfs_create script is just:

Code: Select all

#!/bin/bash
mksquashfs $1 $1.sqfs -noappend -comp zstd -b 256K
Last edited by rych on 22 Nov 2022, 07:51, edited 5 times in total.

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#36 by Rava » 19 Nov 2022, 07:17

^
Thanks for these.

rych, did you try mount-only audacity-linux-3.2.0-x86_64.xzm instead of activating it ?

Added in 33 minutes 42 seconds:
rych wrote:
19 Nov 2022, 06:59
umount $mnt_name
what if that umount failes, e.g. because the user has a terminal open and is in that path of the previously mounted module?
Then

Code: Select all

mkdir $mnt_name
would also fail, and so would

Code: Select all

mount $1 $mnt_name
Also, what would happen if the module name contains whitespace or whitespaces (which is perfectly legal for a UNIX file name, hell, even having more than one line aka also containing a newline character is a valid UNIX filename, even when that would break most scripts that handle filenames)?
You not use " to escape $mnt_name therefore it would also fail when the module name contains whitespace or whitespaces.
Let us not go to the handling of file names that contain a newline character since that breaks many if not most Porteus scripts - but the handling of whitespace character in the module name and thus the created folder name should be made possible, or else your solution fails.

just try it: rename a module into containing a whitespace.

Presuming you have a audacity-linux-3.2.0-x86_64.xzm for testing purposes, copy that as such:

Code: Select all

cp audacity-linux-3.2.0-x86_64.xzm "audacity-linux-3.2.0-x86_64 with whitespace.xzm"
and then try using your above solution for mounting

Code: Select all

audacity-linux-3.2.0-x86_64 with whitespace.xzm
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#37 by rych » 19 Nov 2022, 08:01

Rava wrote:
19 Nov 2022, 07:51
try mount-only audacity-linux-3.2.0-x86_64.xzm instead of activating it
You already have that option in the context menu: "Mount" which mounts it under /mnt/loop
Rava wrote:
19 Nov 2022, 07:51
if that umount failes
I wonder if the subsequent mount fails as well, or will it be mounted over?!
Rava wrote:
19 Nov 2022, 07:51
use " to escape $mnt_name
so should I use it everywhere then for example: mount "$1" "$mnt_name"

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#38 by Rava » 19 Nov 2022, 08:03

rych wrote:
19 Nov 2022, 08:01
so should I use it everywhere then for example: mount "$1" "$mnt_name"
Yes.

Added in 2 minutes 12 seconds:
rych wrote:
19 Nov 2022, 08:01
I wonder if the subsequent mount fails as well, or will it be mounted over?!
Good question, just create the error condition (mount the same module first, then open a terminal and move into its mounted path and stay there) and then try mounting it via the GUI of your DE again and look what happens.

Added in 9 minutes 8 seconds:
To simulate the handling of white spaces using your above squashfs_mount as globally accessible /usr/local/bin/squashfs_mount :

Code: Select all

guest@porteus:/tmp$ mkdir mount-test
guest@porteus:/tmp$ cd mount-test/
guest@porteus:/tmp/mount-test$ cp /Porteus_modules/audacity-linux-3.2.0-x86_64.xzm .
guest@porteus:/tmp/mount-test$ cp audacity-linux-3.2.0-x86_64.xzm "audacity-linux-3.2.0-x86_64 with whitespace.xzm"
guest@porteus:/tmp/mount-test$ ls -o
total 20544
-rw-r--r-- 1 guest 10518528 2022-10-21 13:50 audacity-linux-3.2.0-x86_64\ with\ whitespace.xzm
-rw-r--r-- 1 guest 10518528 2022-10-21 13:50 audacity-linux-3.2.0-x86_64.xzm
guest@porteus:/tmp/mount-test$ cat /usr/local/bin/squashfs_mount 
#!/bin/bash
mnt_name=${1%.*}
umount $mnt_name
mkdir $mnt_name
mount $1 $mnt_name
guest@porteus:/tmp/mount-test$ squashfs_mount audacity-linux-3.2.0-x86_64\ with\ whitespace.xzm 
umount: audacity-linux-3.2.0-x86_64: No such file or directory
mount: bad usage
Try 'mount --help' for more information.
guest@porteus:/tmp/mount-test$ echo $?
1
guest@porteus:/tmp/mount-test$ ls -o
total 20544
drwxr-xr-x 2 guest       40 2022-11-19 09:10 audacity-linux-3.2.0-x86_64
-rw-r--r-- 1 guest 10518528 2022-10-21 13:50 audacity-linux-3.2.0-x86_64\ with\ whitespace.xzm
-rw-r--r-- 1 guest 10518528 2022-10-21 13:50 audacity-linux-3.2.0-x86_64.xzm
drwxr-xr-x 2 guest       40 2022-11-19 09:10 whitespace
drwxr-xr-x 2 guest       40 2022-11-19 09:10 with
guest@porteus:/tmp/mount-test$ 
As you can see, your old version created three folders:

Code: Select all

audacity-linux-3.2.0-x86_64
whitespace
with
instead of only one named

Code: Select all

audacity-linux-3.2.0-x86_64 with whitespace
and the mount did also fail.

Added in 7 minutes 57 seconds:
Tip: mount did fail because it was given too much parameters.
Instead of

Code: Select all

mount filename foldername
it was given

Code: Select all

mount name1 name2 name3 name4 name5 name6
I created a lame echo DEBUG version of /usr/local/bin/squashfs_mount to display what's given to mount:

Code: Select all

guest@porteus:/tmp/mount-test$ rmdir audacity-linux-3.2.0-x86_64 whitespace with
guest@porteus:/tmp/mount-test$ cat /usr/local/bin/squashfs_mount 
#!/bin/bash
mnt_name=${1%.*}
umount $mnt_name
mkdir $mnt_name
echo DEBUG: ██ mount $1 $mnt_name ██
mount $1 $mnt_name
guest@porteus:/tmp/mount-test$ squashfs_mount audacity-linux-3.2.0-x86_64\ with\ whitespace.xzm 
umount: audacity-linux-3.2.0-x86_64: No such file or directory
DEBUG: ██ mount audacity-linux-3.2.0-x86_64 with whitespace.xzm audacity-linux-3.2.0-x86_64 with whitespace ██
mount: bad usage
Try 'mount --help' for more information.
guest@porteus:/tmp/mount-test$ 
HTH.

Added in 4 minutes 18 seconds:
(I use these kinds of

Code: Select all

echo DEBUG [whatever] 
all the time to debug scripts of mine until they are as bug-free as I manage to code them.
Then I either remove the echo DEBUG lines or deactivate them via

Code: Select all

#echo DEBUG [whatever] 
)
Cheers!
Yours Rava

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#39 by Rava » 20 Nov 2022, 14:57

rych, did you manage to make your squashfs_mount more robust?

Added in 10 hours 1 minute 48 seconds:
I pimped my ride my local /usr/local/bin/squashfs_mount a little bit but it still has not enough error checks.

Code: Select all

root@porteus:/usr/local/bin# cat squashfs_mount
#!/bin/bash
mnt_name=${1%.*}
umount "$mnt_name"
mkdir "$mnt_name"
echo DEBUG: ██ mount "$1" "$mnt_name" ██
mount "$1" "$mnt_name"
like when it's called without any parameter we get this:

Code: Select all

root@porteus:/usr/local/bin# squashfs_mount
umount: : no mount point specified.
mkdir: cannot create directory ‘’: No such file or directory
DEBUG: ██ mount   ██
ntfs-3g: Failed to access volume '': No such file or directory

ntfs-3g 2021.8.22 integrated FUSE 27 - Third Generation NTFS Driver
		Configuration type 1, XATTRS are on, POSIX ACLS are off

Copyright (C) 2005-2007 Yura Pakhuchiy
Copyright (C) 2006-2009 Szabolcs Szakacsits
Copyright (C) 2007-2021 Jean-Pierre Andre
Copyright (C) 2009-2020 Erik Larsson

Usage:    ntfs-3g [-o option[,...]] <device|image_file> <mount_point>

Options:  ro (read-only mount), windows_names, uid=, gid=,
          umask=, fmask=, dmask=, streams_interface=.
          Please see the details in the manual (type: man ntfs-3g).

Example: ntfs-3g /dev/sda1 /mnt/windows

Plugin path: /usr/lib64/ntfs-3g

News, support and information:  http://tuxera.com
root@porteus:/usr/local/bin# echo $?
11
now, when we try the same test as in my post above with the module with whitespace in its name squashfs_mount creates the correct folder and also tries mounting the correct module, but it could still fail:

Code: Select all

guest@porteus:/tmp/mount-test$ squashfs_mount audacity-linux-3.2.0-x86_64\ with\ whitespace.xzm 
umount: audacity-linux-3.2.0-x86_64 with whitespace: No such file or directory
DEBUG: ██ mount audacity-linux-3.2.0-x86_64 with whitespace.xzm audacity-linux-3.2.0-x86_64 with whitespace ██
mount: audacity-linux-3.2.0-x86_64 with whitespace: failed to setup loop device for /tmp/mount-test/audacity-linux-3.2.0-x86_64 with whitespace.xzm.
guest@porteus:/tmp/mount-test$ ls
audacity-linux-3.2.0-x86_64\ with\ whitespace
audacity-linux-3.2.0-x86_64\ with\ whitespace.xzm
audacity-linux-3.2.0-x86_64.xzm
As you can see, the echo DEBUG line cannot distinguish between the parameters set in " or not set in "

And of course since the above error is about the loop devices it also fails trying to mount the module without whitespace in its name:

Code: Select all

guest@porteus:/tmp/mount-test$ squashfs_mount audacity-linux-3.2.0-x86_64.xzm 
umount: audacity-linux-3.2.0-x86_64: No such file or directory
DEBUG: ██ mount audacity-linux-3.2.0-x86_64.xzm audacity-linux-3.2.0-x86_64 ██
mount: audacity-linux-3.2.0-x86_64: failed to setup loop device for /tmp/mount-test/audacity-linux-3.2.0-x86_64.xzm.
guest@porteus:/tmp/mount-test$ ls
audacity-linux-3.2.0-x86_64
audacity-linux-3.2.0-x86_64\ with\ whitespace
audacity-linux-3.2.0-x86_64\ with\ whitespace.xzm
audacity-linux-3.2.0-x86_64.xzm
guest@porteus:/tmp/mount-test$ ls -o */
audacity-linux-3.2.0-x86_64 with whitespace/:
total 0

audacity-linux-3.2.0-x86_64/:
total 0
guest@porteus:/tmp/mount-test$ du -scm */
0	audacity-linux-3.2.0-x86_64 with whitespace/
0	audacity-linux-3.2.0-x86_64/
0	total
and both newly created folders for mount are therefore empty.

Added in 15 minutes 1 second:
About the error checking, adding a check for one parameter and one parameter only is no great issue:
The below code uses my colour codes as described here: Ravas coding goodies (Post by Rava #91503)
also, it uses $MYNAME to refer to its own name.

Code: Select all

if [ $# -ne 1 ]; then
	echo -e "$red$bold $MYNAME ERROR -- You must specify exactly one parameter.$off"
	exit 1
fi
if [ ! -f "$1" ]; then
	echo -e "$red$bold $MYNAME ERROR -- file "$1" not exists. Aborting.$off"
	exit 2
fi
thus we get this code (I added a version number and added more info on what the parameter is about):

Code: Select all

#!/bin/bash
# blame Rava, this script is based on an idea by rych - see details here: https://forum.porteus.org/viewtopic.php?p=91537#p91537
MYNAME=squashfs_mount
MYVERSION=2022-11-21
bld="\033[1m"	# bold
red="\033[31m"	# red
off="\033[m"	# off

if [ $# -ne 1 ]; then
	echo -e "$red$bold$MYNAME V$MYVERSION ERROR -- You must specify exactly one parameter (=the file to me mounted).$off"
	exit 1
fi
if [ ! -f "$1" ]; then
	echo -e "$red$bold$MYNAME V$MYVERSION ERROR -- file "$1" not exists. Aborting.$off"
	exit 2
fi

mnt_name=${1%.*}
umount "$mnt_name"
mkdir "$mnt_name"
echo DEBUG: ██ mount "$1" "$mnt_name" ██
mount "$1" "$mnt_name"
It handles file names with spaces
checks if exactly one parameter was given
checks if file exists


TODO
not checks if umount was even necessary or if a needed umount did fail
not checks if folder could be created (could be e.g. the folder-to-be-created exists as a file owned by another user that the current user cannot remove)

Added in 7 minutes 53 seconds:
Here I simulate both errors, one is using a non-existing file name, the other giving no parameter:
guest@porteus:/tmp/mount-test$ squashfs_mount audacity-linux-3.2.0-x86_64\ with\ whitespace-nonexisting.xzm
squashfs_mount V2022-11-21 ERROR -- file audacity-linux-3.2.0-x86_64 with whitespace-nonexisting.xzm not exists. Aborting.
guest@porteus:/tmp/mount-test$ squashfs_mount
squashfs_mount V2022-11-21 ERROR -- You must specify exactly one parameter (=the file to me mounted).
guest@porteus:/tmp/mount-test$
Cheers!
Yours Rava

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#40 by Rava » 21 Nov 2022, 01:34

I could have also added the line breaking of long lines as described here Ravas coding goodies (Post by Rava #90829)
but I leave that out for now, it can be added at the very end, for now it only complicates the code and it is purely aesthetic anyway.

Added in 49 minutes 15 seconds:
info

The issue with loop device failing as guest user is solved when squashfs_mount is run as root.

Done so as test doing the xzm mounting manually:

Code: Select all

guest@porteus:/Porteus_modules$ mkdir /tmp/abiword-3.0.5-x86_64-1_SBo_bundle
guest@porteus:/Porteus_modules$ mount abiword-3.0.5-x86_64-1_SBo_bundle.xzm /tmp/abiword-3.0.5-x86_64-1_SBo_bundle/
mount: /tmp/abiword-3.0.5-x86_64-1_SBo_bundle/: failed to setup loop device for /mnt/sda5/Porteus_modules/5.0/abiword-3.0.5-x86_64-1_SBo_bundle.xzm.
fails as guest

Code: Select all

root@porteus:/Porteus_modules# mount abiword-3.0.5-x86_64-1_SBo_bundle.xzm /tmp/abiword-3.0.5-x86_64-1_SBo_bundle/
root@porteus:/Porteus_modules# df -m /tmp/abiword-3.0.5-x86_64-1_SBo_bundle/
Filesystem     Type     1M-blocks  Used Available Use% Mounted on
/dev/loop18    squashfs         9     9         0 100% /tmp/abiword-3.0.5-x86_64-1_SBo_bundle
succeeds a root
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#41 by rych » 22 Nov 2022, 07:54

I've completed my post above and we now have 4 actions for our .sqfs "modules":
* Mount (here)
* Unmount
* Extract (here)
* Create (from a folder)

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#42 by rych » 22 Nov 2022, 08:53

Example. Let's practice on Blender. First let's use ncmprhnsbl's update-blender to make us a blender xzm. And that's going to be the only updating or changing the installed blender, so a good candidate for a read-only mount. Supppose, we mounted our host fast SSD drive to /H. Let's mount the blender xzm and repackage it's main folder:

Code: Select all

root@porteus:/H# mksquashfs /mnt/loop/opt/blender-3.3 /H/blender-3.3.sqfs -noappend -comp zstd -b 256K
Instead of the 201MB blender-3.3.1-x86_64-en-US.xzm we now have a 227MB blender-3.3.sqfs. Could have probably used a harder compression. But it's still better than installing blender into a folder which would take around 700MB! Now we'll just be mounting our sqfs. The mounting folder will appear as if it occupies 700MB but will really be serving out of the squashfs image.

There are a couple of other important files inside the xzm: a symlinked blender binary and a menu .desktop file which we could manually copy to our system. That's called "integration into the system" and isn't a part of blender itself.

Basically, the .xzm in this case is already fantastic and easily made thanks to ncmprhnsbl and there is no real need for recasting it as a .sqfs. But other examples could include portable installations of TeXLive, conda, Mathematica, Matlab etc. which if stored fully extracted would take up tens of GB. Yet, as mounted .sqfs images, -- take much smaller space, while apparently run as fast.

User avatar
Rava
Contributor
Contributor
Posts: 5401
Joined: 11 Jan 2011, 02:46
Distribution: XFCE 5.01 x86_64 + 4.0 i586
Location: Forests of Germany

why modules

Post#43 by Rava » 22 Nov 2022, 20:28

rych wrote:
22 Nov 2022, 08:53
portable installations of TeXLive
Is there a TeXLive or LaTeX module creation script for Porteus?

What I dearly miss for years is working with TeX again, the output it creates is so many miles ahead in quality compared to all the Office Writing Programs or Suites out there.

TeX or LaTeX is on a whole other level compared to just some office suite. Image

I tried building one myself some year ago but it was not very successful. :(
Cheers!
Yours Rava

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#44 by rych » 03 Dec 2022, 08:32

Rava wrote:
22 Nov 2022, 20:28
Is there a TeXLive or LaTeX module creation script for Porteus?
Almost. There is the official install-tl script which based on your answers will download and install a (portable, minimal, etc.) TeXLive on your system, and I recommend a portable installation into a new folder. Then you just turn that folder into either .xzm (a squashfs module).

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

why modules

Post#45 by rych » 03 Dec 2022, 08:54

I've implemented a faster storage for the porteus modules: on the SSD host disk. It's better to be done as early as other modules, at startup, so I plugged the following logic into linuxrc:

1. Find the host disk
2. Update a certain host folder with modules from a porteus USB folder (e.g. $PORTDIR/hosted)
3. Add the new path as an additional source of modules to be activated

So, most if not all my modules will be running off the faster SSD system disk. If I occasionally boot on a machine without (access to) a host disk, all the modules will be run from the $PORTDIR/hosted.

The idea is that programs portably stored on a USB shouldn't really be running from a slow USB (3.0, 3.1 are still slower than NVMe SSD for example; maybe USB 3.2 will be as fast?) ... even as compressed xzm. When we boot on a machine where we have an access to a faster storage, modules and other data are better be deployed there.

Post Reply