Porteus with zstd compression

[jssouza]

so: .xzms and -zstd.xzms will activate at boot, and a lot of the gui scripts will work oftb .ie activate/deactivate/extract

The only problem here I noticed is that there is no way to check what type of compression has been used to create the module. A file command just says Squashfs filesystem, not the compression used. Which rather makes the extension a necessary indicator to see what compression is used.

[babam]

Why not make it simple.
mymodule.xzm ---> Modules with xz compression
mymodule.zstd.xzm ---> modules with zstd compression

zactivate, zdeactivate and modify initrd (linuxrc) are not needed.

[ncmprhnsbl]

in what context is it needed to determine the compression used?
i mean. it's good to know, that's what the -zstd.xzm tag is for (or .zstd.xzm) and that could be parsed for..
but eg. unsquashfs works out what to do... slax's .sb or puppy's .sfs give no indication of compressor and seem to get by..

i have noticed that if i extract a -zstd.xzm and recompress it , it becomes -zstd-zstd.xzm.. so that's not exactly seamless
but hey, i'm just messing about and seeing what i can break ")

[neko]

==== Porteus "ZSTD compression" Trial Version ====

Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso was converted from Porteus-OPENBOX-v5.0rc1-x86_64.iso.

Step 1.
Kernel of Porteus-OPENBOX-v5.0rc1-x86_64.iso was changed to 5.1.15 (build with config "CONFIG_SQUASHFS_ZSTD=y").
refer to Porteus Kernel Builder (Post by neko #72505)

Step 2.
ISO@/porteus/base/000-kernel.xzm, 001-core.xzm, 002-xorg.xzm, 003-openbox.xzm were rebuilt with "ZSTD compression".

the script of making XZM(the naming is for old interface)

Code: Select all

#! /bin/sh
SQ=`basename $1`
SQ=${SQ}.xzm

rm -f ${SQ}
mksquashfs $1/ ${SQ} -comp zstd -b 256K -Xcompression-level 22

Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso (359 M)
http://www.mediafire.com/file/dw7wft2nl ... x86_64.iso
md5sum: bc9027054f96983eb7205453ac90340b Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso


Thanks.

[ncmprhnsbl]

here's a module with various files/fixes to enable the majority of GUI filemanager actions specific to .zstdm modules..
stick it in modules(or base) or if you're booting the ISO use cheatcode: extramod=/path/to/folder/where/it/is/ (thanks Ed_P)
ZTREE.zstdm
module contains:

Code: Select all

$tree -al ZTREE
├── home
│   └── guest
│       └── .config
│           └── mimeapps.list
├── opt
│   └── porteus-scripts
│       ├── context-menu
│       │   ├── create-zmodule
│       │   ├── extract-zmodule
│       │   └── zconvert_txz
│       ├── mloop
│       ├── txz2zstdm
│       ├── xorg
│       │   └── zxactivate
│       └── zactivate
├── root
│   └── .config
│       └── mimeapps.list
└── usr
    ├── bin
    │   ├── dir2zstdm -> /opt/porteus-scripts/dir2zstdm
    │   └── txz2zstdm -> /opt/porteus-scripts/txz2zstdm
    ├── local
    │   └── bin
    │       └── lsmodules
    └── share
        ├── applications
        │   ├── mimeapps.list
        │   ├── pcmanfm-create-zmodule.desktop
        │   ├── pcmanfm-extract-module.desktop
        │   ├── pcmanfm-mount-module.desktop
        │   ├── pcmanfm-unmount-module.desktop
        │   ├── pcmanfm-zconvert-txz-to-xzm.desktop
        │   ├── zactivate.desktop
        │   └── zdeactivate.desktop
        ├── icons
        │   └── Adwaita
        │       ├── 16x16
        │       │   └── mimetypes
        │       │       └── application-x-zstdm.png -> /usr/share/icons/Adwaita/16x16/mimetypes/application-x-xzm.png
        │       ├── 22x22
        │       │   └── mimetypes
        │       │       └── application-x-zstdm.png -> /usr/share/icons/Adwaita/22x22/mimetypes/application-x-xzm.png
        │       ├── 32x32
        │       │   └── mimetypes
        │       │       └── application-x-zstdm.png -> /usr/share/icons/Adwaita/32x32/mimetypes/application-x-xzm.png
        │       ├── 48x48
        │       │   └── mimetypes
        │       │       └── application-x-zstdm.png -> /usr/share/icons/Adwaita/48x48/mimetypes/application-x-xzm.png
        │       └── 64x64
        │           └── mimetypes
        │               └── application-x-zstdm.png -> /usr/share/icons/Adwaita/64x64/mimetypes/application-x-xzm.png
        └── mime
            └── packages
                └── freedesktop.org.xml

29 directories, 26 files

notes:
these files would normally be spread across 001-core, 002-xorg, 003-openbox
zactivate is included for some edits regarding zxactivate
mloop is expanded to include .zstdm
lsmodules only handles .zstdm. xzm s are not seen or handled.
this is for jssouzas ISO .. not sure i understand the nature of nekos ISO..(edit: ZTREE.zstdm is not useful as is, for nekos ISO)

[Ed_P]

if you're booting the ISO use cheatcode: load=/path/to/where/it/is/ZTREE

Not the extramod= cheatcode?

[neko]

@ncmprhnsbl
[Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso]
The same naming rule (sufix is .xzm) is used for "ZSTD compression" module and "XZ compression" module.
It allows mixed xzm modules.
No more difference is between Porteus-OPENBOX-v5.0rc1-x86_64.iso and Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso.

Thanks.

[ncmprhnsbl]

Not the extramod= cheatcode?

you're right Ed, load= is only for the /optional folder .. i'll fix my post..

No more difference is between Porteus-OPENBOX-v5.0rc1-x86_64.iso and Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso.

thanks neko, that's what i suspected.. which means my ZTREE.zstdm would not be useful for that IS0.

[babam]

Even with the kernel built by neko Porteus Kernel Builder, it allows Porteus 3.2.2 to support modules with zstd compression without modifying anything.

[brokenman]

Do you guys think it is worth converting to zstd?

[ncmprhnsbl]

Do you guys think it is worth converting to zstd?

i'm more of the opinion of having support(kernel,squashfs-tools,scripts) for all of the above (xz, zstd, lz4, even gzip)
keep official iso xz for the smallest size, and then the user can convert the base modules to whatever, if they value speed over size..
i'm currently running a mixture(zstd, xz) simply because i haven't bothered to convert everything to zstd..
my zstd modules are named name-zstd.xzm to keep track of them..
correct me if i'm wrong: decompression times don't have an effect beyond activation(therefore boot times), ie. once loaded to the union there should be no difference in performance..

[burdi01]

correct me if i'm wrong: decompression times don't have an effect beyond activation(therefore boot times), ie. once loaded to the union there should be no difference in performance..

For as far as I am aware squashfses loaded in the union are still in compressed format -- it would take a lot of memory and cpu cycles to decompress them in their entirety at once. Therefore the decompression has to be done upon accessing files in them -- which spreads the load on memory and cpu cycles.
In other words there is a difference in performance but it is less noticeable than one might expect on a first impulse.

[fanthom]

To make things more complicated:

Linux caches everything in RAM so if you have plenty of it then system reads (decompresses) data from the modules only once.
This is pretty much the kiosk case: system loads the browser and nothing more. First load is slower but browser restarts are super fast cause are libs, images, other files are cached in RAM (which is fastest storage device available - maybe Optane storage is equal).

In case of Porteus Destop there can be more apps opened so module compression matters more.
Every time the system is running out of RAM the read from modules (decompression) happens and you can save on CPU cycles if you pick zstd compression.

Situation is also different if you use 'copy2ram' feature.

If you enable copy2ram then zstd is a clear win.

However - if modules are stored on a slow device rather than in RAM (e.g. ancient SD card or usb stick plugged to a USB 1.1 port) then storage reading speed will be a bottleneck and not the CPU.
In this case xz compression may give better results as system must read smaller files from the storage.

I hope you are confused enough now

[ncmprhnsbl]

thanks, that actually clears things up..
if i had thought about it, that's why it's 'loop mounted' and not just 'ordinary mounted'

back to "is it(zstd) worth it?" ..seems it's a fairly clear "it depends" ...
on user wants and needs(how fast is fast enough, how many things to do at once)
on hardware(ram size, storage speed & so on)
accommodating both(or more) options seems reasonable to me..
in the future, as more ram, faster storage becomes more common(presumably(it's a big world with a lot of disparity) maybe then it's time to retire xz..

[raja]

Do you guys think it is worth converting to zstd?

Did you run the Open box ISO with ZSTD compression and experienced it's agility? If yes, choice is clear.

For users, Module extension is abc or xyz does not matter, and may not even be interested to know compression standards used..

I would like to hear from an user having a recent Core I7 m/c on boot speed and applications rendering. Mine is fairly old, Core I3 7th generation.