Porteus with zstd compression

Here is a place for your projects which are not officially supported by the Porteus Team. For example: your own kernel patched with extra features; desktops not included in the standard ISO like Gnome; base modules that are different than the standard ISO, etc...
jssouza
DEV Team
DEV Team
Posts: 1047
Joined: 09 Jul 2015, 14:17
Distribution: Porteus x86 arm
Location: Liechtenstein

Porteus with zstd compression

Post#16 by jssouza » 27 Jun 2019, 09:44

ncmprhnsbl wrote:
27 Jun 2019, 09:08
so: .xzms and -zstd.xzms will activate at boot, and a lot of the gui scripts will work oftb .ie activate/deactivate/extract
The only problem here I noticed is that there is no way to check what type of compression has been used to create the module. A file command just says Squashfs filesystem, not the compression used. Which rather makes the extension a necessary indicator to see what compression is used.

babam
Black ninja
Black ninja
Posts: 67
Joined: 16 Nov 2016, 10:30
Distribution: Porteus 3.2.2
Location: Rainy city

Porteus with zstd compression

Post#17 by babam » 27 Jun 2019, 10:53

Why not make it simple.
mymodule.xzm ---> Modules with xz compression
mymodule.zstd.xzm ---> modules with zstd compression

zactivate, zdeactivate and modify initrd (linuxrc) are not needed.
Last edited by babam on 28 Jun 2019, 02:02, edited 1 time in total.
Sorry, my English is bad.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2263
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

Porteus with zstd compression

Post#18 by ncmprhnsbl » 27 Jun 2019, 12:00

in what context is it needed to determine the compression used?
i mean. it's good to know, that's what the -zstd.xzm tag is for (or .zstd.xzm) and that could be parsed for..
but eg. unsquashfs works out what to do... slax's .sb or puppy's .sfs give no indication of compressor and seem to get by..

i have noticed that if i extract a -zstd.xzm and recompress it , it becomes -zstd-zstd.xzm.. so that's not exactly seamless
but hey, i'm just messing about and seeing what i can break ")
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

neko
DEV Team
DEV Team
Posts: 1568
Joined: 09 Feb 2013, 09:55
Distribution: APorteus-FVWM-ja-x86_64.iso
Location: japan

Porteus with zstd compression

Post#19 by neko » 27 Jun 2019, 16:11

==== Porteus "ZSTD compression" Trial Version ====

Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso was converted from Porteus-OPENBOX-v5.0rc1-x86_64.iso.

Step 1.
Kernel of Porteus-OPENBOX-v5.0rc1-x86_64.iso was changed to 5.1.15 (build with config "CONFIG_SQUASHFS_ZSTD=y").
refer to Porteus Kernel Builder (Post by neko #72505)

Step 2.
ISO@/porteus/base/000-kernel.xzm, 001-core.xzm, 002-xorg.xzm, 003-openbox.xzm were rebuilt with "ZSTD compression".

the script of making XZM(the naming is for old interface)

Code: Select all

#! /bin/sh
SQ=`basename $1`
SQ=${SQ}.xzm

rm -f ${SQ}
mksquashfs $1/ ${SQ} -comp zstd -b 256K -Xcompression-level 22

Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso (359 M)
http://www.mediafire.com/file/dw7wft2nl ... x86_64.iso
md5sum: bc9027054f96983eb7205453ac90340b Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso


Thanks.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2263
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

Porteus with zstd compression

Post#20 by ncmprhnsbl » 28 Jun 2019, 13:04

here's a module with various files/fixes to enable the majority of GUI filemanager actions specific to .zstdm modules..
stick it in modules(or base) or if you're booting the ISO use cheatcode: extramod=/path/to/folder/where/it/is/ (thanks Ed_P)
ZTREE.zstdm
module contains:

Code: Select all

$tree -al ZTREE
├── home
│   └── guest
│       └── .config
│           └── mimeapps.list
├── opt
│   └── porteus-scripts
│       ├── context-menu
│       │   ├── create-zmodule
│       │   ├── extract-zmodule
│       │   └── zconvert_txz
│       ├── mloop
│       ├── txz2zstdm
│       ├── xorg
│       │   └── zxactivate
│       └── zactivate
├── root
│   └── .config
│       └── mimeapps.list
└── usr
    ├── bin
    │   ├── dir2zstdm -> /opt/porteus-scripts/dir2zstdm
    │   └── txz2zstdm -> /opt/porteus-scripts/txz2zstdm
    ├── local
    │   └── bin
    │       └── lsmodules
    └── share
        ├── applications
        │   ├── mimeapps.list
        │   ├── pcmanfm-create-zmodule.desktop
        │   ├── pcmanfm-extract-module.desktop
        │   ├── pcmanfm-mount-module.desktop
        │   ├── pcmanfm-unmount-module.desktop
        │   ├── pcmanfm-zconvert-txz-to-xzm.desktop
        │   ├── zactivate.desktop
        │   └── zdeactivate.desktop
        ├── icons
        │   └── Adwaita
        │       ├── 16x16
        │       │   └── mimetypes
        │       │       └── application-x-zstdm.png -> /usr/share/icons/Adwaita/16x16/mimetypes/application-x-xzm.png
        │       ├── 22x22
        │       │   └── mimetypes
        │       │       └── application-x-zstdm.png -> /usr/share/icons/Adwaita/22x22/mimetypes/application-x-xzm.png
        │       ├── 32x32
        │       │   └── mimetypes
        │       │       └── application-x-zstdm.png -> /usr/share/icons/Adwaita/32x32/mimetypes/application-x-xzm.png
        │       ├── 48x48
        │       │   └── mimetypes
        │       │       └── application-x-zstdm.png -> /usr/share/icons/Adwaita/48x48/mimetypes/application-x-xzm.png
        │       └── 64x64
        │           └── mimetypes
        │               └── application-x-zstdm.png -> /usr/share/icons/Adwaita/64x64/mimetypes/application-x-xzm.png
        └── mime
            └── packages
                └── freedesktop.org.xml

29 directories, 26 files
notes:
these files would normally be spread across 001-core, 002-xorg, 003-openbox
zactivate is included for some edits regarding zxactivate
mloop is expanded to include .zstdm
lsmodules only handles .zstdm. xzm s are not seen or handled.
this is for jssouzas ISO .. not sure i understand the nature of nekos ISO..(edit: ZTREE.zstdm is not useful as is, for nekos ISO)
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

User avatar
Ed_P
Contributor
Contributor
Posts: 5145
Joined: 06 Feb 2013, 22:12
Distribution: 4.0 Cinnamon 64-bit ISO
Location: Western NY, USA

Porteus with zstd compression

Post#21 by Ed_P » 28 Jun 2019, 14:35

ncmprhnsbl wrote:
28 Jun 2019, 13:04
if you're booting the ISO use cheatcode: load=/path/to/where/it/is/ZTREE
Not the extramod= cheatcode?
Ed

neko
DEV Team
DEV Team
Posts: 1568
Joined: 09 Feb 2013, 09:55
Distribution: APorteus-FVWM-ja-x86_64.iso
Location: japan

Porteus with zstd compression

Post#22 by neko » 28 Jun 2019, 14:45

@ncmprhnsbl
[Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso]
The same naming rule (sufix is .xzm) is used for "ZSTD compression" module and "XZ compression" module.
It allows mixed xzm modules.
No more difference is between Porteus-OPENBOX-v5.0rc1-x86_64.iso and Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso.

Thanks.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2263
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

Porteus with zstd compression

Post#23 by ncmprhnsbl » 28 Jun 2019, 22:13

Ed_P wrote:
28 Jun 2019, 14:35
Not the extramod= cheatcode?
you're right Ed, load= is only for the /optional folder .. i'll fix my post..
neko wrote:
28 Jun 2019, 14:45
No more difference is between Porteus-OPENBOX-v5.0rc1-x86_64.iso and Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso.
thanks neko, that's what i suspected.. which means my ZTREE.zstdm would not be useful for that IS0.
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

babam
Black ninja
Black ninja
Posts: 67
Joined: 16 Nov 2016, 10:30
Distribution: Porteus 3.2.2
Location: Rainy city

Porteus with zstd compression

Post#24 by babam » 29 Jun 2019, 09:59

Even with the kernel built by neko Porteus Kernel Builder, it allows Porteus 3.2.2 to support modules with zstd compression without modifying anything.
Sorry, my English is bad.

User avatar
brokenman
Site Admin
Site Admin
Posts: 6096
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil
Contact:

Porteus with zstd compression

Post#25 by brokenman » 09 Jul 2019, 16:34

Do you guys think it is worth converting to zstd?
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2263
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

Porteus with zstd compression

Post#26 by ncmprhnsbl » 10 Jul 2019, 03:20

brokenman wrote:
09 Jul 2019, 16:34
Do you guys think it is worth converting to zstd?
i'm more of the opinion of having support(kernel,squashfs-tools,scripts) for all of the above (xz, zstd, lz4, even gzip)
keep official iso xz for the smallest size, and then the user can convert the base modules to whatever, if they value speed over size..
i'm currently running a mixture(zstd, xz) simply because i haven't bothered to convert everything to zstd..
my zstd modules are named name-zstd.xzm to keep track of them..
correct me if i'm wrong: decompression times don't have an effect beyond activation(therefore boot times), ie. once loaded to the union there should be no difference in performance..
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

burdi01
Samurai
Samurai
Posts: 109
Joined: 18 Aug 2013, 12:09
Distribution: Slackware, Xubuntu
Location: The Netherlands

Porteus with zstd compression

Post#27 by burdi01 » 10 Jul 2019, 09:10

correct me if i'm wrong: decompression times don't have an effect beyond activation(therefore boot times), ie. once loaded to the union there should be no difference in performance..
For as far as I am aware squashfses loaded in the union are still in compressed format -- it would take a lot of memory and cpu cycles to decompress them in their entirety at once. Therefore the decompression has to be done upon accessing files in them -- which spreads the load on memory and cpu cycles.
In other words there is a difference in performance but it is less noticeable than one might expect on a first impulse.
:D

User avatar
fanthom
Site Admin
Site Admin
Posts: 5182
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

Porteus with zstd compression

Post#28 by fanthom » 10 Jul 2019, 11:02

To make things more complicated:

Linux caches everything in RAM so if you have plenty of it then system reads (decompresses) data from the modules only once.
This is pretty much the kiosk case: system loads the browser and nothing more. First load is slower but browser restarts are super fast cause are libs, images, other files are cached in RAM (which is fastest storage device available - maybe Optane storage is equal).

In case of Porteus Destop there can be more apps opened so module compression matters more.
Every time the system is running out of RAM the read from modules (decompression) happens and you can save on CPU cycles if you pick zstd compression.

Situation is also different if you use 'copy2ram' feature.

If you enable copy2ram then zstd is a clear win.

However - if modules are stored on a slow device rather than in RAM (e.g. ancient SD card or usb stick plugged to a USB 1.1 port) then storage reading speed will be a bottleneck and not the CPU.
In this case xz compression may give better results as system must read smaller files from the storage.

I hope you are confused enough now :)
Please add [Solved] to your thread title if the solution was found.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2263
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

Porteus with zstd compression

Post#29 by ncmprhnsbl » 10 Jul 2019, 13:03

:D thanks, that actually clears things up..
if i had thought about it, that's why it's 'loop mounted' and not just 'ordinary mounted'

back to "is it(zstd) worth it?" ..seems it's a fairly clear "it depends" ...
on user wants and needs(how fast is fast enough, how many things to do at once)
on hardware(ram size, storage speed & so on)
accommodating both(or more) options seems reasonable to me..
in the future, as more ram, faster storage becomes more common(presumably(it's a big world with a lot of disparity) maybe then it's time to retire xz..
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

raja
Shogun
Shogun
Posts: 213
Joined: 02 May 2017, 09:51
Distribution: v3.2.2-32 and v5.0-rc1-zstd-64
Location: Chennai,India

Porteus with zstd compression

Post#30 by raja » 11 Jul 2019, 17:29

brokenman wrote:
09 Jul 2019, 16:34
Do you guys think it is worth converting to zstd?
Did you run the Open box ISO with ZSTD compression and experienced it's agility? If yes, choice is clear.

For users, Module extension is abc or xyz does not matter, and may not even be interested to know compression standards used..

I would like to hear from an user having a recent Core I7 m/c on boot speed and applications rendering. Mine is fairly old, Core I3 7th generation.
Linux Kernel-4.4.120-32 bit; Linux kernel-5.2-64 bit.

Post Reply