Page 2 of 3
Porteus with zstd compression
Posted: 27 Jun 2019, 09:44
by jssouza
ncmprhnsbl wrote: ↑27 Jun 2019, 09:08
so: .xzms and -zstd.xzms will activate at boot, and a lot of the gui scripts will work oftb .ie activate/deactivate/extract
The only problem here I noticed is that there is no way to check what type of compression has been used to create the module. A file command just says Squashfs filesystem, not the compression used. Which rather makes the extension a necessary indicator to see what compression is used.
Porteus with zstd compression
Posted: 27 Jun 2019, 10:53
by babam
Why not make it simple.
mymodule.xzm ---> Modules with xz compression
mymodule.zstd.xzm ---> modules with zstd compression
zactivate, zdeactivate and modify initrd (linuxrc) are not needed.
Porteus with zstd compression
Posted: 27 Jun 2019, 12:00
by ncmprhnsbl
in what context is it needed to determine the compression used?
i mean. it's good to know, that's what the -zstd.xzm tag is for (or .zstd.xzm) and that could be parsed for..
but eg. unsquashfs works out what to do... slax's .sb or puppy's .sfs give no indication of compressor and seem to get by..
i have noticed that if i extract a -zstd.xzm and recompress it , it becomes -zstd-zstd.xzm.. so that's not exactly seamless
but hey, i'm just messing about and seeing what i can break ")
Porteus with zstd compression
Posted: 27 Jun 2019, 16:11
by neko
==== Porteus "ZSTD compression" Trial Version ====
Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso was converted from Porteus-OPENBOX-v5.0rc1-x86_64.iso.
Step 1.
Kernel of Porteus-OPENBOX-v5.0rc1-x86_64.iso was changed to 5.1.15 (build with config "CONFIG_SQUASHFS_ZSTD=y").
refer to
Porteus Kernel Builder (Post by neko #72505)
Step 2.
ISO@/porteus/base/000-kernel.xzm, 001-core.xzm, 002-xorg.xzm, 003-openbox.xzm were rebuilt with "ZSTD compression".
the script of making XZM(the naming is for old interface)
Code: Select all
#! /bin/sh
SQ=`basename $1`
SQ=${SQ}.xzm
rm -f ${SQ}
mksquashfs $1/ ${SQ} -comp zstd -b 256K -Xcompression-level 22
Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso (359 M)
http://www.mediafire.com/file/dw7wft2nl ... x86_64.iso
md5sum: bc9027054f96983eb7205453ac90340b Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso
Thanks.
Porteus with zstd compression
Posted: 28 Jun 2019, 13:04
by ncmprhnsbl
here's a module with various files/fixes to enable the majority of GUI filemanager actions specific to .zstdm modules..
stick it in modules(or base) or if you're booting the ISO use cheatcode: extramod=/path/to/folder/where/it/is/ (thanks Ed_P)
ZTREE.zstdm
module contains:
Code: Select all
$tree -al ZTREE
├── home
│ └── guest
│ └── .config
│ └── mimeapps.list
├── opt
│ └── porteus-scripts
│ ├── context-menu
│ │ ├── create-zmodule
│ │ ├── extract-zmodule
│ │ └── zconvert_txz
│ ├── mloop
│ ├── txz2zstdm
│ ├── xorg
│ │ └── zxactivate
│ └── zactivate
├── root
│ └── .config
│ └── mimeapps.list
└── usr
├── bin
│ ├── dir2zstdm -> /opt/porteus-scripts/dir2zstdm
│ └── txz2zstdm -> /opt/porteus-scripts/txz2zstdm
├── local
│ └── bin
│ └── lsmodules
└── share
├── applications
│ ├── mimeapps.list
│ ├── pcmanfm-create-zmodule.desktop
│ ├── pcmanfm-extract-module.desktop
│ ├── pcmanfm-mount-module.desktop
│ ├── pcmanfm-unmount-module.desktop
│ ├── pcmanfm-zconvert-txz-to-xzm.desktop
│ ├── zactivate.desktop
│ └── zdeactivate.desktop
├── icons
│ └── Adwaita
│ ├── 16x16
│ │ └── mimetypes
│ │ └── application-x-zstdm.png -> /usr/share/icons/Adwaita/16x16/mimetypes/application-x-xzm.png
│ ├── 22x22
│ │ └── mimetypes
│ │ └── application-x-zstdm.png -> /usr/share/icons/Adwaita/22x22/mimetypes/application-x-xzm.png
│ ├── 32x32
│ │ └── mimetypes
│ │ └── application-x-zstdm.png -> /usr/share/icons/Adwaita/32x32/mimetypes/application-x-xzm.png
│ ├── 48x48
│ │ └── mimetypes
│ │ └── application-x-zstdm.png -> /usr/share/icons/Adwaita/48x48/mimetypes/application-x-xzm.png
│ └── 64x64
│ └── mimetypes
│ └── application-x-zstdm.png -> /usr/share/icons/Adwaita/64x64/mimetypes/application-x-xzm.png
└── mime
└── packages
└── freedesktop.org.xml
29 directories, 26 files
notes:
these files would normally be spread across 001-core, 002-xorg, 003-openbox
zactivate is included for some edits regarding zxactivate
mloop is expanded to include .zstdm
lsmodules only handles .zstdm. xzm s are not seen or handled.
this is for jssouzas ISO .. not sure i understand the nature of nekos ISO..(edit: ZTREE.zstdm is not useful as is, for nekos ISO)
Porteus with zstd compression
Posted: 28 Jun 2019, 14:35
by Ed_P
ncmprhnsbl wrote: ↑28 Jun 2019, 13:04
if you're booting the ISO use cheatcode: load=/path/to/where/it/is/ZTREE
Not the extramod= cheatcode?
Porteus with zstd compression
Posted: 28 Jun 2019, 14:45
by neko
@ncmprhnsbl
[Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso]
The same naming rule (sufix is .xzm) is used for "ZSTD compression" module and "XZ compression" module.
It allows mixed xzm modules.
No more difference is between Porteus-OPENBOX-v5.0rc1-x86_64.iso and Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso.
Thanks.
Porteus with zstd compression
Posted: 28 Jun 2019, 22:13
by ncmprhnsbl
Ed_P wrote: ↑28 Jun 2019, 14:35
Not the extramod= cheatcode?
you're right Ed, load= is only for the /optional folder .. i'll fix my post..
neko wrote: ↑28 Jun 2019, 14:45
No more difference is between Porteus-OPENBOX-v5.0rc1-x86_64.iso and Porteus-OPENBOX-v5.0rc1-zstd-x86_64.iso.
thanks neko, that's what i suspected.. which means my ZTREE.zstdm would not be useful for that IS0.
Porteus with zstd compression
Posted: 29 Jun 2019, 09:59
by babam
Even with the kernel built by
neko Porteus Kernel Builder, it allows Porteus 3.2.2 to support modules with zstd compression without modifying anything.
Porteus with zstd compression
Posted: 09 Jul 2019, 16:34
by brokenman
Do you guys think it is worth converting to zstd?
Porteus with zstd compression
Posted: 10 Jul 2019, 03:20
by ncmprhnsbl
brokenman wrote: ↑09 Jul 2019, 16:34
Do you guys think it is worth converting to zstd?
i'm more of the opinion of having support(kernel,squashfs-tools,scripts) for all of the above (xz, zstd, lz4, even gzip)
keep official iso xz for the smallest size, and then the user can convert the base modules to whatever, if they value speed over size..
i'm currently running a mixture(zstd, xz) simply because i haven't bothered to convert everything to zstd..
my zstd modules are named name-zstd.xzm to keep track of them..
correct me if i'm wrong: decompression times don't have an effect beyond activation(therefore boot times), ie. once loaded to the union there should be no difference in performance..
Porteus with zstd compression
Posted: 10 Jul 2019, 09:10
by burdi01
correct me if i'm wrong: decompression times don't have an effect beyond activation(therefore boot times), ie. once loaded to the union there should be no difference in performance..
For as far as I am aware squashfses loaded in the union are still in compressed format -- it would take a lot of memory and cpu cycles to decompress them in their entirety at once. Therefore the decompression has to be done upon accessing files in them -- which spreads the load on memory and cpu cycles.
In other words there is a difference in performance but it is less noticeable than one might expect on a first impulse.
Porteus with zstd compression
Posted: 10 Jul 2019, 11:02
by fanthom
To make things more complicated:
Linux caches everything in RAM so if you have plenty of it then system reads (decompresses) data from the modules only once.
This is pretty much the kiosk case: system loads the browser and nothing more. First load is slower but browser restarts are super fast cause are libs, images, other files are cached in RAM (which is fastest storage device available - maybe Optane storage is equal).
In case of Porteus Destop there can be more apps opened so module compression matters more.
Every time the system is running out of RAM the read from modules (decompression) happens and you can save on CPU cycles if you pick zstd compression.
Situation is also different if you use 'copy2ram' feature.
If you enable copy2ram then zstd is a clear win.
However - if modules are stored on a slow device rather than in RAM (e.g. ancient SD card or usb stick plugged to a USB 1.1 port) then storage reading speed will be a bottleneck and not the CPU.
In this case xz compression may give better results as system must read smaller files from the storage.
I hope you are confused enough now
Porteus with zstd compression
Posted: 10 Jul 2019, 13:03
by ncmprhnsbl
thanks, that actually clears things up..
if i had thought about it, that's why it's 'loop mounted' and not just 'ordinary mounted'
back to "is it(zstd) worth it?" ..seems it's a fairly clear "it depends" ...
on user wants and needs(how fast is fast enough, how many things to do at once)
on hardware(ram size, storage speed & so on)
accommodating both(or more) options seems reasonable to me..
in the future, as more ram, faster storage becomes more common(presumably(it's a big world with a lot of disparity) maybe then it's time to retire xz..
Porteus with zstd compression
Posted: 11 Jul 2019, 17:29
by raja
brokenman wrote: ↑09 Jul 2019, 16:34
Do you guys think it is worth converting to zstd?
Did you run the Open box ISO with ZSTD compression and experienced it's agility? If yes, choice is clear.
For users, Module extension is abc or xyz does not matter, and may not even be interested to know compression standards used..
I would like to hear from an user having a recent Core I7 m/c on boot speed and applications rendering. Mine is fairly old, Core I3 7th generation.