Porteus with zstd compression

Here is a place for your projects which are not officially supported by the Porteus Team. For example: your own kernel patched with extra features; desktops not included in the standard ISO like Gnome; base modules that are different than the standard ISO, etc...
jssouza
DEV Team
DEV Team
Posts: 1047
Joined: 09 Jul 2015, 14:17
Distribution: Porteus x86 arm
Location: Liechtenstein

Porteus with zstd compression

Post#1 by jssouza » 22 Jun 2019, 11:42

Here's Porteus v5.0rc1 openbox remastered with zstd compression.
http://www.mediafire.com/file/zwndkxavp ... d.iso/file
md5sum: 54c88d7d8594872471e10cea95f3629d

- kernel built with zstd support for squashfs
- initrd loads only zstdm modules
- 000-kernel.zstdm, 001-core.zstdm, 002-xorg.zstdm and 003-openbox.zstdm modules squashed with zstd compression
- Some scripts added in 001-core: zactivate, zdeactivate, dir2zstdm (command line only) for zstd module creation, activation and deactivation (not fully tested)

I get a boot time of 5 seconds in virtualbox.

@raja please note, if you plan to remaster, you need the 000-kernel.zstdm, vmlinuz as well as initrd.xz.

fulalas
DEV Team
DEV Team
Posts: 1456
Joined: 26 Oct 2016, 15:34
Distribution: Porteus
Location: Brazil

Porteus with zstd compression

Post#2 by fulalas » 22 Jun 2019, 11:52

Pardon my ignorance, but what's the actual difference between this and the official release? :D

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2256
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

Porteus with zstd compression

Post#3 by ncmprhnsbl » 22 Jun 2019, 13:31

fulalas wrote:
22 Jun 2019, 11:52
Pardon my ignorance, but what's the actual difference between this and the official release?
see discussion here: ZSTD compression instead of XZ for the modules
basicly, using zstd compression instead of xz ... advantage: faster decompression disadvantage: larger module size
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

jssouza
DEV Team
DEV Team
Posts: 1047
Joined: 09 Jul 2015, 14:17
Distribution: Porteus x86 arm
Location: Liechtenstein

Porteus with zstd compression

Post#4 by jssouza » 22 Jun 2019, 13:54

I also noticed that there is no support (yet) for compressing kernel and the initrd itself in zstd format. Not sure if this is not possible or might come later in the kernel. I remember xz compression of the kernel also came later, after squashfs xz support had arrived. In old computers, uncompressing the kernel itself takes a long time.
Of course boot loaders (grub, syslinux) would also need to support loading zstd compressed vmlinuz and initrd.

babam
Black ninja
Black ninja
Posts: 67
Joined: 16 Nov 2016, 10:30
Distribution: Porteus 3.2.2
Location: Rainy city

Porteus with zstd compression

Post#5 by babam » 22 Jun 2019, 18:32

jssouza wrote:
22 Jun 2019, 11:42

I get a boot time of 5 seconds in virtualbox.
What if installed on NTFS, is it still fast.
Sorry, my English is bad.

jssouza
DEV Team
DEV Team
Posts: 1047
Joined: 09 Jul 2015, 14:17
Distribution: Porteus x86 arm
Location: Liechtenstein

Porteus with zstd compression

Post#6 by jssouza » 24 Jun 2019, 04:53

babam wrote:
22 Jun 2019, 18:32
What if installed on NTFS, is it still fast.
I did not see any noticeable difference. Boot from ntfs also had same boot time and same response times.

Kulle
Samurai
Samurai
Posts: 195
Joined: 28 Jan 2017, 10:39
Distribution: v4.0 64bit Xfce
Location: Berlin

Porteus with zstd compression

Post#7 by Kulle » 24 Jun 2019, 08:29

Hi,
is a change over from xz to zstd useful just because it is slightly faster to decompress?
I think, what has been tried and tested for a long time should be maintained.
And you can continue to use existing xzm modules.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2256
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

Porteus with zstd compression

Post#8 by ncmprhnsbl » 24 Jun 2019, 12:01

zactivate needs an edit to aufs-insert to work:
@ line 18 (changing .xzm to .zstdm)

Code: Select all

if [[ $(echo "$BASE" | grep .zstdm) = "" ]]; then
    echo "$BASE: Module must end with .zstdm"
    exit 1
fi
of course, simply renaming the (zstd)module *.xzm would work too since unsquashfs detects the protocol, whatever it is..
probably pointing zactivate at an extra "zaufs-insert" script is better(and maybe zxactivate).. then not breaking activate..
what might be interesting, would be to expand the present scripts (and initrd) to handle multiple compression types..
but hey, they already do, provided the modules are named *.xzm (apart from dir2xzm)
perhaps all we need is dir2zstdm that outputs *-zstd.xzm ...
furthermore, modules can be any of: gzip lzma lzo xz(what we use) or zstd .. provided they are named *.xzm they can be activated without modifying porteus..
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

jssouza
DEV Team
DEV Team
Posts: 1047
Joined: 09 Jul 2015, 14:17
Distribution: Porteus x86 arm
Location: Liechtenstein

Porteus with zstd compression

Post#9 by jssouza » 24 Jun 2019, 12:25

ncmprhnsbl wrote:
24 Jun 2019, 12:01
zactivate needs an edit to aufs-insert to work:
Check again, zactivate calls aufs-zinsert

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2256
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

Porteus with zstd compression

Post#10 by ncmprhnsbl » 24 Jun 2019, 12:37

jssouza wrote:
24 Jun 2019, 12:25
Check again, zactivate calls aufs-zinsert
sorry, missed that... but it failed with the "must be .xzm" message .. something else going on..
EDIT:
ok.. this line in xactivate:

Code: Select all

# Insert the module to live filesystem: (changed to)
/opt/porteus-scripts/xorg/aufs-zinsert "$1"
but as i say above none of this is really necessary, if the standard module extension(.xzm) is used..
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

jssouza
DEV Team
DEV Team
Posts: 1047
Joined: 09 Jul 2015, 14:17
Distribution: Porteus x86 arm
Location: Liechtenstein

Porteus with zstd compression

Post#11 by jssouza » 24 Jun 2019, 13:22

It's true, agree with you, but when one facilitates an iso as an experiment, I would think both xzm and zstdm scripts should be present for comparison dont you think?

That is why I compiled the kernel with both xzm as well as zstdm support, and, also instead of overwriting the existing scripts, I (tried) created new zstd variants of them, alongside the xzm scripts.
ncmprhnsbl wrote:
24 Jun 2019, 12:37
ok.. this line in xactivate:
Will try and fix this. Thanks ncmprhnsbl.

jssouza
DEV Team
DEV Team
Posts: 1047
Joined: 09 Jul 2015, 14:17
Distribution: Porteus x86 arm
Location: Liechtenstein

Porteus with zstd compression

Post#12 by jssouza » 24 Jun 2019, 17:20

Kulle wrote:
24 Jun 2019, 08:29
Hi,
is a change over from xz to zstd useful just because it is slightly faster to decompress?
I think, what has been tried and tested for a long time should be maintained.
And you can continue to use existing xzm modules.
I almost agree with you Kulle.
But, here's the thing:
When lzma1 was the thing and lzma2 just came out, Tomas M was just preparing Slax 7, and I in my ignorant beginning just mentioned, why still lzma1 on the kernel? Why not xz on it too?
Tomas replied, Tomas acknowledged me, and changed the kernel to xz. Porteus changed kernel to xz soon. Post is available for ppl who search good enough.

Coming to the point: I made a difference then, both in slax and in porteus. Thought, why not let the community try it again?

ottone
Ronin
Ronin
Posts: 3
Joined: 13 Jun 2019, 06:44
Distribution: slackex

Porteus with zstd compression

Post#13 by ottone » 26 Jun 2019, 07:31

I have to say that on my old Acer Laptop ( T7500) this iso is monsterfast . By the way , the kernel 5xx alone on the normal isos makes a great difference in speed

on my old lappy - surprising .... :showoff:

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2256
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

Porteus with zstd compression

Post#14 by ncmprhnsbl » 27 Jun 2019, 09:08

jssouza wrote:
24 Jun 2019, 13:22
It's true, agree with you, but when one facilitates an iso as an experiment, I would think both xzm and zstdm scripts should be present for comparison dont you think?
absolutely, if zstd is to be incorporated officially, i'd like to see it provided alongside xz, so users can choose for themselves according to their preference for speed or size..
here's what i've done:
swapped your zstd/xz enabled kernel into my standard install - but not your initrd - so renamed to 000-kernel-zstd.xzm
so: .xzms and -zstd.xzms will activate at boot, and a lot of the gui scripts will work oftb .ie activate/deactivate/extract
just had change dir2stdm to output *zstd.xzm, add /opt/porteus-scripts/context-menu/create-zmodule and a corresponding .desktop..

i'm curious about the kernel support for compressors, is this part of the AUFS patching or some other compile flag .. how (not)trivial is this?
i remember back when alphaos was still alive, simargl switched to gzip for a speed increase, i wonder if it might be interesting to play with too, tho size takes a hit..
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

jssouza
DEV Team
DEV Team
Posts: 1047
Joined: 09 Jul 2015, 14:17
Distribution: Porteus x86 arm
Location: Liechtenstein

Porteus with zstd compression

Post#15 by jssouza » 27 Jun 2019, 09:34

Great!
ncmprhnsbl wrote:
27 Jun 2019, 09:08
i'm curious about the kernel support for compressors, is this part of the AUFS patching or some other compile flag .. how (not)trivial is this?
Nope, no aufs patching is necessary. In fact, I think it is now already part of the kernel :)
@raja requested here Porteus Kernel Builder (Post by raja #72503) and @neko added support in the next post. I had added some extra zstd related configs to my kernel in the iso, but I dont think they are necessary. Just CONFIG_SQUASHFS_ZSTD=y should be enough.
ncmprhnsbl wrote:
27 Jun 2019, 09:08
i remember back when alphaos was still alive, simargl switched to gzip for a speed increase, i wonder if it might be interesting to play with too, tho size takes a hit..
I had done similar experiments by creating lzo modules in the raspberry pi variant, to increase speed. The modules did increase in size a lot, but speed increase was quite noticeable.
zlib, lz4, lzo squashfs compressors are already supported in our kernel.

Post Reply