ZSTD compression instead of XZ for the modules

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5666
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

ZSTD compression instead of XZ for the modules

Post#1 by fanthom » 30 May 2019, 17:06

Interesting read:
https://www.phoronix.com/scan.php?page= ... RPM-Change

Seems the XZM modules would be only few percent larger but 5 times faster to decompress:
https://bugzilla.redhat.com/show_bug.cgi?id=1648490#c6

'zstd' support for squash filesystem is in the kernel already so it would be a matter to patch squashfs-tools with 'zstd' support. Did anybody try that?
If yes then please share results :)

Thanks
Please add [Solved] to your thread title if the solution was found.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 3924
Joined: 20 Mar 2012, 03:42
Distribution: v5.0-64bit
Location: australia
Contact:

ZSTD compression instead of XZ for the modules

Post#2 by ncmprhnsbl » 31 May 2019, 09:30

fanthom wrote:
30 May 2019, 17:06
'zstd' support for squash filesystem is in the kernel already so it would be a matter to patch squashfs-tools with 'zstd' support. Did anybody try that?
this repo: https://github.com/plougher/squashfs-to ... shfs-tools seems to indicate that has incorporated zstd support..
indeed mksquashfs --help in porteus yields:

Code: Select all

	zstd
	  -Xcompression-level <compression-level>
		<compression-level> should be 1 .. 22 (default 15)
nothing to do but try it out, i guess

EDIT
quick compression test: mksquashfs %f test.zstdm -comp zstd -b 256K (so that's default compression level 15)
dir = 47.4 MB
xzm = 12.8 MB 27%
zstd = 14.5 MB 30.6%
compression seemed pretty snappy (bit small to tell)
EDIT2
further tests:
at compression levels:
22 = 14.4 MB 30.1%
1 = 17.0 MB 35.9%
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5666
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

ZSTD compression instead of XZ for the modules

Post#3 by fanthom » 31 May 2019, 12:36

Hi ncmprhnsbl,

Please try highest compression level for zstd and check module size then.

You can also measure compression time (however this is less important than decompression):

Code: Select all

time mksquashfs %f test.zstdm -comp zstd -b 256K -Xcompression-level 22
Would be great if you could mount both modules (kernel must have zstd support enabled) under /mnt then copy content of the module to tmpfs (RAM) to measure real decompression speed:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd
time cp -a /mnt/xzm/* /tmp/xzm
Wondering about results :)

Added in 8 minutes 7 seconds:
Well - seems Gentoo also has zstd support enabled in squashfs-tools so i can easily do these tests myself :)

Will post results later today.
Please add [Solved] to your thread title if the solution was found.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 3924
Joined: 20 Mar 2012, 03:42
Distribution: v5.0-64bit
Location: australia
Contact:

ZSTD compression instead of XZ for the modules

Post#4 by ncmprhnsbl » 31 May 2019, 13:13

here's some decompress times: same 47.4 dir
unsquashfs 12.8 MB xzm
real 0m1.399s
user 0m2.762s
sys 0m0.635s

unsquashfs 14.4 MB zstd (compression 22)
real 0m3.634s
user 0m0.436s
sys 0m0.738s
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5666
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

ZSTD compression instead of XZ for the modules

Post#5 by fanthom » 31 May 2019, 13:45

Compression is worse comparing to xz but decompression is 7 time faster in favor of zstd in my tests:

Dir size: 263 MB

Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)

Decompression time:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd

real	0m1.381s
user	0m0.017s
sys	0m1.220s

time cp -a /mnt/xz/* /tmp/xz

real	0m9.709s
user	0m0.016s
sys	0m9.315s
Not sure if the speed is worth bigger module sizes ...
Please add [Solved] to your thread title if the solution was found.

User avatar
babam
Warlord
Warlord
Posts: 526
Joined: 16 Nov 2016, 10:30
Distribution: Porteus 5.0rc3 Xfce K6.1.1
Location: Rainy city

ZSTD compression instead of XZ for the modules

Post#6 by babam » 02 Jun 2019, 20:58

fanthom wrote:
31 May 2019, 13:45
Compression is worse comparing to xz but decompression is 7 time faster in favor of zstd in my tests:

Dir size: 263 MB

Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)

Decompression time:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd

real	0m1.381s
user	0m0.017s
sys	0m1.220s

time cp -a /mnt/xz/* /tmp/xz

real	0m9.709s
user	0m0.016s
sys	0m9.315s
Not sure if the speed is worth bigger module sizes ...
Can you test it by placing zstd and xz modules in the NTFS partition.

I doubt the zstd decompression speed if it is placed in the NTFS partition.
Sorry, my English is bad.

raja
Shogun
Shogun
Posts: 434
Joined: 02 May 2017, 09:51
Distribution: v3.2.2-32 and Porteus-Artix-64
Location: Chennai,India

ZSTD compression instead of XZ for the modules

Post#7 by raja » 03 Jun 2019, 16:48

Zstd compression available in squashfs-tools in Slackware. I got it through USM. Check alien database.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.

Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.
Linux Kernel-4.4.272 -32 bit; Linux Kernel-5.4.185 - 64 bit

raja
Shogun
Shogun
Posts: 434
Joined: 02 May 2017, 09:51
Distribution: v3.2.2-32 and Porteus-Artix-64
Location: Chennai,India

ZSTD compression instead of XZ for the modules

Post#8 by raja » 03 Jun 2019, 16:48

Zstd compression available in squashfs-tools in Slackware. I got it through USM. Check alien database.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.

Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.
Linux Kernel-4.4.272 -32 bit; Linux Kernel-5.4.185 - 64 bit

User avatar
M. Eerie
Moderator
Moderator
Posts: 619
Joined: 31 Aug 2017, 21:18
Distribution: Nemesis Xfce/MATE x64

ZSTD compression instead of XZ for the modules

Post#9 by M. Eerie » 21 May 2020, 11:36

Testing larger memory pages. (mksquashfs ... -b 256K vs -b 1M ...)
SYSTEM SPECS: CPU Intel Core i5, RAM: 8GB's

A compressor using bigger chunks of data should be able to compress more efficiently, but it relies on the memory of the host computer where compressed data will be used. Also, there is no point in using large memory pages if you are going to compress a few Mb's. (<=100 Mb), but...

COMPRESSION TIME:
XZ has clearly better times. While it seems to lag at the beginning, it almost halves the whole process. I.e. The more room is allocated, the faster the CPU will move the data.
ZSTD seems not to benefit of this matter.

COMPRESSION RATIO:
When using large memory pages, final size is around 6% less in both cases, but XZ compression factor was 13% better.

DECOMPRESSION SPEED:
I haven't really tested the decompression speed which seems to be the real deal, but I'd swear I didn't noticed any substantial improvement.

Code: Select all

time mksquashfs 001-core-20200310/ 001-core-20200310.xzm -b 1M -comp xz -Xbcj x86 -noappend
real	2m20,282s
user	6m55,109s
sys	0m1,013s


time mksquashfs 001-core-20200310/ 001-core-20200310.zstd -b 1M -comp zstd -Xcompression-level 22 -noappend
real	1m18,504s
user	3m35,465s
sys	0m0,873s

-rwxrwxrwx 1 root root 115814400 may 21 11:53 001-core-20200310.xzm*
-rwxrwxrwx 1 root root 130097152 may 21 12:03 001-core-20200310.zstd*

time mksquashfs 001-core-20200310/ 001-core-20200310.xzm -b 256K -comp xz -Xbcj x86 -noappend
real	1m34,769s
user	7m36,185s
sys	0m1,793s

time mksquashfs 001-core-20200310/ 001-core-20200310.zstd -b 256K -comp zstd -Xcompression-level 22 -noappend
real	1m8,756s
user	5m17,440s
sys	0m0,832s

-rwxrwxrwx 1 root root 123674624 may 21 12:15 001-core-20200310.xzm*
-rwxrwxrwx 1 root root 139001856 may 21 12:17 001-core-20200310.zstd*
Cheers!
> Does not compute_ 🖖

https://forum.porteus.org/viewtopic.php?p=94310#p94310
https://forum.porteus.org/viewtopic.php?p=84002#p84002
https://forum.porteus.org/viewtopic.php?p=77174#p77174
https://forum.porteus.org/viewtopic.php?f=39&t=8584

User avatar
M. Eerie
Moderator
Moderator
Posts: 619
Joined: 31 Aug 2017, 21:18
Distribution: Nemesis Xfce/MATE x64

ZSTD compression instead of XZ for the modules

Post#10 by M. Eerie » 17 May 2021, 08:02

The only Facebook contribution (imposition) I would admit to use

Last release is 50% faster :shock:
> Does not compute_ 🖖

https://forum.porteus.org/viewtopic.php?p=94310#p94310
https://forum.porteus.org/viewtopic.php?p=84002#p84002
https://forum.porteus.org/viewtopic.php?p=77174#p77174
https://forum.porteus.org/viewtopic.php?f=39&t=8584

fulalas
DEV Team
DEV Team
Posts: 2050
Joined: 26 Oct 2016, 15:34
Distribution: Porteus
Location: Brazil

ZSTD compression instead of XZ for the modules

Post#11 by fulalas » 21 May 2021, 23:28

I did a boot time test here with just the essential modules:
Normal compression (-comp xz -b 256K -Xbcj x86): 14s
Zstd compression (-comp zstd -b 256K -Xcompression-level 22) : 9.5s

It's pretty impressive!

The difference in size was:
kernel: 65.5 MB vs 72.7 MB
core: 127.2 MB vs 147.6 MB
xorg: 101.6 MB vs 117.9 MB
xfce: 11.1 MB vs 12.4 MB

Post Reply