ZSTD compression instead of XZ for the modules

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
Post Reply
User avatar
fanthom
Site Admin
Site Admin
Posts: 5095
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

ZSTD compression instead of XZ for the modules

Post#1 by fanthom » 30 May 2019, 17:06

Interesting read:
https://www.phoronix.com/scan.php?page= ... RPM-Change

Seems the XZM modules would be only few percent larger but 5 times faster to decompress:
https://bugzilla.redhat.com/show_bug.cgi?id=1648490#c6

'zstd' support for squash filesystem is in the kernel already so it would be a matter to patch squashfs-tools with 'zstd' support. Did anybody try that?
If yes then please share results :)

Thanks
Please add [Solved] to your thread title if the solution was found.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2060
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

ZSTD compression instead of XZ for the modules

Post#2 by ncmprhnsbl » 31 May 2019, 09:30

fanthom wrote:
30 May 2019, 17:06
'zstd' support for squash filesystem is in the kernel already so it would be a matter to patch squashfs-tools with 'zstd' support. Did anybody try that?
this repo: https://github.com/plougher/squashfs-to ... shfs-tools seems to indicate that has incorporated zstd support..
indeed mksquashfs --help in porteus yields:

Code: Select all

	zstd
	  -Xcompression-level <compression-level>
		<compression-level> should be 1 .. 22 (default 15)
nothing to do but try it out, i guess

EDIT
quick compression test: mksquashfs %f test.zstdm -comp zstd -b 256K (so that's default compression level 15)
dir = 47.4 MB
xzm = 12.8 MB 27%
zstd = 14.5 MB 30.6%
compression seemed pretty snappy (bit small to tell)
EDIT2
further tests:
at compression levels:
22 = 14.4 MB 30.1%
1 = 17.0 MB 35.9%
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

User avatar
fanthom
Site Admin
Site Admin
Posts: 5095
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

ZSTD compression instead of XZ for the modules

Post#3 by fanthom » 31 May 2019, 12:36

Hi ncmprhnsbl,

Please try highest compression level for zstd and check module size then.

You can also measure compression time (however this is less important than decompression):

Code: Select all

time mksquashfs %f test.zstdm -comp zstd -b 256K -Xcompression-level 22
Would be great if you could mount both modules (kernel must have zstd support enabled) under /mnt then copy content of the module to tmpfs (RAM) to measure real decompression speed:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd
time cp -a /mnt/xzm/* /tmp/xzm
Wondering about results :)

Added in 8 minutes 7 seconds:
Well - seems Gentoo also has zstd support enabled in squashfs-tools so i can easily do these tests myself :)

Will post results later today.
Please add [Solved] to your thread title if the solution was found.

User avatar
ncmprhnsbl
DEV Team
DEV Team
Posts: 2060
Joined: 20 Mar 2012, 03:42
Distribution: 5.0rc1-64bit all-DE+more
Location: australia
Contact:

ZSTD compression instead of XZ for the modules

Post#4 by ncmprhnsbl » 31 May 2019, 13:13

here's some decompress times: same 47.4 dir
unsquashfs 12.8 MB xzm
real 0m1.399s
user 0m2.762s
sys 0m0.635s

unsquashfs 14.4 MB zstd (compression 22)
real 0m3.634s
user 0m0.436s
sys 0m0.738s
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44

User avatar
fanthom
Site Admin
Site Admin
Posts: 5095
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

ZSTD compression instead of XZ for the modules

Post#5 by fanthom » 31 May 2019, 13:45

Compression is worse comparing to xz but decompression is 7 time faster in favor of zstd in my tests:

Dir size: 263 MB

Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)

Decompression time:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd

real	0m1.381s
user	0m0.017s
sys	0m1.220s

time cp -a /mnt/xz/* /tmp/xz

real	0m9.709s
user	0m0.016s
sys	0m9.315s
Not sure if the speed is worth bigger module sizes ...
Please add [Solved] to your thread title if the solution was found.

babam
Black ninja
Black ninja
Posts: 57
Joined: 16 Nov 2016, 10:30
Distribution: Porteus 3.2.2
Location: Rainy city

ZSTD compression instead of XZ for the modules

Post#6 by babam » 02 Jun 2019, 20:58

fanthom wrote:
31 May 2019, 13:45
Compression is worse comparing to xz but decompression is 7 time faster in favor of zstd in my tests:

Dir size: 263 MB

Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)

Decompression time:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd

real	0m1.381s
user	0m0.017s
sys	0m1.220s

time cp -a /mnt/xz/* /tmp/xz

real	0m9.709s
user	0m0.016s
sys	0m9.315s
Not sure if the speed is worth bigger module sizes ...
Can you test it by placing zstd and xz modules in the NTFS partition.

I doubt the zstd decompression speed if it is placed in the NTFS partition.
Sorry, my English is bad.

raja
Shogun
Shogun
Posts: 201
Joined: 02 May 2017, 09:51
Distribution: v3.2.2-32 and v5.0-rc1-zstd-64
Location: Chennai,India

ZSTD compression instead of XZ for the modules

Post#7 by raja » 03 Jun 2019, 16:48

Zstd compression available in squashfs-tools in Slackware. I got it through USM. Check alien database.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.

Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.
Linux Kernel-4.4.120-32 bit; Linux kernel-5.2-64 bit.

raja
Shogun
Shogun
Posts: 201
Joined: 02 May 2017, 09:51
Distribution: v3.2.2-32 and v5.0-rc1-zstd-64
Location: Chennai,India

ZSTD compression instead of XZ for the modules

Post#8 by raja » 03 Jun 2019, 16:48

Zstd compression available in squashfs-tools in Slackware. I got it through USM. Check alien database.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.

Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.
Linux Kernel-4.4.120-32 bit; Linux kernel-5.2-64 bit.

Post Reply