Page 1 of 1

ZSTD compression instead of XZ for the modules

Posted: 30 May 2019, 17:06
by fanthom
Interesting read:
https://www.phoronix.com/scan.php?page= ... RPM-Change

Seems the XZM modules would be only few percent larger but 5 times faster to decompress:
https://bugzilla.redhat.com/show_bug.cgi?id=1648490#c6

'zstd' support for squash filesystem is in the kernel already so it would be a matter to patch squashfs-tools with 'zstd' support. Did anybody try that?
If yes then please share results :)

Thanks

ZSTD compression instead of XZ for the modules

Posted: 31 May 2019, 09:30
by ncmprhnsbl
fanthom wrote:
30 May 2019, 17:06
'zstd' support for squash filesystem is in the kernel already so it would be a matter to patch squashfs-tools with 'zstd' support. Did anybody try that?
this repo: https://github.com/plougher/squashfs-to ... shfs-tools seems to indicate that has incorporated zstd support..
indeed mksquashfs --help in porteus yields:

Code: Select all

	zstd
	  -Xcompression-level <compression-level>
		<compression-level> should be 1 .. 22 (default 15)
nothing to do but try it out, i guess

EDIT
quick compression test: mksquashfs %f test.zstdm -comp zstd -b 256K (so that's default compression level 15)
dir = 47.4 MB
xzm = 12.8 MB 27%
zstd = 14.5 MB 30.6%
compression seemed pretty snappy (bit small to tell)
EDIT2
further tests:
at compression levels:
22 = 14.4 MB 30.1%
1 = 17.0 MB 35.9%

ZSTD compression instead of XZ for the modules

Posted: 31 May 2019, 12:36
by fanthom
Hi ncmprhnsbl,

Please try highest compression level for zstd and check module size then.

You can also measure compression time (however this is less important than decompression):

Code: Select all

time mksquashfs %f test.zstdm -comp zstd -b 256K -Xcompression-level 22
Would be great if you could mount both modules (kernel must have zstd support enabled) under /mnt then copy content of the module to tmpfs (RAM) to measure real decompression speed:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd
time cp -a /mnt/xzm/* /tmp/xzm
Wondering about results :)

Added in 8 minutes 7 seconds:
Well - seems Gentoo also has zstd support enabled in squashfs-tools so i can easily do these tests myself :)

Will post results later today.

ZSTD compression instead of XZ for the modules

Posted: 31 May 2019, 13:13
by ncmprhnsbl
here's some decompress times: same 47.4 dir
unsquashfs 12.8 MB xzm
real 0m1.399s
user 0m2.762s
sys 0m0.635s

unsquashfs 14.4 MB zstd (compression 22)
real 0m3.634s
user 0m0.436s
sys 0m0.738s

ZSTD compression instead of XZ for the modules

Posted: 31 May 2019, 13:45
by fanthom
Compression is worse comparing to xz but decompression is 7 time faster in favor of zstd in my tests:

Dir size: 263 MB

Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)

Decompression time:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd

real	0m1.381s
user	0m0.017s
sys	0m1.220s

time cp -a /mnt/xz/* /tmp/xz

real	0m9.709s
user	0m0.016s
sys	0m9.315s
Not sure if the speed is worth bigger module sizes ...

ZSTD compression instead of XZ for the modules

Posted: 02 Jun 2019, 20:58
by babam
fanthom wrote:
31 May 2019, 13:45
Compression is worse comparing to xz but decompression is 7 time faster in favor of zstd in my tests:

Dir size: 263 MB

Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)

Decompression time:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd

real	0m1.381s
user	0m0.017s
sys	0m1.220s

time cp -a /mnt/xz/* /tmp/xz

real	0m9.709s
user	0m0.016s
sys	0m9.315s
Not sure if the speed is worth bigger module sizes ...
Can you test it by placing zstd and xz modules in the NTFS partition.

I doubt the zstd decompression speed if it is placed in the NTFS partition.

ZSTD compression instead of XZ for the modules

Posted: 03 Jun 2019, 16:48
by raja
Zstd compression available in squashfs-tools in Slackware. I got it through USM. Check alien database.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.

Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.

ZSTD compression instead of XZ for the modules

Posted: 03 Jun 2019, 16:48
by raja
Zstd compression available in squashfs-tools in Slackware. I got it through USM. Check alien database.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.

Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.