ZSTD compression instead of XZ for the modules

[fanthom]

Interesting read:
https://www.phoronix.com/scan.php?page= ... RPM-Change

Seems the XZM modules would be only few percent larger but 5 times faster to decompress:
https://bugzilla.redhat.com/show_bug.cgi?id=1648490#c6

'zstd' support for squash filesystem is in the kernel already so it would be a matter to patch squashfs-tools with 'zstd' support. Did anybody try that?
If yes then please share results

Thanks

[ncmprhnsbl]

'zstd' support for squash filesystem is in the kernel already so it would be a matter to patch squashfs-tools with 'zstd' support. Did anybody try that?

this repo: https://github.com/plougher/squashfs-to ... shfs-tools seems to indicate that has incorporated zstd support..
indeed mksquashfs --help in porteus yields:

Code: Select all

	zstd
	  -Xcompression-level <compression-level>
		<compression-level> should be 1 .. 22 (default 15)

nothing to do but try it out, i guess

EDIT
quick compression test: mksquashfs %f test.zstdm -comp zstd -b 256K (so that's default compression level 15)
dir = 47.4 MB
xzm = 12.8 MB 27%
zstd = 14.5 MB 30.6%
compression seemed pretty snappy (bit small to tell)
EDIT2
further tests:
at compression levels:
22 = 14.4 MB 30.1%
1 = 17.0 MB 35.9%

[fanthom]

Hi ncmprhnsbl,

Please try highest compression level for zstd and check module size then.

You can also measure compression time (however this is less important than decompression):

Code: Select all

time mksquashfs %f test.zstdm -comp zstd -b 256K -Xcompression-level 22

Would be great if you could mount both modules (kernel must have zstd support enabled) under /mnt then copy content of the module to tmpfs (RAM) to measure real decompression speed:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd
time cp -a /mnt/xzm/* /tmp/xzm

Wondering about results

Added in 8 minutes 7 seconds:
Well - seems Gentoo also has zstd support enabled in squashfs-tools so i can easily do these tests myself

Will post results later today.

[ncmprhnsbl]

here's some decompress times: same 47.4 dir
unsquashfs 12.8 MB xzm
real 0m1.399s
user 0m2.762s
sys 0m0.635s

unsquashfs 14.4 MB zstd (compression 22)
real 0m3.634s
user 0m0.436s
sys 0m0.738s

[fanthom]

Compression is worse comparing to xz but decompression is 7 time faster in favor of zstd in my tests:

Dir size: 263 MB

Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)

Decompression time:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd

real	0m1.381s
user	0m0.017s
sys	0m1.220s

time cp -a /mnt/xz/* /tmp/xz

real	0m9.709s
user	0m0.016s
sys	0m9.315s

Not sure if the speed is worth bigger module sizes ...

[babam]

Compression is worse comparing to xz but decompression is 7 time faster in favor of zstd in my tests:

Dir size: 263 MB

Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)

Decompression time:

Code: Select all

time cp -a /mnt/zstd/* /tmp/zstd

real	0m1.381s
user	0m0.017s
sys	0m1.220s

time cp -a /mnt/xz/* /tmp/xz

real	0m9.709s
user	0m0.016s
sys	0m9.315s

Not sure if the speed is worth bigger module sizes ...

Can you test it by placing zstd and xz modules in the NTFS partition.

I doubt the zstd decompression speed if it is placed in the NTFS partition.

[raja]

Zstd compression available in squashfs-tools in Slackware. I got it through USM. Check alien database.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.

Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.

[raja]

Zstd compression available in squashfs-tools in Slackware. I got it through USM. Check alien database.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.

Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.

[M. Eerie]

Testing larger memory pages. (mksquashfs ... -b 256K vs -b 1M ...)
SYSTEM SPECS: CPU Intel Core i5, RAM: 8GB's

A compressor using bigger chunks of data should be able to compress more efficiently, but it relies on the memory of the host computer where compressed data will be used. Also, there is no point in using large memory pages if you are going to compress a few Mb's. (<=100 Mb), but...

COMPRESSION TIME:
XZ has clearly better times. While it seems to lag at the beginning, it almost halves the whole process. I.e. The more room is allocated, the faster the CPU will move the data.
ZSTD seems not to benefit of this matter.

COMPRESSION RATIO:
When using large memory pages, final size is around 6% less in both cases, but XZ compression factor was 13% better.

DECOMPRESSION SPEED:
I haven't really tested the decompression speed which seems to be the real deal, but I'd swear I didn't noticed any substantial improvement.

Code: Select all

time mksquashfs 001-core-20200310/ 001-core-20200310.xzm -b 1M -comp xz -Xbcj x86 -noappend
real	2m20,282s
user	6m55,109s
sys	0m1,013s


time mksquashfs 001-core-20200310/ 001-core-20200310.zstd -b 1M -comp zstd -Xcompression-level 22 -noappend
real	1m18,504s
user	3m35,465s
sys	0m0,873s

-rwxrwxrwx 1 root root 115814400 may 21 11:53 001-core-20200310.xzm*
-rwxrwxrwx 1 root root 130097152 may 21 12:03 001-core-20200310.zstd*

time mksquashfs 001-core-20200310/ 001-core-20200310.xzm -b 256K -comp xz -Xbcj x86 -noappend
real	1m34,769s
user	7m36,185s
sys	0m1,793s

time mksquashfs 001-core-20200310/ 001-core-20200310.zstd -b 256K -comp zstd -Xcompression-level 22 -noappend
real	1m8,756s
user	5m17,440s
sys	0m0,832s

-rwxrwxrwx 1 root root 123674624 may 21 12:15 001-core-20200310.xzm*
-rwxrwxrwx 1 root root 139001856 may 21 12:17 001-core-20200310.zstd*

Cheers!

[M. Eerie]

The only Facebook contribution (imposition) I would admit to use

Last release is 50% faster

[fulalas]

I did a boot time test here with just the essential modules:
Normal compression (-comp xz -b 256K -Xbcj x86): 14s
Zstd compression (-comp zstd -b 256K -Xcompression-level 22) : 9.5s

It's pretty impressive!

The difference in size was:
kernel: 65.5 MB vs 72.7 MB
core: 127.2 MB vs 147.6 MB
xorg: 101.6 MB vs 117.9 MB
xfce: 11.1 MB vs 12.4 MB