ZSTD compression instead of XZ for the modules
- fanthom
- Moderator Team
- Posts: 5667
- Joined: 28 Dec 2010, 02:42
- Distribution: Porteus Kiosk
- Location: Poland
- Contact:
ZSTD compression instead of XZ for the modules
Interesting read:
https://www.phoronix.com/scan.php?page= ... RPM-Change
Seems the XZM modules would be only few percent larger but 5 times faster to decompress:
https://bugzilla.redhat.com/show_bug.cgi?id=1648490#c6
'zstd' support for squash filesystem is in the kernel already so it would be a matter to patch squashfs-tools with 'zstd' support. Did anybody try that?
If yes then please share results
Thanks
https://www.phoronix.com/scan.php?page= ... RPM-Change
Seems the XZM modules would be only few percent larger but 5 times faster to decompress:
https://bugzilla.redhat.com/show_bug.cgi?id=1648490#c6
'zstd' support for squash filesystem is in the kernel already so it would be a matter to patch squashfs-tools with 'zstd' support. Did anybody try that?
If yes then please share results
Thanks
Please add [Solved] to your thread title if the solution was found.
- ncmprhnsbl
- DEV Team
- Posts: 3936
- Joined: 20 Mar 2012, 03:42
- Distribution: v5.0-64bit
- Location: australia
- Contact:
ZSTD compression instead of XZ for the modules
this repo: https://github.com/plougher/squashfs-to ... shfs-tools seems to indicate that has incorporated zstd support..
indeed mksquashfs --help in porteus yields:
Code: Select all
zstd
-Xcompression-level <compression-level>
<compression-level> should be 1 .. 22 (default 15)
EDIT
quick compression test: mksquashfs %f test.zstdm -comp zstd -b 256K (so that's default compression level 15)
dir = 47.4 MB
xzm = 12.8 MB 27%
zstd = 14.5 MB 30.6%
compression seemed pretty snappy (bit small to tell)
EDIT2
further tests:
at compression levels:
22 = 14.4 MB 30.1%
1 = 17.0 MB 35.9%
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44
- fanthom
- Moderator Team
- Posts: 5667
- Joined: 28 Dec 2010, 02:42
- Distribution: Porteus Kiosk
- Location: Poland
- Contact:
ZSTD compression instead of XZ for the modules
Hi ncmprhnsbl,
Please try highest compression level for zstd and check module size then.
You can also measure compression time (however this is less important than decompression):
Would be great if you could mount both modules (kernel must have zstd support enabled) under /mnt then copy content of the module to tmpfs (RAM) to measure real decompression speed:
Wondering about results
Added in 8 minutes 7 seconds:
Well - seems Gentoo also has zstd support enabled in squashfs-tools so i can easily do these tests myself
Will post results later today.
Please try highest compression level for zstd and check module size then.
You can also measure compression time (however this is less important than decompression):
Code: Select all
time mksquashfs %f test.zstdm -comp zstd -b 256K -Xcompression-level 22
Code: Select all
time cp -a /mnt/zstd/* /tmp/zstd
time cp -a /mnt/xzm/* /tmp/xzm
Added in 8 minutes 7 seconds:
Well - seems Gentoo also has zstd support enabled in squashfs-tools so i can easily do these tests myself
Will post results later today.
Please add [Solved] to your thread title if the solution was found.
- ncmprhnsbl
- DEV Team
- Posts: 3936
- Joined: 20 Mar 2012, 03:42
- Distribution: v5.0-64bit
- Location: australia
- Contact:
ZSTD compression instead of XZ for the modules
here's some decompress times: same 47.4 dir
unsquashfs 12.8 MB xzm
real 0m1.399s
user 0m2.762s
sys 0m0.635s
unsquashfs 14.4 MB zstd (compression 22)
real 0m3.634s
user 0m0.436s
sys 0m0.738s
unsquashfs 12.8 MB xzm
real 0m1.399s
user 0m2.762s
sys 0m0.635s
unsquashfs 14.4 MB zstd (compression 22)
real 0m3.634s
user 0m0.436s
sys 0m0.738s
Forum Rules : https://forum.porteus.org/viewtopic.php?f=35&t=44
- fanthom
- Moderator Team
- Posts: 5667
- Joined: 28 Dec 2010, 02:42
- Distribution: Porteus Kiosk
- Location: Poland
- Contact:
ZSTD compression instead of XZ for the modules
Compression is worse comparing to xz but decompression is 7 time faster in favor of zstd in my tests:
Dir size: 263 MB
Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)
Decompression time:
Not sure if the speed is worth bigger module sizes ...
Dir size: 263 MB
Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)
Decompression time:
Code: Select all
time cp -a /mnt/zstd/* /tmp/zstd
real 0m1.381s
user 0m0.017s
sys 0m1.220s
time cp -a /mnt/xz/* /tmp/xz
real 0m9.709s
user 0m0.016s
sys 0m9.315s
Please add [Solved] to your thread title if the solution was found.
- babam
- Warlord
- Posts: 528
- Joined: 16 Nov 2016, 10:30
- Distribution: Porteus 5.0rc3 Xfce K6.1.1
- Location: Rainy city
ZSTD compression instead of XZ for the modules
Can you test it by placing zstd and xz modules in the NTFS partition.fanthom wrote: ↑31 May 2019, 13:45Compression is worse comparing to xz but decompression is 7 time faster in favor of zstd in my tests:
Dir size: 263 MB
Module size:
xzm: 44 800 KB (squashfs block size 256K)
zstd: 50 800 KB (squashfs block size 1024K, compression level: 22 - maximum available)
Decompression time:
Not sure if the speed is worth bigger module sizes ...Code: Select all
time cp -a /mnt/zstd/* /tmp/zstd real 0m1.381s user 0m0.017s sys 0m1.220s time cp -a /mnt/xz/* /tmp/xz real 0m9.709s user 0m0.016s sys 0m9.315s
I doubt the zstd decompression speed if it is placed in the NTFS partition.
Sorry, my English is bad.
-
- Shogun
- Posts: 434
- Joined: 02 May 2017, 09:51
- Distribution: v3.2.2-32 and Porteus-Artix-64
- Location: Chennai,India
ZSTD compression instead of XZ for the modules
Zstd compression available in squashfs-tools in Slackware. I got it through USM. Check alien database.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.
Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.
Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.
Linux Kernel-4.4.272 -32 bit; Linux Kernel-5.4.185 - 64 bit
-
- Shogun
- Posts: 434
- Joined: 02 May 2017, 09:51
- Distribution: v3.2.2-32 and Porteus-Artix-64
- Location: Chennai,India
ZSTD compression instead of XZ for the modules
Zstd compression available in squashfs-tools in Slackware. I got it through USM. Check alien database.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.
Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.
A xzm module, 300MB was , 318MB in zstd. Compresion level 22.
Speed has to be checked when an ISO is made with zst and deployed, in real time opening applications.
Linux Kernel-4.4.272 -32 bit; Linux Kernel-5.4.185 - 64 bit
ZSTD compression instead of XZ for the modules
Testing larger memory pages. (mksquashfs ... -b 256K vs -b 1M ...)
SYSTEM SPECS: CPU Intel Core i5, RAM: 8GB's
A compressor using bigger chunks of data should be able to compress more efficiently, but it relies on the memory of the host computer where compressed data will be used. Also, there is no point in using large memory pages if you are going to compress a few Mb's. (<=100 Mb), but...
COMPRESSION TIME:
XZ has clearly better times. While it seems to lag at the beginning, it almost halves the whole process. I.e. The more room is allocated, the faster the CPU will move the data.
ZSTD seems not to benefit of this matter.
COMPRESSION RATIO:
When using large memory pages, final size is around 6% less in both cases, but XZ compression factor was 13% better.
DECOMPRESSION SPEED:
I haven't really tested the decompression speed which seems to be the real deal, but I'd swear I didn't noticed any substantial improvement.
Cheers!
SYSTEM SPECS: CPU Intel Core i5, RAM: 8GB's
A compressor using bigger chunks of data should be able to compress more efficiently, but it relies on the memory of the host computer where compressed data will be used. Also, there is no point in using large memory pages if you are going to compress a few Mb's. (<=100 Mb), but...
COMPRESSION TIME:
XZ has clearly better times. While it seems to lag at the beginning, it almost halves the whole process. I.e. The more room is allocated, the faster the CPU will move the data.
ZSTD seems not to benefit of this matter.
COMPRESSION RATIO:
When using large memory pages, final size is around 6% less in both cases, but XZ compression factor was 13% better.
DECOMPRESSION SPEED:
I haven't really tested the decompression speed which seems to be the real deal, but I'd swear I didn't noticed any substantial improvement.
Code: Select all
time mksquashfs 001-core-20200310/ 001-core-20200310.xzm -b 1M -comp xz -Xbcj x86 -noappend
real 2m20,282s
user 6m55,109s
sys 0m1,013s
time mksquashfs 001-core-20200310/ 001-core-20200310.zstd -b 1M -comp zstd -Xcompression-level 22 -noappend
real 1m18,504s
user 3m35,465s
sys 0m0,873s
-rwxrwxrwx 1 root root 115814400 may 21 11:53 001-core-20200310.xzm*
-rwxrwxrwx 1 root root 130097152 may 21 12:03 001-core-20200310.zstd*
time mksquashfs 001-core-20200310/ 001-core-20200310.xzm -b 256K -comp xz -Xbcj x86 -noappend
real 1m34,769s
user 7m36,185s
sys 0m1,793s
time mksquashfs 001-core-20200310/ 001-core-20200310.zstd -b 256K -comp zstd -Xcompression-level 22 -noappend
real 1m8,756s
user 5m17,440s
sys 0m0,832s
-rwxrwxrwx 1 root root 123674624 may 21 12:15 001-core-20200310.xzm*
-rwxrwxrwx 1 root root 139001856 may 21 12:17 001-core-20200310.zstd*
> Does not compute_
https://forum.porteus.org/viewtopic.php?p=94310#p94310
https://forum.porteus.org/viewtopic.php?p=84002#p84002
https://forum.porteus.org/viewtopic.php?p=77174#p77174
https://forum.porteus.org/viewtopic.php?f=39&t=8584
https://forum.porteus.org/viewtopic.php?p=94310#p94310
https://forum.porteus.org/viewtopic.php?p=84002#p84002
https://forum.porteus.org/viewtopic.php?p=77174#p77174
https://forum.porteus.org/viewtopic.php?f=39&t=8584
ZSTD compression instead of XZ for the modules
> Does not compute_
https://forum.porteus.org/viewtopic.php?p=94310#p94310
https://forum.porteus.org/viewtopic.php?p=84002#p84002
https://forum.porteus.org/viewtopic.php?p=77174#p77174
https://forum.porteus.org/viewtopic.php?f=39&t=8584
https://forum.porteus.org/viewtopic.php?p=94310#p94310
https://forum.porteus.org/viewtopic.php?p=84002#p84002
https://forum.porteus.org/viewtopic.php?p=77174#p77174
https://forum.porteus.org/viewtopic.php?f=39&t=8584
ZSTD compression instead of XZ for the modules
I did a boot time test here with just the essential modules:
Normal compression (-comp xz -b 256K -Xbcj x86): 14s
Zstd compression (-comp zstd -b 256K -Xcompression-level 22) : 9.5s
It's pretty impressive!
The difference in size was:
kernel: 65.5 MB vs 72.7 MB
core: 127.2 MB vs 147.6 MB
xorg: 101.6 MB vs 117.9 MB
xfce: 11.1 MB vs 12.4 MB
Normal compression (-comp xz -b 256K -Xbcj x86): 14s
Zstd compression (-comp zstd -b 256K -Xcompression-level 22) : 9.5s
It's pretty impressive!
The difference in size was:
kernel: 65.5 MB vs 72.7 MB
core: 127.2 MB vs 147.6 MB
xorg: 101.6 MB vs 117.9 MB
xfce: 11.1 MB vs 12.4 MB