xz via mc folder compression is not able to use multiple CPU cores

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
Post Reply
User avatar
Rava
Contributor
Contributor
Posts: 2480
Joined: 11 Jan 2011, 02:46
Distribution: Porteus 5.0rc1 x86-64 XFCe
Location: Forests of Germany

xz via mc folder compression is not able to use multiple CPU cores

Post#1 by Rava » 22 Mar 2020, 04:46

I discovered that while dir2zxm being able to use multiple CPU cores when compressing to an xzm (aka xz compressed module), when xz via mc folder compression is not able to use multiple CPU cores?

I compressed a folder using the "build in" folder compression ability of mc and this was the result on a 8 core machine:

Code: Select all

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 3041 root      20   0  100588  97436   2064 R  83.3   2.5   0:07.61 xz
[...]
 3041 root      20   0  100588  97436   2064 R  94.1   2.5   0:12.05 xz
Compressing a xz compressed Porteus module usually ranges between 600% and 750% on a 8 core system.

I managed to capture the temp script that mc creates in /tmp/mc-root/ :

Code: Select all

#! /bin/sh
Pwd=`basename /path/to/folder/to/compress /`
echo -n "Name of the compressed file (without extension) [$Pwd]: "
read tar
[ "$tar"x = x ] && tar="$Pwd"
cd .. && \
tar cf - "$Pwd" | xz -f > "$tar.tar.xz" && \
echo "../$tar.tar.xz created."
man xz says this:

Code: Select all

  -T threads, --threads=threads

   Specify the number of worker threads to use. Setting threads to a
   special value 0 makes xz use as many threads as there are CPU cores on
   the system. The actual number of threads can be less than threads if
   the input file is not big enough for threading with the given settings
   or if using more threads would exceed the memory usage limit.
So, I presume when we are able to tweak the mc setup to have mc create the script using

Code: Select all

tar cf - "$Pwd" | xz -fT 0 > "$tar.tar.xz" && \
instead of

Code: Select all

tar cf - "$Pwd" | xz -f > "$tar.tar.xz" && \
should do the trick, and should have no adverse effects, since it only uses as many CPU cores as there are, e.g. on a single core system it would only use one, and on a multi-core system more than one. I presume all users would agree in having compression work as fast as possible is usually appreciated.

I did a manual test of the parameters.
I moved into ".." as in the script PWD above and then did in the terminal:

Code: Select all

tar cf - TARGETDIR|xz -fT 0 >TARGETDIR.tar.xz
and this was the result top-wise:

Code: Select all

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 3480 root      20   0 1559936 355360   1976 S 511.1   9.0   0:05.85 xz
[…]
 3480 root      20   0 1947056 848720   1976 S 711.1  21.4   0:34.19 xz
[…]
 3480 root      20   0 1947056 971480   1976 S 776.5  24.5   0:48.13 xz
What do you developer folks think?

Should be mc script for handling compressions (possible the same is true for other compression utilities that are multi-core able) be altered like I did above?
Cheers!
Yours Rava

Post Reply