xz via mc folder compression is not able to use multiple CPU cores
Posted: 22 Mar 2020, 04:46
I discovered that while dir2zxm being able to use multiple CPU cores when compressing to an xzm (aka xz compressed module), when xz via mc folder compression is not able to use multiple CPU cores?
I compressed a folder using the "build in" folder compression ability of mc and this was the result on a 8 core machine:
Compressing a xz compressed Porteus module usually ranges between 600% and 750% on a 8 core system.
I managed to capture the temp script that mc creates in /tmp/mc-root/ :
man xz says this:
So, I presume when we are able to tweak the mc setup to have mc create the script using
instead of should do the trick, and should have no adverse effects, since it only uses as many CPU cores as there are, e.g. on a single core system it would only use one, and on a multi-core system more than one. I presume all users would agree in having compression work as fast as possible is usually appreciated.
I did a manual test of the parameters.
I moved into ".." as in the script PWD above and then did in the terminal:and this was the result top-wise:
What do you developer folks think?
Should be mc script for handling compressions (possible the same is true for other compression utilities that are multi-core able) be altered like I did above?
I compressed a folder using the "build in" folder compression ability of mc and this was the result on a 8 core machine:
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3041 root 20 0 100588 97436 2064 R 83.3 2.5 0:07.61 xz
[...]
3041 root 20 0 100588 97436 2064 R 94.1 2.5 0:12.05 xz
I managed to capture the temp script that mc creates in /tmp/mc-root/ :
Code: Select all
#! /bin/sh
Pwd=`basename /path/to/folder/to/compress /`
echo -n "Name of the compressed file (without extension) [$Pwd]: "
read tar
[ "$tar"x = x ] && tar="$Pwd"
cd .. && \
tar cf - "$Pwd" | xz -f > "$tar.tar.xz" && \
echo "../$tar.tar.xz created."
Code: Select all
-T threads, --threads=threads
Specify the number of worker threads to use. Setting threads to a
special value 0 makes xz use as many threads as there are CPU cores on
the system. The actual number of threads can be less than threads if
the input file is not big enough for threading with the given settings
or if using more threads would exceed the memory usage limit.
Code: Select all
tar cf - "$Pwd" | xz -fT 0 > "$tar.tar.xz" && \
Code: Select all
tar cf - "$Pwd" | xz -f > "$tar.tar.xz" && \
I did a manual test of the parameters.
I moved into ".." as in the script PWD above and then did in the terminal:
Code: Select all
tar cf - TARGETDIR|xz -fT 0 >TARGETDIR.tar.xz
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3480 root 20 0 1559936 355360 1976 S 511.1 9.0 0:05.85 xz
[…]
3480 root 20 0 1947056 848720 1976 S 711.1 21.4 0:34.19 xz
[…]
3480 root 20 0 1947056 971480 1976 S 776.5 24.5 0:48.13 xz
Should be mc script for handling compressions (possible the same is true for other compression utilities that are multi-core able) be altered like I did above?