converting .txz files into .xzm for a whole folder

For discussions about programming and projects not necessarily associated with Porteus.
Bogomips
Full of knowledge
Full of knowledge
Posts: 2539
Joined: 25 Jun 2014, 15:21
Distribution: 3.2.2 Cinnamon & KDE5
Location: London

Re: converting .txz files into .xzm for a whole folder

Post#16 by Bogomips » 31 Jul 2015, 16:00

francois wrote:So some kind of function to merge xzms.
Would be surprised if francois was unaware of the existing merge script. Anyway, regarding this as a programming exercise, keeping one's Bash in trim, have tried to incorporate the two simple statements into a function, and have come up with a simple script which should do the trick. Hope all bases covered. :Search:

Being somewhat chary of using others scripts, it will also serve as a script in reserve for me. The script writers seem to always make the assumption that /tmp is a reservoir of unlimited free memory, and do not realise that on this system, what goes into temp comes out of my home, which could result in everything grinding to a standstill. :twisted: Due to this assumption, neko's script did not work, but he is changing it now to allow for an option to take this into account.

So if you wish to have yet another script, enjoy mrgmod!
Script Updated: https://forum.porteus.org/viewtopic.php?f=39&t=4896

Code: Select all

#!/bin/bash
# 2015-07-26 00:31:07
mgm()
{
#	Usage:  mrgmod  [-nv]   [./]    [/Path/to/]<merged modules>.xzm  [/tmp]
# mrgmod  [Non-verbose]   [<modules directory>]   [/Path/to/]<merged module>.xzm  [<An Existing Directory for Temporary Files>]
# md: <modules directory>	mf: <merged modules file>.xzm	mt: <Temporary Files Directory>
# Non-verbose Option?
vb=1; ve=0; op=$1;
[[ $1 ]] && ! [[ ${1%%-*} ]] && { shift; vb=0; [[ $op != -nv ]] && ve=1; }
((ve)) && echo "Incorrect option '$op'!";
for o in $*; do ! [[ ${o%%-*} ]] && ve=1 && echo "Error: Option '$o' Out of Place!"; done
np=$#;	# of parameters
if [[ np -eq 0 || np -gt 3 || ve -eq 1 ]]; then [[ np -gt 3 ]] && echo Error: Too many arguments!
  na=${0##*/}; na=${na%.sh}
  printf "Usage:\t%s\t[-nv]\t[./]\t[/Path/to/]<merged module>.xzm\t[/tmp]\n\n" $na
  printf "%s\t[Non-verbose]\t[<modules directory>]\t[/Path/to/]<merged module>.xzm\t[<An Existing Directory for Temporary Files>]\n" $na
  exit $np
fi
# 
pa=($1 $2 $3); let m=np-1; md=""; mt=/tmp;	# Default modules dir and temp files dir 
# Work out arguments. Brginning from last non-null parameter value.
if [[ -d ${pa[m]} ]]; then	[[ np -eq 1 ]] && { echo Error: No File for Merged Module!; exit 1; }
  mf=${pa[m-1]}; [[ np -eq 3 ]] && md=$1;
else	[[ np -eq 3 ]] && { if ! [[ $3 =~ .+\.xzm$ ]]; then echo "Error: '$3' Non-existent Temporary Files Directory!"; else echo "Error: Superfluous Modules Directory '$2'!"; fi; exit 3; }
  mf=${pa[m]}; [[ np -eq 2 ]] && md=$1;
fi
! [[ $mf =~ .+\.xzm$ ]] && { echo "Error: Module File '$mf' hasn't xzm extension!"; exit 2; }
[[ ! -d ${md:-.} ]] && { echo "Error: Modules Directory '$md' Non-existent!"; exit 3; }
! [[ -w $(dirname $mf) ]] && { echo "Permission: Cannot create Merged Modules File '$mf'!"; exit 2; };
# Create fake root directory
mfr=$mt/mfr$$; mkdir $mfr; (($?)) && { echo "Error: Cannot create Temporary Files in '$mt' Directory!"; exit 3; } 
mm()
{
  [[ $# -lt 2 ]] && { echo "Error: Merging Modules of ${md:-Current} Directory. Less than 2 Modules for Merging!"; return 1; }
  echo $'\n'"Processing Modules of ${md:-Current} Directory."; ((vb)) && echo;
  for a in $*;
  do 
    xzm2dir $a $mfr > /dev/null && { ((vb)) && echo "Processed Module: ${a##*/}"; continue; }
    echo "Error: Processing Module '$a'!"; return 1; 
  done
  echo $'\n'"->  Merged Modules File '$mf'"$'\n';
  dir2xzm $mfr $mf && echo $'\n'"Merged Modules File '$mf' Created." && return 0;
  echo Error: Merging Modules!; return 1;
}
  mm ${md:+$md/}*.xzm; (($?)) && [[ -f $mf ]] && rm $mf; 
  rm -fr $mfr
}
mgm $*
Invocation Example:

Code: Select all

root@porteus:/home/guest# sh p10/Por/mrgmod.sh /tmp/usm/openjre-7u79_b14 openjre-7u79_b14.xzm /mnt/sda9/Junk    

Processing Modules of /tmp/usm/openjre-7u79_b14 Directory.

Processed Module: aaa_elflibs-14.1-i486-3.xzm
Processed Module: openjre-7u79_b14-i486-2sl.xzm
Processed Module: openssl-solibs-1.0.1e-i486-1.xzm
Processed Module: rhino-1_7R4-i486-3sl.xzm

->  Merged Modules File 'openjre-7u79_b14.xzm'

Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on openjre-7u79_b14.xzm, block size 262144.
[=========================================================================|] 1219/1219 100%

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 262144
        compressed data, compressed metadata, compressed fragments, compressed xattrs
        duplicates are removed
Filesystem size 48251.71 Kbytes (47.12 Mbytes)
        53.70% of uncompressed filesystem size (89853.29 Kbytes)
Inode table size 9734 bytes (9.51 Kbytes)
        25.73% of uncompressed inode table size (37832 bytes)
Directory table size 11062 bytes (10.80 Kbytes)
        46.50% of uncompressed directory table size (23790 bytes)
Number of duplicate files found 28
Number of inodes 1120
Number of files 945
Number of fragments 66
Number of symbolic links  79
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 96
Number of ids (unique uids + gids) 1
Number of uids 1
        root (0)
Number of gids 1
        root (0)

Merged Modules File 'openjre-7u79_b14.xzm' Created.
______________________________________________________________________________
______________________________________________________________________________
root@porteus:/home/guest# sh p10/Por/mrgmod.sh -nv /tmp/usm/openjre-7u79_b14 openjre-7u79_b14.xzm /mnt/sda9/Junk

Processing Modules of /tmp/usm/openjre-7u79_b14 Directory.

->  Merged Modules File 'openjre-7u79_b14.xzm'

Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on openjre-7u79_b14.xzm, block size 262144.
[=========================================================================|] 1219/1219 100%

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 262144
        compressed data, compressed metadata, compressed fragments, compressed xattrs
        duplicates are removed
Filesystem size 48251.72 Kbytes (47.12 Mbytes)
        53.70% of uncompressed filesystem size (89853.29 Kbytes)
Inode table size 9738 bytes (9.51 Kbytes)
        25.74% of uncompressed inode table size (37832 bytes)
Directory table size 11062 bytes (10.80 Kbytes)
        46.50% of uncompressed directory table size (23790 bytes)
Number of duplicate files found 28
Number of inodes 1120
Number of files 945
Number of fragments 66
Number of symbolic links  79
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 96
Number of ids (unique uids + gids) 1
Number of uids 1
        root (0)
Number of gids 1
        root (0)

Merged Modules File 'openjre-7u79_b14.xzm' Created.
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

User avatar
francois
Contributor
Contributor
Posts: 4942
Joined: 28 Dec 2010, 14:25
Distribution: kde xfce porteus manjaro kubun
Location: Enfin l'été, le changement climatique attendu: le soleil.

Re: converting .txz files into .xzm for a whole folder

Post#17 by francois » 31 Jul 2015, 21:33

Thanks all. This is comforting to have so many hands on that problem of mine. :)
Voltaire: Le mieux est l'ennemi du bien.

User avatar
francois
Contributor
Contributor
Posts: 4942
Joined: 28 Dec 2010, 14:25
Distribution: kde xfce porteus manjaro kubun
Location: Enfin l'été, le changement climatique attendu: le soleil.

Re: converting .txz files into .xzm for a whole folder

Post#18 by francois » 10 Jan 2016, 16:05

Here is the answer to converting a whole folder of txz to a single module:
use usm tools, there is a function to do that exactly.

Also usefull is freestyler mergexzm, that converts a bunch of modules into a single merged module:
https://forum.porteus.org/viewtopic.php?f=75&t=3532
Voltaire: Le mieux est l'ennemi du bien.

Post Reply