[SCRIPT] stxz2zxm (smart? txz converter)

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
Post Reply
User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

[SCRIPT] stxz2zxm (smart? txz converter)

Post#1 by Ahau » 10 Jan 2012, 18:20

The following discussion and script is intended for those with some familiarity in creating slackware packages and converting them to porteus modules.

Let me preface this by saying that I love the default 'txz2xzm' script that is included with Porteus. It is wonderfully simple; it does what you ask it to. It does it's job and you move on with your life. It performs flawlessly, on about 90-95% of slackware packages. However, there are some inherent flaws that can cause some big problems the rest of the time. These are not flaws in txz2xzm however, they are flaws in how some slackware packages are setup to handle installation to directories other than the root of your filesystem (/). Since txz2xzm uses the 'installpkg -root /some/other/dir' function to convert slackware packages into modules, we inherit these flaws all too often.

Lost already? I was! Here's a couple examples outlining what I mean:

The package 'seamonkey-solibs' contains a number of libraries, which must be seen by your system in order to be used. Rather than being placed in /usr/lib/, these libraries are placed in /usr/lib/seamonkey (assuming a 32-bit arch), and a line of text is echoed to /etc/ld.so.conf ("/usr/lib/seamonkey"). The ldconfig utility is run, and your system can now see all of the libraries in the seamonkey directory. All of these actions are carried out by the install script inside the .txz package, at /install/doinst.sh.

When we run txz2xzm on this package, however, the txz is installed to a fakeroot directory inside /tmp/, and one of the following things will happen:
1) if there is no /tmp/fakeroot/etc directory, the 'echo' command will fail. when the user activates his new module, none of his applications will be able to see the newly installed libraries.
2) if there happens to be a /tmp/fakeroot/etc directory, the 'echo' command will create a new ld.so.conf file containing one line (/usr/lib/seamonkey). This file will be placed into the module and, when activated, this will overwrite your existing system-wide /etc/ld.so.conf, and chaos will ensue.

Next example:

The package gnome-keyring depends upon glib2, and needs to have the utility /usr/bin/glib-compile-schemas run before it will function properly. When txz2xzm is run on this package, the install script calls for glib-compile-schemas, but since the installation has chrooted to /tmp/fakeroot, it cannot be found, and the resulting schemas are never compiled.

The above two issues wasted a lot of my time (yes, I freely admit that it was because I was too ignorant (or stubborn) to check the doinst.sh scripts in advance. To avoid further problems, I decided to create an alternate utility that would look for and warn me when "non standard" commands such as these are encountered.

Here's the script. To use it, copy the contents to a text file, name it 'stxz2xzm' and chmod +x it. I put mine in /usr/bin/ so that it can be accessed from anywhere.

Code: Select all

#!/bin/bash
# convert Slackware's TGZ (or TXZ) package
# or a directory full of TGZ/TXZ packages into .xzm compressed file
# which can be used as a LiveCD module
#
# Author: Ahau <ahau@porteus.org>
# Based on the txz2xzm script by Thomas M <http://www.linux-live.org>
# As modified for Porteus by fanthom
# Also borrowing efforts by brokenman, for converting
# xzm's into compliance with the Porteus Package Manager.

# Switch to root
if [ "$DISPLAY" ]; then
   if [ `whoami` != "root" ]; then
      mod=`readlink -f $1`
      mod2=`readlink -f $2`
      xterm -T "Please enter root's password below" -e su - -c "/opt/porteus-scripts/txz2xzm $mod $mod2"
      exit
   fi
else
   if [ `whoami` != "root" ]; then
      echo "Please enter root's password below"
      mod=`readlink -f $1`
      mod2=`readlink -f $2`
      su - -c "/opt/porteus-scripts/txz2xzm $mod $mod2"
      exit
   fi
fi

if [ "$1" = "" -o "$2" = "" ]; then
   echo
   echo "Convert Slackware's TXZ package or a directory with multiple"
   echo "TXZ packages into .xzm compressed module"
   echo "usage: $0 source_filename.txz output_file.xzm"
   echo "or: $0 source_directory output_file.xzm"
   exit 1
fi

# set up variables, and to allow converting multiple txz's in a directory

pathone=`readlink -f $1`
pathtwo=`readlink -f $2`
filename=${pathone##*/}
pkgname=${filename%%.t?z}
pkgbase=${pkgname%%-[0-9]*}
modulename=${pathtwo##*/}
modname=${modulename%%.xzm}
modbase=${modname%%-[0-9]*}

if [ -d $pathone ]; then
  multivert=true
else
  multivert=false
fi


TMPDIR=/tmp/stxz2xzm$$
mkdir -p $TMPDIR

# Create a file that holds the path/txz's to be converted
# This will only hold one value if a single txz was input as $1
if [ $multivert = "true" ]; then
  for x in `ls $pathone | egrep "*.txz|*.tgz"`; do
    echo $pathone/$x >> $TMPDIR/txzlist;
  done
fi

if [ $multivert = "false" ]; then
  echo `readlink -f $1` >> $TMPDIR/txzlist
fi
# dump all instructions from the install script to a single file,
# to be parsed by the next section.
for x in `cat $TMPDIR/txzlist`; do
  txzname=`echo $x | rev | cut -d '/' -f1 | rev`
  mkdir $TMPDIR/explode
  cp $x $TMPDIR/explode
  ( cd $TMPDIR/explode && explodepkg $txzname 1>/dev/null )
  if [ -e $TMPDIR/explode/install/doinst.sh ]; then
    cat $TMPDIR/explode/install/doinst.sh >> $TMPDIR/doinstdump
  fi
  rm -rf $TMPDIR/explode;
done

if [ -e $TMPDIR/doinstdump ]; then

  #clear whitespace from start and finish of all lines
  sed -i 's/^[ \t]*//;s/[ \t]*$//' $TMPDIR/doinstdump
 
  # porteus doesn't use install-info, so we can ignore doinst.sh stuff that messes with it:
  sed -i '/if \[ -x \/usr\/bin\/install-info/,/^fi$/d' $TMPDIR/doinstdump
 
  # all config files will be generated without the .new suffix because
  # we are using installpkg -root, but we want to keep track of these and other
  # files, so we can warn the user if they are going to modify one of their
  # existing config files.
  if egrep -q 'config()' $TMPDIR/doinstdump; then
    cat $TMPDIR/doinstdump | sed '/config()/,/}/d' | grep ^config | cut -d ' ' -f2| sed 's/.new//g' >> /$TMPDIR/doinst-configs
  fi
 
  # now do the same for preserve_perms
  if egrep -q 'preserve_perms()' $TMPDIR/doinstdump; then
    cat $TMPDIR/doinstdump | sed '/preserve_perms()/,/}/d' | grep ^preserve_perms | cut -d ' ' -f2| sed 's/.new//g' >> /$TMPDIR/doinst-configs
  fi
 
  # this function will find files that are being moved around to preserve existing configs
  # and permissions on config files, and add them to the doinst-configs list for checking
 
  removenew() {
   term="$1"
    if grep -q $term $TMPDIR/doinstdump; then
      for x in `grep $term $TMPDIR/doinstdump |tr ' ' '\n' | grep $term | grep / | sed s/$term//g |sort -u`; do
        echo $x >> $TMPDIR/doinst-configs
        echo $x | sed 's/\//\\\//g' > $TMPDIR/sedalt
   y=`cat $TMPDIR/sedalt`
        # remove all the typical commands associated with moving around files to
        # preserve perms.  This would be easier with sed -i /$y/d $TMPDIR/doinstdump
        # but that would delete any other actions on this file
        sed -i "/if \[ -[f,r,e,z,s,L] $y/d" $TMPDIR/doinstdump
        sed -i "/if \[ \! -[f,r,e,z,s,L] $y/d" $TMPDIR/doinstdump
        sed -i "/mv $y/d" $TMPDIR/doinstdump
        sed -i "/cat $y/d" $TMPDIR/doinstdump
        sed -i "/rm $y/d" $TMPDIR/doinstdump
        sed -i "/rm -f $y/d" $TMPDIR/doinstdump
        sed -i "/cp -a $y/d" $TMPDIR/doinstdump;
      done
    fi
}
 
  removenew .new.incoming
  removenew .new
  removenew .old
  removenew .bak
 
  # parse out instructions that are already handled well by
  # installpkg -root, porteus startup scripts, and above commands
  #
  # remove/create links already handled by installpkg
  # fi is just a leftover -- we will add it back in later if needed
  # update-desktop-database and update-icon-cache are handled by startup scripts
  # update-mime-database also handled by startup script;
  # leaving /etc/rc.d/rc.messagebus reload in for now, to reload dbus daemon if modules
  # that require it are activated after startup.

  cat $TMPDIR/doinstdump | grep -v '\(.rm -rf.\)'| grep -v '\(.rm -f.\)' | grep -v '\(.ln -sf.\)'| grep -v 'rm -f [a-z]' | grep -v update-desktop-database |grep -v update-icon-cache |grep -v icons/hicolor | grep -v fc-cache | grep -v "^#" | sed '/config()/,/}/d' | grep -v ^config | sed '/preserve_perms()/,/}/d' | grep -v ^preserve_perms | grep -v update-mime-database | grep -v ldconfig | sed 's/chroot . //g' | grep -v "^$" >> $TMPDIR/doinst-ns
fi

# this section will take care of extra fi's, else's and elif's, while retaining those that are needed
# eventually I may add a section here to properly format the resulting script
# with linebreaks and extra spaces, etc


if [ -s $TMPDIR/doinst-ns ]; then
  linecount=1
  ifcount=0

  while [ $linecount -le `wc -l $TMPDIR/doinst-ns | cut -d ' ' -f1` ]; do
    if cat $TMPDIR/doinst-ns | head -n$linecount | tail -n1 | grep -q '^if '; then
      ifcount=$(($ifcount + 1))
      sed -i "$linecount s/^if /"$ifcount"if /g" $TMPDIR/doinst-ns
    fi
 
    if cat $TMPDIR/doinst-ns | head -n$linecount | tail -n1 | grep -q '^fi$'; then
      ifcount=$(($ifcount - 1))
    fi
 
    if [ $ifcount -lt 0 ]; then
      ifcount=0
    fi
 
    linecount=$(($linecount +1));
  done

  for x in 9 8 7 6 5 4 3 2 1 ;
    do
    sed -i "/^"$x"if /,/^fi$/s/^fi$/"$x"fi/g" $TMPDIR/doinst-ns
    sed -i "/^"$x"if /,/^"$x"fi$/s/^else/"$x"else/g" $TMPDIR/doinst-ns
    sed -i "/^"$x"if /,/^"$x"fi$/s/^elif/"$x"elif/g" $TMPDIR/doinst-ns
  done

  #clear fi's that don't belong
  sed -i s/^fi$//g $TMPDIR/doinst-ns
  #clear else's that don't belong
  sed -i /^else/d $TMPDIR/doinst-ns
  #same for elif's
  sed -i /^elif/d $TMPDIR/doinst-ns
  #change if's back to normal
  sed -i "s/^[0-9]if /if /g" $TMPDIR/doinst-ns
  #change fi's back to normal
  sed -i "s/^[0-9]fi$/fi/g" $TMPDIR/doinst-ns
  #change else back to normal
  sed -i "s/^[0-9]else/else/g" $TMPDIR/doinst-ns
  #change elif back to normal
  sed -i "s/^[0-9]elif/elif/g" $TMPDIR/doinst-ns

  #clear out blank lines
  sed -i /^$/d $TMPDIR/doinst-ns

fi



# install all packages to a new directory

mkdir $TMPDIR/module

for x in `cat $TMPDIR/txzlist`; do
  installpkg -root $TMPDIR/module $x
  if [ $? != 0 ]; then echo "error installing package"; exit; fi
  sleep 2;
done

# optimization procedures, this doesn't hurt
find $TMPDIR/module/usr{/local,/}{man,info} -type l -name "*.gz" 2>/dev/null | xargs -r gunzip -f
find $TMPDIR/module/usr{/local,/}{man,info} -type f -name "*.gz" 2>/dev/null | xargs -r gunzip
rm -f $TMPDIR/module/{usr,usr/local,var}/man/cat*/*

# make sure we don't get these files in our module
moddir=`uname -a |cut -d ' ' -f3`
for x in /etc/init.d \
  /etc/init.d \
  /etc/rc.d/rc.S \
  /etc/rc.d/rc.M \
  /etc/rc.d/rc.K \
  /etc/rc.d/rc.local \
  /lib/modules/$moddir/modules.dep \
  /lib/modules/$moddir/modules.alias \
  /etc/ld.so.conf \
  /etc/ld.so.cache \
  /etc/passwd \
  /etc/group \
  /etc/shadow \
  /etc/shells
  do
  if [ -e $TMPDIR/module/$x ]; then
    echo $x >> $TMPDIR/blacklist
  fi;
done

if [ -s $TMPDIR/blacklist ]; then
  clear
  echo "WARNING: The module you are creating will contains the following"
  echo "files, which should never be included in a porteus module (other"
  echo "than the base modules:"
  echo ""
  for x in `cat $TMPDIR/blacklist`; do
    echo "   $x";
  done
  echo ""
  echo "these files will be removed from your module, and you should"
  echo "check to make sure this file is handled properly by any"
  echo "init scripts."
  echo ""
  echo "Press enter to continue creating this module"
  echo "or press ctrl+c to quit"
  read
fi

for x in `cat $TMPDIR/blacklist`; do
  rm -rf $TMPDIR/module/$x;
done

# Warn user that they will be overwriting config files, if that is the case
if [ -s $TMPDIR/doinst-configs ]; then
  for x in `cat $TMPDIR/doinst-configs | sort -u`; do
    if [ -e /$x ]; then
      if [ "$(cat /$x | md5sum)" != "$(cat $TMPDIR/module/$x | md5sum)" ]; then
        echo /$x >> $TMPDIR/doinst-will-overwrite
      fi
      if [ "$(ls -la /$x | cut -d ' ' -f1,3,4 | md5sum)" != "$(ls -la $TMPDIR/module/$x | cut -d ' ' -f1,3,4 |  md5sum)" ]; then
        echo /$x >> $TMPDIR/doinst-will-modperms
      fi   
    fi;
  done
fi

if [ -s $TMPDIR/doinst-will-overwrite ]; then
  clear
  echo "WARNING: The module you are creating will overwrite the"
  echo "following file(s) with a modified version when activated:"
  echo ""
  for x in `cat $TMPDIR/doinst-will-overwrite`; do
    echo "   $x";
  done
  echo ""
  echo "It is suggested that you compare the two versions"
  echo "of this file in order to understand how this might"
  echo "affect your system."
  echo ""
  echo "Press enter to continue creating this module"
  echo "or press ctrl+c to quit"
  read
fi

if [ -s $TMPDIR/doinst-will-modperms ]; then
  clear
  echo "WARNING: The module you are creating will change the permissions  or ownership"
  echo "of the following file(s) by overwriting them with a modified version:"
  echo ""
  for x in `cat $TMPDIR/doinst-will-modperms`; do
    echo "   $x";
  done
  echo ""
  echo "It is suggested that you compare the two versions"
  echo "of this file in order to understand how this might"
  echo "affect your system."
  echo ""
  echo "Press enter to continue creating this module"
  echo "or press ctrl+c to quit"
  read
fi


# Warn user that atypical commands have been found in the installscript(s)
# And that an init script will be created and shown for modifications

if [ -s $TMPDIR/doinst-ns ]; then

  clear
  echo "NOTICE: The slackware install scripts for the packages"
  echo "you are converting contain the following non-standard"
  echo "instructions:"
  echo ""
  cat $TMPDIR/doinst-ns |sed 's/^/   /g'
  echo ""
  echo "Would you like this to automatically generate"
  echo "an init script to perform these actions each time"
  echo "Porteus is booted up with this module and every time"
  echo "this module is activated?"
  echo ""
  echo "press y or n"
  read makeinit
 
  while [[ $makeinit != ["y","n"] ]]; do
    echo ""
    echo Incorrect selection.  Press y or n, then enter to continue
    read makeinit;
  done
 
  if [ $makeinit = "y" ]; then
    if [ $multivert = "true" ]; then
      initname=$modbase
    else
      initname=$pkgbase
    fi
 
     
    clear
    echo ""
    echo "The init script will now be generated and displayed"
    echo "inside mcedit.  Make any modifications you see fit"
    echo "then press F2 to save and F10 to quit."
    echo ""
    echo "Your module will then be created."
    echo ""
    echo "Press Enter to continue."
    read garbage
 
    mkdir -p $TMPDIR/module/etc/rc.d/init.d $TMPDIR/module/etc/rc.d/rcS.d
   
    initsh=$TMPDIR/module/etc/rc.d/init.d/$initname.sh

    # we'll set this up to handle 'start' command. pushd to /, so all of the
    # commands that use relative paths will be done to the root filesystm
    echo '#!/bin/bash' >> $initsh
    echo '# This script was autogenerated by stxz2xzm' >> $initsh
    echo '' >> $initsh
    echo 'if [ "$1" = "start" ]; then' >> $initsh
    echo '' >> $initsh
    echo 'pushd /' >> $initsh
    echo '' >> $initsh
    cat $TMPDIR/doinst-ns >> $initsh
    echo '' >> $initsh
    echo 'popd' >> $initsh
    echo '' >> $initsh
    echo 'fi' >> $initsh
    echo '' >> $initsh
    chmod +x $initsh

## NOTE:  This script (stxz2xzm) is not set up, as of yet, to handle shutdown commands!

    mcedit $initsh
    ( cd $TMPDIR/module/etc/rc.d/rcS.d ; ln -sf ../init.d/$initname.sh S-$initname )

  fi
fi

mkdir -p $TMPDIR/module/var/porteus

## Create a category file
grep CATEGORY: /opt/ppm/PACKAGES.TXT|awk '{print$NF}' | sort -u > $TMPDIR/.catchoice

clear
echo ""
echo "Please type a category for this module"
echo "or press Enter to proceed with the category"
echo "set to misc (miscellaneous)."
echo ""
echo "Categories:"
echo ""
cat $TMPDIR/.catchoice
echo ""
echo "you should type the category name exactly as it appears above"
echo ""
read category

if [ $category = "" ]; then
  category=misc
fi

if [ $multivert = "true" ]; then
  ppkg=$modname
else
  ppkg=$pkgname
fi

clear
echo ""
echo "Finding dependencies..."

echo "PACKAGE NAME: $ppkg" >> $TMPDIR/module/var/porteus/$ppkg.info
for a in `find $TMPDIR/module -executable -type f | fgrep -v ".so"` `find -type f -name "*.so"`; do
# Run ldd against all executable files in the package
  ldd $a|awk '{print$1}'|sed -e "/\bnot\b/d" -e '/dynamic/d' -e '/ld-linux/d' -e '/found/d' -e '/libc.so/d' -e '/linux-gate/d' -e '/linux-vdso/d' >> $TMPDIR/req
done
ireq=`cat $TMPDIR/req|tr "\n" ","|sort -u`
echo "REQUIRED: $ireq" >> $TMPDIR/module/var/porteus/$ppkg.info

find $TMPDIR/module -type f -name "lib*.so*" | awk -F/ '{print$NF}' >> $TMPDIR/off
find $TMPDIR/module -type l -name "lib*.so*" | awk -F/ '{print$NF}' >> $TMPDIR/off
ioffer=`cat $TMPDIR/off|tr "\n" ","|sort -u`
echo "OFFERED: $ioffer" >> $TMPDIR/module/var/porteus/$ppkg.info


dir2xzm $TMPDIR/module "$2"
if [ $? != 0 ]; then echo "error building compressed image"; exit; fi

rm -Rf $TMPDIR

To use it, simply invoke it like you would the txz2xzm script:

Code: Select all

stxz2xzm filename.txz modulename.xzm
you can also use it to convert multiple txz's inside a directory into one module:

Code: Select all

stxz2xzm /tmp/modulefiles/ /root/desktop/mynewmodule.xzm
the "S" in stxz2xzm stands for 'smart', or 'super'. If, however, you find it doesn't work for you, it can also stand for 'stupid' or 'shitty'.

Features:

convert single or multiple txz's into one module
scans all instructions from 'doinst.sh', eliminates all of the clutter (hopefully) and leaves all of the actions you need to care about when activating your module (again, hopefully).
warns you if configs are going to be created that are different from your existing configs
warns you if there are any files in your module that shouldn't be there (e.g. ld.so.conf), then removes them
automatically generates an init script in /etc/rc.d/init.d and symlinks it to /etc/rc.d/rcS.d, which runs all of the 'non-standard' commands
lets you edit the init script it generates, in case there were errors
prompts you for a category for the module (for ppm compatibility)
creates a /var/porteus/*.info file (not tested much -- I may need some help from brokenman or fanthom here) for ppm compatibility

If you're interested, please give it a shot, and run it against your favorite (or, more importantly, your least favorite) txz package, and let me know how it goes!

Cheers!
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
Hamza
Warlord
Warlord
Posts: 1847
Joined: 28 Dec 2010, 07:41
Distribution: Porteus
Location: France

Re: [SCRIPT] stxz2zxm (smart? txz converter)

Post#2 by Hamza » 10 Jan 2012, 20:46

I am lost ! :shock:

Why are you using 1 instead 2 in this line

Code: Select all

( cd $TMPDIR/explode && explodepkg $txzname 1>/dev/null )
If you want only one > ; you should use > only instead of 1>. When you use 2>..this mean >> but if you uses 1>..this mean also >.

I would like to know why you are using this.

I prefer to read the PPM Code :)
NjVFQzY2Rg==

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: [SCRIPT] stxz2zxm (smart? txz converter)

Post#3 by Ahau » 10 Jan 2012, 22:38

Hi Hamza,

This is just my inexperience as a scripter...my intent was to redirect the standard output to /dev/null, so that all of the text does not scroll across the screen while the packages are exploded. I probably started with 2>/dev/null and when that didn't do what I wanted, I changed it to 1 without much thought :)

I've removed the 1 from my post and will do the same for my local copy.
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
brokenman
Site Admin
Site Admin
Posts: 5456
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

Re: [SCRIPT] stxz2zxm (smart? txz converter)

Post#4 by brokenman » 11 Jan 2012, 00:19

When you use 2>..this mean >>
Ahau was correct with his technique. Traditionally 1> is used to redirect stdout and 2> is used to redirect stderr

These days most people omit the 1> ... but leaving it is absolutely correct. In fact it is more correct. You could consider leaving out the 1 a kind of coding colloquialism.

That aside ... without having experimented with the script and after reading the description my initial thought was .... aha! That's why i am having trouble with some apps that require libmoz but won't find it because it exists in the subdirectory of mozilla-thunderbird. I will certainly be one that takes your script for a spin. Right now!

Posted after 46 minutes 36 seconds:
Does exactly what he said it does. A very nice improvement over the existing conversion script if you would like a perfectly porteus compliant module and don't mind answering a few questions during conversion. Nice job Ahau i will be using this alot.

My only issue is the /var/porteus/*.info file which needs two small tweaks. Will send via PM.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: [SCRIPT] stxz2zxm (smart? txz converter)

Post#5 by Ahau » 11 Jan 2012, 06:29

I've applied brokenman's tweaks and updated the script in the original post.

Thank you very much, brokenman!
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
Hamza
Warlord
Warlord
Posts: 1847
Joined: 28 Dec 2010, 07:41
Distribution: Porteus
Location: France

Re: [SCRIPT] stxz2zxm (smart? txz converter)

Post#6 by Hamza » 11 Jan 2012, 15:47

Apologize me Ahau.

You are beginning to have some strong skills in scripting..and not me :oops:
NjVFQzY2Rg==

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: [SCRIPT] stxz2zxm (smart? txz converter)

Post#7 by Ahau » 11 Jan 2012, 16:17

No worries, Hamza -- there's always more than one way to grep a cat :D
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: [SCRIPT] stxz2zxm (smart? txz converter)

Post#8 by Ahau » 29 Mar 2012, 14:51

Updated the script to work with Porteus 1.2 (it needed some minor retouching, since liblinuxlive was dropped from V1.2). Enjoy!
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
bigbass
Contributor
Contributor
Posts: 151
Joined: 13 Jan 2012, 14:35
Distribution: slackware 14

Re: [SCRIPT] stxz2zxm (smart? txz converter)

Post#9 by bigbass » 10 Sep 2012, 00:01

Hey Ahau!

There are just too many possible commands to filter but you did a great improvement though
and I like that you are trying to improve this it is a good positive step
how about a look ahead without using explodepkg to save some time and space ?

I had to think about this a bit to get the doinst,sh only
some packages may be huge so this would be faster

if you can use it somehow here just a snippet
I have more comments about this I posted in box
a copy is saved here /opt/doinst_check/

I kept this simple so it would be easier to adapt
but a small pop up showing the cat of the file would be safe and easy to add


great job !

some very good news after compiling the backend module making tools on slackware 14
and made modules on slackware 14 can be opened on Porteus 1.2 :)

Joe

before I understood what exactly was a xzm module I was doing it with sfs with lzm2 so just replace .sfs with .xzm
its the same thing the only difference is I used modules only for the base install then packages on top of that :)

start here --->Posted: Mon 23 Aug 2010, 16:16
http://208.109.22.214/puppy/viewtopic.p ... 43bbf04073


Code: Select all

#!/bin/bash

# Joe arose ... bigbass
# This is a drag N drop script that could be used to see if a package 
# contains a script that would be run at install time  
# get the doinst.sh and read it before installing the package 

mkdir -p /opt/doinst_check/


#==============
#  tgz
#==============

if [[ `echo "$1" | grep "tgz$"` ]]; then

	tar xfO "$1" install/doinst.sh  > /opt/doinst_check/`basename "$1" .tgz`"-doinst.txt"
	    
	    
		if [ ! -s /opt/doinst_check/`basename "$1" .tgz`"-doinst.txt" ]; then
			rm -f /opt/doinst_check/`basename "$1" .tgz`"-doinst.txt"		
		fi
		
#==============
#  txz
#==============

elif [[ `echo "$1" | grep "txz$"` ]]; then	
		
	xz -c -d "$1" | tar xO install/doinst.sh  > /opt/doinst_check/`basename "$1" .txz`"-doinst.txt"

		if [ ! -s /opt/doinst_check/`basename "$1" .txz`"-doinst.txt" ]; then
			rm -f /opt/doinst_check/`basename "$1" .txz`"-doinst.txt"		
		fi
	
#==============
#  tlz
#==============

elif [[ `echo $1 | grep "tlz$"` ]]; then

	lzma -c -d "$1" | tar xO install/doinst.sh  > /opt/doinst_check/`basename "$1" .tlz`"-doinst.txt"
	
		if [ ! -s /opt/doinst_check/`basename "$1" .tlz`"-doinst.txt" ]; then
			rm -f /opt/doinst_check/`basename "$1" .tlz`"-doinst.txt"		
		fi
fi





 

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: [SCRIPT] stxz2zxm (smart? txz converter)

Post#10 by Ahau » 10 Sep 2012, 15:50

Ahh, very nice! I think I understand what you've done here, and it would make a substantial speed improvement :)

I agree, there are far too many commands, and the more I parse out, the more I run the risk of pulling out something that should remain flagged. I don't think this script should ever go into the default ISO for general use, but I still use it for development purposes. I'll add this to my ToDo list for updating.
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
pumbaa2
White ninja
White ninja
Posts: 29
Joined: 08 May 2012, 10:39
Distribution: Slackware 13.37
Location: Melbourne, FL
Contact:

Re: [SCRIPT] stxz2zxm (smart? txz converter)

Post#11 by pumbaa2 » 04 Jun 2014, 08:46

I was going to make a comment about this in a new thread until I found this one.

In practice the doinst shell scripts should be packed with the module. Create a folder /var/log/packages just like slackware that shows which packages are already installed. I know this sounds strange but hear me out:

Scenario 1:
The module is mounted and added to the Aufs file structure (Activate modulename.xzm). Porteus is running in Always-Fresh Mode. Under this instance /var/log/packages will not have the module name in that directory. So when the module comes up for the first time, it should execute the doinst.sh script and then delete the file from the fake Root AUFS Filesystem. This will ensure that all changes take effect on the Aufs Filesystem.

Scenario 2:
The module is mounted and added to the Aufs file structure (Activate modulename.xzm) but the module was activated before. Porteus is running with changes being saved (somewhere). In this instance because the doinst.sh script was ran before and a file has already been created in /var/log/packages so when the module goes live this time the doinst.sh script will NOT be run since those changes have already taken place but the script will still be purged from the aufs filesystem (rm -rf /install).

Example (Quoted from Above):

The package gnome-keyring depends upon glib2, and needs to have the utility /usr/bin/glib-compile-schemas run before it will function properly. When txz2xzm is run on this package, the install script calls for glib-compile-schemas, but since the installation has chrooted to /tmp/fakeroot, it cannot be found, and the resulting schemas are never compiled.

This would never be a problem as the doinst.sh script is not ran during package conversion. The need for a fake root is un-needed and explodepkg can extract the contents to the fake directory, repack it as xzm. The installer script will be ran on module activation ensuring that the commands are available.

Example 2 (Quoted from Above):

The package 'seamonkey-solibs' contains a number of libraries, which must be seen by your system in order to be used. Rather than being placed in /usr/lib/, these libraries are placed in /usr/lib/seamonkey (assuming a 32-bit arch), and a line of text is echoed to /etc/ld.so.conf ("/usr/lib/seamonkey"). The ldconfig utility is run, and your system can now see all of the libraries in the seamonkey directory. All of these actions are carried out by the install script inside the .txz package, at /install/doinst.sh.

When we run txz2xzm on this package, however, the txz is installed to a fakeroot directory inside /tmp/, and one of the following things will happen:
1) if there is no /tmp/fakeroot/etc directory, the 'echo' command will fail. when the user activates his new module, none of his applications will be able to see the newly installed libraries.
2) if there happens to be a /tmp/fakeroot/etc directory, the 'echo' command will create a new ld.so.conf file containing one line (/usr/lib/seamonkey). This file will be placed into the module and, when activated, this will overwrite your existing system-wide /etc/ld.so.conf, and chaos will ensue.


This also won't be an issue. Since the script is ran at module activation, ld.so.conf will not be activated as a truncated file only containing /usr/lib/seamonkey. Since the script is only ran once for installations that record changes, you also won't end up with a ld.so.conf that has 100 /usr/lib/seamonkey entries in it.

Package Directory Layout:

The file is /var/log/packages can simply be "touch /var/log/packages/packagename" or can contain a MD5SUM of the module file itself, that way if the module gets modified it will run the doinst.sh script again on activate and then update the MD5SUM inside the packagename file in /var/log/packages.

Conclusions:

This will make your txz2xzm script go from 95% working to 99% working. The clause on the 1% is if the module is activated before its dependencies become activated. The may result in the doinst script running commands that aren't available yet, or commands that require libraries that aren't available yet. Since bootup uses "mass-activate" it maybe useful to activate all modules first and then run the install scripts for each module that needs running after all modules are activated. I realize that the filename for the installer script is the same path and filename for every script, but each installer script instance can be located in its respective mount point /mnt/live/memory/modulename/install/doinst.sh. At the tail end of the module activation script simply "rm -rf /install" to remove such links from the Root AUFS Filesystem.

Also an extra directory can be added to the Porteus boot structure. Right now you activate everything in Base then in Modules. Maybe it should be Libs then Base then Modules. Libs being activated first will ensure that all library dependencies are present before activating everything else.

I understand that this may slow down the boot process and that some doinst.sh scripts will call upon commands like ldconfig which may get run more than once if more than one module script calls upon it. But as far as package compatibility goes, this is your best bet. In scenarios where changes are being logged, this will only be painful the first time the modules come up. After that, module activation will happen as it does now except the rm -rf /install on the AUFS Root.

I hope this helps...

-RayRay-

Post Reply