USM Unified Slackware Package Manager

[brokenman]

Yes I believe it is related to wget download mode or the locale environment when trying to read a text file encoded in UTF-8.

For slackware please try opening /etc/usm/mirror-slackware.txt and changing the mirror to some http address. Then repeat the same problem you have above.

What confuses me is that nobody else has this problem, but because i know you know what you are doing i can't let it slide. Have you changed locale settings?
Please supply the output of: echo $LANG

Also once you have successfully updates and the PACKAGES.TXT.gz file has been unpacked please supply the output of: od -x < /var/usm/slackware/PACKAGES.TXT|head -n1
I want to look directly at the dump to see why this file reporting to be binary.

@ Slaxmax there is indeed a problem with alien repo links for 32bit. They call i486 --> x86 in the website heirarchy so i have added an exception and will push the update today or tomorrow.

[brokenman]

/var/usm/slackware/PACKAGES.TXT: Quake I or II world or extension, 1415074862 entries
Yes, this was the line i encountered when my text file went all binary on me.

root@porteus:/home/guest# file /var/usm/sbo/SLACKBUILDS.TXT.gz
Don't forget after the file downloads it is gunzipped so it loses the .gz extension.

So now SLACKBUILDS.TXT is downloading OK but slackware still looks binary? Odd since nothing there changed server side.
I think I will recreate the database files, check they are UTF-8 and reupload to see if this fixes your problem. Thanks for finding v3.0.0 too!

Updated usm on sourceforge (only a small fix for the 32bit alien links but important). Busy preparing rc2 so will do all this tomorrow.

+-------------------------
Wed Jan 26 01:09:05 UTC 2014
usm-3.0.5-noarch-1.txz: updated
Fixed bad links for 32bit alien repo (thanks slaxmax)

[brokenman]

No, it gets decompressed. It changes form. Nothing is deleted.

Code: Select all

echo test > /tmp/test.txt
gzip /tmp/test.txt
gunzip /tmp/test.txt.gz

[Slaxmax]

New problem
The links are ok but...

Code: Select all

Avaliable options:

  1+) vlc-2.1.2-i486-3alien.txz
  2 ) samba-4.1.4-i486-1_slack14.1.txz
  3+) aalib-1.4rc5-i486-5.txz
  4+) libcaca-0.99.beta18-i486-2.txz
  5+) gnutls-3.1.16-i486-1.txz
  6+) p11-kit-0.16.4-i486-1.txz
  7+) libaio-0.3.109-i486-1.txz
  8+) aaa_elflibs-14.1-i486-3.txz
  9 ) python-2.7.5-i486-1.txz
 10 ) gc-7.4.0-i486-1sl.txz
 11+) guile-2.0.9-i486-1.txz
 12+) libunistring-0.9.3-i486-1.txz
 13+) libtasn1-3.3-i486-1.txz
 14+) openssl-solibs-1.0.1f-i486-1_slack14.1.txz
 15+) tcl-8.6.1-i486-1.txz
 16+) tk-8.6.1-i486-1.txz
Invalid option: 17
Type number and press enter to deselect an item (again to reselect, ENTER when ready):

http://taper.alienbase.nl/mirrors/people/alien/sbrepos/14.1/x86/vlc/vlc-2.1.2-i486-3alien.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/l/aalib-1.4rc5-i486-5.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/l/libcaca-0.99.beta18-i486-2.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/n/gnutls-3.1.16-i486-1.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/n/p11-kit-0.16.4-i486-1.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/l/libaio-0.3.109-i486-1.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/a/aaa_elflibs-14.1-i486-3.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/d/guile-2.0.9-i486-1.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/l/libunistring-0.9.3-i486-1.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/l/libtasn1-3.3-i486-1.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/patches//openssl-solibs-1.0.1f-i486-1_slack14.1.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/tcl/tcl-8.6.1-i486-1.txz
http://ftp.slackware-brasil.com.br//slackware-14.1/slackware/tcl/tk-8.6.1-i486-1.txz
 Exiting for testing 
 FUNCTION:package_get 
 LINE:840 

[Slaxmax]

@ Slaxmax there is indeed a problem with alien repo links for 32bit. They call i486 --> x86 in the website heirarchy so i have added an exception and will push the update today or tomorrow.

I edited the usm script for this problem and work.
But the new usm downloaded doesn't work. See previous post.

[brokenman]

Thanks. Updated 3.0.5 on sourceforge. (you can just remove the 'devexit' command on line 840 and the printf line above it if you want).

I've pushed a new database update. phhpro this is difficult to troubleshoot since i can't reproduce it. The current wget method is correct, wget defaults to binary mode and since the file it is downloading is a gzip file all is good. Last night i thought of something. Not long ago i added a small piece of code to remove some stuff from PACKAGES.TXT that I don't want to show up in the packages available. This is true for alien, salix and slackware. It uses awk to remove a block of text containing a string. I remember now having a headache with this command, and it may be where i remember the file corruption happening for me.

If you get a chance please remove the block of code in /usr/sbin/usm starting at line: 737 and finishing at 744, and test again.

Code: Select all

if [ -z $LEAVEPACKAGES ]; then
  # Massage PACKAGES.TXT file for distros that require something excluded
  if [ `egrep -o "alien|salix|slackware" <<<$1` ]; then
    # Remove excludes from PACKAGES.TXT
    awk -vRS= '$0!~/'$exc'/{print $0RT}' $SDIR/PACKAGES.TXT > $SDIR/PACKAGES2.TXT
    mv $SDIR/PACKAGES2.TXT $SDIR/PACKAGES.TXT
  fi
fi

Hopefully this is the culprit (v3.0.0 did not contain this edit). If not then I am out of ideas.

NOTE: You may want to just remove the 'mv' line in the above code and leave both files in place so you can check the format of both. SDIR is /var/usm/slackware (or whatever distro is being updated in the function).

[Slaxmax]

Thanks works fine now.
An improvement continue downloading others packages even if one fails.

Thanks brokenman

[brokenman]

An improvement continue downloading others packages even if one fails.
Done. Will add in next update.

[brokenman]

Let's take it one step at a time. Put this text into a file and execute it:

Code: Select all

#!/bin/bash
REPO=http://slackbuilds.org/slackbuilds
wget $REPO/14.1/SLACKBUILDS.TXT.gz -P /tmp
file -bi /tmp/SLACKBUILDS.TXT.gz
gunzip /tmp/SLACKBUILDS.TXT.gz
file -bi /tmp/SLACKBUILDS.TXT
grep wmtime-1.0b2.tar.gz /tmp/SLACKBUILDS.TXT

This is essentially what the usm script does with slackbuilds. If this fails then something is wrong on your end.

Perhaps one should file a ticket with the maintainer of that repo, mirror to have a look?
The problem is that nobody else can reproduce the problem, including me. All is hunky dory here.

You seem to pre-create the whole tree? Even with sbo and slackware commented, the folders are there, holding a zero byte OPTIFEST (sbo) and zero byte OPTIFEST, PACKAGES.TXT (slackware) respectively.
No. Only the folder structure should ship with the slackware package. All files in /var/usm are created or downloaded when usm is run. Furhtermore when you run usm -u distro the files in this tree are removed before updating.

I went to comment lines 582 - 587 (sbo)
This will do no good as the rest of the function will try to run (without downloaded files). You need to open the script in geany and look for the function: update_all() which is on line 256 and comment out: update_database sbo
This will get you passed slackbuilds when you run: usm -u all Then you can remove the codeblock as i asked above and see if slackware's PACKAGES.TXT goes binary.

BTW: Are you downloading each new release from sourceforge? We are at 3.0.5 right now.

[brokenman]

Here is my output from the same script. Notice just after the download where: file -bi /tmp/SLACKBUILDS.TXT.gz was issued. The next line after that, the file is gunzipped and it correctly shows as ascii. So your wget is the source of the problem. What version is it? Mine is 1.14 .... in any case it is not functioning as it should.

Code: Select all

--2014-01-28 01:26:24--  http://slackbuilds.org/slackbuilds/14.1/SLACKBUILDS.TXT.gz
Resolving slackbuilds.org (slackbuilds.org)... 208.94.238.115
Connecting to slackbuilds.org (slackbuilds.org)|208.94.238.115|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 363022 (355K) [application/x-gzip]
Saving to: ‘/tmp/SLACKBUILDS.TXT.gz’

100%[==============>] 363,022      316KB/s   in 1.1s   

2014-01-28 01:26:25 (316 KB/s) - ‘/tmp/SLACKBUILDS.TXT.gz’ saved [363022/363022]

application/x-gzip; charset=binary
text/plain; charset=us-ascii
SLACKBUILD DOWNLOAD: http://web.cs.mun.ca/~gstarkes/wmaker/dockapps/files/wmtime-1.0b2.tar.gz

Here are two other scripts to try. Let me know which one works:

Code: Select all

#!/bin/bash
REPO=http://slackbuilds.org/slackbuilds
[ -f /tmp/SLACKBUILDS.TXT* ] && rm /tmp/SLACKBUILDS.TXT*
wget --passive-ftp -O /tmp/SLACKBUILDS.TXT.gz $REPO/14.1/SLACKBUILDS.TXT.gz
echo "The SLACKBUILDS.TXT.gz file is of type:"
file -bi /tmp/SLACKBUILDS.TXT.gz
echo
echo gunzipping now
gunzip /tmp/SLACKBUILDS.TXT.gz
echo
echo Now thw file is of type:
file -bi /tmp/SLACKBUILDS.TXT
echo
grep wmtime-1.0b2.tar.gz /tmp/SLACKBUILDS.TXT

This one will download using curl instead.

Code: Select all

#!/bin/bash
REPO=http://slackbuilds.org/slackbuilds
[ -f /tmp/SLACKBUILDS.TXT* ] && rm /tmp/SLACKBUILDS.TXT*
curl -o /tmp/SLACKBUILDS.TXT.gz $REPO/14.1/SLACKBUILDS.TXT.gz
echo "The SLACKBUILDS.TXT.gz file is of type:"
file -bi /tmp/SLACKBUILDS.TXT.gz
echo
echo gunzipping now
gunzip /tmp/SLACKBUILDS.TXT.gz
echo
echo Now thw file is of type:
file -bi /tmp/SLACKBUILDS.TXT
echo
grep wmtime-1.0b2.tar.gz /tmp/SLACKBUILDS.TXT

[brokenman]

Interesting. The download of a 200K file (0.5 seconds) requires the 'continue' flag. Well, i guess I will add it to the script then. Thanks for persisting. All is well with the slackware archive too?

[brokenman]

Please try the commands in my previous post. wget -O and curl -o also state your version of wget. I can check the version tonight if you don't answer in time.

[brokenman]

The problem is most certainly not server side. If it was, it would be reproducible by many. It is more likely due to a different version of sed or awk between v2.1 which you are using, and v3.0 on which i was coding. Let me downgrade and do some more testing.

[brokenman]

Finally nailed it. I was able to reproduce the problem which was caused by a mixture of two things.

In ponce's repository, the 14.1 tree contains the MANIFEST in the CHECKSUM.md5 file, while the 14.0 tree doesn't. This was causing the 'problem during download' message on 14.0 as the md5sum mismatched (didn't exist). This doesn't concern me because (as previously stated) USM will only be run on slackware 14.1+

The weirdness of binary files (which certainly is a concern) was caused by something in gawk-3.1.8 (I am using 4.1.0). I don't know exactly what and I can't reproduce it on another file, but with the PACKAGES.TXT file gawk-3.1.8 outputs a binary after using the RS= flag. Commenting out the codeblock with awk (I requested in a previous post) proved this. In any case, until I find out the reason I have bypassed this odd(?) behaviour by first putting the output of awk into a variable and then passing it to a file using printf. This seems to work and allows the script to function normally.

I will push another update tomorrow (well later today really). 1:30am and my brain is bruised from the repeated forehead slaps. I'm sure this has frustrated you too phhpro so thanks for persisting. Without it, I would never have discovered the bug until after beta.

[brokenman]

Updated to 3.0.6 on sourceforge

+-------------------------
Wed Jan 29 02:13:05 UTC 2014
usm-3.0.6-noarch-1.txz: updated
Download of packages continues if one fails (thanks slaxmax)
Fixed fatal bug with gawk 3.1.8 outputting binary file (thanks phhpro)