USM bug reports

[donald]

@ port

There is (was?) a mismatch between the file on the server and your
/var/usm/slackonly/PACKAGES.TXT

as you can see, the file on the server is ansible-2.1.2.0-x86_64-2_slonly.txz
whereas the PACKAGES:TXT shows ansible-2.1.2.0-x86_64-1_slonly.txz

2 options:
wait until it is fixed ( usm -u slackonly)
or open /var/usm/slackonly/PACKAGES.TXT
and change the filename.

[Ed_P]

There is (was?) a mismatch between the file on the server and your
/var/usm/slackonly/PACKAGES.TXT

as you can see, the file on the server is ansible-2.1.2.0-x86_64-2_slonly.txz
whereas the PACKAGES:TXT shows ansible-2.1.2.0-x86_64-1_slonly.txz

Good eyes donald.

or open /var/usm/slackonly/PACKAGES.TXT
and change the filename.

And an interesting fix.

[Bogomips]

I have a recurrent bug with usm, sometimes it download a package as a 0 bytes archive, even when it is not zero.

For example:


as you can see usm find an ansible package which is 2328 K, but when trying to download it I get:

Code: Select all

# usm -g ansible

 The following items were found.
 Choose an number to confirm. 
 ctrl+c to quit

1) ansible-2.1.2.0-x86_64-1_slonly.txz
#? 1

Processing:   ansible-2.1.2.0-x86_64-1_slonly.txz 
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: attempt to access field -1
grep: /etc/usm/mirrors-.txt: No such file or directory
grep: /var/usm//PACKAGES.TXT: No such file or directory
grep: /var/usm//PACKAGES.TXT: No such file or directory

 The following packages are required. 
ansible-2.1.2.0-x86_64-1_slonly.txz [] [installed]

Total size: 0 KB
 Would you like to install the package/s? (custom paths are supported) [y/n]

total size 0 Kb and in fact if you proceed with downloading you get a 0 kb file and thus a invalid xzm module. Which I also find strange is the attempt to access /etc/usm/mirrors-.txt which is obviously an inexistent file and clearly an error forming the name.

In the code example, failure of awk resulted in irrecoverable error, as a result of which the repository name was not formed, ie.e. left at null. So looking for /etc/usm/mirrors-$repo.txt, where $repo is ''.

Resolved problem for mytself (test for error made, and then attempt to recover heuristicallly from error) here: http://forum.porteus.org/viewtopic.php?f=140&t=6966

[port]

@ port

There is (was?) a mismatch between the file on the server and your
/var/usm/slackonly/PACKAGES.TXT

as you can see, the file on the server is ansible-2.1.2.0-x86_64-2_slonly.txz
whereas the PACKAGES:TXT shows ansible-2.1.2.0-x86_64-1_slonly.txz

2 options:
wait until it is fixed ( usm -u slackonly)
or open /var/usm/slackonly/PACKAGES.TXT
and change the filename.

thank you for the solution but I think option 2 doesn't work, if I rename the package name to ansible-2.1.2.0-x86_64-1_slonly.txz in slackonly/PACKAGES.TXT I got:

Code: Select all

# usm -i ansible
grep: invalid option -- '['
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.

Package:    (2328 K) [not installed]
ansible: ansible (a ssh-based config management framework)
ansible:
ansible: A radically simple, model-driven orchestration solution that
ansible: automates configuration, software deployment, and other IT needs.
ansible:
ansible: Homepage: http://ansible.com
ansible:
ansible:
ansible:
ansible:
ansible:

# usm -g ansible

 The following items were found.
 Choose an number to confirm. 
 ctrl+c to quit

1) ansible-2.1.2.0-x86_64-1_slonly.txz
#? 1

Processing:   ansible-2.1.2.0-x86_64-1_slonly.txz 
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: attempt to access field -1
grep: /etc/usm/mirrors-.txt: No such file or directory
grep: /var/usm//PACKAGES.TXT: No such file or directory
grep: /var/usm//PACKAGES.TXT: No such file or directory

 The following packages are required. 
ansible-2.1.2.0-x86_64-1_slonly.txz [] [not installed]

Total size: 0 KB
 Would you like to install the package/s? (custom paths are supported) [y/n]

as you can see usm still doesn't construct the names rightly: /etc/usm/mirrors-.txt instead of /etc/mirrors-slackonly.txt and /var/usm/PACKAGES.TXT instead of /var/usm/slackonly/PACKAGES.TXT

[port]

In the code example, failure of awk resulted in irrecoverable error, as a result of which the repository name was not formed, ie.e. left at null. So looking for /etc/usm/mirrors-$repo.txt, where $repo is ''.

Resolved problem for mytself (test for error made, and then attempt to recover heuristicallly from error) here: http://forum.porteus.org/viewtopic.php?f=140&t=6966

maybe it's my lack of understanding but I think usm should be more robust in its behaviour, at least it should manage errors in a proper way such as warning about them rather to try to continue executing as if nothing happened

[port]

another weird usm bug:

Code: Select all

# usm -g ansible

 You need the following programs to continue:  
installpkg
removepkg

but they are there:

Code: Select all

# find / -xdev -name '*installpkg*'
/sbin/installpkg
# find / -xdev -name '*removepkg*'
/sbin/removepkg

The error is due to /sbin is not in my PATH (so easy to solve .

The only difference respect to yesterday when I could do usm -g whatever is the installation of ansible using python (installing pip and then piping ansible) but don't think it is related anyhow (I have no usm install log to check anyway)

[donald]

@ port

just tested ..after an usm -u all..PACKAGES.TXT has (now) the correct filename but
/var/usm/slackonly/LIBS.TXT still has ansible-2.1.2.0-x86_64-1_slonly.txz
rename it to ansible-2.1.2.0-x86_64-2_slonly.txz and it does work.

Code: Select all

root@porteus:/home/guest# usm -g ansible

 The following items were found.
 Choose an number to confirm. 
 ctrl+c to quit

1) ansible-2.1.2.0-x86_64-2_slonly.txz
#? 1

Processing:   ansible-2.1.2.0-x86_64-2_slonly.txz 

 The following packages are required. 
ansible-2.1.2.0-x86_64-2_slonly.txz [2328K] [not installed]

Total size: 2 MB
 Would you like to install the package/s? (custom paths are supported) [y/n]

 Press [r] to remove packages, [q] to quit, or enter to start downloading.
Downloading: ansible-2.1.2.0-x86_64-2_slonly.txz  DONE

 Commencing package conversion ... 
Verifying package ansible-2.1.2.0-x86_64-2_slonly.txz.
Installing package ansible-2.1.2.0-x86_64-2_slonly.txz:
PACKAGE DESCRIPTION:
# ansible (a ssh-based config management framework)
#
# A radically simple, model-driven orchestration solution that
# automates configuration, software deployment, and other IT needs.
#
# Homepage: http://ansible.com
#
Executing install script for ansible-2.1.2.0-x86_64-2_slonly.txz.
Package ansible-2.1.2.0-x86_64-2_slonly.txz installed.

Creating /tmp/usm/ansible-2.1.2.0-x86_64-2_slonly.xzm

Quiet mode: off
Updating shared library links:  /sbin/ldconfig

root@porteus:/home/guest# 

....seems that slackonly is a bit slow updating the relevant files....

[Ed_P]

FWIW

After USM Update all. No changes to any updated PACKAGES.TXT or LIBS.TXT.

Code: Select all

guest@porteus:~$ su
Password: 
root@porteus:/home/guest# usm -i ansible

Package:  ansible-2.1.2.0-x86_64-2_slonly.txz  (2328 K) [not installed]
ansible: ansible (a ssh-based config management framework)
ansible:
ansible: A radically simple, model-driven orchestration solution that
ansible: automates configuration, software deployment, and other IT needs.
ansible:
ansible: Homepage: http://ansible.com
ansible:
ansible:
ansible:
ansible:
ansible:

root@porteus:/home/guest# usm -g ansible

 The following items were found.
 Choose an number to confirm. 
 ctrl+c to quit

1) ansible-2.1.2.0-x86_64-1_slonly.txz
#? 1

Processing:   ansible-2.1.2.0-x86_64-1_slonly.txz 
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: attempt to access field -1
grep: /etc/usm/mirrors-.txt: No such file or directory
grep: /var/usm//PACKAGES.TXT: No such file or directory
grep: /var/usm//PACKAGES.TXT: No such file or directory

 The following packages are required. 
ansible-2.1.2.0-x86_64-1_slonly.txz [] [not installed]

Total size: 0 KB
 Would you like to install the package/s? (custom paths are supported) [y/n]

 Press [r] to remove packages, [q] to quit, or enter to start downloading.

root@porteus:/home/guest# 

Updated LIBS.TXT

Code: Select all

root@porteus:/home/guest# cat /var/usm/slackonly/LIBS.TXT | grep ansible
ansible-2.1.2.0-x86_64-1_slonly.txz
root@porteus:/home/guest# cat /var/usm/slackonly/LIBS.TXT | grep ansible
ansible-2.1.2.0-x86_64-2_slonly.txz

root@porteus:/home/guest# usm -g ansible

 The following items were found.
 Choose an number to confirm. 
 ctrl+c to quit

1) ansible-2.1.2.0-x86_64-2_slonly.txz
#? 1

Processing:   ansible-2.1.2.0-x86_64-2_slonly.txz 

 The following packages are required. 
ansible-2.1.2.0-x86_64-2_slonly.txz [2328K] [not installed]

Total size: 2 MB
 Would you like to install the package/s? (custom paths are supported) [y/n]

 Press [r] to remove packages, [q] to quit, or enter to start downloading.

root@porteus:/home/guest# 

And

Code: Select all

root@porteus:/home/guest# find / -xdev -name '*installpkg*'
/usr/man/de/man8/installpkg.8
/usr/man/el/man8/installpkg.8
/usr/man/es/man8/installpkg.8
/usr/man/fa/man8/installpkg.8
/usr/man/fr/man8/installpkg.8
/usr/man/id/man8/installpkg.8
/usr/man/it/man8/installpkg.8
/usr/man/man8/installpkg.8
/usr/man/nb/man8/installpkg.8
/usr/man/nl/man8/installpkg.8
/usr/man/pl/man8/installpkg.8
/usr/man/pt_BR/man8/installpkg.8
/usr/man/pt_PT/man8/installpkg.8
/usr/man/ru/man8/installpkg.8
/usr/man/sv/man8/installpkg.8
/usr/man/tr/man8/installpkg.8
/usr/man/uk/man8/installpkg.8
find: `/home/guest/.gvfs': Permission denied
/sbin/installpkg
root@porteus:/home/guest# find / -xdev -name '*removepkg*'
/usr/man/de/man8/removepkg.8
/usr/man/el/man8/removepkg.8
/usr/man/es/man8/removepkg.8
/usr/man/fa/man8/removepkg.8
/usr/man/fr/man8/removepkg.8
/usr/man/id/man8/removepkg.8
/usr/man/it/man8/removepkg.8
/usr/man/man8/removepkg.8
/usr/man/nb/man8/removepkg.8
/usr/man/nl/man8/removepkg.8
/usr/man/pl/man8/removepkg.8
/usr/man/pt_BR/man8/removepkg.8
/usr/man/pt_PT/man8/removepkg.8
/usr/man/ru/man8/removepkg.8
/usr/man/sv/man8/removepkg.8
/usr/man/tr/man8/removepkg.8
/usr/man/uk/man8/removepkg.8
find: `/home/guest/.gvfs': Permission denied
/sbin/removepkg
root@porteus:/home/guest# 

Popular names.

[Bogomips]

maybe it's my lack of understanding but I think usm should be more robust in its behaviour, at least it should manage errors in a proper way such as warning about them rather to try to continue executing as if nothing happened

That is also my thinking, but in this case after looking into it in depth, saw there was chance to recover from error. Do you not think that it is better to try and recover from the error, rather than give an error message. as already mentioned in my previous post? After all the aim of the exercise is to get the package that you need.

FWIW, without touching LiBS.TXT or any of the other data files:

• usm 3.2.0

  Code: Select all

  root@porteus:/home/guest# usm -g ansible
  
   The following items were found.
   Choose an number to confirm. 
   ctrl+c to quit
  
  1) ansible-2.1.2.0-x86_64-1_slonly.txz
  #? 1
  
  Processing:   ansible-2.1.2.0-x86_64-1_slonly.txz 
  awk: cmd. line:1: (FILENAME=- FNR=1) fatal: attempt to access field -1
  grep: /etc/usm/mirrors-.txt: No such file or directory
  grep: /home/guest/p10/Por/var64/usm//PACKAGES.TXT: No such file or directory
  grep: /home/guest/p10/Por/var64/usm//PACKAGES.TXT: No such file or directory
  
   The following packages are required. 
  ansible-2.1.2.0-x86_64-1_slonly.txz [] [not installed]
  
  Total size: 0 KB
  
   Press [r] to remove packages, [q] to quit, or enter to start downloading.
  

• usm patched to try and recover from error intelligently:

  Code: Select all

  root@porteus:/home/guest# cp -p /usr/share/usm/funcpackageGet /usr/share/usm/funcpackageGet.bak
  root@porteus:/home/guest# cp  funcpackageGet.hic  /usr/share/usm/funcpackageGet
  
  root@porteus:/home/guest# usm -g ansible
  
   The following items were found.
   Choose an number to confirm.
   ctrl+c to quit
  
  1) ansible-2.1.2.0-x86_64-1_slonly.txz
  #? 1
  
  Processing:   ansible-2.1.2.0-x86_64-1_slonly.txz
  
  =============
  HEURISTIC USM
  =============
  
  ansible-2.1.2.0-x86_64-1_slonly.txz superseded by ansible-2.1.2.0-x86_64-2_slonly.txz
  
   The following packages are required.
  ansible-2.1.2.0-x86_64-2_slonly.txz [2328K] [not installed]
  
  Total size: 2 MB
  
   Press [r] to remove packages, [q] to quit, or enter to start downloading.
  
  root@porteus:/home/guest# usm -v
   You are using USM version:  3.2.0
  

[brokenman]

Thanks bogomips!
Is this functionpackageGet a dropin? Can I simply replace the existing file with this one to test?

[Bogomips]

^ @brokenman
Yes. It's only an edit at lines 90 and 390, so just replacing will be fine, as shown in the code block above. Tried to cover all the bases, but you might be able to spot something I've missed.

[port]

maybe it's my lack of understanding but I think usm should be more robust in its behaviour, at least it should manage errors in a proper way such as warning about them rather to try to continue executing as if nothing happened

That is also my thinking, but in this case after looking into it in depth, saw there was chance to recover from error. Do you not think that it is better to try and recover from the error, rather than give an error message. as already mentioned in my previous post? After all the aim of the exercise is to get the package that you need.

sure it is best to recover from errors but always with informative warnig about there were errors and what they were and the choice selected to recover ;-)

I've replaced the funcpackageGet with the new funcpackageGet.hic to incorporate the improvements and live on the cutting edge ;-)

Anyway I think usm should be more robust and informative. I didn't know about how many files it use, shouldn't be a good idea to refactor it? maybe rewritting in a "real programming language"?

[Bogomips]

Anyway I think usm should be more robust and informative. I didn't know about how many files it use, shouldn't be a good idea to refactor it? maybe rewritting in a "real programming language"?

Going by experience in writing system software at the level between the 'kernel' and user applications, as well as that gained from tuning system file processing routines in another life, bash on my low spec Porteus live system is superfast! Normally the reason for writing in a real programming language is that instructions are compiled leading to much improved performance speed wise.

Do not really think usm will benefit from any increase in speed. For instance at the moment polishing off a project of my own to set the fastest mirror for each of the repositories, and from which incidentally there has been an unexpected spinoff in that no longer have to worry that usm will not be able to complete updating from all the repositories. Script for finding fastest mirrors. before embarking on the speed test, does quite a bit of pre-processing. The other day pressing the Enter key, and before even being able to break the contact, i.e. with finger still on depressed key, informative output and heading for first speed test had already appeared on screen!

In the time in which finger made the contact on the Enter key, bash had read all the usm mirror text files into arrays, pulled out the URLs from between commented lines or out of commented out lines. stored the URLs in an associative array, read Exclusion URLs given on command line, validated protocol and correspondence with mirror.txt URLs, thereafter read URLs for exclusion from a file, validated same and printed command line URLs as well as header giving numbers/stats on the URLs to be excluded from testing!

As for programming scope bash is in some ways reminiscent of a low level language, despite the odd idiosyncracy here or there. As for debugging, it is great, as no special test harness is required. Can just pull out a chunk of code and test it on the terminal without further ado, and vice versa for development. As for debugging on the fly, just need a few strategically placed prints/echos. And with more abstruse bugs, there's as last resort, the set -x/set +x. IMHO the major hold up comes about when accessing remote files.

[port]

Anyway I think usm should be more robust and informative. I didn't know about how many files it use, shouldn't be a good idea to refactor it? maybe rewritting in a "real programming language"?

Going by experience in writing system software at the level between the 'kernel' and user applications, as well as that gained from tuning system file processing routines in another life, bash on my low spec Porteus live system is superfast! Normally the reason for writing in a real programming language is that instructions are compiled leading to much improved performance speed wise.

Do not really think usm will benefit from any increase in speed. For instance at the moment polishing off a project of my own to set the fastest mirror for each of the repositories, and from which incidentally there has been an unexpected spinoff in that no longer have to worry that usm will not be able to complete updating from all the repositories. Script for finding fastest mirrors. before embarking on the speed test, does quite a bit of pre-processing. The other day pressing the Enter key, and before even being able to break the contact, i.e. with finger still on depressed key, informative output and heading for first speed test had already appeared on screen!

In the time in which finger made the contact on the Enter key, bash had read all the usm mirror text files into arrays, pulled out the URLs from between commented lines or out of commented out lines. stored the URLs in an associative array, read Exclusion URLs given on command line, validated protocol and correspondence with mirror.txt URLs, thereafter read URLs for exclusion from a file, validated same and printed command line URLs as well as header giving numbers/stats on the URLs to be excluded from testing!

As for programming scope bash is in some ways reminiscent of a low level language, despite the odd idiosyncracy here or there. As for debugging, it is great, as no special test harness is required. Can just pull out a chunk of code and test it on the terminal without further ado, and vice versa for development. As for debugging on the fly, just need a few strategically placed prints/echos. And with more abstruse bugs, there's as last resort, the set -x/set +x. IMHO the major hold up comes about when accessing remote files.

I agree bash is really fast (I could even say superfast) but my idea of changing to another "real language" is not looking for increasing speed but increasing robustness, I'm thinking about improving quality rather than improving performance. Now you can say, ok and what's quality for you? well, my thinking is a "real language" will provide a higher abstraction to deal with real issues (objects, strategies and so on) and a cleaner and fail-safe programming style. Trusting in several hard-coded regexp or paths or even existing commands is not a portable scheme neither an scalable one, moreover it's a tricky way of programming prone to errors and weak assumptions (I'm not saying usm is using all this strategies but it could be sensible to them since they are common to bash programming)

In short, in bash programming due to its own nature it's easier to programming weak, tricky and messy and not to take into consideration all possible errors or exceptions, and even if considering them all (through several refining steps) it's hard to handle exceptions, at least very harder than other (top level) languages [I don't say you necessary will write tricky weak code using bash, I say its easier you end up doing so]

I'm a big fan of shell scripts and bash but I'm aware of its limitations ;-)

Also I'm not feel very comfortable with the fact usm being scattered around different places on disk neither split into a lot of pieces, I tend to think about a system script as a whole, a one-file-only script doing a task so if you need to break you code into a lot of modules or function files I tend to think it's time to move up another language. Maybe you will reuse shared libraries rather than reinventing the wheel or at least proven code widely used

[Bogomips]

I agree bash is really fast (I could even say superfast) but my idea of changing to another "real language" is not looking for increasing speed but increasing robustness, I'm thinking about improving quality rather than improving performance. Now you can say, ok and what's quality for you? well, my thinking is a "real language" will provide a higher abstraction to deal with real issues (objects, strategies and so on) and a cleaner and fail-safe programming style.

Quality for me is a robust, well thought out design.And I have every faith in brokenman having done a robust design. With a robust design it's easy to deal with exceptions, especially unforeseen exception conditions.

a "real language" will provide a higher abstraction to deal with real issues (objects, strategies and so on)

As far as I can see there are no glaring objects crying out to be used. Also have a fairly good idea of the kind of program flow involved, having myself done a script to get non-slackware dependencies for packages in pkgs,org by sifting through the documentation for dependencies, and which has served me well in putting together modules for quite a few apps that I had need of: http://forum.porteus.org/viewtopic.php? ... bb4#p40573

Trusting in several hard-coded regexp or paths or even existing commands is not a portable scheme

Does not need to be portable, usm being synonymous with porteus. If talking portability to future releases, need to consider working life, which could perhaps be another year or two, given volatile nature of computing.Would be good if could be made to remain productive yet a couple more years.

Trusting in several hard-coded regexp

Just earlier this week completed replacement of almost all hard-coded regex by soft-coded constants within the procedures associated with setting the fastest repo mirrors for usm. Actually with regard to the LIBS.TXT issue the design was sufficiently robust and/or bash was sufficiently flexible to allow resolving an unforeseeable error situation through insertion of a patch which did not interrupt the workflow. And the patch also makes use of a dynamically compiled regex.

In short, in bash programming due to its own nature it's easier to programming weak, tricky and messy and not to take into consideration all possible errors or exceptions

Again I say it's all dependent on the design. Also it looks like the human factor is being forgotten here. A good programmer would take into consideration all possibilities for error, and the design would be sufficiently flexible to allow for resolution of unforeseen exceptions.