[INFO & SCRIPT] Slackware Mirror Settings for USM

[Bogomips]

@ Ed
Seeing as you sort of owe me big time, please do get cracking and testing
badm.sh

Code: Select all

#!/bin/bash
# badm.sh   Display n worst mirrors in repo by Bogomips    2017-06-23 14:54:17
# Repo defaults to 'slackware'
# Arbitrary cutoff points: n<100    &   (# of mirrors - n) > 9
badm ()
{
um="Argument usage:\t< # of Worst Mirrors to Display >\t[Repository(default:slackware)]"
err="Error! "
ms=("# requested $n > 100" "Requested 0 worst mirrors" "Non-numeric 1st argument:  '$1'" "Repo not found: '$rpo'" "Less than 10 remaining mirrors: $mc-$n=$rm" "$rpo.lst not found. Ensure ${rpo^^} has been set thru fastest mirror script ");
local r=0 q=0;
let y=1 n=-1
[[ $1 ]] || { echo -e $um; return 0; }
[[ $1 =~ ^[0-9]+$ ]] && let n=$1;
[[ n -gt 100 ]] && let r=y; let y++;
((n)) || let r=y; let y++;
[[ n -lt 0 ]] && let r=y; let y++;     # Non-numeric 1st Argument
# Default slackware?
# 2nd Argument: Repo (default: slackware)
rpo=${2:-slackware};
#    let z=y;     # Set 2nd Arg error message level
# Check the Repository
eval $(grep DISTROS= /etc/usm/usm.conf);        # Get distros list from usm.conf
[[ $DISTROS ]] || { echo Error\! File usm.conf corrupted; return 9; }
DISTROS+=" sbo";                                # 2017-06-19 23:44:55
[[ $DISTROS =~ ([[:space:]]|^)$rpo([[:space:]]|$) ]] || let q=y; let y++;
# Check n ag mirror count
#   Get mirror count
if ! ((r)); then
    mc=$(sed /Slackware64-current/q  /etc/usm/mirrors-$rpo.txt | grep -Ec  "^\s*#?\s*(http|ftp)");
    # Check if remaining mirros exceed 9
    let rm=mc-n; [[ rm -lt 10 ]]  &&  let r=y; let y++;
    [[ -f /tmp/$rpo.lst ]] ||  let r=y; let y++;
    if ! ((r)); then    echo "# The $n worst mirrors of ${rpo^^}"
    cut -d" " -f2- /tmp/sesm/$rpo.lst | sort -k2 | cut -d" " -f1 | tail -n$n
    return 0;
    fi
fi
# Error Message Output
((r)) && echo $err${ms[r]}
((q)) && echo $err${ms[r]}
return ${q:-$r};
}
badm $*

[Ed_P]

Needs work.

Code: Select all

guest@porteus:~$ sh ./USM/badm.sh
Argument usage:	< # of Worst Mirrors to Display >	[Repository(default:slackware)]
guest@porteus:~$ sh ./USM/badm.sh 5
Error!
guest@porteus:~$ sh ./USM/badm.sh 5 salix
Error!
guest@porteus:~$ sh ./USM/badm.sh 5 slackware
Error!
guest@porteus:~$ 

[Bogomips]

Thanks, Ed, Did not expect rest of error message to not show.

Error message should now show. badm.sh

Code: Select all

#!/bin/bash
# badm.sh   Display n worst mirrors in repo by Bogomips    2017-06-23 14:54:17
# Repo defaults to 'slackware'
# Arbitrary cutoff points: n<100    &   (# of mirrors - n) > 9
# last Update: 2017-06-24 22:19:36  Error Message repetioins fixed
badm ()
{
um="Argument usage:\t< # of Worst Mirrors to Display >\t[Repository(default:slackware)]"
err="Error! "
local r q=0;
let y=1 n=-1
[[ $1 ]] || { echo -e $um; return 0; }
[[ $1 =~ ^[0-9]+$ ]] && let n=$1;
[[ n -gt 100 ]] && let r=y; let y++;
((n)) || let r=y; let y++;
[[ n -lt 0 ]] && let r=y; let y++;    # Non-numeric 1st Argument
# Default slackware?
# 2nd Argument: Repo (default: slackware)
rpo=${2:-slackware};
#    let z=y;     # Set 2nd Arg error message level
# Check the Repository
eval $(grep DISTROS= /etc/usm/usm.conf);        # Get distros list from usm.conf
[[ $DISTROS ]] || { echo Error\! File usm.conf corrupted; return 9; }
DISTROS+=" sbo";                                # 2017-06-19 23:44:55
[[ $DISTROS =~ ([[:space:]]|^)$rpo([[:space:]]|$) ]] || let q=y; let y++;
# Check non-numeric 2nd Arg
[[ ! $rpo =~ ^[a-z]+$ && n -lt 0 ]] && q=0;
# Check n ag mirror count
#   Get mirror count
if ! ((${r:=q})); then
    mc=$(sed /Slackware64-current/q  /etc/usm/mirrors-$rpo.txt | grep -Ec  "^\s*#?\s*(http|ftp)");
    # Check if remaining mirros exceed 9
    let rm=mc-n; [[ rm -lt 10 ]]  &&  let r=y; let y++;
    [[ -f /tmp/sesm/$rpo.lst ]] ||  let r=y; let y++;
    if ! ((r)); then    echo "# The $n worst mirrors of ${rpo^^}"
    cut -d" " -f2- /tmp/sesm/$rpo.lst | sort -k2 | cut -d" " -f1 | tail -n$n
    return 0;
    fi
fi
# Error Message Output
ms=("# requested $n > 100" "Requested 0 worst mirrors" "Non-numeric 1st argument:  '$1'" "Repo not found: '$rpo'" "Less than 10 remaining mirrors: $mc-$n=$rm" "$rpo.lst not found. Ensure ${rpo^^} has been set thru fastest mirror script");
((r)) && echo $err${ms[r-1]}
[[ q -gt r ]] && echo $err${ms[q-1]}
return ${q:-$r};
}
badm $*
  

[Ed_P]

Needs work.

Code: Select all

guest@porteus:~$ ./USM/badm.sh
Argument usage:	< # of Worst Mirrors to Display >	[Repository(default:slackware)]
guest@porteus:~$ ./USM/badm.sh 5
Error! slackware.lst not found. Ensure SLACKWARE has been set thru fastest mirror script
guest@porteus:~$ ./USM/badm.sh 5 /etc/usm/
Error! Repo not found: '/etc/usm/'
guest@porteus:~$ ./USM/badm.sh 5 /etc/usm/slackware.lst
Error! Repo not found: '/etc/usm/slackware.lst'
guest@porteus:~$ ./USM/badm.sh 5 /etc/usm/mirrors-slackware.txt
Error! Repo not found: '/etc/usm/mirrors-slackware.txt'
guest@porteus:~$ ls /etc/usm/mirrors*
/etc/usm/mirrors-alien.txt  /etc/usm/mirrors-slackonly.txt
/etc/usm/mirrors-ponce.txt  /etc/usm/mirrors-slackware.txt
/etc/usm/mirrors-salix.txt  /etc/usm/mirrors-slackwarepatches.txt
/etc/usm/mirrors-sbo.txt    /etc/usm/mirrors-slacky.txt
guest@porteus:~$ 

It appears you are looking for files in the /tmp folder created by masm.sh and before the distro has been rebooted
and the /tmp/ folder cleared. I suggest you change the default msg to be
"< # of Worst Mirrors to Display > [/tmp Repository(default:slackware.lst)]".

Ideally IMHO a Mirror Settings script that would present the user with the options to:
- select all distro mirrors or just one
- then option to list the 3-5 fastest mirrors for the selected distro's mirror/s or the 3-5 slowest mirrors
- if a single distro was selected and the fastest mirrors option selected the option to choose one of
the displayed mirrors and update the distro's file with it
would be a very useful tool.

[Bogomips]

^ Thanks for feedback. Did not say too much because, needed to see how error message came across. Prior to running badm.sh, if you set the slackware mirrors by running sesm.sh, all the files will be there. i.e. after running sesm all , one can then pick up a list of bad mirrors which do not bear testing.

Anyway one can start small by putting them on command line after all. Expect these to be fastest mirrors with some sort of issue e.g. slackwarecatalogs, which crashes the usm slackware update, and fastest slacky with an issue as well. So that will be next small step.

[Bogomips]

Script to set the Slackware usm Mirrors enhanced with Mirror Exclusion List on Command Line:

Code: Select all

guest@porteus:~$ bash   sesm.sh
Argument usage:         all     [<Toggle Debug: 0/1>]   [URL ..]
URL .. Optional Exclusion Mirror List on Command Line

http://forum.porteus.org/viewtopic.php? ... c7a#p55999

[Ed_P]

Code: Select all

guest@porteus:~$ ./USM/sesm.sh
Argument usage:		all	[<Toggle Debug: 0/1>]	[ URL .. ]
URL .. Optional Exclusion Mirror List on Command Line
guest@porteus:~$ ./USM/sesm.sh all slackware

Error! Mirror Parameter neither http nor ftp:
slackware
guest@porteus:~$ 

It works, I guess.

[Bogomips]

Ideally IMHO a Mirror Settings script that would present the user with the options to:
- select all distro mirrors or just one
- then option to list the 3-5 fastest mirrors for the selected distro's mirror/s or the 3-5 slowest mirrors
-

if a single distro was selected and the fastest mirrors option selected the option to choose one of
the displayed mirrors and update the distro's file with it

would be a very useful tool.

Preoccupied with immediate concerns, but hey, DIY! (It's an IKEA sort of thing):

Mirror Settings script that would present the user with the options to:
- select all distro mirrors or just one

sesm.sh or Set fastest mirror for a repo: fasm.sh

then option to list the 3-5 fastest mirrors for the selected distro's mirror/s or the 3-5 slowest mirrors

/tmp repo.lst sort and cut or cut and sort, and then head or tail. Worked out code for this in the first post.

if a single distro was selected and the fastest mirrors option selected the option to choose one of
the displayed mirrors and update the distro's file with it

Run fasm.sh then then URLs from /tmp/fasm|main/repo.lst feed into a select, and after choice made, use manual setting of slackware mirror masm.sh to set it.

QED

[Bogomips]

Pleased to provide Icing on Cake for Delectation of All and Sundry. 8)
http://forum.porteus.org/viewtopic.php? ... 9fb#p55999

[Ed_P]

Updates

Sat Jul 1

Sat Jul 1 = 2017-07-01. Where is the 2017-07-01 change? sesm.sh shows Last Update 2017-06-30.

[*]Script sesm.sh

Code: Select all

#!/bin/bash
# sesm.sh   Set the Slackware Miirors for usm Repos    Bogomips    2017-06-13 19:33:50
# fasm.sh   Adaptation by Bogomips to Set Fastest Slackware Repository Mirror   2017-06-07 21:35:10

# Last Update   2017-06-30 21:49:19 Comments; bad(): Output msgs + formatting

Cut Out can be made from sesm.sh for the Script fasm.sh that sets the Fastest Slackware Mirror for a Repo:

• Code: Select all

  sed "s/\s*sesm\s*().*/fasm \$*/;T;q" $g/sesm.sh > fasm.sh
  guest@porteus:~$ bash fasm.sh
  Argument usage:         <Repository>            [<Toggle Debug: 0/1>]   [ URL .. ]
  URL .. Optional Exclusion Mirror List on Command Line
  Optional Exclusion File of URLs: .usm/badmirrors.txt (following format of 'mirrors-<repo>,txt')
  

  Code: Select all

  guest@porteus:~$ bash fasm.sh slacky
  
  All Exclusion URLs:
  Cmd Line URLs=0         File (.usm/badmirrors.txt) URLs=64      Merged(Unique) URLs=51
  Done!
  slacky mirror set to: http://slack.isper.sk/pub
  

Where is the .usm/ folder for the badmirrors.txt file referenced above suppose to be?

Code: Select all

guest@porteus:~$ ls .usm/
/bin/ls: cannot access '.usm/': No such file or directory
guest@porteus:~$ su
Password: 
root@porteus:/home/guest# ls ./usm
/bin/ls: cannot access './usm': No such file or directory
root@porteus:/home/guest#

[Bogomips]

Updates

Sat Jul 1

Sat Jul 1 = 2017-07-01. Where is the 2017-07-01 change? sesm.sh shows Last Update 2017-06-30.

[*]Script sesm.sh

Code: Select all

#!/bin/bash
# sesm.sh   Set the Slackware Miirors for usm Repos    Bogomips    2017-06-13 19:33:50
# fasm.sh   Adaptation by Bogomips to Set Fastest Slackware Repository Mirror   2017-06-07 21:35:10

# Last Update   2017-06-30 21:49:19 Comments; bad(): Output msgs + formatting

Now when looking for a program and come across one dated 2015, I'm reassured if there is a timestamp on page which says last updated 2017. Then know it's the latest version. So similarly, it is that after updating script a day or two may pass before getting time to finish the post. Once post completed, then update the updates entry. Consequently changed title from Updates to Updates Posted.

Cut Out can be made from sesm.sh for the Script fasm.sh that sets the Fastest Slackware Mirror for a Repo:

• Code: Select all

  sed "s/\s*sesm\s*().*/fasm \$*/;T;q" $g/sesm.sh > fasm.sh
  guest@porteus:~$ bash fasm.sh
  Argument usage:         <Repository>            [<Toggle Debug: 0/1>]   [ URL .. ]
  URL .. Optional Exclusion Mirror List on Command Line
  Optional Exclusion File of URLs: .usm/badmirrors.txt (following format of 'mirrors-<repo>,txt')
  

  Code: Select all

  guest@porteus:~$ bash fasm.sh slacky
  
  All Exclusion URLs:
  Cmd Line URLs=0         File (.usm/badmirrors.txt) URLs=64      Merged(Unique) URLs=51
  Done!
  slacky mirror set to: http://slack.isper.sk/pub
  

Where is the .usm/ folder for the badmirrors.txt file referenced above suppose to be?

This brings us to the matter of choice, and I for one when trying something new or that I'm unfamiliar with do not like having too many choices. Like to be told do this, do this, and Bob's your uncle!

So in this matter the file can be anything you like, just have to edit script accordingly. Taking decision out of hands of user, have decided file should be badmirrors.txt in a directory .usm.

Helpful comments appreciated. Updated post: http://forum.porteus.org/viewtopic.php? ... 40e#p55999


BTW ran today with almost no overlap, resulting in much improved time:

Code: Select all

guest@porteus:~$ time sesm  all http://slack.isper.sk/pub http://mirror.ovh.net/mirrors/ftp.slackware.com/ http://slackware.mirrorcatalogs.com/ http://mirror.internode.on.net/pub/slackware/            

Exclusion URL Specified on Command Line:
http://slack.isper.sk/pub
http://mirror.ovh.net/mirrors/ftp.slackware.com/
http://slackware.mirrorcatalogs.com/
http://mirror.internode.on.net/pub/slackware/

All Exclusion URLs:
Cmd Line URLs=4         File (.usm/badmirrors.txt) URLs=64      Merged(Unique) URLs=67

SLACKWARE
Testing http://mirrors.ucr.ac.cr/slackware/pub/slackware/ 0.616820
Mirrors=152             Processed=32            Remaining=120
Testing ftp://ftp.tu-chemnitz.de/.SAN0/pub/linux/slackware/ 0.676709
Mirrors=152             Processed=46            Remaining=106
Testing ftp://patroklos.noc.ntua.gr/pub/linux/slackware/ 1.573209
Mirrors=152             Processed=56            Remaining=96
Testing http://taper.alienbase.nl/mirrors/slackware/ 0.401148
Mirrors=152             Processed=77            Remaining=75
Testing http://sunsite.icm.edu.pl/packages/linux-slackware/ 0.116343
Mirrors=152             Processed=89            Remaining=63
Testing ftp://ftp.mirrorservice.org/sites/ftp.slackware.com/pub/slackware/ 0.861712
Mirrors=152             Processed=119           Remaining=33
Testing ftp://slackware.mirrors.tds.net/pub/slackware/ 0.614798
Mirrors=152             Processed=145           Remaining=7
Done!
slackware mirror set to: ftp://hpc-mirror.usc.edu/pub/linux/distributions/slackware/

SLACKWAREPATCHES
Done!
slackwarepatches mirror set to: ftp://hpc-mirror.usc.edu/pub/linux/distributions/slackware/

SLACKY
Done!
slacky mirror set to: http://slackware.org.uk/slacky

SALIX
Testing http://download.salixos.org Fail
Mirrors=28              Processed=11            Remaining=17
Testing ftp://ftp.nluug.nl/pub/os/Linux/distr/salix 0.577518
Mirrors=28              Processed=19            Remaining=9
Testing ftp://mirrors.xmission.com/salix 2.207532
Mirrors=28              Processed=25            Remaining=3
Done!
salix mirror set to: http://ftp.nux.ipb.pt/dists/salix

ALIEN
Done!
alien mirror set to: http://bear.alienbase.nl/mirrors/people/alien/sbrepos

PONCE
Done!
ponce mirror set to: http://ponce.cc/slackware

SLACKONLY
Done!
slackonly mirror set to: http://packages.slackonly.com/pub/packages

SBO
Done!
sbo mirror set to: http://slackbuilds.org/slackbuilds

real    1m8.295s
user    0m2.307s
sys     0m1.008s

There is one thing however that is bugging me that only seems to occur with slackware, already happened more than a couple of times, but not with slackware patches: sesm run followed by usm update process

• 1st Update Attempt

  Code: Select all

  root@porteus:/home/guest# time usm -u all
   Starting slackware database update 
  Downloading: vercheck.txt  DONE
  Downloading: CHECKSUMS.md5  DONE
  Downloading: MANIFEST.bz2  DONE
  Downloading: PACKAGES.TXT  DONE
  Downloading: LIBS.TXT.gz  DONE
   There was a problem downloading the file: 
  
  
  real    0m4.486s
  user    0m0.687s
  sys     0m0.919s
  
  guest@porteus:~$ source  crm.sh slackware
  SLACKWARE
  Mirror: ftp://hpc-mirror.usc.edu/pub/linux/distributions/slackware/
  151 Commented Out
  

• 2nd Update Attempt

  Code: Select all

  guest@porteus:~$ source  masm.sh slackware http://mirror.nl.leaseweb.net/slackware
  Done!
  slackware mirror set to: http://mirror.nl.leaseweb.net/slackware/
  
  root@porteus:/home/guest# time usm -u all
   Starting slackware database update 
  Downloading: vercheck.txt --> ^C
  rm: cannot remove '/home/guest/p10/Por/tmp64/usm/vercheck.txt': No such file or directory
  
  real    0m18.573s
  user    0m0.534s
  sys     0m0.286s
  

• Finally

  Code: Select all

  guest@porteus:~$ source  masm.sh slackware http://ftp.nluug.nl/os/Linux/distr/slackware
  Done!
  slackware mirror set to: http://ftp.nluug.nl/os/Linux/distr/slackware/
  
  root@porteus:/home/guest# time usm -u all
   Starting slackware database update 
  ...
   Starting sbo database update
  SLACKBUILDS.TXT.gz
  Downloading: SLACKBUILDS.TXT.gz  DONE
   sbo  database updated.
  
  
  Downloading: extra-deps.txt  DONE
  Downloading: ignore-deps.txt  DONE
  
  
  real    1m54.067s
  user    0m21.265s
  sys     0m5.875s
  

[/list]
Need to make a trite script to Set Alternative Fast Mirror, code name: safm.sh 8)

[francois]

Is it ripe enough so that you could make a gui to download and run these files?

[Ed_P]

So in this matter the file can be anything you like, just have to edit script accordingly. Taking decision out of hands of user, have decided file should be badmirrors.txt in a directory .usm.

Helpful comments appreciated. Updated post: http://forum.porteus.org/viewtopic.php? ... 40e#p55999

Now I understand.

Exclusion File of URLs. As it stands sesm.sh expects to find the URL list in file: .usm/badmirrors.txt. Hence

• Code: Select all

  guest@porteus:~$ [[ -d ,usm ]] || mkdir .usm

For whatever reason I kept thinking the .usm folder was a USM created folder, like /etc/usm/ & /var/usm/, not a user created one. Thanks for clarifying it.

[Bogomips]

Is it ripe enough so that you could make a gui to download and run these files?

Run the scripts pertinent to your situation, and let us know how much riper it has to be.

No idea how to make a gui, so be my guest.

[Bogomips]

Very Final Enhancement: Script to Set Alternative Fast Mirror safm.sh just about crosses the i's and dots the t's, for me at least. http://forum.porteus.org/viewtopic.php? ... 6fa#p55999