[INFO & SCRIPT] Slackware Mirror Settings for USM
USM UPDATE PROCEDURE
Established a Procedure to follow at Start of Session:
- Run script to set the fastest Slackware mirrors for USM (aks sesm.sh) in bash shell:
Code: Select all
guest@porteus:~$ bash sesm.sh all
- In root terminal run usm -u all
- If there is an Issue with a Repository Mirror, run script to set an alternate fast mirror (aka safm.sh) in bash shell. For example:
-
Code: Select all
root@porteus:/home/guest# time usm -u all Starting slackware database update Downloading: vercheck.txt DONE Downloading: CHECKSUMS.md5 DONE Downloading: MANIFEST.bz2 DONE Downloading: PACKAGES.TXT DONE Downloading: LIBS.TXT.gz DONE There was a problem downloading the file:
- Invocation to use: bash safm.sh <repo>, where repo is one of: slackware, slackwarepatches, salix, slacky.
Code: Select all
guest@porteus:~$ bash safm.sh slackware Current Mirror: ftp://elektroni.phys.tut.fi/ Alternative Mirrors 1) http://ftp.nluug.nl/os/Linux/distr/slackware/ 0.087508 2) http://mirror.nl.leaseweb.net/slackware/ 0.099194 3) http://mirror.ovh.net/mirrors/ftp.slackware.com/ 0.099655 4) http://mirrors.dotsrc.org/.disk1/slackware/ 0.106202 5) http://ftp1.uninett.no/slackware/ 0.108231 6) http://sunsite.icm.edu.pl/packages/linux-slackware/ 0.112806 #? 1 Done! slackware mirror set to: http://ftp.nluug.nl/os/Linux/distr/slackware/ Another Alternative? [Y/n] Terminating.
-
- Repeat from 3. until completed:
Code: Select all
Downloading: extra-deps.txt DONE Downloading: ignore-deps.txt DONE
![Smile :)](./images/smilies/smile.gif)
As of Mon Jul 24 17:08:33 UTC 2017:
- Script to Set the fastest Slackware Mirrors (aka sesm.sh)
Code: Select all
#!/bin/bash # sesm.sh Set the Slackware Miirors for usm Repos Bogomips 2017-06-13 19:33:50 # fasm.sh Adaptation by Bogomips to Set Fastest Slackware Repository Mirror 2017-06-07 21:35:10 # Internal Reference: usm/sesm_240717.sh # Last Update 2017-07-20 00:38:14 Failure of All Mirrors of Reposiory (single mirror normally) does not lead to Termination. # 2017-07-19 14:32:46 Uniformity: Replaced bad() with bad() from 'safm.sh'. # 2017-06-30 21:49:19 Comments; bad(): Output msgs + formatting # 2017-06-29 23:22:09 Sourcing: FUNCNAME[1]==source (,|source) <file> 2017-06-29 23:19:54 # 2017-06-29 16:11:24 Streamlinig with Constants: $hash & $proto # 2017-06-29 16:11:24 Impleemented Single Mirror Repo Exception from Exclusion # 2017-06-27 22:50:12 Bad Mirrors list in file '.usm/badmirrors.txt' merge with Command Line URL Exclusion List # mirror_test.sh # This script tests a list of porteus mirrors for speed # # AUTHOR: Lance Rushing <lance_rushing@hotmail.com> # Modified for Porteus by brokenman <brokenman@porteus.org> # This script is covered under the GNU Public License: http://www.gnu.org/licenses/gpl.txt # Constants proto="(http|ftp)://[^[:space:]]+"; hash="[[:space:]]*#?[[:space:]]*" sus="Argument usage:\t\tall\t[<Toggle Debug: 0/1>]\t[ URL .. ]\nURL .. Optional Exclusion Mirror List on Command Line\nOptional Exclusion File of URLs: .usm/badmirrors.txt (following format of 'mirrors-<repo>,txt')" fs0="Argument usage:\t\t<Repository>\t\t"; fus=$fs0[${sus#*[}; # 2017-06-30 22:53:54 fsm="Warning! No Mirror Timings obtained for Repo: \${1^^}\$'\n'No Mirror Alteration made."; # Assoc Arrays to be Global unset allm; declare -A allm; # Used to Validate URL unset bay; declare -A bay; # Used to Skip testing of URL ## Verification by Key Press 2015-02-12 00:11:36 kpv () { local nu; local k; while (( 1 )) do read -s -n1 -p "$1? [Y/n]" k case $k in "$nu" ) echo ""; return 0;; [yY] ) echo ""; return 0;; [nN] ) echo ""; return 1;; * ) echo $'\n'"Invalid Key '$k'" esac done } # Function bad() # Ascertains if a debug option was defaulted, and sets debug toggle accordingly. # Exclusion from Testing # Cmd Line URLs # Checks Protocols of Cmd Line URLs # Checks Relevance of Cmd Line URLs (corresponds to entry in 'mirrors-<repo>.txt') # Lists them out. # Exclusion from Testing # File ('.usm/badmirrors.txt' similar format to 'mirrors-<repo>.txt' files) URLs # Option to Proceed if no URLs found in file. # Checks Relevance of Cmd Line URLs (correspond to entry in 'mirrors-<repo>.txt'?) # List irrelevant URLs found in file together with containing line in file. # Option to Proceed if irrelevant URLs found in file.(not in 'mirrors-<repo>.txt') bad () { err="Error! " bm=("Mirror Parameter neither http nor ftp:" "Command Line URL not found in Mirrors' List:"); # Command Line 2017-06-25 15:01:32 # Debug Option? dfl=""; [[ $2 =~ ^[0-9]+$ ]] && { dfl=$2; shift; } shift; # Split Cmd Line URLs into pass 'pay' and fail 'fay' categories. unset pay fay ret; [[ $* ]] && echo; # No Cmd Line URLs for l in $*; do if [[ $l =~ ^$proto ]]; then pay+=($l); else fay+=($l); fi done # Announce Erronous Mirrors if [[ -v fay ]]; then ret=6; echo $err${bm[ret-6]}; for l in ${fay[*]}; do echo $l; done fi # Check Cmd Line URL for Relevance # Array allm[*] populated with URLs from 'mirrors-<repo>.txt' Time (0.114") for r in $DISTROS; do readarray -t f < <(sed /Slackware64-current/q /etc/usm/mirrors-$r.txt); for l in "${f[@]}"; do [[ $l =~ ^$hash($proto) ]] && allm[${BASH_REMATCH[1]%/}]=; done # for l in "${f[@]}"; do [[ $l =~ ^$hash((http|ftp).*) ]] && allm[${BASH_REMATCH[1]%/}]=; done done # Assoc Array 'allm' of All Mirrors now Set for u in ${pay[*]}; do if [[ ! -v allm[${u%/}] ]]; then [[ ret++ -lt 7 ]] && { ret=7; echo $err${bm[ret-6]}; } echo $u; fi done ((ret)) && return $ret; # Irrelevant URL Error Return. # Announce Cmd Line URLs if ((${#pay[*]})); then echo Exclusion URL Specified on Command Line:; for u in ${pay[*]}; do bay[${u%/}]= ; echo $u; done fi # Keep Count of Cmd Line URLs ucmd=${#pay[*]}; unset ufle umge # Call from safm? 2017-07-11 13:39:16 [[ ${FUNCNAME[1]} == safm ]] && return 0; # Check for file: '.usm/badmirrors.txt' [[ -f .usm/badmirrors.txt ]] || { echo -e "\nAll Exclusion URLs:\nCmd Line URLs=$ucmd\t\tFile (.usm/badmirrors.txt) URLs=0\tMerged(Unique) URLs=${#bay[*]}"; return 0; } mc=$(grep -Ec "^$hash$proto" .usm/badmirrors.txt); # Mirror Count if ! ((mc)); then echo "No URL found in File '.usm/badmirrors.txt'"; kpv Proceed || return 10; return 0; fi # Keep Count of File URLs ufle=$mc; # Validate the URLs in File # Read URLs into Work Array readarray -t way < <(grep -E "^$hash$proto" .usm/badmirrors.txt) # readarray -t way < <(grep -E "^\s*#*\s*$proto" .usm/badmirrors.txt) [[ mc -ne ${#way[*]} ]] && { echo -e "Internal Error! grep/bash count differ: mc=$mc\t\${#way[*]}=${#way[*]}"; return 11; }; # Relevance Check for File URLs unset fay; # Fail Array for w in "${way[@]}"; do [[ $w =~ ^$hash($proto) ]] || { echo "Internal Error! grep/bash RE clash: w='$w'"; return 12; } u=${BASH_REMATCH[1]%/}; # Isolated URL if [[ ! -v allm[$u] ]]; then [[ -v fay ]] || { ret=12; echo $err${bm[1]/ Command Line/File}; } fay+=($u); [[ $u == $w ]] && echo $u || echo $u$'\n'$w; echo; else bay[$u]=; fi done ((ret)) && { kpv Proceed || return $ret; } # Keep Count of Unique URLs umge=${#bay[*]} ((ucmd+ufle)) && echo -e "\nAll Exclusion URLs:\nCmd Line URLs=$ucmd\t\tFile (.usm/badmirrors.txt) URLs=$ufle\tMerged(Unique) URLs=$umge" # "Cmd Line URLs=$ucmd\tFile URLs=$ufle\t\tMerged(Unique) URLs=$umge" return 0; } # Get Fastest Mirror fam() { # 1st Parameter if set, toggles Selection to only Process Bad Mirrors. 2017-06-28 23:01:06 # 2nd Parameter aaumed never to be set ## get list local -n MIRRORS=${2:-may}; # Changed from 1s to 2nd Para. 2017-06-29 15:58:55 ## Number of seconds before the test is considered a failure TIMEOUT="4" ## String to store results in RESULTS="" # Elapsed Time Check Output an Update every 5 sec. let x=SECONDS; for ((i=0; i<${#MIRRORS[*]}; i++)) ; do MIRROR=${MIRRORS[i]}; let y=SECONDS; if [[ y-x -gt 5 && i -gt 0 ]]; then tail -n1 $RAW; echo -e "Mirrors=${#MIRRORS[*]}\t\tProcessed=$i\t\tRemaining=$((${#MIRRORS[*]}-i))"; let x=SECONDS; # Reset fi # Implementing Exclusive OR, Ensuress Processing if TRUE. 2017-06-28 19:51:57 [[ ! $1 && ${bay[${MIRROR%/}]+x} ]] && continue; # Exclude 2017-06-25 22:26:41 [[ $1 && ! ${bay[${MIRROR%/}]+x} ]] && continue; # Exclude 2017-06-28 19:51:57 2nd Part of XOR echo -n "Testing ${MIRROR} " >> $RAW URL="${MIRROR}%{FILE}" TIME=`curl --max-time $TIMEOUT --silent --output /dev/null --write-out %{time_total} $URL` if [[ $TIME =~ ^0.000 ]] ; then echo "Fail" >> $RAW; else echo $TIME >> $RAW RESULTS="${RESULTS}${TIME}\t${MIRROR}\n"; fi done; echo -e $RESULTS | sort -n > $DUMP FMIR=""; FMIR=`sed '/^$/d' $DUMP | head -n1 | awk '{print$2}'`; # Reset 2017-06-28 16:54:03 } # Get Fastest Slackware Mirror for Repository fasm () { if [[ ! $1 ]]; then echo -e $fus; # fasm() Independent 2017-06-27 22:36:55 return 0; fi # Check the Repository 2017-06-15 00:45:36 if [[ ! ${FUNCNAME[1]} || ${FUNCNAME[1]} =~ ^(main|source)$ ]]; then # Sourcing (,|source) <file> 2017-06-29 23:19:54 Functioning Independently eval $(grep DISTROS= /etc/usm/usm.conf); # Get distros list from usm.conf [[ $DISTROS ]] || { echo Error\! File usm.conf corrupted; return 1; } DISTROS+=" sbo"; # 2017-06-19 23:44:55 [[ $DISTROS =~ ([[:space:]]|^)$1([[:space:]]|$) ]] || { echo Error\! Repository not found.; return 2; } bad $* || return $? # fasm() Independent 2017-06-26 13:36:55 fi dbg=${dfl:-0}; # fasm() Independent 2017-06-26 13:36:55 # Introduce Directory Area TED in /tmp 2017-06-15 17:27:27 if [[ ! $RAW ]]; then TED=${FUNCNAME[1]}; [[ -d /tmp/${TED:=fasm} && -w /tmp/$TED ]] || mkdir /tmp/$TED || mkdir /tmp/$TED${SECONDS:0:-2} # Tags the nearest 100 secs of Up Time fi RAW=/tmp/$TED/$1.lst; echo -n > $RAW; # Initialise. 2017-06-15 17:42:11 ((dbg)) && echo -e "Raw Output:\t$RAW\nSorted Output:\t$DUMP"; # Truncate Mirror List at Slackware Current readarray -t f < <(sed /Slackware64-current/q /etc/usm/mirrors-$1.txt); ((${#f[*]})) || { echo Empty Mirror File\!; return 3; } # Isolate Links # Array f now contains Relevant Section of Mirror File # Process f to set up Array of Mirrors 'may': http or ftp Protocol Links. unset may; for l in "${f[@]}"; do [[ $l =~ ^$hash($proto) ]] && may+=(${BASH_REMATCH[1]}); done # Array Correction 2017-06-13 14:26:06 & FTP 2017-06-09 00:32:14 ((dbg)) && echo Links=${#may[*]}; # of Mirror Links ## Vars DUMP=/tmp/.fmr.tmp # Avoid Duplicated Effort in case of Slackwar/Slackwarepatches if [[ $SMIR && $1 =~ slackware ]]; then FNIR=$SMIR; ((dbg)) && echo FMIR=$FMIR; else fam; # Check there is a Fastest Mirror Found. [[ $FMIR ]] || fam rev; # Rerun using Eliminated if Any [[ $FMIR ]] || { eval echo -e $fsm; return 8; } # All Mirrors u/s # Show time aassociated with Fastest Mirror (debug option) ((dbg)) && echo FMIR=$(grep $FMIR $RAW | cut -d " " -f2-); # Updated to allow for own directory in /tmp unsing global RAW 2017-06-16 18:11:42 [[ $1 =~ slackware ]] && SMIR=$FMIR fi # Set the Mirror # COMMENT OUT ALL Others sudo sed -i -E 's/^\s*(http|ftp)/# &/' /etc/usm/mirrors-$1.txt # 2017-06-09 01:10:11 # UNCOOMMENT FASTEST JUST ONCE! sudo sed -i "0,\%^\s*#\s*$FMIR%s??$FMIR?" /etc/usm/mirrors-$1.txt # Internal Error? Check that there is one Unblocked Mirror e=$(grep -E "^\s*$proto" /etc/usm/mirrors-$1.txt); # 2017-06-11 23:49:14 m=$(grep -Ec "^\s*$proto" /etc/usm/mirrors-$1.txt); # 2017-06-11 23:49:14 if [[ m -ne 1 ]]; then echo Internal Error\!; echo "$e"; return 4; fi echo Done!; echo $1 mirror set to: "$e"; # 2017-06-11 23:52:39 } sesm () # Set the Slackware Miirors for All usm Repos { if [[ ! $1 || $1 != all ]]; then echo -e $sus; return 0; fi # Reset Global [Switch] Varables 2017-06-15 17:14:10 SMIR=""; # Slackware duplication avoidance switch RAW=""; # Raw Output File sret=""; # Uniqu Global only used by sesm() 2017-07-20 00:38:14 # Get Distros eval $(grep DISTROS= /etc/usm/usm.conf); # Get distros list from usm.conf [[ $DISTROS ]] || { echo Error\! File usm.conf corrupted; return 5; } DISTROS+=" sbo"; # 2017-06-19 23:44:55 bad $* || return $?; for r in $DISTROS; do echo; echo ${r^^}; sret=""; fasm $r $dfl || [[ ${sret:=$?} -eq 8 ]] || return $sret; done # sret 2017-07-20 00:38:14 dfl 2017-06-25 15:02:34 } sesm $*
- Script to Set Alternative Fastest Mirror (aka safm.sh)
Code: Select all
#!/bin/bash # safm.sh Set Alternative Fast Mirror for Repository by Bogomips 2017-07-02 19:29:46 # Last Update: 2017-07-12 12:59:50 Check if Raw Output from sesm/fasm exists # Internal Reference: usm/safm_040717.sh # masm.sh Manually set Slackware Mirror by Bogomips # Last Update: # 2017-07-11 13:39:16 bad() added for processing of Command Line Exclusion URLs # 2017-07-04 13:07:16 Display Current Mirror followed by Heading # 2017-06-19 23:44:55 Include 'sbo' as Repository masm () { if [[ ! $2 ]]; then echo -e "Argument usage:\t\t<Repository>\t\t<Mirror URL>"; urn 0; fi # Check the Protocol [[ $2 =~ ^(http|ftp):// ]] || { echo URL Error: Neither http nor ftp.; return 1; } # Check the Repository eval $(grep DISTROS= /etc/usm/usm.conf); # Get distros list from usm.conf [[ $DISTROS ]] || { echo Error\! File usm.conf corrupted; return 1; } DISTROS+=" sbo"; # 2017-06-19 23:44:55 [[ $DISTROS =~ ([[:space:]]|^)$1([[:space:]]|$) ]] || { echo Error\! Repository not found.; return 2; } # Match the URL m=$(grep -c "\s*#\?\s*$2" /etc/usm/mirrors-$1.txt); # of Matches ! ((m)) && { echo \Error! No matching Mirror found.; return 3; } url=$2; # Exact Matches? Also check if Single Match is an Exact Match 2017-06-16 01:00:01 # Provide for Missing Trailing Slash 2017-06-17 15:50:45 n=$(grep -c "\s*#\?\s*$2\s*$" /etc/usm/mirrors-$1.txt); # More Exactly Stipulated 2017-06-16 16:57:49 # Two Part test caters for exact match not having trailing /. # 2017-06-18 14:57:55 ! ((n)) && n=$(grep -c "\s*#\?\s*${2%\/}\/\s*$" /etc/usm/mirrors-$1.txt); # More Exactly # No Exact Match if ! ((n)); then w=$(grep "\s*#\?\s*$2" /etc/usm/mirrors-$1.txt | sed s/^\s*#*\s*//) echo Partial Match found. Select a Mirror or Enter 0 to quit.; select p in $w; do break; done [[ $p ]] || return 4; url=$p; # URL confirmed to exactly match. fi # If more than one Exact Match, first one will be used. # Block All Mirrors sudo sed -i -E 's/^\s*(http|ftp)/# &/' /etc/usm/mirrors-$1.txt # 2017-06-09 01:10:11 # Unblock Selected Just Once sudo sed -i "0,\%^\s*#\s*$url%s??$url?" /etc/usm/mirrors-$1.txt echo Done!; echo $1 mirror set to: $(grep -E "^\s*(http|ftp)" /etc/usm/mirrors-$1.txt); # Providing more accurate information 2017-06-17 15:46:39 } kpv () { ## Verification by Key Press 2015-02-12 00:11:36 local nu; local k; while (( 1 )) do read -s -n1 -p "$1? [Y/n]" k case $k in "$nu" ) echo ""; return 0;; [yY] ) echo ""; return 0;; [nN] ) echo ""; return 1;; * ) echo $'\n'"Invalid Key '$k'" esac done } # Constants proto="(http|ftp)://[^[:space:]]+"; hash="[[:space:]]*#?[[:space:]]*" # Assoc Arrays to be Global unset allm; declare -A allm; # Used to Validate URL unset bay; declare -A bay; # Used to Skip testing of URL # Function bad() # Ascertains if a debug option was defaulted, and sets debug toggle accordingly. # Exclusion from Testing # Cmd Line URLs # Checks Protocols of Cmd Line URLs # Checks Relevance of Cmd Line URLs (corresponds to entry in 'mirrors-<repo>.txt') # Lists them out. # Exclusion from Testing # File ('.usm/badmirrors.txt' similar format to 'mirrors-<repo>.txt' files) URLs # Option to Proceed if no URLs found in file. # Checks Relevance of Cmd Line URLs (correspond to entry in 'mirrors-<repo>.txt'?) # List irrelevant URLs found in file together with containing line in file. # Option to Proceed if irrelevant URLs found in file.(not in 'mirrors-<repo>.txt') bad () { err="Error! " bm=("Mirror Parameter neither http nor ftp:" "Command Line URL not found in Mirrors' List:"); # Command Line 2017-06-25 15:01:32 # Debug Option? dfl=""; [[ $2 =~ ^[0-9]+$ ]] && { dfl=$2; shift; } shift; # Split Cmd Line URLs into pass 'pay' and fail 'fay' categories. unset pay fay ret; [[ $* ]] && echo; # No Cmd Line URLs for l in $*; do if [[ $l =~ ^$proto ]]; then pay+=($l); else fay+=($l); fi done # Announce Erronous Mirrors if [[ -v fay ]]; then ret=6; echo $err${bm[ret-6]}; for l in ${fay[*]}; do echo $l; done fi # Check Cmd Line URL for Relevance # Array allm[*] populated with URLs from 'mirrors-<repo>.txt' Time (0.114") for r in $DISTROS; do readarray -t f < <(sed /Slackware64-current/q /etc/usm/mirrors-$r.txt); for l in "${f[@]}"; do [[ $l =~ ^$hash($proto) ]] && allm[${BASH_REMATCH[1]%/}]=; done # for l in "${f[@]}"; do [[ $l =~ ^$hash((http|ftp).*) ]] && allm[${BASH_REMATCH[1]%/}]=; done done # Assoc Array 'allm' of All Mirrors now Set for u in ${pay[*]}; do if [[ ! -v allm[${u%/}] ]]; then [[ ret++ -lt 7 ]] && { ret=7; echo $err${bm[ret-6]}; } echo $u; fi done ((ret)) && return $ret; # Irrelevant URL Error Return. # Announce Cmd Line URLs if ((${#pay[*]})); then echo Exclusion URL Specified on Command Line:; for u in ${pay[*]}; do bay[${u%/}]= ; echo $u; done fi # Keep Count of Cmd Line URLs ucmd=${#pay[*]}; unset ufle umge # Call from safm? 2017-07-11 13:39:16 [[ ${FUNCNAME[1]} == safm ]] && return 0; # Check for file: '.usm/badmirrors.txt' [[ -f .usm/badmirrors.txt ]] || { echo -e "\nAll Exclusion URLs:\nCmd Line URLs=$ucmd\t\tFile (.usm/badmirrors.txt) URLs=0\tMerged(Unique) URLs=${#bay[*]}"; return 0; } mc=$(grep -Ec "^$hash$proto" .usm/badmirrors.txt); # Mirror Count if ! ((mc)); then echo "No URL found in File '.usm/badmirrors.txt'"; kpv Proceed || return 10; return 0; fi # Keep Count of File URLs ufle=$mc; # Validate the URLs in File # Read URLs into Work Array readarray -t way < <(grep -E "^$hash$proto" .usm/badmirrors.txt) # readarray -t way < <(grep -E "^\s*#*\s*$proto" .usm/badmirrors.txt) [[ mc -ne ${#way[*]} ]] && { echo -e "Internal Error! grep/bash count differ: mc=$mc\t\${#way[*]}=${#way[*]}"; return 11; }; # Relevance Check for File URLs unset fay; # Fail Array for w in "${way[@]}"; do [[ $w =~ ^$hash($proto) ]] || { echo "Internal Error! grep/bash RE clash: w='$w'"; return 12; } u=${BASH_REMATCH[1]%/}; # Isolated URL if [[ ! -v allm[$u] ]]; then [[ -v fay ]] || { ret=12; echo $err${bm[1]/ Command Line/File}; } fay+=($u); [[ $u == $w ]] && echo $u || echo $u$'\n'$w; echo; else bay[$u]=; fi done ((ret)) && { kpv Proceed || return $ret; } # Keep Count of Unique URLs umge=${#bay[*]} ((ucmd+ufle)) && echo -e "\nAll Exclusion URLs:\nCmd Line URLs=$ucmd\t\tFile (.usm/badmirrors.txt) URLs=$ufle\tMerged(Unique) URLs=$umge" # "Cmd Line URLs=$ucmd\tFile URLs=$ufle\t\tMerged(Unique) URLs=$umge" return 0; } safm () { if [[ ! $1 ]]; then echo -e "Argument usage:\t\t<Repository>\t\t[<# of Mirrors to Display (default=6)>]"; return 0; fi # Check the Repository eval $(grep DISTROS= /etc/usm/usm.conf); # Get distros list from usm.conf [[ $DISTROS ]] || { echo Error\! File usm.conf corrupted; return 1; } DISTROS+=" sbo"; # 2017-06-19 23:44:55 [[ $DISTROS =~ ([[:space:]]|^)$1([[:space:]]|$) ]] || { echo Error\! Repository not found.; return 2; } # Look for Command Line Exclusion URLs (fixing default in the process) 2017-07-11 14:51:46 bad $* || return $? # fasm() Independent 2017-06-26 13:36:55 # Get Relevant Files: Output of either sesm or fasm in /tmp inits="sesm main source fasm"; unset way; # Initialise 2017-07-12 12:51:42 readarray -t way < <(for d in $inits; do [[ -f /tmp/$d/$1.lst ]] && cat /tmp/$d/$1.lst ; done | sort -k3 | cut -d\ -f2-) # Work Array # Check if Raw Output from sesm/fasm exists 2017-07-12 12:59:50 if [[ ! $way ]]; then echo; echo -n "$err"; echo -e "Raw Timings for Fastest Mirror Settings mot Found.\n\nPlease run either script to set slackware mirrors (aka sesm.sh): bash sesm.sh all\nOr script to set fastest mirror for repository (aka fasm.sh): bash fasm $1"; return 8; fi unset bay; declare -A bay; unset uny # Filter Array; Unique Array # Add Current Mirror to Filter 2017-07-03 15:00:32 [[ $(grep -E "^\s*(http|ftp)" /etc/usm/mirrors-$1.txt) =~ $proto ]] || return 3; # No current mirror! bay[${BASH_REMATCH%/}]=; # Subscript for Current Mirror 2017-07-03 15:00:32 # Display Current Mirror 2017-07-04 13:07:16 echo ; echo Current Mirror: $BASH_REMATCH; echo Alternative Mirrors; # Filter with Trailing Slash Removed 2017-07-03 15:00:32 for w in "${way[@]}"; do u=${w% *}; [[ -v bay[${u%/}] ]] && continue; bay[${u%/}]=; uny+=("$w"); done # 'Select' Array <- Unique Array Slice say=("${uny[@]:0:${dfl:-6}}") # bad() also picks up default for Slice Size 2017-07-11 14:56:32 # say=("${uny[@]:0:${2:-6}}") while ((${#say[*]})); do select v in "${say[@]}"; do [[ $v ]] || continue; unset say[$REPLY-1]; say=("${say[@]}"); # Shrink Array masm $1 ${v% *} || return $?; break; done kpv "Another Alternative" || break; done ((${#say[*]})) || echo -n "Choices Exhausted! "; echo Terminating. } safm $*
At this point no need for any further action, unless there is worry about bandwidth, or one wishes to accomplish the testing and the usm update in series, in which cast the following section on tuning the testing would be of interest.
Tuning the Testing
Untuned the testing of mirrors takes around 3' 40" (See last code block of this post). However this time can be reduced by: (Please refer to Exclusion File of URLs here viewtopic.php?f=39&t=7000&sid=5681d83b3 ... 54a#p55999)
- Description of Stages
- Excluding Mirrors on a Geographical Basis
See Geographical Exclusion - Excluding Mirrors on a Worst Mirror Basis
- Excluding Mirrors on a Geographical Basis
- Stages of Tuning (Example)
- Create directory for exclusion file of URLs
Code: Select all
mkdir .usm
- Using a text editor paste together a .usm/badmirrors.txt file from the mirror.txt files. Example Excerpt:
Code: Select all
#---------------------------------------------------------------- # Slackware64-14.1 #---------------------------------------------------------------- # AUSTRALIA (AU) # ftp://ftp.cc.swin.edu.au/slackware/ # http://ftp.cc.swin.edu.au/slackware/ # ftp://ftp.iinet.net.au/pub/slackware/ # http://ftp.iinet.net.au/pub/slackware/ # ftp://mirror.aarnet.edu.au/pub/slackware/ # http://mirror.aarnet.edu.au/pub/slackware/ # ftp://mirror.as24220.net/pub/slackware/ # http://mirror.as24220.net/pub/slackware/
- Create directory for exclusion file of URLs
- Following a Stage 1 Run of the Scripts, append 30 Worst Slackware Mirrors to badmirrors.txt, using script badm.sh:
- Invocation:
Code: Select all
sh badm.sh 30 slackware >> .usm/badmirrors.txt
- Excerpt from badmirrors.txt Appendage
Code: Select all
#-------- 30 Worst Mirrors ---------------SLACKWARE--------Mon Jul 3 13:33:55 BST 2017 # ftp://ftp.linux.org.tr/slackware/ ftp://mirror.cs.princeton.edu/pub/mirrors/slackware/ ftp://mirror.its.dal.ca/slackware/ ftp://spout.ussg.indiana.edu/linux/slackware/ ftp://mirror.csclub.uwaterloo.ca/slackware/ ftp://mirrors.slackware.com:/slackware/ ftp://ftp.cc.uoc.gr/mirrors/linux/slackware/ ftp://mirrors.us.kernel.org/slackware/ ftp://mirrors.xmission.com/slackware/ http://darkstar.ist.utl.pt/pub/slackware/ ftp://teewurst.cc.columbia.edu/pub/linux/slackware/
- Invocation:
- Following a Stage 2 Run of the Scripts, append 20 Worst Slackware Mirrors to badmirrors.txt, using script badm.sh:
- Invocation:
Code: Select all
badm.sh 20 slackware >> .usm/badmirrors.txt
- Excerpt from badmirrors.txt Appendage:
Code: Select all
#-------- 20 Worst Mirrors ---------------SLACKWARE--------Tue Jul 4 12:57:26 BST 2017 # ftp://perkons.linux.edu.lv/mirrors/ftp.slackware.com/ ftp://mirror.switch.ch/mirror/slackware/ ftp://slackware.mirrors.tds.net/pub/slackware/ ftp://ftp.mirrorservice.org/sites/ftp.slackware.com/pub/slackware/ ftp://mirrors.unixsol.org/slackware/
- Invocation:
- Following a Stage 3 Run of the Scripts, append 3 Worst Salix Mirrors to badmirrors.txt, using script badm.sh:
- Invocation
Code: Select all
badm.sh 3 salix >> .usm/badmirrors.txt
- Excerpt from badmirrors.txt Appendage:
Code: Select all
#-------- 3 Worst Mirrors --------------------SALIX--------Wed Jul 5 14:48:14 BST 2017 # http://download.salixos.org http://salix.enialis.net http://salix.hostingxtreme.com
- Invocation
- Repeat Step 4.
- Repeat Step 4.
- badmirrors.txt finally ends with
Code: Select all
#-------- 3 Worst Mirrors --------------------SALIX--------Wed Jul 5 14:48:14 BST 2017 # http://download.salixos.org http://salix.enialis.net http://salix.hostingxtreme.com -- #-------- 3 Worst Mirrors --------------------SALIX--------Thu Jul 6 15:12:02 BST 2017 # ftp://ftp.gtlib.gatech.edu/pub/salixos ftp://mirrors.xmission.com/salix http://www.gtlib.gatech.edu/pub/salixos -- #-------- 3 Worst Mirrors --------------------SALIX--------Fri Jul 7 18:53:39 BST 2017 # ftp://mirror.its.dal.ca/salix ftp://ftp.heanet.ie/pub/salix ftp://ftp.cc.uoc.gr/mirrors/linux/salix
- Run of under 40 seconds now a possibility:
Code: Select all
guest@porteus:~$ bash sesm.sh all Cmd Line URLs=0 File (.usm/badmirrors.txt) URLs=93 Merged(Unique) URLs=93 SLACKWARE Testing ftp://ftp.fu-berlin.de/unix/linux/slackware/ 0.947702 Mirrors=152 Processed=43 Remaining=109 Testing ftp://ftp.nluug.nl/vol/4/slackware/ 0.411109 Mirrors=152 Processed=73 Remaining=79 Testing http://z-ftp.wcss.wroc.pl/pub/linux/slackware/ 0.144879 Mirrors=152 Processed=91 Remaining=61 Testing http://mirrors.kingrst.com/slackware/ Fail Mirrors=152 Processed=136 Remaining=16 Done! slackware mirror set to: http://mirror.ovh.net/mirrors/ftp.slackware.com/ SLACKWAREPATCHES Done! slackwarepatches mirror set to: http://mirror.ovh.net/mirrors/ftp.slackware.com/ SLACKY Done! slacky mirror set to: http://repository.slacky.eu/ SALIX Testing ftp://ftp.nluug.nl/pub/os/Linux/distr/salix 0.597785 Mirrors=28 Processed=19 Remaining=9 Done! salix mirror set to: http://ftp.nux.ipb.pt/dists/salix ALIEN Done! alien mirror set to: http://bear.alienbase.nl/mirrors/people/alien/sbrepos PONCE Done! ponce mirror set to: http://ponce.cc/slackware SLACKONLY Done! slackonly mirror set to: http://packages.slackonly.com/pub/packages SBO Done! sbo mirror set to: http://slackbuilds.org/slackbuilds real 0m38.200s user 0m1.760s sys 0m1.247s
- Script badm.sh
Code: Select all
#!/bin/bash # badm.sh Display n worst mirrors in repo by Bogomips 2017-06-23 14:54:17 # Requires raw output of script to set fastest usm mirrors (aka sesm.sh) # Repo defaults to 'slackware' # Arbitrary cutoff points: n<100 & (# of mirrors - n) > 9 # Last Update: 2017-06-27 22:11:32 Fixed length Header of 88 chars. badm () { um="Argument usage:\t< # of Worst Mirrors to Display >\t[Repository(default:slackware)]\nOnly run after script to set fatest usm mirrors (aka sesm.sh) has run." err="Error! " local r q=0; let y=1 n=-1 [[ $1 ]] || { echo -e $um; return 0; } [[ $1 =~ ^[0-9]+$ ]] && let n=$1; [[ n -gt 100 ]] && let r=y; let y++; ((n)) || let r=y; let y++; [[ n -lt 0 ]] && let r=y; let y++; # Non-numeric 1st Argument # Default slackware? # 2nd Argument: Repo (default: slackware) rpo=${2:-slackware}; # let z=y; # Set 2nd Arg error message level # Check the Repository eval $(grep DISTROS= /etc/usm/usm.conf); # Get distros list from usm.conf [[ $DISTROS ]] || { echo Error\! File usm.conf corrupted; return 9; } DISTROS+=" sbo"; # 2017-06-19 23:44:55 [[ $DISTROS =~ ([[:space:]]|^)$rpo([[:space:]]|$) ]] || let q=y; let y++; # Check non-numeric 2nd Arg [[ ! $rpo =~ ^[a-z]+$ && n -lt 0 ]] && q=0; # Check n ag mirror count # Get mirror count if ! ((${r:=q})); then mc=$(sed /Slackware64-current/q /etc/usm/mirrors-$rpo.txt | grep -Ec "^\s*#?\s*(http|ftp)"); # Check if remaining mirros exceed 9 let rm=mc-n; [[ rm -lt 10 ]] && let r=y; let y++; [[ -f /tmp/sesm/$rpo.lst ]] || let r=y; let y++; if ! ((r)); then w=""; for j in {0..79}; do w+=-; done # Dash Line h="#${w:0:8} $n Worst Mirrors ${w:0:0}${rpo^^}${w:0:8}$(date)"; # Get Measure echo -e "#\n#${w:0:8} $n Worst Mirrors ${w:0:88-${#h}}${rpo^^}${w:0:8}$(date)\n#"; cut -d" " -f2- /tmp/sesm/$rpo.lst | sort -k2 | cut -d" " -f1 | tail -n$n return 0; fi fi # Error Message Output ms=("# requested $n > 100" "Requested 0 worst mirrors" "Non-numeric 1st argument: '$1'" "Repo not found: '$rpo'" "Less than 10 remaining mirrors: $mc-$n=$rm" "/tmp/sesm/$rpo.lst not found. Ensure ${rpo^^} has been set thru fastest usm mirrors script (aka sesm.sh)."); ((r)) && echo $err${ms[r-1]} [[ q -gt r ]] && echo $err${ms[q-1]} return ${q:-$r}; } badm $*
Information and a Script to set the Slackware Mirrors for USM
Information
There are 8 mirror files but only 7 distros:
Code: Select all
guest@porteus:~$ grep ^DISTROS /etc/usm/usm.conf
DISTROS="slackware slackwarepatches slacky salix alien ponce slackonly"
Code: Select all
guest@porteus:~$ s="slackware slackwarepatches slacky salix alien ponce slackonly"
guest@porteus:~$ for r in $s; do [[ $(grep -Ec "\s*#*\s*(http|ftp)" /etc/usm/mirrors-$r.txt) -eq 1 ]] && echo $r; done
alien
ponce
slackonly
slackware and slackwarepatches mirrors seem to be almost identical:
Code: Select all
guest@porteus:~$ sed /Slackware64-current/q /etc/usm/mirrors-slackware.txt > s
guest@porteus:~$ sed /Slackware64-current/q /etc/usm/mirrors-slackwarepatches.txt > t
guest@porteus:~$ diff t s
50c50
< # http://mirrors.slackware.com/slackware/
---
> http://mirrors.slackware.com/slackware/
73c73
< http://ftp.slackware-brasil.com.br/
---
> # http://ftp.slackware-brasil.com.br/
Code: Select all
guest@porteus:~$ t="slackware slackwarepatches slacky salix"
guest@porteus:~$ for r in $t; do echo $r=$(sed /Slackware64-current/q /etc/usm/mirrors-$r.txt | grep -Ec "^\s*#*\s*(http|ftp)"); done
slackware=152
slackwarepatches=152
slacky=3
salix=28
Code: Select all
guest@porteus:~$ sudo sed -i '/^\s*http/s//# http/' /etc/usm/mirrors-salix.txt
guest@porteus:~$ sudo sed -i s%^\s*#*\s*ftp://salix%ftp//salix% /etc/usm/mirrors-salix.txt
guest@porteus:~$ grep -E "^\s*(http|ftp)" /etc/usm/mirrors-salix.txt
ftp//salix.mirror.garr.it/mirrors/salix
root@porteus:/home/guest# usm -u salix
Starting salix database update
Downloading: vercheck.txt DONE
Downloading: CHECKSUMS.md5 DONE
Downloading: PACKAGES.TXT DONE
Downloading: LIBS.TXT.gz DONE
Downloading: OPTIFEST.gz DONE
salix database updated.
- Paste into a Terminal at the Prompt:
Code: Select all
eval $(grep DISTROS= /etc/usm/usm.conf); echo -e "\n\033[1mDistro Mirrors\033[0m\n"; for d in $DISTROS; do echo ${d^^}; echo -n Set:; grep -E "^\s*(http|ftp)" /etc/usm/mirrors-$d.txt; echo $(sed /Slackware64-current/q /etc/usm/mirrors-$d.txt | grep -Ec "^\s*#\s*(http|ftp)" ) Commented Out; echo; done
- Current Mirror SettingsRemember to save the output you get, just in case.
Code: Select all
Distro Mirrors SLACKWARE Set:ftp://hpc-mirror.usc.edu/pub/linux/distributions/slackware/ 151 Commented Out SLACKWAREPATCHES Set:ftp://hpc-mirror.usc.edu/pub/linux/distributions/slackware/ 151 Commented Out SLACKY Set:http://slack.isper.sk/pub 2 Commented Out SALIX Set:http://ftp.belnet.be/salixos.org 27 Commented Out ALIEN Set:http://bear.alienbase.nl/mirrors/people/alien/sbrepos 0 Commented Out PONCE Set:http://ponce.cc/slackware 0 Commented Out SLACKONLY Set:http://packages.slackonly.com/pub/packages 0 Commented Out
Advisable to take a backup copy before running the script:
Code: Select all
cp -a /etc/usm .
Script Details
- If there is a problem with sudo, edit out the sudo commands and run script as root, or in a root terminal.
- If launching the script from a terminal, it has to be invoked with bash, otherwise
Code: Select all
guest@porteus:~$ sh sesm.sh sesm.sh: line 63: syntax error near unexpected token `<' sesm.sh: line 63: `readarray -t f < <(sed /Slackware64-current/q /etc/usm/mirrors-$1.txt);'
- Example of Invocation using script fasm.sh to set fastest slackware mirror for a repo
- Create Script
Code: Select all
cp /Path/to/sesm.sh , sed "s/\s*sesm\s*().*/fasm \$*/;T;q" sesm.sh > fasm.sh
- Script Invocation
Code: Select all
guest@porteus:~$ sh fasm.sh slacky fasm.sh: line 63: syntax error near unexpected token `<' fasm.sh: line 63: `readarray -t f < <(sed /Slackware64-current/q /etc/usm/mirrors-$1.txt);' guest@porteus:~$ bash fasm.sh slacky Done! slacky mirror set to: http://slack.isper.sk/pub
- Create Script
- Running script in a terminal presents no problem,
- Script can either be pasted into a terminal at the prompt or sourced:
Code: Select all
source /Path/to/sesm.sh
- Thereafter invoked with the bash function (in the example the function sesm is being timed as well):
Code: Select all
guest@porteus:~$ time sesm all SLACKWARE Testing ftp://ftp.cc.swin.edu.au/slackware/ 3.622269 Mirrors=152 Processed=3 Remaining=149 Testing http://ftp.iinet.net.au/pub/slackware/ 0.824413 Mirrors=152 Processed=6 Remaining=146 Testing http://mirror.as24220.net/pub/slackware/ 4.001222 Mirrors=152 Processed=10 Remaining=142 Testing http://mirror.primusdatacentre.com.au/slackware/ 4.000221 Mirrors=152 Processed=13 Remaining=139 Testing http://ftp.slackware.at/data/ 4.000346 Mirrors=152 Processed=15 Remaining=137 Testing ftp://mirror.datacenter.by/pub/slackware/ 1.198030 Mirrors=152 Processed=18 Remaining=134 Testing ftp://mirror.csclub.uwaterloo.ca/slackware/ 1.680054 Mirrors=152 Processed=24 Remaining=128 Testing ftp://mirrors.ucr.ac.cr/slackware/pub/slackware/ 3.282709 Mirrors=152 Processed=31 Remaining=121 Testing ftp://nephtys.lip6.fr/pub/linux/distributions/slackware/ 2.441583 Mirrors=152 Processed=41 Remaining=111 Testing ftp://ftp.cc.uoc.gr/mirrors/linux/slackware/ 1.892984 Mirrors=152 Processed=52 Remaining=100 Testing http://kambing.ui.ac.id/slackware/ 0.607585 Mirrors=152 Processed=60 Remaining=92 Testing ftp://ftp.nara.wide.ad.jp/pub/Linux/slackware/ 3.999801 Mirrors=152 Processed=66 Remaining=86 Testing ftp://riksun.riken.go.jp/Linux/slackware/ 3.921242 Mirrors=152 Processed=70 Remaining=82 Testing ftp://ftp.man.szczecin.pl/pub/Linux/slackware/ 4.001278 Mirrors=152 Processed=80 Remaining=72 Testing ftp://z-ftp.wcss.wroc.pl/pub/linux/slackware/ 0.695328 Mirrors=152 Processed=90 Remaining=62 Testing http://darkstar.ist.utl.pt/pub/slackware/ 4.001580 Mirrors=152 Processed=93 Remaining=59 Testing http://mirror2.etf.bg.ac.rs/slackware/ 4.000899 Mirrors=152 Processed=98 Remaining=54 Testing ftp://ftp.wa.co.za/pub/slackware/ 2.569787 Mirrors=152 Processed=101 Remaining=51 Testing http://slackware.mirror.ac.za/ 2.291520 Mirrors=152 Processed=104 Remaining=48 Testing ftp://ftp.isu.edu.tw/pub/Linux/Slackware/ 3.999778 Mirrors=152 Processed=111 Remaining=41 Testing http://ftp.twaren.net/Linux/Slackware/ 0.750657 Mirrors=152 Processed=114 Remaining=38 Testing ftp://carroll.aset.psu.edu/pub/linux/distributions/slackware/ 4.001532 Mirrors=152 Processed=123 Remaining=29 Testing ftp://ftp.gtlib.gatech.edu/nv/ao2/lxmirror/ftp.slackware.com/ 1.680286 Mirrors=152 Processed=125 Remaining=27 Testing http://marmot.tn.utexas.edu/slackware/ 0.528524 Mirrors=152 Processed=131 Remaining=21 Testing http://mirrors.kingrst.com/slackware/ 0.456996 Mirrors=152 Processed=136 Remaining=16 Testing http://slackware.mirrorcatalogs.com/ 0.042592 Mirrors=152 Processed=143 Remaining=9 Testing http://slackware.virginmedia.com/ 3.503471 Mirrors=152 Processed=148 Remaining=4 Done! slackware mirror set to: ftp://hpc-mirror.usc.edu/pub/linux/distributions/slackware/ SLACKWAREPATCHES Done! slackwarepatches mirror set to: ftp://hpc-mirror.usc.edu/pub/linux/distributions/slackware/ SLACKY Done! slacky mirror set to: http://slack.isper.sk/pub SALIX Testing http://download.salixos.org Fail Mirrors=28 Processed=11 Remaining=17 Testing ftp://salix.mirror.garr.it/mirrors/salix 1.118754 Mirrors=28 Processed=17 Remaining=11 Testing ftp://mirrors.xmission.com/salix 1.941643 Mirrors=28 Processed=25 Remaining=3 Done! salix mirror set to: http://ftp.belnet.be/salixos.org ALIEN Done! alien mirror set to: http://bear.alienbase.nl/mirrors/people/alien/sbrepos PONCE Done! ponce mirror set to: http://ponce.cc/slackware SLACKONLY Done! slackonly mirror set to: http://packages.slackonly.com/pub/packages real 3m40.491s user 0m3.155s sys 0m1.393s
- Script can either be pasted into a terminal at the prompt or sourced:
Follows in adjacent Post
SUMMARY
<> used to denote fixed format (code) block.
Script to set the Slackware Mirrors for USM Referred to here as sesm.sh, but can be given any convenient name of one's choosing:<>
sesm invoked with argument all, If 1st argument not all, this will be taken as request to show usage parameters. If problem with sudo, edit out sudo commands, then run as root. Also, to run the script in a shell, the bash shell has to be invoked:<>
Prior to Running Script recommend:
- Back Up<>
- List current status of all mirrors
- Status of All Mirrors script sam.sh:<>
- Status Output (might pay to keep a note of current settings):<>
Fastest Mirror for Single Repository
- There is also the option of independently setting the fastest mirror for a repository thru script fasm.sh: <>
- Again, has to be invoked via bash shell:<>
- Listings
- List All Mirrors of Repository
- <>
- <>
- Check the Repository Mirror
- Script crm.sh<>
- <>
- List All Mirrors of Repository
- Timintgs
- Directories for Timings Files. Directory holding raw output files will be made in /tmp. Name of directory dictated by function to find fastest mirror of a repository, fasm. If invoked by
- sesm files will be created in /tmp/sesm
- the shell, as a script, calling function is main, files created in /tmp/main
- none, and run directly as a function in terminal, files will be created under /tmp/fasm.
- Raw Output. These files all named <repository>.lst:<>
- Output Refined<>
- Directories for Timings Files. Directory holding raw output files will be made in /tmp. Name of directory dictated by function to find fastest mirror of a repository, fasm. If invoked by
- Made minutes apart<>
- Made minutes apart<>
- Made several hours later:<>
- masm.sh<>
- Invocation<>