Page 1 of 3

wget

Posted: 06 Nov 2017, 04:25
by Jack
Anyone have a script that use wget to download a lot of files at one time from the same place?

wget

Posted: 06 Nov 2017, 05:47
by donald
Hi Jack

you could try

Code: Select all

wget -r -l1 -A.tgz http://whatever.com/directory/
In this example, -r and -l1 options enable 1-level deep recursive retrieval,
and "-A" option specifies file name suffix to accept during recursive download.
(tgz in this case)

EDIT
If needed add the -np, --no-parent switch = don't ascend to the parent directory

wget

Posted: 06 Nov 2017, 10:10
by burdi01
In a 1-line script I have:

Code: Select all

wget -m -e robots=off --no-parent --reject="index.html*" "$@"
where the "$@" represents the "http://<dir>" or "https://<dir>" invocation parameter(s).
:D

wget

Posted: 06 Nov 2017, 15:02
by brokenman
download a lot of files at one time from the same place?
What place? What files?

wget

Posted: 06 Nov 2017, 15:30
by Jack
brokenman wrote:
06 Nov 2017, 15:02
download a lot of files at one time from the same place?
What place? What files?
Your place and your custom files

wget

Posted: 06 Nov 2017, 15:37
by brokenman
Just for variation.

Code: Select all

for a in $(lynx -dump --listonly --nonumbers http://brokenman.porteus.org/current/x86_64/custom | egrep "txz|tgz|xzm"); do
  wget $a
done
Burdi01's line above should also do it nicely.

wget

Posted: 06 Nov 2017, 15:53
by Ed_P
:%) Why download all of them rather than just the ones you want/need??


-update-

And now I know. :o
Jack wrote:
06 Nov 2017, 19:28
here is the link. http://slackware.uk/msb/current/1.19/x86_64/ The directory are base, deps and extra just the txz files.

wget

Posted: 06 Nov 2017, 17:04
by donald
Jack wrote:
06 Nov 2017, 15:30
brokenman wrote:
06 Nov 2017, 15:02
download a lot of files at one time from the same place?
What place? What files?
Your place and your custom files

Code: Select all

wget -r -l1 -A.tgz,txz,xzm http://brokenman.porteus.org/current/x86_64/custom/
does exactly that

wget

Posted: 06 Nov 2017, 18:26
by Jack
Donald, I did what you first posted and it work great. And I did it before burdi01 and brokenman posted but I will try them. Now I see you posted another way to do it.

wget

Posted: 06 Nov 2017, 18:49
by donald
Jack wrote:
06 Nov 2017, 18:26
Now I see you posted another way to do it.
It's basically the same.
I only posted the complete line now, so copy/paste is all one has to do.

wget

Posted: 06 Nov 2017, 19:28
by Jack
Well I did a test on all the ways posted and they all work well so I will have to pick one.

I like to be able to download what in these directory here is the link. http://slackware.uk/msb/current/1.19/x86_64/ The directory are base, deps and extra just the txz files.

wget

Posted: 06 Nov 2017, 22:20
by Jack
Ed_P wrote:
06 Nov 2017, 15:53
:%) Why download all of them rather than just the ones you want/need??


-update-

And now I know. :o
Jack wrote:
06 Nov 2017, 19:28
here is the link. http://slackware.uk/msb/current/1.19/x86_64/ The directory are base, deps and extra just the txz files.
I want to do these in one line if it can be done. Or a script will do to. I don't like to download one file at a time.

Mate Files
wget -r -l1 -A.txz http://slackware.uk/msb/current/1.19/x86_64/base
wget -r -l1 -A.txz http://slackware.uk/msb/current/1.19/x86_64/deps
wget -r -l1 -A.txz http://slackware.uk/msb/current/1.19/x86_64/extra

wget

Posted: 07 Nov 2017, 02:00
by donald
:)
I do use a "list.txt" file then.
A simple txt file containing the URL's

Code: Select all

http://slackware.uk/msb/current/1.19/x86_64/base
http://slackware.uk/msb/current/1.19/x86_64/deps
http://slackware.uk/msb/current/1.19/x86_64/extra
and passed it on to wget

Code: Select all

guest@porteus:~/Downloads$ wget -r -l1 -A.txz -i ./list.txt

wget

Posted: 07 Nov 2017, 02:09
by Jack
Here is the output something must be missing? Where do I get list.txt at?

Code: Select all

guest@porteus:~/Downloads$ wget -r -l1 -A.txz -i ./list.txt
./list.txt: No such file or directory
No URLs found in ./list.txt.
guest@porteus:~/Downloads$

wget

Posted: 07 Nov 2017, 02:15
by donald
the list.txt file must be in the same directory where you run wget
(Downloads in this case)