wget

Non release banter
Jack
Contributor
Contributor
Posts: 1185
Joined: 09 Aug 2013, 14:25
Distribution: Porteus 3.2.rc5 Mate 64 bit
Location: Marysville, OHIO USA

wget

Post#1 by Jack » 06 Nov 2017, 04:25

Anyone have a script that use wget to download a lot of files at one time from the same place?
I just like Slackware because I think it teach you about Linux to build packages where Ubuntu is like Windows you just install programs you want.

donald
Full of knowledge
Full of knowledge
Posts: 1219
Joined: 17 Jun 2013, 13:17
Distribution: Porteus 3.2.2 XFCE 32bit
Location: Germany

wget

Post#2 by donald » 06 Nov 2017, 05:47

Hi Jack

you could try

Code: Select all

wget -r -l1 -A.tgz http://whatever.com/directory/
In this example, -r and -l1 options enable 1-level deep recursive retrieval,
and "-A" option specifies file name suffix to accept during recursive download.
(tgz in this case)

EDIT
If needed add the -np, --no-parent switch = don't ascend to the parent directory

burdi01
Black ninja
Black ninja
Posts: 64
Joined: 18 Aug 2013, 12:09
Distribution: Slackware, Xubuntu
Location: The Netherlands

wget

Post#3 by burdi01 » 06 Nov 2017, 10:10

In a 1-line script I have:

Code: Select all

wget -m -e robots=off --no-parent --reject="index.html*" "$@"
where the "$@" represents the "http://<dir>" or "https://<dir>" invocation parameter(s).
:D

User avatar
brokenman
Site Admin
Site Admin
Posts: 5574
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

wget

Post#4 by brokenman » 06 Nov 2017, 15:02

download a lot of files at one time from the same place?
What place? What files?
How do i become super user?
Wear your underpants on the outside and put on a cape.

Jack
Contributor
Contributor
Posts: 1185
Joined: 09 Aug 2013, 14:25
Distribution: Porteus 3.2.rc5 Mate 64 bit
Location: Marysville, OHIO USA

wget

Post#5 by Jack » 06 Nov 2017, 15:30

brokenman wrote:
06 Nov 2017, 15:02
download a lot of files at one time from the same place?
What place? What files?
Your place and your custom files
Last edited by Jack on 06 Nov 2017, 15:46, edited 2 times in total.
I just like Slackware because I think it teach you about Linux to build packages where Ubuntu is like Windows you just install programs you want.

User avatar
brokenman
Site Admin
Site Admin
Posts: 5574
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

wget

Post#6 by brokenman » 06 Nov 2017, 15:37

Just for variation.

Code: Select all

for a in $(lynx -dump --listonly --nonumbers http://brokenman.porteus.org/current/x86_64/custom | egrep "txz|tgz|xzm"); do
  wget $a
done
Burdi01's line above should also do it nicely.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Ed_P
Contributor
Contributor
Posts: 3320
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 3.2.2 64-bit ISO
Location: Western NY, USA

wget

Post#7 by Ed_P » 06 Nov 2017, 15:53

:%) Why download all of them rather than just the ones you want/need??


-update-

And now I know. :o
Jack wrote:
06 Nov 2017, 19:28
here is the link. http://slackware.uk/msb/current/1.19/x86_64/ The directory are base, deps and extra just the txz files.
Ed

donald
Full of knowledge
Full of knowledge
Posts: 1219
Joined: 17 Jun 2013, 13:17
Distribution: Porteus 3.2.2 XFCE 32bit
Location: Germany

wget

Post#8 by donald » 06 Nov 2017, 17:04

Jack wrote:
06 Nov 2017, 15:30
brokenman wrote:
06 Nov 2017, 15:02
download a lot of files at one time from the same place?
What place? What files?
Your place and your custom files

Code: Select all

wget -r -l1 -A.tgz,txz,xzm http://brokenman.porteus.org/current/x86_64/custom/
does exactly that

Jack
Contributor
Contributor
Posts: 1185
Joined: 09 Aug 2013, 14:25
Distribution: Porteus 3.2.rc5 Mate 64 bit
Location: Marysville, OHIO USA

wget

Post#9 by Jack » 06 Nov 2017, 18:26

Donald, I did what you first posted and it work great. And I did it before burdi01 and brokenman posted but I will try them. Now I see you posted another way to do it.
I just like Slackware because I think it teach you about Linux to build packages where Ubuntu is like Windows you just install programs you want.

donald
Full of knowledge
Full of knowledge
Posts: 1219
Joined: 17 Jun 2013, 13:17
Distribution: Porteus 3.2.2 XFCE 32bit
Location: Germany

wget

Post#10 by donald » 06 Nov 2017, 18:49

Jack wrote:
06 Nov 2017, 18:26
Now I see you posted another way to do it.
It's basically the same.
I only posted the complete line now, so copy/paste is all one has to do.

Jack
Contributor
Contributor
Posts: 1185
Joined: 09 Aug 2013, 14:25
Distribution: Porteus 3.2.rc5 Mate 64 bit
Location: Marysville, OHIO USA

wget

Post#11 by Jack » 06 Nov 2017, 19:28

Well I did a test on all the ways posted and they all work well so I will have to pick one.

I like to be able to download what in these directory here is the link. http://slackware.uk/msb/current/1.19/x86_64/ The directory are base, deps and extra just the txz files.
I just like Slackware because I think it teach you about Linux to build packages where Ubuntu is like Windows you just install programs you want.

Jack
Contributor
Contributor
Posts: 1185
Joined: 09 Aug 2013, 14:25
Distribution: Porteus 3.2.rc5 Mate 64 bit
Location: Marysville, OHIO USA

wget

Post#12 by Jack » 06 Nov 2017, 22:20

Ed_P wrote:
06 Nov 2017, 15:53
:%) Why download all of them rather than just the ones you want/need??


-update-

And now I know. :o
Jack wrote:
06 Nov 2017, 19:28
here is the link. http://slackware.uk/msb/current/1.19/x86_64/ The directory are base, deps and extra just the txz files.
I want to do these in one line if it can be done. Or a script will do to. I don't like to download one file at a time.

Mate Files
wget -r -l1 -A.txz http://slackware.uk/msb/current/1.19/x86_64/base
wget -r -l1 -A.txz http://slackware.uk/msb/current/1.19/x86_64/deps
wget -r -l1 -A.txz http://slackware.uk/msb/current/1.19/x86_64/extra
I just like Slackware because I think it teach you about Linux to build packages where Ubuntu is like Windows you just install programs you want.

donald
Full of knowledge
Full of knowledge
Posts: 1219
Joined: 17 Jun 2013, 13:17
Distribution: Porteus 3.2.2 XFCE 32bit
Location: Germany

wget

Post#13 by donald » 07 Nov 2017, 02:00

:)
I do use a "list.txt" file then.
A simple txt file containing the URL's

Code: Select all

http://slackware.uk/msb/current/1.19/x86_64/base
http://slackware.uk/msb/current/1.19/x86_64/deps
http://slackware.uk/msb/current/1.19/x86_64/extra
and passed it on to wget

Code: Select all

guest@porteus:~/Downloads$ wget -r -l1 -A.txz -i ./list.txt

Jack
Contributor
Contributor
Posts: 1185
Joined: 09 Aug 2013, 14:25
Distribution: Porteus 3.2.rc5 Mate 64 bit
Location: Marysville, OHIO USA

wget

Post#14 by Jack » 07 Nov 2017, 02:09

Here is the output something must be missing? Where do I get list.txt at?

Code: Select all

guest@porteus:~/Downloads$ wget -r -l1 -A.txz -i ./list.txt
./list.txt: No such file or directory
No URLs found in ./list.txt.
guest@porteus:~/Downloads$
I just like Slackware because I think it teach you about Linux to build packages where Ubuntu is like Windows you just install programs you want.

donald
Full of knowledge
Full of knowledge
Posts: 1219
Joined: 17 Jun 2013, 13:17
Distribution: Porteus 3.2.2 XFCE 32bit
Location: Germany

wget

Post#15 by donald » 07 Nov 2017, 02:15

the list.txt file must be in the same directory where you run wget
(Downloads in this case)

Post Reply