Basic usage of wget command under Linux

Preface

Linux wget is a tool for downloading files, which is used in the command line. It is an indispensable tool for Linux users, especially for network administrators, who often need to download some software or restore backups from remote servers to local servers.

If we use a virtual host, to handle such transactions we can only download it from the remote server to our computer disk first, and then upload it to the server using the FTP tool. This is a waste of time and energy, there is nothing you can do about it.

With Linux VPS, it can be downloaded directly to the server without going through the uploading step. The wget tool is small in size but has complete functions. It supports breakpoint downloading, FTP and HTTP downloading methods, proxy servers and is easy to set up.

Below we will explain how to use wget in the form of an example.

1. Download a single file using wget

The following example downloads a file from the network and saves it in the current directory

wget http://cn.wordpress.org/wordpress-3.1-zh_CN.zip

During the download process, a progress bar will be displayed, including (download completion percentage, bytes downloaded, current download speed, and remaining download time).

2. Download using wget -O and save with a different file name

By default, wget will use the last character following the "/" as the command, and the file name for downloading dynamic links will usually be incorrect.

Error: The following example will download a file and save it with the name download.php?id=1080

wget http://www.centos.bz/download?id=1

Even though the downloaded file is in zip format, it still starts with the command download.php?id=1080 .

Correct: To fix this, we can use the -O parameter to specify a file name:

wget -O wordpress.zip http://www.centos.bz/download.php?id=1080

3. Use wget –limit -rate to limit the download speed

When you execute wget, it will use all possible bandwidth to download by default. But when you are going to download a large file and you also need to download other files, it is necessary to limit the speed.

wget –limit-rate=300k http://cn.wordpress.org/wordpress-3.1-zh_CN.zip

4. Use wget -c to resume downloading

Use wget -c to restart an interrupted download:

wget -c http://cn.wordpress.org/wordpress-3.1-zh_CN.zip

It is very helpful for us to download large files and suddenly be interrupted due to network or other reasons. We can continue downloading instead of downloading a file again. You can use the -c parameter to continue an interrupted download.

5. Use wget -b to download in the background

When downloading very large files, we can use the parameter -b to perform background downloading.

wget -b http://cn.wordpress.org/wordpress-3.1-zh_CN.zip

Continuing in background, pid 1840.

Output will be written to `wget-log'.

You can use the following command to check the download progress:

tail -f wget-log

6. Download by camouflaged proxy name

Some websites can deny your download request by judging that the proxy name is not a browser. However, you can disguise it by using the --user-agent parameter.

wget –user-agent="Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16" Download link

7. Use wget –spider to test the download link

When you plan to schedule downloads, you should test whether the download link is valid at the scheduled time. We can add the --spider parameter to check.

wget –spider URL

If the download link is correct, it will be displayed

wget –spider URL
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled — not retrieving.

This ensures that the download will be carried out at the scheduled time, but when you give a wrong link, it will display the following error

wget –spider url
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response… 404 Not Found
Remote file does not exist — broken link!!!

You can use spider parameters in the following situations:

Check before scheduled download

Interval detection of website availability

Check website pages for dead links

8. Use wget –tries to increase the number of retries

It may also fail if there is a problem with the network or if the download is a large file. By default, wget retries 20 times to connect and download files. If needed, you can increase the number of retries using --tries.

wget –tries=40 URL

9. Download multiple files using wget -i

First, save a download link file

cat > filelist.txt
url1
url2
url3
url4

Then use this file and parameter -i to download

wget -i filelist.txt

10. Use wget –mirror to mirror the website

The following example downloads the entire website to local computer.

wget –mirror -p –convert-links -P ./LOCAL URL

–miror: open an account mirror download -p: download all files for normal display of HTML pages –convert-links: after downloading, convert to local links -P ./LOCAL: save all files and directories to the local specified directory

11. Use wget –reject to filter the specified format download

You want to download a website but you don’t want to download images, you can use the following command.

wget –reject=gif url

12. Use wget -o to save download information into a log file

If you don't want the download information to be displayed directly in the terminal but in a log file, you can use the following command:

wget -o download.log URL

13. Use wget -Q to limit the total download file size

When the file you want to download exceeds 5M and you want to exit the download, you can use the following command:

wget -Q5m -i filelist.txt

Note: This parameter does not work for single file downloads, it only works for recursive downloads.

14. Use wget -r -A to download files in the specified format

This feature can be used in the following situations

Download all images from a website

Download all videos from a website

Download all PDF files from a website

wget -r -A.pdf url

15. Download via FTP using wget

You can use wget to download ftp links.

匿名FTP download using wget

wget ftp-url

FTP download using wget username and password authentication

wget –ftp-user=USERNAME –ftp-password=PASSWORD url

This is the end of this article about the basic usage of wget command under linu. For more relevant linu wget command content, please search 123WORDPRESS.COM's previous articles or continue to browse the following related articles. I hope everyone will support 123WORDPRESS.COM in the future!