A detailed introduction to wget command in Linux

[root@network test]# yum install -y wget

[root@network test]# wget --help

GNU Wget 1.14, a non-interactive network file download tool.
Usage: wget [options]... [URL]...

Required arguments for long options are also required when using short options.

start up:
  -V, --version Display Wget version information and exit.
  -h, --help Print this help.
  -b, --background Go to background after startup.
  -e, --execute=COMMAND Run a ".wgetrc" style command.

Log and input files:
  -o, --output-file=FILE Write log information to FILE.
  -a, --append-output=FILE Append information to FILE.
  -d, --debug Print extensive debugging information.
  -q, --quiet Quiet mode (no output).
  -v, --verbose Verbose output (this is the default).
  -nv, --no-verbose Turn off verbose output, but do not enter quiet mode.
       --report-speed=TYPE Output bandwidth as TYPE. TYPE can be bits.
  -i, --input-file=FILE Download URLs from local or external FILE.
  -F, --force-html Treat input files as HTML files.
  -B, --base=URL Parse HTML input files relative to URLs (specified by the -i -F options).
       --config=FILE Specify config file to use.

download:
  -t, --tries=NUMBER Set the number of retries to NUMBER (0 means unlimited).
       --retry-connrefused Retry even if the connection is refused.
  -O, --output-document=FILE Write document to FILE.
  -nc, --no-clobber skip downloads that would download to
                                 existing files (overwriting them).
  -c, --continue Resume downloading files.
       --progress=TYPE Select the progress bar type.
  -N, --timestamping Only retrieve files that are newer than local files.
  --no-use-server-timestamps Do not use server timestamps to set local files.
  -S, --server-response Print server response.
       --spider Do not download any files.
  -T, --timeout=SECONDS Set all timeouts to SECONDS seconds.
       --dns-timeout=SECS Set DNS lookup timeout to SECS seconds.
       --connect-timeout=SECS Set the connection timeout to SECS seconds.
       --read-timeout=SECS Set read timeout to SECS seconds.
  -w, --wait=SECONDS Wait for SECONDS seconds.
       --waitretry=SECONDS Wait 1..SECONDS seconds between retries to get the file.
       --random-wait When getting multiple files, the random waiting interval is 0.5*WAIT...1.5*WAIT seconds each time.
       --no-proxy Disable the use of a proxy.
  -Q, --quota=NUMBER Set the get quota to NUMBER bytes.
       --bind-address=ADDRESS Bind to ADDRESS (hostname or IP) on the local host.
       --limit-rate=RATE Limit download rate to RATE.
       --no-dns-cache Disable DNS lookup caching.
       --restrict-file-names=OS Limit the characters in file names to those allowed by the OS.
       --ignore-case Ignore case when matching files/directories.
  -4, --inet4-only Connect to IPv4 addresses only.
  -6, --inet6-only Connect to IPv6 addresses only.
       --prefer-family=FAMILY First connect to addresses of the specified protocol FAMILY can be IPv6, IPv4 or none.
       --user=USER Set the username for both ftp and http to USER.
       --password=PASS Set the password for both ftp and http to PASS.
       --ask-password Prompt for a password.
       --no-iri disable IRI support.
       --local-encoding=ENC IRI (Internationalized Resource Identifier) ​​Use ENC as the local encoding.
       --remote-encoding=ENC Use ENC as default remote encoding.
       --unlink remove file before clobber.

Table of contents:
  -nd, --no-directories Do not create directories.
  -x, --force-directories Force creation of directories.
  -nH, --no-host-directories Do not create home directories.
       --protocol-directories Use protocol names in directories.
  -P, --directory-prefix=PREFIX Save files as PREFIX/... --cut-dirs=NUMBER Ignore NUMBER directory levels in the remote directory.

HTTP options:
       --http-user=USER Set http user name to USER.
       --http-password=PASS Set http password to PASS.
       --no-cache Do not cache data on the server.
       --default-page=NAME Change the default page (the default page is usually "index.html").
  -E, --adjust-extension Save HTML/CSS documents with appropriate extensions.
       --ignore-length Ignore the 'Content-Length' header field.
       --header=STRING Insert STRING in the header.
       --max-redirect Maximum redirects allowed per page.
       --proxy-user=USER Use USER as the proxy username.
       --proxy-password=PASS Use PASS as proxy password.
       --referer=URL Include 'Referer: URL' in the HTTP request header.
       --save-headers Save HTTP headers to file.
  -U, --user-agent=AGENT Identify as AGENT instead of Wget/VERSION.
       --no-http-keep-alive Disable HTTP keep-alive (persistent connections).
       --no-cookies Do not use cookies.
       --load-cookies=FILE Load cookies from FILE before the session starts.
       --save-cookies=FILE Save cookies to FILE after the session ends.
       --keep-session-cookies Load and save session (non-persistent) cookies.
       --post-data=STRING Use POST method; send STRING as data.
       --post-file=FILE Use POST method; send FILE content.
       --content-disposition Allow Content-Disposition header when local filename is selected (experimental).
       --content-on-error outputs the received content on server errors.
       --auth-no-challenge Send basic HTTP authentication without a first wait server challenge.

HTTPS (SSL/TLS) options:
       --secure-protocol=PR choose secure protocol, one of auto, SSLv2,
                                SSLv3, TLSv1, TLSv1_1 and TLSv1_2.
       --no-check-certificate Do not verify the server's certificate.
       --certificate=FILE Client certificate file.
       --certificate-type=TYPE Client certificate type, PEM or DER.
       --private-key=FILE Private key file.
       --private-key-type=TYPE Private key file type, PEM or DER.
       --ca-certificate=FILE File with a set of CA certificates.
       --ca-directory=DIR Directory to store the list of hashes of CA certificates.
       --random-file=FILE File with random data for generating the SSL PRNG.
       --egd-file=FILE Name of the file for the EGD socket with random data.

FTP options:
       --ftp-user=USER Set the ftp user name to USER.
       --ftp-password=PASS Set ftp password to PASS.
       --no-remove-listing Do not remove '.listing' files.
       --no-glob Do not use wildcard expansion in FTP file names.
       --no-passive-ftp Disable "passive" transfer mode.
       --preserve-permissions Preserve permissions on remote files.
       --retr-symlinks When recursing directories, get linked files (not directories).

WARC options:
       --warc-file=FILENAME save request/response data to a .warc.gz file.
       --warc-header=STRING insert STRING into the warcinfo record.
       --warc-max-size=NUMBER set maximum size of WARC files to NUMBER.
       --warc-cdx write CDX index files.
       --warc-dedup=FILENAME do not store records listed in this CDX file.
       --no-warc-compression do not compress WARC files with GZIP.
       --no-warc-digests do not calculate SHA1 digests.
       --no-warc-keep-log do not store the log file in a WARC record.
       --warc-tempdir=DIRECTORY location for temporary files created by the
                                 WARC writer.

Recursive download:
  -r, --recursive specifies recursive download.
  -l, --level=NUMBER Maximum recursion depth (inf or 0 means unlimited, i.e. download everything).
       --delete-after Delete local files after downloading is complete.
  -k, --convert-links Make links in downloaded HTML or CSS point to local files.
  --backups=N before writing file X, rotate up to N backup files.
  -K, --backup-converted Back up file X as X.orig before converting it.
  Short form of -m, --mirror -N -r -l inf --no-remove-listing.
  -p, --page-requisites Download all elements such as images used to display HTML pages.
       --strict-comments Process HTML comments in strict mode (SGML).

Recursive accept/reject:
  -A, --accept=LIST Comma separated list of acceptable extensions.
  -R, --reject=LIST Comma-separated list of extensions to reject.
       --accept-regex=REGEX regex matching accepted URLs.
       --reject-regex=REGEX regex matching rejected URLs.
       --regex-type=TYPE regex type (posix|pcre).
  -D, --domains=LIST Comma separated list of acceptable domains.
       --exclude-domains=LIST Comma-separated list of domains to exclude.
       --follow-ftp Follow FTP links in HTML documents.
       --follow-tags=LIST Comma separated list of HTML tags to follow.
       --ignore-tags=LIST Comma separated list of HTML tags to ignore.
  -H, --span-hosts Spread over external hosts when recursing.
  -L, --relative Follow only relative links.
  -I, --include-directories=LIST List of allowed directories.
  --trust-server-names use the name specified by the redirection
                                   url last component.
  -X, --exclude-directories=LIST exclude list of directories.
  -np, --no-parent Do not trace back to parent directories.

wget http://cn.wordpress.org/wordpress-4.9.4-zh_CN.tar.gz

[root@network test]# wget https://cn.wordpress.org/wordpress-4.9.4-zh_CN.tar.gz
[root@network test]# ls
wordpress-4.9.4-zh_CN.tar.gz

wget -O wordpress.tar.gz http://cn.wordpress.org/wordpress-4.9.4-zh_CN.tar.gz
wordpress.tar.gz

wget -c https://cn.wordpress.org/wordpress-4.9.4-zh_CN.tar.gz

[root@network test]# wget -b https://cn.wordpress.org/wordpress-4.9.4-zh_CN.tar.gz
Continue to run in the background with pid 1463.
Will write output to 'wget-log'.

[root@network test]# tail -f wget-log
  8550K .......... .......... .......... .......... .......... 96% 814K 0s
  8600K .......... .......... .......... .......... .......... 97% 9.53M 0s
  8650K .......... .......... .......... .......... .......... 98% 86.8M 0s
  8700K .......... .......... .......... .......... .......... 98% 145M 0s
  8750K .......... .......... .......... .......... .......... 99% 67.4M 0s
  8800K .......... .......... .......... .......... .......... 99% 107M 0s
  8850K .......... ......... 100% 1.95M=16s

2018-11-10 15:39:07 (564 KB/s) - Saved "wordpress-4.9.4-zh_CN.tar.gz.2" [9082696/9082696])

wget –spider URL
If the download link is correct, wget –spider URL will be displayed.
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled — not retrieving.
This ensures that the download will be carried out at the scheduled time, but when you give a wrong link, it will display the following error wget –spider url
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response… 404 Not Found
Remote file does not exist — broken link!!!

wget –tries=40 URL

First, save a download link file cat > filelist.txt
url1
url2
url3
url4
Then use this file and parameter -i to download wget -i filelist.txt

wget --mirror --convert-links --adjust-extension --page-requisites 
--no-parent http://example.org

wget -mkEpnp http://example.org

wget –reject=gif url

wget -o download.log URL

wget -O wordpress.zip http://www.minjieren.com/download.aspx?id=1080 

wget -b http://www.minjieren.com/wordpress-3.1-zh_CN.zip

[root@localhost ~]# wget --spider www.baidu.com #Do not download any files 

[root@localhost ~]# wget -S www.baidu.com # Print server response 

[root@localhost ~]# wget -r --tries=2 www.baidu.com (specify 2 attempts, no more attempts after 2 attempts)
[root@localhost ~]# wget -r --tries=2 -q www.baidu.com (specify attempts and do not print intermediate results)