Last year, due to project needs, I wrote a crawler in Python. The crawled data needs to be stored in the PG database of the production environment. Therefore, you need to deploy the script to the CentOS server and set up a scheduled task to automatically start the script. The implementation steps are as follows: 1. Install pip (the operating system comes with python2.6 which can be used directly, but there is no pip) # Download pip installation package wget "https://pypi.python.org/packages/source/p/pip/pip-1.5.4.tar.gz#md5=834b2904f92d46aaa333267fb1c922bb" --no-check-certificate # Unzip the installation package and install tar -xzvf pip-1.5.4.tar.gz cd pip-1.5.4 python setup.py install 2. Install third-party libraries with pip pip install PyGreSQL==5.0.3 pip install requests==2.18.3 3. Set up scheduled tasks # Start the scheduled task service service crond start # View the scheduled task service status service crond status # Open the scheduled task editing window crontab -e # Add two scheduled tasks, which will be executed once at 0:00 and 12:20 every day, and written to the log 0 0 * * * /usr/bin/python /home/longrise/psrd/collect.py > /home/longrise/psrd/collect.log 2>&1 & 20 12 * * * /usr/bin/python /home/longrise/psrd/collect.py > /home/longrise/psrd/collect.log 2>&1 & The syntax of scheduled tasks is as follows: # For details see man 4 crontabs # Example of job definition: # .---------------- minute (0 - 59) # | .------------- hour (0 - 23) # | | .---------- day of month (1 - 31) # | | | .------- month (1 - 12) OR jan,feb,mar,apr ... # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat # | | | | | # * * * * * user-name command to be executed The above method of deploying python crawler scripts in Linux and setting scheduled tasks is all the content that the editor shares with you. I hope it can give you a reference. I also hope that you will support 123WORDPRESS.COM. You may also be interested in:
|
<<: Detailed explanation of JS WebSocket disconnection reasons and heartbeat mechanism
>>: Detailed explanation of MySQL user rights verification and management methods
When using the docker-maven-plugin plug-in, Maven...
As a popular open source database management syst...
Today, due to project requirements, js is needed t...
IIS7 needs to confirm whether the "URL REWRI...
Table of contents 1. Several syntaxes of Insert 1...
Achieve results Implementation Code <h1>123...
Anaconda refers to an open source Python distribu...
Starting and shutting down Tomcat under Linux In ...
I recently watched Rich Harris's <Rethinki...
The final result is like this, isn’t it cute… PS:...
In order to provide high availability of the netw...
Copy code The code is as follows: <html xmlns=...
COALESCE is a function that refers to each parame...
1. Command Introduction The watch command execute...
Preface When we were writing the horse, I guess e...