PrefaceI need to deploy the selenium crawler program to the Linux server for work. I would like to share this with you. If you are interested, you can take a look. 1. What is selenium?Selenium is a tool used for web application testing. Selenium tests run directly in the browser, just like real users are operating, and crawlers use it to crawl some data dynamically loaded by js 2. Usage steps1. Import libraryThe code is as follows from selenium.webdriver import Chrome from selenium.webdriver.chrome.service import Service from selenium.webdriver.chrome.options import Options # Use a headless browser from selenium.webdriver import ChromeOptions chrome_options = Options() options = ChromeOptions() options.add_experimental_option('excludeSwitches', ['enable-automation']) # => Remove the browser being controlled by the automated testing software options.add_experimental_option('useAutomationExtension', False) chrome_options.add_argument("--headless") # => Configure headless mode for Chrome chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-gpu') chrome_options.add_argument('--disable-dev-shm-usage') 2. Test codeThe code is as follows: s = Service(r"/home/driver/chromedriver") driver = Chrome( service=s, options=chrome_options ) driver.get("https://www.baidu.com") print(diiver.title) 3. Deployment Procedure1. Install ChromeThe command is as follows: yum install https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm Check the version of Chrome: google-chrome --version 2. Install chromedriverThe command is as follows: Download the chromedriver driver address according to the corresponding chrome version: https://npm.taobao.org/mirrors/chromedriver My version number is: 96.0.4664.45 wget https://npm.taobao.org/mirrors/chromedriver/96.0.4664.45/chromedriver_linux64.zip yum install -y unzip zip unzip chromedriver_linux64.zip # Unzip the zip file mkdir driver #Create a new folder to store the driver chmod 777 driver/chromedriver # This is the permission. I give it 777 here 3. Run the test codeCreate a new test.py file vi test.py Save test.py and run it. Seeing this, my request is successful. SummarizeThis is the end of this article about deploying selenium crawler program under Linux system. For more relevant content about linux selenium crawler program, please search previous articles of 123WORDPRESS.COM or continue to browse the related articles below. I hope you will support 123WORDPRESS.COM in the future! You may also be interested in:
|
<<: The difference between animation and transition
Table of contents 1. Introduction to platform bus...
1. display:box; Setting this property on an eleme...
Preface The electricity in my residence has been ...
YSlow is a page scoring plug-in developed by Yaho...
The default database of CentOS7 is mariadb, but m...
1.# # represents a location in a web page. The ch...
Table of contents Single thread asynchronous Sing...
Table of contents 1. Installation and introductio...
Docker is really cool, especially because it'...
The MySQL development team officially released th...
Table of contents 1. Solution 2. MySQL character ...
CSS naming rules header: header Content: content/c...
CentOS 8 is officially released! CentOS fully com...
Question: Although the index has been created, wh...
Note: All pictures in this article are collected ...