VirtualBox CentOS7.7.1908 Python3.8 build Scrapy development environment [graphic tutorial]

environment

VirtualBox_v6.0.14 CentOS_v7.1908 Python_v3.8.0 MongoDB_v3.2.22 pip_v19.3.1 windows Pycharm-2019.2.3 Enterprise Edition (Community Edition does not support remote connection to Linux development), need to activate the system's built-in Python2.7.5 without processing, tried to install 3.8 with pyenv, but when installing scrapy, errors kept coming up, and finally uninstalled pyenv and recompiled and installed manually.

Install CentOS

In order to ensure that the virtual machine and the host machine can access each other and communicate normally, a 2-virtual network card configuration is adopted.

In the VirtualBox menu, go to Settings->Network, enable two virtual network cards ( virtual machines are enabled, as shown in the following 2 pictures ), set the connection mode of network card 1 to "Network Address Translation NAT" to ensure that the virtual machine can access the external network and the host machine, and set the connection mode of network card 2 to "Bridged Network Card" to ensure that the host machine can access the virtual machine.

Install CentOS7 on the VirtualBox virtual machine. During the installation process, it is best to configure and enable 2 network cards ( CentOS system enabled ).

The installation process is omitted.

After starting the virtual machine, use # ip addr to view the CentOS system network card usage and IP allocation. enp0s3 is the NAT network card information, and the default IP is 10.0.2.15; enp0s8 is the network card information in bridge mode, and the default assigned IP is in the same network segment as the host machine.

If you cannot see the IP information of a certain network card, it may be that the network card was not enabled during system installation. You can modify the ONBOOT parameter to yes in /etc/sysconfig/network-scripts/ifcfg-enp0s3 and /etc/sysconfig/network-scripts/ifcfg-enp0s8 , and then restart the network service # systemctl start network.service or # service network restart .

After the system is installed using the DVD image, the ssh service is available by default. If the host machine uses ssh to connect to the virtual machine, you need to set up the firewall to open port 22.

# firewall-cmd --zone=public --add-port=22/tcp --permanent # Firewall opens port 22 # firewall-cmd --reload #Reload firewall configuration # firewall-cmd --zone=public --list-port #View open ports

At this point, you can use the Windows client tool SSH to remotely connect to the virtual machine to access operations, mainly to facilitate sharing the clipboard with the host machine.

Configure yum domestic mirror source

Please refer to the following paragraph for using Alibaba yum source

Download wget tool# yum install -y wget
Enter the folder where the yum source configuration file is located# cd /etc/yum.repos.d/
Back up the local yum source# mv CentOS-Base.repo CentOS-Base.repo_bak
Get Ali yum source# wget -O CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
Clean up the yum cache# yum clean all
Rebuild cache # yum makecache 
Upgrade Linux system# yum -y update

# netstat -nltp # Check if the port is occupied # ifconfig # Common commands to view IP These commands are not installed by default. You can execute the following command line to install them # yum install net-tools

Install third-party dependencies

There are a lot of things that need to be installed here, which are collected from the Internet and added by my own practice. If you don’t want to bother, just follow them.

# yum -y groupinstall "Development tools"
# yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel libxml2-devel libxslt-devel libffi-devel

Install mongodb

Add mongodb domestic mirror source# vim /etc/yum.repos.d/mongodb.repo Paste the following content

[mongodb-org]
name=MongoDB Repository
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mongodb/yum/el$releasever/
gpgcheck=0
enabled=1

Refresh yum cache

# yum makecache

Install mongodb

# yum -y install mongodb-org
Modify mongod.conf bindIp parameter# vim /etc/mongod.conf

Restart the mongod service

# /etc/init.d/mongod restart or # systemctl restart mongod.service

Check the mongod version

# mongod --version

Open port 27017 on the firewall

# firewall-cmd --zone=public --add-port=27017/tcp --permanent
# firewall-cmd --reload
# firewall-cmd --zone=public --list-port

Install python3.8

Download Python 3.8 source code, compile and install

# wget https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tgz
# tar -zxvf Python-3.8.0.tgz
#cd Python-3.8.0.tgz

Check dependencies and generate Makefile.

# ./configure --prefix='/usr/local/python3'

Compile

# make

Install

# make install

Make soft link

# ln -s /usr/local/python3/bin/python3 /usr/bin/python3
# ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3

After Python 3.7, --with-ssl is not available. You can install successfully without adding --with-ssl when executing configure. If it fails, you can remove the comments of the following 3 lines in the Python-3.8.0/Modules/Setup file

# _ssl _ssl.c \# -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \# -L$(SSL)/lib -lssl -lcrypto

Install scrapy

You can first modify the pip source path, the url attribute under [source] in the Pipfile file, for example, change it to: url = "https://pypi.tuna.tsinghua.edu.cn/simple" or you can directly use the following command to install

# pip3 install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple # Specify Tsinghua University's pip source to update pip
# pip3 install scrapy -i https://pypi.tuna.tsinghua.edu.cn/simple # Install scrapy

Next, many scrapy dependency packages will be installed. It will be very slow when installing Twisted, so be patient. The installation is successful when you see the following picture!

After the installation is complete, please refer to the configuration of the remote development environment

https://www.cnblogs.com/xiao-apple36/p/8587086.html

I heard pipenv is awesome and can install Scrapy in one step.

It is also recommended to use Anaconda, which can save a lot of cross-border downloads of dependent packages, manage multiple versions of Python, and install Scrapy with one click. Especially in Windows environment, it is highly recommended. Welcome to exchange and try

Attached

CentOS7.7.1908 Tsinghua University mirror path

https://mirrors.tuna.tsinghua.edu.cn/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso oracle VirtualBox

https://download.virtualbox.org/virtualbox/6.0.14/VirtualBox-6.0.14-133895-Win.exe

Windows Enterprise Edition PyCharm-2019.2.3

https://download.jetbrains.com/python/pycharm-professional-2019.2.3.exe?_ga=2.181479010.470914607.1571468237-1358446455.1571468237

Summarize

The above is the introduction of VirtualBox CentOS7.7.1908 Python3.8 to build Scrapy development environment [graphic tutorial], I hope it will be helpful to everyone. If you have any questions, please leave me a message and I will reply to you in time. I would also like to thank everyone for their support of the 123WORDPRESS.COM website! If you find this article helpful, please feel free to reprint it and please indicate the source. Thank you!