Demand scenario: The boss asked me to use the crawler to write or update the data into the MySQL database. Baidu found two methods 1 is to use pymysql to connect to mysql, and add, delete, modify and query data by operating native sql statements; 2 is to use sqlalchemy to connect to mysql, build tables and operate databases through ORM models, without writing native sql statements, which is relatively simple; The following is my experience of using sqlalchemy. Implementation process: Connect to the database > Create a table through the model class > Establish a session > Execute the create table statement > Add, delete, modify and query through the session from sqlalchemy import exists, Column, Integer, String, ForeignKey, exists from sqlalchemy.ext.declarative import declarative_base from sqlalchemy import create_engine from sqlalchemy.orm import sessionmaker # Created database engine engine = create_engine("mysql+pymysql://user:pwd@ip/database name?charset=utf8") #Create session type DBSession = sessionmaker(bind=engine) # Instantiate the official model - Base is the ORM model Base = declarative_base() # Create a service order table class ServiceOrder(Base): __tablename__ = 'serviceOrderTable' id = Column(Integer, primary_key=True, autoincrement=True) serviceOrderId = Column(String(32), nullable=False, index=True, comment='Service Order ID') serviceDesc = Column(String(268), comment='Service Description') oneLevelName = Column(String(32), comment='C Category') twoLevelName = Column(String(32), comment='T subclass') threeLevelName = Column(String(32), comment='I project') fourLevelName = Column(String(32), comment='S sub-item') transferTimes = Column(String(32), comment='Number of transfers') overDueStatus = Column(String(32), comment='Expired status') serviceTimeLimit = Column(String(32), comment='Service time limit') serTimeLimitTypeName = Column(String(16), comment='Time limit type') # One-to-many: # serviceWorkOrder = relationship("ServiceWorkOrder", backref="serviceorder") # Many-to-one: Multiple service orders can belong to the service order class ServiceWorkOrder(Base): __tablename__ = 'serviceWorkOrderTable' id = Column(Integer, primary_key=True, autoincrement=True) serviceWorkOrderId = Column(String(32), nullable=False, index=True, comment='Service Work Order ID') workOrderName = Column(String(268), comment='Work Order Name') fromId = Column(String(32), comment='Service Order ID') createUserSectionName = Column(String(32), comment='Create User Section') createUserName = Column(String(32), comment='creator') handlerName = Column(String(32), comment='Handler') statusName = Column(String(32), comment='Work order status') createTime = Column(String(32), comment='Creation time') # The book table on the "many" side is linked to the user table via a foreign key: # serviceOrder_id = Column(Integer, ForeignKey('serviceOrderTable.id')) # Create a database. If the database already exists, it will not be created. It will directly connect to the existing library according to the library name. def init_db(): Base.metadata.create_all(engine) def drop_db(): Base.metadata.drop_all(engine) def insert_update(): # all_needed_data_lists is the data format that needs to be inserted into the database [{key: value, ... }, { }, { }...] for item in all_needed_data_lists: ServiceOrderRow = ServiceOrder(serviceOrderId=item['serviceOrderId'], serviceDesc=item['serviceDesc'], oneLevelName=item['oneLevelName'], twoLevelName=item['twoLevelName'], threeLevelName=item['threeLevelName'], fourLevelName=item['fourLevelName'], transferTimes=item['transferTimes'], overDueStatus=item['overDueStatus'], serviceTimeLimit=item['serviceTimeLimit'], serTimeLimitTypeName=item['serTimeLimitTypeName'], ) try: # Use exists to determine whether the target object exists and return True or Faults it_exists = session.query( exists().where(ServiceOrder.serviceOrderId == item['serviceOrderId'] ) ).scalar() except Exception as e: self.log.error(e) break try: # If it does not exist, add it; if it does exist, update the existing data if not it_exists: session.add(ServiceOrderRow) else: session.query(ServiceOrder).filter(ServiceOrder.serviceOrderId == item['serviceOrderId'])\ .update(item) except Exception as e: self.log.error(e) break try: session.commit() self.log.info('Data updated successfully!') except: session.rollback() self.log.info('Data update failed!') if __name__ == "__main__": # Create a database. If the database already exists, it will not be created. It will directly connect to the existing library according to the library name. init_db() # Create a session object to add, delete, modify and query: session = DBSession() # Use session to add or modify data. Remember to submit insert_update() The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM. You may also be interested in:
|
>>: How to set the user root password and remote connection method for Alibaba Cloud ECS instance
MySQL replication table detailed explanation If w...
Table of contents 1 The role of Apache 2 Apache I...
Cerebro is an evolution of the Elasticsearch Kopf...
MySQL8.0.12 installation tutorial, share with eve...
1. What is CSS Animations is a proposed module fo...
Build the project Execute the command line in the...
Introduction Today I will share the use of the su...
Since my development environment is to install Ce...
Table of contents 1. What is grub encryption 2. g...
This article mainly introduces the pie chart data...
How to save and exit after editing a file in Linu...
Introduce two methods to view MySQL user permissi...
First of all, we need to know what a state manage...
Table of contents Image capture through svg CSS p...
This article shares the specific code of vue echa...