How to insert 10 million records into a MySQL database table in 88 seconds

How to insert 10 million records into a MySQL database table in 88 seconds

The database I use is MySQL database version 5.7

First prepare the database table yourself

Actually, I encountered some problems when inserting 10 million data. Now I will solve them first. At the beginning, I got an error when inserting 1 million data. The console information is as follows:

com.mysql.jdbc.PacketTooBigException: Packet for query is too large (4232009 > 4194304). You can change this value on the server by setting the max_allowed_packet' variable.

The above error occurs because the max_allowed_packet configuration of the database table is not large enough, because the default is 4M. Later, I adjusted it to 100M and no error was reported.

set global max_allowed_packet = 100*1024*1024*

Remember, you need to log in to the database again after setting it to see the set value

show VARIABLES like '%max_allowed_packet%'

The code is as follows:

package insert;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Date;
import com.mysql.jdbc.PreparedStatement;
public class InsertTest {
   public static void main(String[] args) throws ClassNotFoundException, SQLException {
     final String url = "jdbc:mysql://127.0.0.1/teacher" ; 
     final String name = "com.mysql.jdbc.Driver"; 
     final String user = "root" ; 
     final String password = "123456" ; 
     Connection conn = null; 
     Class.forName(name); //Specify the connection type conn = DriverManager.getConnection(url, user, password); //Get the connection if (conn!= null ) {
       System.out.println("Connection obtained successfully");
       insert(conn);
     } else {
       System.out.println("Failed to obtain connection");
     }
   }
   public static void insert(Connection conn) {
     // Start time Long begin = new Date().getTime();
     // sql prefix String prefix = "INSERT INTO t_teacher (id,t_name,t_password,sex,description,pic_url,school_name,regist_date,remark) VALUES ";
     try {
       // Save sql suffix StringBuffer suffix = new StringBuffer();
       // Set the transaction to non-auto commit conn.setAutoCommit( false );
       // PST is better than ST PreparedStatement pst = (PreparedStatement) conn.prepareStatement( "" ); //Prepare to execute statement// Outer loop, total number of committed transactions for ( int i = 1 ; i <= 100 ; i++) {
         suffix = new StringBuffer();
         // j-th submission step for ( int j = 1 ; j <= 100000 ; j++) {
           // Build SQL suffix suffix.append( "('" + uutil.UUIDUtil.getUUID()+ "','" +i*j+ "','123456'" + ",'Male'" + ",'Teacher'" + ",'www.bbk.com'" + ",'XX University'" + ",'" + "2016-08-12 14:43:26" + "','Note'" + ")," );
         }
         // Build complete SQL
         String sql = prefix + suffix.substring( 0 , suffix.length() - 1 );
         // Add execution SQL
         pst.addBatch(sql);
         // Execute operations pst.executeBatch();
         // Commit transaction conn.commit();
         // Clear the last added data suffix = new StringBuffer();
       }
       //First-class connection pst.close();
       conn.close();
     } catch (SQLException e) {
       e.printStackTrace();
     }
     // End time Long end = new Date().getTime();
     // Time consuming System.out.println( "Time taken to insert 10 million records: " + (end - begin) / 1000 + " s" );
     System.out.println("Insert completed");
   }
}

Summarize

The above is the operation method introduced by the editor to insert 10 million data into the MySQL database table in 88 seconds. I hope it will be helpful to everyone. If you have any questions, please leave me a message and the editor will reply to you in time. I would also like to thank everyone for their support of the 123WORDPRESS.COM website!

You may also be interested in:
  • MySql quick insert tens of millions of large data examples
  • Example code for inserting millions of data into MySQL using JDBC in Java
  • How to quickly insert millions of test data in MySQL
  • Teach you how to insert 1 million records into MySQL in 6 seconds

<<:  ThingJS particle effects to achieve rain and snow effects with one click

>>:  JavaScript offset implements mouse coordinate acquisition and module dragging within the window

Recommend

View MySQL installation information under Linux server

View the installation information of mysql: #ps -...

Detailed explanation of computed properties in Vue

Table of contents Interpolation Expressions metho...

CSS beginner tutorial: background image fills the entire screen

If you want the entire interface to have a backgr...

Detailed explanation of MySQL binlog usage

binlog is a binary log file that records all DML ...

How to modify the initial password of MySQL on MAC

Problem description: I bought a Mac and installed...

jQuery implements breathing carousel

This article shares the specific code of jQuery t...

js canvas realizes circular water animation

This article example shares the specific code of ...

MySQL sorting using index scan

Table of contents Install sakila Index Scan Sort ...

Element uses scripts to automatically build new components

Table of contents background How does element-ui&...

How to generate a free certificate using openssl

1: What is openssl? What is its function? What is...

Implementation of tomcat image created with dockerfile based on alpine

1. Download the alpine image [root@docker43 ~]# d...

Detailed explanation of 5 solutions for CSS intermediate adaptive layout

Preface When making a page, we often encounter co...

Sample code for separating the front-end and back-end using FastApi+Vue+LayUI

Table of contents Preface Project Design rear end...