Implementation of inserting millions of records into MySQL database within 10 seconds

Implementation of inserting millions of records into MySQL database within 10 seconds

First, let’s think about a question:

To insert such a huge amount of data into the database, it must be accessed frequently under normal circumstances, and no machine equipment can handle it. So how can we avoid frequent access to the database? Can we access it once and then execute it?

Java has actually given us the answer.

Two key objects are used here: Statement and PrepareStatement

Let's look at the characteristics of the two:

BaseDao tool class to be used (jar package/Maven dependency) (Maven dependency code is attached at the end of the article) (packaged for easy use)

Note: (Important) rewriteBatchedStatements=true, insert multiple data at a time, and only insert them once! !

public class BaseDao { // Static tool class, used to create database connection objects and release resources for easy calling // Import the driver jar package or add Maven dependencies (Maven is used here, and the Maven dependency code is attached at the end of the article)
    static {
        try {
            Class.forName("com.mysql.cj.jdbc.Driver");
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        }
    }
 
    // Get the database connection object public static Connection getConn() {
        Connection conn = null;
        try {
            // rewriteBatchedStatements=true, insert multiple data at a time, insert only onceconn = DriverManager.getConnection("jdbc:mysql://localhost:3306/million-test?rewriteBatchedStatements=true", "root", "qwerdf");
        } catch (SQLException throwables) {
            throwables.printStackTrace();
        }
        return conn;
    }
 
    // Release resources public static void closeAll(AutoCloseable... autoCloseables) {
        for (AutoCloseable autoCloseable : autoCloseables) {
            if (autoCloseable != null) {
                try {
                    autoCloseable.close();
                } catch (Exception e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }
        }
    }
}

Next are the key codes and comments:

/* Because the processing speed of the database is very fast, the single throughput is very large and the execution efficiency is very high. addBatch() loads several SQL statements together and sends them to the database for execution at once. The execution takes a very short time. preparedStatement.executeUpdate() sends them to the database one by one for execution, and the time is consumed in the transmission of the database connection.*/
public static void main(String[] args) {
    long start = System.currentTimeMillis(); // Get the current time of the system and record it before the method starts executing Connection conn = BaseDao.getConn(); // Call the static tool class just written to get the connection database object String sql = "insert into mymilliontest values(null,?,?,?,NOW())"; // SQL statement to be executed PreparedStatement ps = null;
    try {
        ps = conn.prepareStatement(sql); // Get PreparedStatement object // Continuously generate sql
        for (int i = 0; i < 1000000; i++) {
            ps.setString(1, Math.ceil(Math.random() * 1000000) + "");
            ps.setString(2, Math.ceil(Math.random() * 1000000) + "");
            ps.setString(3, UUID.randomUUID().toString()); // The UUID class is used to randomly generate a string that will not be repeated ps.addBatch(); // Add a set of parameters to the batch command of this PreparedStatement object.
        }
        int[] ints = ps.executeBatch(); // Submit a batch of commands to the database for execution. If all commands are executed successfully, an array of update counts is returned.
        // If the array length is not 0, it means that the SQL statement is executed successfully, that is, one million data are added successfully!
        if (ints.length > 0) {
            System.out.println("One million records have been added successfully!!");
        }
    } catch (SQLException throwables) {
        throwables.printStackTrace();
    finally
        BaseDao.closeAll(conn, ps); // Call the static tool class just written to release resources}
    long end = System.currentTimeMillis(); // Get the system time again System.out.println("Time taken: " + (end - start) / 1000 + "seconds"); // Subtracting the two times is the time taken to execute the method}

Finally, let's run and see the effect:

Hey, the duration here is more than 10 seconds, the equipment is not good enough, I hope you understand~

<!--mysql-connector-java dependency used to connect to the database-->
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>8.0.27</version>
</dependency>

PS: It will be faster after adding threads, and examples will be given in subsequent articles.

This is the end of this article about how to insert one million records into MySQL database within 10 seconds. For more information about how to insert one million records into MySQL, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!

You may also be interested in:
  • MySQL automatically inserts millions of simulated data operation code
  • 4 ways to optimize MySQL queries for millions of data
  • MySQL single table million data records paging performance optimization skills
  • How to quickly insert millions of test data in MySQL

<<:  Two ways to remove the 30-second ad code from Youku video

>>:  Example of using CSS3 to create Pikachu animated wallpaper

Recommend

How to modify the default storage engine in MySQL

mysql storage engine: The MySQL server adopts a m...

12 types of component communications in Vue2

Table of contents 1. props 2..sync 3.v-model 4.re...

CSS 3.0 text hover jump special effects code

Here is a text hovering and jumping effect implem...

Notes on configuring multiple proxies using vue projects

In the development process of Vue project, for th...

Web page production TD can also overflow hidden display

Perhaps when I name this article like this, someon...

Basic usage details of Vue componentization

Table of contents 1. What is componentization? 2....

How to decrypt Linux version information

Displaying and interpreting information about you...

How to use Element in React project

This is my first time using the element framework...

Solution to the inconsistency between crontab execution time and system time

Preface In LINUX, periodic tasks are usually hand...

How to disable ads in the terminal welcome message in Ubuntu Server

If you are using the latest Ubuntu Server version...

Dissecting the advantages of class over id when annotating HTML elements

There are very complex HTML structures in web pag...

Use JS to zoom in and out when you put the mouse on the image

Use JS to zoom in and out when the mouse is on th...

Practical operation of using any font in a web page with demonstration

I have done some research on "embedding non-...

Detailed explanation of how to gracefully delete a large table in MySQL

Preface To delete a table, the command that comes...