Detailed explanation of the process of using docker to build minio and java sdk

1minio is simple
2 Docker builds minio

2.1 Single Node
2.2 Multi-node deployment

3 Java SDK using minio

1minio is simple

MinIO is a high-performance, distributed object storage system. It is a software product that can run 100% on standard hardware. That is, low-cost machines such as X86 can also run MinIO well.

insert image description here

What makes MinIO different from traditional storage and other object storage is that its software architecture is designed from the beginning for private cloud standards with higher performance requirements. Because MinIO was designed only for object storage from the beginning. Therefore, it is designed in a more user-friendly way. It can realize all the functions required for object storage and has stronger performance. It will not compromise for more business functions and lose the ease of use and efficiency of MinIO. The benefit of this result is that it can more easily implement native object storage services with elastic scalability.

insert image description here

MinIO excels in traditional object storage use cases such as secondary storage, disaster recovery, and archiving. At the same time, it is also unique in storage technology in machine learning, big data, private cloud, hybrid cloud, etc. Of course, data analysis, high-performance application loads, and cloud-native support are not excluded.

The minio community version is open source and free, and can be considered when there is no budget to use OSS.

2 Docker builds minio

Minio supports cloud native, so you can directly use docker to build it. Of course, you can also use k8s and directly download the official chart to use.

insert image description here

2.1 Single Node

A single node can be started directly using docker run

docker run \
  -p 9000:9000 \
  -p 9001:9001 \
  minio/minio server /data --console-address ":9001"

You can also use docker-compose to run.

Write docker-compose.yaml

version: '3'
services:
  minio:
    image: minio/minio
    hostname: "minio"
    ports:
      - 9000:9000 
      - 9001:9001
    environment:
      MINIO_ACCESS_KEY: admin #Console login account MINIO_SECRET_KEY: 12345678 #Console login password volumes:
      - ./data:/data #Storage path - ./config:/root/.minio/ #Configuration file command: server --console-address ':9001' /data  
    privileged: true
    restart: always

Create a mounted file directory and run docker-compose to start it.

docker-compser up -d

Enter ip:9001 Enter admin/12345678 to enter the console

insert image description here

Console:

insert image description here

Once you have created a bucket, you can upload files.

insert image description here

Enter a name and save.

insert image description here

It can be configured, but the relevant policies are not explained here.

insert image description here

Can upload and download operation object files.

insert image description here

2.2 Multi-node deployment

Multi-node deployment is simulated using docker-compse. Create 4 nodes and mount two copies of data on each node.

Write docker-compose.yaml

version: '3'

# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio1:
    image: minio/minio
    hostname: minio1
    volumes:
      - ./data1-1:/data1
      - ./data1-2:/data2
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
    command: server --console-address ":9001" http://minio{1...4}/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio2:
    image: minio/minio
    hostname: minio2
    volumes:
      - ./data2-1:/data1
      - ./data2-2:/data2
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
    command: server --console-address ":9001" http://minio{1...4}/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio3:
    image: minio/minio
    hostname: minio3
    volumes:
      - ./data3-1:/data1
      - ./data3-2:/data2
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
    command: server --console-address ":9001" http://minio{1...4}/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio4:
    image: minio/minio
    hostname: minio4
    volumes:
      - ./data4-1:/data1
      - ./data4-2:/data2
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
    command: server --console-address ":9001" http://minio{1...4}/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9001:9001"
    depends_on:
      -minio1
      - minio2
      - minio3
      - minio4

Create the corresponding data directory and nginx directory for mounting.

Use nginx to load balance 4 nodes and create nginx.conf.

user nginx;
worker_processes auto;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
    worker_connections 4096;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /var/log/nginx/access.log main;
    sendfile on;
    keepalive_timeout 65;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server minio1:9000;
        server minio2:9000;
        server minio3:9000;
        server minio4:9000;
    }

    upstream console {
        ip_hash;
        server minio1:9001;
        server minio2:9001;
        server minio3:9001;
        server minio4:9001;
    }

    server {
        listen 9000;
        listen [::]:9000;
        server_name localhost;

        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }
    }

    server {
        listen 9001;
        listen [::]:9001;
        server_name localhost;

        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;
            
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            
            chunked_transfer_encoding off;

            proxy_pass http://console;
        }
    }
}

run.

docker-compser up -d

Then enter the console and operate the same as a single node.

3 Java SDK using minio

To use minio with SDK, you must first obtain AccessKey and SecretKey.

Generate in console.

insert image description here

The project pom file is introduced.

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>8.3.0</version>
</dependency>

Write upload, download, and delete interfaces.

package com.test.minio;

import javax.servlet.http.HttpServletResponse;
import java.io.InputStream;

/**
 * Storage Files *
 * @author jiangyulu
 */
public interface FileService {
    /**
     * Upload Files *
     * @param inputStream inputStream
     * @param fdsFileName fdsFileName
     * @param img img
     * @return UUID
     */
    String upload(InputStream inputStream, String fdsFileName, boolean img);


    /**
     * Download File *
     * @param fdsFileName the name of the file in fds * @param fileName the reassigned file name * @param response response
     */
    void download(String fdsFileName, String fileName, HttpServletResponse response);

    /**
     * delete*
     * @param fdsFileName fdsFileName
     */
    void delete(String fdsFileName);
}

Write the implementation class.

package com.test.minio.impl;


import com.test.minio.FileService;
import io.minio.*;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.IOUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.InputStream;
import java.net.URLEncoder;
import java.util.UUID;


/**
 * @author jaingyulu
 */
@Slf4j
@Service("minio")
public class MinioFileServiceImpl implements FileService {

    @Value("{$minio.endpoint}")
    private String endpoint;

    @Value("{$minio.accessKeyId}")
    private String accessKeyId;

    @Value("{$minio.accessKeySecret}")
    private String accessKeySecret;

    @Value("{$minio.bucketName}")
    private String bucketName;


    @Override
    public String upload(InputStream inputStream, String fdsFileName, boolean img) {

        try {
            MinioClient minioClient =
                    MinioClient.builder()
                            .endpoint(endpoint)
                            .credentials(accessKeyId, accessKeySecret)
                            .build();

            boolean found =
                    minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
            if (found) {
                log.info("Bucket already exists.");
            } else {
                minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
            }

            if (!img) {
                minioClient.putObject(
                        PutObjectArgs.builder()
                                .bucket(bucketName)
                                .object(fdsFileName)
                                .stream(inputStream, inputStream.available(), -1)
                                .build());
            } else {
                minioClient.putObject(
                        PutObjectArgs.builder()
                                .bucket(bucketName)
                                .object(fdsFileName)
                                .stream(inputStream, inputStream.available(), -1)
                                .contentType("image/jpg")
                                .build());
            }

            inputStream.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
        return UUID.randomUUID().toString();
    }

    @Override
    public void download(String fdsFileName, String fileName, HttpServletResponse response) {
        InputStream in = null;
        try {
            MinioClient minioClient =
                    MinioClient.builder()
                            .endpoint(endpoint)
                            .credentials(accessKeyId, accessKeySecret)
                            .build();
            StatObjectResponse objectStat = minioClient.statObject(StatObjectArgs.builder().bucket(bucketName).object(fdsFileName).build());
            response.setContentType(objectStat.contentType());
            //response.setContentType(MediaType.APPLICATION_OCTET_STREAM_VALUE);
            response.setHeader("Content-Disposition", "attachment;filename=" + URLEncoder.encode(fileName, "UTF-8"));
            in = minioClient.getObject(GetObjectArgs.builder().bucket(bucketName).object(fdsFileName).build());
            IOUtils.copy(in, response.getOutputStream());
        } catch (Exception e) {
            log.error(e.getMessage());
        finally
            if (in != null) {
                try {
                    in.close();
                } catch (IOException e) {
                    log.error(e.getMessage());
                }
            }
        }
    }

    @Override
    public void delete(String fdsFileName) {
        try {
            MinioClient minioClient =
                    MinioClient.builder()
                            .endpoint(endpoint)
                            .credentials(accessKeyId, accessKeySecret)
                            .build();
            minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucketName).object(fdsFileName).build());
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

The above completes the basic functions of minio file operations. For other functions, please refer to the official documentation. The 8.3.0 version of SDK has changed significantly compared to 7.x.

This is the end of this article about building minio with docker and using java sdk. For more information about building minio with docker, please search previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!

You may also be interested in: