Based on vue-simple-uploader, encapsulate the global upload plug-in function of file segment upload, instant upload and breakpoint resume

1. Introduction

Previously, the company wanted to make a global upload plug-in in the management system, that is, when switching between pages, the upload interface is still there and the upload will not be affected. This is not a problem in front of a SPA framework such as Vue. However, the backend boss said that we need to implement the functions of multi-part uploading, instant uploading and breakpoint resuming uploading, which sounds overwhelming.

A long time ago, I wrote an article about webuploader, but found many problems when using it, and the official team no longer maintains this plugin. After many days of research and failures, I finally decided to implement this function based on the vue-simple-uploader plugin, which is painless and stable to use in the project.

If you just want to implement basic (non-customized) upload functionality, use vue-simple-uploader directly and read its documentation. No more secondary packaging is needed.
If you just want to implement a global upload plugin, you can also refer to my implementation.
If you have used complex features such as multi-part uploading, instant uploading and breakpoint resuming, congratulations, this is the focus of this article.

The source code of this article is here: https://github.com/shady-xia/Blog/tree/master/vue-simple-uploader

2. About vue-simple-uploader

vue-simple-uploader is a vue upload plug-in encapsulated based on simple-uploader.js . Its advantages include but are not limited to the following:

• Support file, multiple files, and folder upload; support drag and drop file and folder upload
• Pause and resume uploading
• Error handling
• Support "instant transfer", which can be achieved by judging whether the server already exists through the file
• Multi-part upload
• Supports operations such as progress, estimated remaining time, automatic retry when error occurs, and retransmission

Before reading this article, it is recommended to read the documentation of simple-uploader.js first, and then read the documentation of vue-simple-uploader to understand the role of each parameter. I assume that everyone is already familiar with it here. .
vue-simple-uploader documentation

simple-uploader.js documentation

Installation: npm install vue-simple-uploader --save
Usage: In main.js:

import uploader from 'vue-simple-uploader'
Vue.use(uploader)

3. Encapsulate the global upload component based on vue-simple-uploader

After introducing vue-simple-uploader , we start to encapsulate the global upload component globalUploader.vue . The code is relatively long, so I won’t release it in its entirety. The source code is on github, and I will explain it step by step here.

The template part is as follows. I customized the template and style, so the HTML part is relatively long. The CSS part is not listed for the time being. You can modify it according to your own UI. Mainly pay attention to the options parameter of uploader component and the events of file added , success , progress , and error :

<template>
 <div id="global-uploader">

 <!-- Upload -->
 <uploader
  ref="uploader"
  :options="options"
  :autoStart="false"
  @file-added="onFileAdded"
  @file-success="onFileSuccess"
  @file-progress="onFileProgress"
  @file-error="onFileError"
  class="uploader-app">
  <uploader-unsupport></uploader-unsupport>

  <uploader-btn id="global-uploader-btn" :attrs="attrs" ref="uploadBtn">Select file</uploader-btn>

  <uploader-list v-show="panelShow">
  <div class="file-panel" slot-scope="props" :class="{'collapse': collapse}">
   <div class="file-title">
   <h2>File List</h2>
   <div class="operate">
    <el-button @click="fileListShow" type="text" :title="collapse ? 'Expand':'Collapse' ">
    <i class="iconfont" :class="collapse ? 'icon-fullscreen': 'icon-minus-round'"></i>
    </el-button>
    <el-button @click="close" type="text" title="Close">
    <i class="iconfont icon-close"></i>
    </el-button>
   </div>
   </div>

   <ul class="file-list">
   <li v-for="file in props.fileList" :key="file.id">
    <uploader-file :class="'file_' + file.id" ref="files" :file="file" :list="true"></uploader-file>
   </li>
   <div class="no-file" v-if="!props.fileList.length"><i class="nucfont inuc-empty-file"></i> No files to be uploaded</div>
   </ul>
  </div>
  </uploader-list>

 </uploader>

 </div>
</template>

The data part in the component:

data() {
 return {
 options:
  target: 'http://xxxxx/xx', // target upload URL
  chunkSize: '2048000', // Chunk size fileParameterName: 'file', // File parameter name when uploading files, default file
  maxChunkRetries: 3, //Maximum number of automatic failed retry uploads testChunks: true, //Whether to enable server fragmentation verification // Server fragmentation verification function, the basis for instant upload and breakpoint resume checkChunkUploadedByResponse: function (chunk, message) {
  let objMessage = JSON.parse(message);
  if (objMessage.skipUpload) {
   return true;
  }

  return (objMessage.uploaded || []).indexOf(chunk.offset + 1) >= 0
  },
  headers: {
  // Verification added in the header, please set according to the actual business Authorization: "Bearer " + Ticket.get().access_token
  },
 },
 attrs: {
  // Accepted file types, such as ['.png', '.jpg', '.jpeg', '.gif', '.bmp'...] Here I encapsulate accept: ACCEPT_CONFIG.getAll()
 },
 panelShow: false, //After selecting a file, display the upload panel
 }
},

Global references:
Referenced in app.vue , it always exists as a global component, but the upload interface is hidden when not in use

<global-uploader></global-uploader>

4. Overview of file upload process

1. Click the button to trigger the file upload operation:

(If you are not using the global upload function, but directly click upload, ignore this step.)

Because I am making a global upload plug-in, I need to hide the upload window first. When clicking an upload button, I use Bus to send an openUploader event, receive the event in globalUploader.vue , and trigger the click event of our uploader-btn .

On a certain page, click the upload button and bring the parameters to the background (if any). Here I use event bus to pass values ​​between components. Of course, it would be better to use vuex :

Bus.$emit('openUploader', {
 superiorID: this.superiorID
})

Receive the event in globalUploader.vue :

Bus.$on('openUploader', query => {
 this.params = query || {};

 if (this.$refs.uploadBtn) {
	 // This will open the file selection window$('#global-uploader-btn').click();
 }
});

2. After selecting the file, the upload window will be displayed and the md5 calculation will start

onFileAdded(file) {
 this.panelShow = true;
	
	// Calculate MD5, which will be mentioned below this.computeMD5(file);
},

There is a premise here. I set autoStart to false in uploader . Why do I do this?

After selecting the file, I need to calculate MD5 to implement the functions of breakpoint resumption and instant transmission. Therefore, it is definitely not possible to start uploading directly after selecting the file. I have to wait until the MD5 calculation is completed before starting the file upload operation.

The specific MD5 calculation method will be discussed below, but it will be briefly introduced here.

During the upload process, the file-progress upload progress callback will be continuously triggered

// File progress callback onFileProgress(rootFile, file, chunk) {
 console.log(`Uploading ${file.name}, chunk: ${chunk.startByte / 1024 / 1024} ~ ${chunk.endByte / 1024 / 1024}`)
},

3. After the file is uploaded successfully

After the file is uploaded successfully, in the "upload completed" callback, the needMerge field returned by the server is used to determine whether it is necessary to send a request to merge the shards again.
If this field is true, you need to send an ajax request to the backend to request merging, otherwise the upload will succeed directly.

Note: needMerge here is the field name decided by me and the backend

onFileSuccess(rootFile, file, response, chunk) {
 let res = JSON.parse(response);

 // Server-defined error, which cannot be intercepted by Uploader if (!res.result) {
 this.$message({ message: res.message, type: 'error' });
 return
 }
	
	// If the server returns a request for merging if (res.needMerge) {
 api.mergeSimpleUpload({
  tempName: res.tempName,
  fileName: file.name,
  ...this.params,
 }).then(data => {
  // File merge successful Bus.$emit('fileSuccess', data);
 }).catch(e => {});
 // No need to merge } else {
 Bus.$emit('fileSuccess', res);
 console.log('Upload successful');
 }
},

onFileError(rootFile, file, response, chunk) {
	console.log(error)
},

5. File Sharding

vue-simple-uploader automatically splits the file into pieces, and the size of each piece can be set in chunkSize of options .

As shown in the figure: For large files, multiple requests will be sent. After setting testChunks to true (the default setting in the plugin is true ), a request will be sent to the server for fragment verification. The first get request below is this request; each subsequent post request is a request to upload a fragment.

Take a look at the parameters sent to the server. chunkNumber indicates the current shard number, and totalChunks represents the total number of shards. Both parameters are calculated by the plugin based on chunkSize you set.

It should be noted that in the event of successful file upload, the fields returned by the backend are used to determine whether to send another file merge request to the backend.

6. MD5 calculation process

The basis for breakpoint-resume and instant-transfer is to calculate MD5 of the file, which is the unique identifier of the file. The server then determines whether to perform instant-transfer or breakpoint-resume based on MD5 .

After file-added event, MD5 is calculated. Our ultimate goal is to add the calculated MD5 to the parameter and pass it to the backend, and then continue the file upload operation. The detailed idea steps are:

• Set autoStart of the uploader component to false , which means that uploading will not start automatically after selecting a file.
• First pause the file through file.pause() , and then read the file through H5's FileReader interface
• Perform MD5 on the result of asynchronously reading the file. The encryption tool I use here is spark-md5 . You can install it through npm install spark-md5 --save , or you can use other MD5 encryption tools.
• The file has an attribute called uniqueIdentifier , which represents the unique identifier of the file. We assign the calculated MD5 to this attribute file.uniqueIdentifier = md5 , which achieves our ultimate goal.
• Start/continue file upload via file.resume() .

/**
* Calculate md5 to achieve breakpoint resumable and instant transfer* @param file
*/
/**
* Calculate md5 to achieve breakpoint resumable and instant transfer* @param file
*/
 computeMD5(file) {
 let fileReader = new FileReader();
 let time = new Date().getTime();
 let blobSlice = File.prototype.slice || File.prototype.mozSlice || File.prototype.webkitSlice;
 let currentChunk = 0;
 const chunkSize = 10 * 1024 * 1000;
 let chunks = Math.ceil(file.size / chunkSize);
 let spark = new SparkMD5.ArrayBuffer();
 
 //Set the file status to "calculate MD5"
 this.statusSet(file.id, 'md5');
 
 file.pause();
 
 loadNext();
 
 fileReader.onload = (e => {
 spark.append(e.target.result);
 if (currentChunk < chunks) {
  currentChunk++;
  loadNext();
  // Real-time display of MD5 calculation progress this.$nextTick(() => {
  $(`.myStatus_${file.id}`).text('Verify MD5 '+ ((currentChunk/chunks)*100).toFixed(0)+'%')
  })
 } else {
  let md5 = spark.end();
  this.computeMD5Success(md5, file);
  console.log(`MD5 calculation completed: ${file.name} \nMD5: ${md5} \nSegments: ${chunks} Size: ${file.size} Time: ${new Date().getTime() - time} ms`);
 }
 });
 fileReader.onerror = function () {
 this.error(`Error reading file ${file.name}, please check the file`)
 file.cancel();
 };
 function loadNext() {
 let start = currentChunk * chunkSize;
 let end = ((start + chunkSize) >= file.size) ? file.size : start + chunkSize;
 fileReader.readAsArrayBuffer(blobSlice.call(file.file, start, end));
 }
},

computeMD5Success(md5, file) {
 // Load custom parameters directly into the opts of the uploader instance Object.assign(this.uploader.opts, {
 query: {
  ...this.params,
 }
 })
 file.uniqueIdentifier = md5;
 file.resume();
 this.statusRemove(file.id);
},

After assigning a value to the uniqueIdentifier property of the file, the identifier in the request is the MD5 we calculated.

7. Instant upload and resume download

After calculating MD5 , we can talk about the concepts of resuming downloads and transferring in seconds.

The server determines whether to perform instant transmission or breakpoint resumable transmission based on MD5 sent by the front end:

• a. If the server finds that the file has been uploaded successfully, it will directly return the second upload mark.
• b. If the server finds that the file has been uploaded in fragments, it returns the fragment information and tells the front-end to continue uploading, i.e., resume uploading from a breakpoint.

7.1 For the front end

At the beginning of each upload process, vue-simple-uploader will send a get request to ask the server which parts have been uploaded.

There are several possible results for this request:

a. If it is a second upload, there will be a corresponding mark in the request result. For example, here skipUpload is true and url is returned, which means the server tells us that the file already exists. I will give you url directly and you don’t need to upload it again. This is a second upload.

Figure a1: Backend return value in the case of instant transmission

Figure a2: GIF transmission

b. If the backend returns fragment information, this is breakpoint resume. As shown in the figure, there is an uploaded field in the returned data, which means that these segments have been uploaded, and the plug-in will automatically skip uploading these segments.

Figure b1: Backend return value in case of breakpoint resuming

Figure b2: Breakpoint resume gif

c. Maybe nothing will be returned, then this is a brand new file, and the complete multi-part upload logic is followed

7.2 Front-end fragmentation check: checkChunkUploadedByResponse

The previous part talked about the concepts. Now let’s talk about how the front end handles these return values.
The plugin itself will not determine which one needs to be skipped. In the code, it is controlled by checkChunkUploadedByResponse in options . It will check whether each chunk is uploaded successfully based on the XHR response content. Successful chunks will be skipped directly. You need to process it in this function and return true if they can be skipped.

checkChunkUploadedByResponse: function (chunk, message) {
	 let objMessage = JSON.parse(message);
 if (objMessage.skipUpload) {
  return true;
 }

 return (objMessage.uploaded || []).indexOf(chunk.offset + 1) >= 0
},

Note: skipUpload and uploaded are the fields I discussed with the backend. You should follow the field names actually returned by the backend.

8. Source code and postscript

There are several files in total, app.vue , encapsulated global upload component globalUploader.vue , demo.vue that calls the component, and the source code is on github: https://github.com/shady-xia/Blog/tree/master/vue-simple-uploader.

ticket and api in the globalUploader source code are for your own use. One is accesstoken and the other is a request library based on axios encapsulation. Please replace them according to your business needs. In addition, jquery is used to expand and collapse the upload interface, and the notification uses the Element component, please ignore it.

My level is limited, and I am just providing an idea for your reference.

After packaging this plug-in and developing the file resource library, I found that I have basically realized a simple Baidu Netdisk. It is a management system with such complicated functions, which is a rip-off!

8.1 About the first shard loss problem

Regarding the problem that the server cannot receive the first fragment after testChunk is enabled:
The simpleUploader documentation says:

The get request of testChunk brings the first fragment to the server by default. If the server returns a 200 status, it is assumed that the current chunk has been uploaded and will not be uploaded again.
So here the server needs to change to other http status codes, such as 204, so that it is not in the "200, 201, 202" set, which means that the server does not have this block yet and needs to upload it in standard mode, so that the first fragment will be uploaded again

Updated on 2019/8/6

1. Optimized the way to calculate file MD5 and displayed the calculation progress of MD5

The method of calculating MD5 in the previous article is to directly calculate MD5 for the entire file, which consumes a lot of memory and easily causes the browser to crash. I changed it to calculating MD5 by reading the file in pieces to prevent the web page from freezing and crashing due to excessive memory usage when directly reading large files.

2. Added custom status

(I have encapsulated several custom states before. Recently, some friends have asked why there are no "verify MD5" and "merging" states. I wrote my method. The method is stupid, but it can achieve the effect)

The plugin originally only supports the following states: success , error , uploading , paused , and waiting .

Due to business needs, I have added several custom statuses: “校驗MD5” , “合并中” , “轉碼中” , and “上傳失敗”

Since the first few states are already packaged in the plugin, I cannot change the source code and can only use a more hacky approach:
When the custom state starts, you need to manually call the statusSet method to generate a p tag to cover the original state; when the custom state ends, you also need to manually call statusRemove to remove the tag.

this.statusSet(file.id, 'merging');
this.statusRemove(file.id);

For specific usage, please refer to the source code. At the same time, I hope that the plugin author of simple-uploader will support the configuration of custom status in the future.

This is the end of this article about the global upload plug-in function based on vue-simple-uploader encapsulation for file segment upload, instant upload and breakpoint resume. For more related vue simple uploader encapsulation content, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope everyone will support 123WORDPRESS.COM in the future!