Handtrack.js library for real-time monitoring of hand movements (recommended)

【Introduction】: Handtrack.js is a prototype library that can implement real-time hand motion tracking and detection directly in the browser. It is an open source model trained by Tensorflow and does not require users to train themselves. With it, you can detect hand movements in images with just a few lines of code.

GitHub Homepage

https://github.com/victordibi...

1. Introduction

Handtrack.js is built based on the TensorFlow object detection API training model. It can monitor hand movements in real time through the camera. Its main features include:

handtrack.js 

2. Application scenarios

If you are interested in gesture-based interactive experiences, Handtrack.js may be useful. Users can instantly get a gesture-based interactive experience without using any additional sensors or hardware.

Some relevant application scenarios:

• Map mouse movements to hand movements to achieve control purposes;
• When the hand overlaps with other objects, it can indicate some meaningful interaction signals (such as touching or selecting an object);
• Scenarios where human hand movements can serve as an agent for some activity recognition (e.g., automatically tracking the movements of chess players in a video or image), or simply counting how many people are in an image or video frame;
• Create demos that anyone can easily run or experience with these things with minimal setup.

3. Usage

You can include the library URL directly in the script tag, or import it from npm using a build tool.

3.1 Using the script tag

The minified js file of Handtrack.js is currently hosted on jsdelivr, a free open source CDN that allows you to include any npm package in your web application.

<script src="https://cdn.jsdelivr.net/npm/handtrackjs/dist/handtrack.min.js"> </script>

<img id="img" src="hand.jpg"/> 
<canvas id="canvas" class="border"></canvas>

Once you have added the above script tag to your html page, you can reference handtrack.js using the handTrack variable as shown below:

<script>
 const img = document.getElementById('img'); 
 const canvas = document.getElementById('canvas');
 const context = canvas.getContext('2d');
 
 // Load the model.
 handTrack.load().then(model => {
 model.detect(img).then(predictions => {
  console.log('Predictions: ', predictions); 
 });
 });
</script>

The above code snippet will print out the predicted bounding boxes for the image passed in via the img tag. If you are instead feeding in frames via video or webcam, you will be able to “track” the hand as it appears in each frame.

3.2 Using NPM

You can install handtrack.js as an npm package using the following command:

npm install --save handtrackjs

You can then import and use the example in your web application:

import * as handTrack from 'handtrackjs';

const img = document.getElementById('img');

// Load the model.
handTrack.load().then(model => {
 // detect objects in the image.
 console.log("model loaded")
 model.detect(img).then(predictions => {
 console.log('Predictions: ', predictions); 
 });
});

3.3 Handtrack.js API

Handtrack.js provides two main methods, load() and detect() , which are used to load the hand detection model and get the prediction results respectively.

load() method: accepts an optional model parameter and returns a model object, which allows the user to control the performance of the model:

const modelParams = {
 flipHorizontal: true, // flip eg for video 
 imageScaleFactor: 0.7, // reduce input image size for gains in speed.
 maxNumBoxes: 20, // maximum number of boxes to detect
 iouThreshold: 0.5, // ioU threshold for non-max suppression
 scoreThreshold: 0.79, // confidence threshold for predictions.
}

handTrack.load(modelParams).then(model => {

});

The detect() method takes an input source parameter (which can be an img, video, or canvas object) and returns the bounding box prediction result of the hand position in the image:

An array of bounding boxes with class names and confidences.

model.detect(img).then(predictions => { 
  
});

The prediction result format is as follows:

[{
 bbox: [x, y, width, height],
 class: "hand",
 score: 0.8380282521247864
}, {
 bbox: [x, y, width, height],
 class: "hand",
 score: 0.74644153267145157
}]

Handtrack.js also provides other methods:

• model.getFPS() : Get FPS, that is, the number of detections per second;
• model.renderPredictions(predictions, canvas, context, mediasource) : Draws the bounding box (and source image) on the specified canvas. where predictions is the result array of the detect() method. canvas is a reference to html canvas object where predictions are rendered, context is the canvas 2D context object, mediasource is a reference to the input frame (img, video, canvas, etc.) used in predictions (it is rendered first, and the bounding box is drawn on it).
• model.getModelParameters() : returns model parameters;
• model.setModelParameters(modelParams) : Update model parameters;
• dispose() : delete the model instance;
• startVideo(video) : Starts a camera video stream on the given video element. Returns a promise that can be used to verify whether the user has provided video permissions;
• stopVideo(video) : stop video streaming;

4. The next step is computationally expensive, mainly because neural network operations are required when predicting bounding boxes, which is a point that needs to be improved and optimized later; Tracking IDs across frames: assign an ID to each object as it enters a frame and continue tracking it; Add some discrete postures: for example, not just hands, but detect open hands and fists).

5. References

Source code of Handtrack.js library: https://github.com/victordibi...

Online Demo: https://victordibia.github.io...

Egohands dataset: http://vision.soic.indiana.ed…

This is the end of this article about the Handtrack.js library for real-time monitoring of hand movements (recommended). For more JS library content related to monitoring hand movements, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope everyone will support 123WORDPRESS.COM in the future!