Cloud Deep Learning Face Recognition TypeScript WebRTC

Facial Detection and Match with TypeScript and Faceapi.js – 2

March 4, 2019

We are looking to use AI to detect faces and identify users in our user database in this series.

So far we’ve setup our project and setup camera access to achieve getting a streaming video. With that we were able to grab a frame from that video into a canvas. You can catch these posts here:

Part 0 – Introduction
Part 1 – Camera Access

As a reminder, we are wanting to eventually crop the face from the grabbed frame and send it for analysis on the cloud.

Next we will be working on focusing on the users face locally on the client so we can crop the face portion from the image to send to the cloud. For this, we are going to use a client side library call face-api.js which is an open source project here . The library uses Tensorflow.js to create and run models to detect faces, facial comparison and many other features that can be read about on the GitHub project page. We will be using it just simply for detecting a face and cropping.

Setup

First thing is first, install the package into the project by running

npm install face-api.js --save

I decided here to not use the typings of the library and just use the prebuild dist version instead so at the top of my class I added a reference to the global object that gets loaded when added the script.

<script src="dist/face-api.min.js"></script>

declare var faceapi: any;

class App {
..
..

Before we can run the face detection, we have to load the model we want it to use. There are several and I’ll let it be up to you to learn about the ones available or how to create one your own, however I’ve found that the included one that comes with the npm package is sufficient enough for our task. The library needs to pull these down at runtime so make them available in your public space. I put mine in the root of my project (where my web server is running from currently)

Easiest way to find these I found is just search your npm_modules for ssd_mobilenetv1 and place these 3 files in your public server folder.

Afterwards, we’ll need to initialize faceapi in the constructor of our App class:

faceapi.nets.ssdMobilenetv1.load('/');

Which will search the provided location for the files you just placed. You have the option to use other models, which you would specify other models at this time, the default is the one we are using so no need to specify explicit model.

Workflow

After we enable the video, our <video> element will have a stream of frames. What we want to do now is to grab a frame from that stream, then feed that image data to the faceapi library. The library has several methods to choose from to perform many operations but we will only be using the ‘ detectSingleFace ‘ operation. We will also be drawing the found bounding boxes over the video, we will be doing this via an additional canvas that is overlaid on top of the video and clearing this canvas on every iteration so we can see the box moving as out face moves about. Thirdly we will be grabbing the image data from within this bounding box and displaying this cropped image inside another canvas as a preview for our detected face. So the steps are as follows:

1 – Start video from camera feed
2 – Grab the frames from video element and render in canvas
3 – Use 2d context from canvas to feed into faceapi to detect face bounding boxes
4 – Use found bounding box dimensions to render boxes over video to frame the faces we’ve detected
5 – Use this bounding box to crop out the detected face and render to preview canvas (we’ll use this cropped image in later posts)

We’ve already done the setup to load and initialize faceapi, we have the video running with the camera feed already, now we just need to detect and render the faces.

New canvas elements

I added two new canvas elements, one for the video overlay and one for the crop preview:

    canvasOverlay:HTMLCanvasElement;
    canvasFaceCrop:HTMLCanvasElement;

Detection routing – the AI

Below is the code to detect any single face and render the box and preview. First line is grabbing the video frame. Then we use this frame to detect faces which returns a promise. From the result of this promise we get back many properties (I’ll let you explore those on your own) but we are interested in the box property here which contains coordinate and dimensions.

We then draw a red line with these coordinates on our overlay canvas, first clearing whatever was there before. We then extract image data from the grabbed video frame with these coordinates and render that into our preview canvas.

this.canvas.getContext('2d').drawImage(this.video, 0, 0, this.canvas.width, this.canvas.height);
        faceapi.detectSingleFace(this.canvas).run()
        .then((res) => {
            this.fps +=1;
            let ctx = this.canvasOverlay.getContext("2d");
            ctx.clearRect(0, 0, this.canvas.width, this.canvas.height);
            ctx.beginPath();
            if(res && res.box){
                ctx.strokeStyle = "#FF0000";
                ctx.lineWidth = 3;
                ctx.rect(res.box.x, res.box.y, res.box.width , res.box.height);
                ctx.stroke();

                const croppedImage = this.canvas.getContext("2d").getImageData(res.box.x, res.box.y, res.box.width , res.box.height)
                let ctxfc = this.canvasFaceCrop.getContext("2d");
                ctxfc.clearRect(0, 0, this.canvasFaceCrop.width, this.canvasFaceCrop.height);
                ctxfc.beginPath();
                ctxfc.putImageData(croppedImage, 0, 0);

            }
            window.setTimeout(() => {this.processFaceBox();},0);
        })
        .catch((err) => {console.log(err)});

Wrap all this into a function that is ran on an interval for continuous processing: (When the process button is pressed in our case)

async btnProcessClicked(e:Event){
        this.fps = 0;
        this.processFaceBox();
        window.setInterval(()=> {
            this.divFPS.innerHTML = this.fps.toString();
            this.fps = 0;
        },1000);
    }

Result

This is what we are able to use by now:

Detecting face on the client

You can see this in action here in your browser:

https://recaf-io.github.io/PostExamples/face/faceapijs.html

In the next post we will be using Azure to grab facial analysis as well also continue onto making a registration system for people and use that corpus to identify users.

Aside: What got me interested in facial recognition in the first place is deep learning as well as a few books that are sitting on my self that I highly recommend reading if you’re looking to dig deeper into this space. Here are a couple of the resources that I have spent time with which has helped tremendously unravel how these models work:

A book, not on deep learning , but on ho biometric analysis is shaping the future: