2

I'm using Google's Cloud Vision API to detect faces and landmarks within them (like eyes, nose and so on).

If the face is rotated, I'd like to correct the rotation so the face and its landmarks are positioned vertically inside a canvas element.

Google provides the coordinates of the landmarks with their origin in the top left, and roll, tilt and pan properties in degrees:

enter image description here

"landmarks": [
        {
          "position": {
            "x": 371.52585,
            "y": 437.1983,
            "z": 0.0012220144
          },
          "type": "LEFT_EYE"
        },
        ...
        "panAngle": -2.0305812,
        "rollAngle": 26.898327,
        "tiltAngle": -2.6251676,
        ...

I can correct the rotation of the image by converting the rollAngle property to radians using ctx.rotate(degrees*Math.PI/180), but how do I rotate the coordinates so they match the rotated image?

My goal is to have the image and the corresponding coordinates as follows:

enter image description here

Cheers

dave
  • 2,750
  • 1
  • 14
  • 22

2 Answers2

2

I didn't want to have to send two network requests to the API for this. The solution was to rotate both the canvas and the coordinates separately. First I rotate the canvas at its centre, using the rollAngle provided by the Cloud Vision API.

function rotateCanvas(canvas, image) {

    let ctx = canvas.getContext('2d');
    let rollAngle = sampleResponse.faceAnnotations[0].rollAngle; // rotation of face provided by the Cloud Vision API

    ctx.save();
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    ctx.translate(
        canvas.width / 2,
        canvas.height / 2);
    ctx.rotate(Math.PI / 180 * -rollAngle);
    ctx.translate(
        -(canvas.width / 2),
        -(canvas.height / 2));
    ctx.drawImage(
        image,
        canvas.width / 2 - canvas.width / 2,
        canvas.height / 2 - canvas.height / 2,
        canvas.width,
        canvas.height);
    ctx.restore();

    return canvas;

}

Next I used this answer to loop through each landmark provided by the Cloud Vision API and rotate it by the given rollAngle:

function rotateCoordinate(cx, cy, x, y, angle) {

    // rotate the landmarks provided by the cloud vision API if the face in the supplied
    // image isn't vertically aligned

    var radians = (Math.PI / 180) * angle,
        cos = Math.cos(radians),
        sin = Math.sin(radians),
        nx = (cos * (x - cx)) + (sin * (y - cy)) + cx,
        ny = (cos * (y - cy)) - (sin * (x - cx)) + cy;
    return [nx, ny];
}

As with the canvas rotation, I rotated everything from the centre.

function rotateLandmarks(canvas, landmarks) {

    let rollAngle = sampleResponse.faceAnnotations[0].rollAngle;

    for (let i = 0; i < landmarks.length; i++) {

        let rotated = this.rotateCoordinate(
            canvas.width / 2,
            canvas.height / 2,
            landmarks[i].position.x,
            landmarks[i].position.y,
            rollAngle)

        landmarks[i].position.x = rotated[0];
        landmarks[i].position.y = rotated[1];

    }

    return landmarks;

}
dave
  • 2,750
  • 1
  • 14
  • 22
1

I can see two ways:

  1. Determine basic face image rotation, rotate it to the upright position and then determine face landmarks once again;

or

  1. Use Mathemetics:

    x1 = x0 * cos(roll)

    y1 = y0 * sin(roll)

Bohdan
  • 753
  • 1
  • 9
  • 18