1
\$\begingroup\$

I have a dataset containing 2D images of telco towers and metadata. Each 2D image has 3x3 rotation matrix which provides orientation info about the drone when it captured the image.

Now I have a 3D site (tower) constructed in Three.js canvas with Z in up direction and camera looking in Y direction, containing multiple annotation bounding boxes. Whenever one clicks on a bounding box on tower, we want best view 2d images filtered from our dataset relative to our POV of the asset in bounding box on tower. Which is not the case right now.

Right now I'm extracting third row from our 3x3 rotation matrix and checking the angle between that vector and Three.js camera direction vector in world space.

One solution I thought was to use camera.worldInverseMatrix and get the direction vector of camera in view space. Only then I compare the updated direction vector with the direction vector extracted from rotation matrix of 2d image. But I don't exactly know how to implement this.

additional context:

  • in 2d images: y axis is on top, z axis is backward
  • in canvas: z axis is on top with our camera looking in y axis (forward)
    bestViewImages(imgsWithRotationAndCenter: any[], cameraPosition: any, cameraDirection: any, angleThresholdDegrees: number,
    distThreshMeters: number, numViewsToReturn: number) {
    let angleFilteredImages: any[] = [];
    let filteredImagesToDistance: any = {};
    
    for (const image of imgsWithRotationAndCenter) {
      let isRotationMatrixExists: boolean = !!(image?.rotationMatrix.length);
      if (isRotationMatrixExists) {
        console.log('rotation matrxix: ', image.rotationMatrix)
        let directionVector: ICameraAxis = {
          'x': image.rotationMatrix[2][0],
          'y': image.rotationMatrix[2][1],
          'z': image.rotationMatrix[2][2]
        }

        if (this.checkIfAngleBetweenVecsIsWithinThreshold(directionVector, cameraDirection, angleThresholdDegrees)) {
          angleFilteredImages.push(image);
        }
      }
    }

    for (let idx: number = 0; idx < angleFilteredImages.length; idx += 1) {
      let isCameraCenterExists: boolean = !!(imgsWithRotationAndCenter[idx]?.cameraCenter.length);
      if (isCameraCenterExists) {
        let imageCameraCenter: ICameraAxis = {
          'x': angleFilteredImages[idx].cameraCenter[0],
          'y': angleFilteredImages[idx].cameraCenter[1],
          'z': angleFilteredImages[idx].cameraCenter[2]
        }
        let dis: number = this.calculateDistanceBetween3DPoints(imageCameraCenter, cameraPosition);
        if (dis < distThreshMeters) {
          filteredImagesToDistance[angleFilteredImages[idx]?.imageName] = dis;
        }
      }
    }

    let distanceBasedSortedImages: String[] = !this.commonService.isObjectEmpty(filteredImagesToDistance) ? this.sortDictionary(filteredImagesToDistance) : [];
    // let topFiveImages = (distanceBasedSortedImages.length) ? distanceBasedSortedImages.slice(0, numViewsToReturn) : [];
    console.log('distanceBasedSortedImages: ', distanceBasedSortedImages)
    return distanceBasedSortedImages;
}
  

    private checkIfAngleBetweenVecsIsWithinThreshold(vec1: ICameraAxis, vec2: ICameraAxis, threshDegrees: number) {
        let dotProduct = vec1.x * vec2.x + vec1.y * vec2.y + vec1.z * vec2.z;
        let vec1Mag = Math.sqrt(vec1.x * vec1.x + vec1.y * vec1.y + vec1.z * vec1.z);
        let vec2Mag = Math.sqrt(vec2.x * vec2.x + vec2.y * vec2.y + vec2.z * vec2.z);
        let cosTheta = (dotProduct / (vec1Mag * vec2Mag));
        let angle = Math.acos(cosTheta) * (180 / Math.PI);
        return (angle < threshDegrees) ? true : false;
    }


    private calculateDistanceBetween3DPoints(point1: ICameraAxis, point2: ICameraAxis) {
        return Math.sqrt(Math.pow(point2.x - point1.x, 2) + Math.pow(point2.y - point1.y, 2) + Math.pow(point2.z - point1.z, 2));
    }
```
\$\endgroup\$
2
  • \$\begingroup\$ I'm not exactly sure what you are asking. You ask how to calculate the direction vector from a 3x3 rotation matrix and the answer is the (negated?) 3rd column, which you seem to be doing. \$\endgroup\$
    – CPlus
    Commented Apr 27 at 21:02
  • \$\begingroup\$ I'm displaying 2D telco tower images in a Three.js canvas, each with a 3x3 rotation matrix showing the drone's orientation. Aligning this with the canvas's camera view is tricky due to differing coordinate systems. We extract the third row of each rotation matrix as a direction vector and compare it with the canvas's camera direction. I think there's a issue between coordinate system mismatch here. goal is to dynamically choose and display the most fitting image, considering the similarity between the canvas's camera orientation and the drone's during capture. \$\endgroup\$ Commented Apr 28 at 0:25

0

You must log in to answer this question.

Browse other questions tagged .