Repository: opencv
Issue: Assertion error in cv2.fisheye.stereoCalibrate() and cv2.omnidir.stereoCalibrate() when number of points varies across image pairs
Labels: category: calib3d

##### System information (version)
- OpenCV => opencv-contrib-python==4.5.5.62
- Python => 3.8.10
- Operating System / Platform => Ubuntu 20.04

##### Detailed description
cv2.fisheye.stereoCalibrate() and cv2.omnidir.stereoCalibrate() result in 
```OpenCV(4.5.5) /io/opencv/modules/core/src/matrix_wrap.cpp:1219: error: (-215:Assertion failed) !fixedSize() || ((Mat*)obj)->size.operator()() == Size(_cols, _rows) in function 'create'```
when the number of points is not the same across image pairs, but is otherwise consistent across left and right images. For example, this use case arises when using a charuco board for calibration.

Additional information:
- The following support unequal nb of points across images : 
  - intrinsics: ```cv2.calibrateCamera()```, ```cv2.calibrateCameraExtended()``` ```cv2.fisheye.calibrate()``` and ```cv2.omnidir.calibrate()```
  - stereo: ```cv2.stereoCalibrate()```

##### Steps to reproduce
The following code:
<details>
<summary>Code</summary>
<p>

```
import cv2
import numpy as np

# Load calibration example from
# https://github.com/opencv/opencv_contrib/blob/4.x/modules/ccalib/tutorials/data/omni_stereocalib_data.xml
fs = cv2.FileStorage("omni_stereocalib_data.xml", cv2.FILE_STORAGE_READ)


def get_points_from_node(node):
    res = []
    for i in range(node.size()):
        res.append(node.at(i).mat())
    return res


object_points = get_points_from_node(fs.getNode("objectPoints"))
image_points_1 = get_points_from_node(fs.getNode("imagePoints1"))
image_points_2 = get_points_from_node(fs.getNode("imagePoints2"))


def get_size_int_from_node(node):
    return (int(node.at(0).real()), int(node.at(1).real()))


im_size_1 = get_size_int_from_node(fs.getNode("imageSize1"))
im_size_2 = get_size_int_from_node(fs.getNode("imageSize2"))

for make_bug_happen in {False, True}:

    if make_bug_happen:
        # Remove a pt from the dataset so that a single image pair has a different number of points
        print("")
        print("********* First image has 1 less point than others *********")
        object_points[0] = object_points[0][1:, :, :]
        image_points_1[0] = image_points_1[0][1:, :, :]
        image_points_2[0] = image_points_2[0][1:, :, :]
    else:
        print("")
        print("********* All images have same # of pts *********")

    ############################################################
    print("")
    print("PINHOLE MODEL")


    def convert_pts_to_float32(all_pts):
        # From List[np.ndarray] to List[np.ndarray float32],
        all_pts = [im_pts.astype(np.float32) for im_pts in all_pts]

        return all_pts

    object_points_pinhole = convert_pts_to_float32(object_points)
    image_points_1_pinhole = convert_pts_to_float32(image_points_1)
    image_points_2_pinhole = convert_pts_to_float32(image_points_2)

    print("Intrinsics calibration")
    res_int_pinhole = cv2.calibrateCamera(
        objectPoints=object_points_pinhole,
        imagePoints=image_points_1_pinhole,
        imageSize=im_size_1,
        cameraMatrix=None,
        distCoeffs=None,
        flags=0,
    )

    print(f"reprojection error = {res_int_pinhole[0]}")

    print("Stereo calibration")
    res_stereo_pinhole = cv2.stereoCalibrate(
        objectPoints=object_points_pinhole,
        imagePoints1=image_points_1_pinhole,
        imagePoints2=image_points_2_pinhole,
        imageSize=im_size_1,
        cameraMatrix1=None,
        distCoeffs1=None,
        cameraMatrix2=None,
        distCoeffs2=None,
        flags=0,
        criteria=(cv2.TERM_CRITERIA_COUNT + cv2.TERM_CRITERIA_EPS, 200, 0.0001),
    )
    print(f"reprojection error = {res_stereo_pinhole[0]}")

    ############################################################
    print("")
    print("FISHEYE_MODEL")

    print("Intrinsics calibration")
    res_int_fisheye = cv2.fisheye.calibrate(
        objectPoints=object_points,
        imagePoints=image_points_1,
        image_size=im_size_1,
        K=None,
        D=None,
        flags=0,
    )

    print(f"reprojection error = {res_int_fisheye[0]}")

    print("Stereo calibration")
    try:
        res_stereo_fisheye = cv2.fisheye.stereoCalibrate(
            objectPoints=object_points,
            imagePoints1=image_points_2,
            imagePoints2=image_points_2,
            imageSize=im_size_1,
            K1=None,
            D1=None,
            K2=None,
            D2=None,
            flags=0,
            criteria=(cv2.TERM_CRITERIA_COUNT + cv2.TERM_CRITERIA_EPS, 200, 0.0001),
        )
        print(f"reprojection error = {res_stereo_fisheye[0]}")
    except Exception as e:
        print(f"Failed with message: {e}")

    ############################################################
    print("")
    print("OMNIDIRECTIONAL MODEL")

    print("Intrinsics calibration")
    res_int_omnidir = cv2.omnidir.calibrate(
        objectPoints=object_points,
        imagePoints=image_points_1,
        size=im_size_1,
        K=None,
        xi=None,
        D=None,
        flags=0,
        criteria=(cv2.TERM_CRITERIA_COUNT + cv2.TERM_CRITERIA_EPS, 200, 0.0001),
    )

    print(f"reprojection error = {res_int_omnidir[0]}")

    print("Stereo calibration")
    try:
        res_stereo_omnidir = cv2.omnidir.stereoCalibrate(
            objectPoints=object_points,
            imagePoints1=image_points_1,
            imagePoints2=image_points_2,
            imageSize1=im_size_1,
            imageSize2=im_size_2,
            K1=None,
            xi1=None,
            D1=None,
            K2=None,
            xi2=None,
            D2=None,
            flags=0,
            criteria=(cv2.TERM_CRITERIA_COUNT + cv2.TERM_CRITERIA_EPS, 200, 0.0001),
        )

        print(f"reprojection error = {res_stereo_omnidir[0]}")
    except Exception as e:
        print(f"Failed with message: {e}")

```

</p>
</details>

Produces the following output (Note that the data is meant for omnidirectional camera calibration, I just used it for other camera models as well to demonstrate the issue.):
<details>
<summary>output</summary>
<p>

```
********* All images have same # of pts *********

PINHOLE MODEL
Intrinsics calibration
reprojection error = 1.4275702304817834
Stereo calibration
reprojection error = 1.774005739138387

FISHEYE_MODEL
Intrinsics calibration
reprojection error = 21.307470547754154
Stereo calibration
reprojection error = 0.4170925816649913

OMNIDIRECTIONAL MODEL
Intrinsics calibration
reprojection error = 0.45549936679287745
Stereo calibration
reprojection error = 0.44681857144791404

********* First image has 1 less point than others *********

PINHOLE MODEL
Intrinsics calibration
reprojection error = 1.4254661487227347
Stereo calibration
reprojection error = 1.7729079727294896

FISHEYE_MODEL
Intrinsics calibration
reprojection error = 21.313657569855135
Stereo calibration
Failed with message: OpenCV(4.5.5) /io/opencv/modules/core/src/matrix_wrap.cpp:1219: error: (-215:Assertion failed) !fixedSize() || ((Mat*)obj)->size.operator()() == Size(_cols, _rows) in function 'create'


OMNIDIRECTIONAL MODEL
Intrinsics calibration
reprojection error = 0.45529405093749037
Stereo calibration
Failed with message: OpenCV(4.5.5) /io/opencv/modules/core/src/matrix_wrap.cpp:1219: error: (-215:Assertion failed) !fixedSize() || ((Mat*)obj)->size.operator()() == Size(_cols, _rows) in function 'create'
```

</p>
</details>

##### Issue submission checklist

 - [X] I report the issue, it's not a question
   <!--
   OpenCV team works with forum.opencv.org, Stack Overflow and other communities
   to discuss problems. Tickets with questions without a real issue statement will be
   closed.
   -->
 - [X] I checked the problem with documentation, FAQ, open issues,
       forum.opencv.org, Stack Overflow, etc and have not found any solution
   <!--
   Places to check:
   * OpenCV documentation: https://docs.opencv.org
   * FAQ page: https://github.com/opencv/opencv/wiki/FAQ
   * OpenCV forum: https://forum.opencv.org
   * OpenCV issue tracker: https://github.com/opencv/opencv/issues?q=is%3Aissue
   * Stack Overflow branch: https://stackoverflow.com/questions/tagged/opencv
   -->
 - [X] I updated to the latest OpenCV version and the issue is still there
   <!--
   master branch for OpenCV 4.x and 3.4 branch for OpenCV 3.x releases.
   OpenCV team supports only the latest release for each branch.
   The ticket is closed if the problem is not reproduced with the modern version.
   -->
 - [X] There is reproducer code and related data files: videos, images, onnx, etc
   <!--
   The best reproducer -- test case for OpenCV that we can add to the library.
   Recommendations for media files and binary files:
   * Try to reproduce the issue with images and videos in opencv_extra repository
     to reduce attachment size
   * Use PNG for images, if you report some CV related bug, but not image reader
     issue
   * Attach the image as an archive to the ticket, if you report some reader issue.
     Image hosting services compress images and it breaks the repro code.
   * Provide ONNX file for some public model or ONNX file with random weights,
     if you report ONNX parsing or handling issue. Architecture details diagram
     from netron tool can be very useful too. See https://lutzroeder.github.io/netron/
   -->