Repository: opencv
Issue: `detail_BundleAdjusterRay` error: `error: (-215:Assertion failed) !err.empty() in function 'update'`
Labels: bug

### System Information

OpenCV version: 4.9.0
Operating System: macOS Sonoma 14.1.2 (arm64 M1)
Python 3.11.6

### Detailed description

error `modules/calib3d/src/compat_ptsetreg.cpp:125` when trying to apply `cv.detail_BundleAdjusterRay()` on pairwise self-made sequence of `MatchesInfo` objects and a list of KAZE features.

### Steps to reproduce


## 1. get KAZE features from images
```python
import cv2 as cv
from my_module import plot_images, make_features_image


final_imgs = [cv.imread(path) for path in ["...", "...", "..."]]

finder = cv.AKAZE_create(detector="akaze",
                         descriptor_type=cv.AKAZE_DESCRIPTOR_MLDB_UPRIGHT,
                         threshold=0.001,
                         nOctaves=8,
                         nOctaveLayers=4)

features_list = [finder.detect_features(img) for img in final_imgs]

plot_images(make_features_image(features_list, final_imgs))
```

![image](https://github.com/opencv/opencv/assets/9123665/27752984-40ca-4c43-8131-fdb467f51964)


## 2. match features pair-wise
only match adjacent images (`range=1`) because we know order of input image sequence is same as during stitching
```python
matcher = cv.detail_BestOf2NearestRangeMatcher(1)
matches = matcher.apply2(features_list)
matcher.collectGarbage()
```

![image](https://github.com/opencv/opencv/assets/9123665/ba7bc350-44e0-4882-8eec-02a295a35dc2)
![image](https://github.com/opencv/opencv/assets/9123665/d7c40c96-ad36-418a-b4e5-eab8b5ce22cd)


## 3. define rules to filter out matches
- based on use-case and context of our application, i need to filter out background feature matches and other false positives.
- for example, a the tree in the top right corner will ruin stitching
- we only consider matches between left half of source image and right half of destination image
    - truck is moving left to right
- we try to prevent matching of significantly transformed features
    - difference in angle between two features no more than 3 degrees (sufficient for our use)
    - difference in vertical position no more than 10% of image height
```python
from typing import Callable


class ImageKeypoint:
    w: int
    h: int
    x: float
    y: float
    angle: float

    def __init__(self,
                 features: cv.detail.ImageFeatures,
                 keypoint: cv.KeyPoint):
        self.w = features.img_size[0]
        self.h = features.img_size[1]
        self.x = keypoint.pt[0]
        self.y = keypoint.pt[1]
        self.angle = keypoint.angle
        self.size = keypoint.size

    def is_within_x(self, from_to: tuple[float, float]):
        return self.x >= from_to[0]*self.w and self.x <= from_to[1]*self.w

    def is_within_y(self, from_to: tuple[float, float]):
        return self.y >= from_to[0]*self.h and self.y <= from_to[1]*self.h

    def is_within_angle(self, from_to: tuple[float, float]):
        return self.angle >= from_to[0] and self.angle <= from_to[1]

    def is_within_dx(self, other: 'ImageKeypoint', from_to: tuple[float, float]):
         diff = abs(self.x - other.x)
         return diff >= from_to[0]*self.w and diff <= from_to[1]*self.w

    def is_within_dy(self, other: 'ImageKeypoint', from_to: tuple[float, float]):
         diff = abs(self.y - other.y)
         return diff >= from_to[0]*self.h and diff <= from_to[1]*self.h

    def is_within_da(self, other: 'ImageKeypoint', from_to: tuple[float, float]):
         diff = abs(self.angle - other.angle)
         return diff >= from_to[0] and diff <= from_to[1]

    def is_within_size(self, from_to: tuple[float, float]):
        return self.size >= from_to[0] and self.size <= from_to[1]


class MatchRules:
    def __init__(self, *args):
        self.rules: tuple[tuple[Callable, tuple[float, float]]] = args

    def _invert(from_to: tuple):
        return (1-from_to[1], 1-from_to[0])

    def IS_WITHIN_X(from_to: tuple,
                     _src: ImageKeypoint,
                    _dst: ImageKeypoint):
        dst_from_to = MatchRules._invert(from_to)
        return _src.is_within_x(from_to) and _dst.is_within_x(dst_from_to)

    def IS_WITHIN_Y(from_to: tuple,
                     _src: ImageKeypoint,
                    _dst: ImageKeypoint):
        return _src.is_within_y(from_to)

    def IS_WITHIN_ANGLE(from_to: tuple,
                     _src: ImageKeypoint,
                    _dst: ImageKeypoint):
        return _src.is_within_angle(from_to)

    def IS_WITHIN_DX(from_to: tuple,
                     _src: ImageKeypoint,
                     _dst: ImageKeypoint):
        return _src.is_within_dx(_dst, from_to)

    def IS_WITHIN_DY(from_to: tuple,
                     _src: ImageKeypoint,
                     _dst: ImageKeypoint):
        return _src.is_within_dy(_dst, from_to)

    def IS_WITHIN_DA(from_to: tuple,
                     _src: ImageKeypoint,
                     _dst: ImageKeypoint):
        return _src.is_within_da(_dst, from_to)

    def IS_WITHIN_SIZE(from_to: tuple,
                       _src: ImageKeypoint,
                       _dst: ImageKeypoint):
        return _src.is_within_size(from_to) and _dst.is_within_size(from_to)

    def IS_MATCH_WITHIN_DISTANCE(from_to: tuple,
                                 match: cv.DMatch):
        return match.distance >= from_to[0] and match.distance <= from_to[1]

    def is_valid(self,
                 features1: cv.detail.ImageFeatures,
                 features2: cv.detail.ImageFeatures,
                 match: cv.DMatch):
        src = ImageKeypoint(
            features=features1,
            keypoint=features1.keypoints[match.queryIdx])
        dst = ImageKeypoint(
            features=features2,
            keypoint=features2.keypoints[match.trainIdx])

        for rule in self.rules:
            if "MATCH" in rule[0].__name__:
                result = rule[0](rule[1], match)
            else:
                result = rule[0](rule[1], src, dst)

            if not result:
                return False

        return True


MATCH_RULES = MatchRules(
    (MatchRules.IS_WITHIN_X, (0, 0.4)),  # left ~half of source img and right ~half of destination img
    (MatchRules.IS_WITHIN_DA, (0, 3)),  # delta angle between src/dst keypoints in each match
    (MatchRules.IS_MATCH_WITHIN_DISTANCE, (0, 75)),  # match distance is good enough
    (MatchRules.IS_WITHIN_DY, (0, 0.1)),  # don't match keypoints more than 10% of the img height apart
)
```

## 4. match features and filter matches 
i am trying to copy MatchesInfo elements and modify the inliers and actual matches.
don't know if this is the right way to do it...
i also tried actually recalculating the homography matrix using this method below, but got same error:
    ```python
    for i, m in enumerate(matches_list):
        points1[i, :] = features1.getKeypoints()[m.queryIdx].pt
        points2[i, :] = features2.getKeypoints()[m.trainIdx].pt
    cv.findHomography(points1, points2, cv.RANSAC)
    ```
```python
def copy_match(match: cv.detail.MatchesInfo):
        match_copy = cv.detail.MatchesInfo()
        match_copy.dst_img_idx = match.dst_img_idx
        match_copy.src_img_idx = match.src_img_idx
        match_copy.H = match.H.copy() if match.H is not None else None
        match_copy.confidence = match.confidence or 0.65
        match_copy.inliers_mask = np.ndarray(0, dtype=match.inliers_mask.dtype)
        match_copy.matches = tuple()
        match_copy.num_inliers = 0

        return match_copy


def filter_matches(features1: cv.detail.ImageFeatures,
                   features2: cv.detail.ImageFeatures,
                   matches: cv.detail.MatchesInfo):
    match_copy = copy_match(matches)
    matches_list = []
    inliers_list = []
    for i, match in enumerate(matches.matches):
        if MATCH_RULES.is_valid(features1, features2, match):
            matches_list.append(match)
            inliers_list.append(1)
            match_copy.confidence = 1

    match_copy.dst_img_idx = matches.dst_img_idx
    match_copy.src_img_idx = matches.src_img_idx
    match_copy.matches = tuple(matches_list)
    match_copy.inliers_mask = np.array(inliers_list, dtype=matches.inliers_mask.dtype)
    match_copy.num_inliers = len(inliers_list)

    return match_copy


def filter_pairwise_matches(features_list: list[cv.detail.ImageFeatures],
                            matches_list: list[cv.detail.MatchesInfo]):
    filtered_matches_list = []
    for i, matches in enumerate(matches_list):
        num_matches = len(matches.matches)
        filtered = filter_matches(features_list[matches.src_img_idx], features_list[matches.dst_img_idx], matches)
        log.info(f"{i}: {round(100*len(filtered.matches)/num_matches, 3)}%")
        filtered_matches_list.append(filtered)

    return filtered_matches_list


# filtered_matches contains only the matches that meet the criteria
masked_matches = tuple(filter_pairwise_matches(features_list, matches))
```

![image](https://github.com/opencv/opencv/assets/9123665/13165eb9-44d3-4719-b1bb-673061ca1ed0)
![image](https://github.com/opencv/opencv/assets/9123665/9eaa9b88-f65d-4604-ba25-f8a4a2bac910)

## 5. now stitch with camera estimation/adjustment
```python
from stitching.camera_estimator import CameraEstimator
from stitching.camera_adjuster import CameraAdjuster
from stitching.camera_wave_corrector import WaveCorrector

camera_estimator = cv.detail_HomographyBasedEstimator()
camera_adjuster = cv.detail_BundleAdjusterRay()
adjuster.setRefinementMask(np.array([
    [1, 1, 1],
    [1, 1, 1],
    [1, 1, 1], ], dtype=np.uint8))

cameras = camera_estimator.apply(features_list, masked_matches, None)
b, cameras = adjuster.apply(features_list, masked_matches, cameras)
```

# now for the error
```python
error: OpenCV(4.9.0) [/Users/xperience/GHA-OpenCV-Python2/_work/opencv-python/opencv-python/opencv/modules/calib3d/src/compat_ptsetreg.cpp:125](https://file+.vscode-resource.vscode-cdn.net/Users/xperience/GHA-OpenCV-Python2/_work/opencv-python/opencv-python/opencv/modules/calib3d/src/compat_ptsetreg.cpp:125): error: (-215:Assertion failed) !err.empty() in function 'update'
```

i looked for other issues and checked source code `compat_ptsetreg.cpp` but couldn't figure it out.

### Issue submission checklist

- [X] I report the issue, it's not a question
- [X] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- [X] I updated to the latest OpenCV version and the issue is still there
- [X] There is reproducer code and related data files (videos, images, onnx, etc)