Repository: opencv
Issue: matchTemplate only produces NaNs in some cases when using CCOEFF_NORMED and masks
Labels: bug,category: imgproc

<!--
If you have a question rather than reporting a bug please go to https://forum.opencv.org where you get much faster responses.
If you need further assistance please read [How To Contribute](https://github.com/opencv/opencv/wiki/How_to_contribute).

This is a template helping you to create an issue which can be processed as quickly as possible. This is the bug reporting section for the OpenCV library.
-->

##### System information (version)
<!-- Example
- OpenCV => 4.5.3
- Operating System / Platform => Windows 64 Bit
- Compiler => Pre-compiled for Python
-->

- OpenCV => 4.5.3
- Operating System / Platform => Windows 64 Bit
- Compiler => Pre-compiled for Python

##### Detailed description

After using the alpha channel as mask for matching, a few image matches produce Mats full of `NaN`.

I was able to reproduce this in Android and Python, so there must be some issue in the C++ code.

##### Steps to reproduce

```python
import cv2 as cv
import numpy as np

def start():
    screenshot = cv.imread('screenshot.jpg')
    screenshot = cv.cvtColor(screenshot, cv.COLOR_BGR2GRAY)
    template = cv.imread('template.png', cv.IMREAD_UNCHANGED)
    alpha = cv.extractChannel(template, 3)
    alpha = np.float32(alpha * 1.0 / 255)
    template = cv.cvtColor(template, cv.COLOR_BGRA2GRAY)

    result = cv.matchTemplate(screenshot, template, cv.TM_CCOEFF_NORMED, mask=alpha)
    print(result)

if __name__ == "__main__":
    start()
```

If the screenshot and template are converted to BGR instead of GRAY, the result is the same.

If I change the method to CCORR_NORMED, `result` will contain normal values.

Here are the images: [screenshot](https://user-images.githubusercontent.com/12360257/129605734-1b3e4002-434c-47bc-a179-c9b057a5937f.jpg) [template](https://user-images.githubusercontent.com/12360257/129605738-9281bfa2-3b78-40af-aa0d-93fffc4106b1.png)

##### Issue submission checklist

 - [x] I report the issue, it's not a question
   <!--
   OpenCV team works with forum.opencv.org, Stack Overflow and other communities
   to discuss problems. Tickets with question without real issue statement will be
   closed.
   -->
 - [x] I checked the problem with documentation, FAQ, open issues,
       forum.opencv.org, Stack Overflow, etc and have not found solution
   <!--
   Places to check:
   * OpenCV documentation: https://docs.opencv.org
   * FAQ page: https://github.com/opencv/opencv/wiki/FAQ
   * OpenCV forum: https://forum.opencv.org
   * OpenCV issue tracker: https://github.com/opencv/opencv/issues?q=is%3Aissue
   * Stack Overflow branch: https://stackoverflow.com/questions/tagged/opencv
   -->
 - [x] I updated to latest OpenCV version and the issue is still there
   <!--
   master branch for OpenCV 4.x and 3.4 branch for OpenCV 3.x releases.
   OpenCV team supports only latest release for each branch.
   The ticket is closed, if the problem is not reproduced with modern version.
   -->
 - [x] There is reproducer code and related data files: videos, images, onnx, etc
   <!--
   The best reproducer -- test case for OpenCV that we can add to the library.
   Recommendations for media files and binary files:
   * Try to reproduce the issue with images and videos in opencv_extra repository
     to reduce attachment size
   * Use PNG for images, if you report some CV related bug, but not image reader
     issue
   * Attach the image as archive to the ticket, if you report some reader issue.
     Image hosting services compress images and it breaks the repro code.
   * Provide ONNX file for some public model or ONNX file with with random weights,
     if you report ONNX parsing or handling issue. Architecture details diagram
     from netron tool can be very useful too. See https://lutzroeder.github.io/netron/
   -->