Repository: opencv
Issue: initUndistortRectifyMap does not handle distortion correctly
Labels: category: calib3d

<!--
If you have a question rather than reporting a bug please go to http://answers.opencv.org where you get much faster responses.
If you need further assistance please read [How To Contribute](https://github.com/opencv/opencv/wiki/How_to_contribute).

Please:

* Read the documentation to test with the latest developer build.
* Check if other person has already created the same issue to avoid duplicates. You can comment on it if there already is an issue.
* Try to be as detailed as possible in your report.
* Report only one problem per created issue.


This is a template helping you to create an issue which can be processed as quickly as possible. This is the bug reporting section for the OpenCV library.
-->

##### System information (version)

- OpenCV => 3.4.1
- Operating System / Platform => Windows 64 Bit
- Compiler => Visual Studio 2013 + Python Bindings

##### Detailed description

When using `stereoRectify`, the resulting projection matrices depend on the input distortion, which in my understanding should not happen. 

I encountered the problem when using `stereoRectify` on a new pair of cameras. I have used the whole pipeline successfully before on cameras of the same type. This time, the results returned by `stereRectify` are wrong. The camera centers in the new projection matrices have huge values (see values below).

To locate the problem, I run the rectification and undistortion in two individual steps, instead of using `stereRectify`. If I first undistort the images and then feed the new undistorted images and a vector filled with zeros as distortion into `stereRectify` I obtain good results. See the Steps to reproduce.

##### Steps to reproduce
My data input:

```
M_L = [[ 430.            0.          200.23631705]
 [   0.          430.          199.36536477]
 [   0.            0.            1.        ]]
M_L.dtype = float64

dist_L = [-0.52501353  0.50232865  0.01204062 -0.03013492 -0.42135838]
dist_L.dtype = float64

M_R = [[ 430.            0.          199.64316951]
 [   0.          430.          199.40004749]
 [   0.            0.            1.        ]]
M_R.dtype = float64

dist_R = [-0.44271713  0.01572439  0.01272985 -0.02997997  0.31224365]
dist_R.dtype = float64

R = [[ 0.99757046  0.06468998 -0.02585312]
 [-0.06438798  0.99784853  0.01234882]
 [ 0.02659635 -0.01065419  0.99958948]]
R.dtype = float64

T = [[-0.23299669]
 [-3.05171467]
 [ 0.43562338]]
T.dtype = float64
```

The results of `stereoRectify` using 

```
size = image_L.shape[:2]
R_rectify_L, R_rectify_R, P_L, P_R, Q, validPixROI_L, validPixROI_R = cv.stereoRectify(M_L,
            dist_L, M_R, dist_R, size, R, T)
P_L = [[  4.30000000e+02   0.00000000e+00   2.68848224e+03   0.00000000e+00]
 [  0.00000000e+00   4.30000000e+02  -2.09073371e+03   0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   1.00000000e+00   0.00000000e+00]]
P_L.dtype = float64

P_R = [[  4.30000000e+02   0.00000000e+00   2.68848224e+03   0.00000000e+00]
 [  0.00000000e+00   4.30000000e+02  -2.09073371e+03  -1.32932034e+03]
 [  0.00000000e+00   0.00000000e+00   1.00000000e+00   0.00000000e+00]]
P_R.dtype = float64
```

The values for pixel centers in `P_L` and `P_R` are factor 10 to large.  Using these results as in

```
map1_L, map2_L = cv.initUndistortRectifyMap(M_L, dist_L,
                                            R_rectify_L, P_L, size, cv.CV_32FC1)
image_rectified_L = cv.remap(image_L, map1_L, map2_L, cv.INTER_LINEAR)

map1_R, map2_R = cv.initUndistortRectifyMap(M_R, dist_R,
                                            R_rectify_R, P_R, size, cv.CV_32FC1)
image_rectified_R = cv.remap(image_R, map1_R, map2_R, cv.INTER_LINEAR)
```
gives completely black images.  **However**, if I separate the undistort step like:

```
map1_L, map2_L = cv.initUndistortRectifyMap(M_L, dist_L, np.eye(3),
                                            M_L, size, cv.CV_32FC1)
image_interp_L = cv.remap(image_L, map1_L, map2_L, cv.INTER_LINEAR)

map1_R, map2_R = cv.initUndistortRectifyMap(M_R, dist_R, np.eye(3),
                                            M_R, size, cv.CV_32FC1)
image_interp_R = cv.remap(image_R, map1_R, map2_R, cv.INTER_LINEAR)
```

and than use `stereoRectify` on the undistorted images as in:

```
R_rectify_L, R_rectify_R, P_L, P_R, Q, validPixROI_L, validPixROI_R = cv.stereoRectify(M_L, 
                                    np.zeros((1,5)), M_R, np.zeros((1,5)), size, R, T)

map1_L, map2_L = cv.initUndistortRectifyMap(M_L, np.zeros((1,5)),
                                            R_rectify_L, P_L, size, cv.CV_32FC1)
image_rectified_L = cv.remap(image_interp_L, map1_L, map2_L, cv.INTER_LINEAR)
​
map1_R, map2_R = cv.initUndistortRectifyMap(M_R, np.zeros((1,5)),
                                            R_rectify_R, P_R, size, cv.CV_32FC1)
image_rectified_R = cv.remap(image_interp_R, map1_R, map2_R, cv.INTER_LINEAR)
```

The images and resulting projection matrices look fine.

```
P_L = [[ 430.            0.          199.88677025    0.        ]
 [   0.          430.          126.72800636    0.        ]
 [   0.            0.            1.            0.        ]]
P_L.dtype = float64

P_R = [[  4.30000000e+02   0.00000000e+00   1.99886770e+02   0.00000000e+00]
 [  0.00000000e+00   4.30000000e+02   1.26728006e+02  -1.32932034e+03]
 [  0.00000000e+00   0.00000000e+00   1.00000000e+00   0.00000000e+00]]
P_R.dtype = float64
```

![right](https://user-images.githubusercontent.com/43137209/45295686-de0f3980-b4ff-11e8-8214-22ebfdc7b3a2.png)
![left](https://user-images.githubusercontent.com/43137209/45295687-dea7d000-b4ff-11e8-8be1-ed0954fca46c.jpeg)

ps: the images are acutally top and bottom and not left and right.