Repository: opencv
Issue: StereoBM disparity width reduced problem (SSE2)
Labels: bug,category: calib3d,affected: 3.4

<!--
If you have a question rather than reporting a bug please go to http://answers.opencv.org where you get much faster responses.
If you need further assistance please read [How To Contribute](https://github.com/opencv/opencv/wiki/How_to_contribute).

This is a template helping you to create an issue which can be processed as quickly as possible. This is the bug reporting section for the OpenCV library.
-->
##### System information (version)

<!-- Example
- OpenCV => 3.1
- Operating System / Platform => Windows 64 Bit
- Compiler => Visual Studio 2015
-->
- OpenCV => 3.1 (latest commit in master 3d5700655fca14fe075b79f913255d1ab47fb97e)
- Operating System / Platform => Ubuntu 16.04.1 LTS
- Compiler => gcc (Ubuntu 5.4.0-6ubuntu1~16.04.2)
##### Detailed description

Greetings,

I stumbled upon a problem while computing and analysing the disparity map created by StereoBM using SSE2.

For input images I used aloeL.jpg and aloeR.jpg from samples/data of OpenCV.

When computing a disparity map with `cv::Mat_<float>` as the type of the resulting disparity and with `setPreFilterCap` > 31 or `setBlockSize` > 21, I get a normal looking disparity map:

Parameters used:
- disparities: 160
- block size: 21
- pre filter cap: 32

Scale of the resulting map:
- max: 159.059
- min: -1

Result (scaled [0, 255]):
![aloe_disp_cap_32](https://cloud.githubusercontent.com/assets/4293865/17890321/80ba5a3e-6935-11e6-812b-e4a9e35af25a.png)

The problem arises when I change `setPreFilterCap` to 31. The horizontal resolution of the disparity map is reduced and the scale of the resulting map jumped by roughly 4096 times. 

Parameters used:
- disparities: 160
- block size: 21
- pre filter cap: 31

Scale of the resulting map:
- max: 651520
- min: -4096

Result (scaled [0, 255]):
![aloe_disp_cap_31](https://cloud.githubusercontent.com/assets/4293865/17890539/f5e7bfd0-6936-11e6-95ec-0c4a919fe64c.png)

When I traced the cause of this problem I found this line to be the root of the problem:

modules/calib3d/src/stereobm.cpp [line 1170-1172]:

```
#if CV_SSE2
        bool useShorts = params.preFilterCap <= 31 && params.SADWindowSize <= 21 && checkHardwareSupport(CV_CPU_SSE2);
#else
```

**[Edit 1:**
modules/calib3d/src/stereobm.cpp [line 1016-1021]:

```
#if CV_SSE2
        if( useShorts )
            findStereoCorrespondenceBM_SSE2( left_i, right_i, disp_i, cost_i, *state, ptr, row0, rows - row1 );
        else
#endif
            findStereoCorrespondenceBM( left_i, right_i, disp_i, cost_i, *state, ptr, row0, rows - row1 );
```

The condition of `useShorts` matches with my findings, so if `useShorts` is true the problem arises if not, all is good. So it seems the SSE2 implementation is broken.
**Edit 1]**

On the other hand if the type of resulting map is declared as `cv::Mat` the map's horizontal resolution is not reduced but the scale jumps by 16 times.

Parameters used:
- disparities: 160
- block size: 21
- pre filter cap: 31

Scale of the resulting map:
- max: 2544
- min: -16

Result (scaled [0, 255]):
![aloe_disp_cap_31](https://cloud.githubusercontent.com/assets/4293865/17890902/1adf5634-6939-11e6-89bb-27d909f47285.png)

IMHO this is not the supposed behaviour because changing the pre-filter parameters should not affect the width and/or scale of the disparity map.

Cheers
##### Steps to reproduce

<!-- to add code example fence it with triple backticks and optional file extension
    ```.cpp
    // C++ code example
    ```
 or attach as .txt or .zip file
-->

``` .cpp
#include <iostream>

#include <opencv2/calib3d.hpp>
#include <opencv2/highgui.hpp>

int main(int argc, char* argv[]) {

    cv::Mat leftFrame, rightFrame;

    leftFrame = cv::imread(argv[1], cv::IMREAD_GRAYSCALE);
    rightFrame = cv::imread(argv[2], cv::IMREAD_GRAYSCALE);

    int numDisparities = 160;

    cv::Ptr<cv::StereoBM> sbm;
    sbm = cv::StereoBM::create(numDisparities, 21);

    sbm->setPreFilterCap(31);

    cv::Mat_<float> disp;
    cv::Mat disp8;

    sbm->compute(leftFrame, rightFrame, disp);

    double min, max;
    cv::minMaxLoc(disp, &min, &max);
    std::cout << "Max: " << max << " Min: " << min << std::endl;

    disp.convertTo(disp8, CV_8UC1, 255. / (numDisparities - 1));
    cv::imshow("Disparity", disp8);
    cv::waitKey(0);

    return 0;
}
```