Repository: opencv
Issue: solvePnPRansac behavior wrt to the documentation
Labels: feature,category: documentation,category: calib3d

In OpenCV 3.2 (and since OpenCV 3.0), `solvePnPRansac` does not behave as we could expect after reading the corresponding [documentation](http://docs.opencv.org/3.2.0/d9/d0c/group__calib3d.html#ga50620f0e26e02caa2e9adc07b5fbf24e).

The function prototype is: `bool cv::solvePnPRansac(objectPoints, imagePoints, cameraMatrix, 	distCoeffs, rvec, tvec, useExtrinsicGuess = false, iterationsCount = 100, reprojectionError = 8.0, 	confidence = 0.99, inliers = noArray(), flags = SOLVEPNP_ITERATIVE)`.

The parameter `useExtrinsicGuess` will not be used and the choice of the PnP method can be misleading.

The [RANSAC](http://old.vision.ece.ucsb.edu/~zuliani/Research/RANSAC/docs/RANSAC4Dummies.pdf) algorithm implemented currently can be summarized as (when the number of points is `> 4`):

- for a certain number of iterations:
  - select randomly 5 points to construct the "Minimal Sample Sets" (MSSs) and estimate the camera pose using `SOLVEPNP_EPNP` (see [here](https://github.com/opencv/opencv/blob/3.2.0/modules/calib3d/src/solvepnp.cpp#L250)) **[Hypothesize]**
  - compute the consensus set, an inlier is a 3D point with a reprojection error (using `rvec` and `tvec` estimated with MSSs) below a threshold **[Test]**
  - "terminates when the probability of finding a better ranked CS drops below a certain threshold" ([here](https://github.com/opencv/opencv/blob/3.2.0/modules/calib3d/src/ptsetreg.cpp#L237) I think) or after reaching the maximum number of iterations
- estimate the final `rvec` and `tvec` using the inliers from the best consensus set with `flags` provided by the user (if `SOLVEPNP_P3P` it is `SOLVEPNP_EPNP` that is used instead), `useExtrinsicGuess` is ignored and sets to false ([see](https://github.com/opencv/opencv/blob/3.2.0/modules/calib3d/src/solvepnp.cpp#L287))

The documentation should be updated to reflect this behavior (`useExtrinsicGuess` is ignored and about the choice of the PnP method for the two "stages") and maybe we should add a quick description of the implementation of the RANSAC algorithm.

Better, I think the implementation can be improved:

- in the MSSs step, 5 points is used but in my opinion 4 points should be used as it is the minimum number of points required to compute the camera pose, even with `SOLVEPNP_EPNP` (see [here](http://cvlabwww.epfl.ch/~lepetit/papers/lepetit_ijcv08.pdf))

- in the MSSs step, only `SOLVEPNP_EPNP` is used (alternative could be `SOLVEPNP_P3P`, `SOLVEPNP_AP3P`, `SOLVEPNP_ITERATIVE`)

- how to deal with `useExtrinsicGuess=true` and the provided `rvec`, `tvec`?
  - it can be used when estimating the pose with the MSSs if `SOLVEPNP_ITERATIVE` is chosen
  - it can also be used for the final pose estimation with `SOLVEPNP_ITERATIVE`

- how to deal with `useExtrinsicGuess=false` and `SOLVEPNP_ITERATIVE`?
  - for the MSSs step, there is no problem as `SOLVEPNP_ITERATIVE` will call internally another function to provide an estimate of the solution
  - but for the final pose estimation, currently it will call another function that will compute the pose using all the inliers but I think we could use instead the `rvec` and `tvec` computed from the MSSs and from the best consensus set (should be tested, maybe the iterative method does not converge well if too far, that's why `useExtrinsicGuess` is set to `false`?)

- when there is only 4 points, `SOLVEPNP_P3P` is used for the MSSs. I don't know if the full RANSAC method will be done (at least 1 iteration? + the final pose computed with `SOLVEPNP_EPNP`). If this is the case, maybe we could directly call `solvePnP` with `SOLVEPNP_P3P` and return?

- how to use two differents methods for the pose computed from the MSSs and for the pose computed from the best consensus set (it is currently the case implicitly with `SOLVEPNP_EPNP` used for the MSSs and `flags` for the final pose)?

I will try to implement and see if my suppositions give better results (use 4 points, use `rvec` and `tvec` from the best consensus set). Not sure yet how to compare (generate synthetic data?, use real case data?).