Repository: opencv
Issue: projectpoints distortion handling (and possibly ignore some points in the projection)
Labels: category: calib3d,RFC

##### System information (version)
Example
- OpenCV => 4.3
- Operating System / Platform => Ubuntu
- Compiler => gcc version 4.8.5

##### Detailed description

I'm opening this issue after the question I put a couple of days ago in the answers website. 
I'll copy here only the relevant part of the question related to my answer, since after a couple of days thinking about it I do not really believe there is a "real" error in the code, but rather a simplification or something that nobody thought about simply because is a very specific application. 

To recap, I was trying to project 3D points to an image that has still the distortion and for some reason I don't want to correct the distortion before applying the projection, you can find more details on the question (this is the link to the question: https://answers.opencv.org/question/231939/projectpoints-distortion ). The important thing is that I have the intrinsic and extrinsic parameters of the system.
I used the _projectpoints_  routine for this process and I have found what I initially thought as a bug in the code, that is, 3D points that appear in a "random" way in the image plane. In my first attempt to understand what was going on, I've created a 3D plane in front of the camera and used the system parameters to project them; then I gradually started to move this plane near the camera. The following images are the result of this experiment

![image](https://user-images.githubusercontent.com/3965650/86654187-0d9d0900-bfe6-11ea-8db3-a1a317ad7a69.png)
![image](https://user-images.githubusercontent.com/3965650/86654480-45a44c00-bfe6-11ea-938c-9bf6a664649d.png)
![image](https://user-images.githubusercontent.com/3965650/86654514-4a690000-bfe6-11ea-9a4b-fda8ff4e6707.png)
![image](https://user-images.githubusercontent.com/3965650/86654529-4dfc8700-bfe6-11ea-94ba-1cf1d0cc3b09.png)

As you can see, as the plane come closer than a certain threshold, some points start to come back into the image plane. 

Now, as you can read in the answer's post, the fact was even more strange because there're some opened questions in the answer's website saying ([like this one](https://answers.opencv.org/question/227094/cvprojectpoints-giving-bad-results-when-undistorted-2d-projections-are-far/)) stating that in some cases using less distortion coefficients does not introduce "errors". 

I've been digging into the OpenCV code, compiling all the necessary in debug mode and doing a step-by-step debuggin session of the same implementation I previously had in Python now in C++ ([here is the code](https://github.com/trigal/calib-test/blob/master/projectpoints.cpp)). 

My conclusion is that is not a real bug, it is simply the way in which the routine works. But, there's the issue of "returning" points, or whatever name we want to give to those points. 


The fact is that those "strange" points of the images above are nothing less than these ones I've created using a much higher value for K4 (the 5th element of distorsion) in the following image:

![image](https://user-images.githubusercontent.com/3965650/86655892-5b664100-bfe7-11ea-8b7a-d21d33e55088.png)

The issue than can be translated into: can we find a way to automagically remove those points behind some kind of, let's say, distortion limit?

I don't really know how to simply do that, maybe the jacobian evaluated for every point can be used, or maybe not, but certainly it would be an interesting extension of the current behavior of this function.

At this time, I do not have time to investigate for an elegant solution and my way to solve this issue was just to compare the position of the distorsion-free point (using all K's set to zero) and checking whether the point falls inside a specific region, but this is a way too trivial workaround for a real issue.

To summarize the issue, I think that the following two gifs can explain better than all my words.

OpenCV code without any "extra" filtering:
![image](https://github.com/trigal/trigal.github.io/blob/master/images/opencv3d2dprojections/test6.gif?raw=true)

Same code but with some workaround before drawing the points
![image](https://github.com/trigal/trigal.github.io/blob/master/images/opencv3d2dprojections/test9.gif?raw=true)


##### Steps to reproduce
The link to reproduce the code is here: https://github.com/trigal/calib-test/blob/master/projectpoints.cpp and you'll see the super trivial workaround I've used to exclude those points that are coming back to the image plane. Kitti is the image (I'm using images from the KITTI dataset), and the trick is just to check whether the points without the distortion are too far from the image plane limits, and since there is a 1-1 correspondence between the points with/without the distortion, I can simply select those I think are good. 

```.cpp
double x_offset = -150;
double y_offset = -150;
if (
((imagePointsUndistort.at(i).x>.00+x_offset) && (imagePointsUndistort.at(i).y>0.0+y_offset)) &&
((imagePointsUndistort.at(i).x<kitti.size[1]-x_offset) && (imagePointsUndistort.at(i).y<kitti.size[0]-y_offset))
)
```

Bests,
Augusto

##### Issue submission checklist

 - [X] I report the issue, it's not a question
   <!--
   OpenCV team works with answers.opencv.org, Stack Overflow and other communities
   to discuss problems. Tickets with question without real issue statement will be
   closed.
   -->
 - [X] I checked the problem with documentation, FAQ, open issues,
       answers.opencv.org, Stack Overflow, etc and have not found solution
   <!--
   Places to check:
   * OpenCV documentation: https://docs.opencv.org
   * FAQ page: https://github.com/opencv/opencv/wiki/FAQ
   * OpenCV forum: https://answers.opencv.org
   * OpenCV issue tracker: https://github.com/opencv/opencv/issues?q=is%3Aissue
   * Stack Overflow branch: https://stackoverflow.com/questions/tagged/opencv
   -->
 - [X] I updated to latest OpenCV version and the issue is still there
   <!--
   master branch for OpenCV 4.x and 3.4 branch for OpenCV 3.x releases.
   OpenCV team supports only latest release for each branch.
   The ticket is closed, if the problem is not reproduced with modern version.
   -->
 - [X] There is reproducer code and related data files: videos, images, onnx, etc
   <!--
   The best reproducer -- test case for OpenCV that we can add to the library.
   Recommendations for media files and binary files:
   * Try to reproduce the issue with images and videos in opencv_extra repository
     to reduce attachment size
   * Use PNG for images, if you report some CV related bug, but not image reader
     issue
   * Attach the image as archite to the ticket, if you report some reader issue.
     Image hosting services compress images and it breaks the repro code.
   * Provide ONNX file for some public model or ONNX file with with random weights,
     if you report ONNX parsing or handling issue. Architecture details diagram
     from netron tool can be very useful too. See https://lutzroeder.github.io/netron/
   -->