Repository: opencv
Issue: Suggestion for creation of OpenCV technical notes
Labels: category: documentation,pr: Discussion Required

In my opinion, there should be some kind of OpenCV technical notes for certain important algorithms. The idea is to summarize in a document different things like implementation details, implementation choices, deviation from the original algorithm, performance or accuracy benchmarks.

Cons:
- can almost double the development time in order to develop proper benchmark, proper method to analyze the results
- can become obsolete quickly
- more work to do, time consuming, tedious

Pros:
- implementation details can be useful for some users
- better for the research community since it should avoid confusion between OpenCV implementation and original implementation, can be citable maybe?

---

Here some examples I have in mind:

### SIFT features
- 20 years later, it is the "revival" of SIFT features, it would be great to be able to summarize the performance and the accuracy of the OpenCV SIFT implementation with the newest future developments
- references could be the Lowe's SIFT binary and the [`vl_sift`](https://www.vlfeat.org/overview/sift.html) implementation
- from this recent paper ([Image Matching across Wide Baselines: From Paper to Practice](https://arxiv.org/abs/2003.01587)), it looks like the OpenCV SIFT implementation performs correctly
![image](https://user-images.githubusercontent.com/8229425/81021313-316b9380-8e6b-11ea-9d6d-a1207d70d546.png)
- probably the dataset and the methodology for a proper accuracy benchmark will be time consuming

### SURF features
- possible alternative to SIFT is SURF
- there is an [old benchmark page](https://boofcv.org/index.php?title=Performance:SURF) (2012) comparing the OpenCV SURF implementations with other libraries
- it performs badly, this is an old benchmark but a quick look to the [history](https://github.com/opencv/opencv_contrib/commits/master/modules/xfeatures2d/src/surf.cpp) shows not so much new 
changes
- this means that benchmarks using the OpenCV SURF implementation could potentially overperform (since the OpenCV implementation has worse performance in term of accuracy compared to the SURF original implementation)
- lots of efforts are needed to check the implementation, improve it, so this is probably not soon this would happen
- there are also CUDA and OpenCL SURF implementations in OpenCV, ideally the three implementations should give the same results, so more work needed

### Harris corners detection
- in theory the Harris corners method should be rotation invariant
- an user stumbled about [this issue](https://stackoverflow.com/questions/59473193/harris-corner-detector-at-different-rotations) where the OpenCV implementation of the Harris corners is not rotation invariance
- here the reported issue

![](https://i.stack.imgur.com/CtvFN.gif)

- the thing is that the OpenCV implementation deviates from the original method since it uses for instance box blur instead of Gaussian blur for performance reason, here a paper with some info about the OpenCV implementation: [An Analysis and Implementation of the Harris CornerDetector](https://www.ipol.im/pub/art/2018/229/)
- by tweaking the parameters in `goodFeaturesToTrack()` instead of retrieving manually the corners from the output of `cornerHarris()`, better results can be achieved. Here [the link](https://i.imgur.com/PNzow4P.mp4), left is result from [DIPlib](https://github.com/DIPlib/diplib), right is OpenCV

![](https://i.imgur.com/PNzow4P.gif)

- for this kind of issue, it would have been great to have the implementation details of the OpenCV / IPP Harris corners detector summarizes somewhere

### AprilTag
- I like the AprilTag fiducial marker detector
- there is a [GSoC subject](https://github.com/opencv/opencv/wiki/GSoC_2020#idea-create-robust-visual-fiducial-tags) tackling this topic
- ideally, the OpenCV AprilTag implementation should give the exact same results than the original implementation
- here the [official repo](https://github.com/AprilRobotics/apriltag) for AprilTag 3 version described in this paper: [Flexible Layouts for Fiducial Tags](https://april.eecs.umich.edu/papers/details.php?name=krogius2019iros)
- in my opinion, if the OpenCV implementation deviates and gives poorer results than the original code, warning should be put in the documentation to warn the user that the results are inferior to the original code
- and ideally global performance and accuracy results should be documented somewhere
- I would also advice to avoid mixing ArUco and AprilTag methods in the code:
  - there are already some differences between the OpenCV ArUco and the [author ArUco](https://www.uco.es/investiga/grupos/ava/node/26) latest development (Aruco 3)
  - but I think most of the time the ArUco is mentioned, this is for the OpenCV implementation
  - to avoid confusion, it would be better in my opinion to have something like a parent class for fiducial markers and implementation classes for ArUco and AprilTag methods
- from my experience, AprilTag 3 is better for detecting tags and gives more accurate tag corners locations than OpenCV ArUco
- for instance a quick test:

- this is AprilTag 3
![AprilTag_detections](https://user-images.githubusercontent.com/8229425/81023741-a346db80-8e71-11ea-96b0-6df65b0e4843.png)

- this is OpenCV ArUco with `DICT_6x6` and `refine=None`:
![ArUco_detections_refine_0](https://user-images.githubusercontent.com/8229425/81023780-c07baa00-8e71-11ea-852c-78365eab8d72.png)

- this is OpenCV ArUco with `DICT_6x6` and `refine=Subpixel`:
![ArUco_detections_refine_1](https://user-images.githubusercontent.com/8229425/81023792-cbced580-8e71-11ea-8837-74b32dbe99cd.png)

- this is OpenCV ArUco with `DICT_6x6` and `refine=contour`:
![ArUco_detections_refine_2](https://user-images.githubusercontent.com/8229425/81023821-eacd6780-8e71-11ea-8cc2-2729ef135fc8.png)

- this is OpenCV ArUco with `DICT_6x6` and `refine=AprilTag 2`:
![ArUco_detections_refine_3](https://user-images.githubusercontent.com/8229425/81023853-0173be80-8e72-11ea-8d72-61541e399da4.png)

- this is a quick test, I did not try to tweak the ArUco parameters
- detection rate of AprilTag 3 should be a little bit better than OpenCV ArUco but accuracy in corners extraction should be definitively better with AprilTag 3
- no idea why changing the ArUco refine method gives different detection results, I am using `example_aruco_detect_markers` sample

### Pixel coordinates system
- for image resizing, warping and maybe some other operations, OpenCV treats coordinates using top-left pixel coordinates
- there are some info in the doc, but probably more details would be better in the doc since other libraries can use a different convention
- also using top-left coordinates should introduce shift, so for image analysis this is not desirable
- related issues: [9096](https://github.com/opencv/opencv/issues/9096#issuecomment-333673380), [10146](https://github.com/opencv/opencv/issues/10146), and maybe also [12680](https://github.com/opencv/opencv/issues/12680)
- with Deep Learning, I think I have read that different image resizing method can give notable difference?

---

For important new algorithms, new developments, these kind of implementation details or performance/accuracy benchmarks should be made available from the OpenCV doc. This can be simply in a Doxygen form or maybe even in pdf form for easy citation?