Patent application EP20020294A (EP), filed 2020-06-24
Publication number: EP3929870A1
Title: METHOD FOR SUBPIXEL DISPARITY CALCULATION
Abstract: The present invention relates to a computer implemented method (400) comprising receiving a first image data representative of a first image (101L) of a field of view of a first input device generated for a scene and a second image data representative of a second image (101R) of the field of view of a second input device generated for the scene, applying said first image data and said second image data to a machine learning model, computing, by said machine learning model, an output number (104) that stands for a calculated subpixel disparity between said first image (101L) and said second image (101 R). Various examples of the present invention include a neural network (100) that produces accurate and reliable subpixel disparity estimation in a real-time. Synthetic and/or realistically generated data may be used to train the neural network (100) in a supervised manner, wherein training and prediction are performed in an end-to-end way. The neural network-based subpixel disparity calculation method (400) and system of the present invention has the advantages that accuracy can be improved, and feasibility and robustness are better.
IPC codes: G06N   3/    04            A I
G06N   3/    08            A I
G06T   7/    00            A I
G06T   7/   593            A I
Claims: number: 1
text: A computer implemented method (400) comprising:
number: 2
text: The method (400) of claim 1, wherein the first input device and the second input device each include one or more digital cameras.
number: 3
text: The method (400) of claim 1, wherein the first input device and the second input device each include one or more computer readable medium.
number: 4
text: The method (400) of claim 1, wherein said machine learning model is a neural network (100), said neural network (100) is trained in a supervised training mode.
number: 5
text: The method (400) of claim 4, wherein the supervised training mode comprising:
number: 6
text: The method (400) of claim 5, further comprising:
number: 7
text: The method (400) of claim 6, wherein backpropagation is used for updating of one or more parameters of said neural network (100).
number: 8
text: A neural network (100) comprising:
number: 9
text: The neural network (100) of claim 8, wherein said neural network (100) is trained with a realistic and/or synthetically generated data comprising a training set of pairs of training raw image data representative of a pairs of training raw images with an input number representing a ground truth subpixel disparity between each pair of training raw images.
number: 10
text: A computer program directly loadable into an internal memory of a computing device, comprising software for controlling the steps of any of the claims 1-7 when said program is run on the computing device.
number: 11
text: A computer readable medium, having a computer program recorded thereon, where the computer program is to make a computer control the steps of any of the claims 1 - 7.
Office action: search_report, dated 2021-12-29
Rejection reasons: novelty_epc_54
Prior art cited: ref_id: - Q. TIANM. N. HUHNS, COMPUT. VISION, GRAPH. IMAGE PROCESS., vol. 35, 1986, pages 220 - 233
ref_type: npl
severity: background
ref_id: - M. SHIMIZUM. OKUTOMI: "Significance and attributes of subpixel estimation on area-based matching", SYSTEMS AND COMPUTERS IN JAPAN, vol. 33, 2002, pages 1 - 10
ref_type: npl
severity: background
ref_id: - D. NEHABS. RUSINKIEWIEZJ. DAVIS: "Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1", vol. 1, 2005, IEEE, article "Improved sub-pixel stereo correspondences through symmetric refinement", pages: 557 - 563
ref_type: npl
severity: background
ref_id: - WEI TONG: "Subpixel image registration with reduced bias", OPT. LETTERS, vol. 36, no. 5, 2011, pages 763 - 765, XP001560656, DOI: 10.1364/OL.36.000763
ref_type: npl
severity: background
ref_id: - J. ZBONTARY. LECUN ET AL.: "Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches", J. MACH. LEARN. RES., vol. 17, 2016, pages 2
ref_type: npl
severity: background
ref_id: - D. CHENGD. K. PRASADM. S. BROWN: "estimation for color constancy: why spatial-domain methods work as the role of the color distribution", JOSA A, vol. 31, 2014, pages 1049 - 1058
ref_type: npl
severity: background
ref_id: - N. BANICS. LONCARIC: "Flash and Storm: Fast and Highly Practical Tone Mapping based on Naka-Rushton Equation", 13TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION THEORY AND APPLICATIONS (VISAPP 2018, 2018, pages 47 - 53
ref_type: npl
severity: background
ref_id: - D. CHENGD. K. PRASADM. S. BROWN: "llluminant estimation for color constancy: why spatial-domain methods work as the role of the color distribution", JOSA A, vol. 31, 2014, pages 1049 - 1058
ref_type: npl
severity: background
ref_id: - LIDONG YU ET AL: "Deep Stereo Matching with Explicit Cost Aggregation Sub-Architecture", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 12 January 2018 (2018-01-12), XP080852409
ref_type: npl
severity: background
ref_id: - ALEX KENDALL ET AL: "End-to-End Learning of Geometry and Context for Deep Stereo Regression", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 13 March 2017 (2017-03-13), XP080756540, DOI: 10.1109/ICCV.2017.17
ref_type: npl
severity: background
ref_id: - LEE HYUNMIN ET AL: "Real-Time Stereo Matching Network with High Accuracy", 2019 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), IEEE, 22 September 2019 (2019-09-22), pages 4280 - 4284, XP033647494, DOI: 10.1109/ICIP.2019.8803514
ref_type: npl
severity: background
ref_id: - JAMES V. STONE: "Learning Spatio-Temporal Invariances", PROCEDINGS OF THE BRITISH MACHINE VISION CONFERENCE 1994, 1 January 1994 (1994-01-01), pages 67.1 - 67.10, XP055749014, DOI: 10.5244/C.8.67
ref_type: npl
severity: background
ref_id: - "Big data analytics in the social and ubiquitous context : 5th International Workshop on Modeling Social Media, MSM 2014, 5th International Workshop on Mining Ubiquitous and Social Environments, MUSE 2014 and first International Workshop on Machine Learning for Urban Sensor Data, SenseML 2014 : revis", vol. 8753, 1 January 2014, SPRINGER, Berlin, Heidelberg, ISBN: 978-3-642-17318-9, article DANIEL SCHARSTEIN ET AL: "High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth", pages: 31 - 42, XP055278626, 032548, DOI: 10.1007/978-3-319-11752-2_3
ref_type: npl
severity: background
ref_id: - LIDONG YU ET AL, "Deep Stereo Matching with Explicit Cost Aggregation Sub-Architecture", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, (20180112), XP080852409 [X] 1-4,6-11 * abstract * * figure 2 * * Section Introduction * * Section C: Disparity computation * * Sectio Experimental Results * [I] 5
ref_type: npl
severity: novelty_destroying
ref_id: - ALEX KENDALL ET AL, "End-to-End Learning of Geometry and Context for Deep Stereo Regression", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, (20170313), doi:10.1109/ICCV.2017.17, XP080756540 [X] 1-4,6-11 * abstract * * figure 1 * * Section 1. * * Sections 3., 3.4, 3.5 * [I] 5
ref_type: npl
severity: novelty_destroying
ref_id: - LEE HYUNMIN ET AL, "Real-Time Stereo Matching Network with High Accuracy", 2019 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), IEEE, (20190922), doi:10.1109/ICIP.2019.8803514, pages 4280 - 4284, XP033647494 [X] 1-4,6-11 * abstract * * figure 2 * * Sections 2.4, 2.5 * * Sections 3., 3.1 * [I] 5
ref_type: npl
severity: novelty_destroying
ref_id: - JAMES V. STONE, "Learning Spatio-Temporal Invariances", PROCEDINGS OF THE BRITISH MACHINE VISION CONFERENCE 1994, (19940101), doi:10.5244/C.8.67, pages 67.1 - 67.10, XP055749014 [A] 1-11 * abstract * * Sections 3, 3.1, 3.2 *
ref_type: npl
severity: background
ref_id: - Daniel Scharstein ET AL, "High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth", Big data analytics in the social and ubiquitous context : 5th International Workshop on Modeling Social Media, MSM 2014, 5th International Workshop on Mining Ubiquitous and Social Environments, MUSE 2014 and first International Workshop on Machine Learning for Urban Sensor Data, SenseML 2014 : revis, Berlin, Heidelberg, Springer, (20140101), vol. 8753, pages 31 - 42, 032548, doi:10.1007/978-3-319-11752-2_3, ISBN 978-3-642-17318-9, XP055278626 [A] 1-11 * the whole document *
ref_type: npl
severity: background
Final disposition: pending (2021-12-29)