Patent application EP21176774A (EP), filed 2021-05-31
Publication number: EP4099210A1
Title: METHOD FOR TRAINING A NEURAL NETWORK FOR SEMANTIC IMAGE SEGMENTATION
Abstract: The present invention is related to a method, a computer program, and an apparatus for training a neural network for semantic image segmentation. The invention is further related to an in-car control unit or a backend system, which make use of such a method or apparatus, and to a vehicle comprising such an in-car control unit. In a first step, image data of a sequence of image frames are received (10). Then a frame-based evaluation of semantic segmentation predictions of one or more objects in individual image frames is performed (11). Furthermore, a sequence-based evaluation of temporal characteristics of semantic segmentation predictions of said one or more objects in at least two image frames is performed (12). The results of the frame-based evaluation and the sequence-based evaluation are combined (13).
IPC codes: G06K   9/    00            A I
G06K   9/    62            A I
G06T   7/    00            A I
G06F  18/  2413            A I
G06T   7/    11            A I
G06V  20/    56            A I
Claims: number: 1
text: Method for training a neural network for semantic image segmentation, comprising:
number: 2
text: The method according to claim 1, wherein performing (12) the sequence-based evaluation comprises:
number: 3
text: The method according to claim 2, wherein a temporal consistency loss is determined for evaluating the semantic segmentation prediction.
number: 4
text: The method according to claim 2 or 3, wherein performing a frame-based evaluation of semantic segmentation predictions of one or more objects in individual image frames comprises:
number: 5
text: The method according to claim 4, wherein a total loss is determined by combining (13) the results of the frame-based evaluation and the sequence-based evaluation.
number: 6
text: The method according to claim 5, wherein the total loss is a weighted sum of the cross-entropy loss and the temporal consistency loss.
number: 7
text: The method according to claim 6, wherein the total loss is given by  J t total = 1 − α J t CE + αJ t TC
number: 8
text: The method according to claim 7, wherein loss weight parameter α is variable and can be adapted to address the trade-off between the sequence-based evaluation and the frame-based evaluation.
number: 9
text: A computer program comprising instructions which, when executed by a computer, cause the computer to perform a method according to any of claims 1 to 8.
number: 10
text: An apparatus (20) for training a neural network for semantic image segmentation, the apparatus (20) comprising:
number: 11
text: The apparatus (20) according to claim 10, wherein the means for performing the sequence-based evaluation comprises:
number: 12
text: The apparatus (20) according to claim 10 or 11, wherein the means for performing a frame-based evaluation of semantic segmentation predictions of one or more objects in individual image frames comprises:
number: 13
text: An in-car control unit, wherein the in-car control unit comprises an apparatus according to claim 10 or is configured to perform a method according to any of claims 1 to 8.
number: 14
text: A backend system, wherein the backend system comprises an apparatus according to claim 10 or is configured to perform a method according to any of claims 1 to 8.
number: 15
text: A vehicle (40), wherein the vehicle (40) comprises an in-car control unit according to claim 13.
Office action: search_report, dated 2022-12-07
Rejection reasons: novelty_epc_54
Prior art cited: ref_id: US2021073589A1
ref_type: patent
severity: background
ref_id: US2020327334A1
ref_type: patent
severity: background
ref_id: DE102017130194A1
ref_type: patent
severity: background
ref_id: - LIU ET AL.: "Efficient Semantic Video Segmentation With Per-Frame Inference", PROC. OF ECCV, pages 1 - 22
ref_type: npl
severity: background
ref_id: - S. VARGHESE ET AL.: "Unsupervised Temporal Consistency Metric for Video Segmentation in Highly-Automated Driving", PROC. OF CVPR -WORKSHOPS, June 2020 (2020-06-01), pages 336 - 337
ref_type: npl
severity: background
ref_id: [DXI]     - LIU YIFAN ET AL: "Efficient Semantic Video Segmentation with Per-Frame Inference", 23 August 2020, ECCV 2020 WORKSHOP PROCEEDINGS, SPRINGER, LECTURE NOTES IN COMPUTER SCIENCE BOOK SERIES (LNCS, VOLUME 12535), ISSN: 0302-9743, XP047568661
ref_type: npl
severity: background
ref_id: - VARGHESE SERIN ET AL: "Unsupervised Temporal Consistency Metric for Video Segmentation in Highly-Automated Driving", 2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW), IEEE, 14 June 2020 (2020-06-14), pages 1369 - 1378, XP033799076, DOI: 10.1109/CVPRW50498.2020.00176
ref_type: npl
severity: background
ref_id: - VARGHESE SERIN ET AL: "An Unsupervised Temporal Consistency (TC) Loss to Improve the Performance of Semantic Segmentation Networks", 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW), 1 June 2021 (2021-06-01), pages 12 - 20, XP055856344, ISBN: 978-1-6654-4899-4, DOI: 10.1109/CVPRW53098.2021.00010
* the whole document *
ref_type: npl
severity: background
ref_id: - LIU YIFAN ET AL, Efficient Semantic Video Segmentation with Per-Frame Inference, ECCV 2020 workshop proceedings, Springer, Lecture Notes in Computer Science book series (LNCS, volume 12535), (20200823), ISSN 0302-9743, XP047568661 [DX] 1-12 * abstract * * Section 2.1 * * Section 2.2 * * Section 2.3 * * Section 4.2; figure 4 * [I] 13-15
ref_type: npl
severity: novelty_destroying
ref_id: - VARGHESE SERIN ET AL, "Unsupervised Temporal Consistency Metric for Video Segmentation in Highly-Automated Driving", 2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW), IEEE, (20200614), doi:10.1109/CVPRW50498.2020.00176, pages 1369 - 1378, XP033799076 [DA] 1,9-10 * Section 3; figure 2 * * Section 4.4 *
ref_type: npl
severity: background
ref_id: [ ]     - VARGHESE SERIN ET AL, "An Unsupervised Temporal Consistency (TC) Loss to Improve the Performance of Semantic Segmentation Networks", 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW), (20210601), doi:10.1109/CVPRW53098.2021.00010, ISBN 978-1-6654-4899-4, pages 12 - 20, XP055856344 [ ] * the whole document *
ref_type: npl
severity: background
Final disposition: pending (2022-12-07)