Patent application EP19177666A (EP), filed 2019-05-31
Publication number: EP3745299A1
Title: NEURAL NETWORK DEVICE AND METHOD USING A NEURAL NETWORK FOR SENSOR FUSION
Abstract: [0001]    A neural network device and a method using a neural network for sensor fusion are disclosed, wherein the neural network device includes a neural network (136) and wherein the neural network (136) is configured to: process a first grid (132) including a plurality of grid cells, wherein the first grid (132) represents at least a first portion of a field of view (108) of a first sensor (104), wherein at least one grid cell has information about an occupancy of the first portion of the field of view (108) assigned to the at least one grid cell, the information being based on data provided by the first sensor (104); process a second grid (134) including a plurality of grid cells, wherein the second grid (134) represents at least a second portion of a field of view (110) of a second sensor (106), wherein at least one grid cell has information about an occupancy of the second portion of the field of view (110) assigned to the at least one grid cell, the information being based on data provided by the second sensor (106); and fuse the processed first grid (132) with the processed second grid (134) into a fused grid (138), wherein the fused grid (138) includes information about the occupancy of the first portion of the field of view (108) of the first sensor (104) and the occupancy of the second portion of the field of view (110) of the second sensor (106).
IPC codes: G06K   9/    00            A I
G06K   9/    62            A I
G06V  10/    80            A I
G06V  10/    82            A I
G06V  20/    58            A I
Claims: number: 1
text: A neural network device comprising:
a neural network (136) configured to
number: 2
text: The neural network device of claim 1,
wherein at least a portion of the neural network (136) is implemented by one or more processors (122).
number: 3
text: The neural network device of claim 1 or 2,
wherein the neural network (136) is further configured to determine, for at least one grid cell of the fused grid (138), a probability of the occupancy of the first portion of the field of view (108) of the first sensor (104) and the second portion of the field of view (110) of the second sensor (106) assigned to the at least one grid cell of the fused grid (138).
number: 4
text: The neural network device of any one of claims 1 to 3, wherein the neural network (136) is further configured to take into account the information about an occupancy of a portion of the field of view (108) next to the first portion and/or next to the second portion when determining information about the occupancy assigned to a respective fused grid cell of the fused grid (138).
number: 5
text: The neural network device of any one of claims 1 to 4, wherein the neural network (136) comprises a first neural network portion configured to process the first grid (132);
a second neural network portion configured to process the second grid (134); and
a fusion neural network portion configured to fuse the processed first grid (132) with the processed second grid (134) into the fused grid (138).
number: 6
text: The neural network device of any one of claims 1 to 5, wherein the neural network (136) comprises an auto-encoder;
wherein the auto-encoder comprises:
number: 7
text: The neural network device of any one of claims 1 to 6, wherein the neural network (136) comprises one or more skip connections;
wherein the one or more skip connections bypass code from the first neural network portion and/or the second neural network portion to the fusion neural network portion.
number: 8
text: The neural network device of any one of claims 1 to 7,
wherein a first inverse sensor model is applied to the data provided by the first sensor (104) to provide the information about the occupancy of the first portion of the field of view (108) of the first sensor (104); and wherein a second inverse sensor model is applied to the data provided by the second sensor (106) to provide the information about the occupancy of the second portion of the field of view (110) of the second sensor (106).
number: 9
text: The neural network device of claim 1 to 8,
wherein the neural network (136) is further configured to determine a free space within the fused grid (138) based on the fused grid (138).
number: 10
text: A system, comprising:
number: 11
text: A vehicle (100), comprising:
a driver assistance system comprising the system of claim 10.
number: 12
text: A method, comprising:
a neural network (136):
number: 13
text: The method of claim 12,
wherein at least a portion of the neural network (136) is implemented by one or more processors.
number: 14
text: The method of any one of claims 12 or 13,
wherein the neural network (136) determines, for at least one grid cell of the fused grid (138), a probability of the occupancy of the first portion of the field of view (108) of the first sensor (104) and the second portion of the field of view (110) of the second sensor (106) assigned to the at least one grid cell of the fused grid (138).
number: 15
text: The method of any one of claims 12 to 14,
wherein a first inverse sensor model is applied to the data provided by the first sensor (104) to provide the information about the occupancy of the first portion of the field of view (108) of the first sensor (104); and wherein a second inverse sensor model is applied to the data provided by the second sensor (106) to provide the information about the occupancy of the second portion of the field of view (110) of the second sensor (106).
Office action: search_report, dated 2020-12-02
Rejection reasons: novelty_epc_54
Prior art cited: ref_id: - ZOLT�N FAZEKAS ET AL: "ANN-based Classification of Urban Road Environments from Traffic Sign and Crossroad Data", 1 January 2018 (2018-01-01), pages 2018 - 83, XP055641263, Retrieved from the Internet <URL:http://eprints.sztaki.hu/9644/1/Fazekas_83_30550287_ny.pdf> [retrieved on 20191111], DOI: 10.12700/APH.15.8.2018.8.4
ref_type: npl
severity: background
ref_id: - Zolt�n Fazekas ET AL, "ANN-based Classification of Urban Road Environments from Traffic Sign and Crossroad Data", doi:10.12700/APH.15.8.2018.8.4, (20180101), pages 2018 - 83, URL: http://eprints.sztaki.hu/9644/1/Fazekas_83_30550287_ny.pdf, (20191111), XP055641263 [X] 1-3,9-12 * sections 1-5 * * abstract; figures 1-4 * [I] 4-8,13-15
ref_type: npl
severity: novelty_destroying
Final disposition: pending (2020-12-02)