Patent application EP21153747A (EP), filed 2021-01-27
Publication number: EP3923201A1
Title: APPARATUS AND METHOD WITH NEURAL NETWORK
Abstract: A neural network-implementing neuromorphic device includes: a memory configured to store one or more instructions; an on-chip memory comprising a crossbar array circuit including synapse circuits; and one or more processors configured to, by executing instructions to drive a neural network, store binary weight values of the neural network in the synapse circuits, obtain an input feature map from the memory, convert the input feature map into temporal domain binary vectors, provide the temporal domain binary vectors as input values of the crossbar array circuit, and output an output feature map by performing, using the crossbar array circuit, a convolution computation between the binary weight values and the temporal domain binary vectors.
IPC codes: G06N   3/    04            A I
G06N   3/    06            A I
Claims: number: 1
text: A neural network device, the neural network device comprising:
number: 2
text: The device of claim 1, wherein, for the outputting of the output feature map, the one or more processors are further configured to output the output feature map by performing batch normalization on a result of the convolution computation.
number: 3
text: The device of claim 2, wherein, for the performing of the batch normalization, the one or more processors are further configured to
calculate a modified scale value by multiplying an initial scale value of the batch normalization by an average value of absolute values of initial weight values and dividing a result thereof by a number of elements included in each temporal domain binary vector, and
perform the batch normalization based on the modified scale value.
number: 4
text: The device of claim 3, wherein, for the converting of the input feature map, the one or more processors are further configured to convert the input feature map into the temporal domain binary vectors based on quantization levels of the input feature map.
number: 5
text: The device of claim 4, wherein, for the converting of the input feature map, the one or more processors are further configured to divide a range between a maximum value and a minimum value determined for the temporal domain binary vectors to be input to the neural network N quantization levels, wherein N is a natural number, and convert activations of the input feature map into the temporal domain binary vectors based on the quantization levels to which the activations correspond, or
wherein, for the dividing of the range, the one or more processors are further configured to divide the range between the maximum value and the minimum value into non-linear quantization levels.
number: 6
text: The device of claim 3, wherein, for the outputting of the output feature map, the one or more processors are further configured to
perform a multiplication computation by multiplying each of bias values applied to the neural network by the initial scale value, and
outputting the output feature map by determining the output feature map based on a result of the multiplication computation.
number: 7
text: The device of claim 3, wherein, for the outputting of the output feature map, the one or more processors are further configured to output the output feature map by performing the batch normalization on a result of the convolution computation and applying an activation function to a result of the batch normalization.
number: 8
text: The device of one of claims 1 to 7, wherein
the device is a neuromorphic device further comprising an on-chip memory comprising a crossbar array circuit including synapse circuits, and
the one or more processors are configured to
number: 9
text: A processor-implemented method of implementing a neural network in a neuromorphic device, the method comprising:
number: 10
text: A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform a processor-implemented method of implementing a neural network in a neuromorphic device, the method comprising:
number: 11
text: The computer-readable storage medium of claim 10, wherein, for the converting of the input feature map, the one or more processors are configured to generate one of the temporal domain binary vectors by converting an input activation of the input feature map into elements of either a maximum or a minimum binary value.
number: 12
text: The computer-readable storage medium of claim 11, wherein a temporal sequence of the maximum binary values and the minimum binary values of the generated temporal domain binary vector is determined based on a quantization level of the input activation.
number: 13
text: The computer-readable storage medium of one of claims 10 to 12, wherein, for the storing of the weight values, the one or more processors are configured to:
number: 14
text: The computer-readable storage medium of claim 13, wherein the initial weight values are of connections between nodes of a previous layer of the neural network and a node of a current layer of the neural network.
number: 15
text: The computer-readable storage medium of one of claims 10 to 14, wherein
the device is any one of a personal computer, PC, a server device, a mobile device, and a smart device,
the input feature map corresponds to either one of input image data and input audio data, and
the one or more processors are configured to perform any one of image recognition, image classification, and voice recognition based on the generated output feature map.
Office action: search_report, dated 2021-12-15
Prior art cited: ref_id: - SHAAHIN ANGIZI ET AL: "IMC", 20170717; 1077952576 - 1077952576, 17 July 2017 (2017-07-17), pages 1 - 8, XP058407109, ISBN: 978-1-4503-6442-3, DOI: 10.1145/3183584.3183613
ref_type: npl
severity: background
ref_id: - HYUNGJUN KIM ET AL: "In-memory batch-normalization for resistive memory based binary neural network hardware", 20190121; 1077952576 - 1077952576, 21 January 2019 (2019-01-21), pages 645 - 650, XP058424650, ISBN: 978-1-4503-6007-4, DOI: 10.1145/3287624.3287718
ref_type: npl
severity: background
ref_id: - NI LEIBIN ET AL: "An energy-efficient and high-throughput bitwise CNN on sneak-path-free digital ReRAM crossbar", 2017 IEEE/ACM INTERNATIONAL SYMPOSIUM ON LOW POWER ELECTRONICS AND DESIGN (ISLPED), IEEE, 24 July 2017 (2017-07-24), pages 1 - 6, XP033142372, DOI: 10.1109/ISLPED.2017.8009177
ref_type: npl
severity: background
ref_id: - "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks : 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV", 9 May 2016, article RASTEGARI MOHAMMAD ET AL: "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks : 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV", pages: 1 - 17, XP055843447, DOI: 10.1007/978-3-319-46493-0_32
ref_type: npl
severity: background
ref_id: US2019294413A1
ref_type: patent
severity: background
ref_id: US2019347550A1
ref_type: patent
severity: background
ref_id: - SHAAHIN ANGIZI ET AL, "IMC", 20170717; 1077952576 - 1077952576, (20170717), doi:10.1145/3183584.3183613, ISBN 978-1-4503-6442-3, pages 1 - 8, XP058407109 [I] 1-15 * abstract; figures 3-6 * * paragraphs  [0001] - [0004] *
ref_type: npl
severity: background
ref_id: - HYUNGJUN KIM ET AL, "In-memory batch-normalization for resistive memory based binary neural network hardware", 20190121; 1077952576 - 1077952576, (20190121), doi:10.1145/3287624.3287718, ISBN 978-1-4503-6007-4, pages 645 - 650, XP058424650 [I] 1,2,8-10,15 * abstract; figures 1-3 * * paragraphs  [000I] - [0III] * [A] 3-7,11-14
ref_type: npl
severity: background
ref_id: - NI LEIBIN ET AL, "An energy-efficient and high-throughput bitwise CNN on sneak-path-free digital ReRAM crossbar", 2017 IEEE/ACM INTERNATIONAL SYMPOSIUM ON LOW POWER ELECTRONICS AND DESIGN (ISLPED), IEEE, (20170724), doi:10.1109/ISLPED.2017.8009177, pages 1 - 6, XP033142372 [I] 1,2,8-10 * abstract; figure 3 * * paragraphs  [00II] - [00IV] * [A] 3-7,11-15
ref_type: npl
severity: background
ref_id: - Rastegari Mohammad ET AL, "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks : 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV", XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks : 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, (20160509), pages 1 - 17, doi:10.1007/978-3-319-46493-0_32, XP055843447 [I] 1,2 * abstract; figure 3 * * paragraph  [0003] * [A] 3-15
ref_type: npl
severity: background
Final disposition: pending (2021-12-15)