Patent application EP20214398A (EP), filed 2020-12-16
Publication number: EP3926544A1
Title: SYSTEM AND METHOD OF GESTURE RECOGNITION USING A RESERVOIR BASED CONVOLUTIONAL SPIKING NEURAL NETWORK
Abstract: This disclosure relates to method of identifying a gesture from a plurality of gestures using a reservoir based convolutional spiking neural network. A two-dimensional spike streams is received from neuromorphic event camera as an input. The two-dimensional spike streams associated with at least one gestures from a plurality of gestures is preprocessed to obtain plurality of spike frames. The plurality of spike frames is processed by a multi layered convolutional spiking neural network to learn plurality of spatial features from the at least one gesture. A filter block is deactivated from the plurality of filter blocks corresponds to at least one gesture which are not currently being learnt. A spatio-temporal features is obtained by allowing the spike activations from CSNN layer to flow through the reservoir. The spatial feature is classified by classifier from the CSNN layer and the spatio-temporal features from the reservoir to obtain set of prioritized gestures.
IPC codes: G06N   3/    04            A I
G06N   3/    08            A I
G06F   3/    01            A I
G06N   3/   044            A I
G06N   3/   045            A I
G06N   3/   047            A I
G06N   3/   049            A I
G06N   3/   082            A I
G06N   3/   088            A I
Claims: number: 1
text: A processor implemented method of identifying a gesture from a plurality of gestures using a reservoir based convolutional spiking neural network, comprising:
number: 2
text: The processor implemented method of claim 1, wherein the spike streams are compressed per neuronal level by accumulating spikes at a sliding window of time, to obtain a plurality of output frames with reduced time granularity.
number: 3
text: The processor implemented method of claim 1, wherein a plurality of learned different spatially co-located features are distributed on the plurality of filters from the plurality of filter blocks.
number: 4
text: The processor implemented method of claim 1, wherein a special node between filters of the filter block is configured to switch between different filters based on an associated decay constant to distribute learning of different spatially co-located features on the different filters.
number: 5
text: The processor implemented method of claim 1, wherein a plurality of weights of a synapse between input and the CSNN layer are learned using an unsupervised two trace STDP learning rule upon at least one spiking activity of the input layer, wherein the reservoir comprises a sparse random cyclic connectivity which acts as a random projection of the input spikes to an expanded spatio-temporal embedding.
number: 6
text: A system (100) to identify a gesture from a plurality of gestures using a reservoir based convolutional spiking neural network, comprising:
number: 7
text: The system (100) of claim 6, wherein the spike streams are compressed per neuronal level by accumulating spikes at a sliding window of time, to obtain a plurality of output frames with reduced time granularity.
number: 8
text: The system (100) of claim 6, wherein plurality of learned different spatially co-located features are distributed on the plurality of filters from the plurality of filter blocks.
number: 9
text: The system (100) of claim 6, wherein a special node between filters of the filter block is configured to switch between different filters based on an associated decay constant to distribute learning of different spatially co-located features on the different filters.
number: 10
text: The system (100) of claim 6, wherein a plurality of weights of a synapse between input and the CSNN layer are learned using an unsupervised two trace STDP learning rule upon at least one spiking activity of the input layer, wherein the reservoir comprises a sparse random cyclic connectivity which acts as a random projection of the input spikes to an expanded spatio-temporal embedding.
number: 11
text: One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors perform actions comprising:
number: 12
text: The one or more non-transitory machine-readable information storage mediums of claim 11, wherein the spike streams are compressed per neuronal level by accumulating spikes at a sliding window of time, to obtain a plurality of output frames with reduced time granularity.
number: 13
text: The one or more non-transitory machine-readable information storage mediums of claim 11, wherein a plurality of learned different spatially co-located features are distributed on the plurality of filters from the plurality of filter blocks.
number: 14
text: The one or more non-transitory machine-readable information storage mediums of claim 11, wherein a special node between filters of the filter block is configured to switch between different filters based on an associated decay constant to distribute learning of different spatially co-located features on the different filters.
number: 15
text: The one or more non-transitory machine-readable information storage mediums of claim 11, wherein a plurality of weights of a synapse between input and the CSNN layer are learned using an unsupervised two trace STDP learning rule upon at least one spiking activity of the input layer, wherein the reservoir comprises a sparse random cyclic connectivity which acts as a random projection of the input spikes to an expanded spatio-temporal embedding.
Office action: search_report, dated 2021-12-22
Prior art cited: ref_id: - YANG XU ET AL: "Efficient Reservoir Encoding Method for Near-Sensor Classification with Rate-Coding Based Spiking Convolutional Neural Networks", 26 June 2019, ADVANCES IN DATABASES AND INFORMATION SYSTEMS; [LECTURE NOTES IN COMPUTER SCIENCE; LECT.NOTES COMPUTER], SPRINGER INTERNATIONAL PUBLISHING, CHAM, PAGE(S) 242 - 251, ISBN: 978-3-319-10403-4, XP047511778
ref_type: npl
severity: background
ref_id: - AMIR ARNON ET AL: "A Low Power, Fully Event-Based Gesture Recognition System", 2017 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE COMPUTER SOCIETY, US, 21 July 2017 (2017-07-21), pages 7388 - 7397, XP033250107, ISSN: 1063-6919, [retrieved on 20171106], DOI: 10.1109/CVPR.2017.781
ref_type: npl
severity: background
ref_id: - JEAN-MATTHIEU MARO ET AL: "Event-based Gesture Recognition with Dynamic Background Suppression using Smartphone Computational Capabilities", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 19 November 2018 (2018-11-19), XP080936664
ref_type: npl
severity: background
ref_id: - RUTHVIK VAILA ET AL: "Feature Extraction using Spiking Convolutional Neural Networks", NEUROMORPHIC SYSTEMS, ACM, 2 PENN PLAZA, SUITE 701NEW YORKNY10121-0701USA, 23 July 2019 (2019-07-23), pages 1 - 8, XP058442033, ISBN: 978-1-4503-7680-8, DOI: 10.1145/3354265.3354279
ref_type: npl
severity: background
ref_id: - DONG MENG ET AL: "Unsupervised speech recognition through spike-timing-dependent plasticity in a convolutional spiking neural network", PLOS ONE, vol. 13, no. 11, 29 November 2018 (2018-11-29), pages e0204596, XP055786795, DOI: 10.1371/journal.pone.0204596
ref_type: npl
severity: background
ref_id: - THIELE JOHANNES C ET AL: "A Timescale Invariant STDP-Based Spiking Deep Network for Unsupervised Online Feature Extraction from Event-Based Sensor Data", 2018 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN), IEEE, 8 July 2018 (2018-07-08), pages 1 - 8, XP033419468, DOI: 10.1109/IJCNN.2018.8489666
ref_type: npl
severity: background
ref_id: - YANG XU ET AL, Efficient Reservoir Encoding Method for Near-Sensor Classification with Rate-Coding Based Spiking Convolutional Neural Networks, ADVANCES IN DATABASES AND INFORMATION SYSTEMS; [LECTURE NOTES IN COMPUTER SCIENCE; LECT.NOTES COMPUTER], SPRINGER INTERNATIONAL PUBLISHING, CHAM, PAGE(S) 242 - 251, (20190626), ISBN 978-3-319-10403-4, XP047511778 [A] 1-15 * abstract * * Sections 1 - 3 *
ref_type: npl
severity: background
ref_id: - AMIR ARNON ET AL, "A Low Power, Fully Event-Based Gesture Recognition System", 2017 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE COMPUTER SOCIETY, US, doi:10.1109/CVPR.2017.781, ISSN 1063-6919, (20170721), pages 7388 - 7397, (20171106), XP033250107 [A] 1-15 * abstract * * Sections 1, 4, 6 *
ref_type: npl
severity: background
ref_id: - JEAN-MATTHIEU MARO ET AL, "Event-based Gesture Recognition with Dynamic Background Suppression using Smartphone Computational Capabilities", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, (20181119), XP080936664 [A] 1-15 * abstract * * Section 1 *
ref_type: npl
severity: background
ref_id: - RUTHVIK VAILA ET AL, "Feature Extraction using Spiking Convolutional Neural Networks", NEUROMORPHIC SYSTEMS, ACM, 2 PENN PLAZA, SUITE 701NEW YORKNY10121-0701USA, (20190723), doi:10.1145/3354265.3354279, ISBN 978-1-4503-7680-8, pages 1 - 8, XP058442033 [A] 1-15 * abstract * * Section 3 *
ref_type: npl
severity: background
ref_id: - DONG MENG ET AL, "Unsupervised speech recognition through spike-timing-dependent plasticity in a convolutional spiking neural network", PLOS ONE, (20181129), vol. 13, no. 11, doi:10.1371/journal.pone.0204596, page e0204596, XP055786795 [A] 1-15 * page  1  - page  3 *
ref_type: npl
severity: background
ref_id: - THIELE JOHANNES C ET AL, "A Timescale Invariant STDP-Based Spiking Deep Network for Unsupervised Online Feature Extraction from Event-Based Sensor Data", 2018 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN), IEEE, (20180708), doi:10.1109/IJCNN.2018.8489666, pages 1 - 8, XP033419468 [A] 1-15 * abstract * * Sections I - II *
ref_type: npl
severity: background
Final disposition: pending (2021-12-22)