Patent application EP19177833A (EP), filed 2019-06-03
Publication number: EP3748531A1
Title: DEVICE AND METHOD OF DIGITAL IMAGE CONTENT RECOGNITION, TRAINING OF THE SAME
Abstract: Device for and computer implemented method of image content recognition and of training a prototypical neural network for image content recognition, the method comprising collecting (302) a first set of digital images from a database, wherein the first set of digital images is sampled from digital images assigned to a many shot class; creating (304) a first training set comprising the collected first set of digital images; training (306) in a first training stage a first artificial neural network comprising a first feature extractor and a first classifier for classifying digital images using the first training set; and in a second training stage collecting (308) first parameters of the trained first feature extractor, collecting (310) second parameters of the trained classifier, determining (312) third parameters of a second feature extractor of a second artificial neural network depending on the first parameters, determining (314) fourth parameters of a second classifier for classifying digital images of the second artificial neural network, wherein the second artificial neural network is or comprises the prototypical neural network.
IPC codes: G06V  10/   762            A I
G06V  10/   764            A I
G06V  20/    00            A I
Claims: number: 1
text: Computer implemented method of training an prototypical neural network for image content recognition, characterized by collecting (302) a first set of digital images from a database, wherein the first set of digital images is sampled from digital images assigned to a many shot class; creating (304) a first training set comprising the collected first set of digital images; training (306) in a first training stage a first artificial neural network comprising a first feature extractor and a first classifier for classifying digital images using the first training set; and in a second training stage collecting (308) first parameters of the trained first feature extractor, collecting (310) second parameters of the trained classifier, determining (312) third parameters of a second feature extractor of a second artificial neural network depending on the first parameters, determining (314) fourth parameters of a second classifier for classifying digital images of the second artificial neural network, wherein the second artificial neural network is or comprises the prototypical neural network.
number: 2
text: Method according to claim 1, characterized by collecting (316) a second set of digital images from the database, wherein the second set of digital images is sampled from digital images assigned to a few shot class; creating (318) a second training set for a second stage of training comprising the second set of digital images, training (320) the second artificial neural network using the second training set.
number: 3
text: Method according to claim 1 or 2, characterized in that for the first training set digital images assigned to many shot classes are sampled, wherein for the second training set digital images assigned to few shot classes are sampled, the database comprising digital images assigned to the many shot classes and digital images assigned to few shot classes, wherein each many shot class comprises of more digital images than each few shot class.
number: 4
text: Method according to any of claims 1 to 3, characterized in that the third parameters are weights of the first feature extractor, wherein the fourth parameters are prototypes of the second classifier.
number: 5
text: Method according to any of claims 1 to 4, characterized in that either the first classifier and the second classifier are distance based classifiers or the first classifier and the second classifier are similarity based classifiers, in particular either the first classifier is a first cosine norm distance-based classifier and the second classifier is a second cosine norm classifier, or the first classifier is a first Euclidean norm classifier and the second classifier is a second Euclidean norm distance-based classifier, or the first classifier is a first cosine similarity based classifier and the second classifier is a second cosine similarity based classifier.
number: 6
text: Method according to any of claims 1 to 5, characterized in that the first classifier and the second classifier are cosine norm classifier adapted to determine the distance between a digital image feature determined by the second feature extractor and a trained image feature, wherein the second classifier has the weights of the first classifier.
number: 7
text: Method according to any of claims 1 to 5, characterized in that in the second training stage centroids of training images of the second training set are determined.
number: 8
text: Method according to any of claims 1 to 5, characterized in that in the second training stage nearest neighbor of training images of the second training set are determined.
number: 9
text: Method of digital image content recognition, characterized by training (402) a prototypical neural network according to the method of claim 7, wherein depending on the trained prototypical neural network features of a test image are determined (404), wherein a distance in feature space of the features to a plurality of centroids is determined, wherein a nearest centroid in feature space is determined, and wherein a class for the content of the digital image is determined (406) depending on a class of the nearest centroid.
number: 10
text: Method of digital image content recognition, characterized by training (402) a prototypical neural network according to the method of claim 8, wherein depending on the trained prototypical neural network features of a test image are determined (404), wherein a distance in feature space of the features to a plurality of neighbor is determined, wherein a nearest neighbor in feature space is determined, and wherein a class for the content of the digital image is determined (406) depending on a class of the nearest neighbor.
number: 11
text: Method according to claim 9 or 10, wherein a confidence score is determined for the class for the content.
number: 12
text: Device (100) for image content recognition, characterized in that the device (100) comprises a processor (102) and a memory (104) for an artificial neural network, that is configured for image content recognition according to the method of any of the preceding claims.
number: 13
text: Device (100) according to claim 12, characterized in that the device (100) comprises a first neural network (202), a second neural network (212) and a transfer unit (226), configured to transfer parameters of the first neural network (202) to the second neural network (212), in particular to transfer weights of a first feature extractor (204) to a second feature extractor (212) of the second neural network (212) as weights of the second feature extractor (212), and/or to determine from weights of a first classifier (206) prototypes of a second classifier (216) of the second neural network (212), wherein the second neural network is or comprises a prototypical neural network.
number: 14
text: Computer program, characterized in that the computer program comprises computer readable instructions that when executed by a computer cause the computer to execute the method according to claims 1 to 11.
number: 15
text: Computer program product, characterized in that the computer program product comprises a computer readable storage medium comprising the computer program according to claim 14.
Office action: search_report, dated 2020-12-09
Rejection reasons: novelty_epc_54
Prior art cited: ref_id: - WEI-YU CHEN ET AL.: "A CLOSER LOOK AT FEW-SHOTCLASSIFICATION", ICLR, 2019, Retrieved from the Internet <URL:https://openreview.net/pdf?id=HkxLXnAcFQ>
ref_type: npl
severity: background
ref_id: - YU-XIONG WANG ET AL., 31ST CONFERENCE ON NEURAL INFORMATION PROCESSING SYSTEMS (NIPS, vol. Prototypical Networks for Few-shot Learning, 2017, Retrieved from the Internet <URL:http://www.cs.toronto.edu/-zemei/documents/prototypicai_networks_nips_2017.ρdf>
ref_type: npl
severity: background
ref_id: - ZHAO HENG ET AL: "Few-Shot and Many-Shot Fusion Learning in Mobile Visual Food Recognition", 2019 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), IEEE, 26 May 2019 (2019-05-26), pages 1 - 5, XP033574321, ISSN: 2158-1525, ISBN: 978-1-7281-0397-6, [retrieved on 20190429], DOI: 10.1109/ISCAS.2019.8702564
ref_type: npl
severity: background
ref_id: - ZHAO HENG ET AL, "Few-Shot and Many-Shot Fusion Learning in Mobile Visual Food Recognition", 2019 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), IEEE, doi:10.1109/ISCAS.2019.8702564, ISSN 2158-1525, ISBN 978-1-7281-0397-6, (20190526), pages 1 - 5, (20190429), XP033574321 [X] 1-4,6,7,9,11-15 * sections II and III;; figure 1 * [I] 5,8,10
ref_type: npl
severity: novelty_destroying
Final disposition: pending (2020-12-09)