Patent application EP21162287A (EP), filed 2021-03-12
Publication number: EP3926547A1
Title: PROGRAM, LEARNING METHOD, AND INFORMATION PROCESSING APPARATUS
Abstract: A memory holds a model including a plurality of layers including their respective parameters and training data. A processor starts learning processing, which repeatedly calculates an error of an output of the model by using the training data, calculates an error gradient, which indicates a gradient of the error with respect to the parameters, for each of the layers, and updates the parameters based on the error gradients. The processor calculates a difference between a first error gradient calculated in a first iteration in the learning processing and a second error gradient calculated in a second iteration after the first iteration for a first layer among the plurality of layers. In a case where the difference is less than a threshold, the processor skips the calculating of the error gradient and the updating of the parameter for the first layer in a third iteration after the second iteration.
IPC codes: G06N   3/    08            A I
Claims: number: 1
text: A computer program that causes a computer (10) to execute a process comprising:
number: 2
text: The computer program according to claim 1, wherein the computer program further causes the computer (10) to execute a process of determining the threshold, based on an initial error gradient (Δw0), which indicates an error gradient calculated in an iteration executed in response to the starting of the learning processing.
number: 3
text: The computer program according to claim 2, wherein the computer program further causes the computer (10) to execute a process of setting a learning rate (LR) used to update the parameters and changing the learning rate based on a number (m) of iterations in which the learning processing has been executed, and
wherein the determining of the threshold includes changing the threshold, based on a different initial error gradient (Δw0), which indicates an error gradient calculated in an iteration executed in response to change of the learning rate (LR).
number: 4
text: The computer program according to claim 1, wherein, in each iteration in the learning processing, the computer program further causes the computer (10) to execute a process of calculating different errors from different training data by using a plurality of processing nodes (104-1, 104-2), calculating different error gradients from the different errors by using the plurality of processing nodes, synthesizing the different error gradients by allowing the plurality of processing nodes to communicate with each other, and updating the parameters, based on the synthesized error gradients, and
wherein the skipping includes skipping the calculating of the different error gradients, the communicating among the plurality of processing nodes, and the updating of the parameter for the first layer.
number: 5
text: The computer program according to claim 1, wherein the model is a multilayer neural network (310).
number: 6
text: The computer program according to claim 1, wherein the second iteration (m) is immediately after the first iteration (m-1), and the third iteration is immediately after the second iteration.
number: 7
text: The computer program according to claim 1,
wherein the calculating of the difference includes calculating the difference for each of the plurality of layers (311 to 316), and
wherein the skipping includes determining, among the plurality of layers, a layer (313 to 315) whose difference is less than the threshold and skipping the calculation of the error gradient and the updating of the parameter for the determined layer.
number: 8
text: A learning method executed by a computer (10), the learning method comprising:
number: 9
text: An information processing apparatus comprising:
Office action: search_report, dated 2021-12-22
Rejection reasons: novelty_epc_54
Prior art cited: ref_id: WO2017157183A1
ref_type: patent
severity: background
ref_id: JP2019185275A
ref_type: patent
severity: background
ref_id: - XIAO XUELI ET AL: "Fast Deep Learning Training through Intelligently Freezing Layers", 2019 INTERNATIONAL CONFERENCE ON INTERNET OF THINGS (ITHINGS) AND IEEE GREEN COMPUTING AND COMMUNICATIONS (GREENCOM) AND IEEE CYBER, PHYSICAL AND SOCIAL COMPUTING (CPSCOM) AND IEEE SMART DATA (SMARTDATA), IEEE, 14 July 2019 (2019-07-14), pages 1225 - 1232, XP033637632, DOI: 10.1109/ITHINGS/GREENCOM/CPSCOM/SMARTDATA.2019.00205
ref_type: npl
severity: background
ref_id: - IANDOLA FORREST N ET AL: "FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters", 2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 27 June 2016 (2016-06-27), pages 2592 - 2600, XP033021440, DOI: 10.1109/CVPR.2016.284
ref_type: npl
severity: background
ref_id: - TIANYI CHEN ET AL: "LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 25 May 2018 (2018-05-25), XP080882266
ref_type: npl
severity: background
ref_id: - XIAO XUELI ET AL, "Fast Deep Learning Training through Intelligently Freezing Layers", 2019 INTERNATIONAL CONFERENCE ON INTERNET OF THINGS (ITHINGS) AND IEEE GREEN COMPUTING AND COMMUNICATIONS (GREENCOM) AND IEEE CYBER, PHYSICAL AND SOCIAL COMPUTING (CPSCOM) AND IEEE SMART DATA (SMARTDATA), IEEE, (20190714), doi:10.1109/ITHINGS/GREENCOM/CPSCOM/SMARTDATA.2019.00205, pages 1225 - 1232, XP033637632 [X] 1-3,5-9 * Section III * [I] 4
ref_type: npl
severity: novelty_destroying
ref_id: - IANDOLA FORREST N ET AL, "FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters", 2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, (20160627), doi:10.1109/CVPR.2016.284, pages 2592 - 2600, XP033021440 [A] 1-9 * figure 1 *
ref_type: npl
severity: background
ref_id: - TIANYI CHEN ET AL, "LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, (20180525), XP080882266 [A] 1-9 * Sections 1.2 and 2 *
ref_type: npl
severity: background
Final disposition: pending (2021-12-22)