Patent application EP20179099A (EP), filed 2020-06-09
Publication number: EP3923199A1
Title: METHOD AND SYSTEM FOR COMPRESSING A NEURAL NETWORK
Abstract: [0001]    A method for compressing a neural network comprising a plurality of layers, each layer comprising a weight tensor. Said method comprises a step of selecting (E10) hyperparameters to define at least one basis of tensors referred to as "basis tensors", including a rank hyperparameter R, a dimension hyperparameter N and a size hyperparameter representative of the number of basis tensors. Said method also comprises, for each weight tensor, a set of steps of:
- embedding (E20) the weight tensor into an envelope tensor,
- reshaping (E30) said envelope tensor by tensorization,
- determining (E40) a representation of the reshaped envelope tensor in the form of a sequence of elements comprising tensors referred to as "cores",
- parameterizing (E50) each core into a combination of the basis tensors.
IPC codes: G06N   3/    04            A I
G06N   3/    08            A I
G06N   3/   042            A I
G06N   3/  0464            A I
G06N   3/  0495            A I
G06N   3/   082            A I
Claims: number: 1
text: A method for compressing a neural network comprising a plurality of layers, each layer comprising a weight tensor whose size is equal to a product of numbers referred to as "size numbers", said method comprising a step of selecting (E10) hyperparameters to define at least one basis of tensors referred to as "basis tensors", including a rank hyperparameter R, a dimension hyperparameter N and a size hyperparameter representative of the number of basis tensors, each basis tensor being an array belonging to a vector space whose dimension is equal to the product of at least one number belonging to the hyperparameter R and a number N' belonging to the hyperparameter N, said method also comprising, for each weight tensor, a set of steps of:
number: 2
text: The method according to claim 1, wherein, for each weight tensor, the sequence associated with the reshaped envelope tensor also comprises, before each core, a diagonal matrix intended to be multiplied with said core following it in said sequence, the size of a diagonal matrix being defined as a function of the hyperparameter R.
number: 3
text: The method according to any one of claims 1 to 2, wherein, for each weight tensor, the representation determined for the reshaped envelope tensor is a tensor ring decomposition.
number: 4
text: The method according to any one of claims 1 to 2, wherein, for each weight tensor, the representation determined for the reshaped envelope tensor is a tensor train decomposition.
number: 5
text: The method according to any one of claims 1 to 4, wherein said at least one basis is formed of three-dimensional tensors or else four-dimensional tensors.
number: 6
text: The method according to any one of claims 1 to 5, wherein the basis tensors satisfy at least one constraint parameterized by the number of independent parameters defining each of said tensors.
number: 7
text: A data processing method comprising the steps of:
number: 8
text: The method according to claim 7, said method also comprising the steps of:
number: 9
text: The method according to claim 8, wherein the basis tensors learned during the training of the first neural network are considered as trainable parameters during the training of the second compressed network.
number: 10
text: A method according to any one of claims 7 to 9, wherein, when a compression step of a neural network is carried out according to claim 2, the diagonal matrices used for this neural network are also initialized during the initialization step and are also considered as learnable parameters during the training of this neural network.
number: 11
text: A computer program including instructions for executing the compression method according to any one of claims 1 to 6 or a processing data method according to any one of the claims 7 to 10 when said program is executed by a computer.
number: 12
text: A recording medium readable by a computer and having recorded thereon a computer program according to claim 11.
number: 13
text: A data processing system for compressing a neural network comprising a plurality of layers, each layer comprising a weight tensor whose size is equal to a product of numbers referred to as "size numbers", said processing system comprising:
number: 14
text: A system comprising:
Office action: search_report, dated 2021-12-15
Prior art cited: ref_id: - HE K.ZHANG X.REN S.SUN J.: "Deep residual learning for image recognition", PROCEEDINGS OF THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2016, pages 770 - 778, XP055536240, DOI: 10.1109/CVPR.2016.90
ref_type: npl
severity: background
ref_id: - SIMONYAN K.ZISSERMAN A.: "Very deep convolutional networks for large-scale image recognition", INTERNATIONAL CONFERENCE ON LEARNING REPRESENTATIONS, 2015
ref_type: npl
severity: background
ref_id: - ZHAO Q.ZHOU G.XIE S.ZHANG L.CICHOCKI A., TENSOR RING DECOMPOSITION, 2016
ref_type: npl
severity: background
ref_id: - WANG W.SUN Y.ERIKSSON B.AGGARWAL V.: "Wide compression: Tensor ring nets", PROCEEDINGS OF THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2018, pages 9329 - 9338, XP033473859, DOI: 10.1109/CVPR.2018.00972
ref_type: npl
severity: background
ref_id: - LI Y.GU S.GOOL L. V.TIMOFTE R.: "Learning Filter Basis for Convolutional Neural Network Compression", PROCEEDINGS OF THE IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION, 2019, pages 5623 - 5632
ref_type: npl
severity: background
ref_id: - NOVIKOV A.PODOPRIKHIN D.OSOKIN A.VETROV D.: "Tensorizing Neural Networks", ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS, 2015, pages 442 - 450
ref_type: npl
severity: background
ref_id: - HE K.ZHANG X.REN S.SUN J.: "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification", IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION, 2015, pages 1026 - 1034, XP032866428, DOI: 10.1109/ICCV.2015.123
ref_type: npl
severity: background
ref_id: - KINGMA D.BA J., ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION, 2015
ref_type: npl
severity: background
ref_id: - LECUN Y.BOTTOU L.BENGIO Y.HAFFNER P.: "Gradient-based learning applied to document recognition", PROCEEDINGS OF THE IEEE, vol. 86, no. 11, 1998, pages 2278 - 2324, XP000875095, DOI: 10.1109/5.726791
ref_type: npl
severity: background
ref_id: - ALEXANDER NOVIKOV ET AL: "Tensorizing Neural Networks", 22 September 2015 (2015-09-22), XP055415964, Retrieved from the Internet <URL:https://arxiv.org/pdf/1509.06569.pdf>
ref_type: npl
severity: background
ref_id: - AGGARWAL VANEET ET AL: "Wide Compression: Tensor Ring Nets", 2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, IEEE, 18 June 2018 (2018-06-18), pages 9329 - 9338, XP033473859, DOI: 10.1109/CVPR.2018.00972
ref_type: npl
severity: background
ref_id: - LI YAWEI ET AL: "Learning Filter Basis for Convolutional Neural Network Compression", 2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), IEEE, 27 October 2019 (2019-10-27), pages 5622 - 5631, XP033723354, DOI: 10.1109/ICCV.2019.00572
ref_type: npl
severity: background
ref_id: - AGGARWAL VANEET ET AL, "Wide Compression: Tensor Ring Nets", 2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, IEEE, (20180618), doi:10.1109/CVPR.2018.00972, pages 9329 - 9338, XP033473859 [ID] 1-14 * the whole document *
ref_type: npl
severity: background
ref_id: - LI YAWEI ET AL, "Learning Filter Basis for Convolutional Neural Network Compression", 2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), IEEE, (20191027), doi:10.1109/ICCV.2019.00572, pages 5622 - 5631, XP033723354 [A] 1-14 * the whole document *
ref_type: npl
severity: background
Final disposition: pending (2021-12-15)