Original: Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing
Revised: Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing

Revised: Anonymous Author(s)

Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Residual networks have shown great success and become indispensable in today’s deep models.
Revised: Residual networks have shown great success and become indispensable in today’sdeep models.

Original: In this work, we aim to re-investigate the training process of residual networks from a novel social psychology perspective of loafing, and further propose a new training strategy to strengthen the performance of residual networks.
Revised: In this work, we aim to re-investigate the training process of residualnetworks from a novel social psychology perspective of loafing, and further proposea new training strategy to strengthen the performance of residual networks.

Original: As residual networks can be viewed as ensembles of relatively shallow networks (i.e., unraveled view ) in prior works, we also start from such view and consider that the final performance of a residual network is co-determined by a group of sub-networks.
Revised: Asresidual networks can be viewed as ensembles of relatively shallow networks (i.e.,unraveled view ) in prior works, we also start from such view and consider thatthe final performance of a residual network is co-determined by a group of subnetworks.

Original: Inspired by the social loafing problem of social psychology, we find that residual networks invariably suffer from similar problem, where subnetworks in a residual network are prone to exert less effort when working as part of the group compared to working alone.
Revised: Inspired by the social loafing problem of social psychology, we findthat residual networks invariably suffer from similar problem, where sub-networksin a residual network are prone to exert less effort when working as part of thegroup compared to working alone.

Original: We define this previously overlooked problem as network loafing .
Revised: We define this previously overlooked problemas network loafing .

Original: As social loafing will ultimately cause the low individual productivity and the reduced overall performance, network loafing will also hinder the performance of a given residual network and its sub-networks.
Revised: As social loafing will ultimately cause the low individualproductivity and the reduced overall performance, network loafing will also hinderthe performance of a given residual network and its sub-networks.

Original: Referring to the solutions of social psychology, we propose stimulative training , which randomly samples a residual sub-network and calculates the KL-divergence loss between the sampled sub-network and the given residual network, to act as extra supervision for sub-networks and make the overall goal consistent.
Revised: Referring to thesolutions of social psychology, we propose stimulative training , which randomlysamples a residual sub-network and calculates the KL-divergence loss between thesampled sub-network and the given residual network, to act as extra supervisionfor sub-networks and make the overall goal consistent.

Original: Comprehensive empirical results and theoretical analyses verify that stimulative training can well handle the loafing problem, and improve the performance of a residual network by improving the performance of its sub-networks.The code is available at https://github.com/Sunshine-Ye/NIPS22-ST.
Revised: Comprehensive empiricalresults and theoretical analyses verify that stimulative training can well handle theloafing problem, and improve the performance of a residual network by improvingthe performance of its sub-networks.

Original:  Introduction
Revised:  Introduction

Original: Since ResNet [1] wins the first place at the ILSVRC-2015 competition, simple-but-effective residual connections are applied in various deep networks, such as CNN, MLP, and transformer.
Revised: Since ResNet [1] wins the first place at the ILSVRC-2015 competition, simple-but-effective residualconnections are applied in various deep networks, such as CNN, MLP and transformer.

Original: To explore the secrets behind the success of residual networks, numerous studies have been proposed.
Revised: To explorethe secrets behind the success of residual networks, numerous studies have been proposed.

Original: He et. al [1] exploit the residual structure to avoid the performance degradation of deep networks.
Revised: He et.al [1] exploit the residual structure to avoid the performance degradation of deep networks.

Original: Further,
Revised: Further,

Original: He et.
Revised: He et.

Original: al [2] consider that such a structure can transfer any low level features to high level layers in forward propagation and directly transmit the gradients from deep to shallow layers in backward propagation.
Revised: al [2] consider that such a structure can transfer any low level features to high level layers inforward propagation and directly transmit the gradients from deep to shallow layers in backwardpropagation.

Original: Balduzzi et.
Revised: Balduzzi et.

Original: al [3] find that residual networks can alleviate the shattered gradients problem that gradients resemble white noise.
Revised: al [3] find that residual networks can alleviate the shattered gradientsproblem that gradients resemble white noise.

Original: In addition, Veit et.
Revised: In addition, Veit et.

Original: al
Revised: al

Original: [4] experimentally verify that residual networks can be seen as a collection of numerous networks of different lengths, namely unraveled view .
Revised: [4] experimentally verify thatresidual networks can be seen as a collection of numerous networks of different lengths, namelyunraveled view .

Original: Following this view, Sun et.
Revised: Following this view, Sun et.

Original: al [5] further attribute the success of residual networks to shallow sub-networks, which may correspond to the low-degree term when regarding the neural network as a polynomial function.
Revised: al [5] further attribute the success of residual networksto shallow sub-networks, which may correspond to the low-degree term when regarding the neuralnetwork as a polynomial function.

Original: Sincethe unraveled view is supported both experimentally and [Figure] theoretically, we further investigate some interesting mechanism behind residual networks based on this inspiring work.
Revised: Since unraveled view is supported both experimentally andtheoretically, we further investigate some interesting mechanism behind residual networks based onthis inspiring work.

Original: Specifically, we treat a residual network as the ensemble of relatively shallow sub-networks and consider its final performance to be co-determined by a group of sub-networks. 
Revised: Specifically, we treat a residual network as the ensemble of relatively shallowsub-networks and consider its final performance to be co-determined by a group of sub-networks. [Figure]

Original: In social psychology, working in a group is always atricky thing.
Revised: In social psychology, working in a group is always a thing.

Original: Compared with performing tasks alone, group members tend to make less efforts when working as part of a group, which is defined as the social loafing problem [6, 7].
Revised: Compared with performing tasksalone, group members tend to make less efforts when working as part of a group, which is defined asthe social loafing problem [6, 7].

Original: Moreover, social psychology researches find that increasing the group size may aggravate social loafing for the decrease of individual visibility [8, 9].
Revised: Moreover, social psychology researches find that increasing thegroup size may aggravate social loafing for the decrease of individual visibility [8, 9].

Original: Inspired by these, we find that the ensemble-like networks formed by residual connections also have a similar problem and behavior.
Revised: Inspired bythese, we find that the ensemble-like networks formed by residual connections also have a similarproblem and behavior.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, we can see that, different residual networks invariably suffer from the loafing problem, that is, the sub-networks working in a given residual network are prone to exert degraded performances than these sub-networks working individually.
Revised: 1, we can see that, different residual networks invariablysuffer from the loafing problem, that is, the sub-networks working in a given residual network areprone to exert degraded performances than these sub-networks working individually.

Original: For example, a sub-network ResNet32 in the ResNet56 only has a 46.80% Top1 accuracy, much lower than ResNet trained individually with accuracy of 92.63%.
Revised: For example, asub-network ResNet32 in the ResNet56 only has a 46.80% Top1 accuracy, much lower than ResNet32trained individually with accuracy of 92.63%.

Original: Moreover, the loafing problem of deeper residual networks is inclined to be more severe than that of shallower ones, that is, the same sub-network in deeper residual networks constantly presents inferior performance than that in shallower residual networks.
Revised: Moreover, the loafing problem of deeper residualnetworks inclines to be more severe than that of shallower ones, that is, the same sub-network indeeper residual networks constantly presents inferior performance than that in shallower residualnetworks.

Original: For example, ResNet20 within ResNet32 has a 55.28% Top1 accuracy, while ResNet within ResNet56 (deeper than ResNet 32) only has a 23.17% Top1 accuracy.
Revised: For example, ResNet20 within ResNet32 has a 55.28% Top1 accuracy, while ResNet20within ResNet56 (deeper than ResNet 32) only has a 23.17% Top1 accuracy.

Original: Such problems have not been addressed in the literature so far as we know.
Revised: Such problems have notbeen addressed in the literature so far as we know.

Original: Hereafter, we define this previously overlooked problem as network loafing .
Revised: Hereafter, we define this previously overlookedproblem as network loafing .

Original: As social psychology researches show that social loafing will ultimately cause low productivity of each individual and the collective [10, 11], we consider that network loafing may also hinder the performance of given residual network and all of its sub-networks.
Revised: As social psychology researches show that social loafing will ultimatelycause low productivity of each individual and the collective [10, 11], we consider that networkloafing may also hinder the performance of given residual network and all of its sub-networks.

Original: In social psychology, there are two commonly used solutions for preventing social loafing within groups: 1) establishing individual accountability by increasing the individual supervision and 2) making tasks cooperative by setting up the overall goal [8, 9].
Revised: In social psychology, there are two commonly used solutions for preventing social loafing withingroups: 1) establishing individual accountability by increasing the individual supervision and 2)making tasks cooperative by setting up the overall goal [8, 9].

Original: Inspired by this, we propose a novel training strategy for improving residual networks, namely stimulative training .
Revised: Inspired by this, we propose anovel training strategy for improving residual networks, namely stimulative training .

Original: In details, for each mini-batch during stimulative training, besides the main loss of the given residual network in conventional training, we will randomly sample a residual sub-network (individual supervision) and calculate the KL-divergence loss between the sampled sub-network and the given residual network (consistent overall goal).
Revised: In details, foreach mini-batch during stimulative training, besides the main loss of the given residual network inconventional training, we will randomly sample a residual sub-network (individual supervision) andcalculate the KL-divergence loss between the sampled sub-network and the given residual network(consistent overall goal).

Original: This simple yet effective training strategy can relieve the loafing problem of residual networks, by strengthening the individual supervision of sub-networks and making the goals of residual sub-networks and the given residual network more consistent.
Revised: This simple yet effective training strategy can relieve the loafing problem ofresidual networks, by strengthening the individual supervision of sub-networks and making the goalsof residual sub-networks and the given residual network more consistent.

Original: Comprehensive empirical analyses verify that stimulative training can solve the loafing problem of residual networks effectively and efficiently, thus improve both the performance of a given residual network and all of its residual sub-networks by a large margin.
Revised: Comprehensive empirical analyses verify that stimulative training can solve the loafing problem ofresidual networks effectively and efficiently, thus improve both the performance of a given residualnetwork and all of its residual sub-networks by a large margin.

Original: Furthermore, we theoretically show the connection of the proposed stimulative training strategy and the improved performance of a given residual network and all of its residual sub-networks.
Revised: Furthermore, we theoretically showthe connection of the proposed stimulative training strategy and the improved performance of a givenresidual network and all of its residual sub-networks.

Original: Besides, experiments on various benchmark datasets using various residual networks demonstrate the effectiveness of the proposed training strategy.
Revised: Besides, experiments on various benchmarkdatasets using various residual networks demonstrate the effectiveness of the proposed trainingstrategy.

Original: The contributions of our work can be summarized as the following:
Revised: The contributions of our work can be summarized as the following:

Original: • We understand residual networks from a social psychology perspective, and find that different residual networks invariably suffer from the problem of network loafing. • We improve residual networks from a social psychology perspective, and propose a simplebut-effective stimulative training strategy to improve the performance of the given residual network and all of its sub-networks.
Revised: • We understand residual networks from a social psychology perspective, and find that differentresidual networks invariably suffer from the problem of network loafing.• We improve residual networks from a social psychology perspective, and propose a simplebut-effective stimulative training strategy to improve the performance of the given residualnetwork and all of its sub-networks.

Original: • Comprehensive empirical and theoretical analysis verify that stimulative training can well solve the loafing problem of residual networks.
Revised: • Comprehensive empirical and theoretical analysis verify that stimulative training can well solve the loafing problem of residual networks.

Original:  Related Works 2.1 Unraveled View
Revised:  Related Works2.1 Unraveled View

Original: As one pioneer work to investigate residual networks, [4] experimentally shows that residual networks can be seen as a collection of numerous networks of different length, namely unraveled view.
Revised: As one pioneer work to investigate residual networks, [4] experimentally shows that residual networkscan be seen as a collection of numerous networks of different length, namely unraveled view.

Original: Subsequently, [5] follows this view and further attributes the success of residual networks to shallow sub-networks first.
Revised: Subsequently, [5] follows this view and further attributes the success of residual networks to shallowsub-networks first.

Original: Besides, [5] considers the neural network as a polynomial function and corresponds shallow sub-network to low-degree term to explain the working mechanism of residual networks.
Revised: Besides, [5] considers the neural network as a polynomial function and correspondsshallow sub-network to low-degree term to explain the working mechanism of residual networks.

Original: Similar to [4] and [5], this paper also investigates residual networks from the unraveled view.
Revised: Similar to [4] and [5], this paper also investigates residual networks from the unraveled view.

Original: Differently, inspired by social psychology, we further reveal the loafing problem of residual networks under the unraveled view.
Revised: Differently, inspired by social psychology, we further reveal the loafing problem of residual networksunder the unraveled view.

Original: Besides, we propose a novel stimulative training method to relieve this problem and further unleash the potential of residual networks.
Revised: Besides, we propose a novel stimulative training method to relieve thisproblem and further unleash the potential of residual networks.

Original: 2.2 Knowledge Distillation
Revised: 2.2 Knowledge Distillation

Original: As a classical method, knowledge distillation [12, 13] transfers the knowledge from a teacher network to a student network via approximating the logits [12, 14, 15] or features [16, 17, 18, 19] output.
Revised: As a classical method, knowledge distillation [12] transfers the knowledge from a teacher network toa student network via approximating the logits [12, 13, 14] or features [15, 16, 17] output.

Original: To avoid the huge cost of training a high performance teacher, some works abandon the naive teacherstudent framework, like mutual distillation [20] making group of students learn from each other online, and self distillation [21] transferring knowledge from deep layers to shallow layers.
Revised: To avoidthe huge cost of training a high performance teacher, some works abandon the naive teacher-studentframework, like mutual distillation [18] making group of students learn from each other online,and self distillation [19] transferring knowledge from deep layers to shallow layers.

Original: Generally, all these distillation methods need to introduce additional networks or structures, and employ fixed teacher-student pairs.
Revised: Generally, allthese distillation methods need to introduce additional networks or structures, and employ fixedteacher-student pairs.

Original: As a comparison, our method does not require any additional network or structure, and the student network is a randomly sampled sub-network of a network.
Revised: As a comparison, our method does not require any additional network orstructure, and the student network is a randomly sampled sub-network of a network.

Original: Besides, our method is essentially designed to address the loafing problem of residual networks, which is different from knowledge distillation that aims to obtain a compact network with acceptable accuracy.
Revised: Besides, ourmethod is essentially designed to address the loafing problem of residual networks, which is differentfrom knowledge distillation that aims to obtain a compact network with acceptable accuracy.

Original: 2.3 One-
Revised: 2.3 One-shot NAS

Original: One-shot NAS is an important branch of neural architecture search (NAS) [22, 23, 24, 25, 26, 27].
Revised: One-shot NAS is an important branch of neural architecture search (NAS).

Original: Along this direction, [28] trains an once-for-all (OFA) network with progressive shrinking and knowledge distillation to support kinds of architectural settings.
Revised: Along this direction, [20]trains an once-for-all (OFA) network with progressive shrinking and knowledge distillation to supportkinds of architectural settings.

Original: Following this work, BigNAS [29] introduces several technologies to train a high-quality single-stage model, whose child models can be directly deployed without extra retraining or post-processing steps.
Revised: Following this work, BigNAS [21] introduces several technologiesto train a high-quality single-stage model, whose child models can be directly deployed withoutextra retraining or post-processing steps.

Original: Both OFA and BigNAS aim at simultaneously training and searching various networks with different resolutions, depths, widths and operations.
Revised: Both OFA and BigNAS aim at simultaneously training andsearching various networks with different resolutions, depths, widths and operations.

Original: Differently, the proposed method aims at improving a given residual network, thus can be seamlessly applied to the searched model of NAS.
Revised: Differently, theproposed method aims at improving a given residual network, thus can be seamlessly applied to thesearched model of NAS.

Original: As OFA and BigNAS are not designed to solve the loafing problem, their sampling space and supervision signal are also different with the proposed method.
Revised: As OFA and BigNAS are not designed to solve the loafing problem, theirsampling space and supervision signal are also different with the proposed method.

Original: More importantly, the social-psychology-inspired problem of network loafing may explain why OFA and BigNAS work.
Revised: More importantly,the social-psychology-inspired problem of network loafing may explain why OFA and BigNAS work.

Original: 2.4 Stochastic Depth
Revised: 2.4 Stochastic Depth

Original: As a regularization technique, Stochastic Depth [30] randomly disables the convolution layers of residual blocks, to reduce training time and test error substantially.
Revised: As a regularization technique, Stochastic Depth [22] randomly disables the convolution layers ofresidual blocks, to reduce training time and test error substantially.

Original: In Stochastic Depth, the reduction in test error is attributed to strengthening gradients of earlier layers and the implicit ensemble of numerous sub-networks of different depths.
Revised: In Stochastic Depth, the reductionin test error is attributed to strengthening gradients of earlier layers and the implicit ensemble ofnumerous sub-networks of different depths.

Original: In fact, its improved performance can be also interpreted as relieving the network loafing problem defined in this work.
Revised: In fact, its improved performance can be also interpretedas relieving the network loafing problem defined in this work.

Original: The theoretical analysis in this work can be also applied to Stochastic Depth.
Revised: The theoretical analysis in this workcan be also applied to Stochastic Depth.

Original: Besides, for better solving the loafing problem, our method samples sub-networks with ordered depth, and uses an additional KL-divergence loss to provide a more achievable target and make the output of a given network and its sub-networks more consistent.  
Revised: Besides, for better solving the loafing problem, our methodsamples sub-networks with ordered depth, and uses an additional KL-divergence loss to provide amore achievable target and make the output of a given network and its sub-networks more consistent.  Stimulative Training 3.1 Motivation

Original: 2 L f − 1 L f −
Revised: Social loafing is a social psychology phenomenon that individuals lower their productivity whenworking in a group. Based on the novel perspective that a residual network behaves like an ensemblenetwork [4], we find various residual networks invariably exhibit loafing-like behaviors as shownin Fig. 1 and Fig. 4, which we define as network loafing.

Revised: As social loafing is “a kind of socialdisease" that will harm the individual and collective productivity [10], we consider network loafing

Original:  L f 2 L f − 2 L f − 1 L f − 2 L f − 2 L f − 1 L f −
Revised: 2 L f − 1 L f − L f 2 L f − 2 L f − 1 L f − 2 L f − 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f − 77.3% 2 L f − 1 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f − 77.3% 2 L f − 1 L f −

Original: L f
Revised: L f

Original: Skip connection
Revised: Skip connection

Original: Residualmodule
Revised: Residualmodule

Original: Building block
Revised: Building block

Original: I m a g e
Revised: I m a g e

Original: L a b e l Mainloss 2 L f − 1 L f −
Revised: L a b e l Mainloss 2 L f − 1 L f −

Original: L f
Revised: L f

Original: Skip connection
Revised: Skip connection

Original: Residualmodule
Revised: Residualmodule

Original: Building block
Revised: Building block

Original: I m a g e
Revised: I m a g e

Original: L a b e l Mainloss
Revised: L a b e l Mainloss

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 60%(65%) 2 L f − 1 L f −
Revised: 60%(65%) 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f −

Original: [Table]
Revised: [Table]

Original: % 2 L f − 1 L f −
Revised: % 2 L f − 1 L f −

Original: L f
Revised: L f

Original: Skip connection
Revised: Skip connection

Original: Residualmodule
Revised: Residualmodule

Original: Building block
Revised: Building block

Original: I m a g e
Revised: I m a g e

Original: L a b e l Mainloss 70%(75%) 60%(65%) 2 L f − 1 L f −
Revised: L a b e l Mainloss 70%(75%) 60%(65%) 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f − 2 L f − 1 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f − 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f − 81.07% 2 L f − 1 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f − 81.07% 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f −

Original: L a b e l Mainloss
Revised: L a b e l Mainloss

Original: I m a g e
Revised: I m a g e

Original: KLloss
Revised: KLloss

Original: KLloss 2 L f − 1 L f −
Revised: KLloss 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f −

Original: L a b e l Mainloss
Revised: L a b e l Mainloss

Original: I m a g e
Revised: I m a g e

Original: KLloss
Revised: KLloss

Original: KLloss 80%(
Revised: KLloss 80%(

Original: [Table] [Table]
Revised: [Table] [Table]

Original: %) 2 L f − 1 L f −
Revised: %) 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f − 81.07% 2 L f − 1 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f − 81.07% 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f −

Original: L a b e l Mainloss
Revised: L a b e l Mainloss

Original: I m a g e
Revised: I m a g e

Original: KLloss
Revised: KLloss

Original: KLloss 80%(75%) 79%(65%) (a) Common training scheme suffers from severe network loafing problem (b) S timulative training scheme can relieve network loafing and improve the performance 2 L f − 1 L f −
Revised: KLloss 80%(75%) 79%(65%) (a) Common training scheme suffers from severe network loafing problem (b) S timulative training scheme can relieve network loafing and improve the performance 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f − 77.3% 2 L f − 1 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f − 77.3% 2 L f − 1 L f −

Original: L f
Revised: L f

Original: Skip connection
Revised: Skip connection

Original: Residualmodule
Revised: Residualmodule

Original: Building block
Revised: Building block

Original: I m a g e
Revised: I m a g e

Original: L a b e l Mainloss 70%(75%) 60%(65%) 2 L f − 1 L f −
Revised: L a b e l Mainloss 70%(75%) 60%(65%) 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f − 81.07% 2 L f − 1 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f − 81.07% 2 L f − 1 L f −

Original: L f 2 L f − 2 L f − 1 L f − 2 L f −
Revised: L f 2 L f − 2 L f − 1 L f − 2 L f −

Original: L a b e l Mainloss
Revised: L a b e l Mainloss

Original: I m a g e
Revised: I m a g e

Original: KLloss
Revised: KLloss

Original: KLloss 80%(75%) 79%(65%) Stimulative Training 3.1 Motivation Social loafing is a social psychology phenomenon that individuals lower their productivity when working in a group. Based on the novel perspective that a residual network behaves like an ensemble network [4], we find various residual networks invariably exhibit loafing-like behaviors as shown in Fig.
Revised: KLloss 80%(75%) 79%(65%) may also hinder the performance of a residual network and its sub-networks.

Original: 1 and Fig. 4, which we define as network loafing.

Original: As social loafing is “a kind of social disease" that will harm the individual and collective productivity [10], we consider network loafing may also hinder the performance of a residual network and its sub-networks.

Original: To alleviate network loafing, it is intuitive to learn from social psychology.
Revised: To alleviate networkloafing, it is intuitive to learn from social psychology.

Original: There are two common methods for solving social-loafing problem in sociology, namely, establishing individual accountability (i.e., increasing the individual supervision) and making tasks cooperative (i.e., setting up the overall goal) [8, 9].
Revised: There are two common methods for solvingsocial-loafing problem in sociology, namely, establishing individual accountability (i.e., increasingthe individual supervision) and making tasks cooperative (i.e., setting up the overall goal) [8, 9].

Original: In order to increase individual supervision, we sample sub-networks in the whole network and provide extra supervision to train each sub-network sufficiently.
Revised: Inorder to increase individual supervision, we sample sub-networks in the whole network and provideextra supervision to train each sub-network sufficiently.

Original: For the overall goal, we adopt KL divergence loss to constrain the output of sub-networks not far from that of the whole network, which aims at reducing the performance gap and driving the sampled sub-networks to develop cooperatively.
Revised: For the overall goal, we adopt KL divergenceloss to constrain the output of sub-networks not far from that of the whole network, which aims atreducing the performance gap and driving the sampled sub-networks to develop cooperatively.

Original: 3.2 Training Algorithm
Revised: 3.2 Training Algorithm

Original: In this subsection, we briefly illustrate the working scheme of the proposed stimulative training strategy, and show its difference from the common training method.
Revised: In this subsection, we briefly illustrate the working scheme of the proposed stimulative trainingstrategy, and show its difference from the common training method.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, common training only focuses on optimization of main network, thus suffers from severe network loafing, that is, sub-networks lower their performance when working in an ensemble.
Revised: 2, commontraining only focuses on optimization of main network, thus suffers from severe network loafing, thatis, sub-networks lower their performance when working in an ensemble.

Original: For example, as shown in
Revised: For example, as shown in

Original: Fig.
Revised: Fig.

Original: 2(a) sub-networks within the residual network only have an accuracy of 60% and 70%, much lower than their individual accuracy of 65% and 75%.
Revised: 2(a) sub-networks within the residual network only has an accuracy of 60% and 70%, much lowerthan their individual accuracy of 65% and 75%.

Original: As a comparison, stimulative training optimizes the main network and meanwhile uses it to provide extra supervision for a sampled sub-network at each training iteration, thus well handles the loafing problem.
Revised: As a comparison, stimulative training optimizesthe main network and meanwhile uses it to provide extra supervision for a sampled sub-network ateach training iteration, thus well handles the loafing problem.

Original: In the test procedure, our method can adopt the main network or any sub-network as the inference model, thus requiring the same or lower memory and inference cost compared with a given residual network.
Revised: In the test procedure, our method canadopt the main network or any sub-network as the inference model, thus requiring the same or lowermemory and inference cost compared with a given residual network.

Original: Formally, for a given residual network to be optimized, we define the main network as D m and the sub-network as D s .
Revised: Formally, for a given residual network to be optimized, we define the main network as D m and thesub-network as D s .

Original: All the sub-networks share weights with the main network, and make up the sampling space Θ
Revised: All the sub-networks share weights with the main network, and make up thesampling space Θ

Original: = { D s | D s = π
Revised: = { D s | D s = π

Original: ( D m ) } , where π is sampling operator, usually random sampling.
Revised: ( D m ) } , where π is sampling operator, usually random sampling.

Original: In the training process, we randomly sample a sub-network at each iteration to ensure the whole sampling space can be fully explored.
Revised: In the training process, we randomly sample a sub-network at each iteration to ensure the wholesampling space can be fully explored.

Original: To make the training more efficient and effective, we define a new sampling space obeying ordered residual sampling rule to be discussed in Section 3.3.
Revised: To make the training more efficient and effective, we define anew sampling space obeying ordered residual sampling rule to be discussed in Section 3.3.

Original: Denoting θ Dm and θ Ds as weights of the main network and the sampled network respectively, x as the minibatch training sample and y as its label, Z as the output of network, the total loss of stimulative
Revised: Denotingθ Dm and θ Ds as weights of the main network and the sampled network respectively, x as the minibatch training sample and y as its label, Z as the output of network, the total loss of stimulativetraining is computed as

Original: [Table]
Revised: [Equation]

Revised: where the CE and KL are the standard cross entropy loss and KL divergence loss.

Revised: λ is the balancedcoefficient, and CE loss denotes the main supervision for the main network.

Revised: KL loss provides anextra supervision for the sampled sub-network, and guarantees all sub-networks to be optimized inthe same direction with the main network, which can be considered as setting a common goal for the“ensemble network".

Revised: We use the standard stochastic gradient descent for updating the model, denotedas θ t +1 D m = θ tD m − η ∂ L ut ∂θ Dm , where η is the learning rate.

Revised: Thanks to the weight-sharing between

Revised: [Table]

Original: I npu t
Revised: I npu t

Original: O u t pu t I npu t 
Revised: O u t pu t I npu t O u t pu t I npu t O u t pu t I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t I npu t O u t pu t
Revised: O u t pu t I npu t O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t I npu t O u t pu tI npu t
Revised: O u t pu t I npu t O u t pu t

Original: O u t pu t I npu t

Original: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t 2 L f − 2 L f − 2 L f − 1 L f − 1 L f −
Revised: O u t pu t 2 L f − 2 L f − 2 L f − 1 L f − 1 L f −

Original: L f
Revised: L f

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: [Table]
Revised: [Table]

Original: u t pu t 2 L f − 2 L f − 2 L f − 1 L f − 1 L f −
Revised: u t pu t 2 L f − 2 L f − 2 L f − 1 L f − 1 L f −

Original: [Table]
Revised: [Table]

Original: f
Revised: f

Original: [Table]
Revised: [Table]

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t 2 L f − 2 L f − 2 L f − 1 L f − 1 L f −
Revised: O u t pu t 2 L f − 2 L f − 2 L f − 1 L f − 1 L f −

Original: L f
Revised: L f

Original: [Table]
Revised: [Table]

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t L f L f
Revised: O u t pu t L f L f

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t I npu t
Revised: O u t pu t I npu t

Original: O u t pu t
Revised: O u t pu t

Original: L f 1 L f −
Revised: L f 1 L f −

Original: L f
Revised: L f

Original: L f 2 L f − 1 L f − 1 L f − 2 L f − 2 L f − 1 L f − 2 L f −
Revised: L f 2 L f − 1 L f − 1 L f − 2 L f − 2 L f − 1 L f − 2 L f −

Original: [Table]
Revised: [Table]

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: L f
Revised: L f

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: L f 1 L f −
Revised: L f 1 L f −

Original: L f
Revised: L f

Original: L f 2 L f − 1 L f − 1 L f − 2 L f − 2 L f − 1 L f − 2 L f − 2.
Revised: L f 2 L f − 1 L f − 1 L f − 2 L f − 2 L f − 1 L f − 2 L f − 2.

Original: Stochastic
Revised: Stochastic

Original: [Table]
Revised: [Table]

Original: Top1:
Revised: Top1:

Original: [Table]
Revised: [Table]

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t I npu t O u t pu t
Revised: O u t pu t I npu t O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t 2 L f − 1 L f −
Revised: O u t pu t 2 L f − 1 L f −

Original: L f 2 L f − 1 L f − 2 L f − 1 L f − 2 L f − 1 L f −
Revised: L f 2 L f − 1 L f − 2 L f − 1 L f − 2 L f − 1 L f −

Original: L f
Revised: L f

Original: L f L f
Revised: L f L f

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t 2 L f − 1 L f −
Revised: O u t pu t 2 L f − 1 L f −

Original: L f 2 L f − 1 L f −
Revised: L f 2 L f − 1 L f −

Original: [Table]
Revised: [Table]

Original: −
Revised: −

Original: [Table]
Revised: [Table]

Original: −
Revised: −

Original: [Table]
Revised: [Table]

Original: −
Revised: −

Original: [Table]
Revised: [Table]

Original: f
Revised: f

Original: [Table]
Revised: [Table]

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t 2 L f − 1 L f −
Revised: O u t pu t 2 L f − 1 L f −

Original: L f 2 L f − 1 L f − 2 L f − 1 L f − 2 L f − 1 L f −
Revised: L f 2 L f − 1 L f − 2 L f − 1 L f − 2 L f − 1 L f −

Original: L f
Revised: L f

Original: L f L f
Revised: L f L f

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: I npu t
Revised: I npu t

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: npu t
Revised: npu t

Original: [Table]
Revised: [Table]

Original: u t
Revised: u t

Original: [Table]
Revised: [Table]

Original: t 2 L f − 2 L f − 2 L f − 1 L f − 1 L f −
Revised: t 2 L f − 2 L f − 2 L f − 1 L f − 1 L f −

Original: [Table]
Revised: [Table]

Original: f
Revised: f

Original: [Table]
Revised: [Table]

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: f
Revised: f

Original: I npu t
Revised: I npu t

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: I npu t
Revised: I npu t

Original: [Table]
Revised: [Table]

Original: u t
Revised: u t

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: L f 1 L f −
Revised: L f 1 L f −

Original: L f
Revised: L f

Original: L f 2 L f − 1 L f − 1 L f − 2 L f − 2 L f − 1 L f − 2 L f − 2.
Revised: L f 2 L f − 1 L f − 1 L f − 2 L f − 2 L f − 1 L f − 2 L f − 2.

Original: Stochastic Residual Sampling Main Network Top1:80.53%
Revised: Stochastic Residual Sampling Main Network Top1:80.53%

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t
Revised: O u t pu t

Original: I npu t
Revised: I npu t

Original: O u t pu t 2 L f − 1 L f −
Revised: O u t pu t 2 L f − 1 L f −

Original: L f 2 L f − 1 L f − 2 L f − 1 L f − 2 L f − 1 L f −
Revised: L f 2 L f − 1 L f − 2 L f − 1 L f − 2 L f − 1 L f −

Original: L f
Revised: L f

Original: L f L f
Revised: L f L f

Original: Main Network Stage 2 L f − 1 L f −
Revised: Main Network Stage 2 L f − 1 L f −

Original: L f I npu t
Revised: L f I npu t

Original: O u t pu t
Revised: O u t pu t

Original: Reduce sampling space
Revised: Reduce sampling space

Original: Conform to neural differential equation ) ( ) 2 ( n O O n  training is computed as

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: where the CE and KL are the standard cross entropy loss and KL divergence loss.

Original: λ is the balancing coefficient, and CE loss denotes the main supervision for the main network.

Original: KL loss provides an extra supervision for the sampled sub-network, and guarantees all sub-networks to be optimized in the same direction with the main network, which can be considered as setting a common goal for the “ensemble network”.

Original: We use the standard stochastic gradient descent for updating the model, denoted as θ t +1 D m = θ tD m − η ∂ L ut ∂θ Dm , where η is the learning rate.

Original: Thanks to the weight-sharing between sub-networks and the main network, we only need to update θ D m once for each iteration to optimize both the θ D m and θ D s .
Revised: Conform to neural differential equation ) ( ) 2 ( n O O n  sub-networks and the main network, we only need to update θ D m once for each iteration to optimizeboth the θ D m and θ D s .

Original: The pseudo code is shown in Appendix C.1.
Revised: The pseudo code is shown in Appendix C.1.

Original: 3.3 Ordered Residual Sampling
Revised: 3.3 Ordered Residual Sampling

Original: Network loafing is based on the novel view that residual networks can be seen as the ensemble of numerous different sub-networks[4].
Revised: Network loafing is based on the novel view that residual networks can be seen as the ensemble ofnumerous different sub-networks[4].

Original: It is intuitive to adopt raw unraveled view to design sampling space, namely raw sampling.
Revised: It is intuitive to adopt raw unraveled view to design samplingspace, namely raw sampling.

Original: However, there exist two problems in raw sampling.
Revised: However, there exist two problems in raw sampling.

Original: First, it introduces a mass of single branch sub-networks which are hard to optimize, as shown in the left sub-figure of
Revised: First, it introducesa mass of single branch sub-networks which are hard to optimize, as shown in the left sub-figure of

Original: Fig.
Revised: Fig.

Original: Second, with the growth of network depth, the size of sampling space increases exponentially.
Revised: Second, with the growth of network depth, the size of sampling space increases exponentially.

Original: For a residual stage of n blocks, there are 2 n sub-networks, which constitutes a too large space to explore.
Revised: For a residual stage of n blocks, there are 2 n sub-networks, which constitutes a too large space toexplore.

Original: Under limited computation resources, it is difficult to provide sufficient supervision to each sub-network, causing insufficient stimulative training.
Revised: Under limited computation resources, it is difficult to provide sufficient supervision to eachsub-network, causing insufficient stimulative training.

Original: Therefore, in order to facilitate stimulative training, we re-design two different sampling ways: stochastic residual sampling and ordered residual sampling.
Revised: Therefore, in order to facilitate stimulative training, we re-design two different sampling ways:stochastic residual sampling and ordered residual sampling.

Original: Fig.
Revised: Fig.

Original: 3 shows a three-block residual network as a simple illustration for three sampling ways.
Revised: 3 shows a three-block residualnetwork as a simple illustration for three sampling ways.

Original: For raw sampling, we obey the raw definition of [4], that means each sub-network is composed of stacked convolution layers.
Revised: For raw sampling, we obey the raw definitionof [4], that means each sub-network is composed of stacked convolution layers.

Original: For stochastic residual sampling, we keep the basic structure of residual blocks and skip some blocks randomly.
Revised: For stochastic residualsampling, we keep the basic structure of residual blocks and skip some blocks randomly.

Original: For ordered residual sampling, we also keep the residual structure but skip blocks orderly, (i.e., always skip the last several blocks).
Revised: For orderedresidual sampling, we also keep the residual structure but skip blocks orderly, (i.e., always skip thelast several blocks).

Original: We implement these sampling methods for MobileNetV3 [31] on CIFAR-100.
Revised: We implement these sampling methods for MobileNetV3 [23] on CIFAR-100.

Original: Experimental results show that ordered residual sampling performs better than other two methods (by 5.77% and 0.58% respectively).
Revised: Experimental results show that ordered residual sampling performs better than other two methods (by5.77% and 0.58% respectively).

Original: To explain the superiority of ordered residual sampling, we should pay attention to keeping residual structure and ordered sampling.
Revised: To explain the superiority of ordered residual sampling, we shouldpay attention to keeping residual structure and ordered sampling.

Original: For keeping residual structure, it can drive the network training process easier and improve the final performance [1].
Revised: For keeping residual structure, itcan drive the network training process easier and improve the final performance [1].

Original: For ordered sampling, it can noticeably reduce the size of sampling space (from O (2 n ) to O ( n ) ), making it possible to train each sub-network sufficiently.
Revised: For orderedsampling, it can noticeably reduce the size of sampling space (from O (2 n ) to O ( n ) ), making itpossible to train each sub-network sufficiently.

Original: Besides, all deep networks with residual connections can be approximated by neural differential equations [32], which treat discrete network layers  as [Figure] [Figure] dataset.
Revised: Besides, all deep networks with residual connectionscan be approximated by neural differential equations [24], which treat discrete network layers ascontinuous and regard the layer output as the result of adding an integral to the layer input.

Original: Sampling 0 sub-networks denotes the common training strategy.

Original: Stimulative training with more sub-networks can provide higher maximum, mean and lower standard deviation.

Original: continuous and regard the layer output as the result of adding an integral to the layer input.

Original: From this perspective, ordered sampling will maintain the continuity of residual networks.
Revised: From thisperspective, ordered sampling will maintain the continuity of residual networks.

Original: Because of these outstanding features, we will adopt ordered residual sampling in the following experiments.
Revised: Because of theseoutstanding features, we will adopt ordered residual sampling in the following experiments.

Original:  Empirical Analysis
Revised:  Empirical Analysis

Original: In this section, we conduct extensive experiments to analyze and compare the proposed stimulative training and the common training strategy in detail.
Revised: In this section, we conduct extensive experiments to analyze and compare the proposed stimulativetraining and the common training strategy in detail.

Original: We first illustrate the loafing problem of residual networks with different training strategies.
Revised: We first illustrate the loafing problem ofresidual networks with different training strategies.

Original: We further demonstrate statistical characteristics of the performance of the trained residual network and all of its sub-networks. 
Revised: We further demonstrate statistical characteristicsof the performance of the trained residual network and all of its sub-networks. Then, we show [Figure] [Figure]

Original: Then, we show the performance drop when destructing trained residual networks at test time, including deleting individual layers, deleting more layers and permuting layers.
Revised: dataset. Sampling 0 sub-networks denotes the common training strategy. Stimulative training with more sub-networks can provide higher maximum, mean and lower standard deviation. the performance drop when destructing trained residual networks at test time, including deletingindividual layers, deleting more layers and permuting layers.

Original: Besides, we also record the KL distance between a given residual network and all of its sub-networks in the whole training process.
Revised: Besides, we also record the KL distancebetween a given residual network and all of its sub-networks in the whole training process.

Original: For the convenience of analysis, we always use a NAS-searched model MobileNetV3 [31] as the residual network and train it on CIFAR100, unless otherwise specified.
Revised: For theconvenience of analysis, we always use a NAS-searched model MobileNetV3 [23] as the residualnetwork and train it on CIFAR100, unless otherwise specified.

Original: The detailed experimental settings can be found in Appendix C.2.
Revised: The detailed experimental settings canbe found in Appendix C.2.

Original: 4.1 Experiment 1: Checking the loafing problem of residual networks with different training strategies
Revised: 4.1 Experiment 1: Checking the loafing problem of residual networks with different trainingstrategies

Original: First, we find that the NAS-searched residual network MobileNetV3 still suffers from the loafing problem.
Revised: First, we find that the NAS-searched residual network MobileNetV3 still suffers from the loafingproblem.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, when applying the common training strategy to MobileNetV3, the sub-networks from MobileNetV3 (blue solid circle) have much lower accuracies than the same sub-networks individually trained (green five-pointed star), and the performance gap increases when the size of the sub-network decreases.
Revised: 4, when applying the common training strategy to MobileNetV3, thesub-networks from MobileNetV3 (blue solid circle) have much lower accuracies than the samesub-networks individually trained (green five-pointed star), and the performance gap increases whenthe size of the sub-network decreases.

Original: Second, we show that the proposed stimulative training strategy can well handle the loafing problem.
Revised: Second, we show that the proposed stimulative training strategycan well handle the loafing problem.

Original: As shown in Fig. 4(c), when applying the stimulative training strategy to MobileNetV3, the sub-networks from MobileNetV3 (red five-pointed star) have even better performance than the same sub-networks individually trained (green five-pointed star).
Revised: As shown in Fig. 4(c), when applying the stimulative trainingstrategy to MobileNetV3, the sub-networks from MobileNetV3 (red five-pointed star) have evenbetter performance than the same sub-networks individually trained (green five-pointed star).

Original: More interestingly, as shown in Fig. 4(a), even when we apply stimulative training with 6 sub-networks (red five-pointed star) to MobileNetV3, the performance of all the sub-networks (red solid circle) can be also greatly improved compared to that of common training (blue solid circle).
Revised: Moreinterestingly, as shown in Fig. 4(a), even when we apply stimulative training with 6 sub-networks(red five-pointed star) to MobileNetV3, the performance of all the sub-networks (red solid circle) canbe also greatly improved compared to that of common training (blue solid circle).

Original: Besides, As shown in Fig. 4, with more sub-networks involved in stimulative training, the network loafing problem could be relieved better, and the performance of a given residual network and all of its sub-networks could be improved more significantly.
Revised: Besides, As shownin Fig. 4, with more sub-networks involved in stimulative training, the network loafing problem couldbe relieved better, and the performance of a given residual network and all of its sub-networks couldbe improved more significantly.

Original: More similar results of different datasets and residual networks can be found in Appendix A.
Revised: More similar results of different datasets and residual networks canbe found in Appendix A.

Original: 4.2 Experiment 2: Collecting statistical characteristics of the performance of all residual sub-networks
Revised: 4.2 Experiment 2: Collecting statistical characteristics of the performance of all residualsub-networks

Original: To demonstrate that stimulative training can improve the performance of a given residual network and all of its sub-networks, we collect various statistical characteristics of the performance of all residual  sub-networks after the stimulative training and common training.
Revised: To demonstrate that stimulative training can improve the performance of a given residual network andall of its sub-networks, we collect various statistical characteristics of the performance of all residual [Figure] sub-networks after the stimulative training and common training.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, MobileNetV with common training (i.e., sample 0 sub-networks) has the lowest maximum value, mean value, and the highest standard deviation, which is mainly caused by the network loafing problem.
Revised: 5, MobileNetV3with common training (i.e., sample 0 sub-networks) has the lowest maximum value, mean value, andthe highest standard deviation, which is mainly caused by the network loafing problem.

Original: As a contrast,
Revised: As a contrast,

Original: MobileNetV3 with stimulative training (i.e., sample 144 sub-networks) has the highest maximum value, mean value, and the lowest standard deviation.
Revised: MobileNetV3 with stimulative training (i.e., sample 144 sub-networks) has the highest maximumvalue, mean value, and the lowest standard deviation.

Original: Moreover, we can see that, MobileNetV3 using stimulative training with more sub-networks can usually provide higher maximum value, mean value and lower standard deviation.
Revised: Moreover, we can see that, MobileNetV3 usingstimulative training with more sub-networks can usually provide higher maximum value, mean valueand lower standard deviation.

Original: All these results indicate that stimulative training can improve the performance of all residual sub-networks effectively and efficiently.
Revised: All these results indicate that stimulative training can improve theperformance of all residual sub-networks effectively and efficiently.

Original: 4.3 Experiment 3: Destructing residual networks at test time
Revised: 4.3 Experiment 3: Destructing residual networks at test time

Original: In this experiment, we verify that stimulative training can provide stronger robustness in resisting various network destruction operations including deleting individual layers, deleting more layers and permuting layers than common training.
Revised: In this experiment, we verify that stimulative training can provide stronger robustness in resistingvarious network destruction operations including deleting individual layers, deleting more layers andpermuting layers than common training.

Original: Deleting individual layers.
Revised: Deleting individual layers.

Original: We first show the performance drop when deleting individual layers from trained MobileNetV3 at test time.
Revised: We first show the performance drop when deleting individual layersfrom trained MobileNetV3 at test time.

Original: As shown in Fig. 6(a), when applying common training to
Revised: As shown in Fig. 6(a), when applying common training to

Original: MobileNetV3, the performance drop is non-negligible when some layers are deleted (especially for the last several layers of each stage), and the trained MobileNetV3 is sensitive to the index of deleted layer.
Revised: MobileNetV3, the performance drop is non-negligible when some layers are deleted (especially forthe last several layers of each stage), and the trained MobileNetV3 is sensitive to the index of deletedlayer.

Original: As a comparison, when applying stimulative training to MobileNetV3, no matter which layer is deleted, the performance drop of stimulative trained model is always lower than that of the common trained model, and the performance drop is always smaller than or equal to 1%.
Revised: As a comparison, when applying stimulative training to MobileNetV3, no matter which layer isdeleted, the performance drop of stimulative trained model is always lower than that of the commontrained model, and the performance drop is always smaller than or equal to 1%.

Original: Such results verify that stimulative training can reduce the dependence between sub-networks significantly.
Revised: Such results verifythat stimulative training can reduce the dependence between sub-networks significantly.

Original: Deleting more layers.
Revised: Deleting more layers.

Original: We then show the performance drop when deleting more layers from trained
Revised: We then show the performance drop when deleting more layers from trained

Original: MobileNetV3 at test time.
Revised: MobileNetV3 at test time.

Original: Note that, deleting more layers may come in a variety of combinations, thus we show the statistical box plot of performance drops for all combinations.
Revised: Note that, deleting more layers may come in a variety of combinations,thus we show the statistical box plot of performance drops for all combinations.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6(b), when applying common training to MobileNetV3, the medium value, minimum value and maximum value of performance drop for all combinations keeps continuously rising as the number of deleted layers increases.
Revised: 6(b),when applying common training to MobileNetV3, the medium value, minimum value and maximumvalue of performance drop for all combinations keeps continuously rising as the number of deletedlayers increases.

Original: And the difference in performance drop for different combinations when removing the same number of layers can be very large.
Revised: And the difference in performance drop for different combinations when removingthe same number of layers can be very large.

Original: For example, when deleting 4 layers of common trained
Revised: For example, when deleting 4 layers of common trained

Original: MobileNetV3, the Top1 accuracy drop for different combinations can vary from less than 5% to higher than 35%.
Revised: MobileNetV3, the Top1 accuracy drop for different combinations can vary from less than 5% tohigher than 35%.

Original: Differently, when applying stimulative training, no matter how many layers are deleted, the values of performance drop for all combinations always keep very low (i.e., no more than 3%), indicating that stimulative training can bring strong robustness in resisting network destruction.
Revised: Differently, when applying stimulative training, no matter how many layers aredeleted, the values of performance drop for all combinations always keep very low (i.e., no more than3%), indicating that stimulative training can bring strong robustness in resisting network destruction.

Original: Permuting layers.
Revised: Permuting layers.

Original: We further show the performance drop when permuting layers in trained
Revised: We further show the performance drop when permuting layers in trained

Original: MobileNetV3 at test time.
Revised: MobileNetV3 at test time.

Original: Note that, only the layers in the same stage can be permuted since the resolution and width of different stages are varying, and we show the detailed experimental settings in
Revised: Note that, only the layers in the same stage can be permuted since theresolution and width of different stages are varying, and we show the detailed experimental settings in

Original: Appendix C.3.
Revised: Appendix C.3.

Original: As shown in Fig. 6(c), when applying common training to MobileNetV3, the medium value, minimum value and maximum value of performance drop for all combinations increases with the rising number of permuted layers, and the variance of performance drop for different combinations when permuting the same number of layers may be also very large.
Revised: As shown in Fig. 6(c), when applying common training to MobileNetV3, the mediumvalue, minimum value and maximum value of performance drop for all combinations increases withthe rising number of permuted layers, and the variance of performance drop for different combinationswhen permuting the same number of layers may be also very large.

Original: As a comparison, the values of performance drop for all permuting combinations always keep very low (i.e., less than 1%), when stimulative training is applied.
Revised: As a comparison, the values ofperformance drop for all permuting combinations always keep very low (i.e., less than 1%), whenstimulative training is applied.

Original: This further verifies the superiority of stimulative training.
Revised: This further verifies the superiority of stimulative training.

Original: [Figure]
Revised: Experiment : Recording the

Original: 4.4 Experiment 4: Recording the KL divergence between a residual network and all of its sub-networks when training
Revised: KL divergence between a residual network and all of itssub-networks when training

Original: To study the consistency between the output of a given residual network and all of its sub-networks, we record their KL distance in the whole training process.
Revised: To study the consistency between the output of a given residual network and all of its sub-networks, we record their KL distance in the whole training process.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 7, when applying common training to MobileNetV3, the variance, maximum and medium value of KL distance between a given residual network and all of its sub-networks continuously increases with rising training epoch.
Revised: 7, when applying common training to MobileNetV3, the variance, maximum and medium value of KL distance between a given residual network and all of its sub-networks continuously increases with rising training epoch.

Original: For comparison, when stimulative training is applied to MobileNetV3, all values of KL distance between MobileNetV3 and all of its sub-networks keep very small (i.e., no more than 0.25) in the whole training process.
Revised: For comparison, when stimulative training is applied to MobileNetV3, all values of KL distance between MobileNetV3 and all of its sub-networks keep very small (i.e., no more than 0.25) in the whole training process.

Original: Such results imply that the proposed stimulative training can maintain the consistency between the output of a given residual network and all of its sub-networks.
Revised: Such results imply that the proposed stimulative training can maintain the consistency between the output of a given residual network and all of its sub-networks.

Original: [Figure]
Revised: [Figure]

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: Analysis 1: Why the performance of sub-networks can be improved.
Revised: Analysis 1: Why the performance of sub-networks can be improved.

Original: Based on Eq. 1, our method can be treated as optimizing CE ( Z ( θ D m , x ) , y ) and KL ( Z ( θ D m , x ) , Z ( θ D s , x )) at the same time.
Revised: Based on Eq. 1, ourmethod can be treated as optimizing CE ( Z ( θ D m , x ) , y ) and KL ( Z ( θ D m , x ) , Z ( θ D s , x )) at thesame time.

Original: According to the convergence of stochastic gradient descent, after enough number of iterations, both losses can be bounded by a constant, which is formulated as
Revised: According to the convergence of stochastic gradient descent, after enough number ofiterations, both losses can be bounded by a constant, which is formulated as

Revised: CE ( Z ( θ D m , x ) , y )

Revised: = −

Revised: N (cid:88) i =

Revised: y i log p mi

Original: [Equation]
Revised: [Equation]

Original: where N is the number of categories.
Revised: where N is the number of categories.

Original: The ϵ 1 and ϵ 2 are the tiny constants for corresponding losses.
Revised: The ϵ 1 and ϵ 2 are the tiny constants for corresponding losses.

Original: Θ is the set of all sub-networks.
Revised: Θ is the set of all sub-networks.

Original: The p m and p s are the classification probability of main network and sub-network, respectively.
Revised: The p m and p s are the classification probability of main networkand sub-network, respectively.

Original: Further, we can prove that the absolute difference of cross entropy of sub-networks and main network can be bounded by a constant (detailed proof can be found in
Revised: Further, we can prove that the absolute difference of cross entropyof sub-networks and main network can be bounded by a constant (detailed proof can be found in

Original: Appendix B), which can be formulated as
Revised: Appendix B), which can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where ϵ 2 +log N e − ϵ+ ϵ 1 is a small constant after optimization.
Revised: where ϵ 2 +log N e − ϵ+ ϵ 1 is a small constant after optimization.

Original: Eq.
Revised: Eq.

Original: 4 demonstrates that the performance gap between sub-networks and the main network is constrained.
Revised: 4 demonstrates that the performancegap between sub-networks and the main network is constrained.

Original: Thus, as the rising of the main network performance, sub-networks performance will be improved step by step .
Revised: Thus, as the rising of the mainnetwork performance, sub-networks performance will be improved step by step .

Original: [Table] Analysis 2: Why the performance of given residual networks can be improved.
Revised: Analysis 2: Why the performance of given residual networks can be improved.

Original: Neural network training is in fact a form of polynomial regression, and any neural network with activation functions can roughly correspond to a fitted polynomial [33].
Revised: Neural networktraining is in fact a form of polynomial regression, and any neural network with activation functionscan roughly correspond to a fitted polynomial [25].

Original: For simplifying notations, suppose that both input x and output y in a neural network have only one element.
Revised: For simplifying notations, suppose that bothinput x and output y in a neural network have only one element.

Original: Then, we can consider the training of a neural network as the fitting of a polynomial function.
Revised: Then, we can consider the trainingof a neural network as the fitting of a polynomial function.

Original: [Equation]
Revised: [Equation]

Original: (5) where c 1 , c 2 , . . .
Revised: (5) where c 1 , c 2 , . . .

Original: , c n , . .
Revised: , c n , . .

Original: . represent the polynomial coefficients.
Revised: . represent the polynomial coefficients.

Original: Based on training samples, we can establish the polynomial (neural network), which denotes the mapping relationship between inputs and outputs.
Revised: Based on training samples, we canestablish the polynomial (neural network), which denotes the mapping relationship between inputsand outputs.

Original: For a test sample x , there always exists a point x 0 (e.g., a training sample) close enough to x , which can obtain an accurate output via the trained model.
Revised: For a test sample x , there always exists a point x 0 (e.g., a training sample) close enoughto x , which can obtain an accurate output via the trained model.

Original: According to Taylor expansion, F ( x ) at x 0 can be computed as :
Revised: According to Taylor expansion, F ( x )at x 0 can be computed as :

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: where R n ( x ) denotes the higher degree infinitesimal of ( x − x 0 ) n .
Revised: where R n ( x ) denotes the higher degree infinitesimal of ( x − x 0 ) n .

Original: According to Eq.
Revised: According to Eq.

Original: 5 and Eq.
Revised: 5 and Eq.

Original: 6, it is obvious that different terms of the fitted polynomial have different effects, and the low-degree terms tend to have greater effects on the predicted performance.
Revised: 6, it isobvious that different terms of the fitted polynomial have different effects, and the low-degree termstend to have greater effects on the predicted performance.

Original: Based on this point, [5] further figures out that shallow sub-networks in residual networks roughly correspond to low-degree polynomials consisting of low-degree terms, while deep sub-networks are opposite.
Revised: Based on this point, [5] further figuresout that shallow sub-networks in residual networks roughly correspond to low-degree polynomialsconsisting of low-degree terms, while deep sub-networks are opposite.

Original: Following this theory, the proposed stimulative training can not only strengthen the learning of different items of the polynomial (neural network), but also pay more attention to the learning of low-degree items .
Revised: Following this theory, theproposed stimulative training can not only strengthen the learning of different items of the polynomial(neural network), but also pay more attention to the learning of low-degree items .

Original: As a result, the final performance of given residual networks can be improved by stimulative training.
Revised: As a result, thefinal performance of given residual networks can be improved by stimulative training.

Original:  Verification on Various Datasets and Networks
Revised:  Verification on Various Datasets and Networks

Original: CIFAR implementation details.
Revised: CIFAR implementation details.

Original: CIFAR[34] is a classical image classification dataset consisting of 50,000 training images and 10,000 testing images.
Revised: CIFAR[26] is a classical image classification dataset consistingof 50,000 training images and 10,000 testing images.

Original: It includes CIFAR-100 in 100 categories and
Revised: It includes CIFAR-100 in 100 categories and

Original: CIFAR-10 in 10 categories.
Revised: CIFAR-10 in 10 categories.

Original: For MobileNetV3 and ResNet50, the data augmentations follow [35], we use SGD optimizer and train the model for 500 epochs with a batch size of 64.
Revised: For MobileNetV3 and ResNet50, the data augmentations follow [27], weuse SGD optimizer and train the model for 500 epochs with a batch size of 64.

Original: The initial learning rate is 0.05 with cosine decay schedule.
Revised: The initial learningrate is 0.05 with cosine decay schedule.

Original: The weight decay is 3 × 10 − 5 and momentum is 0.9.
Revised: The weight decay is 3 × 10 − 5 and momentum is 0.9.

Original: ImageNet implementation details.
Revised: ImageNet implementation details.

Original: We implement our method on large-scale ImageNet[36] dataset containing 1.2 million training images and 50,000 validation images from 1,000 categories.
Revised: We implement our method on large-scale ImageNet[28] datasetcontaining 1.2 million training images and 50,000 validation images from 1,000 categories.

Original: For
Revised: For

Original: ResNet families, we use standard data augmentations for ImageNet, as done in [37, 38, 39].
Revised: ResNet families, we use standard data augmentations for ImageNet, as done in [29, 30, 31].

Original: We utilize SGD optimizer to train the model for 100 epochs with a batch size of 512, and the learning rate is 0.2 with cosine decay schedule.
Revised: Weutilize SGD optimizer to train the model for 100 epochs with a batch size of 512, and the learningrate is 0.2 with cosine decay schedule.

Original: The weight decay is 1 × 10 − 4 and momentum is 0.9.
Revised: The weight decay is 1 × 10 − 4 and momentum is 0.9.

Original: Main network and sub-networks performance.
Revised: Main network and sub-networks performance.

Original: We further verify the effectiveness of stimulative training (ST) on various networks and datasets.
Revised: We further verify the effectiveness of stimulativetraining (ST) on various networks and datasets.

Original: Table.1 shows its comparison with common training (CT).
Revised: Table.1 shows its comparison with common training(CT).

Original: Besides the Top1 accuracy of main network, we also report the the mean value and standard deviation of Top1 accuracy for all sub-networks.
Revised: Besides the Top1 accuracy of main network, we also report the the mean value and standarddeviation of Top1 accuracy for all sub-networks.

Original: It is obvious that stimulative training can noticeably improve the performance of various networks on various datasets compared to common training (e.g., by 4.43% for ResNet50 on CIFAR-100 and 1.21% for ResNet101 on ImageNet).
Revised: It is obvious that stimulative training can noticeablyimprove the performance of various networks on various datasets compared to common training (e.g.,by 4.43% for ResNet50 on CIFAR-100 and 1.21% for ResNet101 on ImageNet).

Original: Moreover, the mean performance of all sub-networks is dramatically improved and standard deviation greatly drops.
Revised: Moreover, the meanperformance of all sub-networks is dramatically improved and standard deviation greatly drops.

Original: For example, stimulative training for MobileNetV3 on CIFAR100 has a mean performance of 80.01% (much higher than common training of 55.26%) and a standard deviation of 0.59% (much lower than 13.37%).
Revised: Forexample, stimulative training for MobileNetV3 on CIFAR100 has a mean performance of 80.01%(much higher than common training of 55.26%) and a standard deviation of 0.59% (much lowerthan 13.37%).

Original: Such results validate that stimulative training can well alleviate the network loafing problem, and improve the performance of the given residual network and all of its sub-networks.
Revised: Such results validate that stimulative training can well alleviate the network loafingproblem, and improve the performance of the given residual network and all of its sub-networks.

Original: Experimental results on various models and datasets verify the generalization of stimulative training, we advocate utilizing it as a general technology to train residual networks.
Revised: Experimental results on various models and datasets verify the generalization of stimulative training,we advocate utilizing it as a general technology to train residual networks.

Original:  Conclusions
Revised:  Conclusions

Original: The paper understands and improves residual networks from a social psychology perspective of loafing.
Revised: The paper understands and improves residual networks from a social psychology perspective of loafing.

Original: Based on the novel view that a residual network behaves like an ensemble network, we find that various residual networks invariably exhibit loafing-like behaviors that are consistent with the social loafing problem of social psychology.
Revised: Based on the novel view that a residual network behaves like an ensemble network, we find that various residual networks invariably exhibit loafing-like behaviors that are consistent with the social loafing problem of social psychology.

Original: We define this previously overlooked problem as network loafing.
Revised: We define this previously overlooked problem as network loafing.

Original: As the loafing problem hinders the productivity of each individual and the whole collective, we learn from social psychology and propose a stimulative training strategy to solve the network loafing problem.
Revised: As the loafing problem hinders the productivity of each individual and the whole collective, we learn from social psychology and propose a stimulative training strategy to solve the network loafing problem.

Original: Comprehensive empirical analyses show that stimulative training can improve the performance of a given residual network and all of its sub-networks, and provide strong robustness in resisting various network destruction operations.
Revised: Comprehensive empirical analyses show that stimulative training can improve the performance of a given residual network and all of its sub-networks and provide strong robustness in resisting various network destruction operations.

Original: Furthermore, we theoretically show why the performance of a given residual network and its sub-networks can be improved.
Revised: Furthermore, we theoretically show why the performance of a given residual network and its sub-networks can be improved.

Original: Experiments on various datasets and residual networks demonstrate the effectiveness of the proposed method.  Limitations
Revised: Experiments on various datasets and residual networks demonstrate the effectiveness of the proposed method

Original: The proposed method suffers from about 1.4 times of computation cost of the original model training to get better performance and robustness.

Original: As the first research of network loafing problem, the proposed method is a positive pioneer-like exploration.

Original: We believe designing a more efficient method to solve the network loafing problem is a worthy research direction in the future.

Original: Residual structure is widely applied in numerous different types of models including DenseNet and transformer.

Original: It will be of vital value to study whether the loafing problem exists in these models and explore the proper method to solve this problem.

Original: Since most of existing works overlook the loafing problem, it may also be a feasible way to apply the proposed method in pretrained models.

Original: What’s more, the proposed method adopts main network logits as supervision to alleviate loafing and analogously, the proposed method may be applied in self-supervised learning.

Original: We believe that taking full advantage of splendid achievements in interdisciplinary research can help promote the development of deep learning.

Original: We hope that this paper can provide a new perspective to inspire more researchers to comprehend and improve deep neural networks from other fields such as social psychology.

Original: Acknowledgements

Original: This work is supported by National Natural Science Foundation of China (No. U1909207 and 62071127), and Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103).

Original: Wanli Ouyang is supported by the Australian Research Council Grant DP200103223, Australian

Original: Medical Research Future Fund MRFAI000085, CRC-P Smart Material Recovery Facility (SMRF) –

Original: Curby Soft Plastics, and CRC-P ARIA - bionic visual-spatial prosthesis for the blind.

Original: Checklist

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [No] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]