Original: LIT: B LOCK - WISE I NTERMEDIATE R EPRESENTATION T RAINING FOR M ODEL C OMPRESSION
Revised: LIT: B LOCK - WISE I NTERMEDIATE R EPRESENTATION T RAINING FOR M ODEL C OMPRESSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Knowledge distillation (KD) is a popular method for reducing the computational overhead of deep network inference, in which the output of a teacher model is used to train a smaller, faster student model.
Revised: Knowledge distillation (KD) is a popular method for reducing the computational overhead of deep network inference, in which the output of a teacher model is used to train a smaller, faster student model.

Original: Hint training (i.e., FitNets) extends KD by regressing a student model’s intermediate representation to a teacher model’s intermediate representation.
Revised: Hint training (i.e., FitNets) extends KD by regressing a student model’s intermediate representation to a teacher model’s intermediate representation.

Original: In this work, we introduce b L ock-wise
Revised: In this work, we introduce b L ock-wise

Original: I ntermediate representation T raining (LIT), a novel model compression technique that extends the use of intermediate representations in deep network compression, outperforming KD and hint training.
Revised: I ntermediate representation T raining (LIT), a novel model compression technique that extends the use of intermediate representations in deep network compression, outperforming KD and hint training.

Original: LIT has two key ideas: 1) LIT trains a student of the same width (but shallower depth) as the teacher by directly comparing the intermediate representations, and 2) LIT uses the intermediate representation from the previous block in the teacher model as an input to the current student block during training, avoiding unstable intermediate representations in the student network.
Revised: LIT has two key ideas: 1) LIT trains a student of the same width (but shallower depth) as the teacher by directly comparing the intermediate representations, and 2) LIT uses the intermediate representation from the previous block in the teacher model as an input to the current student block during training, avoiding unstable intermediate representations in the student network.

Original: We show that LIT provides substantial reductions in network depth without loss in accuracy — for example, LIT can compress a ResNeXt-110 to a ResNeXt- (5.5 × ) on CIFAR10 and a VDCNN-29 to a VDCNN-9 (3.2 × ) on Amazon Reviews without loss in accuracy, outperforming KD and hint training in network size for a given accuracy.
Revised: We show that LIT provides substantial reductions in network depth without loss in accuracy — for example, LIT can compress a ResNeXt-110 to a ResNeXt- (5.5 × ) on CIFAR10 and a VDCNN-29 to a VDCNN-9 (3.2 × ) on Amazon Reviews without loss in accuracy, outperforming KD and hint training in network size for a given accuracy.

Original: We also show that applying LIT to identical student/teacher architectures increases the accuracy of the student model above the teacher model, outperforming the recently-proposed Born Again Networks procedure on ResNet, ResNeXt, and VDCNN.
Revised: We also show that applying LIT to identical student/teacher architectures increases the accuracy of the student model above the teacher model, outperforming the recently-proposed Born Again Networks procedure on ResNet, ResNeXt, and VDCNN.

Original: Finally, we show that LIT can effectively compress GAN generators.
Revised: Finally, we show that LIT can effectively compress GAN generators.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modern deep networks have achieved increased accuracy by continuing to introduce more layers (Ioffe & Szegedy, 2015;
Revised: Modern deep networks have achieved increased accuracy by continuing to introduce more layers (Ioffe & Szegedy, 2015;

Original: He et al., 2016) at the cost of higher computational overhead.
Revised: He et al., 2016) at the cost of higher computational overhead.

Original: In response, researchers have proposed many techniques to reduce this computational overhead at inference time, which broadly fall under two categories.
Revised: In response, researchers have proposed many techniques to reduce this computational overhead at inference time, which broadly fall under two categories.

Original: First, in deep compression (Han et al., 2015a; Zhu et al., 2016; Li et al., 2016;
Revised: First, in deep compression (Han et al., 2015a; Zhu et al., 2016; Li et al., 2016;

Original: Hubara et al., 2017), parts of a model are removed or quantized to reduce the number of weights and/or the computational footprint.
Revised: Hubara et al., 2017), parts of a model are removed or quantized to reduce the number of weights and/or the computational footprint.

Original: However, deep compression techniques typically require new hardware (Han et al., 2016) to take advantage of the resulting model sparsity.
Revised: However, deep compression techniques typically require new hardware (Han et al., 2016) to take advantage of the resulting model sparsity.

Original: Second, in student/teacher methods—introduced in knowledge distillation (KD) (Hinton et al., 2014) and further extended (Romero et al., 2015; Kim &
Revised: Second, in student/teacher methods—introduced in knowledge distillation (KD) (Hinton et al., 2014) and further extended (Romero et al., 2015; Kim &

Original: Rush, 2016; Furlanello et al., 2018)—a smaller student model learns from a large teacher model through distillation loss, wherein the student model attempts to match the logits of the teacher model.
Revised: Rush, 2016; Furlanello et al., 2018)—a smaller student model learns from a large teacher model through distillation loss, wherein the student model attempts to match the logits of the teacher model.

Original: As there are no constraints on the teacher and student models, KD can produce hardware-friendly models: the student can be a standard model architecture (e.g., ResNet), optimized for a given hardware substrate.
Revised: As there are no constraints on the teacher and student models, KD can produce hardware-friendly models: the student can be a standard model architecture (e.g., ResNet), optimized for a given hardware substrate.

Original: Hint training (i.e., FitNets (Romero et al., 2015)) extends KD by using a teacher’s intermediate representation (IR, i.e., the output from a hidden layer) to guide the training of the student model.
Revised: Hint training (i.e., FitNets (Romero et al., 2015)) extends KD by using a teacher’s intermediate representation (IR, i.e., the output from a hidden layer) to guide the training of the student model.

Original: The authors show that hint training with a single IR outperforms KD in compressing teacher networks (e.g., maxout networks (Goodfellow et al., 2013)) to thinner and deeper student networks.
Revised: The authors show that hint training with a single IR outperforms KD in compressing teacher networks (e.g., maxout networks (Goodfellow et al., 2013)) to thinner and deeper student networks.

Original: We ask the natural question: does hint training compress more modern, highly-structured, very deep networks—such as ResNet (He et al., 2016), ResNeXt (Xie et al., 2017), VDCNN (Conneau et al., 2016), and StarGAN (Choi et al., 2017)?
Revised: We ask the natural question: does hint training compress more modern, highly-structured, very deep networks—such as ResNet (He et al., 2016), ResNeXt (Xie et al., 2017), VDCNN (Conneau et al., 2016), and StarGAN (Choi et al., 2017)?

Original: We find that standard hint training (i.e., with a single hint) and training with multiple hints is not effective for modern deep networks (Section 4.2).
Revised: We find that standard hint training (i.e., with a single hint) and training with multiple hints is not effective for modern deep networks (Section 4.2).

Original: We hypothesize that, for
Revised: We hypothesize that, for

Original: [Figure]
Revised: [Figure]

Original: modern deep networks, hint training causes unstable IRs: the deepest network considered in Romero et al. (2015) was only 17 layers, achieving 91.61% on CIFAR10; in contrast, a modern 110-layer ResNet achieves 93.68% on CIFAR10.
Revised: modern deep networks, hint training causes unstable IRs: the deepest network considered in Romero et al. (2015) was only 17 layers, achieving 91.61% on CIFAR10; in contrast, a modern 110-layer ResNet achieves 93.68% on CIFAR10.

Original: In this work, we extend hint training’s ability to transfer intermediate knowledge from teacher to student to reduce the depth of modern, highly-structured architectures (e.g., compressing a standard ResNeXtto a standard ResNeXt-20 with no loss in accuracy).
Revised: In this work, we extend hint training’s ability to transfer intermediate knowledge from teacher to student to reduce the depth of modern, highly-structured architectures (e.g., compressing a standard ResNeXtto a standard ResNeXt-20 with no loss in accuracy).

Original: We do this via a novel method called
Revised: We do this via a novel method called

Original: b L ock-wise
Revised: b L ock-wise

Original: I ntermediate representation T raining (LIT), a student/teacher compression technique that outperforms training student networks from scratch, hint training, and KD.
Revised: I ntermediate representation T raining (LIT), a student/teacher compression technique that outperforms training student networks from scratch, hint training, and KD.

Original: LIT targets highly structured, modern networks that consist of repetitive blocks (i.e., groups of layers) that can be scaled up/down for accuracy/speed tradeoffs; for example, ResNets have standard configurations from 20 to hundreds of layers.
Revised: LIT targets highly structured, modern networks that consist of repetitive blocks (i.e., groups of layers) that can be scaled up/down for accuracy/speed tradeoffs; for example, ResNets have standard configurations from 20 to hundreds of layers.

Original: LIT leverages two key ideas to reduce unstable IRs in deep networks.
Revised: LIT leverages two key ideas to reduce unstable IRs in deep networks.

Original: First, LIT directly trains student networks of the same width as the teacher model (as opposed to using a single, thinner hint as in hint training).
Revised: First, LIT directly trains student networks of the same width as the teacher model (as opposed to using a single, thinner hint as in hint training).

Original: Second, LIT avoids unstable student IRs deep in the network by using the IR from the previous block in the teacher model as input to the current student block during training; each student block is effectively trained in isolation to match the corresponding (deeper) block in the teacher.
Revised: Second, LIT avoids unstable student IRs deep in the network by using the IR from the previous block in the teacher model as input to the current student block during training; each student block is effectively trained in isolation to match the corresponding (deeper) block in the teacher.

Original: We show that LIT’s block-wise training improves accuracy, allows for copying parts of the teacher model directly to the student model, and permits selective compression of networks (e.g., compressing one out of three blocks in a network and copying the rest).
Revised: We show that LIT’s block-wise training improves accuracy, allows for copying parts of the teacher model directly to the student model, and permits selective compression of networks (e.g., compressing one out of three blocks in a network and copying the rest).

Original: For example, consider compressing a ResNet-56 from a ResNet-110 (Figure 1), each of which have four stages.
Revised: For example, consider compressing a ResNet-56 from a ResNet-110 (Figure 1), each of which have four stages.

Original: The IR loss is applied to the output of each block, and the teacher model’s IRs are used as input to the student blocks.
Revised: The IR loss is applied to the output of each block, and the teacher model’s IRs are used as input to the student blocks.

Original: We show that LIT outperforms standard KD on a range of models (ResNet, ResNeXt, VDCNN, StarGAN) and datasets (CIFAR10, CIFAR100, Amazon Reviews, CelebA): empirically, LIT can reduce model sizes from 1.7 × to 5.5 × with no loss in accuracy.
Revised: We show that LIT outperforms standard KD on a range of models (ResNet, ResNeXt, VDCNN, StarGAN) and datasets (CIFAR10, CIFAR100, Amazon Reviews, CelebA): empirically, LIT can reduce model sizes from 1.7 × to 5.5 × with no loss in accuracy.

Original: Recent work on Born Again networks (Furlanello et al., 2018) uses standard KD to train identical student and teacher models to higher accuracies (i.e., no compression).
Revised: Recent work on Born Again networks (Furlanello et al., 2018) uses standard KD to train identical student and teacher models to higher accuracies (i.e., no compression).

Original: We show that the benefits of this procedure also apply to LIT student/teacher training, and LIT enables up to 0.64% higher accuracy than KD-based Born Again networks on the networks we consider.
Revised: We show that the benefits of this procedure also apply to LIT student/teacher training, and LIT enables up to 0.64% higher accuracy than KD-based Born Again networks on the networks we consider.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Knowledge distillation.
Revised: Knowledge distillation.

Original: Hinton et al.
Revised: Hinton et al.

Original: ; Bucilu et al.
Revised: ; Bucilu et al.

Original: (2006) introduced knowledge distillation (KD) in which a teacher ensemble or models outputs are used to train a smaller student model, which inspired a variety of related methods, e.g., for cross-modal distillation or faster training (Gupta et al., 2016;
Revised: (2006) introduced knowledge distillation (KD) in which a teacher ensemble or models outputs are used to train a smaller student model, which inspired a variety of related methods, e.g., for cross-modal distillation or faster training (Gupta et al., 2016;

Original: Chen et al., 2015; Frosst & Hinton, 2017; Romero et al., 2015; Furlanello et al., 2018).
Revised: Chen et al., 2015; Frosst & Hinton, 2017; Romero et al., 2015; Furlanello et al., 2018).

Original: FitNets extends KD by regressing a student model’s IR to a teacher model’s IR, as the student models they consider are thinner and deeper.
Revised: FitNets extends KD by regressing a student model’s IR to a teacher model’s IR, as the student models they consider are thinner and deeper.

Original: Wang et al.
Revised: Wang et al.

Original: (2018) extends FitNets by training networks iteratively using hints.
Revised: (2018) extends FitNets by training networks iteratively using hints.

Original: In contrast,
Revised: In contrast,

Original: LIT uses the teacher IRs as input to the student model in training and directly penalizes deviations of the student model’s IRs from teacher model’s IRs, which helps guide training for higher accuracy and improved inference performance.
Revised: LIT uses the teacher IRs as input to the student model in training and directly penalizes deviations of the student model’s IRs from teacher model’s IRs, which helps guide training for higher accuracy and improved inference performance.

Original: In Born Again networks (Furlanello et al., 2018), the same network architecture is used as both the teacher and student in standard KD, resulting in higher accuracy.
Revised: In Born Again networks (Furlanello et al., 2018), the same network architecture is used as both the teacher and student in standard KD, resulting in higher accuracy.

Original: We show that LIT outperforms the Born Again procedure on ResNet, ResNeXt, and VDCNN.
Revised: We show that LIT outperforms the Born Again procedure on ResNet, ResNeXt, and VDCNN.

Original: Deep compression.
Revised: Deep compression.

Original: In deep compression, parts of a network (weights, groups of weights, kernels, or filters (Mao et al., 2017)) are removed for efficient inference (Han et al., 2015a), and the weights of the network are quantized, hashed, or compressed (Hubara et al., 2016; Rastegari et al., 2016; Zhu et al., 2016;
Revised: In deep compression, parts of a network (weights, groups of weights, kernels, or filters (Mao et al., 2017)) are removed for efficient inference (Han et al., 2015a), and the weights of the network are quantized, hashed, or compressed (Hubara et al., 2016; Rastegari et al., 2016; Zhu et al., 2016;

Original: Hubara et al., 2017).
Revised: Hubara et al., 2017).

Original: These methods largely do not take advantage of a teacher model and typically require new hardware for efficiency gains (Han et al., 2016).
Revised: These methods largely do not take advantage of a teacher model and typically require new hardware for efficiency gains (Han et al., 2016).

Original: Methods that prune filters (Li et al., 2016) can result in speedups on existing hardware, but largely degrade accuracy.
Revised: Methods that prune filters (Li et al., 2016) can result in speedups on existing hardware, but largely degrade accuracy.

Original: We show that LIT models can be pruned, and thus these methods are complementary to LIT.
Revised: We show that LIT models can be pruned, and thus these methods are complementary to LIT.

Original: Additionally, deep compression does not perform as well on modern networks: Han et al.
Revised: Additionally, deep compression does not perform as well on modern networks: Han et al.

Original: (2015a)
Revised: (2015a)

Original: compressed VGG by ~10 × , but ResNet-110 can only be compressed ~1.6 × (Li et al., 2016), compared to LIT’s 4 × compression.
Revised: compressed VGG by ~10 × , but ResNet-110 can only be compressed ~1.6 × (Li et al., 2016), compared to LIT’s 4 × compression.

Original: Network architectures for fast inference.
Revised: Network architectures for fast inference.

Original: Researchers have proposed network architectures (e.g., MobileNet (Howard et al., 2017)) and new operations for fast inference (e.g., ShuffleNet (Zhang et al., 2017)) on specific hardware.
Revised: Researchers have proposed network architectures (e.g., MobileNet (Howard et al., 2017)) and new operations for fast inference (e.g., ShuffleNet (Zhang et al., 2017)) on specific hardware.

Original: However, these architectures and operations are largely designed for power/resourceconstrained mobile devices and sacrifice accuracy for low power.
Revised: However, these architectures and operations are largely designed for power/resourceconstrained mobile devices and sacrifice accuracy for low power.

Original: We focus on highly accurate, very deep networks in this work.
Revised: We focus on highly accurate, very deep networks in this work.

Original:  M ETHODS
Revised:  M ETHODS

Original: LIT uses an augmented loss function and training procedure to distill a teacher model into a student model.
Revised: LIT uses an augmented loss function and training procedure to distill a teacher model into a student model.

Original: In its training procedure, LIT both 1) penalizes deviations of the student model’s IRs from the teacher model’s IRs (IR loss) and 2) uses the KD loss (for the entire student network).
Revised: In its training procedure, LIT both 1) penalizes deviations of the student model’s IRs from the teacher model’s IRs (IR loss) and 2) uses the KD loss (for the entire student network).

Original: As LIT directly penalizes deviations in IRs, LIT requires that the teacher model and student model have outputs of the same size at some intermediate layer.
Revised: As LIT directly penalizes deviations in IRs, LIT requires that the teacher model and student model have outputs of the same size at some intermediate layer.

Original: A key challenge in the LIT procedure is that the student network will not have meaningful IRs for a large part of the training (e.g., at the start of training when the weights are initialized randomly).
Revised: A key challenge in the LIT procedure is that the student network will not have meaningful IRs for a large part of the training (e.g., at the start of training when the weights are initialized randomly).

Original: To address this issue, LIT uses the teacher model’s IRs as inputs to the student model (described below).
Revised: To address this issue, LIT uses the teacher model’s IRs as inputs to the student model (described below).

Original: We describe the overall LIT procedure, describe the KD loss, describe how IRs are used in LIT, and discuss hyperparameter optimization.
Revised: We describe the overall LIT procedure, describe the KD loss, describe how IRs are used in LIT, and discuss hyperparameter optimization.

Original: LIT.
Revised: LIT.

Original: In LIT, we combine the KD and IR loss.
Revised: In LIT, we combine the KD and IR loss.

Original: We show that combining the losses results in smaller models for a fixed accuracy in Section 4.
Revised: We show that combining the losses results in smaller models for a fixed accuracy in Section 4.

Original: Specifically, for teacher T and student S the full LIT loss is:
Revised: Specifically, for teacher T and student S the full LIT loss is:

Original: [Equation]
Revised: [Equation]

Original: with α,β ∈ [0 , 1] ( α is described below, β is an interpolation parameter).
Revised: with α,β ∈ [0 , 1] ( α is described below, β is an interpolation parameter).

Original: In some cases, we use β =0 , i.e., we only use the IR loss (e.g., for GANs).
Revised: In some cases, we use β =0 , i.e., we only use the IR loss (e.g., for GANs).

Original: As the IRs have matching dimensions, LIT also allows parts of the teacher model to be copied directly into the student model.
Revised: As the IRs have matching dimensions, LIT also allows parts of the teacher model to be copied directly into the student model.

Original: For example, for ResNets, we copy the teacher’s first convolution (before the skip connections) and fully connected layer to the student model.
Revised: For example, for ResNets, we copy the teacher’s first convolution (before the skip connections) and fully connected layer to the student model.

Original: LIT can also be used to compress specific parts of a model, as we do with StarGAN’s generator (Choi et al., 2017).
Revised: LIT can also be used to compress specific parts of a model, as we do with StarGAN’s generator (Choi et al., 2017).

Original: Finally, after we train the student model with combined loss, we fine-tune the student model with the original loss (KD loss for classification, generator loss for GANs).
Revised: Finally, after we train the student model with combined loss, we fine-tune the student model with the original loss (KD loss for classification, generator loss for GANs).

Original: Knowledge distillation loss.
Revised: Knowledge distillation loss.

Original: In KD, a (typically larger) teacher model or ensemble is used to train a (typically smaller) student model.
Revised: In KD, a (typically larger) teacher model or ensemble is used to train a (typically smaller) student model.

Original: Specifically, the KL-divergence between the probabilities of the student and teacher model is minimized, in addition to the standard cross-entropy loss.
Revised: Specifically, the KL-divergence between the probabilities of the student and teacher model is minimized, in addition to the standard cross-entropy loss.

Original: Formally, denote (for the teacher model) q τi = exp
Revised: Formally, denote (for the teacher model) q τi = exp

Original: ( z i /τ )
Revised: ( z i /τ )

Original: (cid:80) j z j /τ where z i are the inputs to the softmax and τ is a hyperparmeter that “softens” the distribution.
Revised: (cid:80) j z j /τ where z i are the inputs to the softmax and τ is a hyperparmeter that “softens” the distribution.

Original: Denote p τi to be the corresponding quantity for the student model.
Revised: Denote p τi to be the corresponding quantity for the student model.

Original: Then, the full KD loss is:
Revised: Then, the full KD loss is:

Original: [Equation]
Revised: [Equation]

Original: for y to be the true labels, H to be the cross-entropy loss, and α to be the interpolation parameter.
Revised: for y to be the true labels, H to be the cross-entropy loss, and α to be the interpolation parameter.

Original: Hinton et al.
Revised: Hinton et al.

Original: (2014) sets α =0 . 5 , but we show that the choice of α can affect performance (Section 4.4).
Revised: (2014) sets α =0 . 5 , but we show that the choice of α can affect performance (Section 4.4).

Original: Training via intermediate representations.
Revised: Training via intermediate representations.

Original: In LIT, we logically divide the student and teacher networks into k sub-networks such that the input and output dimensions match for the corresponding sub-networks (an example is shown in Figure 1).
Revised: In LIT, we logically divide the student and teacher networks into k sub-networks such that the input and output dimensions match for the corresponding sub-networks (an example is shown in Figure 1).

Original: [Table]
Revised: [Table]

Original: Denote the full teacher network and student network to be T and S respectively.
Revised: Denote the full teacher network and student network to be T and S respectively.

Original: Denote the teacher sub-networks to be T i and the student sub-networks to be S i such that T i ,S i : R n i − i
Revised: Denote the teacher sub-networks to be T i and the student sub-networks to be S i such that T i ,S i : R n i − i

Original: → R n i and that the composition of the sub-networks is the full network, namely that T k ( T k − 1 ( ··· T 1 ( x )))
Revised: → R n i and that the composition of the sub-networks is the full network, namely that T k ( T k − 1 ( ··· T 1 ( x )))

Original: =
Revised: =

Original: T ( x ) .
Revised: T ( x ) .

Original: We will omit the argument when convenient.
Revised: We will omit the argument when convenient.

Original: Denote the loss on the IR loss l (e.g., L2 loss).
Revised: Denote the loss on the IR loss l (e.g., L2 loss).

Original: The full intermediate loss (given the set of splits) is:
Revised: The full intermediate loss (given the set of splits) is:

Original: [Equation]
Revised: [Equation]

Original: Concretely, consider a ResNet-110 as the teacher and a ResNet-56 as the student, each with three “stages”, i.e., layers in the network with downsampling, and an L2 intermediate loss.
Revised: Concretely, consider a ResNet-110 as the teacher and a ResNet-56 as the student, each with three “stages”, i.e., layers in the network with downsampling, and an L2 intermediate loss.

Original: Here, the first teacher ResNet “stage” is T 1 , etc.
Revised: Here, the first teacher ResNet “stage” is T 1 , etc.

Original: and the L2 deviation from the feature maps, across all the downsampling feature maps, is the full intermediate loss.
Revised: and the L2 deviation from the feature maps, across all the downsampling feature maps, is the full intermediate loss.

Original: A schematic is shown in Figure 1.
Revised: A schematic is shown in Figure 1.

Original: This procedure has two key decisions: 1) where to logically split the teacher and student models, and 2) the choice of IR loss.
Revised: This procedure has two key decisions: 1) where to logically split the teacher and student models, and 2) the choice of IR loss.

Original: We discuss these settings in the hyperparameter optimization below.
Revised: We discuss these settings in the hyperparameter optimization below.

Original: Hyperparameter optimization.
Revised: Hyperparameter optimization.

Original: LIT inherits two hyperparameters from KD and introduces one more: τ (the temperature in KD), α (the interpolation parameter in KD), and β (the interpolation parameter in LIT), along with an intermediate representation loss and split points.
Revised: LIT inherits two hyperparameters from KD and introduces one more: τ (the temperature in KD), α (the interpolation parameter in KD), and β (the interpolation parameter in LIT), along with an intermediate representation loss and split points.

Original: In this work, we only consider adding the
Revised: In this work, we only consider adding the

Original: IR loss between natural split points, e.g., when a downsampling occurs in a convolutional network.
Revised: IR loss between natural split points, e.g., when a downsampling occurs in a convolutional network.

Original: We have additionally found that L2 loss works well in practice, so we use the L2 loss for all experiments unless otherwise noted (Section 4.4).
Revised: We have additionally found that L2 loss works well in practice, so we use the L2 loss for all experiments unless otherwise noted (Section 4.4).

Original: We have found that iteratively setting τ , then α , then β to work well in practice.
Revised: We have found that iteratively setting τ , then α , then β to work well in practice.

Original: We have found that the same hyperparameters work well for a given student and teacher structure (e.g., ResNet teacher and ResNet student).
Revised: We have found that the same hyperparameters work well for a given student and teacher structure (e.g., ResNet teacher and ResNet student).

Original: Thus, we use the same set of hyperparameters for a given student and teacher structure (e.g., we use the same hyperparameters for a teacher/student of ResNet-110/ResNet-20 and ResNet-110/ResNet-32).
Revised: Thus, we use the same set of hyperparameters for a given student and teacher structure (e.g., we use the same hyperparameters for a teacher/student of ResNet-110/ResNet-20 and ResNet-110/ResNet-32).

Original: To set the hyperparameters for a given structure, we first set τ using a small student model, then α for the fixed τ , then β for the fixed α and τ (all on the validation set).
Revised: To set the hyperparameters for a given structure, we first set τ using a small student model, then α for the fixed τ , then β for the fixed α and τ (all on the validation set).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Experimental setup.
Revised: Experimental setup.

Original: We evaluate LIT’s efficacy at compressing models on a range of tasks and models, including image classification, sentiment analysis, and image-to-image translation (GAN).
Revised: We evaluate LIT’s efficacy at compressing models on a range of tasks and models, including image classification, sentiment analysis, and image-to-image translation (GAN).

Original: Throughout, we use student and teacher networks with the same broad architecture (e.g., ResNet to ResNet).
Revised: Throughout, we use student and teacher networks with the same broad architecture (e.g., ResNet to ResNet).

Original: We consider
Revised: We consider

Original: ResNet (He et al., 2016), ResNeXt (Xie et al., 2017), VDCNN (Conneau et al., 2016), and StarGAN (Choi et al., 2017).
Revised: ResNet (He et al., 2016), ResNeXt (Xie et al., 2017), VDCNN (Conneau et al., 2016), and StarGAN (Choi et al., 2017).

Original: We use standard architecture depths, widths, and learning rate schedules (described in the
Revised: We use standard architecture depths, widths, and learning rate schedules (described in the

Original: Appendix), and perform hyperparameter selection for the KD and LIT interpolation parameters, while also performing a sensitivity analysis of these hyperparameters in the sequel.
Revised: Appendix), and perform hyperparameter selection for the KD and LIT interpolation parameters, while also performing a sensitivity analysis of these hyperparameters in the sequel.

Original: Comparison to pruning.
Revised: Comparison to pruning.

Original: In this work we focus on modern networks, which pruning does not perform well on.
Revised: In this work we focus on modern networks, which pruning does not perform well on.

Original: For example, Han et al.
Revised: For example, Han et al.

Original: (2015a)
Revised: (2015a)

Original: compresses VGG, where the majority of the weights are in the fully connected (FC) layer, achieving a 10 × compression rate for the FC layer, but only a 1.14 × compression rate for the convolutional layers.
Revised: compresses VGG, where the majority of the weights are in the fully connected (FC) layer, achieving a 10 × compression rate for the FC layer, but only a 1.14 × compression rate for the convolutional layers.

Original: In contract, ResNet-110 can only be pruned by ~1.6 × (Li et al., 2016), compared to LIT’s 4 × compression for ResNet-110.
Revised: In contract, ResNet-110 can only be pruned by ~1.6 × (Li et al., 2016), compared to LIT’s 4 × compression for ResNet-110.

Original: 4. LIT SIGNIFICANTLY COMPRESSES MODELS
Revised: 4. LIT SIGNIFICANTLY COMPRESSES MODELS

Original: LIT is effective at compressing a range of datasets and models.
Revised: LIT is effective at compressing a range of datasets and models.

Original: We ran LIT on a variety of models and datasets for image classification and sentiment analysis (Table 1).
Revised: We ran LIT on a variety of models and datasets for image classification and sentiment analysis (Table 1).

Original: We additionally performed KD and hint training on these datasets and models.
Revised: We additionally performed KD and hint training on these datasets and models.

Original: We selected the hyperparameters sequentially (Section 3).
Revised: We selected the hyperparameters sequentially (Section 3).

Original: [Figure]
Revised: [Figure]

Original: layers 91.
Revised: layers 91.

Original: [Figure]
Revised: [Figure]

Original: Number of layers 93.
Revised: Number of layers 93.

Original: A

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: Number
Revised: Number

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: cc
Revised: cc

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Number
Revised: Number

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Figure 2 shows the results for ResNet and ResNeXt for CIFAR10 and CIFAR100, and Figure 3 shows the results for VDCNN on Amazon Reviews (full, polarity).
Revised: Figure 2 shows the results for ResNet and ResNeXt for CIFAR10 and CIFAR100, and Figure 3 shows the results for VDCNN on Amazon Reviews (full, polarity).

Original: LIT can compress models by up to 5.5 × (CIFAR10, ResNeXt 110 to 20) on image classification and up to 3.2 × on sentiment analysis (Amazon
Revised: LIT can compress models by up to 5.5 × (CIFAR10, ResNeXt 110 to 20) on image classification and up to 3.2 × on sentiment analysis (Amazon

Original: Reviews, VDCNN 29 to 9), with no loss in accuracy.
Revised: Reviews, VDCNN 29 to 9), with no loss in accuracy.

Original: LIT outperforms KD and hint training on all settings.
Revised: LIT outperforms KD and hint training on all settings.

Original: Additionally, LIT outperforms the recently proposed Born Again procedure in which the same architecture is used as both the student and teacher model (Furlanello et al., 2018) (i.e., only for improved accuracy, not for compression).
Revised: Additionally, LIT outperforms the recently proposed Born Again procedure in which the same architecture is used as both the student and teacher model (Furlanello et al., 2018) (i.e., only for improved accuracy, not for compression).

Original: We have found that, in some cases, training sequences of models using LIT results in higher performance.
Revised: We have found that, in some cases, training sequences of models using LIT results in higher performance.

Original: Thus, for VDCNN, we additionally compressed using LIT a VDCNN-29 to a
Revised: Thus, for VDCNN, we additionally compressed using LIT a VDCNN-29 to a

Original: VDCNN-17, and using this VDCNN-17, we trained a VDCNN-9 and VDCNN-17.
Revised: VDCNN-17, and using this VDCNN-17, we trained a VDCNN-9 and VDCNN-17.

Original: We also found that in some cases, KD degrades the accuracy of student models when the teacher model is the same architecture (ResNeXt-110 on CIFAR100, VDCNN-29 on Amazon Reviews polarity).
Revised: We also found that in some cases, KD degrades the accuracy of student models when the teacher model is the same architecture (ResNeXt-110 on CIFAR100, VDCNN-29 on Amazon Reviews polarity).

Original: This corroborates prior observations in Mishra & Marr (2017).
Revised: This corroborates prior observations in Mishra & Marr (2017).

Original: LIT can reduce group cardinality.
Revised: LIT can reduce group cardinality.

Original: While LIT requires the size of at least one IR to be the same width between the teacher and student model, several classes of models have an internal width or group cardinality .
Revised: While LIT requires the size of at least one IR to be the same width between the teacher and student model, several classes of models have an internal width or group cardinality .

Original: For example, ResNeXt (Xie et al., 2017) has a “grouped convolution”, which is equivalent to several convolutions with the same input (see Figure 3 in Xie et al.
Revised: For example, ResNeXt (Xie et al., 2017) has a “grouped convolution”, which is equivalent to several convolutions with the same input (see Figure 3 in Xie et al.

Original: The width of the network is not affected by the group size, so LIT is oblivious to the group size.
Revised: The width of the network is not affected by the group size, so LIT is oblivious to the group size.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We show that LIT can reduce the group cardinality for ResNeXt.
Revised: We show that LIT can reduce the group cardinality for ResNeXt.

Original: We train student ResNeXts with cardinality 16 (instead of the default 32) from a ResNeXt-110 (cardinality 32) on CIFAR10.
Revised: We train student ResNeXts with cardinality 16 (instead of the default 32) from a ResNeXt-110 (cardinality 32) on CIFAR10.

Original: Figureillustrates the results.
Revised: Figureillustrates the results.

Original: As before, LIT outperforms KD and hint training in this setting.
Revised: As before, LIT outperforms KD and hint training in this setting.

Original: LIT can compress GANs.
Revised: LIT can compress GANs.

Original: We compressed StarGAN’s generator (Choi et al., 2017) using the LIT procedure with β =

Revised: We compressed StarGAN’s generator (Choi et al., 2017) using the LIT procedure with β =

Original: (i.e., only using the intermediate representation loss) and with an analogous procedure to KD with the L2 loss as a baseline.
Revised: (i.e., only using the intermediate representation loss) and with an analogous procedure to KD with the L2 loss as a baseline.

Original: The original StarGAN has 18 total convolutional layers (including transposed convolutional layers), with 12 of the layers in the residual blocks (for a total of six residual blocks).
Revised: The original StarGAN has 18 total convolutional layers (including transposed convolutional layers), with 12 of the layers in the residual blocks (for a total of six residual blocks).

Original: We compressed the six residual blocks to two residual blocks (i.e., 12 to four layers) while keeping the rest of the layers fixed.
Revised: We compressed the six residual blocks to two residual blocks (i.e., 12 to four layers) while keeping the rest of the layers fixed.

Original: The remaining layers for the teacher model were copied to the student model and fine-tuned.
Revised: The remaining layers for the teacher model were copied to the student model and fine-tuned.

Original: The discriminator remained fixed.
Revised: The discriminator remained fixed.

Original: As shown in Table 2, LIT outperforms all baselines in inception Salimans et al.
Revised: As shown in Table 2, LIT outperforms all baselines in inception Salimans et al.

Original: and FID Heusel et al.
Revised: and FID Heusel et al.

Original: (2017) score.
Revised: (2017) scores.

Original: Additionally, as shown in Figure 5, the student model appears to perceptually outperform both the teacher model and equivalent model trained from scratch, suggesting LIT can both compress
Revised: Additionally, as shown in Figure 5, the student model appears to perceptually outperform both the teacher model and equivalent model trained from scratch, suggesting LIT can both compress

Original: GANs and serve as a form of regularization.
Revised: GANs and serve as a form of regularization.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 4. I MPACT OF T RAINING T ECHNIQUES
Revised: 4. I MPACT OF T RAINING T ECHNIQUES

Original: LIT uses block-wise training with the teacher IRs as input to the student model.
Revised: LIT uses block-wise training with the teacher IRs as input to the student model.

Original: To show the effectiveness of block-wise training, we tried other variations: 1) matching a single IR, with no input from the teacher (i.e., standard hint training/FitNets), 2) a single IR with teacher input, 3) multiple IRs with no teacher input.
Revised: To show the effectiveness of block-wise training, we tried other variations: 1) matching a single IR, with no input from the teacher (i.e., standard hint training/FitNets), 2) a single IR with teacher input, 3) multiple IRs with no teacher input.

Original: We performed these variations on a teacher model of ResNet-110 and a student model of ResNeton CIFAR10 and similarly for ResNeXt.
Revised: We performed these variations on a teacher model of ResNet-110 and a student model of ResNeton CIFAR10 and similarly for ResNeXt.

Original: We used the second block for the single IR experiments.
Revised: We used the second block for the single IR experiments.

Original: As shown in Table 3, none of the three variants are as effective as LIT or KD.
Revised: As shown in Table 3, none of the three variants are as effective as LIT or KD.

Original: Thus, we see that LIT’s block-wise training is critical for high accuracy compression.
Revised: Thus, we see that LIT’s block-wise training is critical for high accuracy compression.

Original: 4. LIT IS COMPLEMENTARY TO PRUNING
Revised: 4. LIT IS COMPLEMENTARY TO PRUNING

Original: Pruning is a key technique in deep compression in which parts of a network are set to zero, which reduces the number of weights and, on specialized hardware, reduces the computational footprint of networks.
Revised: Pruning is a key technique in deep compression in which parts of a network are set to zero, which reduces the number of weights and, on specialized hardware, reduces the computational footprint of networks.

Original: To see if LIT models are amenable to pruning, we pruned ResNets trained via LIT.
Revised: To see if LIT models are amenable to pruning, we pruned ResNets trained via LIT.

Original: We additionally pruned ResNets trained from scratch.
Revised: We additionally pruned ResNets trained from scratch.

Original: All experiments were done on CIFAR10 using a standard pruning procedure Han et al.
Revised: All experiments were done on CIFAR10 using a standard pruning procedure Han et al.

Original: (2015b).
Revised: (2015b).

Original: As shown in Figure 6, LIT models outperform standard pruning for accuracy at a given model size.
Revised: As shown in Figure 6, LIT models outperform standard pruning for accuracy at a given model size.

Original: Additionally, LIT models can be pruned, although less than their trained-from-scratch counterparts.
Revised: Additionally, LIT models can be pruned, although less than their trained-from-scratch counterparts.

Original: However, LIT models are more accurate and are thus likely learning more meaningful representations.
Revised: However, LIT models are more accurate and are thus likely learning more meaningful representations.

Original: Thus, we expect LIT models to be more difficult to prune, as each weight is more important.
Revised: Thus, we expect LIT models to be more difficult to prune, as each weight is more important.

Original: 4. S ENSITIVITY A NALYSIS OF H YPERPARAMETERS
Revised: 4. S ENSITIVITY A NALYSIS OF H YPERPARAMETERS

Original: Intermediate loss penalty.
Revised: Intermediate loss penalty.

Original: To see the affect of the intermediate loss penalty, we performed LIT from a teacher model of ResNet-110 to a student of ResNet-20 with the L1, L2, and smoothed L1 loss (all on CIFAR10).
Revised: To see the affect of the intermediate loss penalty, we performed LIT from a teacher model of ResNet-110 to a student of ResNet-20 with the L1, L2, and smoothed L1 loss (all on CIFAR10).

Original: The results are shown in Table 4.
Revised: The results are shown in Table 4.

Original: As shown, L2 and L1 do not significantly differ, but smoothed L1 degrades accuracy.
Revised: As shown, L2 and L1 do not significantly differ ( p =0 . 78 ), but smoothed L1 degrades accuracy ( p =0 . 02 ).

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: α and β .
Revised: α and β .

Original: Recall that α is the weighting parameter in KD and β is the relative weight of KD vs the intermediate representation loss (Section 3).
Revised: Recall that α is the weighting parameter in KD and β is the relative weight of KD vs the intermediate representation loss (Section 3).

Original: To see the effect of of α (which is a KD hyperparameter), we varied α between 0 and 1 for ResNet and
Revised: To see the effect of of α (which is a KD hyperparameter), we varied α between 0 and 1 for ResNet and

Original: ResNeXt on CIFAR10 and CIFAR100.
Revised: ResNeXt on CIFAR10 and CIFAR100.

Original: As shown in Figure 7, α can significantly affect accuracy.
Revised: As shown in Figure 7, α can significantly affect accuracy.

Original: Thus, we searched for α as opposed to using a static policy of 0.5 as in Hinton et al.
Revised: Thus, we searched for α as opposed to using a static policy of 0.5 as in Hinton et al.

Original: We additionally varied β between 0 and 1 for ResNet and ResNeXt on CIFAR10.
Revised: We additionally varied β between 0 and 1 for ResNet and ResNeXt on CIFAR10.

Original: As shown in Figure 8, the optimal β varies between architectures but appears to be consistent within the same meta-architecture.
Revised: As shown in Figure 8, the optimal β varies between architectures but appears to be consistent within the same meta-architecture.

Original: LIT works with mixed precision.
Revised: LIT works with mixed precision.

Original: To confirm mixed precision training (Micikevicius et al., 2017) works with LIT, we ran LIT on ResNet and ResNeXt (the teacher had 110 layers and the student had 20 layers) on CIFAR10 with both fp32 and mixed precision training.
Revised: To confirm mixed precision training (Micikevicius et al., 2017) works with LIT, we ran LIT on ResNet and ResNeXt (the teacher had 110 layers and the student had 20 layers) on CIFAR10 with both fp32 and mixed precision training.

Original: The results are shown in Table 5.
Revised: The results are shown in Table 5.

Original: As shown, mixed precision  resultsin a limited difference (0.06%) in accuracies forboth ResNet and ResNeXt.
Revised: As shown, mixed precision does not significantly change the results for ResNet or ResNeXt ( p =0 . 5 , 1 . 0 respectively).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce LIT, a novel model compression technique that trains a student model from a teacher model’s intermediate representations.
Revised: We introduce LIT, a novel model compression technique that trains a student model from a teacher model’s intermediate representations.

Original: LIT requires at least one intermediate layer of the student and teacher to match in width, which allows parts of the teacher model to be copied to the student model.
Revised: LIT requires at least one intermediate layer of the student and teacher to match in width, which allows parts of the teacher model to be copied to the student model.

Original: By combining several such intermediate layers, LIT students learn a high quality representation of the teacher state without the associated depth.
Revised: By combining several such intermediate layers, LIT students learn a high quality representation of the teacher state without the associated depth.

Original: To overcome the lack of useful intermediate representations within the student model at the beginning of training, LIT uses the teacher’s intermediate representations as input to the student model during training.
Revised: To overcome the lack of useful intermediate representations within the student model at the beginning of training, LIT uses the teacher’s intermediate representations as input to the student model during training.

Original: We show that LIT can compress models up to 5.5 × with no loss in accuracy on standard classification benchmark tasks and image-to-image translation (i.e., GAN generators), outperforming standard KD and hint training.
Revised: We show that LIT can compress models up to 5.5 × with no loss in accuracy on standard classification benchmark tasks and image-to-image translation (i.e., GAN generators), outperforming standard KD and hint training.

Original: A E XPERIMENTAL D ETAILS
Revised: A E XPERIMENTAL D ETAILS

Original: All network architectures we use are standard architectures for the datasets of choice.
Revised: All network architectures we use are standard architectures for the datasets of choice.

Original: All student/teacher pairs were of the same architecture type (e.g., ResNet to ResNet).
Revised: All student/teacher pairs were of the same architecture type (e.g., ResNet to ResNet).

Original: For each dataset, we detail the architecture and hyperparameters used.
Revised: For each dataset, we detail the architecture and hyperparameters used.

Original: All experiments were performed in PyTorch v0.4.0 with Python 3.5.
Revised: All experiments were performed in PyTorch v0.4.0 with Python 3.5.

Original: Experiments were run on public cloud and custom servers using NVIDIA P100, V100, Titan Xp, and Titan V GPUs.
Revised: Experiments were run on public cloud and custom servers using NVIDIA P100, V100, Titan Xp, and Titan V GPUs.

Original: All weights for architectures were initialized as in the original architecture.
Revised: All weights for architectures were initialized as in the original architecture.

Original: A. CIFAR10/ 
Revised: A. CIFAR10/

Original: We used the same hyperparameters for CIFAR10 and CIFAR100.
Revised: We used the same hyperparameters for CIFAR10 and CIFAR100.

Original: ResNet.
Revised: ResNet.

Original: We use standard ResNets (He et al., 2016) for CIFAR10/100 (Krizhevsky & Hinton, 2009; ? ).
Revised: We use standard ResNets (He et al., 2016) for CIFAR10/100 (Krizhevsky & Hinton, 2009; ? ).

Original: The architectures are parameterized by the number of “computation heavy” layers, i.e., convolutional and fully connected layers, but not batch norm and ReLU layers.
Revised: The architectures are parameterized by the number of “computation heavy” layers, i.e., convolutional and fully connected layers, but not batch norm and ReLU layers.

Original: Thus, a ResNet-110 has 110 convolutional and fully connected layers.
Revised: Thus, a ResNet-110 has 110 convolutional and fully connected layers.

Original: Each ResNet has three blocks, where the convolutional layers in each block have the same number of filters.
Revised: Each ResNet has three blocks, where the convolutional layers in each block have the same number of filters.

Original: The last convolutional layer in each block downsamples by a factor of two.
Revised: The last convolutional layer in each block downsamples by a factor of two.

Original: The ResNets can be parameterized by the number of residual blocks in each block, where each residual block has two convolutional layers.
Revised: The ResNets can be parameterized by the number of residual blocks in each block, where each residual block has two convolutional layers.

Original: This parameterization is the same parameterizing by the number of layers.
Revised: This parameterization is the same parameterizing by the number of layers.

Original: For example, an [18, 18, 18] has 110 layers total, 108 layers in the blocks, along with an additional convolutional layer at the start and a fully connected layer.
Revised: For example, an [18, 18, 18] has 110 layers total, 108 layers in the blocks, along with an additional convolutional layer at the start and a fully connected layer.

Original: We show a table of the number of layers and block parameterizations in Table 6.
Revised: We show a table of the number of layers and block parameterizations in Table 6.

Original: For all experiments, we used a batch size of 32, SGD with a momentum of 0.9, and weight decay of 1e-4.
Revised: For all experiments, we used a batch size of 32, SGD with a momentum of 0.9, and weight decay of 1e-4.

Original: For training from scratch, we trained with a starting learning rate of 0.1 for 200 epochs with milestones at 100 and 150 epochs, decaying the learning rate by a factor of 10.
Revised: For training from scratch, we trained with a starting learning rate of 0.1 for 200 epochs with milestones at 100 and 150 epochs, decaying the learning rate by a factor of 10.

Original: For KD, we trained with a starting learning rate of 0.1 for 250 epochs with milestones at 100 andepochs, decaying the learning rate by a factor of 10.
Revised: For KD, we trained with a starting learning rate of 0.1 for 250 epochs with milestones at 100 andepochs, decaying the learning rate by a factor of 10.

Original: We used τ =6 and α =0 .
Revised: We used τ =6 and α =0 .

Original: For LIT, we trained with a starting learning rate of 0.1 for 175 epochs with milestones at 60, 100, and 125 epochs, decaying the learning rate by a factor of 10.
Revised: For LIT, we trained with a starting learning rate of 0.1 for 175 epochs with milestones at 60, 100, and 125 epochs, decaying the learning rate by a factor of 10.

Original: We then fine-tuned using the KD loss for another 75 epochs with a starting learning rate of 0.01 and milestones at 35 and 55 epochs.
Revised: We then fine-tuned using the KD loss for another 75 epochs with a starting learning rate of 0.01 and milestones at 35 and 55 epochs.

Original: We used β =0 .
Revised: We used β =0 .

Original: ResNeXt.
Revised: ResNeXt.

Original: We use standard ResNeXts (Xie et al., 2017) for CIFAR10/100.
Revised: We use standard ResNeXts (Xie et al., 2017) for CIFAR10/100.

Original: The parameterization in terms of number of layers are the same for ResNet.
Revised: The parameterization in terms of number of layers are the same for ResNet.

Original: ResNeXt has an additional parameter of the group cardinality.
Revised: ResNeXt has an additional parameter of the group cardinality.

Original: We use a group cardinality of 32 for all experiments, except the ones detailed below.
Revised: We use a group cardinality of 32 for all experiments, except the ones detailed below.

Original: For all experiments, we used a batch size of 32, SGD with a momentum of 0.9, and weight decay of 1e-4.
Revised: For all experiments, we used a batch size of 32, SGD with a momentum of 0.9, and weight decay of 1e-4.

Original: For training from scratch, we trained with a starting learning rate of 0.1 for 300 epochs with milestones at 150 and 225 epochs, decaying the learning rate by a factor of 10. For KD, we trained with a starting learning rate of 0.1 for 300 epochs with milestones at 100, 175, and 225 epochs, decaying the learning rate by a factor of 10.
Revised: For training from scratch, we trained with a starting learning rate of 0.1 for 300 epochs with milestones at 150 and 225 epochs, decaying the learning rate by a factor of 10. For KD, we trained with a starting learning rate of 0.1 for 300 epochs with milestones at 100, 175, and 225 epochs, decaying the learning rate by a factor of 10.

Original: We used τ =6 and α =0 .
Revised: We used τ =6 and α =0 .

Original: For LIT, we trained with a starting learning rate of 0.1 for 200 epochs with milestones at 100, 145, and 175 epochs, decaying the learning rate by a factor of 10.
Revised: For LIT, we trained with a starting learning rate of 0.1 for 200 epochs with milestones at 100, 145, and 175 epochs, decaying the learning rate by a factor of 10.

Original: We then fine-tuned using the KD loss for another 125 epochs with a starting learning rate of 0.01 and milestones at 65, 95, and 110 epochs.
Revised: We then fine-tuned using the KD loss for another 125 epochs with a starting learning rate of 0.01 and milestones at 65, 95, and 110 epochs.

Original: We used β =0 .
Revised: We used β =0 .

Original: Reduced cardinality ResNeXt.
Revised: Reduced cardinality ResNeXt.

Original: All hyperparameters were the same as the standard ResNeXt experiments except we used β =0 .
Revised: All hyperparameters were the same as the standard ResNeXt experiments except we used β =0 .

Original: 25 and a student cardinality of 16.
Revised: 25 and a student cardinality of 16.

Original: A. A MAZON R EVIEWS
Revised: A. A MAZON R EVIEWS

Original: We use standard VDCNNs (Conneau et al., 2016) for Amazon Reviews full and polarity (He & McAuley, 2016).
Revised: We use standard VDCNNs (Conneau et al., 2016) for Amazon Reviews full and polarity (He & McAuley, 2016).

Original: VDCNN has an initial convolutional layer and four blocks of convolutional layers (each block has the same number, but vary between types of VDCNNs).
Revised: VDCNN has an initial convolutional layer and four blocks of convolutional layers (each block has the same number, but vary between types of VDCNNs).

Original: Thus, a VDCNN-9 has an initial convolutional layer and two convolutional layers in each subsequent block.
Revised: Thus, a VDCNN-9 has an initial convolutional layer and two convolutional layers in each subsequent block.

Original: We consider VDCNN-9, VDCNN-17, and
Revised: We consider VDCNN-9, VDCNN-17, and

Original: VDCNN-29 as in Conneau et al.
Revised: VDCNN-29 as in Conneau et al.

Original: For all experiments, we used a batch size of 128, SGD with a momentum of 0.9, and weight decay of 1e-4.
Revised: For all experiments, we used a batch size of 128, SGD with a momentum of 0.9, and weight decay of 1e-4.

Original: For training from scratch, we trained with a starting learning rate of 0.01 for 15 epochs, with milestones at 3, 6, 9, 12, and 15 epochs, decaying the learning rate by a factor of 10. For KD, we trained with a starting learning rate of 0.01 for 18 epochs, with milestones at 3, 6, 9, 12, and 15 epochs, decaying the learning rate by a factor of 10.
Revised: For training from scratch, we trained with a starting learning rate of 0.01 for 15 epochs, with milestones at 3, 6, 9, 12, and 15 epochs, decaying the learning rate by a factor of 10. For KD, we trained with a starting learning rate of 0.01 for 18 epochs, with milestones at 3, 6, 9, 12, and 15 epochs, decaying the learning rate by a factor of 10.

Original: We used τ =6 and α =0 . 98 .
Revised: We used τ =6 and α =0 . 98 .

Original: For LIT, we trained with a starting learning rate of 0.01 for 15 epochs, with milestones at 3, 6, 9, and 12 epochs, decaying the learning rate by a factor of 2.
Revised: For LIT, we trained with a starting learning rate of 0.01 for 15 epochs, with milestones at 3, 6, 9, and 12 epochs, decaying the learning rate by a factor of 2.

Original: We then fine-tuned using the KD loss for another 10 epochs with a starting learning rate of 0.000625 and milestones at 4 and 8 epochs.
Revised: We then fine-tuned using the KD loss for another 10 epochs with a starting learning rate of 0.000625 and milestones at 4 and 8 epochs.

Original: We used β =0 . 02 .
Revised: We used β =0 . 02 .

Original: A. S TAR GAN
Revised: A. S TAR GAN

Original: We used the StarGAN as described in Choi et al.
Revised: We used the StarGAN as described in Choi et al.

Original: The original StarGAN has 18 total convolutional layers (including transposed convolutional layers), with 12 of the layers in the residual blocks (each residual block has two convolutional layers).
Revised: The original StarGAN has 18 total convolutional layers (including transposed convolutional layers), with 12 of the layers in the residual blocks (each residual block has two convolutional layers).

Original: We compressed the six residual blocks to two residual blocks.
Revised: We compressed the six residual blocks to two residual blocks.

Original: For all experiments, we used a batch size of 16, SGD with a momentum of 0.9, and weight decay of 1e-4.
Revised: For all experiments, we used a batch size of 16, SGD with a momentum of 0.9, and weight decay of 1e-4.

Original: For training from scratch, we trained with a starting learning rate of 0.0001 for 20 epochs.
Revised: For training from scratch, we trained with a starting learning rate of 0.0001 for 20 epochs.

Original: The learning rate was decayed to 0 over the last 10 epochs.
Revised: The learning rate was decayed to 0 over the last 10 epochs.

Original: As KD does not apply, we did not run KD.

Original: For LIT, we trained with a starting learning rate of 0.0001 for 16 epochs, decaying the learning rate byat epoch 8 (only the IR loss was used).
Revised: For LIT, we trained with a starting learning rate of 0.0001 for 16 epochs, decaying the learning rate byat epoch 8 (only the IR loss was used).

Original: We then fine-tuned with the discriminator with a starting learning rate of 0.00005 for 10 epochs, decaying the learning rate by 10 at epoch 5.
Revised: We then fine-tuned with the discriminator with a starting learning rate of 0.00005 for 10 epochs, decaying the learning rate by 10 at epoch 5.

Revised: A. S TATISTICAL TEST DETAILS

Revised: p-values were computed with a t-test Wasserman (2013) with three samples each.

Revised: All error estimates are the standard deviation with three samples.

Original: B S TAR GAN IMAGES
Revised: B S TAR GAN IMAGES

Original: We show a randomly selected set of images generated from the StarGAN teacher, student, and trained from scratch generators in Figure 9.
Revised: We show a randomly selected set of images generated from the StarGAN teacher, student, and trained from scratch generators in Figure 9.

Original: [Figure]
Revised: [Figure]