Original: B IG -L ITTLE N ET : A N E FFICIENT M ULTI -S CALE F EATURE R EPRESENTATION FOR V ISUAL AND S PEECH R ECOGNITION
Revised: B IG -L ITTLE N ET : A N E FFICIENT M ULTI -S CALE F EATURE R EPRESENTATION FOR V ISUAL AND S PEECH R ECOGNITION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we propose a novel Convolutional Neural Network (CNN) architecture for learning multi-scale feature representations with good tradeoffs between speed and accuracy.
Revised: In this paper, we propose a novel Convolutional Neural Network (CNN) architecture for learning multi-scale feature representations with good tradeoffs between speed and accuracy.

Original: This is achieved by using a multi-branch network, which has different computational complexity at different branches with different resolutions.
Revised: This is achieved by using a multi-branch network, which has different computational complexity at different branches with different resolutions.

Original: Through frequent merging of features from branches at distinct scales, our model obtains multi-scale features while using less computation.
Revised: Through frequent merging of features from branches at distinct scales, our model obtains multi-scale features while using less computation.

Original: The proposed approach demonstrates improvement of model efﬁciency and performance on both object recognition and speech recognition tasks, using popular architectures including
Revised: The proposed approach demonstrates improvement of model efﬁciency and performance on both object recognition and speech recognition tasks, using popular architectures including

Original: ResNet, ResNeXt and SEResNeXt.
Revised: ResNet, ResNeXt and SEResNeXt.

Original: For object recognition, our approach reduces computation by 1 / 3 while improving accuracy signiﬁcantly over 1% point than the baselines, and the computational savings can be higher up to 1 / 2 without compromising the accuracy.
Revised: For object recognition, our approach reduces computation by 1 / 3 while improving accuracy signiﬁcantly over 1% point than the baselines, and the computational savings can be higher up to 1 / 2 without compromising the accuracy.

Original: Our model also surpasses state-of-the-art CNN acceleration approaches by a large margin in terms of accuracy and FLOPs.
Revised: Our model also surpasses state-of-the-art CNN acceleration approaches by a large margin in terms of accuracy and FLOPs.

Original: On the task of speech recognition, our proposed multi-scale CNNs save 30% FLOPs with slightly better word error rates, showing good generalization across domains.
Revised: On the task of speech recognition, our proposed multi-scale CNNs save 30% FLOPs with slightly better word error rates, showing good generalization across domains.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Convolutional Neural Network (CNN) models have achieved substantial performance gains in many computer vision and speech recognition tasks (
Revised: Deep Convolutional Neural Network (CNN) models have achieved substantial performance gains in many computer vision and speech recognition tasks (

Original: He et al., 2016; 2017; Vinyals et al., 2017;
Revised: He et al., 2016; 2017; Vinyals et al., 2017;

Original: Sercu & Goel, 2016).
Revised: Sercu & Goel, 2016).

Original: However, the accuracy obtained by these models usually grows proportionally with their complexity and computational cost.
Revised: However, the accuracy obtained by these models usually grows proportionally with their complexity and computational cost.

Original: This poses an issue for deploying these models in applications that require real-time inferencing and low-memory footprint, such as self-driving vehicles, human-machine interaction on mobile devices, and robotics.
Revised: This poses an issue for deploying these models in applications that require real-time inferencing and low-memory footprint, such as self-driving vehicles, human-machine interaction on mobile devices, and robotics.

Original: Motivated by these applications, many methods have been proposed for model compression and acceleration, including techniques such as pruning (Dong et al., 2017; Li et al., 2017; Han et al., 2015), quantization (Hubara et al., 2016; Li & Liu, 2016), and low-rank factorization (Wen et al., 2017;
Revised: Motivated by these applications, many methods have been proposed for model compression and acceleration, including techniques such as pruning (Dong et al., 2017; Li et al., 2017; Han et al., 2015), quantization (Hubara et al., 2016; Li & Liu, 2016), and low-rank factorization (Wen et al., 2017;

Original: Ioannou et al., 2015; Zhang et al., 2016).
Revised: Ioannou et al., 2015; Zhang et al., 2016).

Original: Most of these methods have been applied to single-scale inputs, without considering multi-resolution processing.
Revised: Most of these methods have been applied to single-scale inputs, without considering multi-resolution processing.

Original: More recently, another line of work applies dynamic routing to allocate different workloads in the networks according to image complexity (Wu et al., 2018;
Revised: More recently, another line of work applies dynamic routing to allocate different workloads in the networks according to image complexity (Wu et al., 2018;

Original: Wang et al., 2018;
Revised: Wang et al., 2018;

Original: Figurnov et al., 2017; Veit & Belongie, 2018).
Revised: Figurnov et al., 2017; Veit & Belongie, 2018).

Original: Multi-scale feature representations have proven successful for many vision and speech recognition tasks compared to single-scale methods (Nah et al., 2017; Chen et al., 2017a; Tóth, 2017; Farabet et al., 2013); however, the computational complexity has not been addressed much in multi-scale networks.
Revised: Multi-scale feature representations have proven successful for many vision and speech recognition tasks compared to single-scale methods (Nah et al., 2017; Chen et al., 2017a; Tóth, 2017; Farabet et al., 2013); however, the computational complexity has not been addressed much in multi-scale networks.

Original: The computational cost of a CNN model has much to do with the input image size.
Revised: The computational cost of a CNN model has much to do with the input image size.

Original: A model, if running at half of the image size, can gain a remarkable computational saving of 75%.
Revised: A model, if running at half of the image size, can gain a remarkable computational saving of 75%.

Original: Based on this fact, we propose an efﬁcient network architecture by combining image information at different scales through a multi-branch network.
Revised: Based on this fact, we propose an efﬁcient network architecture by combining image information at different scales through a multi-branch network.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, our key idea is to use a high-complexity branch ( accurate but costly ) for low-scale feature representation and low-complexity branch ( efﬁcient but less accurate ) for high-scale feature representation.
Revised: 1, our key idea is to use a high-complexity branch ( accurate but costly ) for low-scale feature representation and low-complexity branch ( efﬁcient but less accurate ) for high-scale feature representation.

Original: The two types of features are frequently merged together to complement and enrich each other, leading to a stronger feature representation than either of them individually.
Revised: The two types of features are frequently merged together to complement and enrich each other, leading to a stronger feature representation than either of them individually.

Original: We refer to the deeper branch operating at low image resolution as Big-Branch
Revised: We refer to the deeper branch operating at low image resolution as Big-Branch

Original: [Figure]
Revised: [Figure]

Original: and the shallower one as Little-Branch to reﬂect their differences in computation.
Revised: and the shallower one as Little-Branch to reﬂect their differences in computation.

Original: The new network architecture is thus called Big-Little Net or bL-Net for short in this paper.
Revised: The new network architecture is thus called Big-Little Net or bL-Net for short in this paper.

Original: While being structurally simple, our approach is quite effective.
Revised: While being structurally simple, our approach is quite effective.

Original: We demonstrate later that when bL-Net is integrated into state-of-the-art CNNs such as ResNet, ResNeXt and SEResNeXt, it yields 2 × computational savings over the baselines without losing any accuracy.
Revised: We demonstrate later that when bL-Net is integrated into state-of-the-art CNNs such as ResNet, ResNeXt and SEResNeXt, it yields 2 × computational savings over the baselines without losing any accuracy.

Original: It also outperforms many other recently developed approaches based on more sophisticated architectures at the same FLOP count.
Revised: It also outperforms many other recently developed approaches based on more sophisticated architectures at the same FLOP count.

Original: One work that relates to ours is the Inception model (Szegedy et al., 2016a; 2017), which also leverages parallel pathways in a network building block for efﬁciency.
Revised: One work that relates to ours is the Inception model (Szegedy et al., 2016a; 2017), which also leverages parallel pathways in a network building block for efﬁciency.

Original: However, the efﬁciency of these models relies on substantial use of 1 × 1 and separable ﬁlters (i.e. 1 × 3 and 3 × 1).
Revised: However, the efﬁciency of these models relies on substantial use of 1 × 1 and separable ﬁlters (i.e. 1 × 3 and 3 × 1).

Original: In contrast, bL-Net is general and applicable to many architectures including Inception.
Revised: In contrast, bL-Net is general and applicable to many architectures including Inception.

Original: The main contributions of our paper are summarized as follows:
Revised: The main contributions of our paper are summarized as follows:

Original: • We propose an efﬁcient and effective multi-scale CNN architecture for object and speech recognition.
Revised: • We propose an efﬁcient and effective multi-scale CNN architecture for object and speech recognition.

Original: • We demonstrate that our approach reduces computation by 1 / 3 in models such as ResNet and ResNeXt while improving accuracy over 1% point than the baselines, and the computational savings can be higher up to 1 / 2 without losing any accuracy; these results outperform state-of-the-art networks that focus on CNN acceleration by a large margin at the same FLOPs.
Revised: • We demonstrate that our approach reduces computation by 1 / 3 in models such as ResNet and ResNeXt while improving accuracy over 1% point than the baselines, and the computational savings can be higher up to 1 / 2 without losing any accuracy; these results outperform state-of-the-art networks that focus on CNN acceleration by a large margin at the same FLOPs.

Original: • We validate the proposed method on a speech recognition task, where we also achieve better word error rates while reducing the number of FLOPs by 30%.
Revised: • We validate the proposed method on a speech recognition task, where we also achieve better word error rates while reducing the number of FLOPs by 30%.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Model Compression and Acceleration.
Revised: Model Compression and Acceleration.

Original: Network pruning (Dong et al., 2017; Li et al., 2017) and quantization (Hubara et al., 2016; Li & Liu, 2016) are popular techniques to remove model redundancy and save computational cost.
Revised: Network pruning (Dong et al., 2017; Li et al., 2017) and quantization (Hubara et al., 2016; Li & Liu, 2016) are popular techniques to remove model redundancy and save computational cost.

Original: Another thread of work consists of training a sparse model directly, such as IGCv2 (Xie et al., 2018) and SCConv (Fan et al., 2017).
Revised: Another thread of work consists of training a sparse model directly, such as IGCv2 (Xie et al., 2018) and SCConv (Fan et al., 2017).

Original: Efﬁcient network architectures like
Revised: Efﬁcient network architectures like

Original: MobileNetV2 (Sandler et al., 2018) and ShufﬂeNetV2 (Ma et al., 2018) have also been explored for training compact deep networks.
Revised: MobileNetV2 (Sandler et al., 2018) and ShufﬂeNetV2 (Ma et al., 2018) have also been explored for training compact deep networks.

Original: Other methods include knowledge distillation (Hinton et al., 2015), compression with structured matrices (Cheng et al., 2015; Sindhwani et al., 2015), and hashing (Chen et al., 2015).
Revised: Other methods include knowledge distillation (Hinton et al., 2015), compression with structured matrices (Cheng et al., 2015; Sindhwani et al., 2015), and hashing (Chen et al., 2015).

Original: Dynamic routing (Wu et al., 2018; Wang et al., 2018;
Revised: Dynamic routing (Wu et al., 2018; Wang et al., 2018;

Original: Figurnov et al., 2017; Veit &
Revised: Figurnov et al., 2017; Veit &

Original: Belongie, 2018) has also been explored in residual networks to improve efﬁciency.
Revised: Belongie, 2018) has also been explored in residual networks to improve efﬁciency.

Original: These methods operate with single-resolution inputs, while our approach processes multi-resolution data.
Revised: These methods operate with single-resolution inputs, while our approach processes multi-resolution data.

Original: It could be used in tandem with these methods to further improve efﬁciency.
Revised: It could be used in tandem with these methods to further improve efﬁciency.

Original: Multi-Resolution Feature Representations.
Revised: Multi-Resolution Feature Representations.

Original: The notion of multi-scale feature representation can be dated back to image pyramids (Adelson et al., 1984) and scale-space theory (Lindeberg & ter Haar Romeny, 1994).
Revised: The notion of multi-scale feature representation can be dated back to image pyramids (Adelson et al., 1984) and scale-space theory (Lindeberg & ter Haar Romeny, 1994).

Original: More recently, several methods have proposed multi-scale CNN-based architectures for object detection and recognition.
Revised: More recently, several methods have proposed multi-scale CNN-based architectures for object detection and recognition.

Original: MSCNN (Cai et al., 2016), DAG-CNN (Yang
Revised: MSCNN (Cai et al., 2016), DAG-CNN (Yang

Original: & Ramanan, 2015) and FPN (Lin et al., 2017) use features at different layers to form multi-scale features.
Revised: & Ramanan, 2015) and FPN (Lin et al., 2017) use features at different layers to form multi-scale features.

Original: Hourglass networks (Newell et al., 2016) use a hierarchical multi-resolution model for human pose estimation.
Revised: Hourglass networks (Newell et al., 2016) use a hierarchical multi-resolution model for human pose estimation.

Original: However, this approach induces a heavy workload as the complexity of each sub-network in their model is equal.
Revised: However, this approach induces a heavy workload as the complexity of each sub-network in their model is equal.

Original: Nah et al.
Revised: Nah et al.

Original: (Nah et al., 2017) and Eigen et al.
Revised: (Nah et al., 2017) and Eigen et al.

Original: (Eigen & Fergus, 2015) combine the features from multiple networks working on different resolutions to generate multi-scale features.
Revised: (Eigen & Fergus, 2015) combine the features from multiple networks working on different resolutions to generate multi-scale features.

Original: The overall computational cost grows along with the number of scales, leading to inefﬁcient models.
Revised: The overall computational cost grows along with the number of scales, leading to inefﬁcient models.

Original: In contrast to existing methods, our approach uses different network capacities for different scales, and yields more powerful multi-scale features by fusing them at multiple levels of the network model.
Revised: In contrast to existing methods, our approach uses different network capacities for different scales, and yields more powerful multi-scale features by fusing them at multiple levels of the network model.

Original: Closely related to our work, approaches such as (Huang et al., 2018; Saxena & Verbeek, 2016), apply multiple branches at multi-scales while aiming at reducing computational complexity.
Revised: Closely related to our work, approaches such as (Huang et al., 2018; Saxena & Verbeek, 2016), apply multiple branches at multi-scales while aiming at reducing computational complexity.

Original: In contrast to our work, their computational gain mostly comes from early exit depending on the input image.
Revised: In contrast to our work, their computational gain mostly comes from early exit depending on the input image.

Original: Our approach speeds up a network constantly regardless of the input image.
Revised: Our approach speeds up a network constantly regardless of the input image.

Original:  O UR A PPROACH
Revised:  O UR A PPROACH

Original: We develop a simple, easy-to-implement, yet very efﬁcient and effective network architecture.
Revised: We develop a simple, easy-to-implement, yet very efﬁcient and effective network architecture.

Original: It learns multi-scale feature representations by fusing multiple branches with different image scales and computational complexity.
Revised: It learns multi-scale feature representations by fusing multiple branches with different image scales and computational complexity.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, we design a multi-scale feature module with the following principles: (I) each branch corresponds to a single unique image scale (or resolution); (II) the computational cost of a branch is inversely proportional to the scale.
Revised: 1, we design a multi-scale feature module with the following principles: (I) each branch corresponds to a single unique image scale (or resolution); (II) the computational cost of a branch is inversely proportional to the scale.

Original: Note that the principle (II) implies that we use high-complexity networks at lower resolutions and low-complexity networks at higher resolutions for the sake of efﬁciency.
Revised: Note that the principle (II) implies that we use high-complexity networks at lower resolutions and low-complexity networks at higher resolutions for the sake of efﬁciency.

Original: 3. B IG -L ITTLE N ET
Revised: 3. B IG -L ITTLE N ET

Original: Big-Little Net is a sequence of Big-Little Modules, each one taking input x i and producing output x
Revised: Big-Little Net is a sequence of Big-Little Modules, each one taking input x i and producing output x

Original: i +1 .
Revised: i +1 .

Original: Within a Big-Little Module, assume we have K branches working on K scales [1 , 1 / 2 , 1 / 4 , . .
Revised: Within a Big-Little Module, assume we have K branches working on K scales [1 , 1 / 2 , 1 / 4 , . .

Original: , 1 / 2 K − 1 ] .
Revised: , 1 / 2 K − 1 ] .

Original: We denote a feature map x
Revised: We denote a feature map x

Original: i at scale 1 / 2 k as x ki , indicating the spatial size of x i downsampled by 2 k with respect to the original input dimension.
Revised: i at scale 1 / 2 k as x ki , indicating the spatial size of x i downsampled by 2 k with respect to the original input dimension.

Original: We use a weighted sum to combine all branches into a feature representation at scale 1 / 2 k .
Revised: We use a weighted sum to combine all branches into a feature representation at scale 1 / 2 k .

Original: Mathematically, the module’s output x i +1 can be expressed by
Revised: Mathematically, the module’s output x i +1 can be expressed by

Original: [Equation]
Revised: [Equation]

Original: where f k ( · ) denotes a sequence of convolutional layers.
Revised: where f k ( · ) denotes a sequence of convolutional layers.

Original: Typically for higher k , f k will have more convolutional layers having more feature maps.
Revised: Typically for higher k , f k will have more convolutional layers having more feature maps.

Original: S k ( · ) is the operation that matches the output size of the branches, either: (1) increasing the number feature maps with a 1 × 1 convolution, (2) upsampling to match the output size of the k = 0 branch, or both.
Revised: S k ( · ) is the operation that matches the output size of the branches, either: (1) increasing the number feature maps with a 1 × 1 convolution, (2) upsampling to match the output size of the k = 0 branch, or both.

Original: c k indicates the weighting coefﬁcients of each scale in the merge while F ( · ) is an optional ﬁnal fusion layer like a convolutional layer.
Revised: c k indicates the weighting coefﬁcients of each scale in the merge while F ( · ) is an optional ﬁnal fusion layer like a convolutional layer.

Original: Note that branches are merged at the end of every Big-Little Module and merging the branch outputs happens at the highest resolution and highest number of feature maps between the branches.
Revised: Note that branches are merged at the end of every Big-Little Module and merging the branch outputs happens at the highest resolution and highest number of feature maps between the branches.

Original: Maintaining these large intermediate states avoids information loss.
Revised: Maintaining these large intermediate states avoids information loss.

Original: A crucial aspect of this design is that through consecutive merging and downsampling, the expensive branches operating at low resolution still have access to the high resolution information, processed by the cheaper branches in the previous module.
Revised: A crucial aspect of this design is that through consecutive merging and downsampling, the expensive branches operating at low resolution still have access to the high resolution information, processed by the cheaper branches in the previous module.

Original: While our design is suitable for any number of networks, in this work we primarily focus on the case of two networks, i.e., K = 2 .
Revised: While our design is suitable for any number of networks, in this work we primarily focus on the case of two networks, i.e., K = 2 .

Original: We also experimented with K > 2 in object and speech recognition; however, the K = 2 case provided the best balance between accuracy and computation (See Appendix A.4 and Section 4.2 for details).
Revised: We also experimented with K > 2 in object and speech recognition; however, the K = 2 case provided the best balance between accuracy and computation (See Appendix A.4 and Section 4.2 for details).

Original: Following the principles above, we propose a multinetwork architecture that integrates two branches for multi-scale feature representation.
Revised: Following the principles above, we propose a multinetwork architecture that integrates two branches for multi-scale feature representation.

Original: Figure 1 (b) shows an example Big-Little Net architecture.
Revised: Figure 1 (b) shows an example Big-Little Net architecture.

Original: The module includes two branches, each of which represents a separate network block from a deep model ( accurate but costly ) and a less deep counterpart ( efﬁcient but less accurate ).
Revised: The module includes two branches, each of which represents a separate network block from a deep model ( accurate but costly ) and a less deep counterpart ( efﬁcient but less accurate ).

Original: The two branches are fused at the end through linear combination with unit weights (i.e., c 

Revised: The two branches are fused at the end through linear combination with unit weights (i.e., c

Original: = c 1 = 1 . 0 ).
Revised: = c 1 = 1 . 0 ).

Original: Before fusion, the low resolution feature maps are upsampled using bilinear interpolation to spatially match the higher-resolution counterparts (= S 1 ( · ) ).
Revised: Before fusion, the low resolution feature maps are upsampled using bilinear interpolation to spatially match the higher-resolution counterparts (= S 1 ( · ) ).

Original: Similarly, the high resolution feature map has an additional 1 × 1 convolutional layer to increase the number of output channels (= S 0 ( · ) ).
Revised: Similarly, the high resolution feature map has an additional 1 × 1 convolutional layer to increase the number of output channels (= S 0 ( · ) ).

Original: Furthermore, since our design is based on ResNet, we add a residual block to further fuse the combined features (i.e., F ( · ) is a residual block).
Revised: Furthermore, since our design is based on ResNet, we add a residual block to further fuse the combined features (i.e., F ( · ) is a residual block).

Original: For convenience, we refer to these two branches as Big-Branch (more layers and channels at low resolution) and Little-Branch (fewer layers and channels at high resolution), respectively.
Revised: For convenience, we refer to these two branches as Big-Branch (more layers and channels at low resolution) and Little-Branch (fewer layers and channels at high resolution), respectively.

Original: We also denote the module as Big-Little Module and the entire architecture as Big-Little
Revised: We also denote the module as Big-Little Module and the entire architecture as Big-Little

Original: Net or bL-Net .
Revised: Net or bL-Net .

Original: To control the complexity of bL-Net , we introduce two parameters to specify the complexity of the
Revised: To control the complexity of bL-Net , we introduce two parameters to specify the complexity of the

Original: Little-Branch with respect to the Big-Branch.
Revised: Little-Branch with respect to the Big-Branch.

Original: The Big-Branch typically follows the structure of the original network, but the Little-Branch needs to be heavily slimmed and shortened to reduce computation as it operates on high resolution.
Revised: The Big-Branch typically follows the structure of the original network, but the Little-Branch needs to be heavily slimmed and shortened to reduce computation as it operates on high resolution.

Original: Here we use two parameters α and β to control the width and depth of the Little-Branch, respectively.
Revised: Here we use two parameters α and β to control the width and depth of the Little-Branch, respectively.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1 (b), α speciﬁes the reduction factor of the number of channels in the convolutional layers of Little-Branch with respect to that of the original network while β is the reduction factor of the number of convolutional layers.
Revised: 1 (b), α speciﬁes the reduction factor of the number of channels in the convolutional layers of Little-Branch with respect to that of the original network while β is the reduction factor of the number of convolutional layers.

Original: Larger values of α and β lead to lower complexity in bL-Net .
Revised: Larger values of α and β lead to lower complexity in bL-Net .

Original: As demonstrated later, with an appropriate choice of α and β (See Table 1 and Table 3), the cost of the Little-Branch can be 1 / 6 of the bL-Net and 1 / 12 of the original network while still providing sufﬁcient complementary information to the
Revised: As demonstrated later, with an appropriate choice of α and β (See Table 1 and Table 3), the cost of the Little-Branch can be 1 / 6 of the bL-Net and 1 / 12 of the original network while still providing sufﬁcient complementary information to the

Original: Big-Branch.
Revised: Big-Branch.

Original: 3. N ETWORK M ERGING
Revised: 3. N ETWORK M ERGING

Original: We consider two options for merging the outputs of the branches.
Revised: We consider two options for merging the outputs of the branches.

Original: The ﬁrst option is a linear combination, which joins features from two networks by addition (Newell et al., 2016).
Revised: The ﬁrst option is a linear combination, which joins features from two networks by addition (Newell et al., 2016).

Original: The alternative concatenates the outputs of the two networks along the channel dimension, and if needed, a 1 ×convolution can be subsequently applied to reduce the number of feature maps (Szegedy et al., 2015).
Revised: The alternative concatenates the outputs of the two networks along the channel dimension, and if needed, a 1 ×convolution can be subsequently applied to reduce the number of feature maps (Szegedy et al., 2015).

Original: Both merging approaches have their pros and cons.
Revised: Both merging approaches have their pros and cons.

Original: With linear combination, the branches can easily compensate each other, meaning each branch can activate output neurons not activated by the other.
Revised: With linear combination, the branches can easily compensate each other, meaning each branch can activate output neurons not activated by the other.

Original: However, additional cost is added as both the size of feature maps and the number of channels in the two branches need to be adjusted to be the same before addition.
Revised: However, additional cost is added as both the size of feature maps and the number of channels in the two branches need to be adjusted to be the same before addition.

Original: On the other hand, merging by concatenation only needs to spatially align the feature maps.
Revised: On the other hand, merging by concatenation only needs to spatially align the feature maps.

Original: On the other hand, concatenation only needs to align the feature map size, however requires a 1 × 1 convolution reducing the number of channels after concatenation, which is a more expensive operation than the pointwise addition.
Revised: On the other hand, concatenation only needs to align the feature map size, however requires a 1 × 1 convolution reducing the number of channels after concatenation, which is a more expensive operation than the pointwise addition.

Original: While linear combination provides an immediate exchange of the activations of both branches, concatenation relies on the following layers for this exchange.
Revised: While linear combination provides an immediate exchange of the activations of both branches, concatenation relies on the following layers for this exchange.

Original: This delay in exchange could possibly be problematic if the information from each branch is destructively altered before merging.
Revised: This delay in exchange could possibly be problematic if the information from each branch is destructively altered before merging.

Original: For example, a nonlinearity such as ReLU would discard all activations less than zero, effectively ignoring negative features in both branches before merging.
Revised: For example, a nonlinearity such as ReLU would discard all activations less than zero, effectively ignoring negative features in both branches before merging.

Original: Since linear combination does not cause too much overhead and provides better accuracy, we chose linear combination as our merging approach.
Revised: Since linear combination does not cause too much overhead and provides better accuracy, we chose linear combination as our merging approach.

Original: In Appendix A.4, we empirically show that the linear combination approach performs better than concatenation in object recognition.
Revised: In Appendix A.4, we empirically show that the linear combination approach performs better than concatenation in object recognition.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We conducted extensive experiments, as discussed below, to validate the effectiveness of our proposed bL-Net on object and speech recognition tasks.
Revised: We conducted extensive experiments, as discussed below, to validate the effectiveness of our proposed bL-Net on object and speech recognition tasks.

Original: bL-Net can be easily integrated with many modern
Revised: bL-Net can be easily integrated with many modern

Original: CNNs and here we chose ResNet (He et al., 2016) as the primary architecture to evaluate our approach.
Revised: CNNs and here we chose ResNet (He et al., 2016) as the primary architecture to evaluate our approach.

Original: For simplicity, from now on, we denote by bL- M the bL-Net using a backbone network M .
Revised: For simplicity, from now on, we denote by bL- M the bL-Net using a backbone network M .

Original: For example, bL-ResNet-50 is the Big-Little net based on ResNet-50.
Revised: For example, bL-ResNet-50 is the Big-Little net based on ResNet-50.

Original: 4. O BJECT R ECOGNITION
Revised: 4. O BJECT R ECOGNITION

Original: We used the ImageNet dataset (Russakovsky et al., 2015) for all the experiments below on object recognition.
Revised: We used the ImageNet dataset (Russakovsky et al., 2015) for all the experiments below on object recognition.

Original: This dataset is a common benchmark for object recognition, which contains 1.28 million training images and 50k validation images with labels from 1000 categories.
Revised: This dataset is a common benchmark for object recognition, which contains 1.28 million training images and 50k validation images with labels from 1000 categories.

Original: The details of our experimental setup and the network structures for bL-ResNet-50, 101 and 152 can be found in
Revised: The details of our experimental setup and the network structures for bL-ResNet-50, 101 and 152 can be found in

Original: Appendix A.1.
Revised: Appendix A.1.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ResNet as the backbone network.
Revised: ResNet as the backbone network.

Original: We experimented with different complexity control factors ( α and β ) to better understand their effects on performance.
Revised: We experimented with different complexity control factors ( α and β ) to better understand their effects on performance.

Original: α and β control both the structural and computational complexity of the Little-Branch, which determines the overall computational cost of bL-Net .
Revised: α and β control both the structural and computational complexity of the Little-Branch, which determines the overall computational cost of bL-Net .

Original: As can be seen in Table 1, all the models based on ResNet-50 yield better performance over the baseline with less computation, clearly demonstrating the advantage of combining low- and highcomplexity networks to balance between speed and accuracy.
Revised: As can be seen in Table 1, all the models based on ResNet-50 yield better performance over the baseline with less computation, clearly demonstrating the advantage of combining low- and highcomplexity networks to balance between speed and accuracy.

Original: In addition, the small performance gaps between these models suggest that a computationally light Little-Branch ( < 15% of the entire network) can compensate well for the low resolution representation by providing ﬁner image details.
Revised: In addition, the small performance gaps between these models suggest that a computationally light Little-Branch ( < 15% of the entire network) can compensate well for the low resolution representation by providing ﬁner image details.

Original: We consider α = 2 and β = 4 as the default setting for the following experiments.
Revised: We consider α = 2 and β = 4 as the default setting for the following experiments.

Original: Furthermore, there are more ablation studies on the design of bL-Net in the Appendix A.4.
Revised: Furthermore, there are more ablation studies on the design of bL-Net in the Appendix A.4.

Original: We further evaluated our approach on deeper models by using ResNet-101 and ResNet-152 as the backbone networks.
Revised: We further evaluated our approach on deeper models by using ResNet-101 and ResNet-152 as the backbone networks.

Original: We see from Table 2 that bL-ResNet-101 and bL-ResNet-152 behave similarly to bL-ResNet-50.
Revised: We see from Table 2 that bL-ResNet-101 and bL-ResNet-152 behave similarly to bL-ResNet-50.

Original: As expected, both of them produce better results against the baseline models and achieving notable computational gains.
Revised: As expected, both of them produce better results against the baseline models and achieving notable computational gains.

Original: Interestingly, our approach computationally favors deeper models, as evidenced by the fact that more speedups are observed on bL-ResNet-152 (2.3 × ) than on bL-ResNet-101 (2.0 × ) and bL-ResNet-50 (1.4 × ).
Revised: Interestingly, our approach computationally favors deeper models, as evidenced by the fact that more speedups are observed on bL-ResNet-152 (2.3 × ) than on bL-ResNet-101 (2.0 × ) and bL-ResNet-50 (1.4 × ).

Original: This is mainly because the Little Branch operating on low resolution spends less computation in a deeper model.
Revised: This is mainly because the Little Branch operating on low resolution spends less computation in a deeper model.

Original: ResNeXt and SEResNeXt as the backbone network.
Revised: ResNeXt and SEResNeXt as the backbone network.

Original: We extended bL-Net to ResNeXt and SEResNeXt, two of the more accurate yet compact network architectures.
Revised: We extended bL-Net to ResNeXt and SEResNeXt, two of the more accurate yet compact network architectures.

Original: We also experimented with (SE)ResNeXt-50 and (SE)ResNeXt-101 using the 32 × 4d and 64 × 4d setting (Xie et al., 2017; Hu et al., 2018).
Revised: We also experimented with (SE)ResNeXt-50 and (SE)ResNeXt-101 using the 32 × 4d and 64 × 4d setting (Xie et al., 2017; Hu et al., 2018).

Original: In our case, the Big-Branch follows the same setting of (SE)ResNeXt; however, we changed the cardinality of the Little-Branch to align with the input channels of a group convolution in the Big-Branch.
Revised: In our case, the Big-Branch follows the same setting of (SE)ResNeXt; however, we changed the cardinality of the Little-Branch to align with the input channels of a group convolution in the Big-Branch.

Original: All the results are shown in Table 2.
Revised: All the results are shown in Table 2.

Original: bL-ResNeXt-50 achieves a moderate speedup (1.40 × ) and provides an additional gain of 0.6% in accuracy.
Revised: bL-ResNeXt-50 achieves a moderate speedup (1.40 × ) and provides an additional gain of 0.6% in accuracy.

Original: However, bL-ResNeXt-101 (32 × 4d) gains a much more substantial speedup of 2 × and seeing 0.12% improvement in accuracy.
Revised: However, bL-ResNeXt-101 (32 × 4d) gains a much more substantial speedup of 2 × and seeing 0.12% improvement in accuracy.

Original: The same trend can be seen on bL-ResNeXt-101 (64 × 4d).
Revised: The same trend can be seen on bL-ResNeXt-101 (64 × 4d).

Original: On the other hand, our bL-SEResNeXts also produce better performance while reducing more FLOPs than SEResNeXts.
Revised: On the other hand, our bL-SEResNeXts also produce better performance while reducing more FLOPs than SEResNeXts.

Original: bL-SEResNeXt-101 achieves 2 × speedups and improves accuracy by 0.13%.
Revised: bL-SEResNeXt-101 achieves 2 × speedups and improves accuracy by 0.13%.

Original: Since our bL-Net saves more budget in computation, we can evaluate a model at a larger image scale for better performance, e.g., 256 × 256 .
Revised: Since our bL-Net saves more budget in computation, we can evaluate a model at a larger image scale for better performance, e.g., 256 × 256 .

Original: As illustrated in Table 2, our models evaluated at 256 × 256 is consistently better than their corresponding baselines while still using fewer FLOPs.
Revised: As illustrated in Table 2, our models evaluated at 256 × 256 is consistently better than their corresponding baselines while still using fewer FLOPs.

Original: The advantage becomes more pronounced with deeper models.
Revised: The advantage becomes more pronounced with deeper models.

Original: For instance, with 40% reduction on FLOPs, bL-ResNeXt-101@256 (64 × 4d) boosts the top-1 performance by 1.1%, which is quite impressive given that ResNeXt-101 (64 × 4d) is a very competitive baseline.
Revised: For instance, with 40% reduction on FLOPs, bL-ResNeXt-101@256 (64 × 4d) boosts the top-1 performance by 1.1%, which is quite impressive given that ResNeXt-101 (64 × 4d) is a very competitive baseline.

Original: Table 2 also shows the running times of bL-Net on GPU, which indicate the practical speedups of these models are consistent with the theoretical FLOP reductions reported in Table 2.
Revised: Table 2 also shows the running times of bL-Net on GPU, which indicate the practical speedups of these models are consistent with the theoretical FLOP reductions reported in Table 2.

Original: 4.1. C OMPARISON WITH R ELATED W ORK
Revised: 4.1. C OMPARISON WITH R ELATED W ORK

Original: We ﬁrst compared our method with the approaches that aim to accelerate ResNets or ResNeXts using techniques such as network pruning and adaptive computations.
Revised: We ﬁrst compared our method with the approaches that aim to accelerate ResNets or ResNeXts using techniques such as network pruning and adaptive computations.

Original: The results are shown in Figure 2.
Revised: The results are shown in Figure 2.

Original: Our bL-Net signiﬁcantly outperforms all related works regarding FLOPs reduction and accuracy.
Revised: Our bL-Net signiﬁcantly outperforms all related works regarding FLOPs reduction and accuracy.

Original: Our bL-ResNet-101 is ∼ 5% better than the network pruning approaches such as PFEC (Li et al., 2017) and LCCL (Dong et al., 2017), but still using less computation.
Revised: Our bL-ResNet-101 is ∼ 5% better than the network pruning approaches such as PFEC (Li et al., 2017) and LCCL (Dong et al., 2017), but still using less computation.

Original: When compared to SACT and ACT (Figurnov et al., 2017), our bL-ResNet-101 improves the accuracy by 5% while using the same number of FLOPs.
Revised: When compared to SACT and ACT (Figurnov et al., 2017), our bL-ResNet-101 improves the accuracy by 5% while using the same number of FLOPs.

Original: On the other hand, our bL-ResNet-101 outperforms some of the most recent works including BlockDrop (Wu et al., 2018), SkipNet (Wang et al., 2018) and ConvNet-AIG (Veit & Belongie, 2018) by 3.7%, 2.2%, 1.2% top-1 accuracy at the same FLOPs, respectively.
Revised: On the other hand, our bL-ResNet-101 outperforms some of the most recent works including BlockDrop (Wu et al., 2018), SkipNet (Wang et al., 2018) and ConvNet-AIG (Veit & Belongie, 2018) by 3.7%, 2.2%, 1.2% top-1 accuracy at the same FLOPs, respectively.

Original: This clearly demonstrates the advantages of a simple fusion of two branches at different scales over the more sophisticated dynamic routing techniques developed in these approaches.
Revised: This clearly demonstrates the advantages of a simple fusion of two branches at different scales over the more sophisticated dynamic routing techniques developed in these approaches.

Original: In comparison to
Revised: In comparison to

Original: SPPoint (Kuen et al., 2018) under the same FLOPs, our bL-ResNet-101 surpasses it by 2.2% in accuracy.
Revised: SPPoint (Kuen et al., 2018) under the same FLOPs, our bL-ResNet-101 surpasses it by 2.2% in accuracy.

Original: We also compared bL-Net with the variants of ResNets, like ResAttNe(X)t (Wang et al., 2017),
Revised: We also compared bL-Net with the variants of ResNets, like ResAttNe(X)t (Wang et al., 2017),

Original: SEResNeXt (Hu et al., 2018) and CBAM (Woo et al., 2018).
Revised: SEResNeXt (Hu et al., 2018) and CBAM (Woo et al., 2018).

Original: These models introduces attention mechanisms to enhance feature representations in either the channel or the spatial domain.
Revised: These models introduces attention mechanisms to enhance feature representations in either the channel or the spatial domain.

Original: From
Revised: From

Original: Figure 2, it can be seen that our bL-Net outperforms all of them in both FLOPs and accuracy.
Revised: Figure 2, it can be seen that our bL-Net outperforms all of them in both FLOPs and accuracy.

Original: Our bL-Net achieves better performance than ResAttNeXt while saving 1.5 × FLOPs.
Revised: Our bL-Net achieves better performance than ResAttNeXt while saving 1.5 × FLOPs.

Original: It also surpasses ∼ 1% point with similar FLOPs or the similar accuracy with ∼ 1.7 × FLOPs reduction for both
Revised: It also surpasses ∼ 1% point with similar FLOPs or the similar accuracy with ∼ 1.7 × FLOPs reduction for both

Original: SEResNeXt and CBAM.
Revised: SEResNeXt and CBAM.

Original: It’s worth noting that bL-Net can be potentially integrated with these models to further improve their accuracy and efﬁciency.
Revised: It’s worth noting that bL-Net can be potentially integrated with these models to further improve their accuracy and efﬁciency.

Original: Finally, we ran a benchmark test on bL-ResNeXt@256 under the PyTorch framework with a batch size of 16 on a K80 GPU, and compared against various models that are publicly available.
Revised: Finally, we ran a benchmark test on bL-ResNeXt@256 under the PyTorch framework with a batch size of 16 on a K80 GPU, and compared against various models that are publicly available.

Original: These models include Inception-V3 (Szegedy et al., 2016a), Inception-V4 (Szegedy et al., 2017), Inception-ResNetV2 (Szegedy et al., 2017), PolyNet (Zhang et al., 2017), NASNet (Zoph et al., 
Revised: These models include Inception-V3 (Szegedy et al., 2016a), Inception-V4 (Szegedy et al., 2017), Inception-ResNetV2 (Szegedy et al., 2017), PolyNet (Zhang et al., 2017), NASNet (Zoph et al., 2018)

Original: [Figure]
Revised: [Figure]

Original: From Fig.
Revised: From Fig.

Original: 3, we can see that overall, our bL-ResNeXt gains a better tradeoff between efﬁciency and accuracy.
Revised: 3, we can see that overall, our bL-ResNeXt gains a better tradeoff between efﬁciency and accuracy.

Original: Compared to the Inception networks, bL-Net are better in both FLOPs and GPU running time.
Revised: Compared to the Inception networks, bL-Net are better in both FLOPs and GPU running time.

Original: The bL-ResNeXt is also 2% point better than DenseNet at the same running speed, and 2 × faster than DualPathNet at the same performance.
Revised: The bL-ResNeXt is also 2% point better than DenseNet at the same running speed, and 2 × faster than DualPathNet at the same performance.

Original: NASNet achieves lower FLOPs and higher accuracy than bL-Net ; however, the networks result in slow GPU running time since their operations are divided into small pieces (i.e. a fragmented structure), which are not friendly for parallel computations.
Revised: NASNet achieves lower FLOPs and higher accuracy than bL-Net ; however, the networks result in slow GPU running time since their operations are divided into small pieces (i.e. a fragmented structure), which are not friendly for parallel computations.

Original: On the other hand, bL-Net , although requiring more FLOPs, can still enjoy the computation optimization brought by modern GPU cards.
Revised: On the other hand, bL-Net , although requiring more FLOPs, can still enjoy the computation optimization brought by modern GPU cards.

Original: Moreover, compared with the lowest FLOP conﬁguration of NASNet, our bL-ResNeXt is 0.5% point better while running 1.5 × faster.
Revised: Moreover, compared with the lowest FLOP conﬁguration of NASNet, our bL-ResNeXt is 0.5% point better while running 1.5 × faster.

Original: To further validate the generality of bL-Net , we also integrated the bL-Net with the highly efﬁcient network, ShufﬂeNetV2 (Ma et al., 2018), and the results can be found in Appendix A.5.
Revised: To further validate the generality of bL-Net , we also integrated the bL-Net with the highly efﬁcient network, ShufﬂeNetV2 (Ma et al., 2018), and the results can be found in Appendix A.5.

Original: We also demonstrate the adaptability of bL-Net on the object detection task (see Appendix A.6).
Revised: We also demonstrate the adaptability of bL-Net on the object detection task (see Appendix A.6).

Original: 4. S PEECH R ECOGNITION
Revised: 4. S PEECH R ECOGNITION

Original: We train ResNet style acoustic models in the hybrid framework on Switchboard+Fisher (2000h) and provide results on Hub5 (Switchboard and Call Home portions).
Revised: We train ResNet style acoustic models in the hybrid framework on Switchboard+Fisher (2000h) and provide results on Hub5 (Switchboard and Call Home portions).

Original: Switchboard is a large dataset with 2000 hours of transcribed speech from 28 , 000 speakers, which is actively used as benchmark (Xiong et al., 2016; Saon et al., 2017) akin to ImageNet in the computer vision community.
Revised: Switchboard is a large dataset with 2000 hours of transcribed speech from 28 , 000 speakers, which is actively used as benchmark (Xiong et al., 2016; Saon et al., 2017) akin to ImageNet in the computer vision community.

Original: Our
Revised: Our

Original: ResNet acoustic models are similar to the state of the art models described in (Saon et al., 2017), though slightly simpliﬁed (less fully connected layers) and trained with a simpler procedure (no class balancing).
Revised: ResNet acoustic models are similar to the state of the art models described in (Saon et al., 2017), though slightly simpliﬁed (less fully connected layers) and trained with a simpler procedure (no class balancing).

Original: We provide results only after Cross-Entropy training and after decoding with a small language model (4M n-grams).
Revised: We provide results only after Cross-Entropy training and after decoding with a small language model (4M n-grams).

Original: Gains from this setting are typically maintained in the standard further pipelines like ﬁne-tuning with sequence training, using more complex language models.
Revised: Gains from this setting are typically maintained in the standard further pipelines like ﬁne-tuning with sequence training, using more complex language models.

Original: Appendix B gives a thorough overview of the architecture of the speech acoustic models.
Revised: Appendix B gives a thorough overview of the architecture of the speech acoustic models.

Original: The main difference speech acoustic models have compared to image classiﬁcation networks, is that striding or pooling only happens along the frequency axis, while along in the time direction we need to output dense predictions per frame (Sercu & Goel, 2016).
Revised: The main difference speech acoustic models have compared to image classiﬁcation networks, is that striding or pooling only happens along the frequency axis, while along in the time direction we need to output dense predictions per frame (Sercu & Goel, 2016).

Original: This means that the branches at different resolutions have a fundamentally different view of the signal as it is propagating through the network; the ratio of resolution in frequency (downsampled in the Big-Branch) vs resolution in time (same between branches) is different.
Revised: This means that the branches at different resolutions have a fundamentally different view of the signal as it is propagating through the network; the ratio of resolution in frequency (downsampled in the Big-Branch) vs resolution in time (same between branches) is different.

Original: We can think about this as the convolutional kernels having different “aspect ratios” between branches.
Revised: We can think about this as the convolutional kernels having different “aspect ratios” between branches.

Original: Therefore we not only expect FLOP reductions in bL-Net , but expect to have increased representational power.
Revised: Therefore we not only expect FLOP reductions in bL-Net , but expect to have increased representational power.

Original: In addition, similar to the case in object recognition (Table 2), we could process the speech signal at higher frequency resolution than what is computationally feasible for the baseline ResNets.
Revised: In addition, similar to the case in object recognition (Table 2), we could process the speech signal at higher frequency resolution than what is computationally feasible for the baseline ResNets.

Original: Table 3 shows the results for the different architectures described in Appendix B. Most results are in line with the observations in the object recognition bL-Net .
Revised: Table 3 shows the results for the different architectures described in Appendix B. Most results are in line with the observations in the object recognition bL-Net .

Original: When comparing the baseline
Revised: When comparing the baseline

Original: ResNet-
Revised: ResNet-

Original: [Table]
Revised: [Table]

Original: (line 1) to the best bL-ResNet-22 (line 5), we see not only a reduction in FLOPs, but also a modest gain in Word Error Rate (WER).
Revised: (line 1) to the best bL-ResNet-22 (line 5), we see not only a reduction in FLOPs, but also a modest gain in Word Error Rate (WER).

Original: Comparing lines 2-4, we see that increasing β
Revised: Comparing lines 2-4, we see that increasing β

Original: (i
Revised: (i

Original: [Table]
Revised: [Table]

Original: e. shorter little branches at full resolution) causes no WER degradation, while reducing the number of FLOPs.
Revised: e. shorter little branches at full resolution) causes no WER degradation, while reducing the number of FLOPs.

Original: From line 5 we see that, similar to the object recognition ResNet results, decreasing α from 4 to 2 (i.e. keeping more feature maps in the full-resolution little branches) is important for performance, even though this increases the FLOPs again.
Revised: From line 5 we see that, similar to the object recognition ResNet results, decreasing α from 4 to 2 (i.e. keeping more feature maps in the full-resolution little branches) is important for performance, even though this increases the FLOPs again.

Original: We can summarize the best setting of α = 2 and β =for the little branches at full resolution: make them shorter but with more feature maps.
Revised: We can summarize the best setting of α = 2 and β =for the little branches at full resolution: make them shorter but with more feature maps.

Original: This is consistent with the image classiﬁcation results.
Revised: This is consistent with the image classiﬁcation results.

Original: From line 2 vs. line 6, the concatenation merge mode performs similar to the default additive merging, while increasing the number of FLOPs.
Revised: From line 2 vs. line 6, the concatenation merge mode performs similar to the default additive merging, while increasing the number of FLOPs.

Original: Line(compare to line 2) shows an experiment with additional branches on the lower layers (See Appendix
Revised: Line(compare to line 2) shows an experiment with additional branches on the lower layers (See Appendix

Original: B).
Revised: B).

Original: Although there is some gain in WER, the added parameters and compute on the lower layers may not make this a worthwhile trade-off.
Revised: Although there is some gain in WER, the added parameters and compute on the lower layers may not make this a worthwhile trade-off.

Original: 4. D ISCUSSION ON bL-Net
Revised: 4. D ISCUSSION ON bL-Net

Original: From the results of both tasks, we observe the following common insights, which enable us to design an efﬁcient multi-scale network with competitive performance: (I)
Revised: From the results of both tasks, we observe the following common insights, which enable us to design an efﬁcient multi-scale network with competitive performance: (I)

Original: The Little-Branch can be very light-weight, (II) bL-Net performs better when the Little-Branch is wide and shallow (smaller α and larger β ), (III) merging is effective when the feature dimension has changed, and (IV) branch merging by addition is more effective than concatenation.
Revised: The Little-Branch can be very light-weight, (II) bL-Net performs better when the Little-Branch is wide and shallow (smaller α and larger β ), (III) merging is effective when the feature dimension has changed, and (IV) branch merging by addition is more effective than concatenation.

Original: (I) is because the Big-Branch can extract essential information, a light Little-Branch is good enough to provide sufﬁcient information the Big-Branch lacks.
Revised: (I) is because the Big-Branch can extract essential information, a light Little-Branch is good enough to provide sufﬁcient information the Big-Branch lacks.

Original: Regarding (II), wider networks have been shown to perform better than deep networks while using a similar number of parameters.
Revised: Regarding (II), wider networks have been shown to perform better than deep networks while using a similar number of parameters.

Original: (III) is well-discussed in Appendix A.4.
Revised: (III) is well-discussed in Appendix A.4.

Original: Finally (IV), merging through addition provides better regularization for both branches to learn complementary features to form strong features.
Revised: Finally (IV), merging through addition provides better regularization for both branches to learn complementary features to form strong features.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed an efﬁcient multi-scale feature representation based on integrating multiple networks for object and speech recognition.
Revised: We proposed an efﬁcient multi-scale feature representation based on integrating multiple networks for object and speech recognition.

Original: The Big-Branches gain signiﬁcant computational reduction by working at low-resolution input but still extract meaningful features while the Little-Branch enriches the features from high-resolution input but with light computation.
Revised: The Big-Branches gain signiﬁcant computational reduction by working at low-resolution input but still extract meaningful features while the Little-Branch enriches the features from high-resolution input but with light computation.

Original: On object recognition task, we demonstrated that our approach provides approximately 2 × speedup over baselines while improving accuracy, and the result signiﬁcantly outperforms the state-of-the-art networks by a large margin in terms of accuracy and FLOPs reduction.
Revised: On object recognition task, we demonstrated that our approach provides approximately 2 × speedup over baselines while improving accuracy, and the result signiﬁcantly outperforms the state-of-the-art networks by a large margin in terms of accuracy and FLOPs reduction.

Original: Furthermore, when using the proposed method on speech recognition task, we gained 0.2% WER and saved 30% FLOPs at the same time.
Revised: Furthermore, when using the proposed method on speech recognition task, we gained 0.2% WER and saved 30% FLOPs at the same time.

Original: In pratice, the proposed bL-Net shows that the reduced FLOPs can consistently speed up the running time on GPU.
Revised: In pratice, the proposed bL-Net shows that the reduced FLOPs can consistently speed up the running time on GPU.

Original: That evidence showed that the proposed bL-Net is an efﬁcient multi-scale feature representation structure for competitive performance with less computation.
Revised: That evidence showed that the proposed bL-Net is an efﬁcient multi-scale feature representation structure for competitive performance with less computation.

Original: In this paper, we chose
Revised: In this paper, we chose

Original: ResNet, ResNeXt and SEResNeXt as our backbone networks but bL-Net can be integrated with other advanced network structures, like DenseNet (Huang et al., 2017b), DualPathNet (Chen et al., 2017b) and NASNet (Zoph et al., 2018) to achieve competitive performance while saving computations.
Revised: ResNet, ResNeXt and SEResNeXt as our backbone networks but bL-Net can be integrated with other advanced network structures, like DenseNet (Huang et al., 2017b), DualPathNet (Chen et al., 2017b) and NASNet (Zoph et al., 2018) to achieve competitive performance while saving computations.

Original: Furthermore, bL-Net can be integrated with those CNN acceleration approaches to make models more compact and efﬁcient. 
Revised: Furthermore, bL-Net can be integrated with those CNN acceleration approaches to make models more compact and efﬁcient. A CKNOWLEDGMENTS

Revised: The authors would like to thank Dr. Paul Crumley and Dr. I-Hsin Chung for their help with the hardware infrastructure setup.

Original: Appendix
Revised: Quanfu Fan and Rogerio Feris are supported by IARPA via DOI/IBC contract number D17PC The U S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofﬁcial policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U

Revised: S. Government

Original: The appendix illustrates the details of our experiments on object recognition and speech recognition and more ablation study.
Revised: The appendix illustrates the details of our experiments on object recognition and speech recognition and more ablation study.

Original: A bL-Net FOR O BJECT R ECOGNITION A. E XPERIMENTAL S ETUP
Revised: A bL-Net FOR O BJECT R ECOGNITION A. E XPERIMENTAL S ETUP

Original: We used the ImageNet dataset for all experiments.
Revised: We used the ImageNet dataset for all experiments.

Original: We trained all the models by Tensorpack (Wu, 2017), a higher-level wrapper for Tensorﬂow (Abadi et al., 2015).
Revised: We trained all the models by Tensorpack (Wu, 2017), a higher-level wrapper for Tensorﬂow (Abadi et al., 2015).

Original: All the models were trained with 110 epochs, batch size 256, weight decay 0.0001, momentum 0.9 and Nesterov momentum optimizer.
Revised: All the models were trained with 110 epochs, batch size 256, weight decay 0.0001, momentum 0.9 and Nesterov momentum optimizer.

Original: Furthermore, we used cosine learning-rate schedule as (Huang et al., 2017a; Loshchilov & Sgdr, 2017).
Revised: Furthermore, we used cosine learning-rate schedule as (Huang et al., 2017a; Loshchilov & Sgdr, 2017).

Original: We deployed the popular augmentation technique in (Szegedy et al., 2016b; Gross & Wilber, 2016) to increase the variety of training data, and randomly crop a 224 × 224 patch as training image.
Revised: We deployed the popular augmentation technique in (Szegedy et al., 2016b; Gross & Wilber, 2016) to increase the variety of training data, and randomly crop a 224 × 224 patch as training image.

Original: The validation error is evaluated by resizing the shorter side of an image to 256 and then crop a 224 × 224 from the center.
Revised: The validation error is evaluated by resizing the shorter side of an image to 256 and then crop a 224 × 224 from the center.

Original: Note that the results reported here for the vanilla ResNet, ResNeXt and
Revised: Note that the results reported here for the vanilla ResNet, ResNeXt and

Original: SEResNeXt models are difference from those reported in the original paper (He et al., 2016; Gross & Wilber, 2016;
Revised: SEResNeXt models are difference from those reported in the original paper (He et al., 2016; Gross & Wilber, 2016;

Original: Hu et al., 2018).
Revised: Hu et al., 2018).

Original: Our vanilla ResNet is better than the original paper while vanilla
Revised: Our vanilla ResNet is better than the original paper while vanilla

Original: ResNeXt and SEResNeXt is slightly worse than the original paper.
Revised: ResNeXt and SEResNeXt is slightly worse than the original paper.

Original: [Table]
Revised: [Table]

Revised: ResBlock

Revised: [Table]

Original: A. N ETWORK S TRUCTURE
Revised: [Table] ResBlock, [Table] ResBlock

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ResBlock

Revised: [Table]

Revised: [Table]

Revised: ResBlock,

Revised: [Table]

Revised: [Table]

Original: This section shows the details of network structures of our bL-ResNet
Revised: ResBlock, [Table] [Table] [Table]

Revised: ResBlock,

Original: [Table]
Revised: [Table]

Original:  and the setting of α and β isand 4, respectively.
Revised: [Table] [Table] A. N ETWORK S TRUCTURE This section shows the details of network structures of our bL-ResNet, and the setting of α and β isand 4, respectively.

Original: To understand how do we design bL-ResNet-50 based on ResNet-50, Tableshows the details of network structure.
Revised: To understand how do we design bL-ResNet-50 based on ResNet-50, Tableshows the details of network structure.

Original: We used a bottleneck residual block as a ResBlock
Revised: We used a bottleneck residual block as a ResBlock, and a

Original: [Table] and a ResBlock, C denotes a block composed of 1 × 1 , 3 × 3 , and 1 × 1 convolutions, where the ﬁrst 1 ×and the 3 × 3 have C/ 4 kernels and the last 1 × 1 has C kernels [Table]
Revised: ResBlock, C denotes a block composed of 1 × 1 , 3 × 3 , and 1 × 1 convolutions, where the ﬁrst 1 ×and the 3 × 3 have C/ 4 kernels and the last 1 × 1 has C kernels.

Original: First, the Big-Branch and the Little-Branch shares a residual block at the transition layer, so the number of residual blocks in each branch will be subtracted by 1.
Revised: First, the Big-Branch and the Little-Branch shares a residual block at the transition layer, so the number of residual blocks in each branch will be subtracted by 1.

Original: The number of residual blocks in the Little-Branch is deﬁned as (cid:100) Lβ (cid:101) − 1 and at least one, where L is the number of residual blocks in the Big-Branch [Table] and the number of kernels in a convolutional layer would be Cα , where C is the number of kernels in the Big-Branch.
Revised: The number of residual blocks in the Little-Branch is deﬁned as (cid:100) Lβ (cid:101) − 1 and at least one, where L is the number of residual blocks in the Big-Branch, and the number of kernels in a convolutional layer would be Cα , where C is the number of kernels in the Big-Branch.

Original: Thus, for all stages, the number of blocks in the Little-Branch is only one, and the number of blocks in the Big-Branch would be the number of blocks in ResNetmiuns one.
Revised: Thus, for all stages, the number of blocks in the Little-Branch is only one, and the number of blocks in the Big-Branch would be the number of blocks in ResNetmiuns one.

Original: For bL-ResNet-101 and bL-ResNet-152, we redistributed the residual blocks to different stages to balance the residual blocks at each stage.
Revised: For bL-ResNet-101 and bL-ResNet-152, we redistributed the residual blocks to different stages to balance the residual blocks at each stage.

Original: A ResNet model has 5 stages, and each stage has the same spatial size.
Revised: A ResNet model has 5 stages, and each stage has the same spatial size.

Original: These two models accumulate most of the convolutions (or computations) on the 4 th stage, where the size of feature maps is 14 × 14 when input size is 224 × 224 .
Revised: These two models accumulate most of the convolutions (or computations) on the 4 th stage, where the size of feature maps is 14 × 14 when input size is 224 × 224 .

Original: While such a design may be suitable for a very deep model, it likely limits the ability of Big-Branch to learn information at large scales, which mostly resides at earlier stages.
Revised: While such a design may be suitable for a very deep model, it likely limits the ability of Big-Branch to learn information at large scales, which mostly resides at earlier stages.

Original: Thus, we move some blocks in the 4 th stage of these two models to the 2 nd and 3 rd stages.
Revised: Thus, we move some blocks in the 4 th stage of these two models to the 2 nd and 3 rd stages.

Original: Table 5 shows the details of bL-ResNet-101 and bLResNet-152.
Revised: Table 5 shows the details of bL-ResNet-101 and bLResNet-152.

Original: [Table]
Revised: [Table]

Original: A. P ERFORMANCE ON L OW R ESOLUTION I NPUT
Revised: A. P ERFORMANCE ON L OW R ESOLUTION I NPUT

Original: We analyzed what advantages bL-Net could provide as compared to the network which works on low resolution input directly (ResNet-50- lowres ).
Revised: We analyzed what advantages bL-Net could provide as compared to the network which works on low resolution input directly (ResNet-50- lowres ).

Original: As shown in Table 6, ResNet-50- lowres reduces lots of computations but its accuracy is not acceptable; however, bL-ResNet-50 ( α = 2 and β = 4 ) achieves a better balance between accuracy and performance.
Revised: As shown in Table 6, ResNet-50- lowres reduces lots of computations but its accuracy is not acceptable; however, bL-ResNet-50 ( α = 2 and β = 4 ) achieves a better balance between accuracy and performance.

Original: A similar trend is also observed on a deeper model ResNet-101- lowres .
Revised: A similar trend is also observed on a deeper model ResNet-101- lowres .

Original: While such performance is unsatisfying compared to the state of the art, it is quite reasonable and expected given that almost 3 ∼ 4 × reduction of computation are achieved in such a case.
Revised: While such performance is unsatisfying compared to the state of the art, it is quite reasonable and expected given that almost 3 ∼ 4 × reduction of computation are achieved in such a case.

Original: Figure 4 shows the prediction results from bL-ResNet-50 and ResNet-50- lowres .
Revised: Figure 4 shows the prediction results from bL-ResNet-50 and ResNet-50- lowres .

Original: When both models predict correctly (4 (a) and (b)), the bL-ResNet-50 provides better conﬁdence for the prediction.
Revised: When both models predict correctly (4 (a) and (b)), the bL-ResNet-50 provides better conﬁdence for the prediction.

Original: Because the object only occupies a small portion of an image, the Little-Branch can still capture the object clearly.
Revised: Because the object only occupies a small portion of an image, the Little-Branch can still capture the object clearly.

Original: On the other hand, when the key features of an object is small, like the shape of beak of a bird (c) and the spots of a ladybug (d), bL-ResNet-50 can easily retain that key feature to predict correctly while ResNet-50- lowres provides wrong predicted label.
Revised: On the other hand, when the key features of an object is small, like the shape of beak of a bird (c) and the spots of a ladybug (d), bL-ResNet-50 can easily retain that key feature to predict correctly while ResNet-50- lowres provides wrong predicted label.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  A BLATION S TUDY ON N ETWORK M ERGING AND M ULTI -B RANCH
Revised: . A BLATION S TUDY ON N ETWORK M ERGING AND M ULTI -B RANCH

Original: Is linear combination better than concatenation?
Revised: Is linear combination better than concatenation?

Original: We adopt the simpler addition in bL-Net .
Revised: We adopt the simpler addition in bL-Net .

Original: Nonetheless, if we design the Big-Branch in a way that the output channels is identical to the backbone networks, then the number of kernels in the Big-Branch would be only 1 − α with respect to the total number of kernels of the backbone network; thus, in this case, the overall bL-Net can be more efﬁcient while the performance degradation could be compromised.
Revised: Nonetheless, if we design the Big-Branch in a way that the output channels is identical to the backbone networks, then the number of kernels in the Big-Branch would be only 1 − α with respect to the total number of kernels of the backbone network; thus, in this case, the overall bL-Net can be more efﬁcient while the performance degradation could be compromised.

Original: We compared the performance of these two different merging schemes in Table 7.
Revised: We compared the performance of these two different merging schemes in Table 7.

Original: Although concatenation approach is more efﬁcient, it performs much worse than addition with a gap of almost 1 . 5 %.
Revised: Although concatenation approach is more efﬁcient, it performs much worse than addition with a gap of almost 1 . 5 %.

Original: This leaves addition as a better choice for bL-Net in both visual and speech tasks.
Revised: This leaves addition as a better choice for bL-Net in both visual and speech tasks.

Original: More-branch in bL-Net As mentioned in Section 3.1, our approach can be extended to a scenario with multiple image scales.
Revised: More-branch in bL-Net As mentioned in Section 3.1, our approach can be extended to a scenario with multiple image scales.

Original: We experimented with three scales [1/4, 1/2, 1] on bL-ResNet-50 ( K = 3 ) where ResNet-50 is served as the Big-Branch at the scale of 1/4 of the original input, i.e. 56 × 56 .
Revised: We experimented with three scales [1/4, 1/2, 1] on bL-ResNet-50 ( K = 3 ) where ResNet-50 is served as the Big-Branch at the scale of 1/4 of the original input, i.e. 56 × 56 .

Original: As indicated in Table 7, a 3-scale bL-Net requires more FLOPs and parameters due to the fact that the overhead in merging more branches is signiﬁcant for ResNet-50, but even though, it still cannot provide superior performance of a 2-scale bL-Net .
Revised: As indicated in Table 7, a 3-scale bL-Net requires more FLOPs and parameters due to the fact that the overhead in merging more branches is signiﬁcant for ResNet-50, but even though, it still cannot provide superior performance of a 2-scale bL-Net .

Original: This is because the Big-Branch in the 3-scale bL-Net is downsampled aggressively by 4 times, thus substantially degrade the capability of feature representation in the Big-Branch.
Revised: This is because the Big-Branch in the 3-scale bL-Net is downsampled aggressively by 4 times, thus substantially degrade the capability of feature representation in the Big-Branch.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Number of Merges in bL-Net
Revised: Number of Merges in bL-Net

Original: We also analyzed the number of merges we needed in the bL-Net .
Revised: We also analyzed the number of merges we needed in the bL-Net .

Original: One big difference between our approach and others is that bL-Net merges multiple times as opposed to only once in most of the other approaches.
Revised: One big difference between our approach and others is that bL-Net merges multiple times as opposed to only once in most of the other approaches.

Original: Below we provide an explanation of why more information exchange is encouraged in our approach and when is the best moment for merging operation.
Revised: Below we provide an explanation of why more information exchange is encouraged in our approach and when is the best moment for merging operation.

Original: In the above bL-Net
Revised: In the above bL-Net

Original: [Table]
Revised: [Table]

Original: we merged branches before the feature dimension changes, except for the ﬁrst stride convolution; thus, we used 4 merges ( m = 4 ).
Revised: we merged branches before the feature dimension changes, except for the ﬁrst stride convolution; thus, we used 4 merges ( m = 4 ).

Original: We experimented with a different number of merges for bL-ResNet-50 and bL-ResNet-
Revised: We experimented with a different number of merges for bL-ResNet-50 and bL-ResNet-

Original: [Table]
Revised: [Table]

Original: and the results are shown in Table 8.
Revised: and the results are shown in Table 8.

Original: Since there are fewer layers in bL-ResNet-50, we reduce the number of merges to show their importance; on the other hand, there are more layers in bL-ResNet-101, so we add more merges to show that those additional merges would similarly not improve the performance anymore.
Revised: Since there are fewer layers in bL-ResNet-50, we reduce the number of merges to show their importance; on the other hand, there are more layers in bL-ResNet-101, so we add more merges to show that those additional merges would similarly not improve the performance anymore.

Original: The accuracy of the bL-ResNet-50 models with less number of merges ( m = 1 and m = 2 ) is signiﬁcantly worse than with more ( m = 4 ) and they do not save many FLOPs and parameters at all.
Revised: The accuracy of the bL-ResNet-50 models with less number of merges ( m = 1 and m = 2 ) is signiﬁcantly worse than with more ( m = 4 ) and they do not save many FLOPs and parameters at all.

Original: This justiﬁes frequent information exchange improves the performance.
Revised: This justiﬁes frequent information exchange improves the performance.

Original: On the other hand, bL-ResNet-101 ( m = 7 ) uses more merges; however, it also does not improve the performance and requires more FLOPs, which comes from more merges.
Revised: On the other hand, bL-ResNet-101 ( m = 7 ) uses more merges; however, it also does not improve the performance and requires more FLOPs, which comes from more merges.

Original: This is because the original setting for the amount of merging happened when either the channel number or feature map size is changed, so extra merges happened at the feature dimension.
Revised: This is because the original setting for the amount of merging happened when either the channel number or feature map size is changed, so extra merges happened at the feature dimension.

Original: Thus, those extra merges could be redundant since merging at identical dimension could be reduced to one merging.
Revised: Thus, those extra merges could be redundant since merging at identical dimension could be reduced to one merging.

Original: Hence, it empirically proves that merging before dimension is changed is the most effective.
Revised: Hence, it empirically proves that merging before dimension is changed is the most effective.

Original: A. bL-Net FOR H IGH -E FFICIENCY N ETWORK
Revised: A. bL-Net FOR H IGH -E FFICIENCY N ETWORK

Original: To demenstrate bL-Net can be applied on different types of network, we deploy bL-Net on the high-efﬁciency network, ShufﬂeNetV2 (Ma et al., 2018), and results are shown in Table 9.
Revised: To demenstrate bL-Net can be applied on different types of network, we deploy bL-Net on the high-efﬁciency network, ShufﬂeNetV2 (Ma et al., 2018), and results are shown in Table 9.

Original: Our bL-ShufﬂeNetV2@256 outperforms ShufﬂeNetV2 by up to 0.5% point under similar FLOPs, which suggests that our approach can also improve the high-efﬁciency networks.
Revised: Our bL-ShufﬂeNetV2@256 outperforms ShufﬂeNetV2 by up to 0.5% point under similar FLOPs, which suggests that our approach can also improve the high-efﬁciency networks.

Original: [Table]
Revised: [Table]

Original: A. bL-Net FOR O BJECT D ETECTION
Revised: A. bL-Net FOR O BJECT D ETECTION

Original: We demonstrate the effectiveness of bL-Net on object detection.
Revised: We demonstrate the effectiveness of bL-Net on object detection.

Original: We use bL-Net as a backbone network for FasterRCNN+FPN (Lin et al., 2017) on the PASCAL VOC (Everingham et al., 2010) and MS
Revised: We use bL-Net as a backbone network for FasterRCNN+FPN (Lin et al., 2017) on the PASCAL VOC (Everingham et al., 2010) and MS

Original: COCO datasets (Lin et al., 2014).
Revised: COCO datasets (Lin et al., 2014).

Original: Table 10 shows the comparison with the detector with ResNet-  as the backbone network, and the results show that bL-Net achieves competitive performance while saving about 1.5 × FLOPs 1 , suggesting that bL-Net is transferable to other vision tasks.
Revised: Table 10 shows the comparison with the detector with ResNet-  as the backbone network, and the results show that bL-Net achieves competitive performance while saving about 1.5 × FLOPs 1 , suggesting that bL-Net is transferable to other vision tasks.

Original: [Table]
Revised: [Table]

Original: A. C OMPARISON OF M EMORY R EQUIREMENT
Revised: A. C OMPARISON OF M EMORY R EQUIREMENT

Original: We benchmarked the GPU memory consumption in runtime at both the training and test phases for all the models evaluated in Fig.
Revised: We benchmarked the GPU memory consumption in runtime at both the training and test phases for all the models evaluated in Fig.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: The batch size was set to 8, which is the largest number allowed for NASNet on a P100 GPU card.
Revised: The batch size was set to 8, which is the largest number allowed for NASNet on a P100 GPU card.

Original: The image size for any model in this benchmark experiment is the same as that used in the experiment reported in Fig.
Revised: The image size for any model in this benchmark experiment is the same as that used in the experiment reported in Fig.

Original: For bL-Net , the input image size is 224 × 224 in training and 256 × 256 in test.
Revised: For bL-Net , the input image size is 224 × 224 in training and 256 × 256 in test.

Original: From Fig.
Revised: From Fig.

Original: [Table]
Revised: [Table]

Original: we can see that bL-Net is the most memory-efﬁcient for training among all the approaches.
Revised: we can see that bL-Net is the most memory-efﬁcient for training among all the approaches.

Original: In test, bL-ResNeXt consumes more memory than inception-resnet-v2 and inception-v at the same accuracy, but bL-SEResNeXt outperforms all the approaches.
Revised: In test, bL-ResNeXt consumes more memory than inception-resnet-v2 and inception-v at the same accuracy, but bL-SEResNeXt outperforms all the approaches.

Original: Note that NASNet and
Revised: Note that NASNet and

Original: PNASNet are not memory friendly.
Revised: PNASNet are not memory friendly.

Original: This is largely because they are trained on a larger image size (331 × 331) and these models are composed of many layers.
Revised: This is largely because they are trained on a larger image size (331 × 331) and these models are composed of many layers.

Original: [Table]
Revised: [Table]

Original: B bL-Net FOR S PEECH R ECOGNITION B. E XPERIMENTAL S ETUP
Revised: B bL-Net FOR S PEECH R ECOGNITION B. E XPERIMENTAL S ETUP

Original: In our experiments
Revised: In our experiments

Original: [Table]
Revised: [Table]

Original: we start with an input size of 64 × 49 , where 64 is the number of logmel ﬁlterbanks, calculated for each utterance on-the-ﬂy.
Revised: we start with an input size of 64 × 49 , where 64 is the number of logmel ﬁlterbanks, calculated for each utterance on-the-ﬂy.

Original: We also stack their ﬁrst and second derivatives to get 3 input channels, resulting in our ﬁnal input of dimensionality batch_size × 3 × 64 × 49 .
Revised: We also stack their ﬁrst and second derivatives to get 3 input channels, resulting in our ﬁnal input of dimensionality batch_size × 3 × 64 × 49 .

Original: Our output is of size batch_size × 512 × 4 × 1 , which is then projected to batch_size × 512 × 1 ×and ﬁnally to batch_size × 32 k × 1 × 1 for classiﬁcation.
Revised: Our output is of size batch_size × 512 × 4 × 1 , which is then projected to batch_size × 512 × 1 ×and ﬁnally to batch_size × 32 k × 1 × 1 for classiﬁcation.

Original: We then perform softmax cross-entropy over this output space of 32 k tied CD states from forced alignment, doing phone prediction on the central frame of the input utterance.
Revised: We then perform softmax cross-entropy over this output space of 32 k tied CD states from forced alignment, doing phone prediction on the central frame of the input utterance.

Original: We report results after Cross-Entropy training, on Hub5’ (SWB and CH part) after decoding using the standard small 4M n-gram language model with a 30.5k word vocabulary.
Revised: We report results after Cross-Entropy training, on Hub5’ (SWB and CH part) after decoding using the standard small 4M n-gram language model with a 30.5k word vocabulary.

Original: All models were trained in PyTorch (Paszke et al., 2017) over 16 epochs on 2 GPUs with per-GPU batch size 256 (total batch size of 512 ), gradient clipping 10 . 0 , weight decay 1 × 10 − 6 , nesterov accelerated momentum 0 .
Revised: All models were trained in PyTorch (Paszke et al., 2017) over 16 epochs on 2 GPUs with per-GPU batch size 256 (total batch size of 512 ), gradient clipping 10 . 0 , weight decay 1 × 10 − 6 , nesterov accelerated momentum 0 .

Original: 9 , and learning rate 0 .
Revised: 9 , and learning rate 0 .

Original: 03 (annealed by √ 0 .
Revised: 03 (annealed by √ 0 .

Original: 5 per epoch ≥ 10 ).
Revised: 5 per epoch ≥ 10 ).

Original: B. N ETWORK S TRUCTURES
Revised: B. N ETWORK S TRUCTURES

Original: ResNet-22 Our models follow a ResNet architecture without padding in time (Saon et al., 2017), which accounts for the fact that padding in time adds undesirable artifacts when processing a longer utterance (Sercu & Goel, 2016).
Revised: ResNet-22 Our models follow a ResNet architecture without padding in time (Saon et al., 2017), which accounts for the fact that padding in time adds undesirable artifacts when processing a longer utterance (Sercu & Goel, 2016).

Original: Under this constraint, each convolution operation reduces our input sequence in time by k − 1 , where k is the kernel width used.
Revised: Under this constraint, each convolution operation reduces our input sequence in time by k − 1 , where k is the kernel width used.

Original: This effect can be seen in Table 12, in which the time variable, T , is reduced in accordance with the number of convolutions.
Revised: This effect can be seen in Table 12, in which the time variable, T , is reduced in accordance with the number of convolutions.

Original: For similar reasons, when we stride we only do so in frequency, and not in time.
Revised: For similar reasons, when we stride we only do so in frequency, and not in time.

Original: For the rest of this section, when we refer to striding we are referring only to striding in frequency.
Revised: For the rest of this section, when we refer to striding we are referring only to striding in frequency.

Original: We deﬁne our residual blocks as a series of 3 × 3 convolutions.
Revised: We deﬁne our residual blocks as a series of 3 × 3 convolutions.

Original: When we transition from one stage to the next, we stride by 2 on the ﬁrst convolution of the following block.
Revised: When we transition from one stage to the next, we stride by 2 on the ﬁrst convolution of the following block.

Original: All of our models start with a 5 × 5 convolution with stride 2 , which downsamples the input from 64 melbins to 32 .
Revised: All of our models start with a 5 × 5 convolution with stride 2 , which downsamples the input from 64 melbins to 32 .

Original: Our baseline model, ResNet-22, consists of four stages of two-convolution residual blocks: (3 × 3 , 64) × 3; (3 × 3 , 128 , s 2) × 3; (3 × 3 , 256 , s 2) × 3; (3 × 3 , 512 , s 2) × 2 .
Revised: Our baseline model, ResNet-22, consists of four stages of two-convolution residual blocks: (3 × 3 , 64) × 3; (3 × 3 , 128 , s 2) × 3; (3 × 3 , 256 , s 2) × 3; (3 × 3 , 512 , s 2) × 2 .

Original: The output then goes through a bottleneck projection layer to the 32 k-dimensional output CD state: (4 × 1 , 512) and (1 × 1 , 32 k ) .
Revised: The output then goes through a bottleneck projection layer to the 32 k-dimensional output CD state: (4 × 1 , 512) and (1 × 1 , 32 k ) .

Original: bL-ResNet-22 ( α = 4 , β = 1 ) Table 12 displays two bL-Net architectures that we experimented with based on the ResNet-22 baseline.
Revised: bL-ResNet-22 ( α = 4 , β = 1 ) Table 12 displays two bL-Net architectures that we experimented with based on the ResNet-22 baseline.

Original: The bL-Net baseline, bL-ResNet-22 ( α = 4 , β = 1 ), consists of two branches in each Big-Little Module and is well-deﬁned through the parameters α and β .
Revised: The bL-Net baseline, bL-ResNet-22 ( α = 4 , β = 1 ), consists of two branches in each Big-Little Module and is well-deﬁned through the parameters α and β .

Original: In between each Big-Little Module we downsample our input using a transition layer consisting of a residual block with a single 3 × 3 convolution with stride 2 .
Revised: In between each Big-Little Module we downsample our input using a transition layer consisting of a residual block with a single 3 × 3 convolution with stride 2 .

Original: Therefore, we shift one convolution operation out of the last residual block in each stage that precedes a transition layer.
Revised: Therefore, we shift one convolution operation out of the last residual block in each stage that precedes a transition layer.

Original: Whenever downsampling is performed in the Big-Branch, it is only in the frequency dimension and not in time.
Revised: Whenever downsampling is performed in the Big-Branch, it is only in the frequency dimension and not in time.

Original: Similarly, bilinear upsampling only occurs in the frequency dimension.
Revised: Similarly, bilinear upsampling only occurs in the frequency dimension.

Original: All bL-Net variants end with the same projection and output layer as ResNet-22.
Revised: All bL-Net variants end with the same projection and output layer as ResNet-22.

Original: All merges, unless otherwise speciﬁed, are through linear combination with unit weights per branch.
Revised: All merges, unless otherwise speciﬁed, are through linear combination with unit weights per branch.

Original: For comparison, we experimented with a version of bL-ResNet-22 ( α = 4 , β = 1 ) using concatenation to merge branches, the results of
Revised: For comparison, we experimented with a version of bL-ResNet-22 ( α = 4 , β = 1 ) using concatenation to merge branches, the results of

Original: [Table]
Revised: [Table]

Original: which are presented in Table 3.
Revised: which are presented in Table 3.

Original: Using concatenation instead of linear combination in this model results in each stage having more channels after concatenation than the current stage of the network calls for (i
Revised: Using concatenation instead of linear combination in this model results in each stage having more channels after concatenation than the current stage of the network calls for (i

Original: [Table]
Revised: [Table]

Original: e. in stage 1 we end up with 64 + 16 = 80 channels at the end of the relevant Big-Little
Revised: e. in stage 1 we end up with 64 + 16 = 80 channels at the end of the relevant Big-Little

Original: Module
Revised: Module

Original: [Table]
Revised: [Table]

Original: whereas we only want 64 channels to be outputted).
Revised: whereas we only want 64 channels to be outputted).

Original: To resolve this, we apply a 1 ×convolution to reduce the number of channels accordingly and fuse the two separate feature maps.
Revised: To resolve this, we apply a 1 ×convolution to reduce the number of channels accordingly and fuse the two separate feature maps

Original:  bL-ResNet-22 ( α = 4 , β = 2 , 3 ) We explored two more models where we ﬁx α = 4 , one in which we take β = 2 and another where β
Revised: [Table] bL-ResNet-22 ( α = 4 , β = 2 , 3 ) We explored two more models where we ﬁx α = 4 , one in which we take β = 2 and another where β

Original: = 3 .
Revised: = 3 .

Original: All Big-Branchs in these models are the same as bL-ResNet-22 ( α = 4 , β = 1 ).
Revised: All Big-Branchs in these models are the same as bL-ResNet-22 ( α = 4 , β = 1 ).

Original: The difference in each Little-Branch is based on the setting of β .
Revised: The difference in each Little-Branch is based on the setting of β .

Original: Since the number of convolutions we use in the bL-Net baseline is uneven in the ﬁrst three stages, we take (cid:100) L / β (cid:101) to be the depth of the Little-Branch, where L is the depth of the Big-Branch.
Revised: Since the number of convolutions we use in the bL-Net baseline is uneven in the ﬁrst three stages, we take (cid:100) L / β (cid:101) to be the depth of the Little-Branch, where L is the depth of the Big-Branch.

Original: For β = 2 , the ﬁrst three stages of the network have a Little-Branch consisting of one residual block with two 3 × 3 convolutions and one residual block with one 3 × 3 convolution.
Revised: For β = 2 , the ﬁrst three stages of the network have a Little-Branch consisting of one residual block with two 3 × 3 convolutions and one residual block with one 3 × 3 convolution.

Original: This results in a reduction of the number of convolutions from 5 in the Big-Branch to 3 in the Little-Branch.
Revised: This results in a reduction of the number of convolutions from 5 in the Big-Branch to 3 in the Little-Branch.

Original: For β = 3 , the ﬁrst three stages of the network have a Little-Branch consisting of one residual block with two 3 ×convolutions, resulting in a reduction in the number of convolutions from 5 in the Big-Branch to 2 in the Little-Branch.
Revised: For β = 3 , the ﬁrst three stages of the network have a Little-Branch consisting of one residual block with two 3 ×convolutions, resulting in a reduction in the number of convolutions from 5 in the Big-Branch to 2 in the Little-Branch.

Original: For both models, the Little-Branch in the ﬁnal stage consists of a single residual block with two 3 × 3 convolutions, since there are only four convolutions in the Big-Branch of the ﬁnal stage and (cid:100) 4 / 3 (cid:101) =
Revised: For both models, the Little-Branch in the ﬁnal stage consists of a single residual block with two 3 × 3 convolutions, since there are only four convolutions in the Big-Branch of the ﬁnal stage and (cid:100) 4 / 3 (cid:101) =

Original: (cid:100) 4 / 2 (cid:101) = 2 .
Revised: (cid:100) 4 / 2 (cid:101) = 2 .

Original: In all bL-Net variants in which β > 1 , because we can’t pad in time, we see that the time dimension will get out of sync between the Big-Branch and Little-Branch.
Revised: In all bL-Net variants in which β > 1 , because we can’t pad in time, we see that the time dimension will get out of sync between the Big-Branch and Little-Branch.

Original: Therefore, before merging we need to match the output size of each branch.
Revised: Therefore, before merging we need to match the output size of each branch.

Original: To do this, we crop the shallower branches in time to match the deepest branch (i.e. the Big-Branch will always have a smaller time dimension due to having more convolutions, so we crop to match it).
Revised: To do this, we crop the shallower branches in time to match the deepest branch (i.e. the Big-Branch will always have a smaller time dimension due to having more convolutions, so we crop to match it).

Original: This is similar to the way the shortcut in ResNet is dealt with in (Saon et al., 2017), and does not introduce edge artifacts when processing longer sequences.
Revised: This is similar to the way the shortcut in ResNet is dealt with in (Saon et al., 2017), and does not introduce edge artifacts when processing longer sequences.

Original: bL-ResNet-22 ( α =2 , β =
Revised: bL-ResNet-22 ( α =

Revised: , β =

Original: The last of our two-branch models is where α = 2 and β = 3 , which is also presented in Table 12.
Revised: The last of our two-branch models is where α = 2 and β = 3 , which is also presented in Table 12.

Original: This variant is well-deﬁned in α and β up to the ﬁrst three stages of the network.
Revised: This variant is well-deﬁned in α and β up to the ﬁrst three stages of the network.

Original: In the last stage, however, we opt to not branch and instead follow identically the ﬁnal stage of ResNet-22 with two residual blocks operating at the full input resolution with 512 channels.
Revised: In the last stage, however, we opt to not branch and instead follow identically the ﬁnal stage of ResNet-22 with two residual blocks operating at the full input resolution with 512 channels.

Original: bL-PYR-ResNet-22 ( α = 4 , β = 1 )
Revised: bL-PYR-ResNet-22 ( α = 4 , β = 1 )

Original: We additionally present results on a pyramidal structure, in which the ﬁrst stage of the network operates with four branches, the second with three, the third with two, and the fourth equivalent to the fourth stage of ResNet-22 (and bL-ResNet-22 ( α = 2 , β = 3 )).
Revised: We additionally present results on a pyramidal structure, in which the ﬁrst stage of the network operates with four branches, the second with three, the third with two, and the fourth equivalent to the fourth stage of ResNet-22 (and bL-ResNet-22 ( α = 2 , β = 3 )).

Original: Due to the setting of α = 4 , we increased the number of channels in the ﬁrst stage Big-Branch to have 256 channels (with the three Little-Branches in this stage having 64 , 16 , and 4 channels), avoiding a single channel on the smallest branch.
Revised: Due to the setting of α = 4 , we increased the number of channels in the ﬁrst stage Big-Branch to have 256 channels (with the three Little-Branches in this stage having 64 , 16 , and 4 channels), avoiding a single channel on the smallest branch.

Original: Note that the middle branches require both resolution upsampling and 1 × 1 convolution to match channels.
Revised: Note that the middle branches require both resolution upsampling and 1 × 1 convolution to match channels.

Original: The third stage Big-Little Module operates on two branches and is identical to the analagous stage in bL-ResNet-22 ( α = 4 , β = 1 ).
Revised: The third stage Big-Little Module operates on two branches and is identical to the analagous stage in bL-ResNet-22 ( α = 4 , β = 1 ).