Original: C ONTEXTUAL R ECURRENT C ONVOLUTIONAL M ODEL FOR R OBUST V ISUAL L EARNING
Revised: C ONTEXTUAL R ECURRENT C ONVOLUTIONAL M ODEL FOR R OBUST V ISUAL L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Feedforward convolutional neural network has achieved a great success in many computer vision tasks.
Revised: Feedforward convolutional neural network has achieved a great success in many computer vision tasks.

Original: While it validly imitates the hierarchical structure of biological visual system, it still lacks one essential architectural feature: contextual recurrent connections with feedback, which widely exists in biological visual system.
Revised: While it validly imitates the hierarchical structure of biological visual system, it still lacks one essential architectural feature: contextual recurrent connections with feedback, which widely exists in biological visual system.

Original: In this work, we designed a Contextual Recurrent Convolutional Network with this feature embedded in a standard CNN structure.
Revised: In this work, we designed a Contextual Recurrent Convolutional Network with this feature embedded in a standard CNN structure.

Original: We found that such feedback connections could enable lower layers to “rethink” about their representations given the top-down contextual information.
Revised: We found that such feedback connections could enable lower layers to “rethink” about their representations given the top-down contextual information.

Original: We carefully studied the components of this network, and showed its robustness and superiority over feedforward baselines in such tasks as noise image classiﬁcation, partially occluded object recognition and ﬁne-grained image classiﬁcation.
Revised: We carefully studied the components of this network, and showed its robustness and superiority over feedforward baselines in such tasks as noise image classiﬁcation, partially occluded object recognition and ﬁne-grained image classiﬁcation.

Original: We believed this work could be an important step to help bridge the gap between computer vision models and real biological visual system.
Revised: We believed this work could be an important step to help bridge the gap between computer vision models and real biological visual system.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: It has been long established that the primate’s ventral visual system has a hierarchical structure (Felleman & Van Essen, 1991) including early (V1, V2), intermediate (V4), and higher (IT) visual areas.
Revised: It has been long established that the primate’s ventral visual system has a hierarchical structure (Felleman & Van Essen, 1991) including early (V1, V2), intermediate (V4), and higher (IT) visual areas.

Original: Modern deep convolutional neural networks (CNNs) for image recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014) trained on large image data sets like ImageNet (Russakovsky et al., 2015) imitate this hierarchical structure with multiple layers.
Revised: Modern deep convolutional neural networks (CNNs) for image recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014) trained on large image data sets like ImageNet (Russakovsky et al., 2015) imitate this hierarchical structure with multiple layers.

Original: There is a hierarchical correspondence between internal feature representations of a deep CNN’s different layers and neural representations of different visual areas (Cichy et al., 2016; Yamins & DiCarlo, 2016); lower visual areas (V1, V2) are best explained by a deep CNN’s internal representations from lower layers (Cadena et al., 2017; Khaligh-Razavi & Kriegeskorte, 2014) and higher areas (IT, V4) are best explained by its higher layers (Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014).
Revised: There is a hierarchical correspondence between internal feature representations of a deep CNN’s different layers and neural representations of different visual areas (Cichy et al., 2016; Yamins & DiCarlo, 2016); lower visual areas (V1, V2) are best explained by a deep CNN’s internal representations from lower layers (Cadena et al., 2017; Khaligh-Razavi & Kriegeskorte, 2014) and higher areas (IT, V4) are best explained by its higher layers (Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014).

Original: Deep
Revised: Deep

Original: CNNs explain neuron responses in ventral visual system better than any other model class (Yamins & DiCarlo, 2016; Kriegeskorte, 2015), and this success indicates that deep CNNs share some similarities with the ventral visual system, in terms of architecture and internal feature representations (Yamins & DiCarlo, 2016).
Revised: CNNs explain neuron responses in ventral visual system better than any other model class (Yamins & DiCarlo, 2016; Kriegeskorte, 2015), and this success indicates that deep CNNs share some similarities with the ventral visual system, in terms of architecture and internal feature representations (Yamins & DiCarlo, 2016).

Original: However, there is one key structural component that is missing in the standard feedforward deep
Revised: However, there is one key structural component that is missing in the standard feedforward deep

Original: CNNs: contextual feedback recurrent connections between neurons in different areas (Felleman &
Revised: CNNs: contextual feedback recurrent connections between neurons in different areas (Felleman &

Original: Van Essen, 1991).
Revised: Van Essen, 1991).

Original: These connections greatly contribute to the complexity of the visual system, and may be essential for the success of the visual systems in reality; for example, there are evidences that recurrent connections are crucial for object recognition under noise, clutter, and occlusion (O’Reilly et al., 2013;
Revised: These connections greatly contribute to the complexity of the visual system, and may be essential for the success of the visual systems in reality; for example, there are evidences that recurrent connections are crucial for object recognition under noise, clutter, and occlusion (O’Reilly et al., 2013;

Original: Spoerer et al., 2017; Rajaei et al., 2018).
Revised: Spoerer et al., 2017; Rajaei et al., 2018).

Original: In this paper, we explored a variety of model with different recurrent architectures, contextual modules, and information ﬂows to understand the computational advantages of feedback circuits.
Revised: In this paper, we explored a variety of model with different recurrent architectures, contextual modules, and information ﬂows to understand the computational advantages of feedback circuits.

Original: We are interested in understanding what and how top-down and bottom-up contextual information can be combined to improve in performance in visual tasks.
Revised: We are interested in understanding what and how top-down and bottom-up contextual information can be combined to improve in performance in visual tasks.

Original: We investigated VGG16 (Simonyan &
Revised: We investigated VGG16 (Simonyan &

Original: Zisserman, 2014), a standard CNN that closely approximate the ventral visual hierarchical stream, and its recurrent variants for comparison.
Revised: Zisserman, 2014), a standard CNN that coarsely approximate the ventral visual hierarchical stream, and its recurrent variants for comparison.

Original: To introduce feedback recurrent connections, we divided
Revised: To introduce feedback recurrent connections, we divided

Original: VGG16’s layers into groups and selectively added feedback connections from the groups’ highest layers to their lowest layers.
Revised: VGG16’s layers into groups and selectively added feedback connections from the groups’ highest layers to their lowest layers.

Original: At the end of each feedback connection, there is a contextual module (Section 3.2) that reﬁnes the bottom-up input with gated contextual information.
Revised: At the end of each feedback connection, there is a contextual module (Section 3.2) that reﬁnes the bottom-up input with gated contextual information.

Original: We tested and compared several networks with such contextual modules against VGG16 in several standard image classiﬁcation task, as well as visual tasks in which reﬁnement under feedback guidance is more likely to produce some beneﬁcial effects, such as object recognition under degraded conditions (noise, clutter and occlusion) and ﬁne-grained recognition.
Revised: We tested and compared several networks with such contextual modules against VGG16 in several standard image classiﬁcation task, as well as visual tasks in which reﬁnement under feedback guidance is more likely to produce some beneﬁcial effects, such as object recognition under degraded conditions (noise, clutter and occlusion) and ﬁne-grained recognition.

Original: We found that our network could outperform  baseline feedforward networks  by a large margin.
Revised: We found that our network could outperform all the baseline feedforward networks and surpassed them by a large margin in ﬁnegrained and occlusion tasks.

Original: Wewe also studied the internal feature representations of our network to illustrate the effectiveness of the structure.
Revised: We also studied the internal feature representations of our network to illustrate the effectiveness of the structure.

Original: While much future work has to be done, our work can still be an important step to bridge the gap between biological visual systems and state-of-the-art computer vision models.
Revised: While much future work has to be done, our work can still be an important step to bridge the gap between biological visual systems and state-of-the-art computer vision models.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Although recurrent network modules including LSTM (Hochreiter & Schmidhuber, 1997) and Gated
Revised: Although recurrent network modules including LSTM (Hochreiter & Schmidhuber, 1997) and Gated

Original: Recurrent Unit (Cho et al., 2014) have been widely used in temporal prediction (Wang et al., 2017c) and processing of sequential data (e.g. video classiﬁcation (Donahue et al., 2015)), few works have been done to augment feedforward CNNs with recurrent connections in image-based computer vision tasks.
Revised: Recurrent Unit (Cho et al., 2014) have been widely used in temporal prediction (Wang et al., 2017c) and processing of sequential data (e.g. video classiﬁcation (Donahue et al., 2015)), few works have been done to augment feedforward CNNs with recurrent connections in image-based computer vision tasks.

Original: Image classiﬁcation.
Revised: Image classiﬁcation.

Original: Standard deep CNNs for image classiﬁcation suffer from occlusion and noise (Wang et al., 2017a;b;
Revised: Standard deep CNNs for image classiﬁcation suffer from occlusion and noise (Wang et al., 2017a;b;

Original: Zhang et al., 2017), since heavy occlusion and noise severely corrupt feature representations at lower layers and therefore cause degradation of higher semantic layers.
Revised: Zhang et al., 2017), since heavy occlusion and noise severely corrupt feature representations at lower layers and therefore cause degradation of higher semantic layers.

Original: With the inclusion of feedback connections, a model can “rethink” or reﬁne its feature representations at lower layers using feedback information from higher layers (Li et al., 2018); after multiple rounds of feedback and reﬁnement, input signals from distracting objects (noise, irrelevant objects, etc.) will be suppressed in the ﬁnal feature representation (Cao et al., 2015).
Revised: With the inclusion of feedback connections, a model can “rethink” or reﬁne its feature representations at lower layers using feedback information from higher layers (Li et al., 2018); after multiple rounds of feedback and reﬁnement, input signals from distracting objects (noise, irrelevant objects, etc.) will be suppressed in the ﬁnal feature representation (Cao et al., 2015).

Original: Li et al.
Revised: Li et al.

Original: (2018) used the output posterior possibilities of a CNN to reﬁne its intermediate feature maps; however, their method requires posterior possibilities for reﬁnement and thus cannot be applied in scenarios where supervision is absent.
Revised: (2018) used the output posterior possibilities of a CNN to reﬁne its intermediate feature maps; however, their method requires posterior possibilities for reﬁnement and thus cannot be applied in scenarios where supervision is absent.

Original: Jetley et al.
Revised: Jetley et al.

Original: (2018) used more global and semantic features at higher convolutional layers to sharpen more local feature maps at lower layers for image classiﬁcation on CIFAR datasets; however, our own experimentation suggests that this method only works when the higher and lower layers have a relatively small semantic gap (similarly sized receptive ﬁelds); on highresolution dataset like ImageNet, large semantic gaps between higher and lower layers make this method difﬁcult to work.
Revised: (2018) used more global and semantic features at higher convolutional layers to sharpen more local feature maps at lower layers for image classiﬁcation on CIFAR datasets; however, our own experimentation suggests that this method only works when the higher and lower layers have a relatively small semantic gap (similarly sized receptive ﬁelds); on high- resolution dataset like ImageNet, large semantic gaps between higher and lower layers make this method difﬁcult to work.

Original: Other computer vision tasks.
Revised: Other computer vision tasks.

Original: Linsley et al.
Revised: Linsley et al.

Original: (2018) designed a model with explicit horizontal recurrent connections to solve contour detection problems, and Spoerer et al.
Revised: (2018) designed a model with explicit horizontal recurrent connections to solve contour detection problems, and Spoerer et al.

Original: (2017) evaluated the performance of various models with recurrent connections on digit recognition tasks under clutter.
Revised: (2017) evaluated the performance of various models with recurrent connections on digit recognition tasks under clutter.

Original: The tasks evaluated in these studies are rather simple and contrived, and it remains to be seen whether their models and conclusions can apply to real world computer vision problems.
Revised: The tasks evaluated in these studies are rather simple and contrived, and it remains to be seen whether their models and conclusions can apply to real world computer vision problems.

Revised: (Li et al., 2018) uses posterior possibilities at the last fully connected layer to select intermediate feature map representations, while the posterior possibility vector is not informative enough and the input of the feedback connection is totally ﬁxed, which makes it less ﬂexible to fully mimic the recurrent connections in the visual system.

Revised: Overall, feedback and recurrent connections are present in multiple layers of the visual hierarchy, and this study constrains feedback connections to the output classiﬁcation layer only.

Original: It is worth noting that a recent work (Nayebi et al., 2018) is motivated by recurrent connections in the brain as well; however, their work focuses on exploring the computational beneﬁts of local recurrent connections while ours focuses on feedback recurrent ones.
Revised: It is worth noting that a recent work (Nayebi et al., 2018) is motivated by recurrent connections in the brain as well; however, their work focuses on exploring the computational beneﬁts of local recurrent connections while ours focuses on feedback recurrent ones.

Original: Thus, we believe that our work is complementary to theirs.
Revised: Thus, we believe that our work is complementary to theirs.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we will describe the overall architecture of our proposed model and discuss some design details.
Revised: In this section, we will describe the overall architecture of our proposed model and discuss some design details.

Original: 3. O VERALL M ODEL A RCHITECTURE
Revised: 3. O VERALL M ODEL A RCHITECTURE

Original: The main structure of our Contextual Recurrent Convolutional Network (CRCN) is shown in Figure 1.
Revised: The main structure of our Contextual Recurrent Convolutional Network (CRCN) is shown in Figure 1.

Original: A CRCN model is a standard feedforward convolutional network augmented with feedback connections attached to some layers.
Revised: A CRCN model is a standard feedforward convolutional network augmented with feedback connections attached to some layers.

Original: At the end of each feedback connection, a contextual module fuses top-down and bottom-up information (dashed red lines in Figure 1) to provide reﬁned and sharpened input to the augmented layer.
Revised: At the end of each feedback connection, a contextual module fuses top-down and bottom-up information (dashed red lines in Figure 1) to provide reﬁned and sharpened input to the augmented layer.

Original: Given an input image, the model generates intermediate feature representations and output responses in multiple time steps.
Revised: Given an input image, the model generates intermediate feature representations and output responses in multiple time steps.

Original: At the ﬁrst time step ( t = 0 in Figure 1), the model passes the input through the feedforward route (black arrows in Figure 1) as in a standard CNN.
Revised: At the ﬁrst time step ( t = 0 in Figure 1), the model passes the input through the feedforward route (black arrows in Figure 1) as in a standard CNN.

Original: At later time steps ( t >in Figure 1), each contextual module fuses output representations of lower and higher layers at the previous step (dashed red lines in Figure 1) to generate the reﬁned input at the current time step (red lines in Figure 1).
Revised: At later time steps ( t >in Figure 1), each contextual module fuses output representations of lower and higher layers at the previous step (dashed red lines in Figure 1) to generate the reﬁned input at the current time step (red lines in Figure 1).

Original: Mathematically, we have
Revised: Mathematically, we have

Original: [Equation]
Revised: [Equation]

Original: where S G is the index set of layers augmented with feedback connections and contextual modules, c k ( · , · ) (detailed in Eqs.
Revised: where S G is the index set of layers augmented with feedback connections and contextual modules, c k ( · , · ) (detailed in Eqs.

Original: (2)) is the contextual module for layer k , O ( t ) k denotes the output of layer k at time t , h ( · ) is a function that maps the index of an augmented layer to that of its higher feedback  layer, and f k ( · ) denotes the (feedforward) operation to compute the output of layer k given some input.
Revised: (2)) is the contextual module for layer k , O ( t ) k denotes the output of layer k at time t , h ( · ) is a function that maps the index of an augmented layer to that of its higher feedback [Table] VGG-ATT means the model proposed in (Jetley et al., 2018), VGG-LR2 means the ”rethinking” one-FC-layer VGG model with 2 unrolling times proposed in (Li et al., 2018). CRCN- n means our 2-recurrentconnection model with n unrolling times layer, and f k ( · ) denotes the (feedforward) operation to compute the output of layer k given some input.

Original: 3. C ONTEXTUAL M ODULE D ESIGN
Revised: 3. C ONTEXTUAL M ODULE D ESIGN

Original: The key part of the Contextual Recurrent Convolutional Network model is the contextual module at the end of each feedback connection.
Revised: The key part of the Contextual Recurrent Convolutional Network model is the contextual module at the end of each feedback connection.

Original: Figure 3 shows one possible design of the contextual module,[Table] [Table] which is inspired by traditional RNN modules including LSTM (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Unit (Cho et al., 2014).
Revised: Figure 3 shows one possible design of the contextual module, which is inspired by traditional RNN modules including LSTM (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Unit (Cho et al., 2014).

Original: In this scheme, a contextual map is generated by the concatenation of the bottom-up and the (upsampled) top-down feature map passing through a 3 × 3 convolution (black circle with “C” and black arrows with circle).
Revised: In this scheme, a contextual map is generated by the concatenation of the bottom-up and the (upsampled) top-down feature map passing through a 3 × 3 convolution (black circle with “C” and black arrows with circle).

Original: Then a sigmoid function is applied to the map to generate a gate map .
Revised: Then a sigmoid function is applied to the map to generate a gate map .

Original: The gate map then controls the amount of contextual information that can go through by a point-wise multiplication (red lines).
Revised: The gate map then controls the amount of contextual information that can go through by a point-wise multiplication (red lines).

Original: To make the information ﬂow more stable, we add it with bottom-up feature map (black circle with “+”).
Revised: To make the information ﬂow more stable, we add it with bottom-up feature map (black circle with “+”).

Original: The equations are presented in Eqs.
Revised: The equations are presented in Eqs.

Original: Then we use this new feature representation to replace the old one and continue feedforward calculation as described in Section 3.1.
Revised: Then we use this new feature representation to replace the old one and continue feedforward calculation as described in Section 3.1.

Original: [Equation]
Revised: [Equation]

Original: 2a)

Original: [Equation]

Original: (2b)

Original: 3. L OCATION OF R ECURRENT C ONNECTIONS
Revised: 3. L OCATION OF R ECURRENT C ONNECTIONS

Original: Since there exists a gap between the semantic meanings of feature representations of bottom-up and top-down layers, we argue that recurrent connection across too many layers can do harm to the performance.
Revised: Since there exists a gap between the semantic meanings of feature representations of bottom-up and top-down layers, we argue that recurrent connection across too many layers can do harm to the performance.

Original: Therefore, we derive three sets of connections, conv3 2 to conv2 2 , conv4 2 to conv3 3 , and conv5 2 to conv4 3 respectively.
Revised: Therefore, we derive three sets of connections, conv3 2 to conv2 2 , conv4 2 to conv3 3 , and conv5 2 to conv4 3 respectively.

Original: It is worth noting that all these connections go across pooling layers, for pooling layers can greatly enlarge the receptive ﬁeld of neurons and enrich the contextual information of top-down information ﬂow.
Revised: It is worth noting that all these connections go across pooling layers, for pooling layers can greatly enlarge the receptive ﬁeld of neurons and enrich the contextual information of top-down information ﬂow.

Original: For information ﬂow in networks with multiple recurrent connections, take the network structure in Figure 2 as an example.
Revised: For information ﬂow in networks with multiple recurrent connections, take the network structure in Figure 2 as an example.

Original: The part between conv2 2 and conv5 2 will be unrolled for a certain number of times.
Revised: The part between conv2 2 and conv5 2 will be unrolled for a certain number of times.

Revised: To make the experiments setting consistent, we used model with two recurrent connections(loop1 + loop2) in all the tasks.

Original:  E XPERIMENTS AND A NALYSIS
Revised:  E XPERIMENTS AND A NALYSIS

Original: We ﬁrst tested the Contextual Recurrent Convolutional model on standard image classiﬁcation task including CIFAR-10, CIFAR-100, ImageNet and ﬁne-grained image classiﬁcation dataset CUB-200. 
Revised: We ﬁrst tested the Contextual Recurrent Convolutional model on standard image classiﬁcation task including CIFAR-10, CIFAR-100, ImageNet and ﬁne-grained image classiﬁcation dataset CUB-200. [Table] [Table]

Original: To display the robustness of our model, we showed its performance on noise image classiﬁcation, adversarial attack and occluded images.
Revised: To display the robustness of our model [Table] we showed its performance on noise image classiﬁcation, adversarial attack and occluded images.

Original: We found that our model achieved noticeable performance gain compared with standard feedforward model on all these tasks.
Revised: We found that our model achieved great performance gain compared with standard feedforward model on all these tasks.

Original: 4. S TANDARD I MAGE C LASSIFICATION
Revised: 4. S TANDARD I MAGE C LASSIFICATION

Original: CIFAR-10:
Revised: CIFAR-10:

Original: Because the size of images in CIFAR-10 and CIFAR-100 dataset is only 32px × 32px, the receptive ﬁelds of layers following conv3 2 already cover the entire image.
Revised: Because the size of images in CIFAR-10 and CIFAR-100 dataset is only 32px × 32px, the receptive ﬁelds of layers following conv3 2 already cover the entire image.

Original: Although the real power of contextual modulation is hindered by this limitation, our model can still beat the baseline
Revised: Although the real power of contextual modulation is hindered by this limitation, our model can still beat the baseline

Original: VGG16 network by a large margin (Second column in Table 1).
Revised: VGG16 network by a large margin (Second column in Table 1).

Original: Speciﬁcally, we used one recurrent connection model (loop 2 in Figure 2) for Cifar-10 and Cifar-100 task.
Revised: We also compared our model with another two models which show similar ideas with us.

Original:  CIFAR-100: Based on the assumption that contextual modulation can help layers capture more detailed information, we also tested our model on CIFAR-100 dataset, which has the same image collection as CIFAR-10 but the number of categories increases from 10 to 100.
Revised: Again, we can still show better results. CIFAR-100: Based on the assumption that contextual modulation can help layers capture more detailed information, we also tested our model on CIFAR-100 dataset, which has the same image collection as CIFAR-10 but the number of categories increases from 10 to 100.

Original: Our model got a larger improvement compared with feedforward model (The third column in Table.1). 
Revised: Our model got a larger improvement compared with feedforward and other models (The third column in Table.1). [Table] [Table] +2, 2+3 and 1+2+3.

Original: 4. N OISE I MAGE C LASSIFICATION AND A DVERSARIAL A TTACK
Revised: 4. N OISE I MAGE C LASSIFICATION AND A DVERSARIAL A TTACK

Original: ImageNet: ImageNet (Krizhevsky et al., 2012) is the commonly used large-scale image classiﬁcation dataset.
Revised: ImageNet: ImageNet (Krizhevsky et al., 2012) is the commonly used large-scale image classiﬁcation dataset.

Original: It contains 1.4 million images with 1000 categories.
Revised: It contains 1.4 million images with 1000 categories.

Original: In this task, to test the robustness of our model, we added different levels of Gaussian noise on the 224px × 224px images in the validation set and calculated the performance drop.
Revised: In this task, to test the robustness of our model, we added different levels of Gaussian noise on the 224px × 224px images in the validation set and calculated the performance drop.

Original: In detail, we used the two recurrent connection model for this task(loop1+loop2 in Figure 2).
Revised: In detail, we used the two recurrent connection model for this task(loop1+loop2 in Figure 2).

Original: Notice that all models are not trained on noise images.
Revised: Notice that all models are not trained on noise images.

Revised: The result of top1 error without any noise is shown in Table 7.

Original: We found that the performance gap between our model and feedforward VGG model got larger as the noise level increased.
Revised: We found that the performance gap between our model and feedforward VGG model got larger as the noise level increased.

Original: Results are shown in Figure 4.
Revised: Results are shown in Figure 4.

Original: Additionally, we also tested adversarial attacks on our model.
Revised: Additionally, we also tested adversarial attacks on our model.

Original: Figure 4 shows the results with different L ∞ norm coefﬁcient.
Revised: Figure 4 shows the results with different L ∞ norm coefﬁcient.

Original: We also found that our model had much lower fooling rates than feedforward model  with the increasing of the norms, which successfully proved our model’s robustness.
Revised: We also found that our model had much lower fooling rates than feedforward model and (Li et al., 2018)’s model with the increasing of the norms, which successfully proved our model’s robustness.

Original: 4. F INE - GRAINED I MAGE C LASSIFICATION
Revised: 4. F INE - GRAINED I MAGE C LASSIFICATION

Original: We argued that the contextual module can help the network to preserve more ﬁne-grained details in feature representations, and thus we tested our model on CUB-200 ﬁne-grained bird classiﬁcation dataset (Wah et al., 2011).
Revised: We argued that the contextual module can help the network to preserve more ﬁne-grained details in feature representations, and thus we tested our model on CUB-200 ﬁne-grained bird classiﬁcation dataset (Wah et al., 2011).

Original: We used the  ImageNet pretrained model and ﬁne-tuned it on the dataset.
Revised: We used the same model as ImageNet classiﬁcation task which indicates that our model contains two recurrent connection(loop1+loop2 in Figure 2).

Original: Our model can beat the strong ResNet-34 model (Zagoruyko & Komodakis, 2016)  with the same [Table] [Table]
Revised: As a result, our model can outperform much better than the feed-forward VGG model(Zagoruyko & Komodakis, 2016) and other similar models with the same experimental settings.

Original: to the feedback connection in Figure +3 and + +3.
Revised: The result is shown in 2.

Original: experimental settings.

Original: Our model can get a top-1 error of 25.1, while that of the ResNet-34 model is 26.5.

Original: 4. O CCLUDED I MAGE T ASK
Revised: 4. O CCLUDED I MAGE T ASK

Original: To further prove the robust ability of our model, we tested our model on VehicleOcclusion dataset (Wang et al., 2017b), which contains 4549 training images and 4507 testing images covering six types of vehicles, i.e., airplane, bicycle, bus, car, motorbike and train.
Revised: To further prove the robust ability of our model, we tested our model on VehicleOcclusion dataset (Wang et al., 2017b), which contains 4549 training images and 4507 testing images covering six types of vehicles, i.e., airplane, bicycle, bus, car, motorbike and train.

Original: For each test image in dataset,  some randomly-positioned occluders (irrelevant to the target object) are placed onto the target object, and make sure that the occlusion ratio of the target object is constrained.
Revised: For each test image in dataset, [Table] some randomly-positioned occluders (irrelevant to the target object) are placed onto the target object, and make sure that the occlusion ratio of the target object is constrained.

Original: One example is shown in
Revised: One example is shown in

Original: Figure 6.
Revised: Figure 6.

Original: In this task, we used multi-recurrent model which is similar with the model mentioned in Imagenet task.
Revised: In this task, we used multi-recurrent model which is similar with the model mentioned in Imagenet task.

Original: Here, we found that our model can achieve a huge improvement compared with feed-forward model from 49.3 to 65.5.
Revised: Here, we found that our model can achieve a huge improvement, which is shown in 3.

Original: 4. D ISCUSSION AND A NALYSIS4.5. L OCATION OF R ECURRENT C ONNECTIONS
Revised: 4. D ISCUSSION AND A NALYSIS

Revised: 4.5. L OCATION OF R ECURRENT C ONNECTIONS

Original: We implemented all the possible combinations of recurrent connections listed in Figure 2.
Revised: We implemented all the possible combinations of recurrent connections listed in Figure 2.

Original: We denote connection from conv3 2 to conv2 2 , conv4 2 to conv3 3 , and conv5 2 to conv4as Loop 1, Loop 2 and Loop 3, respectively.
Revised: We denote connection from conv3 2 to conv2 2 , conv4 2 to conv3 3 , and conv5 2 to conv4as Loop 1, Loop 2 and Loop 3, respectively.

Original: The same naming scheme goes for Loop 1+2 and
Revised: The same naming scheme goes for Loop 1+2 and

Original: Loop 1+2+3, etc.
Revised: Loop 1+2+3, etc.

Original: We tested altogether 6 different models on the noise classiﬁcation experiment, the settings of which were completely the same.
Revised: We tested altogether 6 different models on the noise classiﬁcation experiment, the settings of which were completely the same.

Original: In Table 3, by comparing the corresponding columns where one more recurrent connection is added, we can ﬁnd that having more loops yields better classiﬁcation accuracy and robustness, consistent with the reciprocal loops between successive layers in the hierarchical visual cortex.
Revised: In Table 5, by comparing the corresponding columns where one more recurrent connection is added, we can ﬁnd that having more loops yields better classiﬁcation accuracy and robustness, consistent with the reciprocal loops between successive layers in the hierarchical visual cortex.

Original: Especially, we can also ﬁnd that the importance of Loop 1 is slightly better than Loop 2 and Loop 3, indicating the early layers may beneﬁt more from the additional contextual information as an aid.[Table]
Revised: Especially, we can also ﬁnd that the importance of Loop 1 is slightly better than Loop 2 and Loop 3, indicating the early layers may beneﬁt more from the additional contextual information as an aid.

Original: 4.5. C ONTEXTUAL M ODULE S TRUCTURE
Revised: 4.5. C ONTEXTUAL M ODULE S TRUCTURE

Original: In additional to the original contextual module in Figure 3, we implemented three other structures that we thought were all reasonable, so as to further study the effect and importance of top-down information and contextual modulation.
Revised: In additional to the original contextual module in Figure 3, we implemented three other structures that we thought were all reasonable, so as to further study the effect and importance of top-down information and contextual modulation.

Original: Brieﬂy, we refer Module 1 to the scheme that top-down feature map gating contextual map, Module 2 to contextual map gating contextual map itself, Module 3 to the scheme that top-down feature map gating contextual map, as well as contextual map gating top-down feature map, and afterwards the two gating results are added together.
Revised: Brieﬂy, we refer Module 1 to the scheme that top-down feature map gating contextual map, Module 2 to contextual map gating contextual map itself, Module 3 to the scheme that top-down feature map gating contextual map, as well as contextual map gating top-down feature map, and afterwards the two gating results are added together.

Original: The ﬁnal output of all three modules are the gating output added by bottom-up feature map.
Revised: The ﬁnal output of all three modules are the gating output added by bottom-up feature map.

Original: By “contextual map”, we mean the concatenation of top-down and bottom-up feature map undergone a 3 × 3 convolution layer.
Revised: By “contextual map”, we mean the concatenation of top-down and bottom-up feature map undergone a 3 × 3 convolution layer.

Original: By “gating”, we mean the gated map element-wisely multiplied with the Sigmoid responses of the gate map.
Revised: By “gating”, we mean the gated map element-wisely multiplied with the Sigmoid responses of the gate map.

Original: For formulas and further details of the three module structures, we guide readers to read the supplementary materials.
Revised: For formulas and further details of the three module structures, we guide readers to read the supplementary materials.

Original: We did the same noise image classiﬁcation experiments on these different contextual modules to give a comparison.
Revised: We did the same noise image classiﬁcation experiments on these different contextual modules to give a comparison.

Original: We use the Loop 1+2 model as the remaining ﬁxed part.
Revised: We use the Loop 1+2 model as the remaining ﬁxed part.

Original: The performance of these modules are listed in Figure 2.
Revised: The performance of these modules are listed in Figure 4.

Original: The differences among these contextual modules lie in how the gate map is generated and what information is to be gated.
Revised: The differences among these contextual modules lie in how the gate map is generated and what information is to be gated.

Original: The best model is obtained by generating the gate map from contextual map and then use it to gate top-down information.
Revised: The best model is obtained by generating the gate map from contextual map and then use it to gate top-down information.

Original: By comparing it with Module 1, we ﬁnd that using only top-down information to generate the map and control total data ﬂow is not adequate, possibly because top-down information is too abstract and coarse.
Revised: By comparing it with Module 1, we ﬁnd that using only top-down information to generate the map and control total data ﬂow is not adequate, possibly because top-down information is too abstract and coarse.

Original: By comparing the best module with Module 2, we ﬁnd that only top-down information is necessary to be gated.
Revised: By comparing the best module with Module 2, we ﬁnd that only top-down information is necessary to be gated.

Original: A direct addition of bottom-up map with the output of the gate is adequate to keep all the details in lower level feature maps.
Revised: A direct addition of bottom-up map with the output of the gate is adequate to keep all the details in lower level feature maps.

Original: 4.5. F EATURE A NALYSIS
Revised: 4.5. F EATURE A NALYSIS

Original: We drew t-SNE visualization of feature representations of both ﬁnal fully connected layers and layers with recurrent connections attached (e.g. conv2 2 , conv3 3 , conv4 3 ).
Revised: We drew t-SNE visualization of feature representations of both ﬁnal fully connected layers and layers with recurrent connections attached (e.g. conv2 2 , conv3 3 , conv4 3 ).

Original: We selectedout of 1000 categories from ImageNet validation set.
Revised: We selectedout of 1000 categories from ImageNet validation set.

Original: To effectively capture the changes of feature representations of intermediate convolutional layers, we used ImageNet bounding box annotations and did an average pooling of all the feature responses corresponding to the object bounding box.
Revised: To effectively capture the changes of feature representations of intermediate convolutional layers, we used ImageNet bounding box annotations and did an average pooling of all the feature responses corresponding to the object bounding box.

Original: By comparing the representations of both networks, we can ﬁnd that the Contextual Recurrent Network is able to form a more distinct clustering than VGG16 network.
Revised: By comparing the representations of both networks, we can ﬁnd that the Contextual Recurrent Network is able to form a more distinct clustering than VGG16 network.

Original: Notice that we also tested the presentation when a high noise (standard deviation equal to 30) is added to the images.
Revised: Notice that we also tested the presentation when a high noise (standard deviation equal to 30) is added to the images.

Original: We can ﬁnd a consistent improvement over VGG16 network in both intermediate representations and representations directly linked to the ﬁnal classiﬁcation task.
Revised: We can ﬁnd a consistent improvement over VGG16 network in both intermediate representations and representations directly linked to the ﬁnal classiﬁcation task.

Original: 4.5. U NROLLING P ROCESS
Revised: 4.5. U NROLLING P ROCESS

Original: There is another ﬁnding that the contextual module dynamics in recurrent connections not only helps to reﬁne the low-level feature representation during inference, it can also reﬁne the feedforward weights, resulting in better performance in computer vision tasks even in the ﬁrst iteration, acting as a regularizer.
Revised: There is another ﬁnding that the contextual module dynamics in recurrent connections not only helps to reﬁne the low-level feature representation during inference, it can also reﬁne the feedforward weights, resulting in better performance in computer vision tasks even in the ﬁrst iteration, acting as a regularizer.

Original: The results are shown in Figure 4.
Revised: The results are shown in Table 6.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed a novel Contextual Recurrent Convolutional Network.
Revised: In this paper, we proposed a novel Contextual Recurrent Convolutional Network.

Original: Based on the recurrent connections between layers in the hierarchy of a feedforward deep convolutional neural network, the new network can show some robust properties in some computer vision tasks compared with its feedforward baseline.
Revised: Based on the recurrent connections between layers in the hierarchy of a feedforward deep convolutional neural network, the new network can show some robust properties in some computer vision tasks compared with its feedforward baseline.

Original: Moreover, the network shares many common properties with biological visual system.
Revised: Moreover, the network shares many common properties with biological visual system.

Original: We hope this work will not only shed light on the effectiveness of recurrent connections in robust learning and general computer vision tasks, but also give people some inspirations to bridge the gap between computer vision models and real biological visual system.
Revised: We hope this work will not only shed light on the effectiveness of recurrent connections in robust learning and general computer vision tasks, but also give people some inspirations to bridge the gap between computer vision models and real biological visual system.

Original:  S UPPLEMENTARY M ATERIALS 6. D ETAILS OF DIFFERENT CONTEXTUAL MODULES
Revised:  S UPPLEMENTARY M ATERIALS 6. D ETAILS OF DIFFERENT CONTEXTUAL MODULES

Original: We tested three other possible contextual modules in Section 4.
Revised: We tested three other possible contextual modules in Section 4.

Original: Here are the detailed formulations of the three modules.
Revised: Here are the detailed formulations of the three modules.

Original: [Equation]
Revised: [Equation]

Revised: (3a)

Revised: [Equation]

Revised: (3b)

Revised: [Equation]

Revised: [Equation]

Revised: (3c)

Original: 5d
Revised: [Equation] (4a) gate = Sigmoid(Conv [Equation] (Concat(Upsample( O ( t

Original: [Equation]
Revised: [Equation]

Revised: 1) h ( k ) )

Revised: [Equation]

Revised: O ( t

Revised: [Equation]

Revised: 1) k − 1 ))) (4b) contextual =

Original: In the module described by Eqs.
Revised: Conv [Equation] [Equation] (Upsample(

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: 1 )) (4c)

Revised: [Equation]

Revised: [Equation]

Revised: contextual

Revised: [Equation]

Revised: [Equation]

Revised: ( t

Revised: [Equation]

Revised: 1) k (5a) gate = Sigmoid(Conv

Revised: [Equation]

Revised: (Upsample( O ( t

Revised: [Equation]

Revised: [Equation]

Revised: ( k ) )))

Revised: (5b) contextual =

Revised: Conv

Revised: [Equation]

Revised: (Concat(Upsample( O ( t − 1) h

Revised: ( k ) )

Revised: [Equation]

Revised: O ( t − 1) k )) (5c) gate contextual =

Revised: Sigmoid( contextual ) (5d)

Revised: In the module described by Eqs.

Original: (3), we ﬁrst generated the gate by the top-down layer.
Revised: (3), we ﬁrst generated the gate by the top-down layer.

Original: Then we used the gate to control the contextual information generated by concatenating bottom-up layer and top-down layer.
Revised: Then we used the gate to control the contextual information generated by concatenating bottom-up layer and top-down layer.

Original: To stable the information ﬂow, we added it with the bottom-up layer.
Revised: To stable the information ﬂow, we added it with the bottom-up layer.

Original: In the module described by Eqs.
Revised: In the module described by Eqs.

Original: (4), we ﬁrst generated the gate by contextual information which is the same as our proposed module.
Revised: (4), we ﬁrst generated the gate by contextual information which is the same as our proposed module.

Original: Then we used the gate to control the contextual information itself which we thought was a feasible way to store the largest information.
Revised: Then we used the gate to control the contextual information itself which we thought was a feasible way to store the largest information.

Original: To stable the information ﬂow, we also added it with the bottom-up layer.
Revised: To stable the information ﬂow, we also added it with the bottom-up layer.

Original: We generated two gates by both contextual information and top-down layer in the module described by Eqs.
Revised: We generated two gates by both contextual information and top-down layer in the module described by Eqs.

Original: Then we used the gate contextual to control the top-down information and used the gate to control the contextual information.
Revised: Then we used the gate contextual to control the top-down information and used the gate to control the contextual information.

Original: To stable the information ﬂow, we also added it with the bottom-up layer.
Revised: To stable the information ﬂow, we also added it with the bottom-up layer.

Original: 6. I MAGE E XAMPLES OF D IFFERENT T ASKS
Revised: 6. I MAGE E XAMPLES OF D IFFERENT T ASKS

Original: In this section, we showed some examples of image occlusion task and adversarial noise task.
Revised: In this section, we showed some examples of image occlusion task and adversarial noise task.

Original: In the left of Figure 6, we showed one image occlusion example.
Revised: In the left of Figure 6, we showed one image occlusion example.

Original: And we showed one adversarial noise example in the right of Figure 6.
Revised: And we showed one adversarial noise example in the right of Figure 6.

Revised: 6. I MAGE N ET T OP 1 A CCURACY

Revised: In Table 7, we showed the Imagenet Top1 accuracy results.

Revised: Notice that we did not compare our model with VGG-ATT model proposed in (Jetley et al., 2018) because their model is not reasonable on high resolution image dataset.

Revised: Therefore, their model cannot extract effective attention map from the ImageNet images.