Original: L ATERAL I NHIBITION - INSPIRED STRUCTURE FOR C ONVOLUTIONAL N EURAL N ETWORK ON I MAGE C LASSIFICATION
Revised: L ATERAL I NHIBITION - INSPIRED STRUCTURE FOR C ONVOLUTIONAL N EURAL N ETWORK ON I MAGE C LASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Convolutional neural networks (CNNs) have become powerful and popular tools since deep learning emerged for image classification in the computer vision field.
Revised: Convolutional neural networks (CNNs) have become powerful and popular tools since deep learning emerged for image classification in the computer vision field.

Original: For better recognition, the dimension of both depth and width has been explored, leading to convolutional neural networks with more layers and channels.
Revised: For better recognition, the dimension of both depth and width has been explored, leading to convolutional neural networks with more layers and channels.

Original: In addition to these factors, neurobiology suggests lateral inhibition (lateral antagonism, e.g. Mach band effect), a widely existing phenomenon for vision that increases the contrast and sharpness of nearby neuron excitation in the lateral direction to help recognition.
Revised: In addition to these factors, neurobiology suggests lateral inhibition (lateral antagonism, e.g. Mach band effect), a widely existing phenomenon for vision that increases the contrast and sharpness of nearby neuron excitation in the lateral direction to help recognition.

Original: However, such mechanism has not been well explored in the design of convolutional neural network.
Revised: However, such mechanism has not been well explored in the design of convolutional neural network.

Original: In this paper, we explicitly explore the filter dimension in the lateral direction and propose our lateral inhibition-inspired (LI) structure.
Revised: In this paper, we explicitly explore the filter dimension in the lateral direction and propose our lateral inhibition-inspired (LI) structure.

Original: Our naive design uses the low-pass filter to mimic the strength decay of lateral interaction from neighbors regarding the distance.
Revised: Our naive design uses the low-pass filter to mimic the strength decay of lateral interaction from neighbors regarding the distance.

Original: One learnable parameter per channel is applied to set the amplitude of the low-pass filter by multiplication, which is flexible to model various lateral interactions (including lateral inhibition).
Revised: One learnable parameter per channel is applied to set the amplitude of the low-pass filter by multiplication, which is flexible to model various lateral interactions (including lateral inhibition).

Original: The convolution result is then subtracted from the input, which could increase the contrast and sharpness for better recognition.
Revised: The convolution result is then subtracted from the input, which could increase the contrast and sharpness for better recognition.

Original: Furthermore, a learnable scaling factor and shift are applied to adjust the value after subtraction.
Revised: Furthermore, a learnable scaling factor and shift are applied to adjust the value after subtraction.

Original: Our lateral inhibition-inspired (LI) structure works on both plain convolution and the convolutional block with residual connection, while being compatible with the existing modules.
Revised: Our lateral inhibition-inspired (LI) structure works on both plain convolution and the convolutional block with residual connection, while being compatible with the existing modules.

Original: Preliminary results demonstrate obvious improvements on the
Revised: Preliminary results demonstrate obvious improvements on the

Original: ImageNet dataset for AlexNet (7.58%) and ResNet-18 (0.81%), respectively, with little increase in parameters, indicating the effectiveness of our brain-similar design to help feature learning for image classification from a different perspective.
Revised: ImageNet dataset for AlexNet (7.58%) and ResNet-18 (0.81%), respectively, with little increase in parameters, indicating the effectiveness of our brain-similar design to help feature learning for image classification from a different perspective.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, convolutional neural networks (CNNs) (Hinton et al., 2012; Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016) have become powerful and popular tools since deep learning emerged for image classification in the computer vision field.
Revised: In recent years, convolutional neural networks (CNNs) (Hinton et al., 2012; Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016) have become powerful and popular tools since deep learning emerged for image classification in the computer vision field.

Original: They have recorded record-breaking performance and outperformed traditional methods (Quinlan, 1986; Cortes & Vapnik, 1995) with hand-crafted features (Lowe, 1999; Dalal & Triggs, 2005) on the ImageNet Large
Revised: They have recorded record-breaking performance and outperformed traditional methods (Quinlan, 1986; Cortes & Vapnik, 1995) with hand-crafted features (Lowe, 1999; Dalal & Triggs, 2005) on the ImageNet Large

Original: Scale Visual Recognition Challenge (ILSVRC) (Deng et al., 2009).
Revised: Scale Visual Recognition Challenge (ILSVRC) (Deng et al., 2009).

Original: Today, convolutional neural networks still possess unique merits, as they have been studied the most, along with the fact that convolution has a strong connection with the human vision system and image processing, making them good models for feature learning research.
Revised: Today, convolutional neural networks still possess unique merits, as they have been studied the most, along with the fact that convolution has a strong connection with the human vision system and image processing, making them good models for feature learning research.

Original: Different factors have been explored to improve recognition performance of convolutional neural networks.
Revised: Different factors have been explored to improve recognition performance of convolutional neural networks.

Original: VGGNet (Simonyan & Zisserman, 2015) applies a small convolution kernel size ( 3 × 3 ) for increased network depth, while ResNet (He et al., 2016) introduces deep residual learning to make training very deep deep networks feasible.
Revised: VGGNet (Simonyan & Zisserman, 2015) applies a small convolution kernel size ( 3 × 3 ) for increased network depth, while ResNet (He et al., 2016) introduces deep residual learning to make training very deep deep networks feasible.

Original: The success of such networks indicates that depth is a crucial factor for recognition performance.
Revised: The success of such networks indicates that depth is a crucial factor for recognition performance.

Original: Wide Residual Networks (Zagoruyko & Komodakis, 2016), on the other hand, demonstrate width as another important factor to improved performance.
Revised: Wide Residual Networks (Zagoruyko & Komodakis, 2016), on the other hand, demonstrate width as another important factor to improved performance.

Original: In addition to these factors, neurobiology suggests the widely existing lateral inhibition (lateral antagonism, e.g. Mach band effect, shown in Fig. 1), a phenomenon that increases the contrast and sharpness of nearby neuron excitation in the lateral direction, also important to help feature learning.
Revised: In addition to these factors, neurobiology suggests the widely existing lateral inhibition (lateral antagonism, e.g. Mach band effect, shown in Fig. 1), a phenomenon that increases the contrast and sharpness of nearby neuron excitation in the lateral direction, also important to help feature learning.

Original: However, such an important mechanism has not been well explored in convolutional neural network design.
Revised: However, such an important mechanism has not been well explored in convolutional neural network design.

Original: In recent years, Yang et al.
Revised: In recent years, Yang et al.

Original: (2015) propose a traditional method using manual doubleopponent design (antagonism) with spatial sparseness constraint for boundary detection, following the color-opponent mechanism in the human vision system, to highlight the color boundary while suppressing texture edges.
Revised: (2015) propose a traditional method using manual doubleopponent design (antagonism) with spatial sparseness constraint for boundary detection, following the color-opponent mechanism in the human vision system, to highlight the color boundary while suppressing texture edges.

Original: This double-opponent design is further extended (Gao et al., 2015) for color constancy.
Revised: This double-opponent design is further extended (Gao et al., 2015) for color constancy.

Original: Cao et al.
Revised: Cao et al.

Original: (2018) apply lateral inhibition mechanism in the inference stage of the pre-trained VGGNet (Simonyan & Zisserman, 2015) for visual attention map and saliency detection.
Revised: (2018) apply lateral inhibition mechanism in the inference stage of the pre-trained VGGNet (Simonyan & Zisserman, 2015) for visual attention map and saliency detection.

Original: Hasani et al.
Revised: Hasani et al.

Original: (2019) propose their surround modulation design, using a manually defined kernel from the difference of Gaussian (DoG) function, to explore the feasibility of incorporating the lateral inhibition mechanism.
Revised: (2019) propose their surround modulation design, using a manually defined kernel from the difference of Gaussian (DoG) function, to explore the feasibility of incorporating the lateral inhibition mechanism.

Original: This fixed design is only applied to half channels of the initial layer (with better performance than applying to the full channels).
Revised: This fixed design is only applied to half channels of the initial layer (with better performance than applying to the full channels).

Original: It is still worth exploring flexible lateral inhibition-inspired design for convolutional neural network, to incorporate such mechanism and make it more similar to the brain in the training stage for image classification.
Revised: It is still worth exploring flexible lateral inhibition-inspired design for convolutional neural network, to incorporate such mechanism and make it more similar to the brain in the training stage for image classification.

Original: In this paper, inspired by the findings of neurobiology, we explicitly explore the filter dimension in the lateral direction and propose our lateral inhibition-inspired (LI) structure, which incorporates the lateral inhibition mechanism in a flexible manner inside modern convolutional neural networks.
Revised: In this paper, inspired by the findings of neurobiology, we explicitly explore the filter dimension in the lateral direction and propose our lateral inhibition-inspired (LI) structure, which incorporates the lateral inhibition mechanism in a flexible manner inside modern convolutional neural networks.

Original: Our simple naive design applies the low-pass filter with the central weight eliminated, to mimic the decay of inhibition strength from neighbors regarding the distance.
Revised: Our simple naive design applies the low-pass filter with the central weight eliminated, to mimic the decay of inhibition strength from neighbors regarding the distance.

Original: To set the amplitude of the low-pass filter by multiplication, a learnable parameter per channel is used, for the flexibility of modeling various lateral interactions (including lateral inhibition).
Revised: To set the amplitude of the low-pass filter by multiplication, a learnable parameter per channel is used, for the flexibility of modeling various lateral interactions (including lateral inhibition).

Original: The convolution result is then subtracted from the input, which could increase the contrast and sharpness for better recognition.
Revised: The convolution result is then subtracted from the input, which could increase the contrast and sharpness for better recognition.

Original: Furthermore, a learnable scaling factor and shift are applied to adjust the value after subtraction.
Revised: Furthermore, a learnable scaling factor and shift are applied to adjust the value after subtraction.

Original: Our lateral inhibition-inspired (LI) method works on both plain convolution and convolutional block with a residual connection, while being compatible with the existing modules.
Revised: Our lateral inhibition-inspired (LI) method works on both plain convolution and convolutional block with a residual connection, while being compatible with the existing modules.

Original: Preliminary results of our simple naive design demonstrate obvious improvements on the ImageNet dataset for AlexNet (7.58%), ResNet-18 (0.81%), respectively, with little parameter increase, indicating the effectiveness of our brain-similar to help feature learning for image classification from a different perspective.
Revised: Preliminary results of our simple naive design demonstrate obvious improvements on the ImageNet dataset for AlexNet (7.58%), ResNet-18 (0.81%), respectively, with little parameter increase, indicating the effectiveness of our brain-similar to help feature learning for image classification from a different perspective.

Original: Our main contributions can be summarized as:
Revised: Our main contributions can be summarized as:

Original: • We propose a lateral inhibition-inspired structure for modern convolutional neural network design, which could participate in the training stage for image classification.
Revised: • We propose a lateral inhibition-inspired structure for modern convolutional neural network design, which could participate in the training stage for image classification.

Original: • Our design is flexible compared to the fixed difference of Gaussian (DoG) filter, and could be applied to all layers (and channels) with competitive performance.
Revised: • Our design is flexible compared to the fixed difference of Gaussian (DoG) filter, and could be applied to all layers (and channels) with competitive performance.

Original: • We are the few to explicitly model lateral inhibition as well as other lateral interaction to make the network more brain-similar, given the flexible weight design.
Revised: • We are the few to explicitly model lateral inhibition as well as other lateral interaction to make the network more brain-similar, given the flexible weight design.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neurobiology.
Revised: Neurobiology.

Original: Lateral inhibition is common in sensory system.
Revised: Lateral inhibition is common in sensory system.

Original: Mach reports the Mach bands effect RATLIFF (1965) due to lateral inhibition that the human vision system perceives increased contrast and sharpness along the boundary, where the dark area becomes darker and the bright area becomes brighter.
Revised: Mach reports the Mach bands effect RATLIFF (1965) due to lateral inhibition that the human vision system perceives increased contrast and sharpness along the boundary, where the dark area becomes darker and the bright area becomes brighter.

Original: Blakemore et al.
Revised: Blakemore et al.

Original: (1970) discover that lateral inhibition plays an important role in orientation detection within the human vision system.
Revised: (1970) discover that lateral inhibition plays an important role in orientation detection within the human vision system.

Original: It helps to remove redundancy for the existence of inhibitory signals, as large stimulus covering a whole receptive field would be relatively ineffective in excitation of neurons.
Revised: It helps to remove redundancy for the existence of inhibitory signals, as large stimulus covering a whole receptive field would be relatively ineffective in excitation of neurons.

Original: Traditional method.
Revised: Traditional method.

Original: Yang et al.
Revised: Yang et al.

Original: (2015) propose their double-opponent design with spatial sparseness constraint for boundary detection, by mimicking the color-opponent mechanism in the human vision system.
Revised: (2015) propose their double-opponent design with spatial sparseness constraint for boundary detection, by mimicking the color-opponent mechanism in the human vision system.

Original: The single opponent uses a manual antagonism such as red-green and blue-yellow, while the double opponent combines two opposite single opponent in either center-surround or sideby-side manner for increased contrast.
Revised: The single opponent uses a manual antagonism such as red-green and blue-yellow, while the double opponent combines two opposite single opponent in either center-surround or sideby-side manner for increased contrast.

Original: This design highlights the color boundary while suppressing texture edges.
Revised: This design highlights the color boundary while suppressing texture edges.

Original: The double-opponent design is further extended (Gao et al., 2015) for color constancy.
Revised: The double-opponent design is further extended (Gao et al., 2015) for color constancy.

Original: Convolutional Neural Network.
Revised: Convolutional Neural Network.

Original: Local Response Normalization (LRN) (Hinton et al., 2012) is introduced in AlexNetalong and is not trainable.
Revised: Local Response Normalization (LRN) (Hinton et al., 2012) is introduced in AlexNetalong and is not trainable.

Original: Normalization is performed for the value at each position (x, y) considering neighbors at the same position from nearby channels, with the denominator as a sum of the squared neighbors, given a scaling factor, bias, as well as the exponential value for the denominator.
Revised: Normalization is performed for the value at each position (x, y) considering neighbors at the same position from nearby channels, with the denominator as a sum of the squared neighbors, given a scaling factor, bias, as well as the exponential value for the denominator.

Original: It applies lateral inhibition across nearby channels via competition.
Revised: It applies lateral inhibition across nearby channels via competition.

Original: Batch normalization (Ioffe & Szegedy, 2015) applies normalization within individual channels, using an mini-batch mean (shift) and variance, then a scaling factor and re-shift parameter, to constrain all values in a certain range.
Revised: Batch normalization (Ioffe & Szegedy, 2015) applies normalization within individual channels, using an mini-batch mean (shift) and variance, then a scaling factor and re-shift parameter, to constrain all values in a certain range.

Original: Layer normalization (Ba et al., 2016) only consider one single instance, but performs normalization across all channels using the average and variance from each channel.
Revised: Layer normalization (Ba et al., 2016) only consider one single instance, but performs normalization across all channels using the average and variance from each channel.

Original: Group normalization (Wu & He, 2018) further extends layer normalization by dividing channels into groups for consideration.
Revised: Group normalization (Wu & He, 2018) further extends layer normalization by dividing channels into groups for consideration.

Original: Besides normalization methods, Cao et al.
Revised: Besides normalization methods, Cao et al.

Original: (2018) propose their lateral inhibition design using pretrained VGGNet, as top-down step for attention map and weakly supervised saliency detection.
Revised: (2018) propose their lateral inhibition design using pretrained VGGNet, as top-down step for attention map and weakly supervised saliency detection.

Original: The inhibition value consists of an average term from the neighbors within the same channel to protect high response zone, and a differential term only consider the difference from higher activation nearby for inhibition, with weight decay regarding the distance.
Revised: The inhibition value consists of an average term from the neighbors within the same channel to protect high response zone, and a differential term only consider the difference from higher activation nearby for inhibition, with weight decay regarding the distance.

Original: The whole formula including term coefficients are intuitively decided and unlearnable, and only happens in inference stage.
Revised: The whole formula including term coefficients are intuitively decided and unlearnable, and only happens in inference stage.

Original: Hasani et al.
Revised: Hasani et al.

Original: propose surround modulation for lateral inhibition, to make network closer to the brain.
Revised: propose surround modulation for lateral inhibition, to make network closer to the brain.

Original: It applies two gaussian filters (5x5) with 1:1 weight (fixed) from the difference of Gaussian (DoG) function.
Revised: It applies two gaussian filters (5x5) with 1:1 weight (fixed) from the difference of Gaussian (DoG) function.

Original: However, this fixed design lacks flexibility, which is only applied to half channels of the initial layer within a small convolutional neural network, due to the performance drop when applying to full channels or the following layer.
Revised: However, this fixed design lacks flexibility, which is only applied to half channels of the initial layer within a small convolutional neural network, due to the performance drop when applying to full channels or the following layer.

Original: The channelwise normalization (center pool) in Weighted normalization (Pan et al., 2021) improves LRN (Hinton et al., 2012) from AlexNet by adding extra parameters such as a scaling factor and bias, while further adding surround pool for normalization within each channel does not help on ImageNet dataset.
Revised: The channelwise normalization (center pool) in Weighted normalization (Pan et al., 2021) improves LRN (Hinton et al., 2012) from AlexNet by adding extra parameters such as a scaling factor and bias, while further adding surround pool for normalization within each channel does not help on ImageNet dataset.

Original: DivisiveNorm (Miller et al., 2021) also similarly follows the design of LRN at each location along the channel dimension, but changes the numerator to be squared, and adds a weight scaling factor for each unnormalized activations inside the exponentially weighted sum.
Revised: DivisiveNorm (Miller et al., 2021) also similarly follows the design of LRN at each location along the channel dimension, but changes the numerator to be squared, and adds a weight scaling factor for each unnormalized activations inside the exponentially weighted sum.

Original: The hyper parameters are manually set for stable training.
Revised: The hyper parameters are manually set for stable training.

Original:  M ETHOD
Revised:  M ETHOD

Original: Our lateral inhibition-inspired (LI) structure explicitly incorporates the lateral inhibition mechanism into the design of convolutional neural network, to increase contrast and sharpness for more accurate perception.
Revised: Our lateral inhibition-inspired (LI) structure explicitly incorporates the lateral inhibition mechanism into the design of convolutional neural network, to increase contrast and sharpness for more accurate perception.

Original: Unlike methods only involving in the post-processing stage with the original pre-trained model unchanged, our method participates in the training stage.
Revised: Unlike methods only involving in the post-processing stage with the original pre-trained model unchanged, our method participates in the training stage.

Original: Our lateral inhibition-inspired (LI) design takes two steps: first, it generates the lateral interaction value based on the input within the spatial neighborhood of each channel; then it subtracts that value from the original input.
Revised: Our lateral inhibition-inspired (LI) design takes two steps: first, it generates the lateral interaction value based on the input within the spatial neighborhood of each channel; then it subtracts that value from the original input.

Original: Details will be explained in the following parts.
Revised: Details will be explained in the following parts.

Original: 3. L ATERAL I NTERACTION D ESIGN
Revised: 3. L ATERAL I NTERACTION D ESIGN

Original: We design our naive lateral interaction based on neurobiological findings.
Revised: We design our naive lateral interaction based on neurobiological findings.

Original: It follows the principle that the strength of interaction (including inhibition) from an active neuron decreases as the distance to that neuron increases in the lateral direction.
Revised: It follows the principle that the strength of interaction (including inhibition) from an active neuron decreases as the distance to that neuron increases in the lateral direction.

Original: For implementation, we take the exsiting Gaussian low-pass filter with central weight eliminated to mimic the strength decay from neighbors.
Revised: For implementation, we take the exsiting Gaussian low-pass filter with central weight eliminated to mimic the strength decay from neighbors.

Original: Here, we denote the function of lateral interaction as LI , Gaussian low-pass filter as G , the central weight value of the low-pass filter G as C , convolution operation as Conv , the learnable amplitude weights as W (with one parameter per channel for multiplication to separately decide the channelwise amplitude of Gaussian low-pass filter G ), and the input features as x for our lateral inhibition-inspired structure.
Revised: Here, we denote the function of lateral interaction as LI , Gaussian low-pass filter as G , the central weight value of the low-pass filter G as C , convolution operation as Conv , the learnable amplitude weights as W (with one parameter per channel for multiplication to separately decide the channelwise amplitude of Gaussian low-pass filter G ), and the input features as x for our lateral inhibition-inspired structure.

Original: The weights of the 2D Gaussian low-pass filter ( G ) are calculated using Equation 1.
Revised: The weights of the 2D Gaussian low-pass filter ( G ) are calculated using Equation 1.

Original: Here u and v denote the spatial coordinates regarding to the center of the Gaussian low-pass filter, and σ denotes the standard deviation for weight decay speed from the center (we set filter size as 3 and σ as 1):
Revised: Here u and v denote the spatial coordinates regarding to the center of the Gaussian low-pass filter, and σ denotes the standard deviation for weight decay speed from the center (we set filter size as 3 and σ as 1):

Original: [Equation]
Revised: [Equation]

Original: The lateral interaction function given the input feature x at channel i is defined in Equation 2.
Revised: The lateral interaction function given the input feature x at channel i is defined in Equation 2.

Original: First, convolution on input feature using Gaussian low-pass filter G is applied, then the corresponding convolution result form the central weight value C (of the low-pass filter G ) is subtracted, to only have interaction from the neighbors.
Revised: First, convolution on input feature using Gaussian low-pass filter G is applied, then the corresponding convolution result form the central weight value C (of the low-pass filter G ) is subtracted, to only have interaction from the neighbors.

Original: Finally, multiplication is performed by the amplitude parameter (from the learnable weights W ) to decide the amplitude of the Gaussian low-pass filter G per channel.
Revised: Finally, multiplication is performed by the amplitude parameter (from the learnable weights W ) to decide the amplitude of the Gaussian low-pass filter G per channel.

Original: Different from Surround Modulation (Hasani et al., 2019), which applies 1:1 weight using two
Revised: Different from Surround Modulation (Hasani et al., 2019), which applies 1:1 weight using two

Original: Gaussian low-pass filters in the difference of Gaussian (DoG) function to model lateral inhibition, our learnable weights W is more flexible (could be zero, positive or negative) to fit various lateral interactions (for none, lateral inhibition or even the opposite when necessary).
Revised: Gaussian low-pass filters in the difference of Gaussian (DoG) function to model lateral inhibition, our learnable weights W is more flexible (could be zero, positive or negative) to fit various lateral interactions (for none, lateral inhibition or even the opposite when necessary).

Original: [Equation]
Revised: [Equation]

Original: 3. A PPLYING L ATERAL I NTERACTION
Revised: 3. A PPLYING L ATERAL I NTERACTION

Original: We denote the output function as O for applying lateral interaction on input feature x at channel i .
Revised: We denote the output function as O for applying lateral interaction on input feature x at channel i .

Original: Then the output function is defined in Equation 3, with lateral interaction result subtracted.
Revised: Then the output function is defined in Equation 3, with lateral interaction result subtracted.

Original: [Equation]
Revised: [Equation]

Original: 3. A DDITIONAL PARAMETERS
Revised: 3. A DDITIONAL PARAMETERS

Original: Additional parameters are applied since the interaction result will be subtracted from the original feature map.
Revised: Additional parameters are applied since the interaction result will be subtracted from the original feature map.

Original: Parameters include a scaling factor ( v ), shift ( m ), and bias ( b ).
Revised: Parameters include a scaling factor ( v ), shift ( m ), and bias ( b ).

Original: Combination of these parameters can be explored to find the best setting.
Revised: Combination of these parameters can be explored to find the best setting.

Original: [Equation]
Revised: [Equation]

Original: 3. S TRUCTURE P LACE
Revised: 3. S TRUCTURE P LACE

Original: Our lateral inhibition-inspired (LI) structure works on both the plain convolution and the convolutional block with residual connection.
Revised: Our lateral inhibition-inspired (LI) structure works on both the plain convolution and the convolutional block with residual connection.

Original: For the plain convolution, it is placed after that convolution (and the activation function), as shown in Fig.
Revised: For the plain convolution, it is placed after that convolution (and the activation function), as shown in Fig.

Original: For the convolutional block with a residual connection, it is placed inside the residual block before the first convolution, as shown in Fig.
Revised: For the convolutional block with a residual connection, it is placed inside the residual block before the first convolution, as shown in Fig.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Dataset.
Revised: Dataset.

Original: We use the ImageNet-1k data set (Russakovsky et al., 2015) to evaluate our proposed methods.
Revised: We use the ImageNet-1k data set (Russakovsky et al., 2015) to evaluate our proposed methods.

Original: ImageNet-1k dataset consists of 1k object categories, with 1.2 million images for training and 50k images for validation.
Revised: ImageNet-1k dataset consists of 1k object categories, with 1.2 million images for training and 50k images for validation.

Original: Experiment Settings.
Revised: Experiment Settings.

Original: We use the Pytorch (Paszke et al., 2019) deep learning framework and follow the official training example.
Revised: We use the Pytorch (Paszke et al., 2019) deep learning framework and follow the official training example.

Original: We adopt the standard settings: image crop size of 224 × 224, initial learning rate of 0.01 for AlexNet (Hinton et al., 2012) (which decays every 30 epochs by a factor of 10), momentum of 0.9, and weight decay of 0.0001.
Revised: We adopt the standard settings: image crop size of 224 × 224, initial learning rate of 0.01 for AlexNet (Hinton et al., 2012) (which decays every 30 epochs by a factor of 10), momentum of 0.9, and weight decay of 0.0001.

Original: We use standard data augmentation, which includes random horizontal flipping, random scale and ratio changes in the range of [0.08, 1.0] and [0.75, 1.33], respectively.
Revised: We use standard data augmentation, which includes random horizontal flipping, random scale and ratio changes in the range of [0.08, 1.0] and [0.75, 1.33], respectively.

Original: Models are trained with batch size of 128 and batch accumulation of 2.
Revised: Models are trained with batch size of 128 and batch accumulation of 2.

Original: Experiments are performed on 4 GPUs, with 90 epochs for AlexNet following Miller et al.
Revised: Experiments are performed on 4 GPUs, with 90 epochs for AlexNet following Miller et al.

Original: (2021) (more epochs show little difference) and 100 epochs for ResNet following common practice.
Revised: (2021) (more epochs show little difference) and 100 epochs for ResNet following common practice.

Original: Results.
Revised: Results.

Original: Table 1 shows the results of the baseline model AlexNet
Revised: Table 1 shows the results of the baseline model AlexNet

Original: (Hinton et al., 2012), related methods, as well as our naive lateral inhibition-inspired (LI) design on ImageNet dataset.
Revised: (Hinton et al., 2012), related methods, as well as our naive lateral inhibition-inspired (LI) design on ImageNet dataset.

Original: While all listed methods improve the baseline significantly, our method outperforms others by a large margin.
Revised: While all listed methods improve the baseline significantly, our method outperforms others by a large margin.

Original: Layer normalization only consider one instance, which fits better for dynamic input rather than batch input.
Revised: Layer normalization only consider one instance, which fits better for dynamic input rather than batch input.

Original: Group normalization has further division into groups, therefore limits its performance.
Revised: Group normalization has further division into groups, therefore limits its performance.

Original: Weighted normalization and Divisive normalization are similar in the normalization design, except the power number in the numerator and the denominator, as well as the weighted sum.
Revised: Weighted normalization and Divisive normalization are similar in the normalization design, except the power number in the numerator and the denominator, as well as the weighted sum.

Original: However, most of them lack the consideration of neighbors within the same channel for lateral interaction, which our method does.
Revised: However, most of them lack the consideration of neighbors within the same channel for lateral interaction, which our method does.

Original: Table 3 shows the results of the baseline model
Revised: Table 2 shows the results of the baseline model

Original: ResNet (He et al., 2016) and our naive lateral inhibition inspired (LI) design on ImageNet dataset.
Revised: ResNet (He et al., 2016) and our naive lateral inhibition inspired (LI) design on ImageNet dataset.

Original: Although ResNet contains batch normalization to improve performance, our design makes the improvement further by 0.81%. 
Revised: Although ResNet contains batch normalization to improve performance, our design makes the improvement further by 0.81%. Design Choice.

Revised: From Table 3 we can see that, with a single scaling factor, our design already outperforms other methods.

Revised: Among all the design variants, scaling first with bias is better than shift first then scaling.

Revised: While they are both much better than only scaling, the difference between the best two are not large.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose our lateral inhibition-inspired structure for convolutional neural network to make it more brain-similar on image classification task.
Revised: In this paper, we propose our lateral inhibition-inspired structure for convolutional neural network to make it more brain-similar on image classification task.

Original: We explicitly explore neurobiology findings, using the guassian low-pass filter with central weight eliminated to mimic the strength decay regarding the distance.
Revised: We explicitly explore neurobiology findings, using the guassian low-pass filter with central weight eliminated to mimic the strength decay regarding the distance.

Original: With a learnable weight, our deign is flexible to model lateral inhibition as well as other lateral interaction.
Revised: With a learnable weight, our deign is flexible to model lateral inhibition as well as other lateral interaction.

Original: Our design works on both plain convolution and the convolutional block with residual connection, while being compatible with the existing modules.
Revised: Our design works on both plain convolution and the convolutional block with residual connection, while being compatible with the existing modules.

Original: Preliminary results using AlexNet and ResNet demonstrate obvious improvements on the ImageNet dataset for image classification, with little increase in parameters.
Revised: Preliminary results using AlexNet and ResNet demonstrate obvious improvements on the ImageNet dataset for added

Original: We hope that our research could inspire researchers to reconsider the value of this track for better feature learning.

Original: [Table]
Revised: [Table]

Original: Our  is 
Revised: Our structure is applied to both the initial plain convolution and the following residual blocks. Please note that ResNet already contains batch normalization to increase performance

Original: [Table]
Revised: [Table]

Revised: image classification, with little increase in parameters.

Revised: We hope that our research could inspire researchers to reconsider the value of this track for better feature learning.