Original: Bio-inspired Min-Nets Improve the Performance and Robustness of Deep Networks
Revised: Bio-inspired Min-Nets Improve the Performance and Robustness of Deep Networks

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Min-Nets are inspired by end-stopped cortical cells with units that output theminimum of two learned ﬁlters.
Revised: Min-Nets are inspired by end-stopped cortical cells with units that output the minimum of two learned ﬁlters.

Original: We insert such min-units into state-of-the art deepnetworks, such as the popular ResNet and DenseNet, and show that the resulting
Revised: We insert such Min-units into state-of-the-art deep networks, such as the popular ResNet and DenseNet, and show that the resulting

Original: Min-Nets perform better on the Cifar-10 benchmark.
Revised: Min-Nets perform better on the Cifar-10 benchmark.

Original: Moreover, we show that
Revised: Moreover, we show that

Original: Min-Nets are more robust against JPEG compression artifacts.
Revised: Min-Nets are more robust against JPEG compression artifacts.

Original: We argue that theminimum operation is the simplest way of implementing an AND operation onpairs of ﬁlters, and that such AND operations introduce a bias that is appropriategiven the statistics of natural images.
Revised: We argue that the minimum operation is the simplest way of implementing an AND operation on pairs of ﬁlters and that such AND operations introduce a bias that is appropriate given the statistics of natural images

Original:  Introduction
Revised:  Introduction

Original: We present a novel network-architecture that is based on the AND combination of pairs of linearﬁlters.
Revised: We present a novel network architecture that is based on the AND combination of pairs of linear ﬁlters.

Original: The linear ﬁlters in deep networks can be considered as basic measurements of the input signal,each ﬁlter capturing signal energy along some relevant direction in feature space.
Revised: The linear ﬁlters in deep networks can be considered as basic measurements of the input signal, each ﬁlter capturing signal energy along some relevant direction in feature space.

Original: In differentialgeometry, the basic measurements are the main curvatures.
Revised: In differential geometry, the basic measurements are the main curvatures.

Original: In  case of 2-dimensional manifolds,it is known that the structure of the manifold is captured by the Gaussian curvature, which is theproduct of the main curvatures (do Carmo (1976)).
Revised: In the case of 2-dimensional manifolds, it is known that the structure of the manifold is captured by the Gaussian curvature, which is the product of the main curvatures (do Carmo (1976)).

Original: In higher dimensions, the Riemann curvaturetensor captures the structure of the manifold and it is based on sums of products of main curvatures(OR combinations of pairwise AND combinations).
Revised: In higher dimensions, the Riemann curvature tensor captures the structure of the manifold, and it is based on sums of products of main curvatures (OR combinations of pairwise AND combinations).

Original: Inspired by differential geometry, in computervision, different algorithms have been designed to capture interest points in images, for example, byanalysing the invariants and eigenvalues of the Hessian or of the structure tensor (Guiducci (1988);
Revised: Inspired by differential geometry, in computer vision, different algorithms have been designed to capture interest points in images, for example, by analysing the invariants and eigenvalues of the Hessian or of the structure tensor (Guiducci (1988);

Original: Jähne (1995)).
Revised: Jähne (1995)).

Original: Both the product and the minimum of the eigenvalues have been used for the detectionof interest points (Shi & Tomasi (1994)).
Revised: Both the product and the minimum of the eigenvalues have been used for the detection of interest points (Shi & Tomasi (1994)).

Original: In related work on FP-nets (Anonymous (2021))
Revised: In related work on feature product (FP)-nets (Grüning et al.

Original:  the AND operation has been implemented by usingexplicit multiplications.
Revised: (2021)), the AND operation has been implemented by using explicit multiplications.

Original: FP-nets have been shown to improve the performance of state-of-the artnetworks.
Revised: FP-nets have been shown to improve the performance of state-of-the-art networks.

Original: Moreover, FP-units are hyper-selective (for more information on hyper-selectivity, see(Paiton et al.
Revised: Moreover, FP-units are hyperselective (for more information on hyperselectivity, see (Paiton et al.

Original: and (Vilankar & Field (2017))) and thus more robust against compressionartifacts and adversarial attacks (Anonymous (2021)).
Revised: and (Vilankar & Field (2017))) and thus more robust against compression artifacts and adversarial attacks (Grüning et al.

Original: As an alternative to the computationally moreintensive multiplications, the log-nets introduced in Grüning et al.
Revised: As an alternative to the computationally more intensive multiplications, the log-nets introduced in Grüning et al.

Original: (2020) are more efﬁcient by usingconvolutions in log-space.
Revised: (2020) are more efﬁcient by using convolutions in log-space.

Original: However, the main feature of both the FP-nets and the log-nets is that thenetworks learn appropriate ﬁlter pairs to be AND-combined.
Revised: However, the main feature of both the FP-nets and the log-nets is that the networks learn appropriate ﬁlter pairs to be AND-combined.

Original: FP-nets and the here proposed Min-Nets are inspired by what is known about biological vision sincethe seminal work of Hubel and Wiesel (Hubel & Wiesel (1965)).
Revised: FP-nets and the here proposed Min-Nets are inspired by what is known about biological vision since the seminal work of Hubel and Wiesel (Hubel & Wiesel (1965)).

Original: They discovered oriented neuronsin primary visual cortex V1, but also end-stopped cells in V1 and V2.
Revised: They discovered oriented neurons in primary visual cortex V1 but also end-stopped cells in V1 and V2.

Original: End-stopped cells extractfeatures such as corners and line ends, provide an efﬁcient representation of the visual input, andcan be modelled as AND combinations of linear, oriented ﬁlters (Zetzsche & Barth (1990)).
Revised: End-stopped cells extract features such as corners and line ends, provide an efﬁcient representation of the visual input, and can be modelled as AND combinations of linear, oriented ﬁlters (Zetzsche & Barth (1990)).

Original: Motion selective neurons have also been modelled by using AND combinations of linear spatio-temporalﬁlters based on the Riemann tensor (Barth & Watson (2000)).
Revised: Motion selective neurons have also been modelled by using AND combinations of linear spatio-temporal ﬁlters based on the Riemann tensor (Barth & Watson (2000)).

Original: The question of why end-stopped neurons and AND combinations of linear ﬁlters are useful for visioncan be answered based on the statistics of natural images.
Revised: The question of why end-stopped neurons and AND combinations of linear ﬁlters are useful for vision can be answered based on the statistics of natural images.

Original: A ﬁrst useful bias is due to the fact that0D-features, i.e., local image patches of uniform intensity or color, are both redundant and frequentin natural images.
Revised: A ﬁrst useful bias is due to the fact that 0D-features, i.e., local image patches of uniform intensity or color, are both redundant and frequent in natural images.

Original: Therefore, a representation of images in terms of edges is efﬁcient, it reducesthe entropy of the representation.
Revised: Therefore, a representation of images in terms of edges is efﬁcient; it reduces the entropy of the representation.

Original: 1D-features such as straight edges and lines, however, are alsoredundant and more frequent in natural images than 2D-features, such as curved edges and corners.
Revised: 1D-features such as straight edges and lines, however, are also redundant and more frequent in natural images than 2D-features, such as curved edges and corners.

Original: Therefore, a representation provided by end-stopped units is more efﬁcient than one provided byoriented ﬁlters.
Revised: Therefore, a representation provided by end-stopped units is more efﬁcient than one provided by oriented ﬁlters.

Original: Details on the above summarized statistics, and on why AND combinations areneeded to exploit them, can be found in Zetzsche et al.
Revised: Details on the above-summarized statistics and on why AND combinations are needed to exploit them can be found in Zetzsche et al.

Original: However, we do not assume that real neurons would compute ideal multiplications and strict minimumoperations, but that they may approximate such operations in some way and thus generate ANDcombinations of ﬁlters instead of just using point-wise non-linearities (see also Paiton et al.
Revised: However, we do not assume that real neurons would compute ideal multiplications and strict minimum operations, but that they may approximate such operations in some way and thus generate AND combinations of ﬁlters instead of just using point-wise non-linearities (see also Paiton et al.

Original: (2020)regarding the limits of point-wise non-linearities).
Revised: (2020) regarding the limits of point-wise non-linearities).

Original: Here, we present the novel Min-Net architecturethat implements the minimum operation as the simplest kind of AND operation on pairs of learnedﬁlters.
Revised: Here, we present the novel Min-Net architecture that implements the minimum operation as the simplest kind of AND operation on pairs of learned ﬁlters.

Original: Moreover, minimum operations are faster than multiplications and make training easier due tomore conservative gradients.
Revised: Moreover, minimum operations are faster than multiplications and make training easier due to more conservative gradients.

Original:  Methods
Revised:  Methods

Original: Min-Nets are CNNs containing a certain number of Min-Blocks that process an input Tensor T 0 and,via a sequence of different layers, transform them to a tensor T out .
Revised: Min-Nets are CNNs containing a certain number of Min-blocks that process an input Tensor Tand, via a sequence of different layers, transform them to a tensor T out .

Original: The block structure resembles theinverted residual block of the well-known MobileNet-V2 (Sandler et al.
Revised: The block structure resembles the inverted residual block of the well-known MobileNet-V2 (Sandler et al.

Original: (2018)) architecture, with theimportant extension that it contains two ﬁlter operation that are combined by a minimum operation.
Revised: (2018)) architecture, with the important extension that it contains two ﬁlter operations that are combined by a minimum operation.

Original: Inshort, the block consists of (i) a convolution layer with kernel size one, batch normalization and ReLU,increasing the number of input feature maps, (ii) two depth-wise separable (DWS) convolutionseach learning one ﬁlter per feature map with instance normalization and ReLU, the outputs of whichare element-wise combined by the minimum function, (iii) a second 1 × 1 -convolution with batchnormalization and ReLU, recombining the outputs of (ii) to the desired number of feature maps and,ﬁnally (iv) a residual connection.
Revised: In short, the block consists of (i) a convolution layer with kernel size one, batch normalization, and ReLU, increasing the number of input feature maps, (ii) two depth-wise separable (DWS) convolutions, each learning one ﬁlter per feature map with instance normalization and ReLU, the outputs of which are element-wise combined by the minimum function, (iii) a second 1 × 1 -convolution with batch normalization and ReLU, recombining the outputs of (ii) to the desired number of feature maps and, ﬁnally, (iv) a residual connection.

Original: The next subsection gives a more detailed description.
Revised: The next subsection gives a more detailed description.

Revised: The middle panel of Figure 1 depicts a Min-block.

Original: 2.1 Min-blocks
Revised: 2.1 Min-blocks

Original: First, qd out feature maps are computed as weighted sums of the d feature maps of T 0 ∈ R h × w × d .
Revised: First, qd out feature maps are computed as weighted sums of the d feature maps of T 0 ∈ R h × w × d .

Original: qis an expansion factor and d out the desired number of output feature maps.
Revised: q is an expansion factor, and d out the desired number of output feature maps.

Original: Thus, a convolution layerwith kernel size one, and subsequent ReLU, computes the ( i, j ) -th pixel of the m -th feature map as:
Revised: Thus, a convolution layer with kernel size one, and subsequent ReLU, computes the ( i, j ) -th pixel of the m -th feature map as:

Original: [Equation]
Revised: [Equation]

Original: Note that  T 1 issubsequently normalized via batch normalization.
Revised: Note that before the ReLU, T 1 is normalized via batch normalization.

Original: Next, two DWS convolutionsare applied: for each feature map T m 1 a ﬁlter-pair v m and g m  is learned.
Revised: Next, two DWS convolutions are applied: for each feature map T m 1 a ﬁlter-pair V m and G m ∈ R k × k is learned.

Original: For each pixel ( i, j ) ,the ﬁlter operation computes the scalar product of the ﬁlter vectors  and the vectorized feature mappatch x m ∈ R k 2 with ( i, j ) being the pixel-coordinates of the center pixel.
Revised: For each pixel ( i, j ) , the ﬁlter operation computes the scalar product of the ﬁlter vectors (i.e., the vectorized ﬁlter kernels vect ( V m ) = v m and g m ∈ R k 2 ) and the vectorized feature map patch x m ∈ R k 2 with ( i, j ) being the pixel-coordinates of the center pixel.

Original: Subsequently, instancenormalization (Ulyanov et al.
Revised: Subsequently, instance normalization (Ulyanov et al.

Original: (2016)) is applied with µ v m and σ v m being the mean and standarddeviation of the feature map T m 1 ﬁltered with v m .
Revised: (2016)) is applied with µ v m and σ v m being the mean and standard deviation of the feature map T mﬁltered with v m .

Original: Again, a ReLU is applied afterwards.
Revised: Again, a ReLU is applied afterwards.

Original: Using themin-function, we combine these two intermediate outputs to T 2 ∈ R hs × ws × qd out , s being the strideof the DWS-convolution that allows for subsampling.
Revised: Using the min-function, we combine these two intermediate outputs to T 2 ∈ R hs × ws × qd out , s being the stride of the DWS-convolution that allows for subsampling.

Original: The min-unit, as the main building block of theproposed network, is deﬁned as:
Revised: The Min-unit, as the main building block of the proposed network, is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Note that such a min-unit, or min-neuron, will have a signiﬁcant output only if both ﬁlters v m andg m are activated.
Revised: Note that such a Min-unit, or Min-neuron, will have a signiﬁcant output only if both ﬁlters v m and g m are activated.

Original: A second linear recombination withReLU and batch normalization  (see Equation 1) creates thetensor T 3 ∈ R hs × ws × d out .
Revised: A second linear recombination with batch normalization and ReLU (see Equation 1) creates the tensor T 3 ∈ R hs × ws × d out .

Original: Finally, a residual connection adds the original input to T 3 :
Revised: Finally, a residual connection adds the original input to T 3 :

Original: [Equation]
Revised: [Equation]

Original: Some additional computations need to be applied if the dimensions of T 0 and T 3 do not match.
Revised: Some additional computations need to be applied if the dimensions of T 0 and T 3 do not match.

Original: Seethe Appendix for details.
Revised: See the Appendix for details.

Original: 2.2 Transforming state-of-the-art CNNs to Min-Nets
Revised: 2.2 Transforming state-of-the-art CNNs to Min-Nets

Original: In our experiments, we evaluate how the addition of Min-blocks changes the performance of stateof-the-art CNNs.
Revised: In our experiments, we evaluate how the addition of Min-blocks changes the performance of stateof-the-art CNNs.

Original: Networks such as the DenseNet (Huang et al.
Revised: Networks such as the DenseNet (Huang et al.

Original: and the ResNet (He et al.(2016)) have a similar overall structure: after an initial convolution layer, the networks consistsof several stacks that are larger structures containing a number of blocks .
Revised: and the ResNet (He et al. (2016)) have a similar overall structure: after an initial convolution layer, the networks consists of several stacks that are larger structures containing a number of blocks .

Original: Analogously, a blockcontains a number of layers , for example, convolutions, batch normalization, and non-linearities.
Revised: Analogously, a block contains a number of layers , for example, convolutions, batch normalization, and non-linearities.

Original: After the last stack, global average pooling is applied with a subsequent linear layer and softmax tocompute the class probabilities.
Revised: After the last stack, global average pooling is applied with a subsequent linear layer and softmax to compute the class probabilities.

Original: Stacks are separated by a downsampling layer (strided convolution ora max-pooling layer), except for the ﬁrst stack after the initial convolution.
Revised: Stacks are separated by a downsampling layer (strided convolution or a max-pooling layer), except for the ﬁrst stack after the initial convolution.

Original: When applied to smallerdatasets, e.g., Cifar, both the DenseNet and ResNet consist of three stacks, containing N blockseach, with the input heights and widths 32 × 32 , 16 × 16 , and 8 × 8 .
Revised: When applied to smaller datasets, e.g., Cifar, both the DenseNet and ResNet consist of three stacks, containing N blocks each, with the input heights and widths 32 × 32 , 16 × 16 , and 8 × 8 .

Original: Thus, only the second andthird stack employ downsampling.
Revised: Thus, only the second stack and third stack employ downsampling.

Original: We use a simple design rule to enrich these architectures withblocks that directly compute pairwise AND-combinations: Each ﬁrst block of a stack is substitutedby a Min-block.
Revised: We use a simple design rule to enrich these architectures with blocks that directly compute pairwise AND-combinations: Each ﬁrst block of a stack is substituted by a Min-block.

Original: Note that this approach was derived empirically.
Revised: Note that this approach was derived empirically.

Original: In preceding experiments, othercombinations of Min-blocks and convolutional blocks were tested but yielded somewhat inferiorresults.
Revised: In preceding experiments, other combinations of Min-blocks and convolutional blocks were tested but yielded somewhat inferior or equal results.

Original: We chose a ResNet implementation that uses pyramid blocks (Han et al.
Revised: We chose a ResNet implementation that uses pyramid blocks (Han et al.

Original: since itperforms better than the original ResNet and is currently often used for different applications.
Revised: since it performs better than the original ResNet and is currently often used for different applications.

Original: Here,the sequence of layers consists of batch normalization, 3 × 3 convolution, batch normalization, ReLU,3 × 3 convolution, and batch normalization.
Revised: Here, the sequence of layers consists of batch normalization, 3 × 3 convolution, batch normalization, ReLU, 3 × 3 convolution, and batch normalization.

Original: The sequence output is added to the original input of theblock.
Revised: The sequence output is added to the original input of the block.

Original: As a second reference network, we use the DenseNet-BC.
Revised: As a second reference network, we use the DenseNet-BC.

Original: In this architecture, each so-calledbottleneck block consists of batch normalization, ReLU, and a 1 × 1 convolution layer, followed by asecond batch normalization and ReLU and a 3 × 3 convolution layer.
Revised: In this architecture, each so-called bottleneck block consists of batch normalization, ReLU, and a 1 × 1 convolution layer, followed by a second batch normalization and ReLU and a 3 × 3 convolution layer.

Revised: A comparison of the different block structures is given in Figure 1.

Original: 2.3 Hyper-selectivity
Revised: 2.3 Hyperselectivity

Original: CNNs employ linear neurons with points-wise non-linearities, i.e., a ReLU that is applied to eachpixel independently.
Revised: CNNs employ linear neurons with points-wise non-linearities (LN-neurons), e.g., a ReLU that is applied to each pixel independently.

Original: Thus, the direction of maximal activation, i.e., the optimal stimulus, x ∗ pointsin the same direction as the neuron’s weight vector w .
Revised: Thus, the direction of maximal activation, i.e., the optimal stimulus, x ∗ points in the same direction as the neuron’s weight vector w .

Original: Any deviation o orthogonal to the optimalstimulus does not alter the neuron’s output: ReLU ( w T ( x ∗ + o ))
Revised: Any deviation o orthogonal to the optimal stimulus does not alter the neuron’s output: ReLU ( w T ( x ∗ + o ))

Original: =
Revised: =

Original: ReLU ( w T x ∗ ) .
Revised: ReLU ( w T x ∗ ) .

Original: In contrast, aneuron f ( x ) is hyper-selective, if there exist orthogonal perturbations, that reduce the output:f is hyper-selective ⇔ ∃ o : f ( x ∗ + o ) <
Revised: In contrast, a neuron f ( x ) is hyperselective, if there exist orthogonal perturbations, that reduce the output:

Original: f ( x ∗ ); x ∗ T o =(4)
Revised: [Equation]

Original: For a min-neuron (see Equation 2), the optimal stimulus is the bisector of the angle γ between theﬁlter vectors v and g .
Revised: For a Min-neuron (see Equation 2), the optimal stimulus is the bisector of the angle γ between the ﬁlter vectors v and g .

Original: Any perturbation within the plane spanned by v and g that is orthogonal to theoptimal stimulus reduces the neuron’s output, because one of the scalar products is smaller than itwould be without the perturbation, e.g., v T ( x ∗ + o ) <
Revised: Any perturbation within the plane spanned by v and g that is orthogonal to the optimal stimulus reduces the neuron’s output, because one of the scalar products is smaller than it would be without the perturbation, e.g., v T ( x ∗ + o ) <

Original: v T x ∗ .
Revised: v T x ∗ .

Original: Hyper-selective neurons are harderto activate by signals that differ from the optimal stimulus.
Revised: Hyperselective neurons are harder to activate by signals that differ from the optimal stimulus.

Original: Thus, unwanted perturbations, such asthose due to adversarial examples (Szegedy et al.
Revised: Thus, unwanted perturbations, such as those due to adversarial examples (Szegedy et al.

Original: ; Goodfellow et al.
Revised: ; Goodfellow et al.

Original: or compressionartifacts, have a smaller impact on the neuron’s – and consequently, the whole network’s – output.
Revised: or compression artifacts, have a smaller impact on the neuron’s – and consequently, the whole network’s – output.

Revised: Note that an LN-neuron located in the second or higher layer of a CNN can become hyperselective indirectly when regarding the pixel space as input.

Revised: In contrast, a Min-neuron is directly hyperselective to its input.

Original:  Experiments
Revised:  Experiments

Original: We hypothesize that Min-Nets introduce an useful inductive bias to a CNN architecture: the minimumof a learned ﬁlter-pair, a min-neuron, explicitly models end-stopped neurons and leads to efﬁcientrepresentations, which are adapted to the statistics of natural images, thus leading to better generalization.
Revised: We hypothesize that Min-Nets introduce a useful inductive bias to a CNN architecture: the minimum of a learned ﬁlter-pair, a Min-neuron, explicitly models end-stopped neurons and leads to efﬁcient [Figure] representations, which are adapted to the statistics of natural images, thus leading to better generalization.

Original: Furthermore, unlike traditional model neurons, min-neurons are hyper-selective, a propertythat should make them more robust, e.g., to compression artifacts.
Revised: Furthermore, unlike traditional model neurons, Min-neurons are hyperselective, a property that should make them more robust, e.g., to compression artifacts.

Original: We’ve run two experiments toinvestigate these conjectures.
Revised: We ran two experiments to investigate these conjectures.

Original: 3.1 Classiﬁcation performance on Cifar-10 
Revised: 3.1 Classiﬁcation performance on Cifar-

Original: We compared models with different depths created from two reference network architectures: the
Revised: We compared models with different depths created from two reference network architectures: the

Original: ResNet and the DenseNet.
Revised: ResNet and the DenseNet.

Original: For each model, we created a Min-Net version using our design rule.
Revised: For each model, we created a Min-Net version using our design rule.

Original: Wecompared models with different depths, deﬁned by the number of blocks per stack.
Revised: We compared models with different depths, deﬁned by the number of blocks per stack.

Original: For the ResNets,we created models with N ∈ { 3 , 5 , 7 , 9 } blocks.
Revised: For the ResNets, we created models with N ∈ { 3 , 5 , 7 , 9 } blocks.

Original: We trained DenseNets with N ∈ { 3 , 9 , 16 } blocksand a growth-rate k = 12 , corresponding to networks ( L = 22 , k = 12) , ( L = 58 , k = 12) , and( L = 100 , k = 12) in the DenseNet paper’s naming convention.
Revised: We trained DenseNets with N ∈ { 3 , 9 , 16 } blocks and a growth-rate k = 12 , corresponding to networks ( L = 22 , k = 12) , ( L = 58 , k = 12) , and ( L = 100 , k = 12) in the DenseNet paper’s naming convention.

Original: For all models, we set the expansionfactor q = 2  and we trained and tested the models on Cifar-10 (Krizhevsky et al. (2021)) with astandard procedure for data augmentation (see Appendix).
Revised: For all models, we set the expansion factor q = 2 , and we trained and tested the models on Cifar-10 (Krizhevsky et al. (2021)) with a standard procedure for data augmentation (see Appendix).

Original: Each ResNet was trained for 200 epochswith a learning rate of 0.1, a weight decay of 0.0001, and using SGD with a momentum of 0.9.
Revised: Each ResNet was trained for 200 epochs with a learning rate of 0.1, a weight decay of 0.0001, and using SGD with a momentum of 0.9.

Original: Thebatch size was 128.
Revised: The batch size was 128.

Original: After 100 and 150 epochs, the learning rate was divided by 10.
Revised: After 100 and 150 epochs, the learning rate was divided by 10.

Original: Each net wastrained for ﬁve random seeds, altering the batch order and random initialization.
Revised: Each net was trained for ﬁve random seeds, altering the batch order and random initialization.

Original: We trained the
Revised: We trained the

Original: DenseNets for 300 epochs, where the initial learning rate of 0.1 was divided by ten after 150 and 225epochs.
Revised: DenseNets for 300 epochs, where the initial learning rate of 0.1 was divided by ten after 150 andepochs.

Original: Instead of 128, a batch size of 64 was used.
Revised: Instead of 128, a batch size of 64 was used.

Original: In addition, we used the Nesterov momentum.
Revised: In addition, we used the Nesterov momentum.

Original: We trained each network with three different random seeds.
Revised: We trained each network with three different random seeds.

Original: All other parameters were equal to the
Revised: All other parameters were equal to the

Original: ResNet experiments.
Revised: ResNet experiments.

Original: Note that we did not employ an early stopping strategy, but rather report theﬁnal test results after 200 or 300 epochs, respectively.
Revised: Note that we did not employ an early stopping strategy but rather report the ﬁnal test results after 200 or 300 epochs, respectively.

Original: 3.2 Robustness to compression artifacts
Revised: 3.2 Robustness to compression artifacts

Original: Since Min-Nets have neurons that are hyper-selective, on can expect the networks to be more robustagainst perturbations.
Revised: Since Min-Nets have neurons that are hyperselective, one can expect the networks to be more robust against perturbations.

Original: To evaluate robustness, in a second experiment, each model trained in theﬁrst experiment was tested on JPEG-compressed versions of the Cifar-10 test set X test .
Revised: To evaluate robustness, in a second experiment, each model trained in the ﬁrst experiment was tested on JPEG-compressed versions of the Cifar-10 test set X test .

Original: We creatednine sets S Q 
Revised: We created [Figure] nine additional test sets S Q =

Original: = { JPEG ( I , Q ) :
Revised: { JPEG ( I , Q ) :

Original: I ∈ X test } with quality Q ∈ { 90 , 80 , ..., 10 } , JPEG ( I , Q ) beingthe compression function for an input image I .
Revised: I ∈ X test } with quality Q ∈ { 90 , 80 , ..., 10 } , JPEG ( I , Q ) being the compression function for an input image I .

Original: For Q = 100 , the original image is returned (nocompression); Q = 1 would yield maximal compression with images that are barely recognizable(see Figure 4 in the Appendix for examples).
Revised: For Q = 100 , the original image is returned (no compression); Q = 1 would yield maximal compression with images that are barely recognizable (see Figure 5 in the Appendix for examples).

Original: As robustness metric, we report the percentage ofchanged predictions (POCP), i.e., we compared the model’s predictions( f ( I ) ) with full quality tothe predictions on S Q and computed the number of changed predictions divided by the number ofimages (see Eq.
Revised: As robustness metric, we report the percentage of changed predictions (POCP), i.e., we compared the model’s predictions with full quality to the predictions on S Q and computed the number of changed predictions divided by the number of images (see Eq.

Original: 5 in the Appendix).
Revised: 5 in the Appendix).

Original: Using this metric, we avoid offsets introduced by models withlower test errors.
Revised: Using this metric, we avoid offsets introduced by models with lower test errors.

Original: We used the JPEG algorithm from the OpenCV (Bradski (2000)) library.
Revised: We used the JPEG algorithm from the OpenCV (Bradski (2000)) library.

Original:  Results
Revised:  Results

Original: The Cifar-10 test errors of each model are shown in Figure 1.
Revised: The Cifar-10 test errors of each model are shown in Figure 2.

Original: Note that, in all conﬁgurations, the
Revised: Note that in all conﬁgurations, the

Original: Min-Net has a lower mean test error than the corresponding baseline networks while using fewerparameters.
Revised: Min-Net has a lower mean test error than the corresponding baseline networks while using fewer parameters.

Original: Next, we analyse the robustness of the networks.
Revised: Next, we analyse the robustness of the networks.

Original: Figure 2 shows the POCP (Equation 5) for thequality levels Q = 90 , 80 , 70 , 60 .
Revised: Figure 3 shows the POCP (Equation 5) for the quality levels Q = 90 , 80 , 70 , 60 .

Original: Compared to the ResNet and DenseNet, the Min-Nets are lesslikely to change their output when presented with compressed images.
Revised: Compared to the ResNet and DenseNet, the Min-Nets are less likely to change their output when presented with compressed images.

Original: However, even the more robust
Revised: However, even the more robust

Original: Min-Nets remain sensitive to compression artifacts given that 8% of the predictions change with aslightly altered test set (such as S 90 ).
Revised: Min-Nets remain sensitive to compression artifacts given that 8% of the predictions change with a slightly altered test set (such as S 90 ).

Original:  Discussion
Revised:  Discussion

Original: We have presented a novel network architecture and have demonstrated its competitive performance.
Revised: We have presented a novel network architecture and have demonstrated its competitive performance.

Original: We have designed experiments with state-of-the-art and popular deep networks and showed that wecould improve their performance by substituting original blocks in the network architecture with
Revised: We have designed experiments with state-of-the-art and popular deep networks and showed that we could improve their performance by substituting original blocks in the network architecture with

Original: Min-blocks that implement a minimum operation on pairs of feature maps.
Revised: Min-blocks that implement a minimum operation on pairs of feature maps.

Original: Note that by using asimple design rule, any traditional network can easily be transformed into an Min-Net that will mostlikely perform better.
Revised: Note that by using a simple design rule, any traditional network can easily be transformed into a Min-Net that will most likely perform better.

Original: Also note that we did not tune hyperparameters speciﬁc to the Min-Nets butused the hyperparameters of the original networks; additional tuning may thus lead to even betterperformance of the Min-Nets. 
Revised: Also, note that we did not tune hyperparameters speciﬁc to the Min-Nets but used the hyperparameters of the original networks; additional tuning may thus lead to even better performance of the Min-Nets. [Figure]

Original: We belief that a bias that is appropriate for natural images leads to the improvements that wehave obtained.
Revised: We believe that a bias that is appropriate for natural images leads to the improvements that we have obtained.

Original: As argued in the Introduction, this bias allows the network to learn more efﬁcientrepresentations based on model neurons that are end-stopped.[Figure] o r a f t e [Figure]
Revised: As argued in the Introduction, this bias allows the network to learn more efﬁcient representations based on model neurons that are end-stopped.

Original: e p o c h s

Original: [Figure]

Original: e s t

Original: e rr o r a f t

Original: e r

Original: [Figure]

Original: e p

Original: [Figure]

Original: c h s [Figure] [Figure] The pairwise minimum operations that we introduce allow for AND rather than OR combinationsof features and make the resulting neurons more selective than linear ﬁlters with only point-wisenon-linearities.
Revised: The pairwise minimum operations that we introduce allow for AND rather than OR combinations of features and make the resulting neurons more selective than linear ﬁlters with only point-wise non-linearities.

Original: The demonstrated increased robustness of the Min-Nets is most likely due to the factthat Min-units are hyperselective.
Revised: The demonstrated increased robustness of the Min-Nets is most likely due to the fact that Min-units are hyperselective.

Original: It has been argued before that hyperselectivity might be the key togreater robustness (Paiton et al.
Revised: It has been argued before that hyperselectivity might be the key to greater robustness (Paiton et al.

Original: In previous work, similar results have been obtained with Log-nets and FP-nets where logarithms andfeature products have been used instead of the minimum.
Revised: In previous work similar results have been obtained with Log-nets and FP-nets where logarithms and feature products have been used instead of the minimum.

Original: This shows that the key to the improvementsdemonstrated here is mainly the AND combination of ﬁlters pairs and less the way in which
Revised: This shows that the key to the improvements demonstrated here is mainly the AND combination of ﬁlters pairs and less the way in which

Original: the
Revised: the

Original: AND is implemented.
Revised: AND is implemented.

Original: Nevertheless, we are looking forward to more general and even more effectivenetwork architectures that may emerge based on the principle of using the minimum activation indifferent feature maps.Checklist For all authors...
Revised: Nevertheless, we are looking forward to more general and even more effective network architectures that may emerge based on the principle of using the minimum activation in different feature maps.

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes] We provide citations that document the importance of

Original: AND terms in vision and show on well established state-of-the-art networks that using

Original: AND terms can be beneﬁcial

Original: (b) Did you describe the limitations of your work?

Original: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We’ve included

Original: the code and the training logs in the supplemental material.

Original: At the time of submissionno direct upload to OpenReview was possible.

Original: However, the code can be downloadedat https://ufile.io/dutbo6r3 .

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] We disclosed all training details in the paper or Appendix.(c)

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix.4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a)

Original: If your work uses existing assets, did you cite the creators?

Original: [Yes]

Original: We

Original: cited the deeplearning framework and the creators of code we adapted.(b)

Original: Did you mention the license of the assets?

Original: [Yes](c)

Original: Did you include any new assets either in the supplemental material or as a URL?

Original: [No](d) Did you discuss whether

Original: and how consent was obtained from people whose data you’reusing/curating?

Original: [Yes] We use a standard benchmark dataset.(e)

Original: Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?

Original: [Yes]5.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A](b)

Original: Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A](c)

Original: Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A] A AppendixA.1 Residual connections
Revised: A Appendix A.1 Residual connections

Original: If a stride of two is used ( s = 2 ), average pooling with a kernel size of two and stride of two isapplied to T 0 .
Revised: If a stride of two is used ( s = 2 ), average pooling with a kernel size of two and stride of two is applied to T 0 .

Original: If d is smaller than d out , T 0 is padded with ( d out − d ) all-zero feature maps.
Revised: If d is smaller than d out , T 0 is padded with ( d out − d ) all-zero feature maps.

Original: If d islarger than d out , a linear recombination without ReLU and batch normalization computes a reducedversion of T 0 with d out feature maps.
Revised: If d is larger than d out , a linear recombination without ReLU and batch normalization computes a reduced version of T 0 with d out feature maps.

Original: A.2 Data augmentation
Revised: A.2 Data augmentation

Original: During training, the images were ﬂipped horizontally with a 50% chance.
Revised: During training, the images were ﬂipped horizontally with a 50% chance.

Original: Furthermore, they werepadded with four pixels on each side and randomly cropped using a 32 × 32 window.
Revised: Furthermore, they were padded with four pixels on each side and randomly cropped using a 32 × 32 window.

Original: We normalizedeach channel of an input image by subtracting the channel’s mean pixel and dividing by the standarddeviation, all derived from the training set.
Revised: We normalized each channel of an input image by subtracting the channel’s mean pixel and dividing by the standard deviation, all derived from the training set.

Original: A.3 Percentage of changed predictions
Revised: A.3 Percentage of changed predictions

Original: We computed the POCP depending on Q like this:
Revised: We computed the POCP depending on Q like this:

Original: [Equation]
Revised: [Equation]

Original: 1 ( S ) returns a 1 if S is true, as 0 otherwise.
Revised: 1 ( S ) returns a 1 if S is true, as 0 otherwise; f ( I ) is the model’s class prediction for the input image I .

Original: A.4 Technical details for training
Revised: A.4 Technical details for training

Original: All experiments were conducted with Pytorch (BSD license)
Revised: All experiments were conducted with Pytorch (BSD license)

Original: (Paszke et al.
Revised: (Paszke et al.

Original: version"1.9.0+cu102", on a standard computer running on Linux using Nvidia RTX 2080 GPUs with
Revised: (2019)) version "1.9.0+cu102", on a standard computer running on Linux using Nvidia RTX 2080 GPUs with

Original: CUDA version 11.2.
Revised: CUDA version 11.2.

Original: Training a DenseNet ( L = 100 , k = 12) took roughly 21 hours, training a
Revised: Training a DenseNet ( L = 100 , k = 12) took roughly 21 hours, training a

Original: ResNet ( N = 9 ) 3 hours.
Revised: ResNet ( N = 9 ) 3 hours.

Original: The code creating the ResNet was adapted from Idelbayev (2020) (BSD2-Clause "Simpliﬁed" License) and from Han et al.(2017)code 1 .
Revised: The code creating the ResNet was adapted from Idelbayev (2020) (BSD 2-Clause "Simpliﬁed" License) and from Han et al.

Original: We adapted the DenseNet code 2from Pleiss et al.
Revised: We adapted the DenseNet code 2 from Pleiss et al.

Original: (2017) (MIT License).
Revised: (2017) (MIT License).

Original: OpenCV uses the Apache 2 license.
Revised: OpenCV uses the Apache 2 license.

Revised: We published our code on

Revised: Github

Original: A.5 Additional results: Densenet
Revised: A.5 Additional results: Densenet

Original: Table 1 shows additional results for the DenseNet models on the Cifar-10 classiﬁcation task.
Revised: Table 1 shows additional results for the DenseNet models on the Cifar-10 classiﬁcation task.

Original: Whenregarding minimal errors over all epochs, i.e., the best result with early stopping, we obtained similarresults for the DenseNet ( L = 100 , k = 12 ) as reported in the original paper ( 4 . 51% ): the minimumerror reached by the baseline DenseNet was 4 .
Revised: When regarding minimal errors over all epochs, i.e., the best result with early stopping, we obtained similar results for the DenseNet ( L = 100 , k = 12 ) as reported in the original paper ( 4 . 51% ): the minimum error reached by the baseline DenseNet was 4 .

Original: 56% and 4 .
Revised: 56% and 4 .

Original: 36% for the Min-Net.
Revised: 36% for the Min-Net.

Original: A.6 Additional results: JPEG compression
Revised: A.6 Additional results: JPEG compression

Original: Figure 3 shows the Cifar-10 test errors depending on the Q value.
Revised: Figure 4 shows the Cifar-10 test errors depending on the Q value.

Original: Note that these plots only providethe results for Q = 100 , 90 , ..., 60 , and only for the largest models of each experiment (ResNet:
Revised: Note that these plots only provide the results for Q = 100 , 90 , ..., 60 , and only for the largest models of each experiment (ResNet:

Original: N = 9 , DenseNet: k = 12 , L = 100 ).
Revised: N = 9 , DenseNet: k = 12 , L = 100 ).

Original: The results for Q = 100 , 90 , ..., 10 and for all models aregiven in the Tables 2,3, 4, and 5.
Revised: The results for Q = 100 , 90 , ..., 10 and for all models are given in the Tables 2,3, 4, and 5.

Original: Table 2 shows the test error values for the DenseNet experiment and
Revised: Table 2 shows the test error values for the DenseNet experiment and

Original: Table 3 shows the POCP.
Revised: Table 3 shows the POCP.

Original: The test error and POCP for the ResNet experiments are given in the Tables4 and 5, respectively.
Revised: The test error and POCP for the ResNet experiments are given in the Tables 4 and 5, respectively.

Original: An example image compressed with different Q -values is shown in Figure 4.
Revised: An example image compressed with different Q -values is shown in Figure 5.

Original: [Table]
Revised: Note that the POCP of a speciﬁc Q ∗ can be higher than the respective increase of the test error. For example, ∆( Q ∗ ) – i.e., the test error for Q ∗ = 90 minus the original error – is lower than the POCP for Q = 90 since the POCP measures any change in classiﬁcation: apart from correct classiﬁcations for Q = 100 being incorrect for Q ∗ , the POCP also counts classiﬁcations that are already incorrect and are now classiﬁed as another false class. Furthermore, a few previously incorrect examples can become correct and decrease ∆( Q ) .

Revised: _the_performance_and_

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Min-Net 8.
Revised: Min-Net 8.

Original: Min-Net 6. Min-Net 6.
Revised: Min-Net 6. Min-Net 6.

Original: Min-Net 6.
Revised: Min-Net 6.

Original: ResNet 8.
Revised: ResNet 8.

Original: ResNet 7.
Revised: ResNet 7.

Original: ResNet 6. ResNet 6.
Revised: ResNet 6. ResNet 6.

Original: [Figure]
Revised: [Figure]

Original: Model

Original: N 90% 80% 70% 60% 50% 40% 30% 20% 10% Min-Net 9. Min-Net 8. Min-Net 8.
Revised: [Table]

Original: Min-Net 7.

Original: ResNet 10. ResNet 9. ResNet 8. ResNet 8.
Revised: [Table]