Original: R E M IXER : O BJECT - AWARE M IXING L AYER FOR V ISION T RANSFORMERS AND M IXERS
Revised: R E M IXER : O BJECT - AWARE M IXING L AYER FOR V ISION T RANSFORMERS AND M IXERS

Original: A BSTRACT
Revised: A BSTRACT

Original: Patch-based models, e.g., Vision Transformers (ViTs) and Mixers, have shown impressive results on various visual recognition tasks, exceeding classic convolutional networks.
Revised: Patch-based models, e.g., Vision Transformers (ViTs) and Mixers, have shown impressive results on various visual recognition tasks, exceeding classic convolutional networks.

Original: While the initial patch-based models treated all patches equally, recent studies reveal that incorporating inductive biases like spatiality benefits the learned representations.
Revised: While the initial patch-based models treated all patches equally, recent studies reveal that incorporating inductive biases like spatiality benefits the learned representations.

Original: However, most prior works solely focused on the position of patches, overlooking the scene structure of images.
Revised: However, most prior works solely focused on the position of patches, overlooking the scene structure of images.

Original: This paper aims to further guide the interaction of patches using the object information.
Revised: This paper aims to further guide the interaction of patches using the object information.

Original: Specifically, we propose ReMixer, which reweights the patch mixing layers based on the patch-wise object labels extracted from pretrained saliency or classification models.
Revised: Specifically, we propose ReMixer, which reweights the patch mixing layers based on the patch-wise object labels extracted from pretrained saliency or classification models.

Original: We apply ReMixer on various patch-based models using different patch mixing layers:
Revised: We apply ReMixer on various patch-based models using different patch mixing layers:

Original: ViT, MLP-Mixer, and ConvMixer, where our method consistently improves the classification accuracy and background robustness of baseline models.
Revised: ViT, MLP-Mixer, and ConvMixer, where our method consistently improves the classification accuracy and background robustness of baseline models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Patch-based models, i e., models that process an input image as a sequence of visual patches, have arisen as a new paradigm of neural networks for visual data, alternating the prior standard convolutional neural networks (CNNs; LeCun et al.
Revised: Patch-based models, i e., models that process an input image as a sequence of visual patches, have arisen as a new paradigm of neural networks for visual data, alternating the prior standard convolutional neural networks (CNNs; LeCun et al.

Original: Remarkably, patch-based models have achieved state-of-the-art results on various computer vision tasks by favoring the scaling properties (Dosovitskiy et al., 2021; Zhai et al., 2021).
Revised: Remarkably, patch-based models have achieved state-of-the-art results on various computer vision tasks by favoring the scaling properties (Dosovitskiy et al., 2021; Zhai et al., 2021).

Original: They also merit various advantages, including out-ofdistribution generalization (Naseer et al., 2021), natural extension to video domains (Bertasius et al., 2021), integration with other domains like language or speech (Radford et al., 2021) and easily combined with state-of-the-art visual self-supervised learning (He et al., 2021).
Revised: They also merit various advantages, including out-ofdistribution generalization (Naseer et al., 2021), natural extension to video domains (Bertasius et al., 2021), integration with other domains like language or speech (Radford et al., 2021) and easily combined with state-of-the-art visual self-supervised learning (He et al., 2021).

Original: The core concept of patch-based models is to update patch-wise representations by alternating the computation within patches and among patches, called channel mixing and patch (or token) mixing, respectively.
Revised: The core concept of patch-based models is to update patch-wise representations by alternating the computation within patches and among patches, called channel mixing and patch (or token) mixing, respectively.

Original: The design of patch mixing layers is widely investigated: the pioneering Vision
Revised: The design of patch mixing layers is widely investigated: the pioneering Vision

Original: Transformer (ViT; Dosovitskiy et al.
Revised: Transformer (ViT; Dosovitskiy et al.

Original: and its descendants (Touvron et al., 2021a;b) considered self-attention (Vaswani et al., 2017), while other works considered feed-forward (Tolstikhin et al., 2021), convolution (Trockman & Kolter, 2022), or pooling operation (Yu et al., 2021).
Revised: and its descendants (Touvron et al., 2021a;b) considered self-attention (Vaswani et al., 2017), while other works considered feed-forward (Tolstikhin et al., 2021), convolution (Trockman & Kolter, 2022), or pooling operation (Yu et al., 2021).

Original: However, most patch-based models use self-attention or feed-forward mixing layers, which minimizes the inductive biases from the model by employing all patches equally (Khan et al., 2021).
Revised: However, most patch-based models use self-attention or feed-forward mixing layers, which minimizes the inductive biases from the model by employing all patches equally (Khan et al., 2021).

Original: While this data-centric approach is effective on large-scale scenarios, recent works claim that incorporating inductive biases is still essential for patch-based models, especially when learned from limited data (Steiner et al., 2021).
Revised: While this data-centric approach is effective on large-scale scenarios, recent works claim that incorporating inductive biases is still essential for patch-based models, especially when learned from limited data (Steiner et al., 2021).

Original: To this end, many approaches incorporated spatial inductive bias for patch-based models following the analogy of convolution: designing a patch mixing layer utilizing the location of patches (d’Ascoli et al., 2021; Dai et al., 2021; Wu et al., 2021a; Trockman &
Revised: To this end, many approaches incorporated spatial inductive bias for patch-based models following the analogy of convolution: designing a patch mixing layer utilizing the location of patches (d’Ascoli et al., 2021; Dai et al., 2021; Wu et al., 2021a; Trockman &

Original: Kolter, 2022) or building an architecture that combines convolutional or pooling layers with patch mixing layers (Liu et al., 2021; Yuan et al., 2021b;
Revised: Kolter, 2022) or building an architecture that combines convolutional or pooling layers with patch mixing layers (Liu et al., 2021; Yuan et al., 2021b;

Original: Wang et al., 2021; Fan et al., 2021).
Revised: Wang et al., 2021; Fan et al., 2021).

Original: However, both approaches focus on spatial inductive bias, which overlooks the sample-specific object structures.
Revised: However, both approaches focus on spatial inductive bias, which overlooks the sample-specific object structures.

Original: We provide additional discussion on related work in Appendix A.
Revised: We provide additional discussion on related work in Appendix A.

Original: Contribution.
Revised: Contribution.

Original: We propose ReMixer, a novel reweighting scheme for patch mixing layers leveraging the object structure of images.
Revised: We propose ReMixer, a novel reweighting scheme for patch mixing layers leveraging the object structure of images.

Original: We demonstrate that ReMixer improves classification accuracy and background robustness of patch-based models, outperforming the models considering spatiality.
Revised: We demonstrate that ReMixer improves classification accuracy and background robustness of patch-based models, outperforming the models considering spatiality.

Original:  R E M IXER : O BJECT - AWARE M IXING L AYER
Revised:  R E M IXER : O BJECT - AWARE M IXING L AYER

Original: The main idea of ReMixer is to strengthen the interaction of patches containing similar objects while regularizing the connection of different objects (and background).
Revised: The main idea of ReMixer is to strengthen the interaction of patches containing similar objects while regularizing the connection of different objects (and background).

Original: Intuitively, ReMixer improves the discriminability of objects (i.e., better classification) and reduces the spurious correlations between objects and backgrounds (i.e., robust to background and distribution shifts) by learning disentangled representations of objects.
Revised: Intuitively, ReMixer improves the discriminability of objects (i.e., better classification) and reduces the spurious correlations between objects and backgrounds (i.e., robust to background and distribution shifts) by learning disentangled representations of objects.

Original: We first introduce a general framework of object-aware mixing layer in
Revised: We first introduce a general framework of object-aware mixing layer in

Original: Section 2.1, then illustrate the specific instantiations for various architectures in Section 2.2.
Revised: Section 2.1, then illustrate the specific instantiations for various architectures in Section 2.2.

Original: 2. C OMPUTING REWEIGHTING MASK FOR R E M IXER
Revised: 2. C OMPUTING REWEIGHTING MASK FOR R E M IXER

Original: The idea of patch-based models is to reshape a 2D image x ∈ R H × W × C into a sequence of flattened 2D patches x 0 ∈ R N ×
Revised: The idea of patch-based models is to reshape a 2D image x ∈ R H × W × C into a sequence of flattened 2D patches x 0 ∈ R N ×

Original: ( P 2 C ) , where ( H, W ) is the resolution of original image, C is the number of color channels, ( P, P ) is the resolution of each image patch, and N = HW/P 2 is the number of patches.
Revised: ( P 2 C ) , where ( H, W ) is the resolution of original image, C is the number of color channels, ( P, P ) is the resolution of each image patch, and N = HW/P 2 is the number of patches.

Original: Patch-based models first convert the 2D patches into patch (or token) features x 1 :=
Revised: Patch-based models first convert the 2D patches into patch (or token) features x 1 :=

Original: f embed ( x 0 )
Revised: f embed ( x 0 )

Original: ∈ R N × D with latent dimension D using an embedding function f embed , then update the patch features by alternating two operations: (a) patch mixing layers f l mix : R N → R N which mix the features among patches, and (b) channel mixing layers g l mix : R D → R D which mix the features among channels, where l implies the operation of layer l .
Revised: ∈ R N × D with latent dimension D using an embedding function f embed , then update the patch features by alternating two operations: (a) patch mixing layers f l mix : R N → R N which mix the features among patches, and (b) channel mixing layers g l mix : R D → R D which mix the features among channels, where l implies the operation of layer l .

Original: Formally, the l -th layer of patch-based model updates an input vector x l ∈ R N × D to an output vector x l +1 ∈ R N × D following:
Revised: Formally, the l -th layer of patch-based model updates an input vector x l ∈ R N × D to an output vector x l +1 ∈ R N × D following:

Original: [Equation]
Revised: [Equation]

Original: where z n,d and x n,d denotes n -th patch, d -th channel value, and z l +11: N,d ∈ R N and x l +1 n, 1: D ∈ R D denotes row-wise and column-wise subvector of z and x , respectively.
Revised: where z n,d and x n,d denotes n -th patch, d -th channel value, and z l +11: N,d ∈ R N and x l +1 n, 1: D ∈ R D denotes row-wise and column-wise subvector of z and x , respectively.

Original: We introduce ReMixer, a universal framework for improving patch mixing layers f mix by incorporating the object structure of images.
Revised: We introduce ReMixer, a universal framework for improving patch mixing layers f mix by incorporating the object structure of images.

Original: To this end, we utilize the patch-wise labels y ∈ R N × K where
Revised: To this end, we utilize the patch-wise labels y ∈ R N × K where

Original: K is the number of object classes.
Revised: K is the number of object classes.

Original: Using them, we compute the reweighting mask M l ∈ R N × N that strengthens the interaction of patches of similar objects while regularizing the connection of different objects and backgrounds.
Revised: Using them, we compute the reweighting mask M l ∈ R N × N that strengthens the interaction of patches of similar objects while regularizing the connection of different objects and backgrounds.

Original: Formally, we set the ( i, j ) -th value of the reweighting mask M lij as a reverse distance between the object labels of two patches y i and y j :
Revised: Formally, we set the ( i, j ) -th value of the reweighting mask M lij as a reverse distance between the object labels of two patches y i and y j :

Original: [Equation]
Revised: [Equation]

Original: where κ ( l ) ≥ 0 is a learnable mask scale (scalar) parameter for each layer and d ( · , · ) : R K × R K →
Revised: where κ ( l ) ≥ 0 is a learnable mask scale (scalar) parameter for each layer and d ( · , · ) : R K × R K →

Original: R is a distance function for object labels.
Revised: R is a distance function for object labels.

Original: We initialize κ ( l ) by zero for training, i.e., consider the full interaction initially then focus on the objects as κ ( l ) increases.
Revised: We initialize κ ( l ) by zero for training, i.e., consider the full interaction initially then focus on the objects as κ ( l ) increases.

Original: We observe that the model sets higher κ ( l ) for lower layers and lower κ ( l ) for higher layers (especially, κ ( l ) = 0 for the final layer) after training, i.e., ReMixer automatically attends the intra-object relations first then expand to the inter-object relations, which resembles the local-to-global structure of CNNs (see Table 3).
Revised: We observe that the model sets higher κ ( l ) for lower layers and lower κ ( l ) for higher layers (especially, κ ( l ) = 0 for the final layer) after training, i.e., ReMixer automatically attends the intra-object relations first then expand to the inter-object relations, which resembles the local-to-global structure of CNNs (see Table 3).

Original: We aim to calibrate the N × N interaction of patch mixing layers using the reweighting mask M .
Revised: We aim to calibrate the N × N interaction of patch mixing layers using the reweighting mask M .

Original: If the patch mixing layer f mix :
Revised: If the patch mixing layer f mix :

Original: = L mix is linear, one can simply (element-wise) multiply the mask to get the masked linear mixer M ⊙ L mix .
Revised: = L mix is linear, one can simply (element-wise) multiply the mask to get the masked linear mixer M ⊙ L mix .

Original: While ReMixer can be applied to any patch layers in principle, one needs careful design to consider the nonlinear dynamics of each layer.
Revised: While ReMixer can be applied to any patch layers in principle, one needs careful design to consider the nonlinear dynamics of each layer.

Original: We provide specific implementations of ReMixer on various representative models in the next section.
Revised: We provide specific implementations of ReMixer on various representative models in the next section.

Original: Obtaining object labels.
Revised: Obtaining object labels.

Original: We extract object labels from saliency (Voynov et al., 2021) or classification (Yun et al., 2021)
Revised: We extract object labels from saliency (Voynov et al., 2021) or classification (Yun et al., 2021)

Original: models trained from other source datasets.
Revised: models trained from other source datasets.

Original: See Appendix B for details.
Revised: See Appendix B for details.

Original: 2. R E M IXER FOR V I T AND M IXERS
Revised: 2. R E M IXER FOR V I T AND M IXERS

Original: We briefly review three representative patch mixing layers: self-attention, feed-forward, and convolution, and describe the implementations of ReMixer for each layer.
Revised: We briefly review three representative patch mixing layers: self-attention, feed-forward, and convolution, and describe the implementations of ReMixer for each layer.

Original: ReMixer for self-attention.
Revised: ReMixer for self-attention.

Original: Self-attention (Vaswani et al., 2017) mixing layers update patch features by aggregating values with normalized importances (or attentions):
Revised: Self-attention (Vaswani et al., 2017) mixing layers update patch features by aggregating values with normalized importances (or attentions):

Original: [Equation]
Revised: [Equation]

Original: where Q , K , V are query, key, and value, respectively, which are linear projections of input x ∈ R N × D , given by Q := x · W Q ∈ R N × D K , K := x · W K ∈ R N × D K , and V := x · W V ∈ R N × D V .
Revised: where Q , K , V are query, key, and value, respectively, which are linear projections of input x ∈ R N × D , given by Q := x · W Q ∈ R N × D K , K := x · W K ∈ R N × D K , and V := x · W V ∈ R N × D V .

Original: Here, we compute H independent attention heads and aggregate outputs for the final output of size
Revised: Here, we compute H independent attention heads and aggregate outputs for the final output of size

Original: N × D for D = H · D V .
Revised: N × D for D = H · D V .

Original: Recall that self-attention is basically a matrix multiplication of attention A and value V matrices, and one can (element-wise) multiply the reweighting mask M to the attention matrix to calibrate interaction.
Revised: Recall that self-attention is basically a matrix multiplication of attention A and value V matrices, and one can (element-wise) multiply the reweighting mask M to the attention matrix to calibrate interaction.

Original: Then, we renormalize the masked attention M ⊙ A to make the rowwise sum be 1 as the original self-attention.
Revised: Then, we renormalize the masked attention M ⊙ A to make the rowwise sum be 1 as the original self-attention.

Original: To sum up, ReMixer for self-attention is:
Revised: To sum up, ReMixer for self-attention is:

Original: [Equation]
Revised: [Equation]

Original: where ˜ A is the renormalized masked attention.
Revised: where ˜ A is the renormalized masked attention.

Original: We finally remark that patch-based models using self-attention often use the additional [CLS] token to aggregate the global feature.
Revised: We finally remark that patch-based models using self-attention often use the additional [CLS] token to aggregate the global feature.

Original: Here, we define the mask value between the [CLS] token and every other patch to be one and apply Eq.
Revised: Here, we define the mask value between the [CLS] token and every other patch to be one and apply Eq.

Original: ReMixer for feed-forward.
Revised: ReMixer for feed-forward.

Original: Feed-forward (or multi-layer perceptron; MLP)
Revised: Feed-forward (or multi-layer perceptron; MLP)

Original: mixing layers update patch features with a channel-wise MLP.
Revised: mixing layers update patch features with a channel-wise MLP.

Original: Since each channel is computed independently, we only consider a N × 1 vector of a single channel.
Revised: Since each channel is computed independently, we only consider a N × 1 vector of a single channel.

Original: Then, the mixer layer is:
Revised: Then, the mixer layer is:

Original: [Equation]
Revised: [Equation]

Original: where W 1 , ..., W m are weight matrices and σ is a nonlinear activation.
Revised: where W 1 , ..., W m are weight matrices and σ is a nonlinear activation.

Original: However, it is nontrivial to apply the reweighting mask M since f mix is nonlinear.
Revised: However, it is nontrivial to apply the reweighting mask M since f mix is nonlinear.

Original: To tackle this issue, we decompose the mixing layer f mix into a linear approximation L x mix · x ≈ f mix ( x ) for a (possibly data-dependent) matrix L x mix ∈ R N × N and a residual term f mix ( x ) − L x mix · x .
Revised: To tackle this issue, we decompose the mixing layer f mix into a linear approximation L x mix · x ≈ f mix ( x ) for a (possibly data-dependent) matrix L x mix ∈ R N × N and a residual term f mix ( x ) − L x mix · x .

Original: Here, we only calibrate the linear term but omit the residual term.
Revised: Here, we only calibrate the linear term but omit the residual term.

Original: Then, ReMixer for feed-forward is given by:
Revised: Then, ReMixer for feed-forward is given by:

Original: [Equation]
Revised: [Equation]

Original: where ⊙ is an element-wise product.
Revised: where ⊙ is an element-wise product.

Original: While finding a good L x mix is nontrivial in general, we found that a simple trick of dropping nonlinear activations gives an efficient yet effective solution:
Revised: While finding a good L x mix is nontrivial in general, we found that a simple trick of dropping nonlinear activations gives an efficient yet effective solution:

Original: [Equation]
Revised: [Equation]

Original: We observe that this (somewhat crude) approximation performs well in practice.
Revised: We observe that this (somewhat crude) approximation performs well in practice.

Original: We also tried some data-dependent variants but did not see gain despite their computational burdens.
Revised: We also tried some data-dependent variants but did not see gain despite their computational burdens.

Original: ReMixer for convolution.
Revised: ReMixer for convolution.

Original: Convolutional mixing layers update patch features with a channel-wise 2D convolution.
Revised: Convolutional mixing layers update patch features with a channel-wise 2D convolution.

Original: Similar to the feed-forward case, we only consider a single channel input x (which is x 1: N,d formally), reshaped as a 1 × ¯ H × ¯ W tensor where ( ¯ H, ¯ W ) =
Revised: Similar to the feed-forward case, we only consider a single channel input x (which is x 1: N,d formally), reshaped as a 1 × ¯ H × ¯ W tensor where ( ¯ H, ¯ W ) =

Original: ( H/P, W/P ) is the resolution of patch features.
Revised: ( H/P, W/P ) is the resolution of patch features.

Original: Then, the mixer layer is:
Revised: Then, the mixer layer is:

Original: [Equation]
Revised: [Equation]

Original: where W conv ∈ R 1 × 1 × S × S is a kernel matrix with size S and ∗ denotes convolution operation.
Revised: where W conv ∈ R 1 × 1 × S × S is a kernel matrix with size S and ∗ denotes convolution operation.

Original: Here, we consider the linearized version of kernel matrix (i.e., Toeplitz matrix) that substitutes the convolution to the matrix multiplication.
Revised: Here, we consider the linearized version of kernel matrix (i.e., Toeplitz matrix) that substitutes the convolution to the matrix multiplication.

Original: Then, one can interpret the mixer layer as:
Revised: Then, one can interpret the mixer layer as:

Original: [Equation]
Revised: [Equation]

Original: where W linear ∈ R N × N is the corresponding matrix of W conv and ˜ x is a reshaped tensor of x of size N × 1 , where N = ¯ H · ¯ W .
Revised: where W linear ∈ R N × N is the corresponding matrix of W conv and ˜ x is a reshaped tensor of x of size N × 1 , where N = ¯ H · ¯ W .

Original: Here, one can directly multiply the reweighting mask M to define the ReMixer for convolution:
Revised: Here, one can directly multiply the reweighting mask M to define the ReMixer for convolution:

Original: [Equation]
Revised: [Equation]

Original: where ⊙ is an element-wise product.
Revised: where ⊙ is an element-wise product.

Original: We also compare ReMixer with the models using different kernel matrix for each channel, i.e., W conv ∈ R D × 1 × S × S (see Appendix D).
Revised: We also compare ReMixer with the models using different kernel matrix for each channel, i.e., W conv ∈ R D × 1 × S × S (see Appendix D).

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We first verify the efficacy of ReMixer in Section 3.2, outperforming both vanilla patch-based models and models considering spatial inductive bias.
Revised: We first verify the efficacy of ReMixer in Section 3.2, outperforming both vanilla patch-based models and models considering spatial inductive bias.

Original: Then, we demonstrate the robustness of ReMixer on out-of-distribution datasets in Section 3.3.
Revised: Then, we demonstrate the robustness of ReMixer on out-of-distribution datasets in Section 3.3.

Original: We also show that ReMixer outperforms the method that requires the same-level of supervision in Section 3.4, confirming that the remixing of patch mixing layers gives improvements.
Revised: We also show that ReMixer outperforms the method that requires the same-level of supervision in Section 3.4, confirming that the remixing of patch mixing layers gives improvements.

Original: The experimental settings are in Section 3.1.
Revised: The experimental settings are in Section 3.1.

Original: 3. E XPERIMENTAL SETTINGS
Revised: 3. E XPERIMENTAL SETTINGS

Original: Models and training.
Revised: Models and training.

Original: We apply ReMixer on three representative patch-based models: DeiT (Touvron et al., 2021a), MLP-Mixer (Tolstikhin et al., 2021), and ConvMixer (Trockman & Kolter, 2022), using self-attention, feed-forward, and convolutional patch mixing layers, respectively.
Revised: We apply ReMixer on three representative patch-based models: DeiT (Touvron et al., 2021a), MLP-Mixer (Tolstikhin et al., 2021), and ConvMixer (Trockman & Kolter, 2022), using self-attention, feed-forward, and convolutional patch mixing layers, respectively.

Original: Specifically, we use DeiT-S, MLP-Mixer-S/16, and ConvMixer-512/8.
Revised: Specifically, we use DeiT-S, MLP-Mixer-S/16, and ConvMixer-512/8.

Original: ConvMixer-512/8 has kernel size 9 and patch size 16 following MLP-Mixer-S/16 configurations, and we share the convolution kernel over channels by default, while we provide the unshared results in Appendix D.
Revised: ConvMixer-512/8 has kernel size 9 and patch size 16 following MLP-Mixer-S/16 configurations, and we share the convolution kernel over channels by default, while we provide the unshared results in Appendix D.

Original: We follow the default training setup of (Touvron et al., 2021a), but use 256 batch sizes due to memory issues in our GPUs.
Revised: We follow the default training setup of (Touvron et al., 2021a), but use 256 batch sizes due to memory issues in our GPUs.

Original: Object labels.
Revised: Object labels.

Original: We consider three object labels: first two are BigBiGAN (Voynov et al., 2021) and
Revised: We consider three object labels: first two are BigBiGAN (Voynov et al., 2021) and

Original: ReLabel (Yun et al., 2021), representative methods for binary saliency and multi-class prediction maps, respectively.
Revised: ReLabel (Yun et al., 2021), representative methods for binary saliency and multi-class prediction maps, respectively.

Original: We also test Bbox+GrabCut, which extracts binary masks from the ground-truth bounding boxes using the GrabCut (Rother et al., 2004) algorithm.
Revised: We also test Bbox+GrabCut, which extracts binary masks from the ground-truth bounding boxes using the GrabCut (Rother et al., 2004) algorithm.

Original: 3. M AIN RESULTS
Revised: 3. M AIN RESULTS

Original: Setup.
Revised: Setup.

Original: We train the models on the ImageNet-9 (Xiao et al., 2021a), which is a 9 superclass subset of ImageNet.
Revised: We train the models on the ImageNet-9 (Xiao et al., 2021a), which is a 9 superclass subset of ImageNet.

Original: We report the results on the Background Challenge (Xiao et al., 2021a) benchmark to evaluate the background robustness of models.
Revised: We report the results on the Background Challenge (Xiao et al., 2021a) benchmark to evaluate the background robustness of models.

Original: Background Challenge contains 8 datasets:
Revised: Background Challenge contains 8 datasets:

Original: O RIGINAL ( ↑ ), O NLY -BG-B ( ↓ ), O NLY -BG-T ( ↓ ), N O -FG ( ↓ ), O NLY -FG ( ↑ ), M IXED -S AME ( ↑ ),
Revised: O RIGINAL ( ↑ ), O NLY -BG-B ( ↓ ), O NLY -BG-T ( ↓ ), N O -FG ( ↓ ), O NLY -FG ( ↑ ), M IXED -S AME ( ↑ ),

Original: M IXED -R AND ( ↑ ), and M IXED -N EXT ( ↑ ), where the upper or lower arrows indicate the model
Revised: M IXED -R AND ( ↑ ), and M IXED -N EXT ( ↑ ), where the upper or lower arrows indicate the model

Original: [Table] [Table]
Revised: [Table] [Table]

Original: should predict the class well or not, respectively.
Revised: should predict the class well or not, respectively.

Original: We also report BG-G AP ( ↓ ) which measures the accuracy gap between M IXED -S AME and M IXED -R
Revised: We also report BG-G AP ( ↓ ) which measures the accuracy gap between M IXED -S AME and M IXED -R

Original: AND .
Revised: AND .

Original: We omit O NLY -BG-T, N O -FG, and
Revised: We omit O NLY -BG-T, N O -FG, and

Original: M IXED -N EXT results for the brevity of presentation (see Appendix C for discussion).
Revised: M IXED -N EXT results for the brevity of presentation (see Appendix C for discussion).

Original: Results.
Revised: Results.

Original: Table 1 shows that ReMixer consistently improves classification accuracy and background robustness over various patch-based models and object labels.
Revised: Table 1 shows that ReMixer consistently improves classification accuracy and background robustness over various patch-based models and object labels.

Original: Table 2 compares ReMixer with
Revised: Table 2 compares ReMixer with

Original: ConViT (d’Ascoli et al., 2021), a patch mixing layer with spatial prior.
Revised: ConViT (d’Ascoli et al., 2021), a patch mixing layer with spatial prior.

Original: It verifies that the objectcentric structure of ReMixer is more effective than spatiality, yet gives orthogonal benefits.
Revised: It verifies that the objectcentric structure of ReMixer is more effective than spatiality, yet gives orthogonal benefits.

Original: Analysis.
Revised: Analysis.

Original: Table 3 reports the mask scales κ ( l ) of trained models, averaged by each quarter of layers.
Revised: Table 3 reports the mask scales κ ( l ) of trained models, averaged by each quarter of layers.

Original: The models set higher/lower κ ( l ) for lower/higher layers, i.e., see the objects first then expand its view, like the local-to-global structure of CNNs.
Revised: The models set higher/lower κ ( l ) for lower/higher layers, i.e., see the objects first then expand its view, like the local-to-global structure of CNNs.

Original: ConvMixer sets low κ ( l ) for the early layers since it is hard to understand the objects due to the restricted view of convolution.
Revised: ConvMixer sets low κ ( l ) for the early layers since it is hard to understand the objects due to the restricted view of convolution.

Original: Figure 4 visualizes the saliency maps (Chefer et al., 2021), verifying that ReMixer gives more object-centric view.
Revised: Figure 4 visualizes the saliency maps (Chefer et al., 2021), verifying that ReMixer gives more object-centric view.

Original: 3. R OBUSTNESS OF R E M IXER
Revised: 3. R OBUSTNESS OF R E M IXER

Original: Setup.
Revised: Setup.

Original: We evaluate the robustness of ReMixer inferred on  out-of-distribution (OOD) data.
Revised: We evaluate the robustness of ReMixer inferred on unseen out-of-distribution (OOD) data.

Original: To this end, we test the ReMixer trained on ImageNet-9 on various OOD datasets: 9 superclass ( class) subset of ImageNetV2 (Recht et al., 2019),
Revised: To this end, we test the ReMixer trained on ImageNet-9 on various OOD datasets: 9 superclass ( class) subset of ImageNetV2 (Recht et al., 2019),

Original: ImageNet-ReaL (Beyer et al., 2020), ImageNetR (Hendrycks et al., 2021), ImageNet-Stylized (Geirhos et al., 2019), and ImageNet-Sketch (Wang et al., 2019), denoted by adding ’-9’ at the end.
Revised: ImageNet-ReaL (Beyer et al., 2020), ImageNetR (Hendrycks et al., 2021), ImageNet-Stylized (Geirhos et al., 2019), and ImageNet-Sketch (Wang et al., 2019), denoted by adding ’-9’ at the end.

Original: Results.
Revised: Results.

Original: Table 5 shows the OOD generalization results.
Revised: Table 5 shows the OOD generalization results.

Original: ReMixer performs well on OOD samples, confirming that both object annotators and learned masks are transferable.
Revised: ReMixer performs well on OOD samples, confirming that both object annotators and learned masks are transferable.

Original: We use the BigBiGAN annotator since it gives more robust prediction results than ReLabel.
Revised: We use the BigBiGAN annotator since it gives more robust prediction results than ReLabel.

Original: 3. F AIR COMPARISONS ON LARGE - SCALE DATASET
Revised: 3. F AIR COMPARISONS ON LARGE - SCALE DATASET

Original: Setup.
Revised: Setup.

Original: We compare ReMixer with TokenLabeling (TL; Jiang et al.
Revised: We compare ReMixer with TokenLabeling (TL; Jiang et al.

Original: (2021)) on ImageNet.
Revised: (2021)) on ImageNet.

Original: TL uses the same extra patch-level label with ReMixer.
Revised: TL uses the same extra patch-level label with ReMixer.

Original: The experiment settings are same as Section 3.1.
Revised: The experiment settings are same as Section 3.1.

Original: Results.
Revised: Results.

Original: Table 6 presents the comparison of TL and TL+ReMixer (Ours).
Revised: Table 6 presents the comparison of TL and TL+ReMixer (Ours).

Original: TL+ReMixer outperforms
Revised: TL+ReMixer outperforms

Original: TL, demonstrating that the guided patch interaction of ReMixer drives the performance gain further.
Revised: TL, demonstrating that the guided patch interaction of ReMixer drives the performance gain further.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose ReMixer, a novel object-centric framework to refine any existing patch-based models.
Revised: We propose ReMixer, a novel object-centric framework to refine any existing patch-based models.

Original: We demonstrate the efficacy of ReMixer on ViT, MLP-Mixer, and ConvMixer, while showing superior (yet compatible) performance over prior works using spatial inductive bias.
Revised: We demonstrate the efficacy of ReMixer on ViT, MLP-Mixer, and ConvMixer, while showing superior (yet compatible) performance over prior works using spatial inductive bias.

Original: We hope ReMixer could inspire new research directions for patch-based models and object-centric learning.
Revised: We hope ReMixer could inspire new research directions for patch-based models and object-centric learning.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Revised: This research was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF2018R1A5A1059921).

Original: We thank Sihyun Yu, Jihoon Tack, Jaeho Lee, and Jongjin Park for constructive feedback.
Revised: We thank Sihyun Yu, Jihoon Tack, Jaeho Lee, and Jongjin Park for constructive feedback.

Original: A R ELATED WORK
Revised: A R ELATED WORK

Original: Patch-based models.
Revised: Patch-based models.

Original: Inspired by the success of Transformers (or self-attention; Vaswani et al.
Revised: Inspired by the success of Transformers (or self-attention; Vaswani et al.

Original: (2017)) in natural language processing (Devlin et al., 2019; Brown et al., 2020), numerous works have attempted to extend Transformers for computer vision (Khan et al., 2021).
Revised: (2017)) in natural language processing (Devlin et al., 2019; Brown et al., 2020), numerous works have attempted to extend Transformers for computer vision (Khan et al., 2021).

Original: In particular, the seminal work named Vision Transformer (ViT; Dosovitskiy et al.
Revised: In particular, the seminal work named Vision Transformer (ViT; Dosovitskiy et al.

Original: (2021)) discovered that Transformer could achieve state-of-the-art performance, exceeding prior popular convolutional neural networks (CNNs; LeCun et al.
Revised: (2021)) discovered that Transformer could achieve state-of-the-art performance, exceeding prior popular convolutional neural networks (CNNs; LeCun et al.

Original: Thereafter, other studies revealed that different patch mixing layers such as feed-forward (Tolstikhin et al., 2021), convolution (Trockman & Kolter, 2022), or pooling (Yu et al., 2021) show comparable performance to self-attention, hypothesizing that the success of ViT comes from the patch-based architectures.
Revised: Thereafter, other studies revealed that different patch mixing layers such as feed-forward (Tolstikhin et al., 2021), convolution (Trockman & Kolter, 2022), or pooling (Yu et al., 2021) show comparable performance to self-attention, hypothesizing that the success of ViT comes from the patch-based architectures.

Original: Our work proposes an architecture-agnostic framework to improve the patch-based models by reweighting their patch mixing layers.
Revised: Our work proposes an architecture-agnostic framework to improve the patch-based models by reweighting their patch mixing layers.

Original: Inductive bias for patch-based models.
Revised: Inductive bias for patch-based models.

Original: Many patch-based models aim to remove inductive biases by using patch mixing layers without additional structures, e.g., self-attention.
Revised: Many patch-based models aim to remove inductive biases by using patch mixing layers without additional structures, e.g., self-attention.

Original: While they perform well on large data regimes, recent works reveal that inductive biases are still crucial for patch-based models, especially under limited data (Steiner et al., 2021).
Revised: While they perform well on large data regimes, recent works reveal that inductive biases are still crucial for patch-based models, especially under limited data (Steiner et al., 2021).

Original: Consequently, extensive literature proposed approaches to incorporate additional structures for patch-based models, e.g., spatial structures of CNNs.
Revised: Consequently, extensive literature proposed approaches to incorporate additional structures for patch-based models, e.g., spatial structures of CNNs.

Original: One line of work aims to design patch mixing layers reflecting inductive biases.
Revised: One line of work aims to design patch mixing layers reflecting inductive biases.

Original: For example, ConViT (d’Ascoli et al., 2021) and CoAtNet (Dai et al., 2021) calibrate self-attention with spatial distance between patches, CvT (Wu et al., 2021a) and ConvMixer (Trockman & Kolter, 2022) utilize convolution operation for patch mixing, and AS-MLP (Lian et al., 2021) design a structured operation aggregating the values from different axises.
Revised: For example, ConViT (d’Ascoli et al., 2021) and CoAtNet (Dai et al., 2021) calibrate self-attention with spatial distance between patches, CvT (Wu et al., 2021a) and ConvMixer (Trockman & Kolter, 2022) utilize convolution operation for patch mixing, and AS-MLP (Lian et al., 2021) design a structured operation aggregating the values from different axises.

Original: Another line of work build an architecture that combines convolutional or pooling layers with patch mixing layers Liu et al.
Revised: Another line of work build an architecture that combines convolutional or pooling layers with patch mixing layers Liu et al.

Original: ; Yuan et al.
Revised: ; Yuan et al.

Original: (2021b)
Revised: (2021b)

Original: ; Wang et al.
Revised: ; Wang et al.

Original: ; Fan et al.
Revised: ; Fan et al.

Original: (2021); Heo et al.
Revised: (2021); Heo et al.

Original: ; Yuan et al.
Revised: ; Yuan et al.

Original: (2021a);
Revised: (2021a);

Original: Xiao et al.
Revised: Xiao et al.

Original: (2021b).
Revised: (2021b).

Original: Our work falls into the first category; however, we leverage the object structure of images, unlike prior works focused on the spatial inductive bias.
Revised: Our work falls into the first category; however, we leverage the object structure of images, unlike prior works focused on the spatial inductive bias.

Original: Using rich information, our proposed
Revised: Using rich information, our proposed

Original: ReMixer outperforms ConViT and CoAtNet, where using both ConViT and ReMixer gives further improvements, implying that two methods contribute to the model differently (see Table 2).
Revised: ReMixer outperforms ConViT and CoAtNet, where using both ConViT and ReMixer gives further improvements, implying that two methods contribute to the model differently (see Table 2).

Original: We also emphasize that ReMixer can be applied on any patch mixing layers under a common principle, unlike prior works designed for specific layers such as self-attention or feed-forward.
Revised: We also emphasize that ReMixer can be applied on any patch mixing layers under a common principle, unlike prior works designed for specific layers such as self-attention or feed-forward.

Original: Incorporating object structures.
Revised: Incorporating object structures.

Original: Although objects are the atom of visual scenes, only a limited number of research has leveraged the object structure of images for visual recognition (e.g., classification).
Revised: Although objects are the atom of visual scenes, only a limited number of research has leveraged the object structure of images for visual recognition (e.g., classification).

Original: This is mainly due to two reasons: (a) the cost of collecting object labels and (b) non-triviality of reflecting object information to the black-box deep learning models.
Revised: This is mainly due to two reasons: (a) the cost of collecting object labels and (b) non-triviality of reflecting object information to the black-box deep learning models.

Original: However, both challenges have been relaxed by the rapid advance of deep learning.
Revised: However, both challenges have been relaxed by the rapid advance of deep learning.

Original: First, the progress of supervised (He et al., 2017; Carion et al., 2020; Fang et al., 2021), weakly-supervised (Selvaraju et al., 2017; Chefer et al., 2021; Yun et al., 2021), and self-supervised (Voynov et al., 2021; Caron et al., 2021; Mo et al., 2021) detection significantly reduced the cost of object labels.
Revised: First, the progress of supervised (He et al., 2017; Carion et al., 2020; Fang et al., 2021), weakly-supervised (Selvaraju et al., 2017; Chefer et al., 2021; Yun et al., 2021), and self-supervised (Voynov et al., 2021; Caron et al., 2021; Mo et al., 2021) detection significantly reduced the cost of object labels.

Original: We utilize the pretrained BigBiGAN (Voynov et al., 2021) and ReLabel (Yun et al., 2021) models for our experiments; one could also train weakly- or self-supervised object annotators on their datasets.
Revised: We utilize the pretrained BigBiGAN (Voynov et al., 2021) and ReLabel (Yun et al., 2021) models for our experiments; one could also train weakly- or self-supervised object annotators on their datasets.

Original: Second, the patch-based models are well-suited with object information (unlike CNNs) as the patch embeddings preserve their spatial information (Raghu et al., 2021).
Revised: Second, the patch-based models are well-suited with object information (unlike CNNs) as the patch embeddings preserve their spatial information (Raghu et al., 2021).

Original: Using this property, ReMixer adjusts the interaction of patch embeddings using object labels.
Revised: Using this property, ReMixer adjusts the interaction of patch embeddings using object labels.

Original: ORViT (Herzig et al., 2021) also direct ViT to focus on the object regions by creating extra object tokens.
Revised: ORViT (Herzig et al., 2021) also direct ViT to focus on the object regions by creating extra object tokens.

Original: However, their goal is to guide video
Revised: However, their goal is to guide video

Original: Transformers to track the trajectory of objects and are less suited for image classification.
Revised: Transformers to track the trajectory of objects and are less suited for image classification.

Original: On the other hand, TokenLabeling (Jiang et al., 2021) implicitly utilizes the object information by using them as additional supervision for patch embeddings; it provides an orthogonal gain from ReMixer (see Table 6).
Revised: On the other hand, TokenLabeling (Jiang et al., 2021) implicitly utilizes the object information by using them as additional supervision for patch embeddings; it provides an orthogonal gain from ReMixer (see Table 6).

Original: We finally note that several works (Locatello et al., 2020; Wu et al., 2021b; Kipf et al., 2021) aim to disentangle the object features explicitly.
Revised: We finally note that several works (Locatello et al., 2020; Wu et al., 2021b; Kipf et al., 2021) aim to disentangle the object features explicitly.

Original: However, they do not scale yet to the complex real-world images due to the strong constraints in the model.
Revised: However, they do not scale yet to the complex real-world images due to the strong constraints in the model.

Original: In contrast, ReMixer can be applied to any existing patch-based models with minimal modification.
Revised: In contrast, ReMixer can be applied to any existing patch-based models with minimal modification.

Original: B O BTAINING OBJECT LABELS
Revised: B O BTAINING OBJECT LABELS

Original: One possible concern for ReMixer is the labeling cost of patch-wise object labels y ∈ R N × K .
Revised: One possible concern for ReMixer is the labeling cost of patch-wise object labels y ∈ R N × K .

Original: However, we claim that this cost is not a critical issue since one can utilize the pretrained machine annotators.
Revised: However, we claim that this cost is not a critical issue since one can utilize the pretrained machine annotators.

Original: Notably, the object labels extracted from the models trained on some (source) datasets are still helpful for different (target) downstream datasets (see Section 3.3).
Revised: Notably, the object labels extracted from the models trained on some (source) datasets are still helpful for different (target) downstream datasets (see Section 3.3).

Original: In the remaining section, we describe two types of machine annotators: binary saliency and multi-class prediction with discussion on their pros and cons.
Revised: In the remaining section, we describe two types of machine annotators: binary saliency and multi-class prediction with discussion on their pros and cons.

Original: Binary saliency map.
Revised: Binary saliency map.

Original: We first consider binary saliency maps, i.e., indicating whether the given pixel is object or background.
Revised: We first consider binary saliency maps, i.e., indicating whether the given pixel is object or background.

Original: There is a tremendous amount of work on extracting saliency maps in a self-supervised (Voynov et al., 2021; Caron et al., 2021; Mo et al., 2021) or weakly-supervised (i.e., using class labels; Selvaraju et al.
Revised: There is a tremendous amount of work on extracting saliency maps in a self-supervised (Voynov et al., 2021; Caron et al., 2021; Mo et al., 2021) or weakly-supervised (i.e., using class labels; Selvaraju et al.

Original: ; Chefer et al.
Revised: ; Chefer et al.

Original: (2021)) manner.
Revised: (2021)) manner.

Original: We use the saliency model called BigBiGAN (Voynov et al., 2021), which finds the salient region using BigGAN (Brock et al., 2019) trained on the ImageNet (Deng et al., 2009) dataset.
Revised: We use the saliency model called BigBiGAN (Voynov et al., 2021), which finds the salient region using BigGAN (Brock et al., 2019) trained on the ImageNet (Deng et al., 2009) dataset.

Original: We average the pixel-wise saliency values in the patch to get a soft label y n ∈ [0 , 1] , and use the l 1 -distance (between the object labels of two patches) in Eq.
Revised: We average the pixel-wise saliency values in the patch to get a soft label y n ∈ [0 , 1] , and use the l 1 -distance (between the object labels of two patches) in Eq.

Original: Multi-class prediction map.
Revised: Multi-class prediction map.

Original: We also consider multi-class prediction maps, i.e., pixel- or patchlevel semantic segmentation.
Revised: We also consider multi-class prediction maps, i.e., pixel- or patchlevel semantic segmentation.

Original: However, since segmentation labels are expensive, we utilize ReLabel (Yun et al., 2021), which predicts the dense label maps from an image classifier by applying the classifier on penultimate spatial features (i.e., before global average pooling).
Revised: However, since segmentation labels are expensive, we utilize ReLabel (Yun et al., 2021), which predicts the dense label maps from an image classifier by applying the classifier on penultimate spatial features (i.e., before global average pooling).

Original: We use the NFNet-F (Brock et al., 2021) model trained on the ImageNet dataset to extract ReLabel map and apply regionof-interest (RoI) pooling (Girshick, 2015) to extract the patch label y n ∈ R K .
Revised: We use the NFNet-F (Brock et al., 2021) model trained on the ImageNet dataset to extract ReLabel map and apply regionof-interest (RoI) pooling (Girshick, 2015) to extract the patch label y n ∈ R K .

Original: Here, computing l p -distance between the object labels of patches is expensive since it handles a N 2 K tensor.
Revised: Here, computing l p -distance between the object labels of patches is expensive since it handles a N 2 K tensor.

Original: Instead, we use the cosine distance, efficiently computed by matrix multiplication.
Revised: Instead, we use the cosine distance, efficiently computed by matrix multiplication.

Original: Comparison of two approaches.
Revised: Comparison of two approaches.

Original: Multi-class prediction map contains richer semantics and thus provide more informative reweighing masks.
Revised: Multi-class prediction map contains richer semantics and thus provide more informative reweighing masks.

Original: As a result, ReLabel (used as the object labels for
Revised: As a result, ReLabel (used as the object labels for

Original: ReMixer) has shown better results than BigBiGAN in our experiments, especially when the downstream tasks are close to the source dataset, ImageNet.
Revised: ReMixer) has shown better results than BigBiGAN in our experiments, especially when the downstream tasks are close to the source dataset, ImageNet.

Original: However, ReLabel is often prone to distribution shifts, while BigBiGAN provides consistent gain under the same setup.
Revised: However, ReLabel is often prone to distribution shifts, while BigBiGAN provides consistent gain under the same setup.

Original: Intuitively, predicting the salient objects is easier to generalize than predicting classes.
Revised: Intuitively, predicting the salient objects is easier to generalize than predicting classes.

Original: Thus, we suggest the users choose binary vs. multi-class labels following their robustness vs. in-distribution accuracy trade-off.
Revised: Thus, we suggest the users choose binary vs. multi-class labels following their robustness vs. in-distribution accuracy trade-off.

Original: We finally remark that while we utilize the pretrained annotators (from different source datasets) for simplicity, one can also apply our method without external datasets.
Revised: We finally remark that while we utilize the pretrained annotators (from different source datasets) for simplicity, one can also apply our method without external datasets.

Original: Recall that the annotators we consider are trained in an unsupervised or weakly-supervised manner; it does not require groundtruth dense supervision and can be solely extracted from the downstream dataset.
Revised: Recall that the annotators we consider are trained in an unsupervised or weakly-supervised manner; it does not require groundtruth dense supervision and can be solely extracted from the downstream dataset.

Original: C D ISCUSSION ON B ACKGROUND C HALLENGE
Revised: C D ISCUSSION ON B ACKGROUND C HALLENGE

Original: Figure 1 visualizes the datasets in the Background Challenge (Xiao et al., 2021a) benchmark, which provides various combinations of foregrounds and backgrounds: O RIGINAL ( ↑ ),
Revised: Figure 1 visualizes the datasets in the Background Challenge (Xiao et al., 2021a) benchmark, which provides various combinations of foregrounds and backgrounds: O RIGINAL ( ↑ ),

Original: O NLY -BG-B ( ↓ ),
Revised: O NLY -BG-B ( ↓ ),

Original: O NLY -BG-T ( ↓ ), N O -FG ( ↓ ),
Revised: O NLY -BG-T ( ↓ ), N O -FG ( ↓ ),

Original: O NLY -FG ( ↑ ), M IXED -S AME ( ↑ ), M IXED -R AND ( ↑ ), M IXEDN EXT ( ↑ ).
Revised: O NLY -FG ( ↑ ), M IXED -S AME ( ↑ ), M IXED -R AND ( ↑ ), M IXEDN EXT ( ↑ ).

Original: The upper or lower arrows indicate the model should predict the class well or not, respectively.
Revised: The upper or lower arrows indicate the model should predict the class well or not, respectively.

Original: We omit O NLY -BG-T, N O -FG, and M IXED -N EXT results for brevity of presentation and due to the following reasons:
Revised: We omit O NLY -BG-T, N O -FG, and M IXED -N EXT results for brevity of presentation and due to the following reasons:

Original: • M IXED -N EXT shows almost the same trend with M IXED -R
Revised: • M IXED -N EXT shows almost the same trend with M IXED -R

Original: AND .
Revised: AND .

Original: • N O -FG is controversial to be predicted or not since the image contains the black shape of the object.
Revised: • N O -FG is controversial to be predicted or not since the image contains the black shape of the object.

Original: • O NLY -BG-T accuracy is proportional to the O RIGINAL accuracy.
Revised: • O NLY -BG-T accuracy is proportional to the O RIGINAL accuracy.

Original: However, we remark that ReMixer only increases the O NLY -BG-T accuracy a little while O RIGINAL a lot, compared to the baseline models.
Revised: However, we remark that ReMixer only increases the O NLY -BG-T accuracy a little while O RIGINAL a lot, compared to the baseline models.

Original: D S HARING WEIGHTS FOR C ONV M IXER
Revised: D S HARING WEIGHTS FOR C ONV M IXER

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We adopt ConvMixer (Trockman & Kolter, 2022) to share the convolution kernels over channels to reduce memory and computation.
Revised: We adopt ConvMixer (Trockman & Kolter, 2022) to share the convolution kernels over channels to reduce memory and computation.

Original: Remarkably, ReMixer applied on ConvMixer is comparable or even outperforms ConvMixer-Full, which uses different kernels for each channel, i.e., uses D =times parameters for patch mixing layers.
Revised: Remarkably, ReMixer applied on ConvMixer is comparable or even outperforms ConvMixer-Full, which uses different kernels for each channel, i.e., uses D =times parameters for patch mixing layers.