Original:  On the Relationship between Self-Attention and Convolutional LayersML Reproducibility Challenge
Revised: [Re]: On the Relationship between Self-Attention and Convolutional Layers

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress 
Revised: AfﬁliationAddress email

Original: Abstract Networks based on self-attention are now paving the way towards achieving state-of-the-art performance in image recognition.
Revised: Scope of Reproducibility

Original: In this report, we present a detailed study on the paper titled " On the 
Revised: In this report, we perform a detailed study on the paper "On the Relationship between Self-Attention and Convolutional

Original: Relationship between Self-Attention and Convolutional Layers" by Cordonnier et. al. [2020] whichprovides a theoretical  evidence that  attention layers canindeed behave like convolutions.
Revised: Layers" [1] which provides theoretical and experimental evidence that self attention layers can behave like convolutionallayers. The proposed method does not obtain state-of-the-art performance but rather answers an interesting question - doself-attention layers process images in a similar manner to convolutional layers? .

Original:  We exploredifferent variations of self-attention from other recent works andassess their effectiveness for imagerecognition. Further, we propose a new variant of the attention operation - Hierarchical Attentionwhich shows improved performance with signiﬁcantly lesser parameters.
Revised: This has inspired many recent workslike [2, 3] which propose a fully-attentional model for image recognition. We focus on experimentally validating theclaims of the original paper, highlight differences with other similar works, and propose a new variant of the attentionoperation - Hierarchical Attention which shows improved performance with signiﬁcantly lesser parameters.

Original: To facilitate further study,all the code used in our experiments are publicly available here.
Revised: To facilitatefurther study, all the code used in our experiments are publicly available here 1 . Methodology

Revised: We implement the original paper [1] from scratch in Pytorch and refer to the author’s source code 2 for veriﬁcation.

Revised: Inour experiments involving SAN [2], we utilize the ofﬁcial implementation 3 due to the available faster CUDA kernelswhile we implement VIT [3] from scratch referring to the author’s source code 4 .

Revised: We then incorporate our proposedhierarchical operation in all three methods for comparison.

Revised: For all our experiments mentioned in this report, we usethe CIFAR10 dataset to benchmark the performance of the model in the image classiﬁcation task.

Original: Introduction
Revised: Each training runrequired around 20 hours while the corresponding hierarchical versions took around 10-12 hours for convergence in an Nvidia RTX 2060 GPU. Results We were able to reproduce all the results from the paper within 1% of the reported value, hence validating the claimsof the original paper.

Revised: However, there seems to be some differences in the attention ﬁgures which lead to interestinginsights and the proposed Hierarchical Attention.

Revised: In the case of VIT and SAN, we do not have a comparative baselineas the corresponding papers do not evaluate performance on the CIFAR10 dataset (without pre-training).

Revised: What was easy

Revised: We did not face any major challenges in reproducing the results in the paper

Revised: What was difﬁcult

Revised: Most of the code in the ofﬁcial implementation seems to be borrowed from HuggingFace’s repository 5 which also brought along a lot of unnecessary code making it difﬁcult to read and understand quickly.

Revised: Further the training time for each run is quite signiﬁcant making it difﬁcult for us to experiment with multiple datasets and hyperparameter settings

Revised: Communication with original authors

Revised: We have tried contacting the authors regarding the differences in the attention ﬁgures since the code for the same was not available on the repository for veriﬁcation.

Original:  In computer vision, convolutional architectures [1, 2, 3, 4] have dominated across various image recognition taskslike classiﬁcation, segmentation, etc.
Revised: However, we have not received any response regarding the same  Introduction In computer vision, convolutional architectures [4, 5, 6, 7] have dominated across various image recognition taskslike classiﬁcation, segmentation, etc.

Original: However, they have some limitations like lack of rotation invariance, inability toaggregate information based on the image content.
Revised: However, they have some limitations like lack of rotation invariance, inability toaggregate information based on the image content, etc.

Original: This has inspired researchers to explore a different design spaceand introduce models with interesting new capabilities.
Revised: This has inspired researchers to explore a different design spaceand introduce models with interesting new capabilities.

Original: Self-attention based networks, in particular Transformers [5]have become the model of choice in various Natural Language Processing (NLP) tasks.
Revised: Self-attention based networks, in particular Transformers [8]have become the model of choice for various Natural Language Processing (NLP) tasks.

Original: The major difference betweentransformers and previous methods, such as recurrent neural networks and convolutional neural networks (CNN), is thatthe former can simultaneously learn to attend to various parts of the input sequence.
Revised: The major difference betweentransformers and previous methods, such as recurrent neural networks and Convolutional Neural Networks (CNN), isthat the former can simultaneously learn to attend to various parts of the input sequence.

Original: To utilize this capacity to learnmeaningful inter-dependencies, many recent works have tried to incorporate self-attention [6, 7], some even replacingconvolutions entirely [8, 9].
Revised: To utilize this capacity to learnmeaningful inter-dependencies, many recent works have tried to incorporate self-attention [9, 10], some even replacingconvolutions entirely [11, 12], in networks for vision tasks.

Original: The main highlights of the paper (Cordonnier et. al. [2020]) is the following:
Revised: The main highlights of the paper "On the Relationshipbetween Self-Attention and Convolutional Layers" [1] are:

Original: Theoretical proof that self-attention layers can behave similar to convolution layers.
Revised: Theoretical proof that self-attention layers can behave similar to convolution layers.

Original: Empirically validate the performance of the self-attention model on the image classiﬁcation task using theCIFAR10 dataset.
Revised: Empirical validation of the performance of the self-attention model on the image classiﬁcation task using theCIFAR10 dataset.

Original: Visually show that self-attention applied to images learns convolutional ﬁlters around thequery pixels.
Revised: Visual evidence that self-attention applied to images learns convolutional ﬁlters around thequery pixels.

Original: In this study, we perform a detailed analysis on the various experiments outlined in the paper.
Revised: In this study, we perform a detailed analysis on the various experiments outlined in the paper.

Original: Due to the emerging hypein utilizing attention methods for various vision tasks, we compare differences between various other works which tryto utilize self-attention operation as the basic building block for image recognition models.
Revised: We observe certaindifferences from the original paper which lead to interesting observations.

Original: We go beyond verifyingtheir claims  and introduce a new variant of the attention mechanism called Hierarchical Attention (HA) which can beeasily adapted to any existing self-attention model.
Revised: We go beyond verifying the claims by tryingto solve the observed problems and propose a novel attention operation, which we refer to as Hierarchical Attention(HA).

Original: Detailed experiments show that the proposed method signiﬁcantlyoutperforms existing self-attention variants with roughly 1 / 5 th the number of parameters.
Revised: Detailed experiments show that the proposed method signiﬁcantly outperforms existing self-attention variantswith roughly 1 / 5 th the number of parameters and can be easily adapted to any existing attention-based architecture.

Original: 1.1 Outline of this report
Revised: 1.1 Outline of this report

Original: We structure the report as follows:
Revised: We structure the report as follows:

Original: Section. 2 formally introduces the attention operation, followed by the fundamental working principle of the
Revised: Section 2 formally introduces the attention operation, followed by the fundamental working principle of the

Original: transformer.
Revised: transformer.

Original: We validate the claims made by the paper (Cordonnier et.
Revised: We validate the claims made by the paper in Section 3.

Original: al.
Revised: We

Original: [2020]) in Section. In Section. 4, we introduce other recent works who have tried to completely replace convolutions with self-attention and highlight differences between them.
Revised: visualize the attention patterns of the suggestedmodels and compare with those mentioned in the paper, commenting on any similarities or differences.3.

Original: During our experiments, we observe a ﬂaw in the existing attention operation and introduce a novel Hierarchical Attention operation described in Section.
Revised: We introduce a novel Hierarchical Attention operation described in Section 4.

Original: We compare the modiﬁed operation on various methods andempirically show signiﬁcant performance gains.5.
Revised: We compare the modiﬁedoperation on various methods and empirically show signiﬁcant performance gains.4.

Original: Section. 6 concludes and suggests possible improvements for future research.
Revised: Section 5 concludes and suggests possible improvements for future research.

Original:  Fundamentals
Revised:  Fundamentals

Original: In this section, we introduce the attention operation, ﬁrst in terms of its origin in NLP and how it can be extended forimages.
Revised: In this section, we introduce the attention operation, ﬁrst in terms of its origin in NLP and how it can be extended forimages.

Original: We also provide a short introduction to transformers since most methods described in this report heavily relyon it.
Revised: We also provide a short introduction to transformers since most methods described in this report heavily relyon it.

Original: 2.1 Attention
Revised: 2.1 Attention

Original: Attention has been ﬁrst proposed for NLP, where the goal is to focus on a subset of important words.
Revised: Attention was ﬁrst proposed for NLP, where the goal is to focus on a subset of important words.

Original: Consequently,relations between inputs are highlighted that can be used to capture context and higher-order dependencies.
Revised: Consequently, relationsbetween inputs are highlighted that can be used to capture context and higher-order dependencies.

Original: Theattention matrix A ( . ) indicates a score between N queries Q, and N k keys which indicates which part of the inputsequence to focus on and σ ( . ) is any activation function (generallyσ ( . ) = softmax ( . ) ).
Revised: The attention matrix A ( . ) indicates a score between N queries Q and N k keys, which indicates which part of the input sequence to focus on.σ ( . ) is an activation function (generally softmax ( . ) ).

Original: [Equation]
Revised: [Equation]

Original: To capture the relations among the input sequence, the values V are weighted by the scores from Equation 1.
Revised: To capture the relations among the input sequence, the values V are weighted by the scores from Equation 1.

Original: Therefore,we have
Revised: Therefore,we have

Original: [Equation]
Revised: [Equation]

Original: While in NLP, each element in the sequence corresponds to a word, the same idea is applicable for a sequence of Ndiscrete objects, e.g. pixels.
Revised: While in NLP each element in the sequence corresponds to a word, the same idea is applicable for a sequence of N discrete objects, like pixels of an image.

Original: A key property of the self-attention model described above is that it is equivariant tothe input order, that is, it gives the same output independent of how the N input tokens are shufﬂed.
Revised: A key property of the self-attention model described above is that it isequivariant to the input order, i.e. it gives the same output independent of how the N input tokens are shufﬂed.

Original: This is quiteproblematic in cases where the order actually matters like in images.
Revised: This is quite problematic in cases where the order actually matters like in images.

Original: Hence, a positional encoding is learnt for eachtoken in the sequence and added before the self-attention operation.
Revised: Hence, a positional encoding is learnt foreach token in the sequence and added before the self-attention operation.

Original: Hence, the Q and K vectors are derived from thesummation of input X and positional encoding P .
Revised: Hence, the Q and K vectors are derived fromthe summation of input X and positional encoding P .

Original: 2.2 Transformer Attention
Revised: 2.2 Transformer Attention

Original: The Transformer network is an extension of the attention mechanism from  2 based on the Multi-Head Attention (MHA)operation.
Revised: The Transformer network is an extension of the attention mechanism from Eqn. 2 based on the Multi-Head Attention(MHA) operation.

Original: Rather than computing the attention once, the MHA operation computes it multiple times (heads).
Revised: Rather than computing the attention once, the MHA operation computes it multiple times (heads).

Original: This helpsthe transformer jointly attend to different information derived from each head.
Revised: This helps the transformer jointly attend to different information derived from each head.

Original: The output from each of these heads areconcatenated before projecting onto a ﬁnal output dimension.
Revised: The output from each of theseheads are concatenated before projecting onto a ﬁnal output dimension.

Original: A transformer layer also contains a residual connectionfollowed by a layer normalization and the overall operation can be summarized as:
Revised: A transformer layer also contains a residualconnection followed by a layer normalization. The overall operation can be summarized as:

Original: [Equation]
Revised: [Equation]

Original: 2.3 Positional Encoding for Images
Revised: 2.3 Positional Encoding for Images

Original: There are two types of positional encoding that has been used in transformer-based architectures:the absolute andrelative encoding.
Revised: There are two types of positional encodings used in transformer-based architectures: absolute and relative encoding.

Original: With absolute encodings, a (ﬁxed or learned) vector P p, is assigned to each pixel p . The relativepositional encoding, introduced by [10] considers only the position difference between the query pixel (pixel wecompute the representation of) and the key pixel (pixel we attend) instead of the absolute position of the key pixel.
Revised: Absolute encodings assign a (ﬁxed or learned) vector P p to to every pixel p whereas the relative positional encoding[13] considers only the position difference between the query pixel (pixel we compute the representation of) and thekey pixel (pixel we attend to).

Original: The authors from the paper[11] have elegantly proved how an attention operation mimics a convolution.
Revised: The authors from the paper have elegantly proved how the attention operation mimics a convolution.

Original: The mainresult is the following: A multi-head self-attention layer with N h heads of dimension D h , output dimension D out and arelative positional encoding of dimension D p > = 3 can express any convolutional layer of kernel size √ N h ∗ √ N h andmin ( D h , D out ) output channels . 
Revised: The main resultis the following: A multi-head self-attention layer with N h heads of dimension D h , output dimension D out and arelative positional encoding of dimension D p > = 3 can express any convolutional layer of kernel size √ N h ∗ √ N hand min ( D h , D out ) output channels . In this proposed construction, the attention scores of each head must attend todifferent relative pixel shifts within a kernel (Lemma 1 from the paper).

Revised: The above condition is satisﬁed for the relativepositional encoding referred to as Quadratic Encoding .

Revised: However, experiments suggest that a relative position encodinglearned by a neural network ( Learned Relative Position Encoding ) can also satisfy the conditions of the lemma.

Original: We strongly urge the readers to refer to their paper to get a complete understanding.
Revised: Westrongly urge the readers to refer to the original paper to get a complete understanding.

Original: Experiments for  Reproducibility
Revised:  Reproducibility

Original: The aim of this section is to validate the results claimed by the paper - to examine whether self-attention layers inpractice do actually learn to operate like convolutional layers when trained on  standard image classiﬁcation tasks.
Revised: The aim of this section is to validate the results claimed by the paper - to examine whether self-attention layers inpractice do actually learn to operate like convolutional layers when trained on the standard image classiﬁcation task.

Original: Forall our experiments mentioned in this report, we use the CIFAR10 dataset to benchmark the performance of the modelfor the classiﬁcation task.
Revised: Forall our experiments mentioned in this report, we use the CIFAR10 dataset to benchmark the performance of the model.

Original: 3.1 Dataset - CIFAR
Revised: 3.1 Dataset - CIFAR10

Original: The CIFAR-10 dataset [12] consists of 60,00032x32 colour images  in 10 classes, with 6000 images per class.
Revised: The CIFAR-10 dataset [14] consists of 60,000 colour images of size 32x32 split in 10 classes.

Original: Thereare 50,000 trainingimages and 10,000 test images. The classes are completely mutually exclusive.
Revised: There are 6,000 imagesper class split into 5,000 training and 1,000 validation samples. 3.2 Experiments and Results

Original: The results mentioned in the paper uses a fully-attentional model consisting of 6 multi-head self-attention layers eachwith 9 heads.
Revised: The results mentioned in the paper uses a fully-attentional model consisting of 6 multi-head self-attention layers eachwith 9 heads.

Original: In all the experiments, the input image undergoes a 2x2 down-sampling operation to reduce its size.
Revised: In all the experiments, the input image undergoes a 2x2 down-sampling operation to reduce its size.

Original: Theﬁnal image vector is derived by average-pooling the representations derived from the last layer and then passed to alinear layer for classiﬁcation.
Revised: Theﬁnal image vector is derived by average-pooling the representations derived from the last layer and then passed to alinear layer for classiﬁcation.

Original: Please refer to Table.
Revised: Please refer to Table.

Original: 4  for a detailed list of hyper parameters used in each experiment.
Revised: 4 (Appendix) for a detailed list of hyperparameters used in eachexperiment.

Revised: We closely refer to the ofﬁcial implementation and were able to reproduce all the results within 1% of thereported value.

Original: The paper uses three different implementations of the attention mechanism to completely validate their claims.
Revised: Table. 1 compares the results mentioned in the paper and the ones obtained using our implementation. Fig. 1a visualizes the test accuracy on CIFAR10 at every 10 epochs for each model and it is quite evident that fullyconvolutional networks like ResNet18 tend to converge faster.

Original: We shallnow describe them individually, followed by the results section.
Revised: The following subsections describe these results in detail. Model [Table] Learned

Original: 3.2 Quadratic encoding
Revised: [Table] w/ [Table] Learned embedding w/

Revised: [Table]

Original: Lemma 1 from the paper [11], derives a representation of the attention matrix which can be modelled as an isotropicbivariate Gaussian distribution.
Revised: [Figure] 3.2.1 Quadratic Encoding The authors show that the attention probabilities in the quadratic positional encoding is similar to an isotropic bivariate Gaussian distribution with bounded support.

Original: Hence to validate their claims, all the attention matrices in the model are replacedwith these Gaussian priors.
Revised: Hence to validate their claims, all the attention matrices in the model arereplaced with these Gaussian priors, with learnable parameters to determine the center and width of each attentionhead.

Original: The authors also provide ﬁgures to intuitively prove that self-attention indeed tries tolearnconvolutional ﬁlters around a  query pixel.
Revised: Further, this is extended to a non-isotropic distribution over pixel positions as it might be interesting to see if themodel would learn to attend to such groups of pixels - thus forming unseen representations in CNNs. Fig. 2 visualizesthe attention centers for each head for all the layers and at different epochs. After optimization, we can see that theheads attend to a speciﬁc pixel of the image forming a grid around the query pixel.

Original: 3.3 Learned Relative Position Encoding
Revised: This conﬁrms the intuition thatself-attention applied to images learns convolution-like ﬁlters around the query pixel. Also, it can be seen that theinitial layers (1-2) focus at local patterns while the deeper layers (3-6) attend to larger patterns by positioning the centerof attention further from the queried pixel position. Fig. 2b shows that the network did learn non-isotropic attentionpatterns especially in the last layers.

Original:  Learned Relative Position Encoding (also referred to as Learned embedding) is another experiment where the authorstry to understand therelevance of positional encoding  in fully-attentional models.
Revised: However, there is no performance improvement suggesting that it is not particularlyhelpful in practice. 3.2.2 Learned Relative Position Encoding In this experiment, the authors try to study the positional encoding generally used in fully-attentional models [11].

Original: Theauthors implement a 2D relativepositional encoding scheme where a positional encoding vectoris learned for each row and column pixel shift.
Revised: The positional encoding vector for each row and column pixel shift is learnt. The ﬁnal relative position encodingof a key pixel with a query pixel is derived as the concatenation of row and column shift embeddings.

Original: In oneexperiment (Learned embedding w/o content), the authors completely discard the input data and compute the attentionweights solely with the derived encoding.
Revised: First, theauthors completely discard the input data and compute the attention weights solely with the derived encoding (Learnedembedding w/o content).

Original: In another setting (Learned embedding w/ content), both the positional andcontent based attention information is used which corresponds to a fully-blown stand alone self-attention model. We closely follow the ofﬁcial implementation for reproducing the three embedding schemes and attention mechanisms. The obtained performance is very similar to what has been reported in the paper.
Revised: Fig.

Original: Table 1 compares the results mentionedin the paper and the results obtained using our implementation. Further, we try increasing the number of heads to 16 butthere seems to be no signiﬁcant impact to  the model’s performance (seen in Table. 2). [Table] [Table]
Revised: 3a visualizes the attention probabilities for a given query pixel, conﬁrming the hypothesisthat: even when left to learn positional encoding from randomly initialized vectors, certain self-attention heads learn toattend to individual pixels while the others learn non-localized patterns and long range dependencies.

Original: The performance of the fully-blown self attention model (Learned embedding w/ content) is still quite far away fromstandard convolutional neural networks like ResNets.
Revised: In another setting(Learned embedding w/ content), both the positional and content based attention information is used which correspondsto a full-blown stand alone self-attention model.

Original: Quadratic encoding, Learned embedding w/o content performbetter as information is explicitly given, avoiding the need for the model to learn extra features. We discuss on howfurther improvements can be improved in the later sections of this report (Section. 5). As seen from Fig.
Revised: Fig.

Original: 1, the ResNetmodel tends to converge faster than the attention-based methods. Increasing the number of heads seems to slow downthe convergence even more as the model needs to learn more number of parameters.
Revised: 3b visualizes the attention probabilities for a given query pixel inthis setting and it is interesting to note that even when left to learn the encoding from the data, some attention headsexploit positional information like CNNs while the others focus on the content.

Original: [Figure] L a y e r L a y e r L a y e r
Revised: In Fig.

Original: L a y e r L a y e r L a y e r [Figure]
Revised: 4, we visualize the attention

Original: L a y e r [Figure] L a y e r L a y e r
Revised: E p o c h

Original: L a y e r L a y e r [Figure] Fig.
Revised: E p o c h

Original: 2 visualises the attention probabilities for a given query pixel (in ﬁg: center of the given image).
Revised: E p o c h E p o c h E p o c h E p o c h

Revised: E p o c h

Revised: [Figure]

Revised: E p o c h

Original: We can see that inboth the cases, the attention layers try act like convolutional ﬁlters.
Revised: E p o c h E p o c h E p o c h E p o c h

Original: The ﬁrst few layers (1,2) tend to focus on localpatterns, while deeper layers shift the attention further away from the query pixel.
Revised: E p o c h

Original: One important thing to note is that inthe fully blown attention model (Fig. 2b), the last two layers seem to not contribute much to overall features learnt. Weshall come back to this in Section 5.
Revised: E p o c h [Figure] probabilities averaged across an entire batch of images to understand the focus of each head and remove dependency onthe input image for both the experiments.

Original: Due to therecent hype in replacing convolutions with attention layers, there is a strong requirement to highlightdifferences among these recent methods. Among these works, we choose two popular and unique papers
Revised: Average Attention Visualization : The authors from the original paper visualize the attention probabilities for a singleimage or across a batch images for a speciﬁc query pixel. A single query pixel does not convey information regardingwhere the model focuses on in the entire image and it is not practical to plot individual ﬁgures for every query pixel.

Original: The pairwise self-attention, is essentially a general representation of Eqn.
Revised: Hence, we also visualize the attention probabilities using what we refer to as Average Attention , to identify whichportions of the entire image the model attends to.

Original: It is fundamentally a set operation, doesnot attach stationary weights to speciﬁc locations and  is invariant to permutation and cardinality.
Revised: Given a softmax normalized attention matrix of size N × N , every rowrepresents the relationships between a query pixel and the others. First, every element ( α i,j ) is divided by the row-wisesum to ensure that all the values are in scale.

Original: The paper presents anumber of variants of the pairwise attention that have greater expressive power than dot-product attention. Speciﬁcally,the weight computation does not collapse the channel dimension and allows the feature aggregation to adapt to eachchannel.
Revised: Then the row-wise mean is computed to determine the importance valuefor each pixel. If a pixel is strongly correlated to multiple pixels, the importance value will be higher determiningthat the model has a stronger focus on that given pixel.

Original: It can be mathematically formulated as follows:
Revised: We mathematically describe the operation in Eqn. 4.

Original: [Equation]
Revised: Fig.

Original: Here, i is the spatial index of feature vector x i , δ ( . )
Revised: 5visualizes the average attention for the learned embedding with and without content. In Fig.

Original:  is  the relation function mapped onto another vector by γ ( . ) to createadaptive weight vectors α ( x i , x j ) and then aggregate feature vectors obtained from β ( . ) .
Revised: 5a, since the content data isdiscarded, the model is clearly focusing on positional patterns while in Fig. 5b, the model attends to both positional andcontent information. We visualize additional ﬁgures in Sections. B.1-B.3 (Appendix).

Original: An important point to notehere is that δ can produce vectors of different dimensions when compared to β , allowing for a more expressive weightconstruction. The patch-wise self attention is a variant of the pairwise operation, where x i is replaced by a patch of feature vectorsx R ( i ) which allows the weight vector to incorporate information from all the feature vectors in the patch. The equationsare hence rewritten as:
Revised: [Equation] N

Original: [Equation]
Revised: [Equation]

Revised: L a y e r

Original: However, this operation is no longer permutation or cardinality invariant.
Revised: L a y e r L a y e r L a y e r L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Original: Both the above methods have shown strongperformance in image classiﬁcation speciﬁcally in large datasets (explained more in Section 4.3).
Revised: L a y e r L a y e r L a y e r( [Figure]

Original: One major bottlenecksin this method is the size of the attention matrices leading to huge memory requirements.
Revised: L a y e r L a y e r L a y e r L a y e r

Original: 4.2 Vision Transformer
Revised: L a y e r L a y e r [Figure] L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r(

Revised: [Figure]

Revised: 3.3 Increasing the number of heads

Original: A recent work Vision Transformer, has successfully shown that reliance on convolutions is no longer necessary and apure transformer outperforms all convolution based techniques signiﬁcantly when pre-trained on large amounts of data.
Revised: As per the analogy derived between self-attention and convolutions, the number of heads is directly related to the kernelsize in a convolution operation. Hence, we increase the number of heads from 9 to 16. It is important to note here thatthe unlike the general procedure of setting D h = D out /N h in transformer-based architectures

Revised: , the paper suggests toconcatenate heads of dimension D h

Revised: = D out since the effective number of learned ﬁlters is min ( D h , D out ) .

Original: An image is split into patches which is then projected onto a another representation using a trainable layer and thenpassed through a standard set of transformer operations as described in Eqn. 3.
Revised: Given thelimited compute, we reduced D out from 400 to 256 while increasing the number of heads to 16. As seen in Table. 2there seems to be no signiﬁcant impact to the model’s performance. However the model takes longer to converge due to increased

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r(a

Original: 4.3 Inductive biases of Transformers
Revised: [Figure] embedding w/ [Figure] content

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Original: The above two methods tend to perform really well with sufﬁcient training and yield modest accuracies in small datasets.
Revised: L a y e r(b) Learned embedding w/ content Model Accuracy

Revised: [Table]

Revised: 3.4 Additional Observations

Revised: [Figure]

Revised: 3.4.1 Inductive biases in Transformers

Original:  As mentioned in the Vision Transformer paper, transformers lack inherent biases inherent to CNNs like translationequivariance, two-dimensional neighborhood structure, and hence do not generalize well when trained on insufﬁcientamounts of data.
Revised: [Figure] As seen from the results mentioned in Table. 1, the fully attentional model utilizing learnable embeddings with imagecontent performs poorly when compared to the other methods. As mentioned in the paper [3], transformers lack biasesinherent to CNNs like translation equivariance and retention of 2D neighborhood structure, etc.

Original: Only the Multi-Layer Perceptron (MLP) layers used in these methods are local and translationequivariant, while the self attention layers are global.
Revised: Only the Multi-Layer Perceptron (MLP) layers used in these methods are local and translation equivariant, while the self attention layers areglobal.

Original: Hence its almost equivalent to learning these spatial relationshipsfrom scratch which requires sufﬁcient training.
Revised: Even in the case of NLP, almost 75-90% of the predictions are correct even when the input words are randomlyshufﬂed [15].

Original: This  couldalso be the reason for the improved performance in the caseof Learned embedding w/o content, Quadratic encoding (Table. 2 ), asthey are directly replacing the attention matriceswith positional information.However, this changes completely when provided sufﬁcient training.
Revised: This implies that transformer-based methods do not sufﬁciently capture spatial information even withpositional encodings and require a large amount of training data to do so. This could be the reason for improvedperformance in the case of Learned embedding w/o content and Quadratic embedding as the attention matrices aredirectly replaced with positional information.

Original: Hierarchical Attention
Revised: 3.4.2 Over-expressive power of attention matrices

Original: We now introduce a novel attention operation Hierarchical Attention (HA) which tries to overcome some problems inthe normal attention operation. We try to build an intuition of the problem, introduce the alternate attention method andperform detailed experiments to showcase its superiority. 5.1 The problem

Original: The most important step of the self-attention operation is the generation of the attention matrix of size N ∗ N .
Revised: The most important step of the self-attention operation is the generation of the attention matrix of size N × N .

Original: In NLP,the value of N tends to be small (<100) in most cases as we are dealing with words in a sentence.
Revised: In NLP,the value of N tends to be small (<100) in most cases as we are dealing with words in a sentence.

Original: On the contrary,images when broken down into a sequence of pixels are very large in number, hence creatinga large attention matrix.
Revised: On the contrary,images when broken down result in very long sequences of pixels, hence creating large attention matrices. Therefore,these attention matrices can be sparse and the model has a very high tendency of focusing on very high level information.

Original: This has also been talked about in the case of applying transformers in point clouds where the number of points are very large (>1000).
Revised: This can lead to over-ﬁtting and is talked about in the case of pointclouds 6 where the number of points are very large(>1000). This has also been observed in our experiments.

Original: Section 4.1 in  the paper [11] also mentions that Learned embedding with content is harder to trainwhich was also noticed inour experiments. We believe that this is mainly due to the increased expressive power of theattention matrix when N is a very large number.
Revised: As seen in Figs. 3b, 4b, 5b, the attention heads in the last few2 layers are very sparse and do not capture any information.

Original: Although not mentioned in the paper [13], we would like to highlightthat splitting an image into patches makes the value of N smaller which could possibly validate our intuition.
Revised: This can also be seen in the case of Quadratic encoding(Fig. 2), where certain attention heads focus on "non-intuitive" portions of the image (i.e) a thin strip of pixels or attenduniformly across a large patch of pixels.

Original: A simpleand naive way to overcome this problem is to reduce the number of heads or layers butthe this is not an effectivemethod as the model loses its capacity to learn strong features.  
Revised: A simple and naive way to overcome this problem is to reduce the number ofheads or layers but this is not an effective method as the model loses its capacity to learn strong features.  Hierarchical Attention

Original: As an alternative we  propose a Hierarchical Attention(HA) operation which helps the model iteratively reﬁne its features.
Revised: [Figure] Given the problems described above, we need to propose a method which can avoid the over-expressive nature ofindependent attention heads while still being able to learn and derive strong features from the input image across layers. We now introduce a novel attention operation which we refer to as Hierarchical Attention (HA) operation. In thefollowing sections, we explain the core idea behind the operation and perform detailed experiments to showcase itseffectiveness.

Original: 5.2 Methodology
Revised: 4.1 Methodology

Original:  In all the above mentioned methods, independent self-attention layers are stacked sequentially and the output fromone is passed onto the next to derive the Q , K and V vectors.
Revised: [Figure] In most of the transformer-based methods, independent self-attention layers are stacked sequentially and the outputfrom one is passed onto the next to derive the Q , K and V vectors.

Revised: This allows each attention head to freely attendto speciﬁc features and derive a better representation.

Original: However, in the HA operation, the weights are sharedacross the different layers and only the Q , V vectorsare updated while  K remains the same.
Revised: Deviating from these methods, the HA operation updates onlythe Q , V vectors while the K remains the same after each attention block. Further, the weights are shared acrossthese attention layers inducing the transformer model to iteratively reﬁne its representation across layers.

Original: In this case, the querypixel tries to reﬁne its representation in every layer by focusing onthe corresponding portions from the K vector, andaggregates the required information from the V vector.This helps the model hierarchically learn complex features.
Revised: This canbe considered analogous to an unrolled recurrent neural network (RNN) as we are trying to sequentially improve therepresentation across layers based on the previous hidden state. This helps the model hierarchically learn complexfeatures by focusing on corresponding portions of the K vector, and aggregating required information from the V vector.

Original: Figs. 3, 4 visualize the normal attention and the hierarchical attention operations.
Revised: Figs. 6, 7 visualize the normal attention and the hierarchical attention operations respectively.

Original: For the sake of simplicity, we do notvisualize the entire inner workings of the transformer like residual connections, layer normalization, etc.
Revised: For the sake of simplicity,we do not visualize the entire inner workings of the transformer like residual connections, layer normalization, etc.

Original: Please notethat the HA operation can be easily adapted to any existing attention-based network as described in the next section.Q
Revised: Thisis a very simple yet effective method which can be easily adapted to any existing attention-based network as describedin the next section.

Original: [Figure]
Revised: [Figure]

Original: V
Revised: V

Original: [Figure]
Revised: [Figure]

Original: Q
Revised: Q

Original: K
Revised: K

Original: V
Revised: V

Original: Q
Revised: Q

Original: K
Revised: K

Original: V ... h
Revised: V ... h

Original: Q
Revised: Q

Original: K
Revised: K

Original: V
Revised: V

Original: [Figure]
Revised: [Figure]

Original: h
Revised: h

Original: Q
Revised: Q

Original: K
Revised: K

Original: V
Revised: V

Original: Q
Revised: Q

Original: K
Revised: K

Original: V
Revised: V

Original: Q
Revised: Q

Original: K
Revised: K

Original: V ...  updated while the K remain the same in each layer.
Revised: V ... vectors are updated while the K remain the same in each layer.

Revised: As mentioned earlier, there has been a lot of recent hype in replacing convolutions with attention layers.

Original: 5.3 Results
Revised: Hence, wechoose two other popular and similar papers - "Exploring Self-attention for Image Recognition" [2], "An Image isworth 16x16 words: Transformers for Image Recognition at scale" [3] and apply the proposed HA operation to justifyits effectiveness. For better understanding, we brieﬂy introduce these papers in the following subsections. 4.2 Pairwise and Patchwise self attention (SAN) Introduced by [2], pairwise self-attention is essentially a general representation of Eqn.

Revised: It is fundamentally a setoperation, does not attach stationary weights to speciﬁc locations and is invariant to permutation and cardinality.

Revised: Thepaper presents a number of variants of the pairwise attention that have greater expressive power than dot-productattention.

Revised: Speciﬁcally, the weight computation does not collapse the channel dimension and allows the feature aggregation to adapt to each channel.

Revised: It can be mathematically formulated as follows:

Revised: [Equation]

Revised: Here, i is the spatial index of feature vector x i , δ ( . )

Revised: is the relation function mapped onto another vector by γ ( . ) .

Revised: Theadaptive weight vectors α ( x i , x j ) aggregate feature vectors obtained from β ( . ) .

Revised: An important point to note here is thatδ can produce vectors of different dimensions when compared to β , allowing for a more expressive weight construction.

Revised: The patch-wise self attention is a variant of the pairwise operation, where x i is replaced by a patch of feature vectorsx R ( i ) which allows the weight vector to incorporate information from all the feature vectors in the patch.

Revised: The equationsare hence rewritten as:

Revised: [Equation]

Revised: 4.3 Vision Transformer

Revised: The Vision Transformer [3], has successfully shown that reliance on convolutions is no longer necessary and a puretransformer outperforms all convolution based techniques signiﬁcantly when pre-trained on large amounts of data.

Revised: Inthis method, an image is split into patches which is then projected onto another representation using a trainable layerand then passed through a standard set of transformer operations as described in Eqn. 3.

Revised: 4.4 Results

Original: To validate our intuition and to prove the effectiveness of the proposed method, we incorporate HA in all the methodsdescribed above without modifying the overall structure and compare it with the normal case.
Revised: To validate our intuition and to prove the effectiveness of the proposed method, we incorporate HA in all the methodsdescribed above without modifying the overall structure of the architecture.

Original: Table 3 comparesaccuracy for each model with its corresponding HA variant.
Revised: Table 3 compares accuracy for each modelwith its corresponding HA variant.

Original: We see signiﬁcant improvements in the performance (atleast 5% gain) in each case while being able to reduce the  parameters by almost 1 / 5 th of the original model.
Revised: We see signiﬁcant improvements in the performance (at least 5% gain) in eachcase while being able to reduce the number of parameters to almost 1 / 5 th of the original model.

Revised: As mentioned earlier,transformers require sufﬁcient training to perform equally well as convolution-based architectures.

Revised: When pretrained onlarge datasets (14M-300M images), transformer-based architectures achieve excellent performance and transfer to taskswith fewer datapoints [3].

Revised: However, for all our experiments in this report we only focus on training these models fromscratch on the CIFAR10 dataset.

Original: We alsovisualize the attention probabilities for the Hierarchical Learned 2D w/ embedding in Fig.
Revised: [Table] [Table] In Fig. 8a, we visualize the attention probabilities for a given query pixel.

Revised: The relationship between self-attention andconvolutions is striking as the model is attending to distinct pixels at a ﬁxed shift from the query pixel reproducing thereceptive ﬁeld of the convolution operation.

Original:  Fig. 
Revised: The initial layers attend to local patterns while the deeper layers focus onlarger patterns positioned further away from the query pixel. Similarly, in Fig. 8b, the attention heads from the last twolayers are no longer sparse and help capture more information. Hence the visual correctness veriﬁes the operation andits increased performance.

Original: 5a represents theattention weights for  a query pixel located at the center of the image, while Fig.
Revised: We also visualize the attention probabilities for SAN, VIT in Sections. B.7-B.8, B.9-B.10(Appendix) respectively. L a y e r L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Original: 5b represents the average focus ofthe model in a given image.
Revised: L a y e r [Figure] L a y e r L a y e r

Original: Further details on how visualizationis done along with corresponding ﬁgures for othermethods are given in the Appendix. 7.2.
Revised: L a y e r L a y e r L a y e r L a y e r(b) Average Attention visualization

Original: We summarize the hierarchical operation as follows:
Revised: We summarize the hierarchical operation as follows:

Original: Enable weight sharing between the layers of the model by reusing the K vector and updating the Q, V vectorsonly.
Revised: Enable weight sharing between the layers of the model by reusing the K vector and updating the Q, V vectors only.

Original: This helps the model progressively extract higher level features.
Revised: This helps the model progressively extract higher level features.

Original: The method of progressive reﬁnement ensures that the attention matrices do not get over expressive.
Revised: The method of progressive reﬁnement ensures that the attention matrices do not get over expressive.

Original: This leadsto a signiﬁcant improvement over the corresponding non-hierarchical cases.[Table] L a y e r
Revised: This leads to a signiﬁcant improvement over the corresponding non-hierarchical cases.

Original: L a y e r L a y e r L a y e r L a y e r
Revised: The total number of parameters in the model is independent of the number of layers and this property helpssigniﬁcantly reduce the number of parameters when compared to the non-hierarchical versions.

Original: L a y e r( [Figure] L a y e r L a y e r
Revised: Also, thishelps make deeper models without worrying about memory constraints.

Original: L a y e r

Original: L a y e r

Original: L a y e r

Original: L a y e r [Figure] and can Even if the model is provided more layers than are necessary, every layer learns to attend to a differentpattern.
Revised: Even if the model is provided more layers than are necessary, every layer learns to attend to a differentpattern.

Original: The new features learnt at every layer add on to those learnt by the previous ones, which provides itscharacteristic hierarchical nature.
Revised: The new features learnt at every layer add on to those learnt by the previous ones, which provides its characteristic hierarchical nature.

Original: By visualizing the attention scores on a test image, we obtain convincingevidence to support this hypothesis.
Revised: By visualizing the attention scores on a test image, we obtain convincingevidence to support this hypothesis.

Original: (Fig. 5 where the model gradually reﬁnes the regions each head attends to,which is in contrary to the case in Fig. 2b)

Original:  Conclusion
Revised:  Conclusion

Original: In this report, we study the application of self-attention for image recognition, speciﬁcally image classiﬁcation.
Revised: In this report, we study the application of self-attention for image recognition, speciﬁcally image classiﬁcation.

Original: We validate the author’s claims by performing detailed experiments on the CIFAR10 dataset.
Revised: Wevalidate the original paper’s claims by performing detailed experiments on the CIFAR10 dataset.

Original: We discuss two recent papers which propose stand-alone attention based models for image classiﬁcation.
Revised: We discuss tworecent papers which propose stand-alone attention based models for image classiﬁcation.

Original: Finally, we introduce a novelHierarchical
Revised: Finally, we introduce a novel

Original:  Attention operation which signiﬁcantly improves performance across various models while also reducing the number ofparameters.
Revised: Hierarchical Attention operation which signiﬁcantly improves performance across various models while also reducingthe number of parameters.

Original: These preliminary results raises various questions like: do we actually need multiple independent layers inlarge transformers?, does this improved performance also translate to large datasets and across various other image recognition tasks like object detection, image segmentation etc?
Revised: The preliminary results raises various questions: do we actually need multiple independentlayers in large transformers? Does this improved performance also translate to large datasets and across various otherimage recognition tasks like object detection and image segmentation?

Original: We would like to answer all these questions and provide a more rigorous understanding of the proposed method 
Revised: We would like to answer all these questions andprovide a more rigorous understanding of the proposed method in the future.

Original: Appendix7.1 Hyperparameters
Revised: Appendix

Revised: A Hyperparameters

Original: [Table]
Revised: [Table]

Original: 7.2 Attention Visualization
Revised: B Attention Visualization

Original: [Table]
Revised: [Figure]

Original: The attention maps included in this report are visualized in two ways - attention probabilities for a given query in animage, average attention to determine the overall focus of the model in the given image.
Revised: We present more examples for visualising the attention probabilities in various models.

Original: Given an N ∗ N attentionmatrix, every row/column (based on the softmax normalization axis) represents the relationships between a query pixeland the others.
Revised: [Figure] L a y e r L a y e r L a y e r

Original: To visualize the average attention, every element is ﬁrst divided by its row-wise sum to ensure that allvalues are in scale. Then the mean of the attention probabilities in each column is computed to determine an importancevalue for each pixel. If a point is strongly related to multiple points, the importance value will be higher determiningthat the model has a stronger focus on that given pixel. We now visualize the attention maps for different methods forboth the hierarchical and non-hierarchical case.
Revised: L a y e r

Original: L a y e r 
Revised: L a y e r L a y e r [Figure] [Figure]

Revised: [Figure]

Original: L a y e r 
Revised: L a y e r L a y e r L a y e r L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Original: L a y e r 
Revised: L a y e r [Figure] L a y e r L a y e r

Original: L a y e r 
Revised: L a y e r L a y e r [Figure] [Figure]

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Revised: [Figure]

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Revised: [Figure]

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Revised: [Figure]

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Revised: [Figure]

Revised: [Figure]

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Revised: [Figure]

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Revised: [Figure]

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]