Original: [Re]: On the Relationship between Self-Attention and Convolutional Layers
Revised: [Re]: On the Relationship between Self-Attention and Convolutional Layers

Original: Anonymous Author(s) AfﬁliationAddress email Scope of Reproducibility
Revised: Abstract

Original: In this report, we perform a detailed study on the paper "On the Relationship between Self-Attention and Convolutional Layers" [1] which provides theoretical and experimental evidence that self attention layers can behave like convolutionallayers.
Revised: In this report, we perform a detailed study on the paper "On the Relationship between Self-Attention and Convolutional Layers", which provides theoretical and experimental evidence that self attention layers can behave like convolutional layers.

Original: The proposed method does not obtain state-of-the-art performance but rather answers an interesting question - doself-attention layers process images in a similar manner to convolutional layers?.
Revised: The proposed method does not obtain state-of-the-art performance but rather answers an interesting question - do self-attention layers process images in a similar manner to convolutional layers?

Original: This has inspired many recent workslike [2, 3] which proposea fully-attentional model for image recognition.
Revised: This has inspired many recent works which propose fully-attentional models for image recognition.

Original: We focus on experimentally validating theclaims of the original paper, highlight differences with other similar works, and  propose a new variant of the attentionoperation - Hierarchical Attentionwhich shows improved performance with signiﬁcantly lesser parameters. To facilitatefurther study, all the code used in our experiments are publicly available here 1 .Methodology We implement the original paper [1] from scratch in Pytorch and refer to the author’s source code 2 for veriﬁcation.
Revised: We focus on experimentally validating the claims of the original paper and our inferences from the results led us to propose a new variant of the attention operation - Hierarchical Attention .

Original: Inour experiments involving SAN [2], we utilize the ofﬁcial implementation 3 due to the available faster CUDA kernelswhile we implement VIT [3] from scratch referring to the author’s source code 4 .

Original: We then incorporate our proposedhierarchical operation in all three methods for comparison.

Original: For all our experiments mentioned in this report, we usethe CIFAR10 dataset to benchmark the performance of the model in the image classiﬁcation task.

Original: Each training runrequired around 20 hours while the corresponding hierarchical versions took around 10-12 hours for convergence in an

Original: Nvidia RTX 2060 GPU. Results We were able to reproduce all the results from the paper within 1% of the reported value, hence validating the claimsof the original paper. However, there seems to be some differences in the attention ﬁgures which lead to interestinginsights and the proposed Hierarchical Attention.
Revised: The proposed method shows signiﬁcantly improved performance with fewer parameters, hence validating our hypothesis.

Original: In the case of VIT and SAN, we do not have a comparative baselineas the corresponding papers do not evaluate performance on the CIFAR10 dataset (without pre-training).

Original: What was easy

Original: We did not face any major challenges in reproducing the results in the paper What was difﬁcult Most of the code  in the ofﬁcial implementation seems to be borrowed from HuggingFace’s repository 5 which also brought along a lot of unnecessary code making it difﬁcult to read and understand quickly. Further the training time for each run is quite signiﬁcant making it difﬁcult for us to experiment with multiple datasets and hyperparameter settings
Revised: To facilitate further study, all the code used in our experiments are publicly available here 1 .

Original: Communication with original authors

Original: We have tried contacting the authors regarding the differences in the attention ﬁgures since the code for the same was not available on the repository for veriﬁcation.

Original: However, we have not received any response regarding the same

Original:  Introduction
Revised:  Introduction

Original: In computer vision, convolutional architectures [4, 5, 6, 7] have dominated across various image recognition taskslike classiﬁcation, segmentation, etc.
Revised: In computer vision, convolutional architectures [1, 2, 3, 4] have dominated across various image recognition tasks like classiﬁcation, segmentation etc.

Original: However, they have some limitations like lack of rotation invariance, inability toaggregate information based on the image content, etc.
Revised: However, they have some limitations like lack of rotation invariance, inability to aggregate information based on the image content, etc.

Original: This has inspired researchers to explore a different design spaceand introduce models with interesting new capabilities.
Revised: This has inspired researchers to explore a different design space and introduce models with interesting new capabilities.

Original: Self-attention based networks, in particular Transformers [8]have become the model of choice for various Natural Language Processing (NLP) tasks.
Revised: Self-attention based networks, in particular Transformers [5] have become the model of choice for various Natural Language Processing (NLP) tasks.

Original: The major difference betweentransformers and previous methods, such as recurrent neural networks and Convolutional Neural Networks (CNN), isthat the former can simultaneously learn to attend to various parts of the input sequence.
Revised: The major difference between transformers and previous methods, such as recurrent neural networks and Convolutional Neural Networks (CNN), is that the former can simultaneously learn to attend to various parts of the input sequence.

Original: To utilize this capacity to learnmeaningful inter-dependencies, many recent works have tried to incorporate self-attention [9, 10], some even replacingconvolutions entirely [11, 12], in networks for vision tasks.
Revised: To utilize this capacity to learn meaningful inter-dependencies, many recent works have tried to incorporate self-attention [6, 7], some even replacing convolutions entirely [8, 9], in networks for vision tasks.

Original: The main highlights of the paper "On the Relationshipbetween Self-Attention and Convolutional Layers" [1] are:
Revised: The main highlights of the paper "On the Relationship between Self-Attention and Convolutional Layers" [10] are:

Original: Theoretical proof that self-attention layers can behave similar to convolution layers.
Revised: Theoretical proof that self-attention layers can behave similar to convolution layers.

Original: Empirical validation of the performance of the self-attention model on the image classiﬁcation task using theCIFAR10 dataset.
Revised: Empirical validation of the performance of the self-attention model on the image classiﬁcation task using the CIFAR10 dataset.

Original: Visual evidence that self-attention applied to images learns convolutional ﬁlters around thequery pixels.
Revised: Visual evidence that self-attention applied to images learns convolutional ﬁlters around the query pixels.

Original: In this study, we perform a detailed analysis on the various experiments outlined in the paper.
Revised: In this study, we perform a detailed analysis on the various experiments outlined in the paper.

Original: We observe certaindifferences from the original paper which lead to interesting observations.
Revised: We observe certain differences from the original paper which lead to interesting observations.

Original: We go beyond verifying the claims by tryingto solve the observed problems and propose a novel attention operation, which we refer to as Hierarchical Attention(HA).
Revised: We go beyond verifying the claims by trying to solve the observed problems and propose a novel attention operation, which we refer to as Hierarchical Attention (HA).

Original: Detailed experiments show that the proposed method signiﬁcantly outperforms existing self-attention variantswith roughly 1 / 5 th the number of parameters and can be easily adapted to any existing attention-based architecture.
Revised: We incorporate HA in various existing architectures [10, 11, 12] and our detailed experiments suggest signiﬁcantly improved performance ( ≈ 5% ) with roughly 1 / 5 th the number of parameters.

Original: 1.1 Outline of this report
Revised: 1.1 Outline of this report

Original: We structure the report as follows:
Revised: We structure the report as follows:

Original: Section 2 formally introduces the attention operation, followed by the fundamentalworking principle of the transformer.
Revised: Section 2 formally introduces the attention operation, followed by the fundamental principle of the transformer.

Original: We validate the claims made by the paper in Section 3.
Revised: We validate the claims made by the paper in Section 3.

Original: We visualize the attention patterns of the suggestedmodels and compare with those mentioned in the paper, commenting on any similarities or differences.3.
Revised: We visualize the attention patterns of the suggested models and compare with those mentioned in the paper, commenting on any similarities or differences.

Original: We introduce a novel Hierarchical Attention operation described in Section 4.
Revised: We introduce a novel Hierarchical Attention operation described in Section 4.

Original: We compare the modiﬁedoperation on various methods and empirically show signiﬁcant performance gains.4.
Revised: We compare the modiﬁed operation on various methods and empirically show signiﬁcant performance gains.

Original: Section 5 concludes and suggests possible improvements for future research.
Revised: Section 5 concludes and suggests possible improvements for future research.

Original:  Fundamentals
Revised:  Fundamentals

Original: In this section, we introduce the attention operation, ﬁrst in terms of its origin in NLP and how it can be extended forimages.
Revised: In this section, we introduce the attention operation, ﬁrst in terms of its origin in NLP and how it can be extended for images.

Original: We also provide a short introduction to transformers since most methods described in this report heavily relyon it.
Revised: We also provide a short introduction to transformers since most methods described in this report heavily rely on it.

Original: 2.1 Attention
Revised: 2.1 Attention

Original: Attention was ﬁrst proposed for NLP, where the goal is to focus on a subset of important words.
Revised: Attention was ﬁrst proposed for NLP, where the goal is to focus on a subset of important words.

Original: Consequently, relationsbetween inputs are highlighted that can be used to capture context and higher-order dependencies.
Revised: Consequently, relations between inputs are highlighted that can be used to capture context and higher-order dependencies.

Original: The attention matrix
Revised: The attention matrix

Original: A ( . ) indicates a score between N queries Q and N k keys, which indicates which part of the input sequence to focus on.σ ( . ) is an activation function (generally softmax ( . ) ).
Revised: A ( . ) indicates a score between N queries Q and N k keys, which indicates which part of the input sequence to focus on. σ ( . ) is an activation function (generally softmax ( . ) ).

Original: [Equation]
Revised: [Equation]

Original: To capture the relations among the input sequence, the values V are weighted by the scores from Equation 1.
Revised: To capture the relations among the input sequence, the values V are weighted by the scores from Equation 1.

Original: Therefore,we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: While in NLP each element in the sequence corresponds to a word, the same idea is applicable for a sequence of
Revised: While in NLP each element in the sequence corresponds to a word, the same idea is applicable for a sequence of

Original: N discrete objects, like pixels of an image.
Revised: N discrete objects, like pixels of an image.

Original: A key property of the self-attention model described above is that it isequivariant to the input order, i.e. it gives the same output independent of how the N input tokens are shufﬂed.
Revised: A key property of the self-attention model described above is that it is equivariant to the input order, i.e. it gives the same output independent of how the N input tokens are shufﬂed.

Original: This is quite problematic in cases where the order actually matters like in images.
Revised: This is quite problematic in cases where the order actually matters like in images.

Original: Hence, a positional encoding is learnt foreach token in the sequence and added before the self-attention operation.
Revised: Hence, a positional encoding is learnt for each token in the sequence and added before the self-attention operation.

Original: Hence, the Q and K vectors are derived fromthe summation of input X and positional encoding P .
Revised: Hence, the Q and K vectors are derived from the summation of input X and positional encoding P .

Original: 2.2 Transformer Attention
Revised: 2.2 Transformer Attention

Original: The Transformer network is an extension of the attention mechanism from Eqn.
Revised: The Transformer network is an extension of the attention mechanism from Eqn.

Original: 2 based on the Multi-Head Attention(MHA) operation.
Revised: 2 based on the Multi-Head Attention (MHA) operation.

Original: Rather than computing the attention once, the MHA operation computes it multiple times (heads).
Revised: Rather than computing the attention once, the MHA operation computes it multiple times (heads).

Original: This helps the transformer jointly attend to different information derived from each head.
Revised: This helps the transformer jointly attend to different information derived from each head.

Original: The output from each of theseheads are concatenated before projecting onto a ﬁnal output dimension.
Revised: The output from each of these heads are concatenated before projecting onto a ﬁnal output dimension.

Original: A transformer layer also contains a residualconnection followed by a layer normalization.
Revised: A transformer layer also contains a residual connection followed by a layer normalization.

Original: The overall operation can be summarized as:
Revised: The overall operation can be summarized as:

Original: [Equation]
Revised: [Equation]

Original: 2.3 Positional Encoding for Images
Revised: 2.3 Positional Encoding for Images

Original: There are two types of positional encodings used in transformer-based architectures: absolute and relative encoding.
Revised: There are two types of positional encodings used in transformer-based architectures: absolute and relative encoding.

Original: Absolute encodings assign a (ﬁxed or learned) vector P p to to every pixel p whereas the relative positional encoding[13] considers only the position difference between the query pixel (pixel we compute the representation of) and thekey pixel (pixel we attend to).
Revised: Absolute encodings assign a (ﬁxed or learned) vector P p to to every pixel p whereas the relative positional encoding [13] considers only the position difference between the query pixel (pixel we compute the representation of) and the key pixel (pixel we attend to).

Original: The authors from the paper have elegantly proved how the attention operation mimics a convolution.
Revised: The authors from the paper have elegantly proved how the attention operation mimics a convolution.

Original: The main resultis the following: A multi-head self-attention layer with N h heads of dimension D h , output dimension D out and arelative positional encoding of dimension D p > = 3 can express any convolutional layer of kernel size √ N h ∗ √ N hand min ( D h , D out ) output channels .
Revised: The main result is the following: A multi-head self-attention layer with N h heads of dimension D h , output dimension D out and a relative positional encoding of dimension D p > = 3 can express any convolutional layer of kernel size √ N h ∗ √ N h and min ( D h , D out ) output channels .

Original: In this proposed construction, the attention scores of each head must attend todifferent relative pixel shifts within a kernel (Lemma 1 from the paper).
Revised: In this proposed construction, the attention scores of each head must attend to different relative pixel shifts within a kernel (Lemma 1 from the paper).

Original: The above condition is satisﬁed for the relativepositional encoding referred to as Quadratic Encoding .
Revised: The above condition is satisﬁed for the relative positional encoding referred to as Quadratic Encoding .

Original: However, experiments suggest that a relative position encodinglearned by a neural network ( Learned Relative Position Encoding ) can also satisfy the conditions of the lemma.
Revised: However, experiments suggest that a relative position encoding learned by a neural network ( Learned Relative Position Encoding ) can also satisfy the conditions of the lemma.

Original: Westrongly urge the readers to refer to the original paper to get a complete understanding.
Revised: We strongly urge the readers to refer to the original paper to get a complete understanding.

Original:  Reproducibility
Revised:  Reproducibility

Original: The aim of this section is to validate the results claimed by the paper - to examine whether self-attention layers inpractice do actually learn to operate like convolutional layers when trained on the standard image classiﬁcation task.
Revised: The aim of this section is to validate the results claimed by the paper - to examine whether self-attention layers in practice do actually learn to operate like convolutional layers when trained on the standard image classiﬁcation task.

Original: Forall our experiments mentioned in this report, we use the CIFAR10 dataset to benchmark the performance of the model.
Revised: For all our experiments mentioned in this report, we use the CIFAR10 dataset to benchmark the performance of the model.

Revised: We implement the original paper from scratch in PyTorch and refer to the author’s source code for veriﬁcation 2 .

Original: 3.1 Dataset - CIFAR10 
Revised: 3.1 Dataset - CIFAR

Original: The CIFAR-10 dataset [14] consists of 60,000 colour images of size 32x32 split in 10 classes.
Revised: The CIFAR-10 dataset [14] consists of 60,000 colour images of size 32x32 split in 10 classes.

Original: There are 6,000 imagesper class split into 5,000 training and 1,000 validation samples.
Revised: There are 6,000 images per class split into 5,000 training and 1,000 validation samples.

Original: 3.2 Experiments and Results
Revised: 3.2 Computational Requirements

Revised: Experiments involving smaller models, namely ResNet18 and Quadratic embedding, were trained on an 8GB NVIDIA

Revised: RTX 2060 GPU.

Revised: Learned embedding-based models were trained on 16GB NVIDIA V100 virtual GPUs rented from

Revised: Amazon Web Services (AWS).

Revised: Each training run for the smaller models required around 20 hours while the larger experiments took over 2 days for convergence.

Revised: 3.3 Experiments and Results

Original: The results mentioned in the paper uses a fully-attentional model consisting of 6 multi-head self-attention layers eachwith 9 heads.
Revised: The results mentioned in the paper uses a fully-attentional model consisting of 6 multi-head self-attention layers each with 9 heads.

Original: In all the experiments, the input image undergoes a 2x2 down-sampling operation to reduce its size.
Revised: In all the experiments, the input image undergoes a 2x2 down-sampling operation to reduce its size.

Original: Theﬁnal image vector is derived by average-pooling the representations derived from the last layer and then passed to alinear layer for classiﬁcation.
Revised: The ﬁnal image vector is derived by average-pooling the representations derived from the last layer and then passed to a linear layer for classiﬁcation.

Original: Please refer to Table.
Revised: Please refer to Table.

Original: 4 (Appendix) for a detailed list of hyperparameters used in eachexperiment.
Revised: 4 (Appendix) for a detailed list of hyperparameters used in each experiment.

Original: We closely refer to the ofﬁcial implementation and were able to reproduce all the results within 1% of thereported value.
Revised: We closely refer to the ofﬁcial implementation and were able to reproduce all the results within 1% of the reported value.

Original: Table.
Revised: Table.

Original: 1 compares the results mentioned in the paper and the ones obtained using our implementation.
Revised: 1 compares the results mentioned in the paper and the ones obtained using our implementation.

Original: Fig.
Revised: Fig.

Original: 1a visualizes the test accuracy on CIFAR10 at every 10 epochs for each model and it is quite evident that fullyconvolutional networks like ResNet18 tend to converge faster.
Revised: 1a visualizes the test accuracy on CIFAR10 at every 10 epochs for each model and it is quite evident that fully convolutional networks like ResNet18 tend to converge faster.

Original: The following subsections describe these results in detail.
Revised: The following subsections describe these results in detail.

Original: Model

Original: [Table]
Revised: [Table]

Original: Learned

Original: [Table]
Revised: [Figure]

Original: w/ [Table] Learned embedding w/ [Table]
Revised: T e s t a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: 3.2.1 Quadratic Encoding
Revised: 3.3.1 Quadratic Encoding

Original: The authors show that the attention probabilities in the quadratic positional encoding is similar to an isotropic bivariate
Revised: The authors show that the attention probabilities in the quadratic positional encoding is similar to an isotropic bivariate

Original: Gaussian distribution with bounded support.
Revised: Gaussian distribution with bounded support.

Original: Hence to validate their claims, all the attention matrices in the model arereplaced with these Gaussian priors, with learnable parameters to determine the center and width of each attentionhead.
Revised: Hence to validate their claims, all the attention matrices in the model are replaced with these Gaussian priors, with learnable parameters to determine the center and width of each attention head.

Original: Further, this is extended to a non-isotropic distribution over pixel positions as it might be interesting to see if themodel would learn to attend to such groups of pixels - thus forming unseen representations in CNNs.
Revised: Further, this is extended to a non-isotropic distribution over pixel positions as it might be interesting to see if the model would learn to attend to such groups of pixels - thus forming unseen representations in CNNs.

Original: Fig.
Revised: Fig.

Original: 2 visualizesthe attention centers for each head for all the layers and at different epochs.
Revised: 2 visualizes the attention centers for each head for all the layers and at different epochs.

Original: After optimization, we can see that theheads attend to a speciﬁc pixel of the image forming a grid around the query pixel.
Revised: After optimization, we can see that the heads attend to a speciﬁc pixel of the image forming a grid around the query pixel.

Original: This conﬁrms the intuition thatself-attention applied to images learns convolution-like ﬁlters around the query pixel.
Revised: This conﬁrms the intuition that self-attention applied to images learns convolution-like ﬁlters around the query pixel.

Original: Also, it can be seen that theinitial layers (1-2) focus at local patterns while the deeper layers (3-6) attend to larger patterns by positioning the centerof attention further from the queried pixel position.
Revised: Also, it can be seen that the initial layers (1-2) focus at local patterns while the deeper layers (3-6) attend to larger patterns by positioning the center of attention further from the queried pixel position.

Original: Fig.
Revised: Fig.

Original: 2b shows that the network did learn non-isotropic attentionpatterns especially in the last layers.
Revised: 2b shows that the network did learn non-isotropic attention patterns especially in the last layers.

Original: However, there is no performance improvement suggesting that it is not particularlyhelpful in practice. 
Revised: However, there is no performance improvement suggesting that it is not particularly helpful in practice. E p o c h E p o c h E p o c h

Original: 3.2.2 Learned Relative Position Encoding
Revised: E p o c h E p o c h E p o c h E p o c h

Revised: [Figure]

Revised: p

Revised: [Figure]

Revised: c

Revised: [Figure]

Revised: E p o c h

Revised: [Figure]

Revised: E p o c h

Revised: [Figure]

Revised: E p o c h

Revised: [Figure]

Revised: p

Revised: [Figure]

Revised: c h

Revised: [Figure]

Revised: c h

Revised: [Figure]

Revised: h

Revised: [Figure]

Revised: 3.3.2 Learned Relative Position Encoding

Original: In this experiment, the authors try to study the positional encoding generally used in fully-attentional models [11].
Revised: In this experiment, the authors try to study the positional encoding generally used in fully-attentional models [8].

Original: The positional encoding vector for each row and column pixel shift is learnt.
Revised: The positional encoding vector for each row and column pixel shift is learnt.

Original: The ﬁnal relative position encodingof a key pixel with a query pixel is derived as the concatenation of row and column shift embeddings.
Revised: The ﬁnal relative position encoding of a key pixel with a query pixel is derived as the concatenation of row and column shift embeddings.

Original: First, theauthors completely discard the input data and compute the attention weights solely with the derived encoding (Learnedembedding w/o content).
Revised: First, the authors completely discard the input data and compute the attention weights solely with the derived encoding (Learned embedding w/o content).

Original: Fig.
Revised: Fig.

Original: 3a visualizes the attention probabilities for a given query pixel, conﬁrming the hypothesisthat: even when left to learn positional encoding from randomly initialized vectors, certain self-attention heads learn toattend to individual pixels while the others learn non-localized patterns and long range dependencies.
Revised: 3a visualizes the attention probabilities for a given query pixel, conﬁrming the hypothesis that: even when left to learn positional encoding from randomly initialized vectors, certain self-attention heads learn to attend to individual pixels while the others learn non-localized patterns and long range dependencies.

Original: In another setting(Learned embedding w/ content), both the positional and content based attention information is used which correspondsto a full-blown stand alone self-attention model.
Revised: In another setting (Learned embedding w/ content), both the positional and content based attention information is used which corresponds to a full-blown stand alone self-attention model.

Original: Fig.
Revised: Fig.

Original: 3b visualizes the attention probabilities for a given query pixel inthis setting and it is interesting to note that even when left to learn the encoding from the data, some attention headsexploit positional information like CNNs while the others focus on the content.
Revised: 3b visualizes the attention probabilities for a given query pixel in this setting and it is interesting to note that even when left to learn the encoding from the data, some attention heads exploit positional information like CNNs while the others focus on the content.

Original: In Fig.
Revised: In Fig.

Original: 4, we visualize the attention 
Revised: 4, we visualize the attention probabilities averaged across an entire batch of images to understand the focus of each head and remove dependency on the input image for both the experiments.

Original: E p o c h E p o c h
Revised: L a y e r L a y e r

Original: E p o c h E p o c h E p o c h
Revised: L a y e r L a y e r

Original: E p o c h E p o c h
Revised: L a y e r L a y e r(

Original: [Figure]
Revised: [Figure]

Original: E p o c h E p o c h
Revised: embedding

Original: E p o c h
Revised: [Figure] o content

Original: E p o c h E p o c h
Revised: L a y e r L a y e r

Original: E p o c h E p o c h
Revised: L a y e r L a y e r

Original:  [Figure]probabilities averaged across an entire batch of images to understand the focus of each head and remove dependency onthe input image for both the experiments.
Revised: L a y e r L a y e r [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Original:  Average Attention Visualization : The authors from the original paper visualize the attention probabilities for a singleimage or across a batch images for a speciﬁc query pixel.
Revised: L a y e r L a y e r( [Figure] Average Attention Visualization : The authors from the original paper visualize the attention probabilities for a single image or across a batch images for a speciﬁc query pixel.

Original: A single query pixel does not convey information regardingwhere the model focuses on in the entire image and it is not practical to plot individual ﬁgures for every query pixel.
Revised: A single query pixel does not convey information regarding where the model focuses on in the entire image and it is not practical to plot individual ﬁgures for every query pixel.

Original: Hence, we also visualize the attention probabilities using what we refer to as Average Attention , to identify whichportions of the entire image the model attends to.
Revised: Hence, we also visualize the attention probabilities using what we refer to as Average Attention , to identify which portions of the entire image the model attends to.

Original: Given a softmax normalized attention matrix of size N × N , every rowrepresents the relationships between a query pixel and the others.
Revised: Given a softmax normalized attention matrix of size N × N , every row represents the relationships between a query pixel and the others.

Original: First, every element ( α i,j ) is divided by the row-wisesum to ensure that all the values are in scale.
Revised: First, every element ( α i,j ) is divided by the row-wise sum to ensure that all the values are in scale.

Original: Then the row-wise mean is computed to determine the importance valuefor each pixel.
Revised: Then the row-wise mean is computed to determine the importance value for each pixel.

Original: If a pixel is strongly correlated to multiple pixels, the importance value will be higher determiningthat the model has a stronger focus on that given pixel.
Revised: If a pixel is strongly correlated to multiple pixels, the importance value will be higher determining that the model has a stronger focus on that given pixel.

Original: We mathematically describe the operation in Eqn. 4.
Revised: We mathematically describe the operation in Eqn. 4.

Original: Fig. 5visualizes the average attention for the learned embedding with and without content.
Revised: Fig.visualizes the average attention for the learned embedding with and without content.

Original: In Fig.
Revised: In Fig.

Original: 5a, since the content data isdiscarded, the model is clearly focusing on positional patterns while in Fig.
Revised: 5a, since the content data is discarded, the model is clearly focusing on positional patterns while in Fig.

Original: 5b, the model attends to both positional andcontent information.
Revised: 5b, the model attends to both positional and content information.

Original: We visualize additional ﬁgures in Sections.
Revised: We visualize additional ﬁgures in Sections.

Original: B.1-B.3 (Appendix).
Revised: B.1-B.3 (Appendix).

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e rL a y e r L a y e r L a y e r
Revised: L a y e r

Original: [Figure]

Original: L a y e r
Revised: L a y e r

Original: L a y e rL a y e r L a y e r L a y e r
Revised: L a y e r

Original: L a y e r(
Revised: L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e rL a y e r L a y e r L a y e r
Revised: L a y e r

Original: [Figure]

Original: L a y e r
Revised: L a y e r

Original: L a y e rL a y e r L a y e r L a y e r
Revised: L a y e r

Original: L a y e r(
Revised: L a y e r

Original: [Figure]
Revised: [Figure]

Original: 3.3 Increasing the number of heads
Revised: 3.4 Increasing the number of heads

Original: As per the analogy derived between self-attention and convolutions, the number of heads is directly related to the kernelsize in a convolution operation.
Revised: As per the analogy derived between self-attention and convolutions, the number of heads is directly related to the kernel size in a convolution operation.

Original: Hence, we increase the number of heads from 9 to 16.
Revised: Hence, we increase the number of heads from 9 to 16.

Original: It is important to note here thatthe unlike the general procedure of setting D h
Revised: It is important to note here that the unlike the general procedure of setting D h

Original: = D out /N h in transformer-based architectures , the paper suggests toconcatenate heads of dimension D h
Revised: = D out /N h in transformer-based architectures, the paper suggests to concatenate heads of dimension D h

Original: = D out since the effective number of learned ﬁlters is min ( D h , D out ) .
Revised: = D out since the effective number of learned ﬁlters is min ( D h , D out ) .

Original: Given thelimited compute, we reduced D out from 400 to 256 while increasing the number of heads to 16.
Revised: Given the limited compute, we reduced D out from 400 to 256 while increasing the number of heads to 16.

Original: As seen in Table. 2there seems to be no signiﬁcant impact to the model’s performance.
Revised: As seen in Table.there seems to be no signiﬁcant impact to the model’s performance.

Original: However the model takes longer to converge due to  increased L a y e r L a y e r L a y e r
Revised: However, the model takes longer to converge due to the increased number of parameters (Fig. 1b).

Original: L a y e r

Original: L a y e r

Original: L a y e r(a

Original: [Figure]

Original: embedding w/

Original: [Figure] content L a y e r L a y e r
Revised: We visualize the attention probabilities in Sections.

Original: L a y e r L a y e r L a y e r L a y e r(b)
Revised: B.4-B.6 (Appendix).

Original: Learned embedding w/ content

Original: Model
Revised: [Figure]

Original: Accuracy

Original: [Table]
Revised: [Table]

Original: 3.4 Additional Observations 
Revised: 3.5 Additional Observations 3.5.1 Inductive biases in Transformers

Original: [Figure] 3.4.1 Inductive biases in Transformers [Figure]

Original: As seen from the results mentioned in Table.
Revised: As seen from the results mentioned in Table.

Original: 1, the fully attentional model utilizing learnable embeddings with imagecontent performs poorly when compared to the other methods.
Revised: 1, the fully attentional model utilizing learnable embeddings with image content performs poorly when compared to the other methods.

Original: As mentioned in the paper [3], transformers lack biasesinherent to CNNs like translation equivariance and retention of 2D neighborhood structure, etc.
Revised: As mentioned in the paper [11], transformers lack biases inherent to CNNs like translation equivariance and retention of 2D neighborhood structure, etc.

Original: Only the Multi-Layer
Revised: Only the Multi-Layer

Original: Perceptron (MLP) layers used in these methods are local and translation equivariant, while the self attention layers areglobal.
Revised: Perceptron (MLP) layers used in these methods are local and translation equivariant, while the self attention layers are global.

Original: Even in the case of NLP, almost 75-90% of the predictions are correct even when the input words are randomlyshufﬂed [15].
Revised: Even in the case of NLP, almost 75-90% of the predictions are correct even when the input words are randomly shufﬂed [15].

Original: This implies that transformer-based methods do not sufﬁciently capture spatial information even withpositional encodings and require a large amount of training data to do so.
Revised: This implies that transformer-based methods do not sufﬁciently capture spatial information even with positional encodings and require a large amount of training data to do so.

Original: This could be the reason for improvedperformance in the case of Learned embedding w/o content and Quadratic embedding as the attention matrices aredirectly replaced with positional information.
Revised: This could be the reason for improved performance in the case of Learned embedding w/o content and Quadratic embedding as the attention matrices are directly replaced with positional information.

Original: 3.4.2 Over-expressive power of attention matrices
Revised: 3.5.2 Over-expressive power of attention matrices

Original: The most important step of the self-attention operation is the generation of the attention matrix of size N × N .
Revised: The most important step of the self-attention operation is the generation of the attention matrix of size N × N .

Original: In NLP,the value of N tends to be small (<100) in most cases as we are dealing with words in a sentence.
Revised: In NLP, the value of N tends to be small (<100) in most cases as we are dealing with words in a sentence.

Original: On the contrary,images when broken down result in very long sequences of pixels, hence creating large attention matrices.
Revised: On the contrary, images when broken down result in very long sequences of pixels, hence creating large attention matrices.

Original: Therefore,these attention matrices can be sparse and the model has a very high tendency of focusing on very high level information.
Revised: Therefore, these attention matrices can be sparse and the model has a very high tendency of focusing on very high level information.

Original: This can lead to over-ﬁtting and is talked about in the case of pointclouds 6 where the number of points are very large(>1000).
Revised: This can lead to over-ﬁtting and is talked about in the case of pointclouds 3 where the number of points are very large (>1000).

Original: This has also been observed in our experiments.
Revised: This has also been observed in our experiments.

Original: As seen in Figs.
Revised: As seen in Figs.

Original: 3b, 4b, 5b, the attention heads in the last few2 layers are very sparse and do not capture any information.
Revised: 3b, 4b, 5b, the attention heads in the last few 2 layers are very sparse and do not capture any information.

Original: This can also be seen in the case of Quadratic encoding(Fig. 2), where certain attention heads focus on "non-intuitive" portions of the image (i.e) a thin strip of pixels or attenduniformly across a large patch of pixels.
Revised: This can also be seen in the case of Quadratic encoding (Fig. 2), where certain attention heads focus on "non-intuitive" portions of the image (i.e) a thin strip of pixels or attend uniformly across a large patch of pixels.

Original: A simple and naive way to overcome this problem is to reduce the number ofheads or layers but this is not an effective method as the model loses its capacity to learn strong features.
Revised: A simple and naive way to overcome this problem is to reduce the number of heads or layers but this is not an effective method as the model loses its capacity to learn strong features.

Original:  Hierarchical Attention
Revised:  Hierarchical Attention

Original: [Figure] Given the problems described above, we need to propose a method which can avoid the over-expressive nature ofindependent attention heads while still being able to learn and derive strong features from the input image across layers.
Revised: Given the problems described above, we need to propose a method which can avoid the over-expressive nature of independent attention heads while still being able to learn and derive strong features from the input image across layers.

Original: We now introduce a novel attention operation which we refer to as Hierarchical Attention (HA) operation.
Revised: We now introduce a novel attention operation which we refer to as Hierarchical Attention (HA) operation.

Original: In thefollowing sections, we explain the core idea behind the operation and perform detailed experiments to showcase itseffectiveness.
Revised: In the following sections, we explain the core idea behind the operation and perform detailed experiments to showcase its effectiveness.

Original: 4.1 Methodology
Revised: 4.1 Methodology

Original: [Figure] In most of the transformer-based methods, independent self-attention layers are stacked sequentially and the outputfrom one is passed onto the next to derive the Q , K and V vectors.
Revised: In most of the transformer-based methods, independent self-attention layers are stacked sequentially and the output from one is passed onto the next to derive the Q , K and V vectors.

Original: This allows each attention head to freely attendto speciﬁc features and derive a better representation.
Revised: This allows each attention head to freely attend to speciﬁc features and derive a better representation.

Original: Deviating from these methods, the HA operation updates onlythe Q , V vectors while the K remains the same after each attention block.
Revised: Deviating from these methods, the HA operation updates only the Q , V vectors while the K remains the same after each attention block.

Original: Further, the weights are shared acrossthese attention layers inducing the transformer model to iteratively reﬁne its representation across layers.
Revised: Further, the weights are shared across these attention layers inducing the transformer model to iteratively reﬁne its representation across layers.

Original: This canbe considered analogous to an unrolled recurrent neural network (RNN) as we are trying to sequentially improve therepresentation across layers based on the previous hidden state.
Revised: This can be considered analogous to an unrolled recurrent neural network (RNN) as we are trying to sequentially improve the representation across layers based on the previous hidden state.

Original: This helps the model hierarchically learn complexfeatures by focusing on corresponding portions of the K vector, and aggregating required information from the V vector.
Revised: This helps the model hierarchically learn complex features by focusing on corresponding portions of the K vector, and aggregating required information from the V vector.

Original: Figs. 6, 7 visualize the normal attention and the hierarchical attention operations respectively.
Revised: Figs. 6, 7 visualize the normal attention and the hierarchical attention operations respectively.

Original: For the sake of simplicity,we do not visualize the entire inner workings of the transformer like residual connections, layer normalization, etc.
Revised: For the sake of simplicity, we do not visualize the entire inner workings of the transformer like residual connections, layer normalization, etc.

Original: Thisis a very simple yet effective method which can be easily adapted to any existing attention-based network as describedin the next section.
Revised: This is a very simple yet effective method which can be easily adapted to any existing attention-based network as described in the next section.

Revised: As mentioned earlier, there has been a lot of recent hype in replacing convolutions with attention layers.

Original: [Figure]
Revised: Hence, we choose two other popular and similar papers - "Exploring Self-attention for Image Recognition" [12], "An Image is worth 16x16 words: Transformers for Image Recognition at scale" [11] and apply the proposed HA operation to justify its effectiveness. For better understanding, we brieﬂy introduce these papers in the following subsections. Q

Revised: K

Original: V
Revised: V

Original: [Figure]
Revised: X

Original: Q
Revised: Q

Original: K
Revised: K

Original: V
Revised: V

Original: Q
Revised: Q

Original: K
Revised: K

Original: V ... h
Revised: V ... h

Original: Q
Revised: Q

Original: K
Revised: K

Original: V
Revised: V

Original: [Figure]
Revised: Q K V

Original:  h
Revised: X h

Original: Q
Revised: Q

Original: K
Revised: K

Original: V
Revised: V

Original: Q
Revised: Q

Original: K
Revised: K

Original: V
Revised: V

Original: Q
Revised: Q

Original: K
Revised: K

Original: V ...vectors are updated while the K remain the same in each layer. As mentioned earlier, there has been a lot of recent hype in replacing convolutions with attention layers. Hence, wechoose two other popular and similar papers - "Exploring Self-attention for Image Recognition" [2], "An Image isworth 16x16 words: Transformers for Image Recognition at scale" [3] and apply the proposed HA operation to justifyits effectiveness. For better understanding, we brieﬂy introduce these papers in the following subsections.
Revised: V ...

Original: 4.2 Pairwise and Patchwise self attention (SAN)
Revised: 4.2 Pairwise and Patchwise self attention (SAN)

Original: Introduced by [2], pairwise self-attention is essentially a general representation of Eqn.
Revised: Introduced by [12], pairwise self-attention is essentially a general representation of Eqn.

Original: It is fundamentally a setoperation, does not attach stationary weights to speciﬁc locations and is invariant to permutation and cardinality.
Revised: It is fundamentally a set operation, does not attach stationary weights to speciﬁc locations and is invariant to permutation and cardinality.

Original: Thepaper presents a number of variants of the pairwise attention that have greater expressive power than dot-productattention.
Revised: The paper presents a number of variants of the pairwise attention that have greater expressive power than dot-product attention.

Original: Speciﬁcally, the weight computation does not collapse the channel dimension and allows the feature aggregation to adapt to each channel.
Revised: Speciﬁcally, the weight computation does not collapse the channel dimension and allows the feature aggregation to adapt to each channel.

Original: It can be mathematically formulated as follows:
Revised: It can be mathematically formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, i is the spatial index of feature vector x i , δ ( . )
Revised: Here, i is the spatial index of feature vector x i , δ ( . )

Original: is the relation function mapped onto another vector by γ ( . ) .
Revised: is the relation function mapped onto another vector by γ ( . ) .

Original: Theadaptive weight vectors α ( x i , x j ) aggregate feature vectors obtained from β ( . ) .
Revised: The adaptive weight vectors α ( x i , x j ) aggregate feature vectors obtained from β ( . ) .

Original: An important point to note here is thatδ can produce vectors of different dimensions when compared to β , allowing for a more expressive weight construction.
Revised: An important point to note here is that δ can produce vectors of different dimensions when compared to β , allowing for a more expressive weight construction.

Original: The patch-wise self attention is a variant of the pairwise operation, where x i is replaced by a patch of feature vectorsx R ( i ) which allows the weight vector to incorporate information from all the feature vectors in the patch.
Revised: The patch-wise self attention is a variant of the pairwise operation, where x i is replaced by a patch of feature vectors x R ( i ) which allows the weight vector to incorporate information from all the feature vectors in the patch.

Original: The equationsare hence rewritten as:
Revised: The equations are hence rewritten as:

Original: [Equation]
Revised: [Equation]

Original: 4.3 Vision Transformer 
Revised: 4.3 Vision Transformer (VIT)

Original: The Vision Transformer [3], has successfully shown that reliance on convolutions is no longer necessary and a puretransformer outperforms all convolution based techniques signiﬁcantly when pre-trained on large amounts of data.
Revised: The Vision Transformer [11], has successfully shown that reliance on convolutions is no longer necessary and a pure transformer outperforms all convolution based techniques signiﬁcantly when pre-trained on large amounts of data.

Original: Inthis method, an image is split into patches which is then projected onto another representation using a trainable layerand then passed through a standard set of transformer operations as described in Eqn. 3.
Revised: In this method, an image is split into patches which is then projected onto another representation using a trainable layer and then passed through a standard set of transformer operations as described in Eqn. 3.

Original: 4.4 Results
Revised: 4.4 Results

Original: To validate our intuition and to prove the effectiveness of the proposed method, we incorporate HA in all the methodsdescribed above without modifying the overall structure of the architecture.
Revised: To validate our intuition and to prove the effectiveness of the proposed method, we incorporate HA in all the methods described above without modifying the overall structure of the architecture.

Revised: In our experiments involving SAN, we utilize the ofﬁcial implementation 4 due to the available faster CUDA kernels while we implement VIT from scratch referring to the author’s source code 5 .

Original: Table 3 compares accuracy for each modelwith its corresponding HA variant.
Revised: Table 3 compares accuracy for each model with its corresponding HA variant.

Original: We see signiﬁcant improvements in the performance (at least 5% gain) in eachcase while being able to reduce the number of parameters to almost 1 / 5 th of the original model.
Revised: We see signiﬁcant improvements in the performance (at least 5% gain) in each case while being able to reduce the number of parameters to almost 1 / 5 th of the original model.

Original: As mentioned earlier,transformers require sufﬁcient training to perform equally well as convolution-based architectures.
Revised: As mentioned earlier, transformers require sufﬁcient training to perform equally well as convolution-based architectures.

Original: When pretrained onlarge datasets (14M-300M images), transformer-based architectures achieve excellent performance and transfer to taskswith fewer datapoints [3].
Revised: When pretrained on large datasets (14M-300M images), transformer-based architectures achieve excellent performance and transfer to tasks with fewer datapoints [11].

Original: However, for all our experiments in this report we only focus on training these models fromscratch on the CIFAR10 dataset.
Revised: However, for all our experiments in this report we only focus on training these models from scratch on the CIFAR dataset.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: In Fig.
Revised: In Fig.

Original: 8a, we visualize the attention probabilities for a given query pixel.
Revised: 8a [Table] we visualize the attention probabilities for a given query pixel.

Original: The relationship between self-attention andconvolutions is striking as the model is attending to distinct pixels at a ﬁxed shift from the query pixel reproducing thereceptive ﬁeld of the convolution operation.
Revised: The relationship between self-attention and convolutions is striking as the model is attending to distinct pixels at a ﬁxed shift from the query pixel reproducing the receptive ﬁeld of the convolution operation.

Original: The initial layers attend to local patterns while the deeper layers focus onlarger patterns positioned further away from the query pixel.
Revised: The initial layers attend to local patterns while the deeper layers focus on larger patterns positioned further away from the query pixel.

Original: Similarly, in Fig.
Revised: Similarly, in Fig.

Original: 8b, the attention heads from the last twolayers are no longer sparse and help capture more information.
Revised: 8b, the attention heads from the last two layers are no longer sparse and help capture more information.

Original: Hence the visual correctness veriﬁes the operation andits increased performance.
Revised: Hence the visual correctness veriﬁes the operation and its increased performance.

Original: We also visualize the attention probabilities for SAN, VIT in Sections.
Revised: We also visualize the attention probabilities for SAN, VIT in Sections.

Original: B.7-B.8, B.9-B.10(Appendix) respectively.
Revised: B.7-B.8, B.9-B. (Appendix) respectively.

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r 
Revised: L a y e r L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r
Revised: L a y e r

Original: L a y e r(b) Average Attention visualization
Revised: [Figure]

Original: We summarize the hierarchical operation as follows:
Revised: We summarize the hierarchical operation as follows:

Original: Enable weight sharing between the layers of the model by reusing the K vector and updating the Q, V vectors only.
Revised: Enable weight sharing between the layers of the model by reusing the K vector and updating the Q, V vectors only.

Original: This helps the model progressively extract higher level features.
Revised: This helps the model progressively extract higher level features.

Original: The method of progressive reﬁnement ensures that the attention matrices do not get over expressive.
Revised: The method of progressive reﬁnement ensures that the attention matrices do not get over expressive.

Original: This leads to a signiﬁcant improvement over the corresponding non-hierarchical cases.
Revised: This leads to a signiﬁcant improvement over the corresponding non-hierarchical cases.

Original: The total number of parameters in the model is independent of the number of layers and this property helpssigniﬁcantly reduce the number of parameters when compared to the non-hierarchical versions.
Revised: The total number of parameters in the model is independent of the number of layers and this property helps signiﬁcantly reduce the number of parameters when compared to the non-hierarchical versions.

Original: Also, thishelps make deeper models without worrying about memory constraints.
Revised: Also, this helps make deeper models without worrying about memory constraints.

Original: Even if the model is provided more layers than are necessary, every layer learns to attend to a differentpattern.
Revised: Even if the model is provided more layers than are necessary, every layer learns to attend to a different pattern.

Original: The new features learnt at every layer add on to those learnt by the previous ones, which provides its characteristic hierarchical nature.
Revised: The new features learnt at every layer add on to those learnt by the previous ones, which provides its characteristic hierarchical nature.

Original: By visualizing the attention scores on a test image, we obtain convincingevidence to support this hypothesis.
Revised: By visualizing the attention scores on a test image, we obtain convincing evidence to support this hypothesis.

Original:  Conclusion
Revised:  Conclusion

Original: In this report, we study the application of self-attention for image recognition, speciﬁcally image classiﬁcation.
Revised: In this report, we study the application of self-attention for image recognition, speciﬁcally image classiﬁcation.

Original: Wevalidate the original paper’s claims by performing detailed experiments on the CIFAR10 dataset.
Revised: We validate the original paper’s claims by performing detailed experiments on the CIFAR10 dataset.

Revised: We were able to reproduce all the results from the paper within 1% of the reported value, hence validating the claims of the original paper.

Original: We discuss tworecent papers which propose stand-alone attention based models for image classiﬁcation.
Revised: However, there seems to be some differences in the attention ﬁgures which lead to interesting insights and the proposed Hierarchical Attention.

Original: Finally, we introduce a novel Hierarchical Attention operation which  signiﬁcantly improves performanceacross various models while also reducingthe number of parameters.
Revised: To validate our hypothesis, we perform detailed experiments by incorporating HA with various methods which helps signiﬁcantly improve the performance while reducing the number of parameters.

Original: The preliminary results raises various questions: do we actually need multiple independentlayers in large transformers?
Revised: These preliminary results raise various questions: Do we actually need multiple independent layers in large transformers?

Original: Does this improved performance also translate to large datasets and across various otherimage recognition tasks like object detection and image segmentation?
Revised: Does this improved performance also translate to large datasets and across various other image recognition tasks like object detection and image segmentation?

Original: We would like to answer all these questions andprovide a more rigorous understanding of the proposed method in the future.
Revised: We would like to answer all these questions and provide a more rigorous understanding of the proposed method in the future.

Original: Appendix
Revised: What was easy The original paper is well written, quite complete and mentioned the required details to carry out various experiments mentioned in the paper. What was difﬁcult The authors have possibly ported code from the transformers repository by HuggingFace 6 and left behind lots of unnecessary code making it difﬁcult to follow. We observed major differences in the attention plots from our implementation and those reported in the paper. We attempted to contact the authors to discuss these ﬁndings, but we received no response from them.

Revised: However, we provide reasoning that supports our results, which also led us to conceptualize Hierarchical Attention.

Revised: For computational reasons, our scope was limited to smaller datasets like

Revised: CIFAR10 with limited hyper-parameter study.

Original:  A Hyperparameters
Revised: Appendix A Hyperparameters

Original: [Table]
Revised: [Table]

Original: B Attention Visualization
Revised: B Attention Visualization

Original: [Figure]

Original: We present more examples for visualising the attention probabilities in various models.
Revised: We present more examples for visualising the attention probabilities in various models.

Original: [Figure]
Revised: B.1 Learned embedding w/ content; 9 heads

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Revised: B.2 Learned embedding w/o content; 9 heads

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: B.8 Hierarchical SAN Patchwise

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a ye r
Revised: L a y e r L a y

Revised: [Figure]

Original: L a ye r L a y e r
Revised: y

Original: [Figure]
Revised: [Figure]

Revised: L a y

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: L a y e r
Revised: L a y e r

Original: L a y e rL a y e r L a y e r
Revised: L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: a y

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r L a y e r L a y e r
Revised: r

Original: L a y e r

Original: L a y e r
Revised: L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: [Figure]
Revised: [Figure]