Original: Peripheral Vision Transformer
Revised: Peripheral Vision Transformer

Original: Abstract
Revised: Abstract

Original: Human vision possesses a special type of visual processing systems called peripheral vision .
Revised: Human vision possesses a special type of visual processing systems called peripheral vision .

Original: Partitioning the entire visual field into multiple contour regions based on the distance to the center of our gaze, the peripheral vision provides us the ability to perceive various visual features at different regions.
Revised: Partitioning the entire visual field into multiple contour regions based on the distance to the center of our gaze, the peripheral vision provides us the ability to perceive various visual features at different regions.

Original: In this work, we take a biologically inspired approach and explore to model peripheral vision in deep neural networks for visual recognition.
Revised: In this work, we take a biologically inspired approach and explore to model peripheral vision in deep neural networks for visual recognition.

Original: We propose to incorporate peripheral position encoding to the multi-head self-attention layers to let the network learn to partition the visual field into diverse peripheral regions given training data.
Revised: We propose to incorporate peripheral position encoding to the multi-head self-attention layers to let the network learn to partition the visual field into diverse peripheral regions given training data.

Original: We evaluate the proposed network, dubbed PerViT, on ImageNet-1K and systematically investigate the inner workings of the model for machine perception, showing that the network learns to perceive visual data similarly to the way that human vision does.
Revised: We evaluate the proposed network, dubbed PerViT, on ImageNet-1K and systematically investigate the inner workings of the model for machine perception, showing that the network learns to perceive visual data similarly to the way that human vision does.

Original: The performance improvements in image classification over the baselines across different model sizes demonstrate the efficacy of the proposed method.
Revised: The performance improvements in image classification over the baselines across different model sizes demonstrate the efficacy of the proposed method.

Original:  Introduction
Revised:  Introduction

Original: [Figure]
Revised: [Figure]

Original: For the last ten years, convolution has been a dominant feature transformation in neural networks for visual recognition due to its superiority in modelling spatial configurations of images [21, 32, 34].
Revised: For the last ten years, convolution has been a dominant feature transformation in neural networks for visual recognition due to its superiority in modelling spatial configurations of images [21, 32, 34].

Original: Despite the efficacy in learning visual patterns, the local and stationary nature of convolutional kernels limited the maximum extent of representation ability in flexible processing, e g. , dynamic transformations with global receptive fields.
Revised: Despite the efficacy in learning visual patterns, the local and stationary nature of convolutional kernels limited the maximum extent of representation ability in flexible processing, e g. , dynamic transformations with global receptive fields.

Original: Originally devised for natural language processing (NLP), self-attention [63] shed a light on this direction; equipped with adaptive input processing and the ability to capture long-range interactions, it has emerged as an alternative feature transform for computer vision, being widely adopted as a core building block [18].
Revised: Originally devised for natural language processing (NLP), self-attention [63] shed a light on this direction; equipped with adaptive input processing and the ability to capture long-range interactions, it has emerged as an alternative feature transform for computer vision, being widely adopted as a core building block [18].

Original: The stand-alone self-attention models, e.g. , ViT [18], however, demand significantly more training data [57] for competitive performance with its convolutional counterparts [6, 25, 27, 72] since they miss certain desirable property which convolution possesses, e.g. , locality.
Revised: The stand-alone self-attention models, e.g. , ViT [18], however, demand significantly more training data [57] for competitive performance with its convolutional counterparts [6, 25, 27, 72] since they miss certain desirable property which convolution possesses, e.g. , locality.

Original: These inherent pros and cons of convolution and self-attention encourage recent researches toward combinations of both so as to enjoy the best of the both worlds but which one suits the best for effective visual processing is yet controversial in literature [8, 9, 10, 12, 35, 37, 38, 40, 41, 49, 50, 59, 60, 62, 67, 69, 71, 73, 77].
Revised: These inherent pros and cons of convolution and self-attention encourage recent researches toward combinations of both so as to enjoy the best of the both worlds but which one suits the best for effective visual processing is yet controversial in literature [8, 9, 10, 12, 35, 37, 38, 40, 41, 49, 50, 59, 60, 62, 67, 69, 71, 73, 77].

Original: Unlike the dominant visual feature transformations in machine vision, human vision possesses a special type of visual processing systems called peripheral vision [36]; it partitions the entire visual field into multiple contour regions based on the distances to the center of a gaze where each region identifies different visual aspects.
Revised: Unlike the dominant visual feature transformations in machine vision, human vision possesses a special type of visual processing systems called peripheral vision [36]; it partitions the entire visual field into multiple contour regions based on the distances to the center of a gaze where each region identifies different visual aspects.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 1, we have high-resolution processing near the center of our gaze, i.e. , central and para-central regions, to identify highly-detailed visual elements such as geometric shapes, and low-level details.
Revised: 1, we have high-resolution processing near the center of our gaze, i.e. , central and para-central regions, to identify highly-detailed visual elements such as geometric shapes, and low-level details.

Original: For the regions more distant from the gaze, i.e. , mid and far peripheral regions, the resolution decreases to recognize abstract visual features such as motion, and high-level contexts.
Revised: For the regions more distant from the gaze, i.e. , mid and far peripheral regions, the resolution decreases to recognize abstract visual features such as motion, and high-level contexts.

Original: This systematic strategy enables us to effectively perceive important details within a small fraction (1%) of the visual field while minimizing unnecessary processing of background clutter in the rest (99%), thus facilitating efficient visual processing for human brain.
Revised: This systematic strategy enables us to effectively perceive important details within a small fraction (1%) of the visual field while minimizing unnecessary processing of background clutter in the rest (99%), thus facilitating efficient visual processing for human brain.

Original: According to recent study on inner workings of vision transformers [12, 18, 49, 58, 71], their behaviors are related to the aforementioned visual processing strategies in the following respect; the attention maps of early layers are learned to locally capture fine-grained geometric details at central regions while those of later layers perform global attentions to identify coarse-grained semantics and contexts from the whole visual field, covering peripheral regions .
Revised: According to recent study on inner workings of vision transformers [12, 18, 49, 58, 71], their behaviors are related to the aforementioned visual processing strategies in the following respect; the attention maps of early layers are learned to locally capture fine-grained geometric details at central regions while those of later layers perform global attentions to identify coarse-grained semantics and contexts from the whole visual field, covering peripheral regions .

Original: This finding reveals that imitating biological designs may potentially help in modelling an effective machine vision, and also support recent approaches towards a hybrid [5, 10, 12, 24, 39] of convolution and self-attention beyond stand-alone visual processing to take advantages of the two different perception strategies: fine-grained/local and coarse-grained/global, similarly to the human visual processing as in Fig.
Revised: This finding reveals that imitating biological designs may potentially help in modelling an effective machine vision, and also support recent approaches towards a hybrid [5, 10, 12, 24, 39] of convolution and self-attention beyond stand-alone visual processing to take advantages of the two different perception strategies: fine-grained/local and coarse-grained/global, similarly to the human visual processing as in Fig.

Original: In this work, we take a biologically inspired approach and propose to inject the peripheral inductive biases 1 to deep neural networks for image recognition.
Revised: In this work, we take a biologically inspired approach and propose to inject the peripheral inductive biases 1 to deep neural networks for image recognition.

Original: We propose to incorporate peripheral attention mechanism to the multi-head self-attention [63] to let the network learn to partition the visual field into diverse peripheral regions given training data where each region captures different visual features.
Revised: We propose to incorporate peripheral attention mechanism to the multi-head self-attention [63] to let the network learn to partition the visual field into diverse peripheral regions given training data where each region captures different visual features.

Original: We experimentally show that the proposed network models effective visual periphery for reliable visual recognition.
Revised: We experimentally show that the proposed network models effective visual periphery for reliable visual recognition.

Original: Our main contributions can be summarized as follows:
Revised: Our main contributions can be summarized as follows:

Original: • This work explores to narrow the gap between human and machine vision by injecting peripheral inductive biases to self-attention layers, and presents a new form of feature transformation named M ulti-head P eripheral A ttention (MPA).
Revised: • This work explores to narrow the gap between human and machine vision by injecting peripheral inductive biases to self-attention layers, and presents a new form of feature transformation named M ulti-head P eripheral A ttention (MPA).

Original: • Based on the MPA, we introduce Per ipheral Vi sion T ransformer (PerViT), and systematically study the inner workings of PerViT by qualitatively and quantitatively analyzing its learned attentions, which reveal that the network learns to perceive visual elements similarly to the way that human vision does without any special supervisions.
Revised: • Based on the MPA, we introduce Per ipheral Vi sion T ransformer (PerViT), and systematically study the inner workings of PerViT by qualitatively and quantitatively analyzing its learned attentions, which reveal that the network learns to perceive visual elements similarly to the way that human vision does without any special supervisions.

Original: • The performance improvements in image classification over columnar Transformer baselines, e.g. , DeiT, across different model sizes demonstrate the efficacy of the proposed approach.
Revised: • The performance improvements in image classification over columnar Transformer baselines, e.g. , DeiT, across different model sizes demonstrate the efficacy of the proposed approach.

Original:  Related Work
Revised:  Related Work

Original: Feature transformations in computer vision.
Revised: Feature transformations in computer vision.

Original: With notable success in NLP, Transformers [14, 63] introduced a paradigm shift in computer vision [3, 4, 7, 18, 30, 50, 55, 56, 58, 59, 62].
Revised: With notable success in NLP, Transformers [14, 63] introduced a paradigm shift in computer vision [3, 4, 7, 18, 30, 50, 55, 56, 58, 59, 62].

Original: Despite their generalization capability, pure Vision Transformers [18] require extensive amount of training data to capture spatial layout of images due to lack of certain desirable property, e.g. , locality.
Revised: Despite their generalization capability, pure Vision Transformers [18] require extensive amount of training data to capture spatial layout of images due to lack of certain desirable property, e.g. , locality.

Original: This encouraged many recent ViT work to incorporate local inductive biases via distillation of convolutional biases [58], local self-attention [35, 40, 73], a hybridization [10, 12, 37], and augmenting convolutions [8, 60, 67, 69], all of which convey a unified message: “Despite high generalizibility of self-attentions, a sufficient amount of convolutional processing must be incorporated to capture the spatial configurations of images for reliable visual processing.”
Revised: This encouraged many recent ViT work to incorporate local inductive biases via distillation of convolutional biases [58], local self-attention [35, 40, 73], a hybridization [10, 12, 37], and augmenting convolutions [8, 60, 67, 69], all of which convey a unified message: “Despite high generalizibility of self-attentions, a sufficient amount of convolutional processing must be incorporated to capture the spatial configurations of images for reliable visual processing.”

Original: Position encoding for Transformer.
Revised: Position encoding for Transformer.

Original: Witnessing the efficacy of position encoding in capturing input structures in NLP [11, 28, 54], recent vision models [18, 50, 70] have begun employing position encodings for images to model spatial structures of images.
Revised: Witnessing the efficacy of position encoding in capturing input structures in NLP [11, 28, 54], recent vision models [18, 50, 70] have begun employing position encodings for images to model spatial structures of images.

Original: In particular, relative position encoding (RPE) plays a vital role for the purpose: The work of Cordonnier et al.
Revised: In particular, relative position encoding (RPE) plays a vital role for the purpose: The work of Cordonnier et al.

Original: [9] proves that self-attention has close relationship with convolution when equipped with certain form of RPE.
Revised: [9] proves that self-attention has close relationship with convolution when equipped with certain form of RPE.

Original: Wu et al.
Revised: Wu et al.

Original: [70] explore the existing RPE methods used in vision transformers [11, 28, 54, 65] and draws a conclusion that RPEs impose convolutional processing on vision transformers.
Revised: [70] explore the existing RPE methods used in vision transformers [11, 28, 54, 65] and draws a conclusion that RPEs impose convolutional processing on vision transformers.

Original: Dai et al.
Revised: Dai et al.

Original: [10] observe that depthwise convolution and self-attention can naturally be unified via RPEs.
Revised: [10] observe that depthwise convolution and self-attention can naturally be unified via RPEs.

Original: While offering promising directions, the previous RPE work, however, is limited in the sense that the focus of RPE utilization is restricted to only local attention, e.g. , convolution.
Revised: While offering promising directions, the previous RPE work, however, is limited in the sense that the focus of RPE utilization is restricted to only local attention, e.g. , convolution.

Original: This work exploits RPEs to devise an original visual feature transformation which naturally generalizes convolution and self-attention layers, thus enjoying the benefits of both via imitation of human visual processing system: peripheral vision .
Revised: This work exploits RPEs to devise an original visual feature transformation which naturally generalizes convolution and self-attention layers, thus enjoying the benefits of both via imitation of human visual processing system: peripheral vision .

Original: Peripheral vision for machine perception.
Revised: Peripheral vision for machine perception.

Original: Along with central vision, peripheral vision plays a vital role in a wide range of visual recognition tasks [33].
Revised: Along with central vision, peripheral vision plays a vital role in a wide range of visual recognition tasks [33].

Original: The fundamental mechanisms of peripheral visual processing, however, have not been fully disclosed in human vision literature [52] which stimulated many researchers to reveal its inner workings and deep implications [2, 15, 16, 43, 53, 68].
Revised: The fundamental mechanisms of peripheral visual processing, however, have not been fully disclosed in human vision literature [52] which stimulated many researchers to reveal its inner workings and deep implications [2, 15, 16, 43, 53, 68].

Original: The work of Rosenholtz [52] discusses pervasive myths and current findings about peripheral vision, suggesting that peripheral vision is more crucial for human perception than previously thought to perform diverse important tasks.
Revised: The work of Rosenholtz [52] discusses pervasive myths and current findings about peripheral vision, suggesting that peripheral vision is more crucial for human perception than previously thought to perform diverse important tasks.

Original: Inspired by its importance, a number of pioneering work [16, 20, 22, 23, 44, 66] investigate the linkage between peripheral vision and machine vision, e.g. , CNNs, while some [31, 64] devise biologically-inspired models for the creation of stronger machine vision.
Revised: Inspired by its importance, a number of pioneering work [16, 20, 22, 23, 44, 66] investigate the linkage between peripheral vision and machine vision, e.g. , CNNs, while some [31, 64] devise biologically-inspired models for the creation of stronger machine vision.

Original: Continuing previous study, this paper explores to blend human peripheral vision with attention-based neural networks, e.g. , vision transformer [18, 58], and introduces a new network called Peripheral Vision Transformer.
Revised: Continuing previous study, this paper explores to blend human peripheral vision with attention-based neural networks, e.g. , vision transformer [18, 58], and introduces a new network called Peripheral Vision Transformer.

Original:  Our Approach
Revised:  Our Approach

Original: In this section, we introduce the Peripheral Vision Transformer (PerViT) that learns to model peripheral vision for effective image recognition.
Revised: In this section, we introduce the Peripheral Vision Transformer (PerViT) which learns to model peripheral vision for effective image recognition.

Original: We first revisit the mathematical formulation of a self-attention layer and then describe how we improve it with peripheral inductive biases.
Revised: We first revisit the mathematical formulation of a self-attention layer and then describe how we improve it with peripheral inductive biases.

Original: Background: Multi-Head Self-Attention.
Revised: Background: Multi-Head Self-Attention.

Original: The multi-head self-attention (MHSA) [63] with N h heads performs an attention-based feature transformation by aggregating N h self-attention outputs:
Revised: The multi-head self-attention (MHSA) [63] with N h heads performs an attention-based feature transformation by aggregating N h self-attention outputs:

Original: [Equation]
Revised: [Equation]

Original: where X ∈ R HW × D emb is a set of input tokens and W out ∈ R N h D h × D emb and b out ∈ R D emb are the transformation parameters.
Revised: where X ∈ R HW × D emb is a set of input tokens and W out ∈ R N h D h × D emb and b out ∈ R D emb are the transformation parameters.

Original: The N h outputs of self-attention are designed to extract a diverse set of features from the input representation.
Revised: The N h outputs of self-attention are designed to extract a diverse set of features from the input representation.

Original: Formally, the self-attention at head h is defined as
Revised: Formally, the self-attention at head h is defined as

Original: [Equation]
Revised: [Equation]

Original: where Normalize [ · ] denotes a row-wise normalization and Φ ( h ) ( · ) ∈ R
Revised: where Normalize [ · ] denotes a row-wise normalization and Φ ( h ) ( · ) ∈ R

Original: HW × HW is a function that provides spatial attentions based on content information to aggregate the values V ( h ) =
Revised: HW × HW is a function that provides spatial attentions based on content information to aggregate the values V ( h ) =

Original: XW ( [Equation] ) val :
Revised: XW ( h ) val :

Original: [Equation]
Revised: [Equation]

Original: using linear projections of W ( h ) qry , W ( h ) key , W ( h ) val ∈ R D
Revised: using linear projections of W ( h ) qry , W ( h ) key , W ( h ) val ∈ R D

Original: emb × D h for queries, keys, and values respectively where τ is softmax temperature.
Revised: emb × D h for queries, keys, and values respectively where τ is softmax temperature and exp ( · ) applies an element-wise exponential to the input matrix.

Original: 3.1 Multi-head Peripheral Attention
Revised: 3.1 Multi-head Peripheral Attention

Original: Based on the formulation of the MHSA, we define M ulti-head P eripheral A ttention (MPA) as
Revised: Based on the formulation of MHSA in Eq. 1, we define M ulti-head P eripheral A ttention (MPA) as

Original: [Equation]
Revised: [Equation]

Original: where R ∈ R HW × HW × D r is the relative position encoding with D r channel dimension.
Revised: where R ∈ R HW × HW × D r is the relative position encoding with D r channel dimension.

Original: The selfattention in MHSA is now replaced with Peripheral-Attention, consisting of content- and positionbased attention functions( Φ ( h ) c  , Φ ( h ) p 
Revised: The self-attention in MHSA is now replaced with Peripheral-Attention ( · ) , consisting of content- and position-based attention functions Φ ( h ) c ( X ) , Φ ( h ) p ( R )

Original: [Equation]
Revised: ∈ R

Original: R HW × HW ), which is defined as 
Revised: HW × HW , which is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original:  where ⊙ is Hadamard product which mixes the given pair of attentions to provide a mixed attention  a network that aims to imitate human vision
Revised: ) , (5) where ⊙ is Hadamard product which mixes the given pair of attentions to provide a mixed attention Φ ( h ) a ( X , R ) : = Φ ( h ) c ( X ) ⊙ Φ ( h ) p ( R ) ∈ R HW × HW .

Revised: For the content-based attention Φ c , we use exponentiated (scaled) dot-product between queries and keys as in Eq.

Original:  Modelling peripheral vision: a Roadmap.
Revised: 3: Φ ( h ) c ( X ) : = exp( τ Q ( h ) K ( h ) ⊤ ) . For the position-based attention Φ p , we design a neural network that aims to imitate human visual processing system, e.g. , peripheral vision, which we discuss next. Modelling peripheral vision: a Roadmap.

Original: Human visual field can be grouped into several regions based on the Euclidean distances from the center of gaze, each forming ring-shaped region as seen in
Revised: Human visual field can be grouped into several regions based on the Euclidean distances from the center of gaze, each forming ring-shaped region as seen in

Original: Fig.
Revised: Fig.

Original: 1, where each region captures different visual aspects; the closer to the gaze, the more complex features we process, and further from the gaze, the simpler visual features we perceive.
Revised: 1, where each region captures different visual aspects; the closer to the gaze, the more complex features we process, and further from the gaze, the simpler visual features we perceive.

Original: In the context of 2-dimensional attention map Φ ∗ ( · ) q , : ∈ R HW , we refer the query position q ∈ R 2 as the center of gaze, i.e. , the position where feature of our interest lies at for  transformation.
Revised: In the context of 2-dimensional attention map Φ ∗ ( · ) q , : ∈ R HW , we refer the query position q ∈ R 2 as the center of gaze, i.e. , the position where feature of our interest lies at for the transformation.

Original: We refer the local regions around  q as central/para-central regions and the rest as mid/far peripheral regions.
Revised: We refer the local regions around the query q as central/para-central regions and the rest as mid/far peripheral regions.

Original: Perhaps the simplest approach to divide the visual field into multiple subregions is to perform a single linear projection on the Euclidean distances, i.e. , Φ ( h ) p ( R ) =
Revised: Perhaps the simplest approach to divide the visual field into multiple subregions is to perform a single linear projection on the Euclidean distances, i.e. , Φ ( h ) p ( R ) =

Original: σ (cid:2) RW ( h ) p (cid:3) where W ( h ) p ∈ R D r , similarly  the previous work of Wu et al.
Revised: σ (cid:2) RW ( h ) p (cid:3) where W ( h ) p ∈ R D r and σ [ · ] is non-linearity, similarly to the previous work of Wu et al.

Original: For straightforward imitation of peripheral vision, we use Euclidean distance for relative position input R and weigh the distances in D r different ways for the network to learn the mapping in multiple scales: R q , k , : :=
Revised: For straightforward imitation of peripheral vision, we use Euclidean distance for relative position input R and weigh the distances in D r different ways for the network to learn the mapping in multiple scales: R q , k , : :=

Original: concat r ∈
Revised: concat r ∈

Original: [ D r ] [ w r · R euc q , k ] ∈ R D r where { w r } r ∈
Revised: [ D r ] [ w r · R euc q , k ] ∈ R D r where { w r } r ∈

Original: [ D r ] is a set of learnable parameters shared across layers and heads, and
Revised: [ D r ] is a set of learnable parameters shared across layers and heads, and

Original: R euc q , k = ∥
Revised: R euc q , k = ∥

Original: q − k ∥is the Euclidean distance between query and key positions q , k ∈ R 2 .
Revised: q − k ∥ 2 is the Euclidean distance between query and key positions q , k ∈ R 2 .

Original: For σ , we choose sigmoid  to provide normalized (positive) weights to the content-based attention Φ c .
Revised: In our experiments, we choose sigmoid function for σ to provide normalized (positive) weights to the content-based attention Φ c .

Original: A main drawback of this single-layer formulation is that Φ p is only able to provide Gaussian-like attention map as seen in top-left in Fig.
Revised: A main drawback of this single-layer formulation is that Φ p is only able to provide Gaussian-like attention map as seen in top-left in Fig.

Original: 2, thus being unable to representdiverse peripheral regions.
Revised: 2, thus being unable to represent peripheral regions in diverse shapes.

Original: For the encoding function to represent diverse (torus-shaped) peripheral regions, the distances must be processed by an MLP: Φ
Revised: For the encoding function to represent various (torus-shaped) peripheral regions, the distances must be processed by an MLP: Φ

Original: [Equation]
Revised: [Equation]

Original: ReLU
Revised: p ( R ) = σ (cid:104) Linear

Original: [Equation]
Revised: [Equation]

Original: where W p1 ∈ R D r × D hid and W ( h ) p2 ∈ R D hid are the linear projection parameters 3 , and ReLU gives non-linearity to the function.
Revised: where W p1 ∈ R D r × D hid and W ( h ) p2 ∈ R D hid are the linear projection parameters 3 , and ReLU gives non-linearity to the function.

Original: The first projection W p1 is shared across the heads in order to exchange information so each function is able to provide attention that are effective or complementary to other heads’ attention.
Revised: The first projection W p1 is shared across the heads in order to exchange information so each function is able to provide attention that are effective or complementary to other heads’ attention.

Original: Note that given identical relative distances between a fixed query point q ∈ R 2 and key points k
Revised: Note that given identical relative distances between a fixed query point q ∈ R 2 and key points k

Original: i , k j ∈ R 2 , i.e. , R q , k i =
Revised: i , k j ∈ R 2 , i.e. , R q , k i =

Original: R q , k j , Eq. 6 provides the same attention scores: Φ p ( R ) q , k i =
Revised: R q , k j , Eq. 6 provides the same attention scores: Φ p ( R ) q , k i =

Original: Φ p ( R ) q , k j as seen in top-right of Fig.
Revised: Φ p ( R ) q , k j as seen in top-right of Fig.

Original: This property, however, is not always desired in practical scenarios because the rotational symmetric property hardly holds for most real-world objects.
Revised: This property, however, is not always desired in practical scenarios because the rotational symmetric property hardly holds for most real-world objects.

Original: To break the symmetric property in Eq.
Revised: To break the symmetric property in Eq.

Original: 6 while preserving peripheral design to sufficient extent, we introduce peripheral projections in which the transformation parameters are given small spatial resolutions, e.g. , K × K window, such that W p1 ∈ R K 2 × D r × D hid and W ( h ) p2 ∈ R K 2 × D hid so that they provide similar but different attention scores, Φ p ( R ) q , k i ̸
Revised: 6 while preserving peripheral design to sufficient extent, we introduce peripheral projections in which the transformation parameters are given small spatial resolutions, e.g. , K × K window, such that W p1 ∈ R K 2 × D r × D hid and W ( h ) p2 ∈ R K 2 × D hid so that they provide similar but different attention scores, Φ p ( R ) q , k i ̸

Original: = Φ p ( R ) q , k j , given R q , k i
Revised: = Φ p ( R ) q , k j , given R q , k i

Original: = R q , k j , by aggregating neighboring relative distances around the key location k as follows:
Revised: = R q , k j , by aggregating neighboring relative distances around the key location k as follows:

Original: [Equation]
Revised: [Equation]

Original: where N ( k ) := (cid:2) k − ⌊ K2 ⌋ , . . . , k + ⌊ K 2 ⌋ (cid:3) ×
Revised: where N ( k ) := (cid:2) k − ⌊ K

Original: [Equation]
Revised: [Equation]

Original: k − ⌊ K2 ⌋
Revised: k + ⌊ K

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: × (cid:2) k − ⌊ K 2 ⌋ , . . .

Original:  k
Revised: , k

Original: [Equation] ⌊ K 2 ⌋ (cid:3) is a set of K 2 neighbors around input position k .
Revised: + ⌊ K 2 ⌋ (cid:3) is a set of K 2 neighbors around input position k .

Original: We set K = 3 for all layers and heads as K > 3 hardly brought improvements.
Revised: We set K = 3 for all layers and heads as K > 3 hardly brought improvements.

Original: Note that this formulation can be considered as avariant of 4-dimensional convolution [51] in the sense that the function Φ p takes 4-dimensional input R ∈ R
Revised: Note that each linear projection in Eq. 7 is equivalent to a 4-dimensional convolution [51], taking 4-dimensional input R ∈ R

Original: HW × HW × D r and processes it in convolutional manner with 4-dimensional kernels in size of K × K × 1 × 1 , i.e. , W p1 ∈ R K × K × 1 × 1 × D r × D hid .
Revised: HW × HW × D r to process in convolutional manner using 4-dimensional kernels in size of K × K × 1 × 1 , i.e. , W p1 ∈ R K × K × 1 × 1 × D r × D hid .

Original: Precisely, each peripheral projectiononly considers a  subset of 4D local neighbors that pivots the query position q , similarly to the center-pivot 4D convolution [45, 46].
Revised: Precisely, the peripheral projection considers a small subset of 4D local neighbors that pivots the query position q , similarly to the center-pivot 4D convolution [45, 46].

Original: After each peripheral projection, we add an instance normalization layer [61] for stable optimization:
Revised: After each peripheral projection, we add an instance normalization layer [61] for stable optimization:

Original: [Equation]
Revised: [Equation]

Original: where γγγ p1 ,βββ p1 ∈ R D hid and γ ( h ) p2 , β ( h ) p2 ∈ R are weights/biases of the instance norms and PP ( · ) denotes the peripheral projection: PP ( R , W ) q , k , : := (cid:80) n ∈N ( k ) R q , n , : W n − k , : , : .
Revised: where γγγ p1 ,βββ p1 ∈ R D hid and γ ( h ) p2 , β ( h ) p2 ∈ R are weights/biases of the instance norms and PP ( · ) denotes the peripheral projection: PP ( R , W ) q , k , : := (cid:80) n ∈N ( k ) R q , n , : W n − k , : , : .

Original: The middle row of Fig.depicts learned attentions of Φ p with peripheral projections, which provides peripheral attention maps in greater diversity compared to single- and multi-layer counterparts without N . 
Revised: The middle row of Fig.depicts learned attentions of Φ p with peripheral projections, which provides peripheral attention maps in greater diversity compared to single- and multi-layer counterparts without N . Given

Original: Multi-layer projection
Revised: Multi-layer projection

Original: Layer
Revised: Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Peripheral initialization
Revised: Peripheral initialization

Original: [Figure]
Revised: [Figure]

Original: MPA
Revised: MPA

Original: FFN
Revised: FFN

Original: Convolutionalpatch embedding
Revised: Convolutionalpatch embedding

Original: GAP
Revised: GAP

Original: Image embedding
Revised: Image embedding

Original: Input image ×𝑁 !
Revised: Input image ×𝑁 !

Original: Scaled dot product 𝑸 (#) 𝑲 (#) 𝑽 (#) 𝐗∈ ℝ %&×( !"
Revised: Scaled dot product 𝑸 (#) 𝑲 (#) 𝑽 (#) 𝐗∈ ℝ %&×( !"

Original: Peripheralprojection 𝐑∈ ℝ %&×%&×( $
Revised: Peripheralprojection 𝐑∈ ℝ %&×%&×( $

Original: Peripheralprojection
Revised: Peripheralprojection

Original: IN & ReLU
Revised: IN & ReLU

Original: IN & Sigmoid
Revised: IN & Sigmoid

Original: Exp
Revised: Exp

Original: Matrix mult.
Revised: Matrix mult.

Original: 𝛗 !
Revised: 𝛗 !

Original: 𝛗 "
Revised: 𝛗 "

Original: CPE
Revised: CPE

Original: Normalize
Revised: Normalize

Original: Linear Linear
Revised: Linear Linear

Original: Linear 𝒉 (𝐗,𝐑) Peripheral-Attention ×𝑁 #
Revised: Linear 𝒉 (𝐗,𝐑) Peripheral-Attention ×𝑁 #

Original: Peripheral initialization.
Revised: Peripheral initialization.

Original: Recent study [49, 58] observe that early layers of trained vision transformer learn to attend locally whereas late layers perform global attentions.
Revised: Recent study [49, 58] observe that early layers of trained vision transformer learn to attend locally whereas late layers perform global attentions.

Original: To facilitate training of our network, we inject this property in the beginning of the training stage by initializing parameters of Φ p for the purpose, making attention scores near the queries larger than the distant ones in the early layers while uniformly distributing them in the late layers as seen in bottom row of
Revised: To facilitate training of our network, we inject this property in the beginning of the training stage by initializing parameters of Φ p for the purpose, making attention scores near the queries larger than the distant ones in the early layers while uniformly distributing them in the late layers as seen in bottom row of

Original: Fig.
Revised: Fig.

Original: We refer this method as peripheral initialization for its resemblance to the functioning of peripheral vision [36] which also operates either locally or globally to perceive different visual features [76].
Revised: We refer this method as peripheral initialization for its resemblance to the functioning of peripheral vision [36] which also operates either locally or globally to perceive different visual features [76].

Original: To formally put, given two arbitrarily chosen distances R euc q , k i , R euc q , k j ∈ R which satisfy
Revised: To formally put, given two arbitrarily chosen distances R euc q , k i , R euc q , k j ∈ R which satisfy

Original: R euc q , k i < R euc q , k j , we want Φ ( l,h ) p ( R ) q , k
Revised: R euc q , k i < R euc q , k j , we want Φ ( l,h ) p ( R ) q , k

Original: i ≫ Φ ( l,h ) p ( R ) q , k j 4 as l → 1 , i.e. , local attention in early layers, and Φ ( l,h ) p
Revised: i ≫ Φ ( l,h ) p ( R ) q , k j 4 as l → 1 , i.e. , local attention in early layers, and Φ ( l,h ) p

Original: ( R ) q , k
Revised: ( R ) q , k

Original: i ≈ Φ ( l,h ) p ( R ) q , k j as l → N l , i.e. , global attention in late layers, where N l is the total number of MPA layers.
Revised: i ≈ Φ ( l,h ) p ( R ) q , k j as l → N l , i.e. , global attention in late layers, where N l is the total number of MPA layers.

Original: We first initialize the parameters of Φ ( l,h ) p and { w r } r ∈
Revised: We first initialize the parameters of Φ ( l,h ) p and { w r } r ∈

Original: [ D r ] to particular values.
Revised: [ D r ] to particular values.

Original: Specifically, for all layers l ∈ [ N l ] and heads h ∈
Revised: Specifically, for all layers l ∈ [ N l ] and heads h ∈

Original: [ N h ] ,
Revised: [ N h ] ,

Original: [Equation]
Revised: [Equation]

Original: ) where c 1 , c 2 ∈ R + are positive reals, and J N,M ∈ R N × M refers to all-one matrix in size N ×
Revised: ) where c 1 , c 2 ∈ R + are positive reals, and J N,M ∈ R N × M refers to all-one matrix in size N ×

Original: M .
Revised: M .

Original: The above initialization provides local attention after the second peripheral projection, i.e. ,
Revised: The above initialization provides local attention after the second peripheral projection, i.e. ,

Original: PP ( R ′ ; W ( h ) p2 ) q , k i , : > PP ( R ′ ; W ( h ) p2 ) q , k j , : given R euc q , k i < R euc q , k j .
Revised: PP ( R ′ ; W ( h ) p2 ) q , k i , : > PP ( R ′ ; W ( h ) p2 ) q , k j , : given R euc q , k i < R euc q , k j .

Original: Next, based on our findings that biases β ( l,h ) p and the weights γ ( l,h ) p in the second instance norm control the sizes and strengths of local attention respectively, we simulate peripheral initialization by setting their initial values as β ( l,h ) p :=
Revised: Next, based on our findings that biases β ( l,h ) p and the weights γ ( l,h ) p in the second instance norm control the sizes and strengths of local attention respectively, we simulate peripheral initialization by setting their initial values as β ( l,h ) p :=

Original: s l and γ ( l,h ) p :=
Revised: s l and γ ( l,h ) p :=

Original: v l where respective { s l } l ∈ [ N l ] and { v l } l ∈ [ N l ] are sets of initial values for attention sizes and strengths.
Revised: v l where respective { s l } l ∈ [ N l ] and { v l } l ∈ [ N l ] are sets of initial values for attention sizes and strengths.

Original: We set their values collected from uniform intervals: s l ∈ [ − 5 . 0 , 4 . 0] and v l ∈ [3 . 0 , 0 .
Revised: We set their values collected from uniform intervals: s l ∈ [ − 5 . 0 , 4 . 0] and v l ∈ [3 . 0 , 0 .

Original: 01] where s l − 1 < s l and v l − 1 > v l to give stronger local attentions to shallow layers compared to deep ones as seen in bottom row of Fig.
Revised: 01] where s l − 1 < s l and v l − 1 > v l to give stronger local attentions to shallow layers compared to deep ones as seen in bottom row of Fig.

Original: We set c 1 , c 2 = 0 .
Revised: We set c 1 , c 2 = 0 .

Original: 02 in our experiments and leave the details on the peripheral initialization to the supplementary.
Revised: 02 in our experiments. We refer the readers to the supplementary for the complete derivation of the peripheral initialization.

Original: 3.2 Peripheral Vision Transformer
Revised: 3.2 Peripheral Vision Transformer

Original: Based on the proposed peripheral projections and initialization, we develop image classification models, dubbed Peripheral Vision Transformer, which is illustrated in Fig.
Revised: Based on the proposed peripheral projections and initialization, we develop image classification models, dubbed Peripheral Vision Transformer, which is illustrated in Fig.

Original: We follow similar architecture to DeiT [58] with convolutional patch embedding stem; as the original patchify stem [18] exhibits substandard optimizability due to its coarse-grained early visual processing [71], many recent
Revised: We follow similar architecture to DeiT [58] with convolutional patch embedding stem; as the original patchify stem [18] exhibits substandard optimizability due to its coarse-grained early visual processing [71], many recent

Original: ViT models adopt multi-resolution pyramidal designs [40, 67, 69, 73] to mitigate the issue.
Revised: ViT models adopt multi-resolution pyramidal designs [40, 67, 69, 73] to mitigate the issue.

Original: While the pyramidal models have shown their efficacy in learning reliable image embeddings, we stick with the original single-resolution columnar design for PerViT because features in multiple resolution make our study less interpretable, which further requires additional techniques for combining our
Revised: While the pyramidal models have shown their efficacy in learning reliable image embeddings, we stick with the original single-resolution columnar design for PerViT because features in multiple resolution make our study less interpretable, which further requires additional techniques for combining our

Original: [Figure]
Revised: [Figure]

Original: H e a d s l o c a l g l o b a l
Revised: H e a d s l o c a l g l o b a l

Original: [Figure]
Revised: [Figure]

Original: Central
Revised: Central

Original: Para [Figure] central
Revised: Para-central

Original: Mid peripheral
Revised: Mid peripheral

Original: Far peripheral
Revised: Far peripheral

Original: Tiny
Revised: Tiny

Original: Small
Revised: Small

Original: Medium 0 26 812 141 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 

Revised: Medium 0 26 812 141 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8

Original: [Figure]
Revised: [Figure]

Original: peripheral attention Φ p with the existing cost-effective self-attentions mechanisms such as factorized attention [73], shifted-window [40], and cross-shaped window [17].
Revised: peripheral attention Φ p with the existing cost-effective self-attentions mechanisms such as factorized attention [73], shifted-window [40], and cross-shaped window [17].

Original: To carry out fine-grained early processing while keeping single-resolution features across the layers, we adopt convolutional patch embedding layer [71] with multi-stage layouts for channel dimensions.
Revised: To carry out fine-grained early processing while keeping single-resolution features across the layers, we adopt convolutional patch embedding layer [71] with multi-stage layouts for channel dimensions.

Original: The convolutional embedding layer consists of four 3 × 3 and one 1 × 1 convolutions where the 3 × 3 convolutions are followed by batch norm [29] and ReLU [47].
Revised: The convolutional embedding layer consists of four 3 × 3 and one 1 × 1 convolutions where the 3 × 3 convolutions are followed by batch norm [29] and ReLU [47].

Original: For additional details, we refer to the supplementary materials.
Revised: For additional details, we refer to the supplementary materials.

Revised: Overall pipeline.

Original: Given an image, the convolutional patch embedding provides token embeddings X (1) ∈ R HW × D emb .
Revised: Given an image, the convolutional patch embedding provides token embeddings X (1) ∈ R HW × D emb .

Original: Similarly to [18, 58], the embeddings are fed to a series of N l blocks each of which consists of an MPA layer and a feed-forward network with residual pathways:
Revised: Similarly to [18, 58], the embeddings are fed to a series of N l blocks each of which consists of an MPA layer and a feed-forward network with residual pathways:

Original: [Equation]
Revised: [Equation]

Original:  where LN is layer normalization [1], and FFN is an MLP consisting of two linear transformations with a GELU activation [26].
Revised: ) where LN is layer normalization [1], and FFN is an MLP consisting of two linear transformations with a GELU activation [26].

Original: Following the work of [35, 73], we adopt convolutional position encoding (CPE), i.e. , a 3 × 3 depth-wise convolution, before first layer norm for its efficacy with negligible computational cost.
Revised: Following the work of [35, 73], we adopt convolutional position encoding (CPE), i.e. , a 3 × 3 depth-wise convolution, before first layer norm for its efficacy with negligible computational cost.

Original: The output X ( N h ) is global-average pooled to form an image embedding.
Revised: The output X ( N h ) is global-average pooled to form an image embedding.

Original:  Experiments
Revised:  Experiments

Original: In this section, we first investigate the inner workings of PerViT trained on ImageNet-1K classification dataset to examine how it benefits from the proposed peripheral projections and initialization, and then compare the method with previous state of the arts under comparable settings.
Revised: In this section, we first investigate the inner workings of PerViT trained on ImageNet-1K classification dataset to examine how it benefits from the proposed peripheral projections and initialization, and then compare the method with previous state of the arts under comparable settings.

Original: Experimental setup.
Revised: Experimental setup.

Original: Our experiments focus on image classification on ImageNet-1K [13].
Revised: Our experiments focus on image classification on ImageNet-1K [13].

Original: Following training recipes of DeiT [58], we train our model on ImageNet-1K from scratch with batch size of 1024, learning rate of 0.001 using AdamW [42] optimizer, cosine learning rate decay scheduler, and the same data augmentations [14] for 300 epochs, including warm-up epochs.
Revised: Following training recipes of DeiT [58], we train our model on ImageNet-1K from scratch with batch size of 1024, learning rate of 0.001 using AdamW [42] optimizer, cosine learning rate decay scheduler, and the same data augmentations [14] for 300 epochs, including warm-up epochs.

Original: We evaluate our model with three different sizes, e.g. , Tiny (T), Small (S), and Medium (M).
Revised: We evaluate our model with three different sizes, e.g. , Tiny (T), Small (S), and Medium (M).

Original: We use stochastic depths of 0.0, 0.1, and 0.2 for T, S, and M respectively.
Revised: We use stochastic depths of 0.0, 0.1, and 0.2 for T, S, and M respectively.

Original: Refer to the supplmentary for additional details.
Revised: We refer to the supplementary for additional details.

Original: 4.1 The inner workings of PerViT
Revised: 4.1 The inner workings of PerViT

Original: Learning peripheral vision.
Revised: Learning peripheral vision.

Original: We begin by investigating how PerViT models peripheral vision by qualitatively analyzing its learned attention of Φ p .
Revised: We begin by investigating how PerViT models peripheral vision by qualitatively analyzing its learned attention of Φ p .

Original: Figure 4 depicts the learned attention map of Φ ( l,h ) p q , : ∈ R HW for all layers and heads where the query position is given at the center, i.e. , q = [7 , 7] ⊤ 5 .
Revised: Figure 4 depicts the learned attention map of Φ ( l,h ) p q , : ∈ R HW for all layers and heads where the query position is given at the center, i.e. , q = [7 , 7] ⊤ 5 .

Original: We observe that the attentions are learned to be in diverse shapes of peripheral regions.
Revised: We observe that the attentions are learned to be in diverse shapes of peripheral regions.

Original: Interestingly, without any special supervisions, the four attended regions ( N h = 4 ) in most layers are learned to complement each other to cover the entire visual field, capturing different visual aspects at each region (head), similarly to human peripheral vision illustrated in Fig.
Revised: Interestingly, without any special supervisions, the four attended regions ( N h = 4 ) in most layers are learned to complement each other to cover the entire visual field, capturing different visual aspects at each region (head), similarly to human peripheral vision illustrated in Fig.

Original: For example, first two heads in Layer 3 attend the central regions while the others cover the rest peripheral regions.
Revised: For example, first two heads in Layer 3 attend the central regions while the others cover the rest peripheral regions.

Original: The second and third heads in Layer 8 cover top and bottom hemicircles respectively, forming a circular-shaped semi-global receptive field.
Revised: The second and third heads in Layer 8 cover top and bottom hemicircles respectively, forming a circular-shaped semi-global receptive field.

Original: Moreover, a large number of early attentions is in form of central/para-central regions while those of late layers are learned to cover mid to far peripheral regions.
Revised: Moreover, a large number of early attentions is in form of central/para-central regions while those of late layers are learned to cover mid to far peripheral regions.

Original: To quantitatively inspect how PerViT models the peripheral visual system, we classify every feature transformation layer in the network into one of the four visual regions P ∈ { c , p , m , f } where respective elements refer to central, para-central, mid, and far peripheral regions.
Revised: To quantitatively inspect how PerViT models the peripheral visual system, we classify every feature transformation layer in the network into one of the four visual regions P ∈ { c , p , m , f } where respective elements refer to central, para-central, mid, and far peripheral regions.

Original: PerViT-Attention of head h at layer l is classified as peripheral region p if the average of its attention scores which fall in visual region p is the largest among the others:
Revised: PerViT-Attention of head h at layer l is classified as peripheral region p if the average of its attention scores which fall in visual region p is the largest among the others:

Original: [Equation]
Revised: [Equation]

Original: where P is a set of spatial positions ( |P| = HW ) and I p is distance range (real-valued interval) of peripheral region p 6 .
Revised: where P is a set of spatial positions ( |P| = HW ) and I p is distance range (real-valued interval) of peripheral region p 6 .

Original: The pie charts of Fig.
Revised: The pie charts of Fig.

Original: 5 describe the proportions of peripheral regions for Tiny,
Revised: 5 describe the proportions of peripheral regions for Tiny,

Original: Small, and Medium models where the bar graphs show them in layer-wise manner 7 .
Revised: Small, and Medium models where the bar graphs show them in layer-wise manner 7 .

Original: Similarly to the visualized attention maps in Fig.
Revised: Similarly to the visualized attention maps in Fig.

Original: 4, the early layers attends central/para-central regions whereas deeper ones focus on outer region.
Revised: 4, the early layers attends central/para-central regions whereas deeper ones focus on outer region.

Original: We observe that, as the model size grows, the number of mid/far peripheral attention increases whereas that of central/para-central attention stays similar, suggesting that the models no longer require local attentions once sufficient amount of processing is done in the central region because, we hypothesize, identifying geometric patterns, e.g. , corners and edges, is relatively simpler process than understanding high-level semantics.
Revised: We observe that, as the model size grows, the number of mid/far peripheral attention increases whereas that of central/para-central attention stays similar, suggesting that the models no longer require local attentions once sufficient amount of processing is done in the central region because, we hypothesize, identifying geometric patterns, e.g. , corners and edges, is relatively simpler process than understanding high-level semantics.

Original: Inspecting the impact of attentions (static vs. dynamic).
Revised: Inspecting the impact of attentions (static vs. dynamic).

Original: To study how position-based attentions Φ p contribute to the mixed attentions Φ a = Φ c ⊙ Φ p , we collect sample images and visualize their attention maps of Layers 3, 4 and 8 in Fig.
Revised: To study how position-based attentions Φ p contribute to the mixed attentions Φ a = Φ c ⊙ Φ p , we collect sample images and visualize their attention maps of Layers 3, 4 and 8 in Fig.

Original: The mixed attentions Φ a at Layer 4 are formed dynamically ( Φ c ) within statically-formed region ( Φ p ) while the attentions Φ a at Layer 8 weakly exploit position information ( Φ p ) to form dynamic attentions ( Φ c ).
Revised: The mixed attentions Φ a at Layer 4 are formed dynamically ( Φ c ) within statically-formed region ( Φ p ) while the attentions Φ a at Layer 8 weakly exploit position information ( Φ p ) to form dynamic attentions ( Φ c ).

Original: The results reveal that Φ p plays two different roles; it imposes semi-dynamic attention if the attended region is focused in a small area whereas it serves as position bias injection when the attended region is relatively broad.
Revised: The results reveal that Φ p plays two different roles; it imposes semi-dynamic attention if the attended region is focused in a small area whereas it serves as position bias injection when the attended region is relatively broad.

Original: In the supplementary, we constructively prove that an MPA layer in extreme case of semi-dynamic attention/position bias injection is in turn convolution/multi-head self-attention , naturally generalizing the both transformations.
Revised: In the supplementary, we constructively prove that an MPA layer in extreme case of semi-dynamic attention/position bias injection is in turn convolution/multi-head self-attention , naturally generalizing the both transformations.

Original: To quantitatively examine the contributions of Φ c and Φ p to the mixed attention Φ a , we define a measure of ‘impact’ by taking inverse of difference between two attentions:
Revised: To quantitatively examine the contributions of Φ c and Φ p to the mixed attention Φ a , we define a measure of ‘impact’ by taking inverse of difference between two attentions:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ∥·∥ F is Frobenius norm.
Revised: where ∥·∥ F is Frobenius norm.

Original: The higher the measure Ψ ( l,h ) p , the larger the impact of position-based attention Φ ( l,h ) p .
Revised: The higher the measure Ψ ( l,h ) p , the larger the impact of position-based attention Φ ( l,h ) p .

Original: Being averaged over all test samples, Ψ ( l,h ) c is similarly defined.
Revised: Being averaged over all test samples, Ψ ( l,h ) c is similarly defined.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 7, we observe a clear tendency that the impact of position-based attention is significantly higher in early processing, transforming features semi-dynamically , while the later layers require less position information, regarding Φ p as a minor position bias .
Revised: 7, we observe a clear tendency that the impact of position-based attention is significantly higher in early processing, transforming features semi-dynamically , while the later layers require less position information, regarding Φ p as a minor position bias .

Original: This tendency becomes more visible with larger models as seen in right of Fig 7; Small and Medium models exploit dynamic transformations much
Revised: This tendency becomes more visible with larger models as seen in right of Fig 7; Small and Medium models exploit dynamic transformations much

Original: I m p a c t m e t
Revised: I m p a c t m e t

Original: r
Revised: r

Original: i c Ψ ∗ ( # , % )
Revised: i c Ψ ∗ ( # , % )

Original: ( T i n y ) 0.
Revised: ( T i n y ) 0.

Original: Tiny Small
Revised: Tiny Small

Original: Medium 0.
Revised: Medium 0.

Original: [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Equation]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: more compared to Tiny model, especially in the later layers.
Revised: more compared to Tiny model, especially in the later layers.

Original: Moreover, we note that the impact measures of four heads (bar graphs) within each layer are unevenly distributed, showing high variance, which imply that the network evenly utilizes both position and content information simultaneously within each MPA layer as seen in Layer 3 in Fig. 6, performing both static/local and dynamic/global transformation in a single-shot .
Revised: Moreover, we note that the impact measures of four heads (bar graphs) within each layer are unevenly distributed, showing high variance, which imply that the network evenly utilizes both position and content information simultaneously within each MPA layer as seen in Layer 3 in Fig. 6, performing both static/local and dynamic/global transformation in a single-shot .

Original: These results reveal that feature transformations for effective visual recognition should not be restricted to be in either position-only [25, 41] or content-only [18, 58] design
Revised: These results reveal that feature transformations for effective visual recognition should not be restricted to be in either position-only [25, 41] or content-only [18, 58] design

Original: but they should be in the form of a hybrid [10, 12].
Revised: but they should be in the form of a hybrid [10, 12].

Original: Inspecting the locality (local vs. global).
Revised: Inspecting the locality (local vs. global).

Original: We further investigate the inner workings of PerViT by quantifying how locally Φ ∗ attends.
Revised: We further investigate the inner workings of PerViT by quantifying how locally Φ ∗ attends.

Original: Following the work of [12], we define the measure of nonlocality for Φ ( l,h ) p by summing all pair-wise query-key distances weighted by their attention scores:
Revised: Following the work of [12], we define the measure of nonlocality for Φ ( l,h ) p by summing all pair-wise query-key distances weighted by their attention scores:

Original: [Equation]
Revised: [Equation]

Original: This metrics Ω c and Ω a are similarly defined, being averaged over all test samples.
Revised: The metrics Ω c and Ω a are similarly defined, being averaged over all test samples.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 8, we observe a similar trend of locality between Φ p and Φ a , which reveals the position information play more dominant role over the content information in forming spatial attentions ( Φ a ) for feature transformation.
Revised: 8, we observe a similar trend of locality between Φ p and Φ a , which reveals the position information play more dominant role over the content information in forming spatial attentions ( Φ a ) for feature transformation.

Original: Interestingly, we also observe that content- and position-based attentions behave conversely; Φ c attends globally in early layers, i.e. , large scores are distributed over the whole spatial region, while being relatively local in deeper layers.
Revised: Interestingly, we also observe that content- and position-based attentions behave conversely; Φ c attends globally in early layers, i.e. , large scores are distributed over the whole spatial region, while being relatively local in deeper layers.

Original: We hypothesize that the proposed Φ p in early layers is trained to effectively suppress unnecessary scores of Φ c at distant positions, thus exploiting only a few relevant ones within its local region of interest.
Revised: We hypothesize that the proposed Φ p in early layers is trained to effectively suppress unnecessary scores of Φ c at distant positions, thus exploiting only a few relevant ones within its local region of interest.

Original: Meanwhile, Φ p at later layers gives Φ c higher freedom in forming the spatial attention Φ a as described in the plots of Fig. 7, which allows the attention scores of Φ c to be clustered in semantically meaningful parts, e.g. , eyes of the animals as seen in attentions of the first head at Layer 8 (Fig. 6), which makes Φ c relatively local.
Revised: Meanwhile, Φ p at later layers gives Φ c higher freedom in forming the spatial attention Φ a as described in the plots of Fig. 7, which allows the attention scores of Φ c to be clustered in semantically meaningful parts, e.g. , eyes of the animals as seen in attentions of the first head at Layer 8 (Fig. 6), which makes Φ c relatively local.

Original: 4.2 Quantitative evaluation on ImageNet-1K
Revised: 4.2 Quantitative evaluation on ImageNet-1K

Original: Ablation study on main components.
Revised: Ablation study on main components.

Original: In Tab. 1, we analyze the impact of each component in PerViT, which is denoted as (a), where C-stem refers to convolutional patch embedding stem 8 .
Revised: In Tab. 1, we analyze the impact of each component in PerViT, which is denoted as (a), where C-stem refers to convolutional patch embedding stem 8 .

Original: We observe that the proposed attention Φ p brings consistent gains to models (b, f, g, i) with relative improvements of 1.4 ∼ 4.2%p for (a, d, e, h) respectively.
Revised: We observe that the proposed attention Φ p brings consistent gains to models (b, f, g, i) with relative improvements of 1.4 ∼ 4.2%p for (a, d, e, h) respectively.

Original: Among three main components ( Φ p , C-stem, CPE), Φ p has the most significant impact on PerViT (a), losing 1.5%p Top-1 accuracy without Φ p , i.e. , model (b).
Revised: Among three main components ( Φ p , C-stem, CPE), Φ p has the most significant impact on PerViT (a), losing 1.5%p Top-1 accuracy without Φ p , i.e. , model (b).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Surprisingly, PerViT without content-based attention Φ c , model (c), achieves decent performance, almost equalling to the accuracy of PerViT without position-based attention Φ p , model (b) (-0.5%p).
Revised: Surprisingly, PerViT without content-based attention Φ c , model (c), achieves decent performance, almost equalling to the accuracy of PerViT without position-based attention Φ p , model (b) (-0.5%p).

Original: The results verify that the proposed peripheral attention, which achieves comparable level of efficacy to the content-based attention, learns to generate reliable spatial attentions for visual recognition.
Revised: The results verify that the proposed peripheral attention, which achieves comparable level of efficacy to the content-based attention, learns to generate reliable spatial attentions for visual recognition.

Original: We also implement the proposed position-based attention Φ p on DeiT [58] baseline and compare the results with recent state-of-the-art RPE methods.
Revised: We also implement the proposed position-based attention Φ p on DeiT [58] baseline and compare the results with recent state-of-the-art RPE methods.

Original: As seen in Tab. 2, the large improvements over the previous RPE methods [8, 70] further verify the efficacy of the proposed peripheral position encoding (PPE).
Revised: As seen in Tab. 2, the large improvements over the previous RPE methods [8, 70] further verify the efficacy of the proposed peripheral position encoding (PPE).

Original: To confirm that the impact of Φ p is consistent with large models, we conduct similar ablations using PerViT-S/M in Tab. 3; without Φ p , the accuracy consistently drops for all the three models.
Revised: To confirm that the impact of Φ p is consistent with large models, we conduct similar ablations using PerViT-S/M in Tab. 3; without Φ p , the accuracy consistently drops for all the three models.

Original: Comparing (b) with (c), we observe that C-Stem and CPE are less effective for large models, bringing 1.3%p and 0.1%p gains for Small and Medium respectively whereas they improve the Tiny model by 5.1%p.
Revised: Comparing (b) with (c), we observe that C-Stem and CPE are less effective for large models, bringing 1.3%p and 0.1%p gains for Small and Medium respectively whereas they improve the Tiny model by 5.1%p.

Original: In contrast, the impact of Φ p is consistent across different model sizes, bringing 1%p gains for all the three models.
Revised: In contrast, the impact of Φ p is consistent across different model sizes, bringing 1%p gains for all the three models.

Original: The better efficacy of Φ p for larger models, we hypothesize, is due to its flexibility in modeling local/global spatial attentions while C-Stem/CPE are designed only to be local.
Revised: The better efficacy of Φ p for larger models, we hypothesize, is due to its flexibility in modeling local/global spatial attentions while C-Stem/CPE are designed only to be local.

Original: Sample-efficiency of PerViT. To investigate the training sample efficiency of our model, we train
Revised: Sample-efficiency of PerViT. To investigate the training sample efficiency of our model, we train

Original: PerViT-S with ImageNet subsampled by fractions of 50% and 25% 9 and evaluate it on full-sized test set of ImageNet-1K. Table 4 compare our results with DeiT [58]; our model consistently surpasses the baseline for all subsampled datasets, showing its robustness under limited training data.
Revised: PerViT-S with ImageNet subsampled by fractions of 50% and 25% 9 and evaluate it on full-sized test set of ImageNet-1K. Table 4 compare our results with DeiT [58]; our model consistently surpasses the baseline for all subsampled datasets, showing its robustness under limited training data.

Original: Ablation study on Φ p .
Revised: Ablation study on Φ p .

Original: The top section of Tab.
Revised: The top section of Tab.

Original: 5 reports results of PerViT-T with different parameter initialization methods for Φ p where peripheral denotes the proposed peripheral initialization, conv refers to convolutional initialization such that s l = − 5 . 

Revised: 5 reports results of PerViT-T with different parameter initialization methods for Φ p where peripheral denotes the proposed peripheral initialization, conv refers to convolutional initialization such that s l = − 5 .

Original: and v l = 3 . 0 for all l ∈ [ N l ] , and rand refers to random initialization for all parameters in Φ p : w r , W ( l ) p1 , W ( l,h ) p2 , γγγ ( l ) p1 , βββ ( l ) p1 , γ ( l,h ) p2 , and β ( l,h ) p2 .
Revised: and v l = 3 . 0 for all l ∈ [ N l ] , and rand refers to random initialization for all parameters in Φ p : w r , W ( l ) p1 , W ( l,h ) p2 , γγγ ( l ) p1 , βββ ( l ) p1 , γ ( l,h ) p2 , and β ( l,h ) p2 .

Original: The results show the efficacy of our peripheral initialization which is also supported by the results in Fig.
Revised: The results show the efficacy of our peripheral initialization which is also supported by the results in Fig.

Original: 4 and 8: Φ p provides early local and late global attentions, suggesting that peripheral initialization effectively reduces burden in learning such form of attentions.
Revised: 4 and 8: Φ p provides early local and late global attentions, suggesting that peripheral initialization effectively reduces burden in learning such form of attentions.

Original: The bottom section of Tab.
Revised: The bottom section of Tab.

Original: 5 studies network designs for Φ p where N represents the proposed peripheral projection, i.e. , projecting input distance representation by referring neighbors N , ML refers to multi-layer design of Φ p , and Euc & Lrn indicate the type of embedding R :
Revised: 5 studies network designs for Φ p where N represents the proposed peripheral projection, i.e. , projecting input distance representation by referring neighbors N , ML refers to multi-layer design of Φ p , and Euc & Lrn indicate the type of embedding R :

Original: Euc is relative Euclidean distances ( R q , k , : = concat r ∈
Revised: Euc is relative Euclidean distances ( R q , k , : = concat r ∈

Original: [ D r ] [ w r · R ] ) where Lrn is relative distances between learnable vectors ( R ∈ R
Revised: [ D r ] [ w r · R ] ) where Lrn is relative distances between learnable vectors ( R ∈ R

Original: HW × HW × D r ).
Revised: HW × HW × D r ).

Original: Without N and ML, we observe consistent accuracy drops for Euc and Lrn by 0.8%p and 0.2%p respectively.
Revised: Without N and ML, we observe consistent accuracy drops for Euc and Lrn by 0.8%p and 0.2%p respectively.

Original: A sole multi-layer projection hardly improves accuracy but the model performs the best when N is jointly used, meaning that both need to complement each other to provide diverse attention shapes as in Fig 4.
Revised: A sole multi-layer projection hardly improves accuracy but the model performs the best when N is jointly used, meaning that both need to complement each other to provide diverse attention shapes as in Fig 4.

Original: Furthermore, Euc models consistently surpasses Lrn models, implying the Euclidean distance is more straightforward encoding type than learnable vectors in capturing spatial configurations of images.
Revised: Furthermore, Euc models consistently surpasses Lrn models, implying the Euclidean distance is more straightforward encoding type than learnable vectors in capturing spatial configurations of images.

Original: network de signs bottom based attention
Revised: network de signs bottom based attention

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Comparison with state of the arts.
Revised: Comparison with state of the arts.

Original: Table 6 summarizes the results of our method and recent state of the arts.
Revised: Table 6 summarizes the results of our method and recent state of the arts.

Original: For fair comparison, the baselines used in our comparison are trained using 224 × 224 input resolution without distillations, and are grouped into either pyramidal or columnar ViT based on the network designs, i.e. , multi- or single-resolution feature processing, where the results are partitioned according to model sizes within each group.
Revised: For fair comparison, the baselines used in our comparison are trained using 224 × 224 input resolution without distillations, and are grouped into either pyramidal or columnar ViT based on the network designs, i.e. , multi- or single-resolution feature processing, where the results are partitioned according to model sizes within each group.

Original: As shown in the bottom section of Tab 6, the proposed method achieves consistent improvements over the recent columnar ViT methods [12, 19, 58, 75] while showing competitive results to the pyramidal counterparts.
Revised: As shown in the bottom section of Tab 6, the proposed method achieves consistent improvements over the recent columnar ViT methods [12, 19, 58, 75] while showing competitive results to the pyramidal counterparts.

Original:  Scope and Limitations
Revised:  Scope and Limitations

Original: Despite the interpretability and effectiveness of PerViT, it still leaves much room for improvements.
Revised: Despite the interpretability and effectiveness of PerViT, it still leaves much room for improvements.

Original: First, PerViT-Attention (Eq. 5) is based on the original self-attention formulation [63], thus directly inheriting its limitations [18, 58], e.g. , quadratic complexity w.r.t.
Revised: First, PerViT-Attention (Eq. 5) is based on the original self-attention formulation [63], thus directly inheriting its limitations [18, 58], e.g. , quadratic complexity w.r.t.

Original: input resolution.
Revised: input resolution.

Original: The computational efficiency could be further improved by approximating low-rank matrices as in [7, 37, 73].
Revised: The computational efficiency could be further improved by approximating low-rank matrices as in [7, 37, 73].

Original: Second, given the ability to process high-resolution input with feasible complexity, the efficacy of PerViT could be improved by adopting multi-resolution pyramidal design following recent trend of ViT designs [17, 35, 37, 40, 67, 69, 73, 74].
Revised: Second, given the ability to process high-resolution input with feasible complexity, the efficacy of PerViT could be improved by adopting multi-resolution pyramidal design following recent trend of ViT designs [17, 35, 37, 40, 67, 69, 73, 74].

Original: Third, the focus of this paper is model development & exploration for image classification task but we believe the proposed idea is broadly generalizable to other vision applications such as object detection and segmentation.
Revised: Third, the focus of this paper is model development & exploration for image classification task but we believe the proposed idea is broadly generalizable to other vision applications such as object detection and segmentation.

Original: We leave this to future work.
Revised: We leave this to future work.

Original:  Conclusion
Revised:  Conclusion

Original: This paper explores blending human peripheral vision with machine vision for effective visual recognition, and introduces Peripheral Vision Transformer which learns to provide diverse positionbased attentions to model peripheral vision using peripheral projections and initialization.
Revised: This paper explores blending human peripheral vision with machine vision for effective visual recognition, and introduces Peripheral Vision Transformer which learns to provide diverse positionbased attentions to model peripheral vision using peripheral projections and initialization.

Original: We have systematically investigated the inner workings of the proposed network and observed that the network enjoys the benefits of both convolution and self-attention by learning to decide level of the locality and dynamicity for the feature transformations, by the network itself given training data.
Revised: We have systematically investigated the inner workings of the proposed network and observed that the network enjoys the benefits of both convolution and self-attention by learning to decide level of the locality and dynamicity for the feature transformations, by the network itself given training data.

Original: The consistent improvements over the baseline models on ImageNet-1K classification across different model sizes and in-depth ablation study confirm the efficacy of the proposed approach.
Revised: The consistent improvements over the baseline models on ImageNet-1K classification across different model sizes and in-depth ablation study confirm the efficacy of the proposed approach.

Original:  Acknowledgments and Disclosure of Funding
Revised:  Acknowledgments and Disclosure of Funding

Original: This work was supported by the IITP grants (No.2021-0-01696: High-Potential Individuals Global
Revised: This work was supported by the IITP grants (No.2021-0-01696: High-Potential Individuals Global

Original: Training Program (40%), No.2022-0-00290:
Revised: Training Program (40%), No.2022-0-00290:

Original: Visual Intelligence for Space-Time Understanding and
Revised: Visual Intelligence for Space-Time Understanding and

Original: Generation based on Multi-layered Visual Common Sense (50%), No.2019-0-01906:
Revised: Generation based on Multi-layered Visual Common Sense (50%), No.2019-0-01906:

Original: AI Graduate
Revised: AI Graduate

Original: School Program - POSTECH (10%)) funded by Ministry of Science and ICT, Korea.
Revised: School Program - POSTECH (10%)) funded by Ministry of Science and ICT, Korea.

Original: This work was done while Juhong Min was working as an intern at Microsoft Research Asia.
Revised: This work was done while Juhong Min was working as an intern at Microsoft Research Asia.