Original: Peripheral Vision Transformer
Revised: Peripheral Vision Transformer

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Human vision possesses a special type of visual processing systems called peripheral vision .
Revised: Human vision possesses a special type of visual processing systems called peripheral vision .

Original: Partitioning the entire visual field into multiple contour regions basedon the distance to the center of our gaze, the peripheral vision provides us theability to perceive various visual features at different regions.
Revised: Partitioning the entire visual field into multiple contour regions based on the distance to the center of our gaze, the peripheral vision provides us the ability to perceive various visual features at different regions.

Original: In this work, wetake a biologically inspired approach and explore to model peripheral vision indeep neural networks for visual recognition.
Revised: In this work, we take a biologically inspired approach and explore to model peripheral vision in deep neural networks for visual recognition.

Original: We propose to incorporate peripheralposition encoding to the multi-head self-attention layers to let the network learnto partition the visual field into diverse peripheral regions given training data.
Revised: We propose to incorporate peripheral position encoding to the multi-head self-attention layers to let the network learn to partition the visual field into diverse peripheral regions given training data.

Original: Weevaluate the proposed network, dubbed PerViT, on ImageNet-1K and systematicallyinvestigate the inner workings of the model for machine perception, showing thatthe network learns to perceive visual data similarly to the way that human visiondoes.
Revised: We evaluate the proposed network, dubbed PerViT, on ImageNet-1K and systematically investigate the inner workings of the model for machine perception, showing that the network learns to perceive visual data similarly to the way that human vision does.

Original: The performance improvements in image classification over the baselinesacross different model sizes demonstrate the efficacy of the proposed method
Revised: The performance improvements in image classification over the baselines across different model sizes demonstrate the efficacy of the proposed method.

Original:  Introduction
Revised:  Introduction

Original: [Figure]
Revised: [Figure]

Original: For the last ten years, convolution has been a dominant feature transformation in neural networks for visual recognitiondue to its superiority in modelling spatial configurations ofimages [21, 32, 34].
Revised: For the last ten years, convolution has been a dominant feature transformation in neural networks for visual recognition due to its superiority in modelling spatial configurations of images [21, 32, 34].

Original: Despite the efficacy in learning visualpatterns, the local and stationary nature of convolutional kernels limited the maximum extent of representation abilityin flexible processing, e.g. , dynamic transformations withglobal receptive fields.
Revised: Despite the efficacy in learning visual patterns, the local and stationary nature of convolutional kernels limited the maximum extent of representation ability in flexible processing, e g. , dynamic transformations with global receptive fields.

Original: Originally devised for natural language processing (NLP), self-attention [58] shed a light onthis direction; equipped with adaptive input processing andthe ability to capture long-range interactions, it has emergedas an alternative feature transform for computer vision, beingwidely adopted as a core building block [18].
Revised: Originally devised for natural language processing (NLP), self-attention [63] shed a light on this direction; equipped with adaptive input processing and the ability to capture long-range interactions, it has emerged as an alternative feature transform for computer vision, being widely adopted as a core building block [18].

Original: The stand-aloneself-attention models, e.g. , ViT [18], however, demand significantly more training data [53] for competitive performancewith its convolutional counterparts [6, 25, 27, 67] since theymiss certain desirable property which convolution possesses,e.g. , locality.
Revised: The stand-alone self-attention models, e.g. , ViT [18], however, demand significantly more training data [57] for competitive performance with its convolutional counterparts [6, 25, 27, 72] since they miss certain desirable property which convolution possesses, e.g. , locality.

Original: These inherent pros and cons of convolutionand self-attention encourage recent researches toward combinations of both so as to enjoy the best of the both worlds butwhich one suits the best for effective visual processing is yetcontroversial in literature [8, 9, 10, 12, 35, 37, 39, 40, 46, 47, 55, 57, 62, 64, 66, 68, 72].
Revised: These inherent pros and cons of convolution and self-attention encourage recent researches toward combinations of both so as to enjoy the best of the both worlds but which one suits the best for effective visual processing is yet controversial in literature [8, 9, 10, 12, 35, 37, 38, 40, 41, 49, 50, 59, 60, 62, 67, 69, 71, 73, 77].

Original: Unlike the dominant visual feature transformations in machine vision, human vision possesses aspecial type of visual processing systems called peripheral vision [36]; it partitions the entire visual field into multiple contour regions based on the distances to the center of a gaze where each regionidentifies different visual aspects.
Revised: Unlike the dominant visual feature transformations in machine vision, human vision possesses a special type of visual processing systems called peripheral vision [36]; it partitions the entire visual field into multiple contour regions based on the distances to the center of a gaze where each region identifies different visual aspects.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 1, we have high-resolution processing near thecenter of our gaze, i.e. , central and para-central regions, to identify highly-detailed visual elementssuch as geometric shapes, and low-level details.
Revised: 1, we have high-resolution processing near the center of our gaze, i.e. , central and para-central regions, to identify highly-detailed visual elements such as geometric shapes, and low-level details.

Original: For the regions more distant from the gaze, i.e. ,mid and far peripheral regions, the resolution decreases to recognize abstract visual features such asmotion, and high-level contexts.
Revised: For the regions more distant from the gaze, i.e. , mid and far peripheral regions, the resolution decreases to recognize abstract visual features such as motion, and high-level contexts.

Original: This systematic strategy enables us to effectively perceive importantdetails within a small fraction (1%) of the visual field while minimizing unnecessary processing ofbackground clutter in the rest (99%), thus facilitating efficient visual processing for human brain.
Revised: This systematic strategy enables us to effectively perceive important details within a small fraction (1%) of the visual field while minimizing unnecessary processing of background clutter in the rest (99%), thus facilitating efficient visual processing for human brain.

Original: Based on recent study on inner workings of vision transformers [12, 18, 46, 54, 66], we speculate thattheir behavior is loosely related to the aforementioned visual processing strategies.
Revised: Based on recent study on inner workings of vision transformers [12, 18, 49, 58, 71], we speculate that their behavior is fairly related to the aforementioned visual processing strategies.

Original: The attention mapsof early layers are learned to locally capture fine-grained geometric details at central regions whilethose of later layers perform global attentions to identify coarse-grained semantics and contexts fromthe whole visual field, covering peripheral regions .
Revised: The attention maps of early layers are learned to locally capture fine-grained geometric details at central regions while those of later layers perform global attentions to identify coarse-grained semantics and contexts from the whole visual field, covering peripheral regions .

Original: The connections reveal that imitating biologicaldesigns may potentially help in modelling an effective machine vision, and also support recentapproaches towards a hybrid [5, 10, 12, 24, 38] of convolution and self-attention beyond stand-alonevisual processing to take advantages of the two different perception strategies: fine-grained/local andcoarse-grained/global, similarly to the  visual processing illustrated in Fig.
Revised: The connections reveal that imitating biological designs may potentially help in modelling an effective machine vision, and also support recent approaches towards a hybrid [5, 10, 12, 24, 39] of convolution and self-attention beyond stand-alone visual processing to take advantages of the two different perception strategies: fine-grained/local and coarse-grained/global, similarly to the human visual processing illustrated in Fig.

Original: In this work, we take a biologically inspired approach and propose to inject the peripheral inductivebiases 1 to deep neural networks for image recognition.
Revised: In this work, we take a biologically inspired approach and propose to inject the peripheral inductive biases 1 to deep neural networks for image recognition.

Original: We propose to incorporate peripheral attentionmechanism to the multi-head self-attention [58] to let the network learn to partition the visual fieldinto diverse peripheral regions given training data where each region captures different visual features.
Revised: We propose to incorporate peripheral attention mechanism to the multi-head self-attention [63] to let the network learn to partition the visual field into diverse peripheral regions given training data where each region captures different visual features.

Original: We experimentally show that the proposed network models effective visual periphery for reliablevisual recognition.
Revised: We experimentally show that the proposed network models effective visual periphery for reliable visual recognition.

Original: Our main contributions can be summarized as follows:
Revised: Our main contributions can be summarized as follows:

Original: • This work explores to narrow the gap between human and machine vision by injectingperipheral inductive biases to self-attention layers, and presents a new form of featuretransformations named M ulti-head P eripheral A ttention (MPA).• Based on the MPA, we introduce Per ipheral Vi sion T ransformer (PerViT), and systematically study the inner workings of PerViT by qualitatively and quantitatively analyzing itslearned attentions, which reveal that the network learns to perceive visual elements similarlyto the way that human vision does without any special supervisions.
Revised: • This work explores to narrow the gap between human and machine vision by injecting peripheral inductive biases to self-attention layers, and presents a new form of feature transformations named M ulti-head P eripheral A ttention (MPA). • Based on the MPA, we introduce Per ipheral Vi sion T ransformer (PerViT), and systematically study the inner workings of PerViT by qualitatively and quantitatively analyzing its learned attentions, which reveal that the network learns to perceive visual elements similarly to the way that human vision does without any special supervisions.

Original: • The performance improvements in image classificationtask overthe columnar Transformerbaselines, e.g. , DeiT, across different model sizes demonstrate the efficacy of the proposedapproach.
Revised: • The performance improvements in image classification over columnar Transformer baselines, e.g. , DeiT, across different model sizes demonstrate the efficacy of the proposed approach.

Original:  Related Work
Revised:  Related Work

Original: Feature transformations in computer vision.
Revised: Feature transformations in computer vision.

Original: With notable success in NLP, Transformers [14, 58]introduced a paradigm shift in computer vision [3, 4, 7, 18, 30, 47, 51, 52, 54, 55, 57].
Revised: With notable success in NLP, Transformers [14, 63] introduced a paradigm shift in computer vision [3, 4, 7, 18, 30, 50, 55, 56, 58, 59, 62].

Original: Despitetheir generalization capability, pure Vision Transformers [18] require extensive amount of trainingdata to capture spatial layout of images due to lack of certain desirable property, e.g. , locality.
Revised: Despite their generalization capability, pure Vision Transformers [18] require extensive amount of training data to capture spatial layout of images due to lack of certain desirable property, e.g. , locality.

Original: Thisencouraged many recent ViT work to incorporate local inductive biases via distillation of convolutional biases [54], local self-attention [35, 39, 68], a hybridization [10, 12, 37], and augmentingconvolutions [8, 62, 64], all of which convey a unified message: “Despite high generalizibility ofself-attentions, a sufficient amount of convolutional processing must be incorporated to capture thespatial configurations of images for reliable visual processing.”
Revised: This encouraged many recent ViT work to incorporate local inductive biases via distillation of convolutional biases [58], local self-attention [35, 40, 73], a hybridization [10, 12, 37], and augmenting convolutions [8, 60, 67, 69], all of which convey a unified message: “Despite high generalizibility of self-attentions, a sufficient amount of convolutional processing must be incorporated to capture the spatial configurations of images for reliable visual processing.”

Original: Position encoding for Transformer.
Revised: Position encoding for Transformer.

Original: Witnessing the efficacy of position encoding in capturing inputstructures in NLP [11, 28, 50], recent vision models [18, 47, 65] have begun employing positionencodings for images to model spatial structures of images.
Revised: Witnessing the efficacy of position encoding in capturing input structures in NLP [11, 28, 54], recent vision models [18, 50, 70] have begun employing position encodings for images to model spatial structures of images.

Original: In particular, relative position encoding(RPE) plays a vital role for the purpose: The work of Cordonnier et al.
Revised: In particular, relative position encoding (RPE) plays a vital role for the purpose: The work of Cordonnier et al.

Original: [9] proves that self-attentionhas close relationship with convolution when equipped with certain form of RPE.
Revised: [9] proves that self-attention has close relationship with convolution when equipped with certain form of RPE.

Original: Wu et al.
Revised: Wu et al.

Original: [65]explores the existing RPE methods used in vision transformers [11, 28, 50, 60] and draws a conclusionthat RPEs imposes convolutional processing on vision transformers.
Revised: [70] explores the existing RPE methods used in vision transformers [11, 28, 54, 65] and draws a conclusion that RPEs imposes convolutional processing on vision transformers.

Original: Dai et al.
Revised: Dai et al.

Original: [10] observe that depthwise convolution and self-attention can naturally be unified via RPEs.
Revised: [10] observe that depthwise convolution and self-attention can naturally be unified via RPEs.

Original: While offering promisingdirections, the previous RPE work, however, is limited in the sense that the focus of RPE utilizationis restricted to only local attention, e.g. , convolution.
Revised: While offering promising directions, the previous RPE work, however, is limited in the sense that the focus of RPE utilization is restricted to only local attention, e.g. , convolution.

Original: We exploit RPEs to devise an original visualfeature transformation which is more general than convolution while being more structured thanself-attention via imitation of human visual system: peripheral vision .
Revised: We exploit RPEs to devise an original visual feature transformation which is more general than convolution while being more structured than self-attention via imitation of human visual system: peripheral vision .

Original: Peripheral vision for machine perception.
Revised: Peripheral vision for machine perception.

Original: Along with central vision, peripheral vision plays anessential role in a wide range of visual recognition tasks [33].
Revised: Along with central vision, peripheral vision plays an essential role in a wide range of visual recognition tasks [33].

Original: The fundamental mechanisms ofperipheral visual processing, however, have not been fully disclosed in  vision literature [48] whichstimulated many researchers to reveal its inner workings and deep implications [2, 15, 16, 42, 49, 63].
Revised: The fundamental mechanisms of peripheral visual processing, however, have not been fully disclosed in human vision literature [52] which stimulated many researchers to reveal its inner workings and deep implications [2, 15, 16, 43, 53, 68].

Original: The work of Rosenholtz [48] discusses pervasive myths and current findings about peripheral vision,suggesting that peripheral vision is more crucial for human perception than previously thoughtto perform diverse important tasks.
Revised: The work of Rosenholtz [52] discusses pervasive myths and current findings about peripheral vision, suggesting that peripheral vision is more crucial for human perception than previously thought to perform diverse important tasks.

Original: Inspired by its importance, a number of pioneering work[16, 22, 23, 20, 43, 61] investigate the linkage between peripheral vision and machine vision, e.g. , CNNs, while some [31, 59] devise biologically-inspired models for the creation of stronger machinevision.
Revised: Inspired by its importance, a number of pioneering work [16, 20, 22, 23, 44, 66] investigate the linkage between peripheral vision and machine vision, e.g. , CNNs, while some [31, 64] devise biologically-inspired models for the creation of stronger machine vision.

Original: Continuing previous study, this paper explores to blend human peripheral vision with attentionbased neural networks, e.g. , vision transformer [18].
Revised: Continuing previous study, this paper explores to blend human peripheral vision with attention-based neural networks, e.g. , vision transformer [18, 58].

Original:  Our Approach
Revised:  Our Approach

Revised: In this section, we introduce the Peripheral Vision Transformer (PerViT) that learns to model peripheral vision for effective image recognition.

Original: We omit the existing text (Background: Multi-Head Self-Attention) in this draft due to 9-page limitfor the revised manuscript.
Revised: We first revisit the mathematical formulation of self-attention and then describe how we improve it with peripheral inductive biases. Background: Multi-Head Self-Attention. The multi-head self-attention (MHSA) [63] with N h heads performs an attention-based feature transformation by aggregating N h self-attention outputs: [Equation]

Revised: where X ∈ R HW × D emb is a set of input tokens and W out ∈ R N h D h × D emb and b out ∈ R D emb are the transformation parameters.

Revised: The N h outputs of self-attention are designed to extract a diverse set of features from the input representation.

Revised: Formally, the self-attention at head h is defined as

Revised: [Equation]

Original: We will include the text in our final manuscript (10-page limit) if accepted.
Revised: where Normalize [ · ] denotes a row-wise normalization and Φ ( h ) ( · ) ∈ R HW × HW is a function that provides spatial attentions based on content information to aggregate the values V ( h ) = XW ( h ) val : [Equation]

Revised: using linear projections of W ( h ) qry , W ( h ) key , W ( h ) val ∈ R D

Revised: emb × D h for queries, keys, and values respectively.

Original: 3.1 Peripheral Vision Transformer
Revised: 3.1 Peripheral Vision Transformer

Original: Based on the formulation of the MHSA, we define M ulti-head P eripheral A ttention (MPA) as
Revised: Based on the formulation of the MHSA, we define M ulti-head P eripheral A ttention (MPA) as

Original: [Equation]
Revised: [Equation]

Original: where R ∈ R HW × HW × D r is the relative position encoding with D r channel dimension.
Revised: where R ∈ R HW × HW × D r is the relative position encoding with D r channel dimension.

Original: The selfattention in MHSA is now replaced with Peripheral-Attention, consisting of content- and positionbased attention functions ( Φ ( h ) c , Φ ( h ) p ∈ R
Revised: The selfattention in MHSA is now replaced with Peripheral-Attention, consisting of content- and positionbased attention functions ( Φ ( h ) c , Φ ( h ) p ∈ [Equation]

Original: HW × HW ), which is defined as
Revised: HW × HW ), which is defined as

Original: Peripheral-
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ⊙ is Hadamard product which mixes the given pair of attentions to provide a mixed attention Φ ( h ) a ( X , R ) :
Revised: where ⊙ is Hadamard product which mixes the given pair of attentions to provide a mixed attention Φ ( h ) a ( X , R ) :

Original: = Φ ( h ) c ( X ) ⊙ Φ ( h ) p (R )
Revised: = Φ ( h ) c ( X ) ⊙ Φ ( h ) p (

Original: ∈ R HW × HW . For thecontent-based attention Φ c ,we use exponentiated (scaled) dot-product between queries and keys as in Eqn. ?? : Φ ( h ) c ( X ) :=exp( τ Q ( h ) K ( h ) ⊤ ) .
Revised: For the

Original: For the position-based attention Φ p , we design a neural network that aims toimitate human visual system, e.g. , peripheral vision.
Revised: -based attention Φ p , we design a neural network that aims to imitate human visual system, e.g. , peripheral vision.

Original: Modelling peripheral vision: a Roadmap.
Revised: Modelling peripheral vision: a Roadmap.

Original: Human visual field can be grouped into several regionsbased on the Euclidean distances from the center of gaze, each forming ring-shaped region as seen in
Revised: Human visual field can be grouped into several regions based on the Euclidean distances from the center of gaze, each forming ring-shaped region as seen in

Original: Fig.
Revised: Fig.

Original: 1, where each region captures different visual aspects; the closer to the gaze, the more complexfeatures we process, and further from the gaze, the simpler visual features we perceive.
Revised: 1, where each region captures different visual aspects; the closer to the gaze, the more complex features we process, and further from the gaze, the simpler visual features we perceive.

Original: In the contextof 2-dimensional attention map Φ ∗ ( · ) q , : ∈ R HW , we refer the query position q ∈ R 2 as the centerof gaze, i.e. , the position where feature of our interest lies at for transformation.
Revised: In the context of 2-dimensional attention map Φ ∗ ( · ) q , : ∈ R HW , we refer the query position q ∈ R 2 as the center of gaze, i.e. , the position where feature of our interest lies at for transformation.

Original: We refer the localregions around q as central/para-central regions and the rest as mid/far peripheral regions.
Revised: We refer the local regions around q as central/para-central regions and the rest as mid/far peripheral regions.

Original: Perhaps the simplest approach to divide the visual field into multiple subregions is to perform asingle linear projection on the Euclidean distances, i.e. , Φ ( h ) p ( R ) =
Revised: Perhaps the simplest approach to divide the visual field into multiple subregions is to perform a single linear projection on the Euclidean distances, i.e. , Φ ( h ) p ( R ) =

Original: σ (cid:2) RW ( h ) p (cid:3) where W ( h ) p ∈ [Equation] ,similarly the previous work of Wu et al.
Revised: σ (cid:2) RW ( h ) p (cid:3) where W ( h ) p ∈ R D r , similarly the previous work of Wu et al.

Original: For straightforward imitation of peripheral vision, weuse Euclidean distance for relative position input R and weigh the distances in D r different ways forthe network to learn the mapping in multiple scales: R q , k , : :=concat r ∈ [ D r ] [ w r · R euc q , k ] ∈ R D r where{ w r } r ∈ [ D r ] is a set of learnable parameters shared across layers and heads, and R euc q , k = ∥
Revised: For straightforward imitation of peripheral vision, we use Euclidean distance for relative position input R and weigh the distances in D r different ways for the network to learn the mapping in multiple scales: R q , k , : :=

Original: [-q − k ∥ 
-]
Revised: [-q − k ∥ 
-]

Original: Given σ [ · ] := exp( · ) , Peripheral-Attention ( h ) =

Original: Normalize [exp( Q ( h ) K ( h ) ⊤ ) ⊙ exp( RW ( h ) p )] V ( h ) = softmax ( Q ( h ) K ( h ) ⊤ + RW rpe ) V ( h ) , which is equivalent to the bias mode RPE presented in [65].
Revised: concat r ∈

Original: Multi

Original: [Figure] Layer Layer Layer
Revised: [ D r ] [ w r · R euc q , k ] ∈ R D r where { w r } r ∈

Original: Layer Layer Peripheral initialization [Figure]
Revised: [ D r ] is a set of learnable parameters shared across layers and heads, and

Original: MPA

Original: FFN

Original: Convolutionalpatch embedding

Original: GAP

Original: Image embedding

Original: Input image ×𝑁 !

Original: Scaled dot product 𝑸 (#) 𝑲 (#) 𝑽 (#) 𝐗∈ ℝ %&×( !"

Original: Peripheralprojection 𝐑∈ ℝ %&×%&×( $

Original: Peripheralprojection

Original: IN & ReLU IN & Sigmoid Exp Matrix mult.
Revised: R euc q , k = ∥

Original: 𝛗 !

Original: 𝛗 "

Original: CPE Normalize Linear Linear Linear 𝒉 (𝐗,𝐑) Peripheral-Attention ×𝑁 # is the Euclidean distance between query and key positions q , k ∈ R 2 .
Revised: q − k ∥is the Euclidean distance between query and key positions q , k ∈ R 2 .

Original: For σ , we choose sigmoid toprovide normalized (positive) weights to the content-based attention Φ c .
Revised: For σ , we choose sigmoid to provide normalized (positive) weights to the content-based attention Φ c .

Original: A main drawback of thissingle-layer formulation is that Φ p is only able to provide Gaussian-like attention map as seen intop-left in Fig.
Revised: A main drawback of this single-layer formulation is that Φ p is only able to provide Gaussian-like attention map as seen in top-left in Fig.

Original: 2, thus being unable to represent diverse peripheral regions.
Revised: 2, thus being unable to represent diverse peripheral regions.

Original: For the encoding functionto represent diverse (torus-shaped) peripheral regions, the distances must be processed by an MLP: 
Revised: For the encoding function to represent diverse (torus-shaped) peripheral regions, the distances must be processed by an MLP: Φ

Original: [Equation]
Revised: [Equation]

Original: (3) where W p1 ∈ R D r × D hid and W ( h ) p2 ∈ R D hid are the linear projection parameters 3 , and ReLU givesnon-linearity to the function.
Revised: ReLU [Equation] where W p1 ∈ R D r × D hid and W ( h ) p2 ∈ R D hid are the linear projection parameters 3 , and ReLU gives non-linearity to the function.

Original: The first projection W p1 is shared across the heads in order to exchangeinformation so each function is able to provide attention that are effective or complementary toother heads’ attention.
Revised: The first projection W p1 is shared across the heads in order to exchange information so each function is able to provide attention that are effective or complementary to other heads’ attention.

Original: Note that given identical relative distances between a fixed query pointq ∈ R 2 and key points k
Revised: Note that given identical relative distances between a fixed query point q ∈ R 2 and key points k

Original: i , k j ∈ R 2 , i.e. , R q , k i =
Revised: i , k j ∈ R 2 , i.e. , R q , k i =

Original: R q , k j , Eq. 3 provides the same attention scores:Φ p ( R ) q , k i =
Revised: R q , k j , Eq. 6 provides the same attention scores: Φ p ( R ) q , k i =

Original: Φ p ( R ) q , k j as seen in top-right of Fig.
Revised: Φ p ( R ) q , k j as seen in top-right of Fig.

Original: This property, however, is not always desiredin practical scenarios because the rotational symmetric property hardly holds for most real-worldobjects.
Revised: This property, however, is not always desired in practical scenarios because the rotational symmetric property hardly holds for most real-world objects.

Original: To break the symmetric property in Eq.
Revised: To break the symmetric property in Eq.

Original: 3 while preserving peripheral design to sufficientextent, we introduce peripheral projections in which the transformation parameters are given smallspatial resolutions, e.g. , K × K window, such that W p1 ∈ R K 2 × D r × D hid and W ( h ) p2 ∈ R K 2 × D hid so thatthey provide similar but different attention scores, Φ p ( R ) q , k i ̸
Revised: 6 while preserving peripheral design to sufficient extent, we introduce peripheral projections in which the transformation parameters are given small spatial resolutions, e.g. , K × K window, such that W p1 ∈ R K 2 × D r × D hid and W ( h ) p2 ∈ R K 2 × D hid so that they provide similar but different attention scores, Φ p ( R ) q , k i ̸

Original: = Φ p ( R ) q , k j , given R q , k i
Revised: = Φ p ( R ) q , k j , given R q , k i

Original: = R q , k j ,146 by aggregating neighboring relative distances around the key location k as follows:
Revised: = R q , k j , by aggregating neighboring relative distances around the key location k as follows:

Original: [Equation]
Revised: [Equation]

Original: where N ( k ) := (cid:2) k − ⌊ K 2 ⌋ , . . .
Revised: where N ( k ) := (cid:2) k − ⌊ K 2 ⌋ , . . .

Original: , k
Revised: , k

Original: + ⌊ K 2 ⌋ (cid:3) × (cid:2) k − ⌊ K 2 ⌋ , . . .
Revised: + ⌊ K 2 ⌋ (cid:3) × [Equation] k − ⌊ K 2 ⌋ [Equation]

Original: , k
Revised: [Equation] k

Original: + ⌊ K 2 ⌋ (cid:3) is a set of K 2 neighborsaround input position k .
Revised: [Equation] ⌊ K 2 ⌋ (cid:3) is a set of K 2 neighbors around input position k .

Original: We set K = 3 for all layers and heads as K > 3 hardly brought improvements.
Revised: We set K = 3 for all layers and heads as K > 3 hardly brought improvements.

Revised: Note that this formulation can be considered as a variant of 4-dimensional convolution [51] in the sense that the function Φ p takes 4-dimensional input R ∈ R

Revised: HW × HW × D r and processes it in convolutional manner with 4-dimensional kernels in size of K × K × 1 × 1 , i.e. ,

Revised: W p1 ∈ R K × K × 1 × 1 × D r × D hid .

Revised: Precisely, each peripheral projection only considers a subset of 4D local neighbors that pivots the query position q , similarly to the center-pivot 4D convolution [45, 46].

Original: After each peripheral projection, we add an instance normalization layer [56] for stableoptimization:
Revised: After each peripheral projection, we add an instance normalization layer [61] for stable optimization:

Original: [Equation]
Revised: [Equation]

Original: where γγγ p1 ,βββ p1 ∈ R D hid and γ ( h ) p2 , β ( h ) p2 ∈ R are weights/biases of the instance norms and PP ( · ) denotesthe peripheral projection: PP ( R , W ) q , k , : := (cid:80) n ∈N ( k ) R q , n , : W n − k , : , : .
Revised: where γγγ p1 ,βββ p1 ∈ R D hid and γ ( h ) p2 , β ( h ) p2 ∈ R are weights/biases of the instance norms and PP ( · ) denotes the peripheral projection: PP ( R , W ) q , k , : := (cid:80) n ∈N ( k ) R q , n , : W n − k , : , : .

Original: The middle row of Fig. 2depicts learned attentions of Φ p with peripheral projections, which provides peripheral attention mapsin greater diversity compared to single- and multi-layer counterparts without N . 
Revised: The middle row of Fig.depicts learned attentions of Φ p with peripheral projections, which provides peripheral attention maps in greater diversity compared to single- and multi-layer counterparts without N . Multi-layer projection Layer

Revised: Layer

Revised: Layer

Revised: Layer

Original:  Peripheral initialization.
Revised: Layer Peripheral initialization

Revised: [Figure]

Revised: MPA

Revised: FFN

Revised: Convolutionalpatch embedding

Revised: GAP

Revised: Image embedding

Revised: Input image ×𝑁 !

Revised: Scaled dot product 𝑸 (#) 𝑲 (#) 𝑽 (#) 𝐗∈ ℝ %&×( !"

Revised: Peripheralprojection 𝐑∈ ℝ %&×%&×( $

Revised: Peripheralprojection

Revised: IN & ReLU

Revised: IN & Sigmoid

Revised: Exp

Revised: Matrix mult.

Revised: 𝛗 !

Revised: 𝛗 "

Revised: CPE

Revised: Normalize

Original:  Recent study [46, 54] observe that early layers of trained vision transformerlearn to attend locally whereas late layers perform global attentions.
Revised: Linear Linear Linear 𝒉 (𝐗,𝐑) Peripheral-Attention ×𝑁 # Peripheral initialization. Recent study [49, 58] observe that early layers of trained vision transformer learn to attend locally whereas late layers perform global attentions.

Original: To facilitate training of ournetwork, we inject this property in the beginning of the training stage by initializing parameters of Φ p for the purpose, making attention scores near the queries larger than the distant ones inthe early layers while uniformly distributing them in the late layers as seen in bottom row of
Revised: To facilitate training of our network, we inject this property in the beginning of the training stage by initializing parameters of Φ p for the purpose, making attention scores near the queries larger than the distant ones in the early layers while uniformly distributing them in the late layers as seen in bottom row of

Original: Fig.
Revised: Fig.

Original: We refer this method as peripheral initialization for its resemblance to the functioningof peripheral vision [36] which also operates either locally or globally to perceive different visualfeatures [71].
Revised: We refer this method as peripheral initialization for its resemblance to the functioning of peripheral vision [36] which also operates either locally or globally to perceive different visual features [76].

Original: To formally put, given two arbitrarily chosen distances R euc q , k i , R euc q , k j ∈ R which satisfy
Revised: To formally put, given two arbitrarily chosen distances R euc q , k i , R euc q , k j ∈ R which satisfy

Original: R euc q , k i < R euc q , k j , we want Φ ( l,h ) p ( R ) q , ki ≫ Φ [Equation] l,
Revised: R euc q , k i < R euc q , k j , we want Φ ( l,h ) p ( R ) q , k

Original: [Equation] ) p ( R ) q , k j 4 as l → 1 , i.e. , local attention in earlylayers, and Φ ( l,h ) p( R ) q , [Equation] i ≈ Φ
Revised: i ≫ Φ ( l,h ) p ( R ) q , k j 4 as l → 1 , i.e. , local attention in early layers, and Φ ( l,h ) p

Original: [Equation] l,
Revised: ( R ) q , k

Original: [Equation] ) p ( R ) q , k j as l → N l , i.e. , global attention in late layers, where N lis the total number of MPA layers.
Revised: i ≈ Φ ( l,h ) p ( R ) q , k j as l → N l , i.e. , global attention in late layers, where N l is the total number of MPA layers.

Original: We first initialize the parameters of Φ ( l,h ) p and { w r } r ∈
Revised: We first initialize the parameters of Φ ( l,h ) p and { w r } r ∈

Original: [ D r ] toparticular values.
Revised: [ D r ] to particular values.

Original: Specifically, for all layers l ∈ [ N l ] and heads h ∈
Revised: Specifically, for all layers l ∈ [ N l ] and heads h ∈

Original: [ N h ] ,
Revised: [ N h ] ,

Original: [Equation]
Revised: [Equation]

Original: ) where c 1 , c 2 ∈ R + are positive reals, and J N,M ∈ R N × M refers to all-one matrix in size N ×
Revised: ) where c 1 , c 2 ∈ R + are positive reals, and J N,M ∈ R N × M refers to all-one matrix in size N ×

Original: M .
Revised: M .

Original: The above initialization provides local attention after the second peripheral projection, i.e. ,
Revised: The above initialization provides local attention after the second peripheral projection, i.e. ,

Original: PP ( R ′ ; W ( h ) p2 ) q , k i , : > PP ( R ′ ; W ( h ) p2 ) q , k j , : given R euc q , k i < R euc q , k j .
Revised: PP ( R ′ ; W ( h ) p2 ) q , k i , : > PP ( R ′ ; W ( h ) p2 ) q , k j , : given R euc q , k i < R euc q , k j .

Original: Next, based on our findings thatbiases β ( l,h ) p and the weights γ ( l,h ) p in the second instance norm control the sizes and strengths oflocal attention respectively, we simulate peripheral initialization by setting their initial values asβ ( l,h ) p :=
Revised: Next, based on our findings that biases β ( l,h ) p and the weights γ ( l,h ) p in the second instance norm control the sizes and strengths of local attention respectively, we simulate peripheral initialization by setting their initial values as β ( l,h ) p :=

Original: s l and γ ( l,h ) p :=
Revised: s l and γ ( l,h ) p :=

Original: v l where respective { s l } l ∈ [ N l ] and { v l } l ∈ [ N l ] are sets of initial values forattention sizes and strengths.
Revised: v l where respective { s l } l ∈ [ N l ] and { v l } l ∈ [ N l ] are sets of initial values for attention sizes and strengths.

Original: We set their values collected from uniform intervals: s l ∈ [ − 5 . 0 , 4 . 0]and v l ∈ [3 . 0 , 0 .
Revised: We set their values collected from uniform intervals: s l ∈ [ − 5 . 0 , 4 . 0] and v l ∈ [3 . 0 , 0 .

Original: 01] where s l − 1 < s l and v l − 1 > v l to give stronger local attentions to shallow layerscompared to deep ones as seen in bottom row of Fig.
Revised: 01] where s l − 1 < s l and v l − 1 > v l to give stronger local attentions to shallow layers compared to deep ones as seen in bottom row of Fig.

Original: We set c 1 , c 2 = 0 .
Revised: We set c 1 , c 2 = 0 .

Original:  in our experiments andleave all the details on the peripheral initialization to the supplementary.
Revised: 02 in our experiments and leave the details on the peripheral initialization to the supplementary.

Original: 3.2 Overall Architecture
Revised: 3.2 Overall Architecture

Original: Based on the proposed peripheral projections and initialization, we develop image classificationmodels, dubbed Peripheral Vision Transformer, which is illustrated in Fig.
Revised: Based on the proposed peripheral projections and initialization, we develop image classification models, dubbed Peripheral Vision Transformer, which is illustrated in Fig.

Original: We follow similararchitecture to DeiT [54] with convolutional patch embedding stem; as the original patchify stem [18]exhibits substandard optimizability due to its coarse-grained early visual processing [66], many recent
Revised: We follow similar architecture to DeiT [58] with convolutional patch embedding stem; as the original patchify stem [18] exhibits substandard optimizability due to its coarse-grained early visual processing [71], many recent

Original: ViT models adopt multi-resolution pyramidal designs [39, 62, 64, 68] to mitigate the issue.
Revised: ViT models adopt multi-resolution pyramidal designs [40, 67, 69, 73] to mitigate the issue.

Original: Whilethe pyramidal models have shown their efficacy in learning reliable image embeddings, we stick withthe original single-resolution columnar design for PerViT because features in multiple resolutionmake our study less interpretable, which further brings additional technical efforts in combining ourperipheral attention Φ p with the existing cost-effective self-attentions mechanisms such as factorizedattention [68], shifted-window [39], and cross-shaped window [17].
Revised: While the pyramidal models have shown their efficacy in learning reliable image embeddings, we stick with the original single-resolution columnar design for PerViT because features in multiple resolution make our study less interpretable, which further requires additional techniques for combining our [Figure] H e a d s l o c a l g l o b a l [Figure]

Revised: Central

Original: To carry out fine-grained earlyprocessing while keeping single-resolution features across the layers, we adopt convolutional patchembedding layer [66] with multi-stage layouts for channel dimensions.
Revised: Para-central Mid peripheral Far peripheral Tiny

Revised: Small

Original: {+Medium 0 26 812 141 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 
+}
Revised: {+Medium 0 26 812 141 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 
+}

Revised: [Figure]

Revised: peripheral attention Φ p with the existing cost-effective self-attentions mechanisms such as factorized attention [73], shifted-window [40], and cross-shaped window [17].

Revised: To carry out fine-grained early processing while keeping single-resolution features across the layers, we adopt convolutional patch embedding layer [71] with multi-stage layouts for channel dimensions.

Original: The convolutional embeddinglayer consists of four 3 × 3 and one 1 × 1 convolutions where the 3 × 3 convolutions are followed bybatch norm [29] and ReLU [44].
Revised: The convolutional embedding layer consists of four 3 × 3 and one 1 × 1 convolutions where the 3 × 3 convolutions are followed by batch norm [29] and ReLU [47].

Original: For additional details, we refer to the supplementary.
Revised: For additional details, we refer to the supplementary.

Original: Peripheral Vision Transformer.
Revised: Peripheral Vision Transformer.

Original: Given an image, the convolutional patch embedding provides tokenembeddings X (1) ∈ R HW × D emb .
Revised: Given an image, the convolutional patch embedding provides token embeddings X (1) ∈ R HW × D emb .

Original: Similarly to [18, 54], the embeddings are fed to a series of N lblocks each of which consists of an MPA layer and a feed-forward network with residual pathways:
Revised: Similarly to [18, 58], the embeddings are fed to a series of N l blocks each of which consists of an MPA layer and a feed-forward network with residual pathways:

Original: [Equation]
Revised: [Equation]

Original:  where LN is layer normalization [1], and FFN is an MLP consisting of two linear transformations witha GELU activation [26].
Revised: 10) where LN is layer normalization [1], and FFN is an MLP consisting of two linear transformations with a GELU activation [26].

Original: Following the work of [35, 68], we adopt convolutional position encoding(CPE), i.e. , a 3 × 3 depth-wise convolution, before first layer norm for its efficacy with negligiblecomputational cost.
Revised: Following the work of [35, 73], we adopt convolutional position encoding (CPE), i.e. , a 3 × 3 depth-wise convolution, before first layer norm for its efficacy with negligible computational cost.

Original: The output X ( N h ) is global-average pooled to form an image embedding.
Revised: The output X ( N h ) is global-average pooled to form an image embedding.

Original:  Experiments
Revised:  Experiments

Original: In this section, we first investigate the inner workings of PerViT trained on ImageNet-1K classificationdataset to examine how it benefits from the proposed peripheral projections and initialization, andthen compare the method  state  the arts settings [Figure] H e a d s l o c a l g l o b a l [Figure]
Revised: In this section, we first investigate the inner workings of PerViT trained on ImageNet-1K classification dataset to examine how it benefits from the proposed peripheral projections and initialization, and then compare the method with previous state of the arts under comparable settings.

Original: [Figure]

Original: Mid peripheral

Original: Far peripheral

Original: Tiny

Original: Small

Original: [-Medium 0 26 812 141 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 
-]
Revised: [-Medium 0 26 812 141 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 
-]

Original: [Figure]

Original: Experimental setup.
Revised: Experimental setup.

Original: Our experiments focus on image classification on ImageNet-1K [13].
Revised: Our experiments focus on image classification on ImageNet-1K [13].

Original: Followingtraining recipes of DeiT [54], we train our model on ImageNet-1K from scratch with batch size of1024, learning rate of 0.001 using AdamW [41] optimizer, cosine learning rate decay scheduler, andthe same data augmentations [14] for 300 epochs, including5 warm-up epochs.
Revised: Following training recipes of DeiT [58], we train our model on ImageNet-1K from scratch with batch size of 1024, learning rate of 0.001 using AdamW [42] optimizer, cosine learning rate decay scheduler, and the same data augmentations [14] for 300 epochs, including warm-up epochs.

Original: We evaluate ourmodel with three different sizes, e.g. , Tiny (T), Small (S), and Medium (M).
Revised: We evaluate our model with three different sizes, e.g. , Tiny (T), Small (S), and Medium (M).

Original: We use stochastic depthsof 0.0, 0.1, and 0.2 for T, S, and M respectively.
Revised: We use stochastic depths of 0.0, 0.1, and 0.2 for T, S, and M respectively.

Original: Refer to the supplmentary for additional details.
Revised: Refer to the supplmentary for additional details.

Original: 4.1 The inner workings of PerViT
Revised: 4.1 The inner workings of PerViT

Original: Learning peripheral vision.
Revised: Learning peripheral vision.

Original: We omit the existing text in this draft due to 9-page limit for the revisedmanuscript.
Revised: We begin by investigating how PerViT models peripheral vision by qualitatively analyzing its learned attention of Φ p . Figure 4 depicts the learned attention map of Φ ( l,h ) p q , : ∈ R HW for all layers and heads where the query position is given at the center, i.e. , q = [7 , 7] ⊤ 5 . We observe that the attentions are learned to be in diverse shapes of peripheral regions. Interestingly, without any special supervisions, the four attended regions ( N h = 4 ) in most layers are learned to complement each other to cover the entire visual field, capturing different visual aspects at each region (head), similarly to human peripheral vision illustrated in Fig.

Revised: For example, first two heads in Layer 3 attend the central regions while the others cover the rest peripheral regions.

Revised: The second and third heads in Layer 8 cover top and bottom hemicircles respectively, forming a circular-shaped semi-global receptive field.

Revised: Moreover, a large number of early attentions is in form of central/para-central regions while those of late layers are learned to cover mid to far peripheral regions.

Revised: To quantitatively inspect how PerViT models the peripheral visual system, we classify every feature transformation layer in the network into one of the four visual regions P ∈ { c , p , m , f } where respective elements refer to central, para-central, mid, and far peripheral regions.

Original: We will include the text in our final manuscript (10-page limit) if accepted.
Revised: PerViT-Attention of head h at layer l is classified as peripheral region p if the average of its attention scores which fall in visual region p is the largest among the others: [Equation] where P is a set of spatial positions ( |P| = HW ) and I p is distance range (real-valued interval) of peripheral region p 6 . The pie charts of Fig.

Revised: 5 describe the proportions of peripheral regions for Tiny,

Revised: Small, and Medium models where the bar graphs show them in layer-wise manner 7 .

Revised: Similarly to the visualized attention maps in Fig.

Revised: 4, the early layers attends central/para-central regions whereas deeper ones focus on outer region.

Revised: We observe that, as the model size grows, the number of mid/far peripheral attention increases whereas that of central/para-central attention stays similar, suggesting that the models no longer require local attentions once sufficient amount of processing is done in the central region because, we hypothesize, identifying geometric patterns, e.g. , corners and edges, is relatively simpler process than understanding high-level semantics.

Original: Inspecting the impact of attentions (static vs. dynamic).
Revised: Inspecting the impact of attentions (static vs. dynamic).

Original: To study how position-based attentionsΦ p contribute to the mixed attentions Φ a = Φ c ⊙ Φ p , we collect sample images and visualize theirattention maps of Layers 3, 4 and 8 in Fig.
Revised: To study how position-based attentions Φ p contribute to the mixed attentions Φ a = Φ c ⊙ Φ p , we collect sample images and visualize their attention maps of Layers 3, 4 and 8 in Fig.

Original: The mixed attentions Φ a at Layer 4 are formeddynamically ( Φ c ) within statically-formed region ( Φ p ) while the attentions Φ a at Layer 8 weaklyexploit position information ( Φ p ) to form dynamic attentions ( Φ c ).
Revised: The mixed attentions Φ a at Layer 4 are formed dynamically ( Φ c ) within statically-formed region ( Φ p ) while the attentions Φ a at Layer 8 weakly exploit position information ( Φ p ) to form dynamic attentions ( Φ c ).

Original: The results reveal that Φ p playstwo different roles; it imposes semi-dynamic attention if the attended region is focused in a smallarea whereas it serves as position bias injection when the attended region is relatively broad.
Revised: The results reveal that Φ p plays two different roles; it imposes semi-dynamic attention if the attended region is focused in a small area whereas it serves as position bias injection when the attended region is relatively broad.

Original: Inthe supplementary, we constructively prove that an MPA layer in extreme case of semi-dynamicattention/position bias injection is in turn convolution/multi-head self-attention , naturally generalizingthe both transformations.
Revised: In the supplementary, we constructively prove that an MPA layer in extreme case of semi-dynamic attention/position bias injection is in turn convolution/multi-head self-attention , naturally generalizing the both transformations.

Original: To quantitatively examine the contributions of Φ c and Φ p to the mixedattention Φ a , we define a measure of ‘impact’ by taking inverse of difference between two attentions: 
Revised: To quantitatively examine the contributions of Φ c and Φ p to the mixed attention Φ a , we define a measure of ‘impact’ by taking inverse of difference between two attentions: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ∥·∥ F is Frobenius norm.
Revised: where ∥·∥ F is Frobenius norm.

Original: The higher the measure Ψ ( l,h ) p , the larger the impact of position-basedattention Φ ( l,h ) p .
Revised: The higher the measure Ψ ( l,h ) p , the larger the impact of position-based attention Φ ( l,h ) p .

Original: Being averaged over all test samples, Ψ ( l,h ) c is similarly defined.
Revised: Being averaged over all test samples, Ψ ( l,h ) c is similarly defined.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 7,we observe a clear tendency that the impact of position-based attention is significantly higher inearly processing, transforming features semi-dynamically , while the later layers require less positioninformation, regarding Φ p as a minor position bias .
Revised: 7, we observe a clear tendency that the impact of position-based attention is significantly higher in early processing, transforming features semi-dynamically , while the later layers require less position information, regarding Φ p as a minor position bias .

Original: This tendency becomes more visible with largermodels as seen in right of Fig 7; Small and Medium models exploit dynamic transformations much
Revised: This tendency becomes more visible with larger models as seen in right of Fig 7; Small and Medium models exploit dynamic transformations much

Original: I m p a c t m e t
Revised: I m p a c t m e t

Original: r
Revised: r

Original: i c Ψ ∗ ( # , % )
Revised: i c Ψ ∗ ( # , % )

Original: ( T i n y ) 0.
Revised: ( T i n y ) 0.

Original: Tiny Small
Revised: Tiny Small

Original: Medium 0.
Revised: Medium 0.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation] 
Revised: [Equation] [Figure]

Original: more compared to Tiny model, especially in the later layers.
Revised: more compared to Tiny model, especially in the later layers.

Original: Moreover, we note that the impactmeasures of four heads (bar graphs) within each layer are unevenly distributed, showing high variance,which imply that the network evenly utilizes both position and content information simultaneouslywithin each MPA layer as seen in Layer 3 in Fig. 6, performing both static/local and dynamic/globaltransformation in a single-shot .
Revised: Moreover, we note that the impact measures of four heads (bar graphs) within each layer are unevenly distributed, showing high variance, which imply that the network evenly utilizes both position and content information simultaneously within each MPA layer as seen in Layer 3 in Fig. 6, performing both static/local and dynamic/global transformation in a single-shot .

Original: These results reveal that feature transformations for effective visualrecognition should not be restricted to be in either position-only [25, 40] or content-only [18, 54]design
Revised: These results reveal that feature transformations for effective visual recognition should not be restricted to be in either position-only [25, 41] or content-only [18, 58] design

Original: but they should be in the form of a hybrid [10, 12].
Revised: but they should be in the form of a hybrid [10, 12].

Original: Inspecting the locality (local vs. global).
Revised: Inspecting the locality (local vs. global).

Original: We further investigate the inner workings of PerViT byquantifying how locally Φ ∗ attends.
Revised: We further investigate the inner workings of PerViT by quantifying how locally Φ ∗ attends.

Original: Following the work of [12], we define the measure of nonlocalityfor Φ ( l,h ) p by summing all pair-wise query-key distances weighted by their attention scores:
Revised: Following the work of [12], we define the measure of nonlocality for Φ ( l,h ) p by summing all pair-wise query-key distances weighted by their attention scores:

Original: [Equation]
Revised: [Equation]

Original: This metrics Ω c and Ω a are similarly defined, being averaged over all test samples.
Revised: This metrics Ω c and Ω a are similarly defined, being averaged over all test samples.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 8,we observe a similar trend of locality between Φ p and Φ a , which reveals the position informationplay more dominant role over the content information in forming spatial attentions ( Φ a ) for featuretransformation.
Revised: 8, we observe a similar trend of locality between Φ p and Φ a , which reveals the position information play more dominant role over the content information in forming spatial attentions ( Φ a ) for feature transformation.

Original: Interestingly, we also observe that content- and position-based attentions behaveconversely; Φ c attends globally in early layers, i.e. , large scores are distributed over the whole spatialregion, while being relatively local in deeper layers.
Revised: Interestingly, we also observe that content- and position-based attentions behave conversely; Φ c attends globally in early layers, i.e. , large scores are distributed over the whole spatial region, while being relatively local in deeper layers.

Original: We hypothesize that the proposed Φ p in earlylayers is trained to effectively suppress unnecessary scores of Φ c at distant positions, thus exploitingonly a few relevant ones within its local region of interest.
Revised: We hypothesize that the proposed Φ p in early layers is trained to effectively suppress unnecessary scores of Φ c at distant positions, thus exploiting only a few relevant ones within its local region of interest.

Original: Meanwhile, Φ p at later layers gives Φ chigher freedom in forming the spatial attention Φ a as described in the plots of Fig. 7, which allowsthe attention scores of Φ c to be clustered in semantically meaningful parts, e.g. , eyes of the animalsas seen in attentions of the first head at Layer 8 (Fig. 6), which makes Φ c relatively local.
Revised: Meanwhile, Φ p at later layers gives Φ c higher freedom in forming the spatial attention Φ a as described in the plots of Fig. 7, which allows the attention scores of Φ c to be clustered in semantically meaningful parts, e.g. , eyes of the animals as seen in attentions of the first head at Layer 8 (Fig. 6), which makes Φ c relatively local.

Original: 4.2 Quantitative evaluation on ImageNet-1K
Revised: 4.2 Quantitative evaluation on ImageNet-1K

Original: Ablation study on main components.
Revised: Ablation study on main components.

Original: In Tab. 1, we analyze the impact of each component in PerViT,which is denoted as (a), where C-stem refers to convolutional patch embedding stem 5 .
Revised: In Tab. 1, we analyze the impact of each component in PerViT, which is denoted as (a), where C-stem refers to convolutional patch embedding stem 8 .

Original: We observethat the proposed attention Φ p brings consistent gains to models (b, f, g, i) with relative improvementsof 1.4 ∼ 4.2%p for (a, d, e, h) respectively.
Revised: We observe that the proposed attention Φ p brings consistent gains to models (b, f, g, i) with relative improvements of 1.4 ∼ 4.2%p for (a, d, e, h) respectively.

Original: Among three main components ( Φ p , C-stem, CPE), Φ p hasthe most significant impact on PerViT (a), losing 1.5%p Top-1 accuracy without Φ p , i.e. , model (b).
Revised: Among three main components ( Φ p , C-stem, CPE), Φ p has the most significant impact on PerViT (a), losing 1.5%p Top-1 accuracy without Φ p , i.e. , model (b).

Original: Surprisingly, PerViT without content-based attention Φ c , model (c), achieves decent performance,almost equalling to the accuracy of PerViT without position-based attention Φ p , model (b) (-0.5%p).

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: Surprisingly, PerViT without content-based attention Φ c , model (c), achieves decent performance, almost equalling to the accuracy of PerViT without position-based attention Φ p , model (b) (-0.5%p).

Original: The results verify that the proposed peripheral attention, which achieves comparable level of efficacyto the content-based attention, learns to generate reliable spatial attentions for visual recognition.
Revised: The results verify that the proposed peripheral attention, which achieves comparable level of efficacy to the content-based attention, learns to generate reliable spatial attentions for visual recognition.

Original: Wealso implement the proposed position-based attention Φ p on DeiT [54] baseline and compare theresults with recent state-of-the-art RPE methods.
Revised: We also implement the proposed position-based attention Φ p on DeiT [58] baseline and compare the results with recent state-of-the-art RPE methods.

Original: As seen in Tab. 2, the large improvements over theprevious RPE methods [8, 65] further verify the efficacy of the proposed peripheral position encoding(PPE).
Revised: As seen in Tab. 2, the large improvements over the previous RPE methods [8, 70] further verify the efficacy of the proposed peripheral position encoding (PPE).

Original: To confirm that the impact of Φ p is consistent with large models, we conduct similar ablationsusing PerViT-S/M in Tab. 3; without Φ p , the accuracy consistently drops for all the three models.
Revised: To confirm that the impact of Φ p is consistent with large models, we conduct similar ablations using PerViT-S/M in Tab. 3; without Φ p , the accuracy consistently drops for all the three models.

Original: Comparing (b) with (c), we observe that C-Stem and CPE are less effective for large models, bringing1.3%p and 0.1%p gains for Small and Medium respectively whereas they improve the Tiny model by5.1%p.
Revised: Comparing (b) with (c), we observe that C-Stem and CPE are less effective for large models, bringing 1.3%p and 0.1%p gains for Small and Medium respectively whereas they improve the Tiny model by 5.1%p.

Original: In contrast, the impact of Φ p is consistent across different model sizes, bringing 1%p gainsfor all the three models.
Revised: In contrast, the impact of Φ p is consistent across different model sizes, bringing 1%p gains for all the three models.

Original: The better efficacy of Φ p for larger models, we hypothesize, is due to itsflexibility in modeling local/global spatial attentions while C-Stem/CPE are designed only to be local.
Revised: The better efficacy of Φ p for larger models, we hypothesize, is due to its flexibility in modeling local/global spatial attentions while C-Stem/CPE are designed only to be local.

Original: Sample-efficiency of PerViT. To investigate the training sample efficiency of our model, we train
Revised: Sample-efficiency of PerViT. To investigate the training sample efficiency of our model, we train

Original: PerViT-S with ImageNet subsampled by fractions of 50% and 25% 6 and evaluate it on full-sized testset of ImageNet-1K. Table 4 compare our results with DeiT [54]; our model consistently surpassesthe baseline for all subsampled datasets, showing its robustness under limited training data.
Revised: PerViT-S with ImageNet subsampled by fractions of 50% and 25% 9 and evaluate it on full-sized test set of ImageNet-1K. Table 4 compare our results with DeiT [58]; our model consistently surpasses the baseline for all subsampled datasets, showing its robustness under limited training data.

Original: Ablation study on Φ p .
Revised: Ablation study on Φ p .

Original: The top section of Tab.
Revised: The top section of Tab.

Original: 5 reports results of PerViT-T with different parameterinitialization methods for Φ p where peripheral denotes the proposed peripheral initialization, convrefers to convolutional initialization such that s l = − 5 . 

Revised: 5 reports results of PerViT-T with different parameter initialization methods for Φ p where peripheral denotes the proposed peripheral initialization, conv refers to convolutional initialization such that s l = − 5 .

Original: and v l = 3 . 0 for all l ∈ [ N l ] , and randrefers to random initialization for all parameters in Φ p : w r , W ( l ) p1 , W ( l,h ) p2 , γγγ ( l ) p1 , βββ ( l ) p1 , γ ( l,h ) p2 , andβ ( l,h ) p2 .
Revised: and v l = 3 . 0 for all l ∈ [ N l ] , and rand refers to random initialization for all parameters in Φ p : w r , W ( l ) p1 , W ( l,h ) p2 , γγγ ( l ) p1 , βββ ( l ) p1 , γ ( l,h ) p2 , and β ( l,h ) p2 .

Original: The results show the efficacy of our peripheral initialization which is also supported bythe results in Fig.
Revised: The results show the efficacy of our peripheral initialization which is also supported by the results in Fig.

Original: 4 and 8: Φ p provides early local and late global attentions, suggesting thatperipheral initialization effectively reduces burden in learning such form of attentions.
Revised: 4 and 8: Φ p provides early local and late global attentions, suggesting that peripheral initialization effectively reduces burden in learning such form of attentions.

Original: The bottomsection of Tab.
Revised: The bottom section of Tab.

Original: 5 studies network designs for Φ p where N represents the proposed peripheralprojection, i.e. , projecting input distance representation by referring neighbors N , ML refers tomulti-layer design of Φ p , and Euc & Lrn indicate the type of embedding R : Euc is relative Euclideandistances ( R q , k , : = concat r ∈
Revised: 5 studies network designs for Φ p where N represents the proposed peripheral projection, i.e. , projecting input distance representation by referring neighbors N , ML refers to multi-layer design of Φ p , and Euc & Lrn indicate the type of embedding R : Euc is relative Euclidean distances ( R q , k , : = concat r ∈

Original: [ D r ] [ w r · R ] ) where Lrn is relative distances between learnable vectors( R ∈ R
Revised: [ D r ] [ w r · R ] ) where Lrn is relative distances between learnable vectors ( R ∈ R

Original: HW × HW × D r ).
Revised: HW × HW × D r ).

Original: Without N and ML, we observe consistent accuracy drops for Euc and Lrnby 0.8%p and 0.2%p respectively.
Revised: Without N and ML, we observe consistent accuracy drops for Euc and Lrn by 0.8%p and 0.2%p respectively.

Original: A sole multi-layer projection hardly improves accuracy but themodel performs the best when N is jointly used, meaning that both need to complement each otherto provide diverse attention shapes as in Fig 4.
Revised: A sole multi-layer projection hardly improves accuracy but the model performs the best when N is jointly used, meaning that both need to complement each other to provide diverse attention shapes as in Fig 4.

Original: Furthermore, Euc models consistently surpasses Lrn models, implying the Euclidean distance is more straightforward encoding type than learnablevectors in capturing spatial configurations of images.
Revised: Furthermore, Euc models consistently surpasses Lrn models, implying the Euclidean distance is more straightforward encoding type than learnable vectors in capturing spatial configurations of images.

Original: To further highlight the benefits of learningΦ p , we conduct experiments using PerViT-T with parameters of Φ p fixed during training under threedifferent initialization methods of peripheral, conv, and rand. Table 7 summarizes the results. FixingΦ p parameters damages performance for all three intializations, verifying the efficacy of learning de signs bottom based attention
Revised: network de signs bottom based attention

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: diverse position-based attentions across different layers and heads.

Original: [-We observe that conv and randinits perform poorly compared to PerViT-T without Φ p , e.g. , model (b) in Tab. 
-]
Revised: [-We observe that conv and randinits perform poorly compared to PerViT-T without Φ p , e.g. , model (b) in Tab. 
-]

Original: (77.3%); we suspectthat fixed Φ p with conv and rand inits only provide local and noisy attentions respectively while

Original: PerViT without Φ p has no such strong restrictions.

Original: Comparison with state of the arts.
Revised: Comparison with state of the arts.

Original: Table 6 summarizes the results of our method and recent state ofthe arts.
Revised: Table 6 summarizes the results of our method and recent state of the arts.

Original: For fair comparison, the baselines used in our comparison are trained using 224 × 224 inputresolution without distillations, and are grouped into either pyramidal or columnar ViT based on thenetwork designs, i.e. , multi- or single-resolution feature processing, where the results are partitionedaccording to model sizes within each group.
Revised: For fair comparison, the baselines used in our comparison are trained using 224 × 224 input resolution without distillations, and are grouped into either pyramidal or columnar ViT based on the network designs, i.e. , multi- or single-resolution feature processing, where the results are partitioned according to model sizes within each group.

Original: As shown in the bottom section of Tab 6, the proposedmethod achieves consistent improvements over the recent columnar ViT methods [12, 19, 54, 70]while showing competitive results to the pyramidal counterparts.  
Revised: As shown in the bottom section of Tab 6, the proposed method achieves consistent improvements over the recent columnar ViT methods [12, 19, 58, 75] while showing competitive results to the pyramidal counterparts.  Scope and Limitations

Original: To further verify the robustnessof the proposed method, we compare the PerViT-M with baseline models [54, 18] on differenttransfer learning task with ImageNet pre-training in Tab. 8.
Revised: Despite the interpretability and effectiveness of PerViT, it still leaves much room for improvements. First, PerViT-Attention (Eq. 5) is based on the original self-attention formulation [63], thus directly inheriting its limitations [18, 58], e.g. , quadratic complexity w.r.t.

Original: We finetune trained PerViT-M on CIFAR10, CIFAR-100, and iNaturalist-19, following the same training recipes of DeiT [54].
Revised: input resolution.

Original: Even withsignificantly lower complexity than [54, 18], our method surpasses baselines by approx.
Revised: The computational efficiency could be further improved by approximating low-rank matrices as in [7, 37, 73].

Original: 1%p on CIFAR-100 and iNaturalist19 while performing on par with [54] on CIFAR-10.
Revised: Second, given the ability to process high-resolution input with feasible complexity, the efficacy of PerViT could be improved by adopting multi-resolution pyramidal design following recent trend of ViT designs [17, 35, 37, 40, 67, 69, 73, 74].

Original: We omit the existing text in this draft due to 9-page limit for  the revised manuscript. We will includethe text in our final manuscript (10-page limit) if accepted.
Revised: Third, the focus of this paper is model development & exploration for image classification task but we believe the proposed idea is broadly generalizable to other vision applications such as object detection and segmentation. We leave this to future work.

Original:  Conclusion
Revised:  Conclusion

Original: This paper explores blending human peripheral vision with machine vision for effective visualrecognition, and introduces Peripheral Vision Transformer which learns to provide diverse positionbased attentions to model peripheral vision using peripheral projections and initialization.
Revised: This paper explores blending human peripheral vision with machine vision for effective visual recognition, and introduces Peripheral Vision Transformer which learns to provide diverse positionbased attentions to model peripheral vision using peripheral projections and initialization.

Original: We havesystematically investigated the inner workings of the proposed network and observed that the networkenjoys the benefits of both convolution and self-attention by learning to decide level of the locality& dynamicity in feature transformation, by the network itself given training data.
Revised: We have systematically investigated the inner workings of the proposed network and observed that the network enjoys the benefits of both convolution and self-attention by learning to decide level of the locality and dynamicity for the feature transformations, by the network itself given training data.

Original: The consistentimprovements over the baseline models on ImageNet-1K  across different model sizes and in-depthablation study confirm the efficacy of the proposed approach.
Revised: The consistent improvements over the baseline models on ImageNet-1K classification across different model sizes and in-depth ablation study confirm the efficacy of the proposed approach.

Revised:  Acknowledgments and Disclosure of Funding

Revised: This work was supported by the IITP grants (IITP-2021-0-01696: High-Potential Individuals Global

Revised: Training Program, IITP-2021-0-00537:

Revised: Visual Commonsense, IITP-2019-0-01906:

Revised: AI Graduate

Revised: School Program - POSTECH) funded by Ministry of Science and ICT, Korea.

Revised: This work was done while Juhong Min was working as an intern at Microsoft Research Asia.