Original: Peripheral Vision Transformer
Revised: Peripheral Vision Transformer

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Human vision possesses a special type of visual processing systems called peripheral vision .
Revised: Human vision possesses a special type of visual processing systems called peripheral vision .

Original: Partitioning the entire visual field into multiple contour regions basedon the distance to the center of our gaze, the peripheral vision provides us theability to perceive various visual features at different regions.
Revised: Partitioning the entire visual field into multiple contour regions basedon the distance to the center of our gaze, the peripheral vision provides us theability to perceive various visual features at different regions.

Original: In this work, wetake a biologically inspired approach and explore to model peripheral vision indeep neural networks for visual recognition.
Revised: In this work, wetake a biologically inspired approach and explore to model peripheral vision indeep neural networks for visual recognition.

Original: We propose to incorporate peripheralposition encoding to the multi-head self-attention layers to let the network learnto partition the visual field into diverse peripheral regions given training data.
Revised: We propose to incorporate peripheralposition encoding to the multi-head self-attention layers to let the network learnto partition the visual field into diverse peripheral regions given training data.

Original: Weevaluate the proposed network, dubbed PerViT, on the large-scale ImageNet datasetand systematically investigate the inner workings of the model for machine perception, showing that the network learns to perceive visual data similarly to the waythat human vision does.
Revised: Weevaluate the proposed network, dubbed PerViT, on ImageNet-1K and systematicallyinvestigate the inner workings of the model for machine perception, showing thatthe network learns to perceive visual data similarly to the way that human visiondoes.

Original: Thestate-of-the-art performance  in image classificationtask across various model sizes demonstrates the efficacy of the proposed method
Revised: The performance improvements in image classification over the baselinesacross different model sizes demonstrate the efficacy of the proposed method

Original:  Introduction
Revised:  Introduction

Original: [Figure]
Revised: [Figure]

Original: For the last ten years, convolution has been a dominant fea- ture transformation in neural networks for visual recognitiondue to its superiority in modelling spatial configurations ofimages [17, 25, 26].
Revised: For the last ten years, convolution has been a dominant feature transformation in neural networks for visual recognitiondue to its superiority in modelling spatial configurations ofimages [21, 32, 34].

Original: Despite the efficacy in learning visualpatterns, the local and stationary nature of convolutional kernels limited the maximum extent of representation abilityin flexible processing, e.g. , dynamic transformations withglobal receptive fields.
Revised: Despite the efficacy in learning visualpatterns, the local and stationary nature of convolutional kernels limited the maximum extent of representation abilityin flexible processing, e.g. , dynamic transformations withglobal receptive fields.

Original: Originally devised for natural language processing (NLP), self-attention [46] shed a light onthis direction; equipped with adaptive input processing andthe ability to capture long-range interactions, it has emergedas an alternative feature transform for computer vision, beingwidely adopted as a core building block [15].
Revised: Originally devised for natural language processing (NLP), self-attention [58] shed a light onthis direction; equipped with adaptive input processing andthe ability to capture long-range interactions, it has emergedas an alternative feature transform for computer vision, beingwidely adopted as a core building block [18].

Original: The stand-aloneself-attention models, e.g. , ViT [15], however, demand significantly more training data [41] for competitive performancewith its convolutional counterparts [5, 19, 52, 21] since theymiss certain desirable property which convolution possesses,e.g. , locality.
Revised: The stand-aloneself-attention models, e.g. , ViT [18], however, demand significantly more training data [53] for competitive performancewith its convolutional counterparts [6, 25, 27, 67] since theymiss certain desirable property which convolution possesses,e.g. , locality.

Original: These inherent pros and cons of convolutionand self-attention encourage recent researches toward combinations of both so as to enjoy the best of the both worlds butwhich one suits the best for effective visual processing is yetcontroversial in literature [7, 8, 9, 11, 27, 29, 31, 32, 36, 37, 43, 45, 48, 49, 51, 53, 57].
Revised: These inherent pros and cons of convolutionand self-attention encourage recent researches toward combinations of both so as to enjoy the best of the both worlds butwhich one suits the best for effective visual processing is yetcontroversial in literature [8, 9, 10, 12, 35, 37, 39, 40, 46, 47, 55, 57, 62, 64, 66, 68, 72].

Original: Unlike the dominant visual feature transformations in machine vision, human vision possesses aspecial type of visual processing systems called peripheral vision [28]; it partitions the entire visual field into multiple contour regions based on the distances to the center of a gaze where each regionidentifies different visual aspects.
Revised: Unlike the dominant visual feature transformations in machine vision, human vision possesses aspecial type of visual processing systems called peripheral vision [36]; it partitions the entire visual field into multiple contour regions based on the distances to the center of a gaze where each regionidentifies different visual aspects.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 1, we have high-resolution processing near thecenter of our gaze, i.e. , central and para-central regions, to identify highly-detailed visual elementssuch as geometric shapes, and low-level details.
Revised: 1, we have high-resolution processing near thecenter of our gaze, i.e. , central and para-central regions, to identify highly-detailed visual elementssuch as geometric shapes, and low-level details.

Original: For the regions more distant from the gaze, i.e. ,mid and far peripheral regions, the resolution decreases to recognize abstract visual features such asmotion, and high-level contexts.
Revised: For the regions more distant from the gaze, i.e. ,mid and far peripheral regions, the resolution decreases to recognize abstract visual features such asmotion, and high-level contexts.

Original: This systematic strategy enables us to effectively perceive importantdetails within a small fraction (1%) of the visual field while minimizing unnecessary processing ofbackground clutter in the rest (99%), thus facilitating efficient visual processing for human brain.
Revised: This systematic strategy enables us to effectively perceive importantdetails within a small fraction (1%) of the visual field while minimizing unnecessary processing ofbackground clutter in the rest (99%), thus facilitating efficient visual processing for human brain.

Original: According to recent study on inner workings of vision transformers [11, 15, 36, 42, 51], their behavioris in fact closely related tohow the peripheral vision functions.
Revised: Based on recent study on inner workings of vision transformers [12, 18, 46, 54, 66], we speculate thattheir behavior is loosely related to the aforementioned visual processing strategies.

Original: The attention maps of early layersare learned to locally capture fine-grained geometric details at central regions while those of laterlayers perform global attentions to identify coarse-grained semantics and contexts from the wholevisual field, covering peripheral regions .
Revised: The attention mapsof early layers are learned to locally capture fine-grained geometric details at central regions whilethose of later layers perform global attentions to identify coarse-grained semantics and contexts fromthe whole visual field, covering peripheral regions .

Original: These findings reveal that imitating biological designsmay potentially help in modelling an effective machine vision, and also support recent approachestowards a hybrid [4, 9, 18, 11, 30] of convolution and self-attention beyond stand-alone visualprocessing to take advantages of the two different perception strategies: fine-grained/local andcoarse-grained/global, similarly to human vision as illustrated in Fig.
Revised: The connections reveal that imitating biologicaldesigns may potentially help in modelling an effective machine vision, and also support recentapproaches towards a hybrid [5, 10, 12, 24, 38] of convolution and self-attention beyond stand-alonevisual processing to take advantages of the two different perception strategies: fine-grained/local andcoarse-grained/global, similarly to the visual processing illustrated in Fig.

Original: In this work, we take a biologically inspired approach and proposes to inject the peripheral inductivebiases  to deep neural networks for image recognition.
Revised: In this work, we take a biologically inspired approach and propose to inject the peripheral inductivebiases 1 to deep neural networks for image recognition.

Original: We propose to incorporate peripheral attentionmechanism to the multi-head self-attention [46] to let the network learn to partition the visual fieldinto diverse peripheral regions given training data where each region captures different visual features.
Revised: We propose to incorporate peripheral attentionmechanism to the multi-head self-attention [58] to let the network learn to partition the visual fieldinto diverse peripheral regions given training data where each region captures different visual features.

Original: We experimentally show that the proposed network models effective visual periphery for reliablevisual recognition.
Revised: We experimentally show that the proposed network models effective visual periphery for reliablevisual recognition.

Original: Our main contributions can be summarized as follows:
Revised: Our main contributions can be summarized as follows:

Original: • This work explores to narrow the gap between human and machine vision by injectingperipheral inductive biases to self-attention layers, and presents a new form of featuretransformations named M ulti-head P eripheral A ttention (MPA).• Based on the MPA, we introduce Per ipheral Vi sion T ransformer (PerViT), and systematically study the inner workings of PerViT by qualitatively and quantitatively analyzing itslearned attentions, which reveal that the network learns to perceive visual elements similarly to the way that human vision does without any special supervisions.
Revised: • This work explores to narrow the gap between human and machine vision by injectingperipheral inductive biases to self-attention layers, and presents a new form of featuretransformations named M ulti-head P eripheral A ttention (MPA).• Based on the MPA, we introduce Per ipheral Vi sion T ransformer (PerViT), and systematically study the inner workings of PerViT by qualitatively and quantitatively analyzing itslearned attentions, which reveal that the network learns to perceive visual elements similarlyto the way that human vision does without any special supervisions.

Original: • Thestate-of-the-art performance on image classification task  across different model sizesverify the effectiveness of the proposed approach.
Revised: • The performance improvements in image classification task over the columnar Transformerbaselines, e.g. , DeiT, across different model sizes demonstrate the efficacy of the proposedapproach.

Original:  Related Work
Revised:  Related Work

Original: Feature transformations in computer vision.
Revised: Feature transformations in computer vision.

Original: With notable success in NLP, Transformers [13, 46]introduced a paradigm shift in computer vision [2, 3, 6, 15, 24, 37, 39, 40, 42, 43, 45].
Revised: With notable success in NLP, Transformers [14, 58]introduced a paradigm shift in computer vision [3, 4, 7, 18, 30, 47, 51, 52, 54, 55, 57].

Original: Despitetheir generalization capability, pure Vision Transformers [15] require extensive amount of trainingdata to capture spatial layout of images due to lack of certain desirable property, e.g. , locality.
Revised: Despitetheir generalization capability, pure Vision Transformers [18] require extensive amount of trainingdata to capture spatial layout of images due to lack of certain desirable property, e.g. , locality.

Original: This encouraged many recent ViT work to incorporate local inductive biases via distillation ofconvolutional biases [42], local self-attention [27, 31, 53], a hybridization [9, 11, 29], and augmentingconvolutions [7, 48, 49], all of which convey a unified message: “Despite high generalizibility ofself-attentions, a sufficient amount of convolutional processing must be incorporated to capture thespatial configurations of images for reliable visual processing.”
Revised: Thisencouraged many recent ViT work to incorporate local inductive biases via distillation of convolutional biases [54], local self-attention [35, 39, 68], a hybridization [10, 12, 37], and augmentingconvolutions [8, 62, 64], all of which convey a unified message: “Despite high generalizibility ofself-attentions, a sufficient amount of convolutional processing must be incorporated to capture thespatial configurations of images for reliable visual processing.”

Original: Position encoding for Transformer.
Revised: Position encoding for Transformer.

Original: Witnessing the efficacy of position encoding in capturinginput structures in NLP [10, 38, 22], recent vision models [15, 37, 50] have begun employingposition encodings for images to model spatial structures of images.
Revised: Witnessing the efficacy of position encoding in capturing inputstructures in NLP [11, 28, 50], recent vision models [18, 47, 65] have begun employing positionencodings for images to model spatial structures of images.

Original: In particular, relative positionencoding (RPE) plays a vital role for the purpose: The work of Cordonnier et al.
Revised: In particular, relative position encoding(RPE) plays a vital role for the purpose: The work of Cordonnier et al.

Original: [8] proves thatself-attention has close relationship with convolution when equipped with certain form of RPE.
Revised: [9] proves that self-attentionhas close relationship with convolution when equipped with certain form of RPE.

Original: Wu etal.
Revised: Wu et al.

Original: [50] explores the existing RPE methods used in vision transformers [10, 22, 38, 47] and draws aconclusion that RPEs imposes convolutional processing on vision transformers.
Revised: [65]explores the existing RPE methods used in vision transformers [11, 28, 50, 60] and draws a conclusionthat RPEs imposes convolutional processing on vision transformers.

Original: Dai et al.
Revised: Dai et al.

Original: [9] observethat depthwise convolution and self-attention can naturally be unified via RPEs.
Revised: [10] observe that depthwise convolution and self-attention can naturally be unified via RPEs.

Original: While offeringpromising directions, the previous RPE work, however, is limited in the sense that the focus of RPEutilization is restricted to only local attention, e.g. , convolution.
Revised: While offering promisingdirections, the previous RPE work, however, is limited in the sense that the focus of RPE utilizationis restricted to only local attention, e.g. , convolution.

Original: In this work, we exploit RPEs todevise an original visual feature transformation which is more general than convolution while beingmore structured than self-attention via imitation of human visual system: peripheral vision .
Revised: We exploit RPEs to devise an original visualfeature transformation which is more general than convolution while being more structured thanself-attention via imitation of human visual system: peripheral vision .

Original: Our Approach
Revised: Peripheral vision for machine perception. Along with central vision, peripheral vision plays anessential role in a wide range of visual recognition tasks [33].

Original: In this section, we introduce the Peripheral Vision Transformer (PerViT) that learns to modelperipheral vision for effective image recognition. We first revisit the mathematical formulation ofself-attention and then describe how we improve it with peripheral inductive biases.
Revised: The fundamental mechanisms ofperipheral visual processing, however, have not been fully disclosed in vision literature [48] whichstimulated many researchers to reveal its inner workings and deep implications [2, 15, 16, 42, 49, 63].

Original: Background: Multi-Head Self-Attention. The multi-head self-attention (MHSA) [46] with N hheads performs an attention-based feature transformation by aggregating N h self-attention outputs: [Equation] where X ∈ R HW × D emb is a set of input tokens and W out ∈ R N h D h × D emb and b out ∈ R D emb are thetransformation parameters.
Revised: The work of Rosenholtz [48] discusses pervasive myths and current findings about peripheral vision,suggesting that peripheral vision is more crucial for human perception than previously thoughtto perform diverse important tasks.

Original: The N h outputs of self-attention are designed to extract a diverse set offeatures from the input representation. Formally, the self-attention at head h is defined as Self-Attention ( [Equation]
Revised: Inspired by its importance, a number of pioneering work[16, 22, 23, 20, 43, 61] investigate the linkage between peripheral vision and machine vision, e.g. ,

Original: Normalize [Equation] where Normalize [ · ] denotes a row-wise normalization and Φ ( h ) ( · ) ∈ R
Revised: CNNs, while some [31, 59] devise biologically-inspired models for the creation of stronger machinevision.

Original: HW × HW is a function thatprovides spatial attentions based on content information to aggregate the values V ( h ) = XW ( h ) val :Φ ( h ) ( X ) := exp [Equation]
Revised: Continuing previous study, this paper explores to blend human peripheral vision with attentionbased neural networks, e.g. , vision transformer [18].

Original:  τ [Equation] h [Equation]
Revised:  Our Approach

Original: qry

Original: [Equation]

Original: key [Equation] = exp( [Equation]
Revised: We omit the existing text (Background: Multi-Head Self-Attention) in this draft due to 9-page limitfor the revised manuscript.

Original: Q [Equation] h ) , K ( h ) ⊤ ) , (3) using linear projections of W ( h ) qry , W ( h ) key , W ( h ) val ∈ R D emb × D h for queries, keys, and values respectively.
Revised: We will include the text in our final manuscript (10-page limit) if accepted.

Original: 3.1 Peripheral Vision Transformer
Revised: 3.1 Peripheral Vision Transformer

Original: Based on the formulation of the MHSA, we define M ulti-head P eripheral A ttention (MPA) as
Revised: Based on the formulation of the MHSA, we define M ulti-head P eripheral A ttention (MPA) as

Original: [Equation]
Revised: [Equation]

Original: where R ∈ R HW × HW × D r is the relative position encoding with D r channel dimension.
Revised: where R ∈ R HW × HW × D r is the relative position encoding with D r channel dimension.

Original: The selfattention in MHSA is now replaced with Peripheral-Attention, consisting of content- and positionbased attention functions ( Φ ( h ) c , Φ ( h ) p ∈ R
Revised: The selfattention in MHSA is now replaced with Peripheral-Attention, consisting of content- and positionbased attention functions ( Φ ( h ) c , Φ ( h ) p ∈ R

Original: HW × HW ), which is defined as
Revised: HW × HW ), which is defined as

Original: Peripheral
Revised: Peripheral-

Original: [Equation]
Revised: [Equation]

Original: where ⊙ is Hadamard product which mixes the given pair of attentions to provide a mixed attentionΦ ( h ) a ( X , R ) :
Revised: where ⊙ is Hadamard product which mixes the given pair of attentions to provide a mixed attention Φ ( h ) a ( X , R ) :

Original: = Φ ( h ) c ( X ) ⊙ Φ ( h ) p ( R )
Revised: = Φ ( h ) c ( X ) ⊙ Φ ( h ) p ( R )

Original: ∈
Revised: ∈

Original: R HW × HW .
Revised: R HW × HW .

Original: For the content-based attention Φ c , we use exponentiated (scaled) dot-product between queries and keys as in Eqn. 3: Φ ( h ) c ( X ) :
Revised: For the content-based attention Φ c ,we use exponentiated (scaled) dot-product between queries and keys as in Eqn. ?? :

Original: = exp( τ Q ( h ) K ( h ) ⊤ ) .
Revised: Φ ( h ) c ( X ) :=exp( τ Q ( h ) K ( h ) ⊤ ) .

Original: For the position-based attention Φ p , we design a neural network that aims to imitate human visualsystem, e.g. , peripheral vision.
Revised: For the position-based attention Φ p , we design a neural network that aims toimitate human visual system, e.g. , peripheral vision.

Original: Modelling peripheral vision: a Roadmap.
Revised: Modelling peripheral vision: a Roadmap.

Original: Human visual field can be grouped into several regionsbased on the Euclidean distances from the center of gaze, each forming ring-shaped region as seen in
Revised: Human visual field can be grouped into several regionsbased on the Euclidean distances from the center of gaze, each forming ring-shaped region as seen in

Original: Fig.
Revised: Fig.

Original: 1, where each region captures different visual aspects; the closer to the gaze, the more complexfeatures we process, and further from the gaze, the simpler visual features we perceive.
Revised: 1, where each region captures different visual aspects; the closer to the gaze, the more complexfeatures we process, and further from the gaze, the simpler visual features we perceive.

Original: In the contextof 2-dimensional attention map Φ ∗ ( · ) q , : ∈ R HW , we refer the query position q ∈ R 2 , i.e. , theposition where feature of our interest lies at for transformation, as the center of gaze, local regionsaround the query as central/para-central regions, and the rest as mid/far peripheral regions.
Revised: In the contextof 2-dimensional attention map Φ ∗ ( · ) q , : ∈ R HW , we refer the query position q ∈ R 2 as the centerof gaze, i.e. , the position where feature of our interest lies at for transformation. We refer the localregions around q as central/para-central regions and the rest as mid/far peripheral regions.

Original: Perhaps the simplest approach to divide the visual field into multiple subregions is to perform asingle linear projection on the Eucliden distances, i.e. , Φ ( h ) p ( R )
Revised: Perhaps the simplest approach to divide the visual field into multiple subregions is to perform asingle linear projection on the Euclidean distances, i.e. , Φ ( h ) p ( R )

Original: =
Revised: =

Original: σ (cid:2) RW ( h ) p (cid:3) where W ( h ) p ∈ R D r ,similarly the previous work of Wu et al.
Revised: σ (cid:2) RW ( h ) p (cid:3) where W ( h ) p ∈ [Equation] ,similarly the previous work of Wu et al.

Original: For straightforward imitation of peripheral vision, weuse Euclidean distance for relative position input R and weigh the distances in D r different ways forthe network to learn the mapping in multiple scales: R q , k , : := concat r ∈
Revised: For straightforward imitation of peripheral vision, weuse Euclidean distance for relative position input R and weigh the distances in D r different ways forthe network to learn the mapping in multiple scales: R q , k , : := concat r ∈

Original: [ D r ] [ w r · R euc q , k ] ∈ R D r where{ w r } r ∈
Revised: [ D r ] [ w r · R euc q , k ] ∈ R D r where{ w r } r ∈

Original: [ D r ] is a set of learnable parameters shared across layers and heads, and
Revised: [ D r ] is a set of learnable parameters shared across layers and heads, and

Original: R euc q , k = ∥
Revised: R euc q , k = ∥

Original: {+q − k ∥ 
+}
Revised: {+q − k ∥ 
+}

Revised: Given σ [ · ] := exp( · ) , Peripheral-Attention ( h ) =

Revised: Normalize [exp( Q ( h ) K ( h ) ⊤ ) ⊙ exp( RW ( h ) p )]

Original: q − k ∥ 21 Given σ [ softmax  to the bias mode RPE presented in [50].
Revised: V ( h ) = softmax ( Q ( h ) K ( h ) ⊤ + RW rpe ) V ( h ) , which is equivalent to the bias mode RPE presented in [65].

Original: Multi-layer projection
Revised: Multi [Figure]

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer
Revised: Layer

Original: Peripheral initialization
Revised: Peripheral initialization

Original: [Figure]
Revised: [Figure]

Original: Single

Original: [Figure]

Original: MPA
Revised: MPA

Original: FFN
Revised: FFN

Original: Convolutionalpatch embedding
Revised: Convolutionalpatch embedding

Original: GAP
Revised: GAP

Original: Image embedding
Revised: Image embedding

Original: Input image ×𝑁 !
Revised: Input image ×𝑁 !

Original: Scaled dot product 𝑸 (#) 𝑲 (#) 𝑽 (#) 𝐗∈ ℝ %&×( !"
Revised: Scaled dot product 𝑸 (#) 𝑲 (#) 𝑽 (#) 𝐗∈ ℝ %&×( !"

Original: Peripheralprojection 𝐑∈ ℝ %&×%&×( $
Revised: Peripheralprojection 𝐑∈ ℝ %&×%&×( $

Original: Peripheralprojection
Revised: Peripheralprojection

Original: IN & ReLU
Revised: IN & ReLU

Original: IN & Sigmoid
Revised: IN & Sigmoid

Original: Exp
Revised: Exp

Original: Matrix mult.
Revised: Matrix mult.

Original: 𝛗 !
Revised: 𝛗 !

Original: 𝛗 "
Revised: 𝛗 "

Original: CPE
Revised: CPE

Original: Normalize
Revised: Normalize

Original: Linear Linear
Revised: Linear Linear

Original: Linear 𝒉 (𝐗,𝐑) Peripheral-Attention ×𝑁 # is the Euclidean distance between query and key positions q , k ∈ R 2 .
Revised: Linear 𝒉 (𝐗,𝐑) Peripheral-Attention ×𝑁 # is the Euclidean distance between query and key positions q , k ∈ R 2 .

Original: For σ , we choose sigmoid toprovide normalized (positive) weights to the content-based attention Φ c .
Revised: For σ , we choose sigmoid toprovide normalized (positive) weights to the content-based attention Φ c .

Original: A main drawback of thissingle-layer formulation is that Φ p is only able to provide Gaussian-like attention map as seen intop-left in Fig.
Revised: A main drawback of thissingle-layer formulation is that Φ p is only able to provide Gaussian-like attention map as seen intop-left in Fig.

Original: 2, thus being unable to represent diverse peripheral regions.
Revised: 2, thus being unable to represent diverse peripheral regions.

Original: For the encoding functionto represent diverse (torus-shaped) peripheral regions, the distances must be processed by an MLP:
Revised: For the encoding functionto represent diverse (torus-shaped) peripheral regions, the distances must be processed by an MLP:

Original: [Equation]
Revised: [Equation]

Original: (6) where W p1 ∈ R D r × D hid and W ( h ) p2 ∈ R D hid are the linear projection parameters 2 , and ReLU givesnon-linearlity to the function.
Revised: (3) where W p1 ∈ R D r × D hid and W ( h ) p2 ∈ R D hid are the linear projection parameters 3 , and ReLU givesnon-linearity to the function.

Original: The first projection W p1 is shared across the heads in order to exchangeinformation so each function is able to provide attention that are effective or complementary toother heads’ attention.
Revised: The first projection W p1 is shared across the heads in order to exchangeinformation so each function is able to provide attention that are effective or complementary toother heads’ attention.

Original: Note that given identical relative distances between a fixed query pointq ∈ R 2 and key points k
Revised: Note that given identical relative distances between a fixed query pointq ∈ R 2 and key points k

Original: i , k j ∈ R 2 , i.e. , R q , k i =
Revised: i , k j ∈ R 2 , i.e. , R q , k i =

Original: R q , k j , Eq. 6 provides the same attention scores:Φ p ( R ) q , k i =
Revised: R q , k j , Eq. 3 provides the same attention scores:Φ p ( R ) q , k i =

Original: Φ p ( R ) q , k j as seen in top-right of Fig.
Revised: Φ p ( R ) q , k j as seen in top-right of Fig.

Original: This property, however, is not always desiredin practical scenarios because the rotational symmetric property hardly holds for most real-worldobjects.
Revised: This property, however, is not always desiredin practical scenarios because the rotational symmetric property hardly holds for most real-worldobjects.

Original: To break the symmetric property in Eq.
Revised: To break the symmetric property in Eq.

Original: 6 while preserving peripheral design to sufficientextent, we introduce peripheral projections in which the transformation parameters are given smallspatial resolutions such that W p1 ∈ R K 2 × D r × D hid and W ( h ) p2 ∈ R K 2 × D hid so the they providesimilar but different attention scores, Φ p ( R ) q , k i ̸
Revised: 3 while preserving peripheral design to sufficientextent, we introduce peripheral projections in which the transformation parameters are given smallspatial resolutions, e.g. , K × K window, such that W p1 ∈ R K 2 × D r × D hid and W ( h ) p2 ∈ R K 2 × D hid so thatthey provide similar but different attention scores, Φ p ( R ) q , k i ̸

Original: = Φ p ( R ) q , k j , given R q , k i
Revised: = Φ p ( R ) q , k j , given R q , k i

Original: = R q , k j ,  by referring 143 neighboring relative distances around the keys:
Revised: = R q , k j , 146 by aggregating neighboring relative distances around the key location k as follows:

Original: [Equation]
Revised: [Equation]

Revised: where N ( k ) := (cid:2) k − ⌊ K 2 ⌋ , . . .

Original: where N ( · ) is afunction that provides a set of neighbors around input position.
Revised: , k + ⌊ K 2 ⌋ (cid:3) × (cid:2) k − ⌊ K 2 ⌋ , . . . , k + ⌊ K 2 ⌋ (cid:3) is a set of K 2 neighborsaround input position k .

Revised: We set K = 3 for all layers and heads as K > 3 hardly brought improvements.

Original: After each peripheralprojection, we add an instance normalization layer [44] for stable optimization:
Revised: After each peripheral projection, we add an instance normalization layer [56] for stableoptimization:

Original: [Equation]
Revised: [Equation]

Original: ( IN ( PP ( R ; W p1 ); γγγ p1 ,βββ p1 )) , [Equation] 8) where γγγ p1 ,βββ p1 ∈ R D hid and γ ( h ) p2 , β ( h ) p2 ∈ R are weights/biases of the instance norms.
Revised: where γγγ p1 ,βββ p1 ∈ R D hid and γ ( h ) p2 , β ( h ) p2 ∈ R are weights/biases of the instance norms and PP ( · ) denotesthe peripheral projection: PP ( R , W ) q , k , : := (cid:80) n ∈N ( k ) R q , n , : W n − k , : , : .

Original: The middlerow of Fig.
Revised: The middle row of Fig.

Original: 2 depicts learned attentions of Φ p with peripheral projections, which provides peripheralattention maps in greater diversity compared to single- and multi-layer counterparts without N .
Revised: 2depicts learned attentions of Φ p with peripheral projections, which provides peripheral attention mapsin greater diversity compared to single- and multi-layer counterparts without N .

Original: Peripheral initialization.
Revised: Peripheral initialization.

Original: Recent study [36, 42] observe that early layers of trained vision transformerlearn to attend locally whereas late layers perform global attentions.
Revised: Recent study [46, 54] observe that early layers of trained vision transformerlearn to attend locally whereas late layers perform global attentions.

Original: To facilitate training of ournetwork, we inject this property in the beginning of the training stage by initializing parametersof Φ p for the purpose, making attention scores near the queries larger than the distant ones inthe early layers while uniformly distributing them in the late layers as seen in bottom row of
Revised: To facilitate training of ournetwork, we inject this property in the beginning of the training stage by initializing parameters of Φ p for the purpose, making attention scores near the queries larger than the distant ones inthe early layers while uniformly distributing them in the late layers as seen in bottom row of

Original: Fig.
Revised: Fig.

Original: We refer this method as peripheral initialization for its resemblance to the functioningof peripheral vision [28] which also operates either locally or globally to perceive different visual features [56].
Revised: We refer this method as peripheral initialization for its resemblance to the functioningof peripheral vision [36] which also operates either locally or globally to perceive different visualfeatures [71].

Original: To formally put, given two arbitrarily chosen distances R euc q , k i , R euc q , k j ∈ R which satisfy
Revised: To formally put, given two arbitrarily chosen distances R euc q , k i , R euc q , k j ∈ R which satisfy

Original: R euc q , k i < R euc q , k j , we want Φ ( l,h ) p ( R ) q , k
Revised: R euc q , k i < R euc q , k j , we want Φ ( l,h ) p ( R ) q , k

Original: i ≫ Φ
Revised: i ≫ Φ

Original: [Equation]
Revised: [Equation]

Original: l,
Revised: l,

Original: [Equation]
Revised: [Equation]

Original:  p ( R ) q , k j 3 as l → 1 , i.e. , local attention in earlylayers, and Φ ( l,h ) p ( R ) q , k i ≈ Φ ( l,h ) p ( R ) q , k j as l → N l , i.e. , global attention in late layers, where N lis the total number of MPA layers.
Revised: ) p ( R ) q , k j 4 as l → 1 , i.e. , local attention in earlylayers, and Φ ( l,h ) p ( R ) q , [Equation] i ≈ Φ [Equation]

Revised: l,

Revised: [Equation]

Revised: ) p ( R ) q , k j as l → N l , i.e. , global attention in late layers, where N lis the total number of MPA layers.

Original: We first initialize the parameters of Φ ( l,h ) p and { w r } r ∈
Revised: We first initialize the parameters of Φ ( l,h ) p and { w r } r ∈

Original: [ D r ] toparticular values.
Revised: [ D r ] toparticular values.

Original: Specifically, for all layers l ∈ [ N l ] and heads h ∈
Revised: Specifically, for all layers l ∈ [ N l ] and heads h ∈

Original: [ N h ] ,
Revised: [ N h ] ,

Original: [Equation]
Revised: [Equation]

Original:  where c 1 , c 2 ∈ R + are positive reals, and J N,M ∈ R N × M refers to all-one matrix in size N ×
Revised: ) where c 1 , c 2 ∈ R + are positive reals, and J N,M ∈ R N × M refers to all-one matrix in size N ×

Original: M .
Revised: M .

Original: The above initialization provides local attention after the second peripheral projection, i.e. ,
Revised: The above initialization provides local attention after the second peripheral projection, i.e. ,

Original: PP ( R ′ ; W ( h ) p2 ) q , k i , : > PP ( R ′ ; W ( h ) p2 ) q , k j , : given R euc q , k i < R euc q , k j .
Revised: PP ( R ′ ; W ( h ) p2 ) q , k i , : > PP ( R ′ ; W ( h ) p2 ) q , k j , : given R euc q , k i < R euc q , k j .

Original: Next, based on our findings thatbiases β ( l,h ) p and the weights γ ( l,h ) p in the second instance norm control the sizes and strengths oflocal attention respectively, we simulate peripheral initialization by setting their initial values asβ ( l,h ) p :=
Revised: Next, based on our findings thatbiases β ( l,h ) p and the weights γ ( l,h ) p in the second instance norm control the sizes and strengths oflocal attention respectively, we simulate peripheral initialization by setting their initial values asβ ( l,h ) p :=

Original: s l and γ ( l,h ) p :=
Revised: s l and γ ( l,h ) p :=

Original: v l where respective { s l } l ∈ [ N l ] and { v l } l ∈ [ N l ] are sets of initial values forattention sizes and strengths.
Revised: v l where respective { s l } l ∈ [ N l ] and { v l } l ∈ [ N l ] are sets of initial values forattention sizes and strengths.

Original: We set their values collected from uniform intervals: s l ∈ [ − 5 . 0 , 4 .
Revised: We set their values collected from uniform intervals: s l ∈ [ − 5 . 0 , 4 .

Original: 0]and v l ∈ [3 . 0 , 0 .
Revised: 0]and v l ∈ [3 . 0 , 0 .

Original: 01] where s l − 1 < s l and v l − 1 > v l to give stronger local attentions to shallow layerscompared to deep ones as seen in bottom row of Fig.
Revised: 01] where s l − 1 < s l and v l − 1 > v l to give stronger local attentions to shallow layerscompared to deep ones as seen in bottom row of Fig.

Original: We set c 1 , c 2 = 0 .
Revised: We set c 1 , c 2 = 0 .

Original: in our experiments andleave all the details on the peripheral initialization to the supplementary.
Revised: in our experiments andleave all the details on the peripheral initialization to the supplementary.

Original: 3.2 Overall Architecture
Revised: 3.2 Overall Architecture

Original: Based on the proposed peripheral projections and initialization, we develop image classificationmodels, dubbed Peripheral Vision Transformer, which is illustrated in Fig.
Revised: Based on the proposed peripheral projections and initialization, we develop image classificationmodels, dubbed Peripheral Vision Transformer, which is illustrated in Fig.

Original: We follow similararchitecture to DeiT [42] with convolutional patch embedding stem; as the original patchify stem [15]exhibits substandard optimizability due to its coarse-grained early visual processing [51], many recent
Revised: We follow similararchitecture to DeiT [54] with convolutional patch embedding stem; as the original patchify stem [18]exhibits substandard optimizability due to its coarse-grained early visual processing [66], many recent

Original: ViT models adopt multi-resolution pyramidal designs [31, 48, 49, 53] to mitigate the issue.
Revised: ViT models adopt multi-resolution pyramidal designs [39, 62, 64, 68] to mitigate the issue.

Original: Whilethe pyramidal models have shown their efficacy in learning reliable image embeddings, we stick withthe original single-resolution cylindrical design for PerViT because features in multiple resolutionmake our study less interpretable, which further brings additional technical efforts in combining ourperipheral attention Φ p with the existing cost-effective self-attentions mechanisms such as factorizedattention [53], shifted-window [31], and cross-shaped window [14].
Revised: Whilethe pyramidal models have shown their efficacy in learning reliable image embeddings, we stick withthe original single-resolution columnar design for PerViT because features in multiple resolutionmake our study less interpretable, which further brings additional technical efforts in combining ourperipheral attention Φ p with the existing cost-effective self-attentions mechanisms such as factorizedattention [68], shifted-window [39], and cross-shaped window [17].

Original: To carry out fine-grained earlyprocessing while keeping single-resolution features across the layers, we adopt convolutional patchembedding layer [51] with multi-stage layouts for channel dimensions.
Revised: To carry out fine-grained earlyprocessing while keeping single-resolution features across the layers, we adopt convolutional patchembedding layer [66] with multi-stage layouts for channel dimensions.

Original: The convolutional embeddinglayer consists of four 3 × 3 and one 1 × 1 convolutions where the 3 × 3 convolutions are followed bybatch norm [23] and ReLU [34].
Revised: The convolutional embeddinglayer consists of four 3 × 3 and one 1 × 1 convolutions where the 3 × 3 convolutions are followed bybatch norm [29] and ReLU [44].

Original: For additional details, we refer to the supplementary.
Revised: For additional details, we refer to the supplementary.

Original: Peripheral Vision Transformer.
Revised: Peripheral Vision Transformer.

Original: Given an image, the convolutional patch embedding provides tokenembeddings X (1) ∈ R HW × D emb .
Revised: Given an image, the convolutional patch embedding provides tokenembeddings X (1) ∈ R HW × D emb .

Original: Similarly to [15, 42], the embeddings are fed to a series of N lblocks each of which consists of an MPA layer and a feed-forward network with residual pathways:
Revised: Similarly to [18, 54], the embeddings are fed to a series of N lblocks each of which consists of an MPA layer and a feed-forward network with residual pathways:

Original: [Equation]
Revised: [Equation]

Original: where LN is layer normalization [1], and FFN is an MLP consisting of two linear transformations witha GELU activation [20].
Revised: where LN is layer normalization [1], and FFN is an MLP consisting of two linear transformations witha GELU activation [26].

Original: Following the work of [27, 53], we adopt convolutional position encoding(CPE), i.e. , a 3 × 3 depth-wise convolution, before first layer norm for its efficacy with negligiblecomputational cost.
Revised: Following the work of [35, 68], we adopt convolutional position encoding(CPE), i.e. , a 3 × 3 depth-wise convolution, before first layer norm for its efficacy with negligiblecomputational cost.

Original: The output X ( N h ) is global-average pooled to form an image embedding.
Revised: The output X ( N h ) is global-average pooled to form an image embedding.

Original:  Experiments
Revised:  Experiments

Original: In this section, we first investigate the inner workings of PerViT trained on large-scale imageclassification dataset to examine how it benefits from the proposed peripheral projectionsandinitialization, and then compare the methodwith previous stateof the arts under comparable settings. Experimental setup. Our experiments focus on image classification on ImageNet-1K [12]. Followingtraining recipes of DeiT [42], we train our model on ImageNet-1K from scratch with batch size of
Revised: In this section, we first investigate the inner workings of PerViT trained on ImageNet-1K classificationdataset to examine how it benefits from the proposed peripheral projections and initialization, andthen compare the method state the arts settings

Original: [Figure]
Revised: [Figure]

Original: H e a d s l o c a l g l o b a l
Revised: H e a d s l o c a l g l o b a l

Original: [Figure]
Revised: [Figure]

Original: Central Para-central
Revised: [Figure]

Original: Mid peripheral
Revised: Mid peripheral

Original: Far peripheral
Revised: Far peripheral

Original: Tiny
Revised: Tiny

Original: Small
Revised: Small

Original: Medium 0 26 812 141 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 

Revised: Medium 0 26 812 141 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8 9 101112 04 610 12 1 2 3 4 5 6 7 8

Original: [Figure]
Revised: [Figure]

Original: 1024, learning rate of 0.001 using AdamW [33] optimizer, cosine learning rate decay scheduler, andthe same data augmentations [13] for 300 epochs, including 5 warm-up epochs.
Revised: Experimental setup. Our experiments focus on image classification on ImageNet-1K [13]. Followingtraining recipes of DeiT [54], we train our model on ImageNet-1K from scratch with batch size of1024, learning rate of 0.001 using AdamW [41] optimizer, cosine learning rate decay scheduler, andthe same data augmentations [14] for 300 epochs, including 5 warm-up epochs.

Original: We evaluate ourmodel with three different sizes, e.g. , Tiny (T), Small (S), and Medium (M).
Revised: We evaluate ourmodel with three different sizes, e.g. , Tiny (T), Small (S), and Medium (M).

Original: We use stochastic depthsof 0.0, 0.1, and 0.2 for T, S, and M respectively.
Revised: We use stochastic depthsof 0.0, 0.1, and 0.2 for T, S, and M respectively.

Original: Refer to the supplmentary for additional details.
Revised: Refer to the supplmentary for additional details.

Original: 4.1 The inner workings of PerViT
Revised: 4.1 The inner workings of PerViT

Original: [Figure]

Original: Learning peripheral vision.
Revised: Learning peripheral vision.

Original: We begin by investigating how PerViT models peripheral vision

Original: [Figure]

Original: by qualitatively analyzing its learned attention of Φ p . Figure 4 depicts the learned attention map [Figure] of Φ ( l,h ) p q , : ∈ R HW forall layers and heads where the query position is given at the center, i.e. ,q = [7 , 7] ⊤ 4 .
Revised: We omit the existing text in this draft due to 9-page limit for the revisedmanuscript.

Original: We observe that the attentions are learned to be in diverse shapes of peripheral regions.

Original: Interestingly, without any special supervisions, the four attended regions ( N h = 4 ) in most layers arelearned to complement each other to cover the entire visual field, capturing different visual aspectsat each region (head), similarly to human peripheral vision illustrated in Fig.

Original: For example, firsttwo heads in Layer 3 attend the central regions while the others cover the rest peripheral regions.

Original: The second and third heads in Layer 8 cover top and bottom hemicircles respectively, forming acircular-shaped semi-global receptive field.

Original: Moreover, a large number of early attentions is in formof central/para-central regions while those of late layers are learned to cover mid to far peripheralregions.

Original: To quantitatively inspect how PerViT models the peripheral visual system, we classify everyfeature transformation layer in the network into one of the four visual regions P ∈ { c , p , m , f } whererespective elements refer to central, para-central, mid, and far peripheral regions.

Original: PerViT-Attention ofhead h at layer l is classified as peripheral region p if the average of its attention scores which fall invisual region p is the largest among the others:

Original: [Equation]

Original: I m p a c t m e t r i c Ψ ∗ ( # , % ) ( T i n y ) 0.
Revised: We will include the text in our final manuscript (10-page limit) if accepted.

Original: Tiny Small

Original: Medium 0.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: where P is a set of spatial positions ( |P| = HW ) and I p is distance range (real-valued interval) ofperipheral region p 5 .

Original: The pie charts of Fig.

Original: 5 describe the proportions of peripheral regions for Tiny,

Original: Small, and Medium models where the bar graphs show them in layer-wise manner 6 .

Original: Similarly tothe visualized attention maps in Fig.

Original: 4, the early layers attends central/para-central regions whereasdeeper ones focus on outer region.

Original: We observe that, as the model size grows, the number of mid/farperipheral attention increases whereas that of central/para-central attention stays similar, suggestingthat the models no longer require local attentions once sufficient amount of processing is done in thecentral region because, we hypothesize, identifying geometric patterns, e.g. , corners and edges, isrelatively simpler process than understanding high-level semantics.

Original: Inspecting the impact of attentions (static vs. dynamic).
Revised: Inspecting the impact of attentions (static vs. dynamic).

Original: To study how position-based attentionsΦ p contribute to the mixed attentions Φ a = Φ c ⊙ Φ p , we collect sample images and visualize theirattention maps of Layers 3, 4 and 8 in Fig.
Revised: To study how position-based attentionsΦ p contribute to the mixed attentions Φ a = Φ c ⊙ Φ p , we collect sample images and visualize theirattention maps of Layers 3, 4 and 8 in Fig.

Original: The mixed attentions Φ a at Layer 4 are formeddynamically ( Φ c ) within statically-formed region ( Φ p ) while the attentions Φ a at Layer 8 weaklyexploit position information ( Φ p ) to form dynamic attentions ( Φ c ).
Revised: The mixed attentions Φ a at Layer 4 are formeddynamically ( Φ c ) within statically-formed region ( Φ p ) while the attentions Φ a at Layer 8 weaklyexploit position information ( Φ p ) to form dynamic attentions ( Φ c ).

Original: The results reveal that Φ p playstwo different roles; it imposes semi-dynamic attention if the attended region is focused in a smallarea whereas it serves as position bias injection when the attended region is relatively broad.
Revised: The results reveal that Φ p playstwo different roles; it imposes semi-dynamic attention if the attended region is focused in a smallarea whereas it serves as position bias injection when the attended region is relatively broad.

Original: Inthe supplementary, we constructively prove that an MPA layer in extreme case of semi-dynamicattention/position bias injection is in turn convolution/multi-head self-attention , naturally generalizingthe both transformations.
Revised: Inthe supplementary, we constructively prove that an MPA layer in extreme case of semi-dynamicattention/position bias injection is in turn convolution/multi-head self-attention , naturally generalizingthe both transformations.

Original: To quantitatively examine the contributions of Φ c and Φ p to the mixedattention Φ a , we define a measure of ‘impact’ by taking inverse of difference between two attentions:[Equation]
Revised: To quantitatively examine the contributions of Φ c and Φ p to the mixedattention Φ a , we define a measure of ‘impact’ by taking inverse of difference between two attentions:

Original: [Equation]
Revised: [Equation]

Original: where ∥·∥ F is Frobenius norm.
Revised: where ∥·∥ F is Frobenius norm.

Original: The higher the measure Ψ ( l,h ) p , the larger the impact of position-basedattention Φ ( l,h ) p .
Revised: The higher the measure Ψ ( l,h ) p , the larger the impact of position-basedattention Φ ( l,h ) p .

Original: Being averaged over all test samples, Ψ ( l,h ) c is similarly defined.
Revised: Being averaged over all test samples, Ψ ( l,h ) c is similarly defined.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 7,we observe a clear tendency that the impact of position-based attention is significantly higher inearly processing, transforming features semi-dynamically , while the later layers require less positioninformation, regarding Φ p as a minor position bias .
Revised: 7,we observe a clear tendency that the impact of position-based attention is significantly higher inearly processing, transforming features semi-dynamically , while the later layers require less positioninformation, regarding Φ p as a minor position bias .

Original: This tendency becomes more visible with largermodels as seen in right of Fig 7; Small and Medium models exploit dynamic transformations muchmore compared to Tiny model, especially in the later layers.
Revised: This tendency becomes more visible with largermodels as seen in right of Fig 7; Small and Medium models exploit dynamic transformations much I m p a c t m e t r i c Ψ ∗ ( # , % )

Original: Moreover, we note that the impactmeasures of four heads (bar graphs) within each layer are unevenly distributed, showing high variance,which imply that the network evenly utilizes both position and content information simultaneouslywithin each MPA layer as seen in Layer 3 in Fig. 6, performing both static/local and dynamic/global
Revised: ( T i n y ) 0. Tiny Small Medium 0. [Figure]

Original: [Table]
Revised: [Figure] [Figure] [Equation] [Figure]

Original: [Table]
Revised: [Equation]

Original: transformation in a single-shot .
Revised: more compared to Tiny model, especially in the later layers. Moreover, we note that the impactmeasures of four heads (bar graphs) within each layer are unevenly distributed, showing high variance,which imply that the network evenly utilizes both position and content information simultaneouslywithin each MPA layer as seen in Layer 3 in Fig. 6, performing both static/local and dynamic/globaltransformation in a single-shot .

Original: These results reveal that feature transformations for effective visualrecognition should not be restricted to be in neither position-only [19, 32] nor content-only [15, 42]design
Revised: These results reveal that feature transformations for effective visualrecognition should not be restricted to be in either position-only [25, 40] or content-only [18, 54]design

Original: but they should be in the form of a hybrid [9, 11].
Revised: but they should be in the form of a hybrid [10, 12].

Original: Inspecting the locality (local vs. global).
Revised: Inspecting the locality (local vs. global).

Original: We further investigate the inner workings of PerViT byquantifying how locally Φ ∗ attends.
Revised: We further investigate the inner workings of PerViT byquantifying how locally Φ ∗ attends.

Original: Following the work of [11] [Table] we define the measure of nonlocalityfor Φ ( l,h ) p by summing all pair-wise query-key distances weighted by their attention scores:Ω ( l,h ) p := [Equation]
Revised: Following the work of [12], we define the measure of nonlocalityfor Φ ( l,h ) p by summing all pair-wise query-key distances weighted by their attention scores:

Original: [Equation]
Revised: [Equation]

Original: This metrics Ω c and Ω a are similarly defined, being averaged over all test samples.
Revised: This metrics Ω c and Ω a are similarly defined, being averaged over all test samples.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 8,we observe a similar trend of locality between Φ p and Φ a , which reveals the position informationplay more dominant role over the content information in forming spatial attentions ( Φ a ) for featuretransformation.
Revised: 8,we observe a similar trend of locality between Φ p and Φ a , which reveals the position informationplay more dominant role over the content information in forming spatial attentions ( Φ a ) for featuretransformation.

Original: Interestingly, we also observe that content- and position-based attentions behaveconversely; Φ c attends globally in early layers, i.e. , large scores are distributed over the whole spatialregion, while being relatively local in deeper layers.
Revised: Interestingly, we also observe that content- and position-based attentions behaveconversely; Φ c attends globally in early layers, i.e. , large scores are distributed over the whole spatialregion, while being relatively local in deeper layers.

Original: We hypothesize that the proposed Φ p in earlylayers is trained to effectively suppress unnecessary scores of Φ c at distant positions, thus exploitingonly a few relevant ones within its local region of interest.
Revised: We hypothesize that the proposed Φ p in earlylayers is trained to effectively suppress unnecessary scores of Φ c at distant positions, thus exploitingonly a few relevant ones within its local region of interest.

Original: Meanwhile, Φ p at later layers gives Φ chigher freedom in forming the spatial attention Φ a as described in the plots of Fig. 7, which allowsthe attention scores of Φ c to be clustered in semantically meaningful parts, e.g. , eyes of the animalsas seen in attentions of the first head at Layer 8 (Fig. 6), which makes Φ c relatively local.
Revised: Meanwhile, Φ p at later layers gives Φ chigher freedom in forming the spatial attention Φ a as described in the plots of Fig. 7, which allowsthe attention scores of Φ c to be clustered in semantically meaningful parts, e.g. , eyes of the animalsas seen in attentions of the first head at Layer 8 (Fig. 6), which makes Φ c relatively local.

Original: 4.2 Quantitative evaluation on ImageNet-1K
Revised: 4.2 Quantitative evaluation on ImageNet-1K

Original: Comparison with state of the arts.
Revised: Ablation study on main components.

Original: Table 3 summarizes the results of our method and recent stateof the arts.
Revised: In Tab. 1, we analyze the impact of each component in PerViT,which is denoted as (a), where C-stem refers to convolutional patch embedding stem 5 . We observethat the proposed attention Φ p brings consistent gains to models (b, f, g, i) with relative improvementsof 1.4 ∼ 4.2%p for (a, d, e, h) respectively.

Original: For fair comparison, the baselines used in our comparison are trained using 224 × 224input resolution without distillations, and are grouped into either pyramidal or cylindrical ViT basedon the network designs, i.e. , multi- or single-resolution feature processing, where the results arepartitioned according to  model sizes within each group.
Revised: Among three main components ( Φ p , C-stem, CPE), Φ p hasthe most significant impact on PerViT (a), losing 1.5%p Top-1 accuracy without Φ p , i.e. , model (b). Surprisingly, PerViT without content-based attention Φ c , model (c), achieves decent performance,almost equalling to the accuracy of PerViT without position-based attention Φ p , model (b) (-0.5%p). [Table]

Original: As shown in thebottom section of Tab 3,the proposed method achieves consistent improvements over the state-of-the-art cylindrical ViTmethods [11, 16, 42, 55] while showing competitive results to the pyramidal counterparts.
Revised: [Table] [Table] [Table] The results verify that the proposed peripheral attention, which achieves comparable level of efficacyto the content-based attention, learns to generate reliable spatial attentions for visual recognition.

Original: We alsoimplement the proposed position-based attention Φ p on DeiT [42] baseline and compare the resultswith recent state-of-the-art RPE methods.
Revised: Wealso implement the proposed position-based attention Φ p on DeiT [54] baseline and compare theresults with recent state-of-the-art RPE methods.

Original: As seen in Tab. 2, the large improvements over the previous RPE methods [7, 50] further verify the efficacy of the proposed peripheral position encoding (PPE).
Revised: As seen in Tab. 2, the large improvements over theprevious RPE methods [8, 65] further verify the efficacy of the proposed peripheral position encoding(PPE).

Revised: To confirm that the impact of Φ p is consistent with large models, we conduct similar ablationsusing PerViT-S/M in Tab. 3; without Φ p , the accuracy consistently drops for all the three models.

Revised: Comparing (b) with (c), we observe that C-Stem and CPE are less effective for large models, bringing1.3%p and 0.1%p gains for Small and Medium respectively whereas they improve the Tiny model by5.1%p.

Revised: In contrast, the impact of Φ p is consistent across different model sizes, bringing 1%p gainsfor all the three models.

Original:  Ablation study on Φ p .
Revised: The better efficacy of Φ p for larger models, we hypothesize, is due to itsflexibility in modeling local/global spatial attentions while C-Stem/CPE are designed only to be local. Sample-efficiency of PerViT. To investigate the training sample efficiency of our model, we train PerViT-S with ImageNet subsampled by fractions of 50% and 25% 6 and evaluate it on full-sized testset of ImageNet-1K. Table 4 compare our results with DeiT [54]; our model consistently surpassesthe baseline for all subsampled datasets, showing its robustness under limited training data. Ablation study on Φ p .

Original: The top section of Tab.
Revised: The top section of Tab.

Original: 4 reports results of PerViT-T with different parameterinitialization methods for Φ p where peripheral denotes the proposed peripheral initialization, convrefers to convolutional initialization such that s l = − 5 . 

Revised: 5 reports results of PerViT-T with different parameterinitialization methods for Φ p where peripheral denotes the proposed peripheral initialization, convrefers to convolutional initialization such that s l = − 5 .

Original: and v l = 3 . 0 for all l ∈ [ N l ] , and randrefers to random initialization for all parameters in Φ p : w r , W ( l ) p1 , W ( l,h ) p2 , γγγ ( l ) p1 , βββ ( l ) p1 , γ ( l,h ) p2 , andβ ( l,h ) p2 .
Revised: and v l = 3 . 0 for all l ∈ [ N l ] , and randrefers to random initialization for all parameters in Φ p : w r , W ( l ) p1 , W ( l,h ) p2 , γγγ ( l ) p1 , βββ ( l ) p1 , γ ( l,h ) p2 , andβ ( l,h ) p2 .

Original: The results show the efficacy of our peripheral initialization which is also supported bythe results in Fig.
Revised: The results show the efficacy of our peripheral initialization which is also supported bythe results in Fig.

Original: 4 and 8: Φ p provides early local and late global attentions, suggesting thatperipheral initialization effectively reduces burden in learning such form of attentions.
Revised: 4 and 8: Φ p provides early local and late global attentions, suggesting thatperipheral initialization effectively reduces burden in learning such form of attentions.

Original: The bottomsection of Tab.
Revised: The bottomsection of Tab.

Original: 4 studies network designs for Φ p where N represents the proposed peripheralprojection, i.e. , projecting input distance representation by referring neighbors N , ML refers tomulti-layer design of Φ p , and Euc & Lrn indicate the type of embedding R : Euc is relative Euclideandistances ( R q , k , : = concat r ∈
Revised: 5 studies network designs for Φ p where N represents the proposed peripheralprojection, i.e. , projecting input distance representation by referring neighbors N , ML refers tomulti-layer design of Φ p , and Euc & Lrn indicate the type of embedding R : Euc is relative Euclideandistances ( R q , k , : = concat r ∈

Original: [ D r ] [ w r · R ] ) where Lrn is relative distances between learnable vectors( R ∈ R
Revised: [ D r ] [ w r · R ] ) where Lrn is relative distances between learnable vectors( R ∈ R

Original: HW × HW × D r ).
Revised: HW × HW × D r ).

Original: Without N and ML, we observe consistent accuracy drops for Euc and Lrn [Table] [Table] by 0.8%p and 0.2%p respectively.
Revised: Without N and ML, we observe consistent accuracy drops for Euc and Lrnby 0.8%p and 0.2%p respectively.

Original: A sole multi-layer projection hardly improves accuracy but themodel performs the best when N is jointly used, meaning that both need to complement each otherto provide diverse attention shapes as in Fig 4.
Revised: A sole multi-layer projection hardly improves accuracy but themodel performs the best when N is jointly used, meaning that both need to complement each otherto provide diverse attention shapes as in Fig 4.

Original: Furthermore, Euc models consistently surpasses Lrnmodels, implying the Euclidean distance is more straightforward encoding type than learnable vectorsin capturing spatial configurations of images.
Revised: Furthermore, Euc models consistently surpasses Lrn models, implying the Euclidean distance is more straightforward encoding type than learnablevectors in capturing spatial configurations of images.

Original: Ablation study on main components. In Tab. 1, we analyze the impact of each component in PerViT,which is denoted as (a), where C-stem refers to convolutional patch embedding stem 7 . We observethat the proposed attention Φ p brings consistent gains to models (b, f, g, i) with relative improvementsof 1.4 ∼ 4.2%p for (a, d, e, h) respectively.
Revised: To further highlight the benefits of learningΦ p , we conduct experiments using PerViT-T with parameters of Φ p fixed during training under threedifferent initialization methods of peripheral, conv, and rand. Table 7 summarizes the results.

Original: Among three main components ( Φ p , C-stem, CPE), Φ p hasthe most significant impact on PerViT (a), losing 1.5%p Top-1 accuracy without Φ p , i.e. , model (b).
Revised: FixingΦ p parameters damages performance for all three intializations, verifying the efficacy of learning de signs bottom based attention [Table] [Table]

Original: Surprisingly, PerViT without content-based attention Φ c , model (c), achieves decent performance,almost equalling to the accuracy of PerViT withoutposition-based attention Φ p ,  model (b) (-0.5%p).{+in Tab. 
+}
Revised: [Table] [Table] diverse position-based attentions across different layers and heads. We observe that conv and randinits perform poorly compared to PerViT-T without Φ p , e.g. , model (b) {+in Tab. 
+}

Original: The results verify that the proposed peripheral attention, which achieves comparable level ofefficacyto the content-based attention, learns to generate reliable spatial attentions for visual recognition.
Revised: (77.3%); we suspectthat fixed Φ p with conv and rand inits only provide local and noisy attentions respectively while PerViT without Φ p has no such strong restrictions. Comparison with state of the arts.

Original: Scope and Limitations
Revised: Table 6 summarizes the results of our method and recent state ofthe arts. For fair comparison, the baselines used in our comparison are trained using 224 × 224 inputresolution without distillations, and are grouped into either pyramidal or columnar ViT based on thenetwork designs, i.e. , multi- or single-resolution feature processing, where the results are partitionedaccording to model sizes within each group. As shown in the bottom section of Tab 6, the proposedmethod achieves consistent improvements over the recent columnar ViT methods [12, 19, 54, 70]while

Original: Despite the interpretability and effectiveness of PerViT, it still leaves much room for improvements. First, PerViT-Attention (Eq. 5) bases on the original self-attention formulation [46], thus directlyinheriting its limitations [15, 42], e.g. , quadratic complexity w.r.t.
Revised: showing competitive results to the pyramidal counterparts. To further verify the robustnessof the proposed method, we compare the PerViT-M with baseline models [54, 18] on differenttransfer learning task with ImageNet pre-training in Tab. 8.

Original: input resolution.
Revised: We finetune trained PerViT-M on CIFAR10, CIFAR-100, and iNaturalist-19, following the same training recipes of DeiT [54].

Original: The computationalefficiency could be further improved by approximating low-rank matrices as in [6, 29, 53].
Revised: Even withsignificantly lower complexity than [54, 18], our method surpasses baselines by approx.

Original: Second,given the ability to process high-resolution input with feasible complexity, the efficacy of PerViTcould be improved by adopting multi-resolution pyramidal design following recent trend of ViTdesigns [14, 27, 29, 31, 48, 49, 53, 54].
Revised: 1%p on CIFAR-100 and iNaturalist19 while performing on par with [54] on CIFAR-10.

Original: Third, the focus of this paper is model development &exploration forimage classification task but we believe the proposed idea is broadly generalizable toother vision applications such as object detection and segmentation. We leave this to future work.
Revised: We omit the existing text in this draft due to 9-page limit for the revised manuscript. We will includethe text in our final manuscript (10-page limit) if accepted.

Original:  Conclusion
Revised:  Conclusion

Original: This paper explores blending human peripheral vision with machine vision for effective visualrecognition, and introduces Peripheral Vision Transformer which learns to provide diverse positionbased attentions to model peripheral vision using peripheral projections and initialization.
Revised: This paper explores blending human peripheral vision with machine vision for effective visualrecognition, and introduces Peripheral Vision Transformer which learns to provide diverse positionbased attentions to model peripheral vision using peripheral projections and initialization.

Original: We havesystematically investigated the inner workings of the proposed network and observed that the networkenjoys the benefits of both convolution and self-attention by learning to decide level of the locality& dynamicity in feature transformation, by the network itself given training data.
Revised: We havesystematically investigated the inner workings of the proposed network and observed that the networkenjoys the benefits of both convolution and self-attention by learning to decide level of the locality& dynamicity in feature transformation, by the network itself given training data.

Original: The consistentimprovements over the state of the arts on ImageNet across different model sizes and in-depth ablationstudy confirm the efficacy of the proposed approach.
Revised: The consistentimprovements over the baseline models on ImageNet-1K across different model sizes and in-depthablation study confirm the efficacy of the proposed approach.