Visual recognition has been dominated by convolutionalneural networks (CNNs) for years. Though recently the pre-vailing vision transformers (ViTs) have shown great poten-tial of self-attention based models in ImageNet classifica-tion , their performance is still inferior to latest SOTA CNNsif no extra data are provided. In this work, we aim to closethe performance gap and demonstrate that attention-basedmodels are indeed able to outperform CNNs. We found thatthe main factor limiting the performance of ViTs for Ima-geNet classification is their low efficacy in encoding fine-level features into the token representations. To resolvethis , we introduce a noveloutlook attentionand present asimple and general architecture, termed Vision Outlooker (VOLO). Unlike self-attention that focuses on global depen-dency modeling at a coarse level, the outlook attention aimsto efficiently encode finer-level features and contexts intotokens, which are shown to be critical for recognition per-formance but largely ignored by the self-attention. Experi-ments show that our VOLO achieves 87.1\% top-1 accuracyon ImageNet-1K classification, being the first model exceed-ing 87\% accuracy on this competitive benchmark, withoutusing any extra training data . In addition, the pre-trainedVOLO transfers well to downstream tasks, such as seman-tic segmentation. We achieve 84.3\% mIoU score on thecityscapes validation set and 54.3\% on the ADE20K valida-tion set. Code is available at URL
Visual recognition has been dominated by convolutional neural networks (CNNs) for years. Though recently the prevailing vision transformers (ViTs) have shown great potential of self-attention based models in ImageNet classification , their performance is still inferior to that of the latest SOTA CNNs if no extra data are provided. In this work, we try to close the performance gap and demonstrate that attention-based models are indeed able to outperform CNNs. We find a major factor limiting the performance of ViTs for ImageNet classification is their low efficacy in encoding fine-level features into the token representations. To resolve this , we introduce a novel outlook attention and present a simple and general architecture, termed Vision Outlooker (VOLO). Unlike self-attention that focuses on global dependency modeling at a coarse level, the outlook attention efficiently encodes finer-level features and contexts into tokens, which is shown to be critically beneficial to recognition performance but largely ignored by the self-attention. Experiments show that our VOLO achieves 87.1\% top-1 accuracy on ImageNet-1K classification, which is the first model exceeding 87\% accuracy on this competitive benchmark, without using any extra training data In addition, the pre-trained VOLO transfers well to downstream tasks, such as semantic segmentation. We achieve 84.3\% mIoU score on the cityscapes validation set and 54.3\% on the ADE20K validation set. Code is available at URL
R
convolutionalneural
convolutional neural
R
pre-vailing
prevailing
R
poten-tial
potential
R
classifica-tion
classification
R
latest SOTA CNNsif
that of the latest SOTA CNNs if
R
aim to closethe
try to close the
R
attention-basedmodels
attention-based models
R
found thatthe main
find a major
R
Ima-geNet
ImageNet
R
resolvethis
resolve this
R
noveloutlook attentionand present asimple
novel outlook attention and present a simple
R
depen-dency
dependency
R
aimsto efficiently encode
efficiently encodes
R
intotokens, which are
into tokens, which is
R
critical for recognition per-formance
critically beneficial to recognition performance
R
Experi-ments
Experiments
R
accuracyon
accuracy on
R
being
which is
R
exceed-ing
exceeding
R
withoutusing
without using
D
.
R
pre-trainedVOLO
pre-trained VOLO
R
seman-tic
semantic
R
thecityscapes
the cityscapes
R
valida-tion
validation