  Introducing the transformer structure into computer vision tasks holds the
promise of yielding a better speed-accuracy trade-off than traditional
convolution networks. However, directly training vanilla transformers on vision
tasks has been shown to yield unstable and sub-optimal results. As a result,
recent works propose to modify transformer structures by incorporating
convolutional layers to improve the performance on vision tasks. This work
investigates how to stabilize the training of vision transformers
\emph{without} special structure modification. We observe that the instability
of transformer training on vision tasks can be attributed to the over-smoothing
problem, that the self-attention layers tend to map the different patches from
the input image into a similar latent representation, hence yielding the loss
of information and degeneration of performance, especially when the number of
layers is large. We then propose a number of techniques to alleviate this
problem, including introducing additional loss functions to encourage
diversity, prevent loss of information, and discriminate different patches by
additional patch classification loss for Cutmix. We show that our proposed
techniques stabilize the training and allow us to train wider and deeper vision
transformers, achieving 85.0\% top-1 accuracy on ImageNet validation set
without introducing extra teachers or additional convolution layers. Our code
will be made publicly available at
https://github.com/ChengyueGongR/PatchVisionTransformer .
