  In this paper, we present Co-scale conv-attentional image Transformers
(CoaT), a Transformer-based image classifier equipped with co-scale and
conv-attentional mechanisms. First, the co-scale mechanism maintains the
integrity of Transformers' encoder branches at individual scales, while
allowing representations learned at different scales to effectively communicate
with each other; we design a series of serial and parallel blocks to realize
the co-scale attention mechanism. Second, we devise a conv-attentional
mechanism by realizing a relative position embedding formulation in the
factorized attention module with an efficient convolution-like implementation.
CoaT empowers image Transformers with enriched multi-scale and contextual
modeling capabilities. On ImageNet, relatively small CoaT models attain
superior classification results compared with the similar-sized convolutional
neural networks and image/vision Transformers. The effectiveness of CoaT's
backbone is also illustrated on object detection and instance segmentation,
demonstrating its applicability to the downstream computer vision tasks.
