  While Transformer architectures have show remarkable success, they are bound
to the computation of all pairwise interactions of input element and thus
suffer from limited scalability. Recent work has been successful by avoiding
the computation of the complete attention matrix, yet leads to problems down
the line. The absence of an explicit attention matrix makes the inclusion of
inductive biases relying on relative interactions between elements more
challenging. An extremely powerful inductive bias is translational
equivariance, which has been conjectured to be responsible for much of the
success of Convolutional Neural Networks on image recognition tasks. In this
work we show how translational equivariance can be implemented in efficient
Transformers based on kernelizable attention - Performers. Our experiments
highlight that the devised approach significantly improves robustness of
Performers to shifts of input images compared to their naive application. This
represents an important step on the path of replacing Convolutional Neural
Networks with more expressive Transformer architectures and will help to
improve sample efficiency and robustness in this realm.
