  With the rise of Transformers as the standard for language processing, and
their advancements in computer vision, along with their unprecedented size and
amounts of training data, many have come to believe that they are not suitable
for small sets of data. This trend leads to great concerns, including but not
limited to: limited availability of data in certain scientific domains and the
exclusion of those with limited resource from research in the field. In this
paper, we dispel the myth that transformers are "data hungry" and therefore can
only be applied to large sets of data. We show for the first time that with the
right size and tokenization, transformers can perform head-to-head with
state-of-the-art CNNs on small datasets. Our model eliminates the requirement
for class token and positional embeddings through a novel sequence pooling
strategy and the use of convolutions. We show that compared to CNNs, our
compact transformers have fewer parameters and MACs, while obtaining similar
accuracies. Our method is flexible in terms of model size, and can have as
little as 0.28M parameters and achieve reasonable results. It can reach an
accuracy of 94.72% when training from scratch on CIFAR-10, which is comparable
with modern CNN based approaches, and a significant improvement over previous
Transformer based models. Our simple and compact design democratizes
transformers by making them accessible to those equipped with basic computing
resources and/or dealing with important small datasets. Our code and
pre-trained models will be made publicly available at
https://github.com/SHI-Labs/Compact-Transformers.
