  Recent developments in Natural Language Processing (NLP) demonstrate that
large-scale, self-supervised pre-training can be extremely beneficial for
downstream tasks. These ideas have been adapted to other domains, including the
analysis of the amino acid sequences of proteins. However, to date most
attempts on protein sequences rely on direct masked language model style
pre-training. In this work, we design a new, adversarial pre-training method
for proteins, extending and specializing similar advances in NLP. We show
compelling results in comparison to traditional MLM pre-training, though
further development is needed to ensure the gains are worth the significant
computational cost.
