  In this paper, we investigate whether the power of the models pre-trained on
text data, such as BERT, can be transferred to general token sequence
classification applications. To verify pre-trained models' transferability, we
test the pre-trained models on (1) text classification tasks with meanings of
tokens mismatches, and (2) real-world non-text token sequence classification
data, including amino acid sequence, DNA sequence, and music. We find that even
on non-text data, the models pre-trained on text converge faster than the
randomly initialized models, and the testing performance of the pre-trained
models is merely slightly worse than the models designed for the specific
tasks.
