Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models’ Transferability (2021)

Abstract: This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. To verify pre-trained models’ transferability, we test the pre-trained models on text classification tasks with meanings of tokens mismatches, and real-world non-text token sequence classification data, including amino acid, DNA, and music. We find that even on non-text data, the models pre-trained on text converge faster, perform better than the randomly initialized models, and only slightly worse than the models using task-specific knowledge. We also find that the representations of the text and non-text pre-trained models share non-trivial similarities.

## introduction in recent nlp research, pre-trained masked language models (mlms) such as bert @xcite are widely used by practitioners. after pretrained on large corpora such as wikipedia, the models can be fine-tuned quickly on nlp tasks like text classification and question answering and generalize well on small datasets such as rte in @xcite . to apply and improve bert in a more specialized domain such as scientific articles or clinical data, several mlms are proposed by pre-training on the domain-specific text data @xcite . the concept of mlm can also be extended to other disciplines (maybe non-linguistic) as long as the input is discrete. for example, @xcite pretrain mlms called plus on amino acid sequence data and achieve state-of-the-art performance on several protein classification tasks. this paper examines whether the model pretrained on large text corpora, such as bert, can be efficiently adapted to data with numbers of tokens, token distribution, labels, and structure very different from natural language (the target data could even be non-text). we refer to this ability as discipline adaptability foot_0 . previous work (ethayarajh and jurafsky, 2020) only shows that language models (lms) pre-trained on non-text data can be adapted to lms of human languages. this work is the first to examines if the pre-trained mlms can learn the relation between the label and the data never seen during pre-training. our contributions are the following. we believe the findings of discipline adaptation will intrigue the nlp community to ponder what is learned in the pre-training procedure. the findings can also be helpful to the disciplines that largescale datasets are not available, which are essential for practitioners. to examine the discipline adaptability of the models pre-trained on text corpora, we fine-tune the pretrained models on two types of downstream data. the first type (section 2.1) are synthetic datasets, which are generated by permuting tokens in common nlp datasets. so the meaning of each token is changed. the second type of data (section 2.2) is a more challenging situation in which the downstream tasks are not relevant to human language. ## discussion the results in section 3.2 validate the potential of the pre-trained models as strong cross-disciplinary ## conclusion in this paper, we investigate the potential of bert as a cross-disciplinary knowledge learner. by finetuning bert on the synthetic text data with meanings of tokens changed and the non-text data, we verify that bert can be adapted to data of different disciplines efficiently and generalizes well. besides, we discover the non-trivial similarity between the models pre-trained on text and protein before fine-tuning by pwcca, which helps to explain the reasons behind bert's discipline adaptability. we hope that the proposed settings can act as new analysis tools for researchers and provide new insight into the power of pre-trained models.