Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models (2022)

Abstract: There is growing evidence that pretrained language models improve task-specific fine-tuning even where the task examples are radically different from those seen in training. We study an extreme case of transfer learning by providing a systematic exploration of how much transfer occurs when models are denied any information about word identity via random scrambling. In four classification tasks and two sequence labeling tasks, we evaluate LSTMs using GloVe embeddings, BERT, and baseline models. Among these models, we find that only BERT shows high rates of transfer into our scrambled domains, and for classification but not sequence labeling tasks. Our analyses seek to explain why transfer succeeds for some tasks but not others, to isolate the separate contributions of pretraining versus fine-tuning, to show that the fine-tuning process is not merely learning to unscramble the scrambled inputs, and to quantify the role of word frequency. Furthermore, our results suggest that current benchmarks may overestimate the degree to which current models actually understand language.

## introduction fine-tuning pretrained language models has proven to be highly effective across a wide range of nlp tasks; the leaderboards for standard benchmarks are currently dominated by models that adopt this general strategy @xcite @xcite @xcite . recent work has extended these findings in even more surprising ways: @xcite , @xcite , and tran (2020) find evidence of transfer between natural languages, and @xcite show that pretraining language models on non-linguistic data such as music and computer code can improve test performance on natural language. recently, @xcite show that bert's performance on downstream glue tasks suffers starting with a model (e.g., pretrained bert, gloveinitialized lstm, etc.), we copy it and fine-tune it on the regular and scrambled train set using a scrambling function f. the model is then evaluated on regular and scrambled test sets. our paper explores different options for f and a number of variants of our models to try to quantity the amount of transfer and identify its sources. figure 1 gives an overview of our core experimental paradigm: starting with two identical copies of a single pretrained model for english, we finetune one on english examples and the other on scrambled english sentences, using a scrambling function f (section 3), and then we evaluate the resulting models. we apply this paradigm to four classification tasks and two sequence modeling tasks, and we evaluate bag-of-words baselines, lstms with glove initialization and rich attention mechanisms, and bert. our central finding is that, for bert, high rates of transfer occur on classification tasks, but not sequence labeling tasks to better understand why such transfer is successful for some tasks but not others, we pursue a number of hypotheses. first, we assess whether the transfer occurs if only word identities are scrambled among words with similar frequencies. second, we assess whether our matching methods might actually be inserting semantic consistency into the scrambling process by matching synonyms. third, we analyze the learning dynamics behind such transfer by studying the effects of model pretraining. our findings suggest that performance on existing tasks may be less informative than previously thought about the degree to which a model understands language. our pretrained models transfer knowledge in our tasks even when they are denied any information about word identity. thus, a large percentage of their success might trace to factors that have nothing to do with communication or understanding. after all, our scrambled data do not have the semantics of english, or indeed of any language. ## experimental paradigm we now describe the evaluation paradigm summarized in figure 1 (section 3.1), with special attention to the scrambling functions f that we consider (sections 3.2-3.3). ## models in this section, we describe the models we evaluated within our paradigm. our supplementary materials provide additional details about how the models were designed, optimized, and evaluated. bert for our bert model @xcite , we import weights from the pretrained bert-base model through the huggingface transformers library @xcite . for sequence classification tasks, we append a classification head after the [cls] token embedding in the last layer of the bert model. if an input example contains a pair of sentences, we concatenate them using a [sep] token in between. for sequence labeling tasks, we append a shared classification head to each token embedding in the last layer of the bert model. our supplementary materials provide results for deberta models @xcite as well. lstm we contextualize our results by comparing them against a strong lstm-based model @xcite . we lowercase each input sentence and tokenize it by separating on spaces and punctuation. we then use 300-dimensional glove embeddings @xcite foot_1 as inputs to a single-layer recurrent neural network with lstm cells, with a hidden size of 64. we use dot-product attention @xcite to formulate a context vector for each sentence. finally, we pass the context vector through a multilayer perceptron (mlp) layer with a hidden size of 64 to get the final prediction. for an input example with a pair of sentences, we concatenate two sentences together with a separator token before feeding them into our lstm encoder. for sequence labeling tasks, we directly feed the hidden state at each position to the mlp layer to get the final prediction. ## results in this section, we analyze the fine-tuning performance of bert on scrambled datasets. ## conclusion in this paper, we propose an evaluation pipeline for pretrained models by testing their transfer abilities when they are denied all information about word identity. specifically, we take an english pretrained bert off-the-shelf and fine-tune it with a scrambled english dataset. we conduct analyses across six tasks covering both classification and sequence labeling. by evaluating performance against multiple baselines, we aim to assess where bert can transfer knowledge even without word identities. we find considerable transfer for bert as compared to even powerful baselines, but only for classification tasks. what is the source of successful cross-domain transfer with bert? we find that word frequency contributes, but only to a limited extent: scrambling with matched word frequencies consistently outperforms scrambling with unmatched word frequencies, but transfer still occurs robustly even with random scrambling. we are also able to determine that both pretraining and fine-tuning are important and interacting factors in this transfer; freezing bert weights during task-specific training leads to much less transfer, but too much task-specific training erodes the benefits of pretraining and in turn reduces the amount of transfer observed. these analyses begin to piece together a full account of these surprising transfer results for bert, but they do not fully explain our experimental results. recent literature suggests at least two new promising avenues to explore. first, @xcite seek to help characterize the rich distributional prior that models like bert may be learning, which suggests that higher-order notions of frequency play a significant role in transfer. second, the findings of @xcite may be instructive: through successful layers, bert seems to perform specific kinds of dimensionality reduction that help with low-dimensional classification tasks. our results concerning layer-wise variation are consistent with this. our results are also highly relevant to questions of benchmarking in nlp. it is widely assumed that the benchmark tasks we considered here can help illuminate the capacity of modern nlp systems to process and understand language. however, in our experiments, fine-tuned bert models are successful at these tasks even in scrambled conditions that render all the examples meaningless, which should lead us to think critically about whether success in the usual unscrambled conditions is reliable evidence of understanding.