  There is growing evidence that pretrained language models improve
task-specific fine-tuning not just for the languages seen in pretraining, but
also for new languages and even non-linguistic data. What is the nature of this
surprising cross-domain transfer? We offer a partial answer via a systematic
exploration of how much transfer occurs when models are denied any information
about word identity via random scrambling. In four classification tasks and two
sequence labeling tasks, we evaluate baseline models, LSTMs using GloVe
embeddings, and BERT. We find that only BERT shows high rates of transfer into
our scrambled domains, and for classification but not sequence labeling tasks.
Our analyses seek to explain why transfer succeeds for some tasks but not
others, to isolate the separate contributions of pretraining versus
fine-tuning, and to quantify the role of word frequency. These findings help
explain where and why cross-domain transfer occurs, which can guide future
studies and practical fine-tuning efforts.
