  In this paper, we study how the intrinsic nature of pre-training data
contributes to the fine-tuned downstream performance. To this end, we pre-train
different transformer-based masked language models on several corpora with
certain features, and we fine-tune those language models on GLUE benchmarks. We
find that models pre-trained on unstructured data beat those trained directly
from scratch on downstream tasks. Our results also show that pre-training on
structured data does not always make the model acquire ability that can be
transferred to natural language downstream tasks. To our great astonishment, we
uncover that pre-training on certain non-human language data gives GLUE
performance close to performance pre-trained on another non-English language.
