M 2 D 2: A Massively Multi-Domain Language Modeling Dataset (2022)

Abstract: We present M2D2, a fine-grained, massively multi-domain corpus for studying domain adaptation in language models (LMs). M2D2 consists of 8.5B tokens and spans 145 domains extracted from Wikipedia and Semantic Scholar. Using ontologies derived from Wikipedia and ArXiv categories, we organize the domains in each data source into 22 groups. This two-level hierarchy enables the study of relationships between domains and their effects on in- and out-of-domain performance after adaptation. We also present a number of insights into the nature of effective domain adaptation in LMs, as examples of the new types of studies M2D2 enables. To improve in-domain performance, we show the benefits of adapting the LM along a domain hierarchy; adapting to smaller amounts of fine-grained domain-specific data can lead to larger in-domain performance gains than larger amounts of weakly relevant data. We further demonstrate a trade-off between in-domain specialization and out-of-domain generalization within and across ontologies, as well as a strong correlation between out-of-domain performance and lexical overlap between domains.

## introduction even though they can contain a wide variety of different types of domains, the texts that make up the corpora used to train and evaluate language models (lms) are often treated as if they are all the same. this makes it challenging to characterize lm performance under diverse data distributions and understand how to effectively adapt lms to new ones. to address these challenges, we develop m2d2, a massively multi-domain dataset, with 145 subdomains and a human-curated hierarchy for studying fine-grained domain adaptation. prior work on domain transfer focuses on a small number of broad domains (typically 4-20; @xcite @xcite . in contrast, domains in m2d2 are fine-grained and organized into a hierarchy derived from human-curated ontologies in wikipedia (figure 1 ) and semantic scholar (figure 2 ). unlike prior work, the fine granularity of m2d2 enables the study of transfer to naturally occurring datascarce domains recognized by human curators (e.g. philosophy, public health, transport). this hierarchy enables the study of domain transfer at varying levels of topic granularity. for instance, how should we combine widely available internet text (entire corpus), text on computer science (coarse domain), and scarce corpus on machine learning (fine domain) to improve performance in the machine learning domain? to the best of our knowledge, m2d2 is the first dataset that combines fine domain granularity with human-curated domain hierarchy in a massively multi-domain setting. using m2d2, we investigate the following questions, as examples of the broad classes of new questions that can be asked: (1) how well do coarse and fine domains transfer to each other across the hierarchy? (2) which features and aspects of a domain are important for transfer? (3) how important is domain specificity versus breadth? we perform preliminary experiments analyzing transfer between similar domains, disparate domains, and hierarchically related domains. moreover, we explore how to select source domains to improve transfer performance. we present baseline experiments using a gpt2 @xcite language model. we find that (1) more specific data is often more important for performance than larger, less-specific data, shown by our comparison of coarse-grained, fine-grained and coarse-to-fine adaptation comparison (in which coarse-to-fine performed best) , (2) vocabulary overlap is a surprising good indicator for transfer, and (3) data source provenance information is often a better predictor than ontology when predicting transferability, perhaps indicating a more multi-faceted definition of domain could be developed in future work. given the importance of fine granularity domains in language modeling, we hope that m2d2 will encourage the community to further study domain transfer: how do we identify hierarchical finegrained domains in naturally occurring text, and how do we leverage this fine-grained domain hier-archy to improve domain transfer. ## m2d2 m2d2 consists of a large quantity of fine-grain domains. unlike prior work that defines the domain of a corpus using its source (e.g. the web text domain; @xcite , we derive domains from a human-curated wikipedia and arxiv ontologies. in this section, we describe how m2d2 is collected and organized. ## experiments as examples of the types of new studies m2d2 enables, we explore a number of key questions about the nature of effective domain adaptation in language models. for example, how does one best specialize a language model to a domain, given an ontology? how well can adapted models be applied out-of-domain, within and across ontologies? what features of target domains are predictive of out-ofdomain transfer? in this section, we present a set of experiments that begin to answer these questions. first, we study the impact of adapting to the l1 and l2 domains of our dataset on in-domain ( §3.2) and outof-domain ( §3.3) language modeling performance. then, we perform an analysis of lexical features in domains that are predictive of out-of-domain performance ( §3.4). ## conclusion we developed m2d2, a new massively multidomain language modeling dataset for studying domain adaptation in language models. m2d2 consists of 145 fine-grained domains (curated from wikipedia and semantic scholar) that are hierarchically organized using domain-specific ontologies. using m2d2, we find that domain precision is more important than data quantity to improve in-domain performance, a tradeoff between specialization and out-of-domain generalization. we release m2d2 publicly to spur further research on building effective language models on highly heterogeneous data.