Cross-Domain Language Modeling: An Empirical Investigation (2021)

Abstract: Transformer encoder models exhibit strong performance in single-domain applications. However, in a cross-domain situation, using a sub-word vocabulary model results in sub-word overlap. This is an issue when there is an overlap between sub-words that share no semantic similarity between domains. We hypothesize that alleviating this overlap allows for a more effective modeling of multi-domain tasks; we consider the biomedical and general domains in this paper. We present a study on reducing sub-word overlap by scaling the vocabulary size in a Transformer encoder model while pretraining with multiple domains. We observe a significant increase in downstream performance in the general-biomedical cross-domain from a reduction in sub-word overlap.

## introduction contemporary language models are pretrained on massive, linguistically diverse corpora @xcite . it is not uncommon for these models to excel at benchmark downstream tasks @xcite , given the use of contextual representations @xcite ) that are trained on a variety of source domains-a term used to describe a distribution of language on a given topic or genre (for example biomedical, scientific)-or general domain. however, the benefit of general domain pretraining for specialized application is questionable, as applying these language models @xcite to specialized tasks is worse than using specialized counterparts @xcite . this degradation still occurs after sequential pretraining on specialized domains @xcite when fine-tuned (updates to pretraining) to downstream tasks. we hypothesize some of this degradation lies in the use of a sub-word vocabulary @xcite . sub-word vocabularies @xcite allow for efficient modeling of a source language distribution with a limited vocabulary size. however, problematically sub-words can be shared between different words-for example hypotension and hypocritical-with different meanings. this potentially conflates the vector representation of a sub-word (or wordpiece) causing sub-word overlap. when this overlap occurs with sub-words appearing in multiple domain contexts we call this cross-domain sub-word overlap. as a pilot empirical study, we investigate reducing cross-domain sub-word overlap, by increasing vocabulary size, in language models pretrained in the general and biomedical cross-domain. to evaluate the effect of sub-word overlap, general and biomedical domain benchmarks are used in this study as the task distribution includes different linguistic phenomena such as grammar, sentiment, textual similarity, natural language inference @xcite . interestingly, we find that disjoint sub-word vocabulary sets are not ideal. some sub-word overlap is necessary and unavoidable, and a different level of overlap is ideal for each target domain. we also find a positive trend occurs when reducing cross-domain sub-word overlap, suggesting that there is a trade-off depending on the target downstream task and domain. to better understand the results, we look at the impact of the pretraining data domain on downstream benchmark performance. surprisingly, we found that inclusion of the general domain with a specialized domain improves downstream performance for that specialized domain's tasks, but not the other way around. this suggests that specialized domains should be trained in tandem with a general one. our contribution is a pilot study that investigates a pretraining strategy to reduce cross-domain subword overlap between general and biomedical domains. we train cross-domain language models with varied vocabulary sizes and evaluate them on downstream classification tasks. we show that a significant improvement can be achieved on two benchmark datasets ( @xcite , @xcite ) when reducing overlap. further experiments point to the importance of selecting appropriate pretraining data for specialized domains. ## related work we discuss strategies from the literature to adapt the general domain language model, in particular a transformer @xcite encoder @xcite , to a specialized domain. ## datasets and tasks we use the combined english snapshot of wikipedia (a proxy for the general domain) and pubmed open access full-text corpora (biomedical domain) taken on the 1st of april 2020 for pretraining the language models and tokenizers. the pubmed corpus, consisting of 8.3 billion tokens, is preprocessed to remove references, while the wikipedia corpus, consisting of 2.0 billion tokens, is extracted and cleaned with wikiextractor (attardi, 2015). we use this pretraining data combination as a cross-domain proxy of the general and biomedical domain. we use the training and validation sets of the glue benchmark @xcite to fine-tune our models for general domain benchmarking. likewise, we use the publicly available subset of the blue tasks collection for the biomedical domain @xcite . ## experiments we perform pretraining with a cross-domain corpus with the albert model, which results in a high degree of cross-domain sub-word overlap. in addition, we experiment with models that have different vocabulary sizes (5000 to 100,000), each with a varying degree of sub-word overlap during pretraining. in transformer models, the embedding dimension is coupled with the model's hidden dimension, causing the vocabulary size to control the model size-a larger vocabulary size exponentially increases the model's size. to remedy this, we use the albert model @xcite , which projects the embedding dimension to a latent vocabulary dimension before projecting it to the model's hidden dimension. this projection allows scaling of the vocabulary size without significantly impacting the model's size. task performance and vocabulary size after pretraining, for each vocabulary size, we then evaluate our language models on downstream blue and glue benchmark datasets to determine how downstream performance is affected by the amount of sub-word overlap. determining sub-word overlap to determine the amount of sub-word overlap in relation to vocabulary size, we tokenize each general domain and biomedical task in glue and blue for each vocabulary size and compute the jaccard index (jaccard, 1912). the glue and blue tasks, are used as a cross-domain proxy between the general and biomedical domains. experimental setup for each model (vocabulary size |v |), we train a separate tokenizer using byte-pair encoding @xcite . we use masked language modeling to train the largest model, albert |@xmath0,000| , on the combined corpora of wikipedia and pubmed for two weeks using four v100 gpus with an effective batch size of 256. we use the lamb @xcite optimizer and a maximum model sequence length of 512. all other hyperparameters are left as default, as described by @xcite . for each model, we select the checkpoint such that validation performance (perplexity) is equal for all models. we then evaluate each model on both general domain and biomedical benchmark tasks. specifically, we fine-tune each model for a maximum of 15 epochs for all the biomedical tasks, taking the best model on the validation set for inference over the test set. for the general domain tasks, to reduce overfitting (false convergence), we train each task for five epochs and report the validation performance as the test set labels are not publicly available. however, scaling vocabulary size itself can lead to performance increases @xcite . hence, we use the checkpoint where validation performance for masked language modeling is equal across all models; meaning that all models have similar capacity for language modeling with the only difference being vocabulary size during downstream updates via fine-tuning. the increase in parameter count due to vocabulary embeddings is negligible as the embeddings are all projected into the same sized latent dimension before being used by the model. the classification layer used is created for each individual task and is not shared by any model. we use the default classification layer, with the correct label output layer as provided by the huggingface library @xcite . ## results and discussion we trained masked language models of varying vocabulary sizes, each with its own degree of subword overlap and evaluate on downstream general and biomedical language understanding benchmarks. we found that cross-domain sub-word overlap reduction benefited the cross-domain between the general ( in terms of sub-word overlap, we find that the jaccard index decreases sharply with vocabulary size ( we also report the sub-word overlap proportional to vocabulary size ( blue tasks seem to benefit from a larger separation of vocabularies, as suggested by an improved f1-score with increased vocabulary size (|v |) in figure 1 . however, this benefit is less significant for glue tasks, as validation model selection (used in blue) could not be applied. we find that glue results are worse when using combined (pubmed+wiki) rather than individual pretraining corpora (see we observe that inference tasks fared better with a larger vocabulary ( ## limitations this study only considers the biomedical and general domains; we hypothesize these principles can be applied to other domains, such as multilingual machine translation. one particular observation relevant to our setup is that the general domain corpus is smaller than that of the target domain, which should also be considered when extrapolating our findings. another limitation is that training of the language models was not performed to completion. however, language modeling effectiveness was fixed for a fair comparison. these limitations will be explored in future work. we are also aware that the fixed perplexity does not fully disentangle the impacts of vocabulary overlap and vocabulary size on the downstream effectiveness. we plan to extend our study with further experiments to ensure the robustness of results presented here. ## conclusions when applying general domain transformer language models to specialized ones, the use of sub-word modeling results causes sub-word overlap leading to decreased performance. we showed that increasing the vocabulary size of the model alleviates this performance penalty and improves downstream task performance on general and biomedical benchmarks. furthermore, we show that specialized domains improve significantly from a combination of specialized and general domain pretraining data. our work is a pilot study into improving downstream performance on specialized domains with potential application in crossdomain tasks. in the future, we would extend this study to other applications such as machine translation and cross-lingual language modeling.