Automatic Document Selection for Efficient Encoder Pretraining (2022)

Abstract: Building pretrained language models is considered expensive and data-intensive, but must we increase dataset size to achieve better performance? We propose an alternative to larger training sets by automatically identifying smaller yet domain-representative subsets. We extend Cynical Data Selection, a statistical sentence scoring method that conditions on a representative target domain corpus. As an example, we treat the OntoNotes corpus as a target domain and pretrain a RoBERTa-like encoder from a cynically selected subset of the Pile. On both perplexity and across several downstream tasks in the target domain, it consistently outperforms random selection with 20x less data, 3x fewer training iterations, and 2x less estimated cloud compute cost, validating the recipe of automatic document selection for LM pretraining.

## introduction large pretrained language models have achieved state-of-the-art performance in nlp tasks @xcite . these studies find that increasing pretraining data size usually leads to better task performance. for many tasks, additional task (in-domain) data helps improve the performance further @xcite @xcite . several studies have found that directly pretraining on task data is more effective : science texts @xcite , tweets @xcite , legal texts @xcite or code @xcite . notably, these domains are known a priori, and identifying data sources for curation is straightforward. in other instances where the domain is less clear, like "offensive online content" @xcite , more complicated data sampling is employed to guess at the desired data distribution suitable for training a downstream classifier. to address such scenarios, we propose automatically identifying relevant domain-specific training data for a large corpus and subsequently pretraining a model on the selected data. specifically, we use cynical data selection @xcite , an approach that advanced moore-lewis sampling @xcite , to select data from the pile dataset @xcite . this automatic selection method can include possibly overlooked yet relevant documents from domains that may not be too close to the target domain. figure 1 illustrates this method which achieves higher performance on tasks in the target domain by using only 2.5gb (0.5%) of cynically selected data. specifically, we experiment with pretraining encoders with varying amounts of data sampled from the pile. foot_0 with our "target corpus" of ontonotes @xcite , we compare language models trained with cynical and random selection at various data levels. we find that the cynically selected encoder achieves consistently lower target corpus perplexity than one trained with random selection. we further finetune the encoders on a suite of tasks, some of which are derived from ontonotes. again, we find that models pretrained with cynical selection perform best. we suggest this as a viable method for inexpensively pretraining effective domain-specific encoders. ## cynical data selection methods for data selection for language-related tasks have been widely studied, usually to select in-domain data @xcite @xcite . one such method is cynical data selection @xcite . the intuition behind cynical selection is greedily ranking sentences from the text corpus, based on its score computed against text representative of the target domain, which is based on how much information gained by selecting it. concretely, given representative text from the target domain, cynical selection uses the crossentropy of the selected text against the representative text and calculates the information gain of each sentence in the general corpus. it then picks the most useful sentence relative to what has already been selected and its similarity to the representative text. this also leads to a bias towards shorter sentences and preferring sentences that contain words with high probability in the representative text. our work extends the cynical selection to the document level selection. sentences are still scored at the sentence level, but the average sentence-level gain determines the information gain of a document. 2 we demonstrate its advantages in efficiently selecting related documents to the target domain. ## experiments and results in this work, we set ontonotes 5.0 @xcite as our target corpus, and we use a smaller sample from the training corpus of the conll 2012 shared task @xcite as the representative corpus for data selection. we first train an encoder based on the selected data and use the edge probing suite @xcite for the downstream task evaluation, which has previously been used to probe and evaluate language models @xcite @xcite . ## related work due to the huge computational cost of training large models, both researchers and engineers have sought alternatives to using data more efficiently. some prior works use statistical methods to select relevant data from a large corpus @xcite @xcite . some other studies introduce additional classifiers or language models to help the data selection @xcite @xcite . also, data selection could be more efficiently involved in the active learning approaches @xcite @xcite @xcite . this work applies a simple statistical method to find the most related text to a target domain. it incrementally constructs a dataset out of a large corpus for the goal of training language models. ## conclusion this work builds the connection from corpus subselection in statistical lm construction to neural 9525 lms. we extend cynical data selection to efficiently select task-related documents for encoder pretraining and achieve lower perplexity in the target domain. we also demonstrate its effectiveness on downstream tasks by achieving comparable or even better results with 20x less data, 3x fewer training iterations, and 2x less computational cost on 8 edge probing tasks. we believe this fills the gap in the literature on an important topic in training powerful lms. we purposefully keep this work in the space of methods used in the days of stat nlp to highlight their out-of-the-box applicability, for which that line of research is still salient. based on our findings, this line is resurrected, suggesting new novel approaches should be studied. we anticipate that with this connection, researchers could explore this topic, investigate various subselection methods, and extend it to other domains.