Domain Specific Augmentations as Low Cost Teachers for Large Students (2022)

Abstract: Current neural network solutions in scientific document processing employ models pretrained on domain-specific corpi, which are usually limited in model size, as pretraining can be costly and limited by training resources. We introduce a framework that uses data augmentation from such domain-specific pretrained models to transfer domain specific knowledge to larger general pretrained models and improve performance on downstream tasks. Our method improves the performance of Named Entity Recognition in the astrophysical domain by more than 20% compared to domain-specific pretrained models finetuned to the target dataset.

## introduction scientific document processing (sdp) is an emerging field in natural language processing (nlp) that proves to have more obstacles than everyday text due to the extensive scientific jargon and long text spans. recent work in sdp on transformer architectures @xcite has placed emphasis on constructing pretrained models in scientific corpi, such as biobert @xcite , scibert @xcite , and as-trobert @xcite . however, such models are usually trained on the base size of its corresponding architectures, limiting the potential inference performances due to the smaller number of trainable parameters compared to the large-size models usually used in state-of-the-art (sota) performance for benchmarks in everyday text. are we able to achieve similar or better results with finetuning models larger in size whilst transferring knowledge from such pretrained scientific models to increase robustness? in this paper, we propose a training method inspired by the unsupervised data augmentation @xcite and the noisy student @xcite framework. we first augment the training data with model that is trained on a corpus that is more closely aligned with the context domain of the target dataset. we then train a larger model on both the original training data and the augmented training data, combining the computational availability of the larger model with the domain-specific trained knowledge of the smaller domain-pretrained model. we describe the shared task deal @xcite and its dataset in section 2 and briefly review the previous work we used in section 3. we detail our model architecture and methodology in section 4, and go through our experimental setup and results in section 5. finally, we go through an in depth discussion of our results in section 6 and conclude our findings in section 7. ## task description and dataset named entity recognition (ner) refers to the identification and recognition of entities from a string of text. although this task is well explored in everyday text in benchmarks such as @xcite and wnut2017 @xcite , the focus of scientific text is not prominently showcased in such work. even in benchmarks that focus on scientific document processing (sdp), the corpi in question often lie in the domain of biology and chemistry, such as the ncbi-disease @xcite and biocreative v cdr @xcite corpi, with a lack of evaluation and state-of-the-art models in the astrophysics domain. the shared task deal (detecting entities in the astrophysics literature; @xcite ) is a sequence labeling task that aims to increase the accuracy in named entity recognition in the domain of astrophysics. given the overlapping usage of historical names and acronyms in different types of astrophysical entities, it may be difficult to extract named entities in astrophysics purely by carefully constructed systematic rules. for exam-ple, maxwell may refer to either the physician james clerk maxwell, a crater on the far side of the moon, or a series of equations. deal aims not only to discern such entities, but also to discern between such different types of entities. the training dataset consists of 1,753 samples of text fragments from the text and acknowledgments of astrophysics papers provided by the nasa astrophysical data system (nasa ads; @xcite . for evaluation purposes, the labeled development dataset consists of 20 samples, while the unlabeled validation and test dataset consists of 1,366 and 2,505 samples, respectively. we evaluate the performances based on the seqeval @xcite f1 score at the entity level and matthew's correlation coefficient @xcite at the token level in the validation and test dataset. ## literature review we briefly review some previous work that are utilized in our proposed system. ## architecture and methodology we propose a system that uses data augmentation as a low-cost method of teacher-student training to transfer domain-specific knowledge to a larger adapter-based model. ## experiments we describe the experimental setup and the results in this section. ## analysis we now present a more detailed analysis of the performance of different variants of our model and some considerations between experimental setups. ## conclusion in this paper, we show that we are able to surpass models pretrained on domain-specific knowledge by utilizing general corpus pretrained adapter models of larger sizes. furthermore, such a method can by used in conjunction to the aforementioned domain-specific pretrained models via data augmentation to transfer such knowledge to the backbone model. further work may explore other methods of augmentation to act as teacher models or combining multiple augmentations in training.