Tri-Train: Automatic Pre-Fine Tuning between Pre-Training and Fine-Tuning for S ci NER (2020)

Abstract: The training process of scientific NER models is commonly performed in two steps: i) Pre-training a language model by self-supervised tasks on huge data and ii) fine-tune training with small labelled data. The success of the strategy depends on the relevance between the data domains and between the tasks. However, gaps are found in practice when the target domains are specific and small. We propose a novel framework to introduce a “pre-fine tuning” step between pre-training and fine-tuning. It constructs a corpus by selecting sentences from unlabeled documents that are the most relevant with the labelled training data. Instead of predicting tokens in random spans, the pre-fine tuning task is to predict tokens in entity candidates identified by text mining methods. Pre-fine tuning is automatic and light-weight because the corpus size can be much smaller than pre-training data to achieve a better performance. Experiments on seven benchmarks demonstrate the effectiveness.

## introduction in many scientific domains such as biomedicine and computer science (cs), named entity recognition (ner) is a fundamental information extraction task @xcite @xcite . like many other natural language processing (nlp) tasks, language modeling plays an essential role in learning to perform like a domain expert @xcite @xcite . two-step training process has been widely used in nlp research, especially for domain-specific ner. step 1: pre-train a language model by self-supervised task(s) such as masked token prediction and next sentence prediction on large-scale datasets of billions of tokens. step 2: fine-tune the neural model on a target task with a carefully labelled domain-specific dataset. here, the premises of the strategy's success are that i) the pre-training corpora and fine-tuning labelled data are domain-relevant; ii) the pre-training task(s) and fine-tuning task are also relevant. data relevance becomes an issue for scientific ner (sciner). language models like bert @xcite were pre-trained on general corpora such as english wikipedia and books corpus. therefore, scibert @xcite was developed on paper corpus (3.2b tokens) in cs and bio domains from semantic scholar. biobert @xcite added the pubmed central (pmc) full-text corpus (13.5b tokens). they both applied the same pre-training tasks as the regular bert. the pre-training cost week(s) on 8 tpus or v100 gpus to win over the general-domain bert. however, the data relevance gap was not fully fixed because the target data were often collected in a very specific and small research field such as brca1-related breast cancer and vision ai, while the pre-training corpora were too broad to adapt the model effectively to the target task. moreover, the pre-training aimed at predicting random masked tokens or tokens in random masked spans @xcite , which could be too weakly associated with the target task of entity recognition and typing. it is important to bridge the gaps of data and tasks for accurate sciner. in this work, we propose a novel framework called tri-train that introduces a training step between heavy pre-training (pt) and small-data fine tuning (ft). we name it "pre-fine tuning" (pft). the framework is illustrated in figure 1 . first, it constructs a corpus by selecting a set of sentences relevant with the labelled training data (the specific research field) from unlabeled auxiliary corpora (like those used for pre-training). the pft corpus is of a medium size compared to those in pt and ft (300k tokens). second, we optimize the pretrained model parameters on two tasks. instead of random masked tokens or tokens in random masked we evaluate our framework based on four kinds of bert models: regular bert, spanbert @xcite , biobert, and scibert. experiments on seven sciner benchmarks demonstrate the effectiveness of the proposed framework. ## conclusion in this work, we proposed a novel framework to introduce a "pre-fine tuning" step between pretraining and fine-tuning. it constructed a corpus by selecting sentences from unlabeled documents that were the most relevant with the labelled training data. instead of predicting tokens in random spans, the pre-fine tuning task was to predict tokens in entity candidates identified. pre-fine tuning was automatic and light-weight because the corpus size could be much smaller than pre-training data to achieve a better performance. experiments on seven benchmarks demonstrate the effectiveness. we further investigated settings of pre-fine tuning.