Auxiliary Learning for Named Entity Recognition with Multiple Auxiliary Biomedical Training Data (2022)

Abstract: Named entity recognition (NER) is one of the elemental technologies, which has been used for knowledge extraction from biomedical text. As one of the NER improvement approaches, multi-task learning that learns a model from multiple training data has been used. Among multi-task learning, an auxiliary learning method, which uses an auxiliary task for improving its target task, has shown higher NER performance than conventional multi-task learning for improving all the tasks simultaneously by using only one auxiliary task in the auxiliary learning. We propose Multiple Utilization of NER Corpora Helpful for Auxiliary BLESsing (MUNCH ABLES). MUNCHABLES utilizes multiple training datasets as auxiliary training data by the following methods; the first one is to finetune the NER model of the target task by sequentially performing auxiliary learning for each auxiliary training dataset, and the other is to use all training datasets in one auxiliary learning. We evaluate MUNCHABLES on eight biomedical-related domain NER tasks, where seven training datasets are used as auxiliary training data. The experiment results show that MUNCHABLES achieves higher accuracy than conventional multi-task learning methods on average while showing state-of-the-art accuracy.

## introduction named entity recognition (ner) is a fundamental natural language processing technology for extracting named entity (ne) and technical terms from input texts and has been put to practical use in various situations. for example, ner is used as one of the core technologies for structuring and accumulating information on interrelationships among chemical substances and physical properties of chemical substances, which are reported daily in papers and patents, to develop new materials and products. ner has been actively studied for a long time, and many ner methods have been proposed. in recent years, neural network (nn)-based methods have become dominant, and a bilstm-crf model (e.g., @xcite ), composed of two recurrent neural networks (rnns) and conditional random fields (crf), and a transformerbased model (e.g., @xcite ) have achieved high performance in ner. in addition, it has been reported that the performance of an ner model is improved by multi-task learning, which uses training data of a task different from the target task and simultaneously learns features from multiple ner training datasets @xcite @xcite @xcite . remarkably, @xcite have shown that, an ner in the biotechnology field (bioner) with an auxiliary learning method, which is a variant of multi-task learning, achieves higher performance in the target task, compared to a standard multi-task learning method. the auxiliary learning uses a task other than the target task as an auxiliary task for improving the target task performance, in contrast the standard multi-task learning learns models for multiple tasks to improve performance of the multiple tasks. we propose a new auxiliary learning paradigm that uses multiple ner datasets as auxiliary training data, multiple utilization of ner corpora helpful for auxiliary blessing (munch-ables), whereas existing auxiliary learning uses only one type of auxiliary training data. specifically, we propose two types of multi-auxiliary learning: the first one is to fine-tune the ner model of the target task by sequentially performing auxiliary learning for each auxiliary training dataset (munchables-stack model), and the other is to use all types of training data in single auxiliary learning. as for the latter, we propose two models: one is to concatenate all the multiple auxiliary training datasets and make a batch by randomly selecting data from the auxiliary training dataset (munchables-concatenation model), and the other is to change auxiliary training datasets every epoch (munchablesiteration model). we compare the proposed munchables models with standard multi-task learning and single auxiliary learning on eight chemical/ biomedical/ scientific domain ner tasks. as for our proposed models, seven training datasets are used as auxiliary training data in each task. the experiment results show that the f1-scores of the proposed models are higher than those of the baselines on average and ner performance can be improved by using multiple auxiliary training datasets. in addition, the proposed models achieve state-of-the-art performance in chemical/biomedical/scientific ner. ## existing multi-task learning this section describes existing multi-task learning methods which use training data of a different task other than the target task. we first outline the ner model used as the base model, and then describe an extension of the ner model to multitask learning, where multiple tasks are trained simultaneously. in this multi-task learning, the target task and the other tasks are treated equally. then, we explain an existing auxiliary learning model, which uses training data for a different task from the target task as auxiliary training data. ## munchables: multi-auxiliary learning an existing auxiliary learning method uses only one auxiliary training dataset. in this section, we propose a new auxiliary learning paradigm, multi-auxiliary learning munch-ables, that utilizes multiple training datasets as auxiliary training data. we first propose two munchables models that use multiple auxiliary training datasets in single auxiliary learning (munchables-concatenation model and munchables-iteration model), and then propose a munchables model that sequentially fine-tunes a main model by auxiliary learning with each auxiliary training dataset (munchablesstack model). ## conclusion this paper proposed a new auxiliary learning paradigm for ner, munchables, that utilizes multiple training datasets as auxiliary training data for improving the performance of its target task. the experiments on eight chemical/biomedical/scientific domain ner datasets, showed that our proposed models achieved higher performance on average than conventional multitask learning methods and an auxiliary learning method using only one auxiliary training dataset. moreover, our proposed models achieved the state-of-the-art performance on chemical/biomedical/scientific ner tasks.