Multi-Domain Named Entity Recognition with Genre-Aware and Agnostic Inference (2020)

Abstract: Named entity recognition is a key component of many text processing pipelines and it is thus essential for this component to be robust to different types of input. However, domain transfer of NER models with data from multiple genres has not been widely studied. To this end, we conduct NER experiments in three predictive setups on data from: a) multiple domains; b) multiple domains where the genre label is unknown at inference time; c) domains not encountered in training. We introduce a new architecture tailored to this task by using shared and private domain parameters and multi-task learning. This consistently outperforms all other baseline and competitive methods on all three experimental setups, with differences ranging between +1.95 to +3.11 average F1 across multiple genres when compared to standard approaches. These results illustrate the challenges that need to be taken into account when building real-world NLP applications that are robust to various types of text and the methods that can help, at least partially, alleviate these issues.

## introduction accurately identifying named entities and their type in texts is a key processing step for many nlp applications. named entity recognition (ner) is an important component in several tasks including named entity linking @xcite , co-reference resolution @xcite , question answering @xcite , relation extraction @xcite and usually sits upstream of analytics such as sentiment @xcite or stance @xcite . building robust ner models to accurately tag and adapt to heterogeneous types of text is thus paramount. recent research focused on * *equal contribution improving the overall performance of ner models on specific data sets. yet ner models show relatively high variance even when trained on the same data @xcite and poorly generalize when tested on data from different genres foot_0 , especially if these contain entity mentions unseen in the test data @xcite . despite this, research on ner models robust to different types of input is usually limited to the standard domain adaptation scenario: a single source domain rich in training data and a single target domain with limited or no training data @xcite . we argue that this is an over-simplified experimental setup that is not typical for how ner models are used in real-world applications. ideally, ner models use all available data, regardless of genre, and perform inference on data from any genre, even if this was not encountered in training. in this scenario, simply pooling all the available data is likely sub-optimal as genre-specific differences in named entity mentions are useful to model. conversely, models limited to only data from the same genre as the test set are likely to underperform, as using more data is usually beneficial. this work introduces three experimental setups for the ner task where models are trained on data from multiple genres and evaluated as follows: a) multi-domain -evaluation is performed across multiple genres, all seen in training. b) multi-domain with unknown domain labels -evaluation is carried out across multiple genres, all seen in training, but the genre label for each document is unknown at inference time. c) zero-shot domain -evaluation is performed on documents from genres unseen in training. we propose a neural architecture for ner tailored to these three experimental setups, based on the popular bilstm-crf architecture @xcite . we augment the base architecture to learn both domain-specific and independent features through shared and private domain components including projections and crfs. further, we add a multi-task learning objective for domain prediction to guide this separation. this model can perform inference on a text without knowledge of its corresponding domain label by using the shared components. we compare this model with several competitive methods that use a similar base architecture while holding the embeddings constant (i.e. glove embeddings). these include models trained on data from each domain independently, models that pool all data and models that use domain identities as features through to source-target domain adaptation methods. extensive results on all three experimental setups on a collection of data from a total of twelve genres demonstrate that our proposed architecture outperforms all others by a respectable margin. finally, through an error analysis of our results, we aim to understand the contributions of each proposed component and the margins for future improvements. domain adaptation for ner models for domain adaptation in ner using neural architectures were studied recently, albeit mostly for covering the single-source and single-target setup. the init method trains a model using the source domain data, and its parameters are used to initialize a target model which is fine-tuned on the target data @xcite . the mult method trains jointly one model for each domain with shared parameters @xcite . for sequence tagging, one crf for each of the two domains is used to obtain the predictions @xcite . adaptation can also be made at the embeddings stage (lin and lu, 2018) or by using additional unlabeled data from the source domain and out-of-domain annotated data @xcite . however, as mentioned above, this assumes that unlabeled training data can be provided for each domain, which may not be realistic. the model adds layers between embeddings and the bilstm layers, between the bil-stm and the crf for the target domain and separate crf layers, the latter two of which we adapt to our proposed architecture for multi-domain adaptation. a hierarchical bayesian prior approach is used in @xcite to tie feature weights across domains when information is sparse and also allow the model to take advantage if substantial data is available in one domain. their experiments on ner focused only on three data sets: conll, muc-6 and muc-7 and only the first of our three setups. a multi-task domain adaptation method for ner and word segmentation is used in @xcite . the proposed architecture learns a shared representation across domains and experiments with linear domain projections for each domain to guide learning of shared representations. the output of these linear layers is fed to a crf. we adopt the linear domain projection method, but extend this to also include a shared projection, followed by domain-specific crfs and multi-task learning. finally, another type of domain adaptation is temporal adaptation of models tested on data that is more recent than the training data, when each temporal slice can be considered as a different domain @xcite . ## methods this section describes the proposed ner architecture tailored the architecture to our multi-domain experimental setups, which is independent of input embedding representation. ## results in this section, we present and compare the results of all the methods introduced previously. experiments are conducted first on the open data collection introduced in section 4.1 in the multi-domain and multi-domain with unknown label setups. following, we evaluate the performance of our model on the data used for zero-shot genre ner. the goal of these experiments is to examine the ner performance across the three proposed experimental setups which focus on model generalizability across multiple domains. we note that the results below can not be directly compared to the state-of-the-art results on each data set, as we restrict the entity types to per, org, loc, such that these types are constant across all data sets. ## runtime comparison finally, we compare the runtime difference across various methods listed in the experiment section to test the practical implications of using our pro- in addition to inference runtime, we also find that the training time is not significantly more than the combined training time of n in-domain models. the main additions are that of the shared layers and the auxiliary task to the components of the n in-domain models and is thus a constant addition in the number of parameters to the total of n indomain models. hence, the model would scale by a constant with respect to the number of input domains (n+1 number of components, where n is the number of domains). this should allow our pro-posed model to scale to a large number of domains. this highlights that the proposed multdomain-sp-aux model is a viable option for real-world applications. ## conclusions robustness of nlp models is essential to their wider adoption and usability. existing ner approaches are widely faced with limited scalability when applied to data that spans multiple domains. this paper introduced three experimental setups that provide a framework for evaluating the robustness of ner models. these include learning from data in multiple domains and testing on all domains, when the domain label of the test point is unknown and when this does not belong to a domain seen in training. building on past research, we proposed a new neural architecture that achieves substantial improvements of up to 5 f1 points when compared to standard methods. future work will focus on domain adaptation at the embedding layer.