PDALN : Progressive Domain Adaptation over a Pre-trained Model for Low-Resource Cross-Domain Named Entity Recognition (2021)

Abstract: Cross-domain Named Entity Recognition (NER) transfers the NER knowledge from high-resource domains to the low-resource target domain. Due to limited labeled resources and domain shift, cross-domain NER is a challenging task. To address these challenges, we propose a progressive domain adaptation Knowledge Distillation (KD) approach – PDALN. It achieves superior domain adaptability by employing three components: (1) Adaptive data augmentation techniques, which alleviate cross-domain gap and label sparsity simultaneously; (2) Multi-level Domain invariant features, derived from a multi-grained MMD (Maximum Mean Discrepancy) approach, to enable knowledge transfer across domains; (3) Advanced KD schema, which progressively enables powerful pre-trained language models to perform domain adaptation. Extensive experiments on four benchmarks show that PDALN can effectively adapt high-resource domains to low-resource target domains, even if they are diverse in terms and writing styles. Comparison with other baselines indicates the state-of-the-art performance of PDALN.

## introduction named entity recognition (ner) is typically framed as a sequence labeling task that targets to locate and classify named entities in text into predefined semantic types, such as person, organization, location, etc. ner is a fundamental task in information extraction @xcite and text understanding @xcite . the effectiveness of most existing ner models depends on sufficient labeled data, which is time-consuming and labor-intensive. current research proposes cross-domain ner, which enables ner on the low-resource target domain by transferring knowledge from other high-resource source domains. however, it is challenging to build a crossdomain ner component with high precision and recall, due to the domain shift problem @xcite . when casting the cross-domain ner as a transfer learning problem, most solutions (he and @xcite @xcite @xcite require high-quality cross-domain features for knowledge transfer. limited labeled data prohibit transfer learning from extracting informative features. besides, it is hard to find a single training dataset covering all the required ner types. even if words overlap across domains, their combination or usage is different from each other. domain adaptation @xcite is widely studied to solve the domain shift issue. existing approaches mainly introduce either word-level or discourse-level domain adaptations to enable crossdomain ner. to mitigate the word-level discrepancy, previous endeavors propose distributed word embedding @xcite , label-aware maximum mean discrepancy estimation @xcite , and projecting learning @xcite . as to the discourse-level discrepancy, existing approaches introduce multi-level adaptation layers @xcite , tensor decomposition @xcite , and multi-task learning with external information @xcite . however, those methods require sufficient labeled data, which hinders their performances under low-resource scenarios. to tackle both label sparsity and domain shift problem, existing approaches @xcite @xcite exploit external resources to generate pseudo labels for the low-resource domain. nevertheless, the less confident labels may deteriorate the robustness of models because of noise. in this paper, we propose a progressive domain adaptation cross-domain ner model pdaln. it introduces a novel domain adaptation component, which is enhanced by a progressive kd framework. pdaln addresses both word-and discourse-level domain adaptation on two low-resource scenarios: unsupervised and semi-supervised cross-domain ner. we first augment mix-domain training data by cross-domain anchor pairs, which alleviates the sparsity of annotated target domain. next, we enable knowledge transfer across domains through domain invariant features learned from a multigrained mmd adaptation metric. additionally, we fuse contrastive learning @xcite with a pre-trained model to extract robust features. finally, instead of directly fine-tuning the model on the augmented adaptive data under the mmd-based metric, we integrate the cross-domain ner model into a sequential kd framework to learn a lowcapacity student model. the low-capacity student can avoid over-fitting on limited annotated data because it progressively only cares about general cross-domain features retrieved by its sequential teachers to increase model confidence over domain invariant features. our main contributions are summarized as follows: ner is typically formulated as a sequence labeling task. based on the bio schema 1 , ner is to assign a sequence of labels @xmath0, ..., y n ] to a given sentence @xmath1, ..., x n ] with n tokens. an entity is a span of tokens @xmath2, ...x j ](1 ≤ @xmath3. 1 https://en.wikipedia.org/wiki/inside-outsidebeginning_(tagging) in unsupervised ner domain adaptation, we are given source domain {(x s m , y s m )} ns @xmath4, and target domain data {x t m } nt @xmath5. the source domain and target domain are characterized by probability distributions p s and p t , respectively. we aim to construct a model which can learn transferable features to bridge the cross-domain discrepancy, and build a classifier @xmath6; θ) which can minimize target prediction error using source supervision. for low-resource cross-domain ner, it is a semi-supervised adaptation where the target has a few labeled examples. we denote the source domain data d @xmath7, y s m )} ns @xmath8, unannotated target data d @xmath9, and annotated target data d @xmath10, y ta j )} na @xmath11. d @xmath12. ## preliminary 3.1 base model to obtain expressive sentence features, we adopt a pre-trained language model (e.g. bert @xcite ) to encode the sentence @xmath13, x 1 , ..., x n , x sep ] (after padding tokens in bert) into sentence representation @xmath14, h 1 , ..., h n , h sep ]. the task objective is denote as crf loss, where l @xmath15. ## the proposed model in this section, we present the structure of the proposed model. we first introduce domain adaptation components. on the one hand, we design an adaptive data augmentation to tackle the label sparsity issue. on the other hand, we introduce a multigrained mmd metric on the augmented adaptive data to extract domain invariant features. there is an intuitive illustration in figure 1 to show how our domain adaption approach mitigates the domain shifting. besides, we exploit the power of the pretrained model to capture expressive data features. we integrate a sequential self-training strategy to progressively and effectively perform our domain adaption components, as shown in figure 2 . we describe the details of cross-domain adaptation in section 4.1 and progressive self-training for lowresource domain adaptation in section 4.2. ## experiments in this section, we evaluate pdaln and other baselines on four public benchmarks. we conduct two groups of comparison experiments for unsupervised and semi-supervised cross-domain ner separately. we also conduct further ablation studies and hyperparameter studies to validate the efficacy of the domain adaptation approaches. ## related work recently, label sparsity has achieved great success in many research frontiers @xcite @xcite @xcite @xcite . one of the widely adopted strategies is a cross-domain transfer which mainly deals with the domain shift problem. the causes for domain shift in ner are mainly twofold including the discrepancies of word distributions and sentence patterns between source and target domain. on the one hand, word distributions are not compatible between different domain datasets. therefore, existing works equip the model with diverse domain adaptation components to alleviate domain shift. @xcite propose distributed word embedding methods to leverage domain-specific knowledge to boost their cross-domain ner performance. @xcite introduce a label-aware mechanism into maximum mean discrepancy (mmd) to explicitly reduce domain shift between the same labels across domains in medical data. lin and lu (2018) employ projecting learning to obtain a transfer matrix that maps target domain words into the word space of the source domain. on the other hand, diverse sentence patterns are usually caused by various factors, like written styles, publication categories, data quality, etc. the solutions for mitigating the discourse-level discrepancy mainly include multi-level adaptation layers (lin and lu, 2018), tensor decomposition @xcite and multi-task learning with external information @xcite . as we mentioned before, lin and lu (2018) construct the word adaptation component in their model. besides, they construct another sentence-adaptation layer, which takes in the adapted word embedding to extract another adaptation sentence feature. @xcite use multi-task learning and tensor decomposition to extract latent factors. through latent factors, knowledge can be transferred across source and target domains. @xcite employ ner label experts to guide model learning between domains. the label-aware guidance layer is key to enable domain adaptation. jia and zhang (2020) a multi-cell compositional lstm structure for cross-domain ner under the multi-task learning strategy. besides, those @xcite @xcite exploit external resources to generate pseudo labels for the low-resource domain with the assistance of a pretrained language model. however, those methods either lack the capability to capture expressive text features for the adaptation or require sufficient labeled target data, which impedes their performances under both zeroresource and minimal-resource scenarios. for the pre-trained model assisted approaches mainly rely on external knwledge bases which introduces too much noise. ## conclusion in this paper, we propose a progressive adaptation knowledge distillation framework, including anchor-guided adaptive data to address data sparsity, multi-grained mmd to bridge the domain adaptation, and progressive kd to stably distill cross-domain knowledge. the results exhibit the model's superiority over the most state-of-the-arts.