Multi-Cell Compositional LSTM for NER Domain Adaptation (2020)

Abstract: Cross-domain NER is a challenging yet practical problem. Entity mentions can be highly different across domains. However, the correlations between entity types can be relatively more stable across domains. We investigate a multi-cell compositional LSTM structure for multi-task learning, modeling each entity type using a separate cell state. With the help of entity typed units, cross-domain knowledge transfer can be made in an entity type level. Theoretically, the resulting distinct feature distributions for each entity type make it more powerful for cross-domain transfer. Empirically, experiments on four few-shot and zero-shot datasets show our method significantly outperforms a series of multi-task learning methods and achieves the best results.

## introduction named entity recognition (ner) is a fundamental task in information extraction, providing necessary information for relation classification @xcite , event detection @xcite , sentiment classification @xcite , etc. ner is challenging because entity mentions are an open set and can be ambiguous in the context of a sentence. due to relatively high cost in manual labeling, cross-domain ner has received increasing research attention. recently, multi-task learning methods @xcite @xcite @xcite have achieved great success for cross-domain ner. other methods such as fine-tuning @xcite , share-private @xcite and knowledge distill @xcite also show effectivenesses for cross-domain ner. there are three main source of challenges in cross-domain ner. first, instances of the same type entities can be different across domains. for example, typical person names can include "trump" and "clinton" in the political news domain, but "james" and "trout" in the sports domain. second, different types of entities can exhibit different degrees of dissimilarities across domains. for example, a large number of location names are shared in the political news domain and the sports domain, such as "barcelona" and "los angeles", but the case is very different for organization names across these domains. third, even types of entities can be different across domains. for example, while disease names are a type of entities in the medical domain, it is not so in the biochemistry domain. we investigate a multi-cell compositional lstm structure to deal with the above challenges by separately and simultaneously considering the possibilities of all entity types for each word when processing a sentence. as shown in figure 1 , the main idea is to extend a standard lstm structure by using a separate lstm cell to model the state for each entity type in a recurrent step. intuitively, the model differs from the baseline lstm by simultaneously considering all possible entity types. a compositional cell (c cell) combines the entity typed cells (et cells) for the next recurrent state transition by calculating a weighted sum of each et cell, where the weight of each et cell corresponds to the probability of its corresponding entity type. different from naive parameter sharing on lstm @xcite , source domain and target domain in our multi-task learning framework share only the et cells corresponding to the same entity types and the same c cell, but not for the domain-specific et cells. in this way, our model learns domain-invariant in the entity level. intuitively, our model addresses the above challenges by modeling entity type sequences more explicity, which are relatively more robust across domains compared with entity instances. for example, the pattern "per o per o loc" can exist in both the political and sports domains, despite that the specific per instances can be different. in addition, thanks to the merging operation at each step, our method effectively encodes multiple entity type sequences in linear time by having a sausage shaped multi-cell lstm. thus it allows us to learn distributional differences between entity type chains across domains. this effectively reduces the confusions of different entities when source domain and target domain have different entity types in few-shot transfer, where the target domain has a few training data. in zero-shot transfer where the target domain has no training data, a target-domain lm transfers source-domain knowledge. this knowledge transfer is also in the entity level thanks to the compositional weights which are supervised by gold-standard entity type knowledge in source-domain training. theoretically, our method creates distinct feature distributions for each entity type across domains, which can give better transfer learning power compared to representation networks that do not explicitly differentiate entity types ( §3.4). empirically, experiments on four fewshot and zero-shot datasets show that our method gives significantly better results compared to standard bilstm baselines with the same numbers of parameters. in addition, we obtain the best resutls on four cross-domain ner datasets. the code is released at https://github.com/ jiachenwestlake/multi-cell_lstm . ## method given a sentence @xmath0, . . . , x m ], the vector representation w t for each word x t is the concatenation of its word embedding and the output of a character level cnn, following @xcite . a bi-directional lstm encoder is used to obtain sequence level features @xmath1, . . . , h m ]. we use the forward lstm component to explain the de-tails in the following subsections. finally, a crf layer outputs the label sequence @xmath2, . . . , l m . ## conclusion we have investigated a multi-cell compositional lstm structure for cross-domain ner under the multi-task learning strategy. theoretically, our method benefits from the distinct feature distributions for each entity type across domains. results on a range of cross-domain datasets show that multi-cell compositional lstm outperforms bil-stm under the multi-task learning strategy. as the common setting of transfer learning, we set 1>α ≥ 1 2 and then α 1-α ≥ 1, further obtaining: using these conclusions to the previous inequalities, we have: setting λ * = λ 1 + λ 2 , which is the shared error of ideal joint hypothesis and use the precondition, finally, we obtain the lemma 1: