MINER : Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective (2022)

Abstract: NER model has achieved promising performance on standard NER benchmarks. However, recent studies show that previous approaches may over-rely on entity mention information, resulting in poor performance on out-of-vocabulary(OOV) entity recognition. In this work, we propose MINER, a novel NER learning framework, to remedy this issue from an information-theoretic perspective. The proposed approach contains two mutual information based training objectives: i) generalizing information maximization, which enhances representation via deep understanding of context and entity surface forms; ii) superfluous information minimization, which discourages representation from rotate memorizing entity names or exploiting biased cues in data. Experiments on various settings and datasets demonstrate that it achieves better performance in predicting OOV entities.

## introduction named entity recognition (ner) aims to identify and classify entity mentions from unstructured text, e.g., extracting location mention "berlin" from the sentence "berlin is wonderful in the winter". ner is a key component in information retrieval @xcite , question answering @xcite , dialog systems @xcite , etc. traditional ner models are feature-engineering and machine learning based @xcite @xcite . benefiting from the development of deep learning, neuralnetwork-based ner models have achieved stateof-the-art results on several public benchmarks @xcite @xcite @xcite . there are three classical strategies to alleviate the oov problem: external knowledge, oov word embedding, and contextualized embedding. the first one is to introduce additional features, e.g., entity lexicons @xcite , part-ofspeech tags @xcite , which alleviates the model's dependence on word embeddings. however, the external knowledge is not always easy to obtain. the second strategy is to get a better oov word embedding @xcite . the strategy is learning a static oov embedding representation, but not directly utilizing the context. last one is fine-tune pre-trained models, e.g., elmo @xcite , bert @xcite , which provide contextualized word representations. unfortunately, @xcite shows that the higher performance of pretrained models could be the results of learning the subword structure better. how do we make the model focus on contextual information to tackle the oov problem? motivated by the information bottleneck principle @xcite , we propose a novel learning framework -mutual information based named entity recognition (miner). the proposed method provides an information-theoretic perspective to the oov problem by training an encoder to minimize task-irrelevant nuisances while keeping predictive information. specifically, miner contains two mutual information based learning objectives: i) generalizing information maximization, which aims to maximize the mutual information between representations and well-generalizing features, i.e., context and entity surface forms; ii) superfluous information minimization, which prevents the model from rote memorizing the entity names or exploiting biased cues via eliminating entity name information. our codes 1 are publicly available. our main contributions are summarized as follows: ## background in this section, we highlight the information bottleneck principle. subsequently, the analysis of possible issues was provided when applying it to oov entity recognition. furthermore, we review related techniques in deriving our framework. information bottleneck (ib) principle originated in information theory, and provides a theoretical framework for analyzing deep neural networks. it formulates the goal of representation learning as an information trade-off between predictive power and representation compression. given the input dataset (x,y), it seeks to learn the internal representation z of some intermediate layers by: on the one hand, it has been shown that it is challenging to find a trade-off between high compression and high predictive power @xcite @xcite . when compressing task-irrelevant nuisances, however, useful information will inevitably be left out. on the other hand, it is unclear for the ib principle which parts of features are well-generalizing and which are not, as we usually train a classifier to solely maximize accuracy. consequently, neural networks tend to use any accessible signal to do so @xcite , which is referred to as a shortcut learning problem @xcite . for training sets with limited size, it may be easier for neural networks to memorize entity names rather than to classify them by context and common entity features @xcite . in section 4, we demonstrate how we extend ib to the ner task and address these issues. ## model architecture in recent years, ner systems have undergone a paradigm shift from sequence labeling, which formulates ner as a token-level tagging task @xcite @xcite , to span prediction (spanner), which regards ner as a span-level classification task @xcite @xcite . we choose spanner as base architecture for two reasons: overall, spanner consists of three major modules: token representation layer, span representation layer, and span classification layer. besides, our method inserts a bottleneck layer to the architecture for information optimization. ## mi-based objectives motivated by ib @xcite , we can subdivide i(x; z) into two components by using the chain rule of mutual information(mi): the first term determines how much information about y is accessible from z. while the second term, conditional mutual information term i(x; z|y ), denotes the information in z that is not predictive of y . for ner, which parts of the information retrieved from input are useful and which are redundant? from human intuition, text context should be the main predictive information for ner. for example, "the ceo of x resigned", the type of x in each of these contexts should always be "org". besides, entity mentions also provide much information for entity recognition. for example, nearly all person names capitalize the first letter and follow the "firstname lastname" or "lastname figure 1 : visualization of miner, where x 1 and x 2 share the same context and entity labels, while their entity words are different. z 1 and z 2 are compressed entity representations sampled by p(z 1 |x 1 ) and p(z 2 |x 2 ), respectively, which are implemented by information bottleneck(ib) layer. our method add two additional learning objectives to basic architecture. the first one is to maximize the mutual information, i.e., i(z 1 ; z 2 ), to enhance context information and entity surface form information of z 1 and z 2 . the second objective is to minimize the jensen-shannon divergence, representing an upper bound of i(x 1 ; z 1 |x 2 ), aiming to eliminate task-irrelevant nuisances. we convert the targets of eq. ( 6 ) into a form that is easier to solve via a contrastive strategy. specifically, consider x 1 and x 2 are two contrastive samples of similar context, and contains different entity mentions of the same entity category, i.e., s 1 and s 2 , respectively. assuming both x 1 and x 2 are both sufficient for inferring label y. the mutual information between x 1 and z 1 can be factorized to two parts. thus any representation z containing all information shared from both sentences would also contain the necessary label information, and sentencespecific information is superfluous. so eq. ( 6 ) can be approximated by eq. ( 7 ) by: the target of eq. ( 8 ) is defined as generaliz-ing information maximization. we proved that i(z 1 ; z 2 ) is a lower bound of i(z 1 ; x 2 )(proof could be found in appendix 7). @xcite was used as a lower bound on mi and can be used to approximate i(z 1 ; z 2 ). subsequently, it can be optimized by: the target of eq. ( 9 ) is defined as superfluous information minimization. to restrict this term, we can minimize an upper bound of i(x 1 ; z 1 |x 2 ) (proofs could be found in appendix 7) as follows: ## experiment in this section, we verify the performance of the proposed method on five oov datasets, and compared it with other methods. in addition, we tested the universality of the proposed method in various pre-trained models. ## conclusion based on the recent studies of ner, we analyze how to improve the oov entity recognition. in this work, we propose a novel and flexible learning framework -miner, to tackle oov entities recognition issue from an information-theoretic perspective. on the one hand, this method can enhance the context information of the output of the encoder. on the other hand, it can safely eliminate task-irrelevant nuisances and prevents the model from rote memorizing the entities. specifically, the proposed approach contains two mutual information based training objectives: generalizing information maximization, and superfluous information minimization. experiments on various datasets demonstrate that miner achieves much better performance in predicting out-of-vocabulary entities.