Reconciliation of Pre-trained Models and Prototypical Neural Networks in Few-shot Named Entity Recognition (2022)

Abstract: Incorporating large-scale pre-trained models with the prototypical neural networks is a de-facto paradigm in few-shot named entity recognition. Existing methods, unfortunately, are not aware of the fact that embeddings from pre-trained models contain a prominently large amount of information regarding word frequencies, biasing prototypical neural networks against learning word entities. This discrepancy constrains the two models’ synergy. Thus, we propose a one-line-code normalization method to reconcile such a mismatch with empirical and theoretical grounds. Our experiments based on nine benchmark datasets show the superiority of our method over the counterpart models and are comparable to the state-of-the-art methods. In addition to the model enhancement, our work also provides an analytical viewpoint for addressing the general problems in few-shot name entity recognition or other tasks that rely on pre-trained models or prototypical neural networks.

## introduction named entity recognition (ner) is a classical task in natural language processing (nlp) which aims to automatically identify entities in the plain text by classifying each word to a set of pre-defined entities, e.g. "person/location", or to the "others" (no-entity) @xcite . as a crucial sub-component of many language understanding tasks, ner has been widely adopted to different applications, e.g. news @xcite and the medical @xcite . neural networks (nns) have achieved great success in ner @xcite . however, nns face the adaptation challenge @xcite as words in different entities can change to a great extent @xcite , e.g. "mr. bush" in the "person" v.s. "budgets" in the "money", and obtaining sufficient annotations of new entities can be expensive @xcite . few-shot ner, a cost-efficient solution, aims at training a model to be aware of unseen entities given few labeled examples @xcite . few-shot ner has received a rising interest in the nlp community, where new datasets @xcite and methods @xcite @xcite have been constantly proposed. low-dim manifold encodes more adaptive information @xcite . prototypical neural networks (pnns) @xcite learn an embedding space where the same-entity datapoints are clustered around a center, called the prototype, and distances between the query data to all prototypes represent its entity probabilities. in addition to using an embedding network, pnns calculate the prototypes and distances via a non-parameteric algorithm, gaining popularity for the flexibility and low computing cost @xcite . a supplementary enhancement will be using embeddings from large-scale pre-trained models (ptms), like bert @xcite , to provide extra knowledge that helps pnns' learning of entities. as such, incorporating ptms with pnns has become a de-facto paradigm for few-shot ner that achieves competitive results to state-of-the-arts @xcite @xcite . related works consider ner-specific properties @xcite or new learning algorithms @xcite to enhance the model, but they tend not to examine the coordinating effects between ptms and pnns in terms of the information contained in embeddings. it should be reminded that pnns calculate distances between word embeddings and prototypes to represent entity probabilities. however, ptms embeddings may not effectively provide entity information as they prominently contain information on word frequencies @xcite , and we find frequencies are shallow statistics that can cause a loss of in-depth and useful entity-denoting information. by probing into pnns, we find that words tend to be classified to the entity centred with words of higher frequencies. therefore, the distance measure is biased towards focusing on frequencies. such a bias can cause the over-fitting of the pnns and the unreliability on classifying new entities. as a consequence, when frequencies are changed on a new corpus, the distances can no longer effectively represent the entity probabilities. form a mathematical view, the biased distance is mainly caused by the varying prototype ℓ2-norms. however, we argue that those ℓ2-norms contribute little to but actually undermine the correct classification. we propose to normalize all prototypes to unit vectors as a simple yet effective remedy to reconcile pnns and ptms for few-shot ner. our experiments on nine few-shot ner datasets @xcite demonstrate the effectiveness of our one-line-code remedy. the normalized pnns achieve competitive results compared to the state-of-the-art methods while retaining all the pnns' advantages, such as easy implementation and low computation cost. we also demonstrate normalization can make pnns learn more effectively about correctly classifying entities, and conduct ablation studies on different normalization strategies. our study on reconciling ptms and pnns, and the promising performance of the simple normalization method may inspire new research motivations to the few-shot ner, as well as other fields that involve the use of ptms /or pnns. ## distance in prototypical neural networks in this section, we describe pnns' feed-forward propagation from the mathematical viewpoint focusing on the pnns' distance function. in k-way n -shot, let s k denote the small support set containing n labeled examples with the class k. pnns calculate the prototype of each class through meanaggregating the embedded support examples: theorem 1. assume data embeddings of the support and query set are independent and identically distributed. let c k be the class prototype calculated by an aggregation function proto(•) : n @xmath0, the problem: min proto(•) j , where j is the classification loss, achieves minimization given by proto(•) being the arithmetic mean. corollary 1.1. based on the support set, pnns estimate a gaussian distribution n k (c k , σ foot_0 ) for the embeddings in the class k (σ is a constant vector). the corresponding choice of the bregman divergence d should be the squared euclidean distance. proofs are provided in the appendix b. while d is proposed to be any bregman divergence @xcite , we prove the optimal distance function should be the squared euclidean distance: ∥zz ′ ∥ 2 . 2 pnns consider that distances between embeddings and prototypes represent the entity probabilities. therefore, we count on the distance to capture the sharing entity information between the word and the prototypes. factorizing the distance (-∥f ϕ (x)c k ∥ 2 ) to (-∥f ϕ (x)∥ 2 + 2f ϕ (x) t c k -∥c k ∥ 2 ), the entity probabilities are not only proportional to the dot production, but are also reversely proportional to the two ℓ2-norms. while ∥f ϕ (x)∥ 2 represents query data information, different ∥c k ∥ 2 implies part of the probabilities are priorly determined, and the word is more likely to be classified to the entity that has the smaller prototype ℓ2-norm. unfortunately, because of the representation degeneration of ptms, these priorly determined probabilities tend to introduce non-entity information, and bias the pnns' distance towards frequencies. ## representation degeneration of pre-trained models in this section, we introduce the concept of representation degeneration in ptms and explain its associated effects to pnns. small-scale ptms, like glove @xcite and word2vec @xcite , are argued by researchers as low-capacity models for representing the richness in semantics of natural languages @xcite . both theoretical @xcite and empirical @xcite results in literature have proven: the learned word embeddings contain substantial non-semantic statistics information, i.e. the frequencies of the words, causing a lower performance on various downstream tasks, like the task of word classification @xcite . recent transformer @xcite -based large-scale ptms @xcite are groundbreaking in modeling natural language. however, we are concerned that the learned embeddings might also contain the information regarding the non-semantic word frequencies. in line with @xcite , we use the online statistics data foot_1 , get the word embeddings from bert and roberta, and do principal component analysis (pca) to extract the first two coefficients, and plot them on point diagrams. figure 2 displays the result. the results show both models' embeddings have correlation to frequencies. in addition, @xcite finds in the embedding space, word embedding ℓ2-norms are inversely proportional to their frequencies. as in pnns, the prototypes are the mean-aggregation of the words in the support set. therefore, the prototype ℓ2-norms are also correlated to word frequencies as well as the priorly determined probabilities we find in section 3. however, we hypothesize that word frequencies are shallow statistics that are irrelevant to word entities, and the priorly determined probabilities represent little entity information. word frequencies ratio word frequencies ratio figure 4: a histogram displaying the word frequencies in two entities. ## distance bias of prototypical neural networks in section 3, we have shown that pnns have a priori on the distances between word embeddings and different entities: embeddings are more likely to be close to the entity that has a smaller prototype ℓ2norm, and the word is more likely to be classified to that entity. however, in section 4, we argue this priori will introduce non-entity information that confuses the calculation of probabilities in pnns. we have shown frequencies and entities are irrelevant. in the following two figures, we further show the prototype ℓ2-norms vary in a manner that is also irrelevant to entities. figure 5 displays the average prototype l2-norms of all classes. the ℓ2-norms vary greatly among different classes (@xmath1.25, @xmath2.13, coefficient of @xmath3.202). in figure 6 , the blue column represents the largest class-prototype ℓ2-norm, the orange one the smallest and the green one the average. even within the same class, the prototypes ℓ2-norms demonstrate large variance due to the contrasting difference among episodes. 0 1 2 3 4 5 6 7 8 9 10111213141516171819 entity id 0 2 4 6 8 10 12 l2-norm values figure 6: max(blue)/avg.(green)/min(orange) prototypes ℓ2-norms within a same class. distances between prototypes and word embeddings should represent entity probabilities. unfor-tunately, with respect to the above problem, the distances in the original pnns are biased towards frequencies instead of being entity-oriented. as a result, pnns tend to overfit the training data and be trained with unreliable loss minimization. ## normalizing the prototypes in this section, we aim to provide a solution to the above-mentioned problems through a normalizing method. varying ℓ2-norms mainly causes frequency-biased distances and the above two problems. as a result, we consider normalizing the prototypes to ℓ2-norm-invariant vectors. earlier works in computer vision find normalizing both prototypes and the query data embeddings can achieve better and more stable results @xcite . however, we do not normalize the query data embeddings, because word embeddings represent more detail and other useful information that may be eliminated by the normalization. representing high-level entity information, prototypes should not be priorly distinguished from each other. furthermore, observing the following evidence, we argue that prototype ℓ2-norms have limited contribution to the correct classification. in both the bert's pre-training (1) and the original pnns (2), we find the ℓ2-norms of class features play limited roles to the correct classification. based on the above analysis, we propose to normalize the prototypes to unit vectors before calculating the class probabilities. algorithm 1 normalizing the prototypes *** pseudo-code in pytorch *** import torch.nn.functional as f @xmath4.normalize (c, @xmath5... the same as the original pnns ... connection to the adaptive loss: different data may associate with different difficulties to be classified. adaptive loss is proposed to be able to change dynamically in magnitude so as to capture the difficult ones @xcite @xcite . humans are prone to processing high-frequency words as reported in psychological studies @xcite . applying this psychological finding to the named entity recognition in natural language processing, we postulate that if a word appears more frequently, its entity should be easier to be classified. to this end, pnns well adapt to task difficulty through the frequencyrelated embedding ℓ2-norms of the query data. ## experiments & results to demonstrate the effectiveness of our normalized pnns, we conduct experiments on nine fewshot named entity recognition datasets proposed by @xcite and @xcite . datasets: being a classical and basic natural language understanding task, dozens of supervised ner datasets have been proposed, including wikigold @xcite @xcite , @xcite , mit movie @xcite , mit restaurant @xcite , snips @xcite , atis @xcite , multiwoz @xcite . based on these datasets, researchers @xcite restructure them to the "k-way n -shot" few-shot setting into a comprehensive few-shot ner benchmark. however, except for the formatting change of data, the simple and direct re-structuring shall lose track of some critical ner properties, such as the task-difficulty differences between the finegrained and coarse-grained entities @xcite . therefore, a new expert and challenging dataset has been proposed as a benchmark in fewshot ner @xcite . experimental settings: without special notations, we basically follow the original implementations in the two open sources 67 , including models, training/testing pipelines, hyper-parameters, and sampled episodes. we report results using the standard evaluation metrics: micro averaged f1 score. we re-run all the experiments of the origin pnns to examine the performance improvements by our normalization method based on the same hardware device. we add early stop constraints when reproducing results of @xcite ) and relocate the comparable results from the peer models @xcite . 8 all the experiments are conducted on a single 3090ti gpu. ## conclusion we examine the synergistic effects of the largescale ptms and the classical pnns in the few-shot ner. our theoretical analysis of pnns shows that pnns' distances that represent the query data's entity probabilities are partly priorily determined in terms of the prototype ℓ2-norms. however, on the embeddings of the ptms, we empirically verify that embedding ℓ2-norms contain little entity information, being a type of ptms' representation degeneration. furthermore, we show that such representation degeneration makes pnns' distance biased towards frequencies instead of entity-denoting. this distance bias prevents pnns from learning useful entity information and causes pnns to overfit the training corpus and become unreliable on new entities. therefore, we propose a one-linecode normalization remedy to reconcile ptms and pnns for few-shot ner. the experimental results based on nine datasets suggest that the normalized pnns proposed in this work achieve significant performance improvements over the original pnns and get competitive results compared with the latest sophisticated methods while maintaining pnns' all advantages, such as easy implementation and low computation cost. considering the promising results and the innovation in normalizing the existing models, our results and analysis may be an interest of reference study for researchers and practitioners working with few-shot ner or other relevant tasks that involve the use of ptms or pnns.