Boundary Smoothing for Named Entity Recognition (2022)

Abstract: Neural named entity recognition (NER) models may easily encounter the over-confidence issue, which degrades the performance and calibration. Inspired by label smoothing and driven by the ambiguity of boundary annotation in NER engineering, we propose boundary smoothing as a regularization technique for span-based neural NER models. It re-assigns entity probabilities from annotated spans to the surrounding ones. Built on a simple but strong baseline, our model achieves results better than or competitive with previous state-of-the-art systems on eight well-known NER benchmarks. Further empirical analysis suggests that boundary smoothing effectively mitigates over-confidence, improves model calibration, and brings flatter neural minima and more smoothed loss landscapes.

## introduction named entity recognition (ner) is one of the fundamental natural language processing (nlp) tasks with extensive investigations. as a common setting, an entity is regarded as correctly recognized only if its type and two boundaries exactly match the ground truth. the annotation of boundaries is more ambiguous, error-prone, and raises more inconsistencies than entity types. for example, the conll 2003 task contains four entity types (i.e., person, location, organization, miscellaneous), which are easy to distinguish between. however, the boundaries of a entity mention could be ambiguous, because of the "boundary words" (e.g., articles or modifiers). considerable efforts are required to specify the "gold standard practice" case by case. notation guidelines. 2 in addition, some studies have also reported that incorrect boundary is a major source of entity recognition error @xcite . recently, span-based models have gained much popularity in ner studies, and achieved state-ofthe-art (sota) results @xcite @xcite . this approach typically enumerates all candidate spans and classifies them into entity types (including a "non-entity" type); the annotated spans are scarce and assigned with full probability to be an entity, whereas all other spans are assigned with zero probability. this creates noticeable sharpness between the classification targets of adjacent spans, and may thus plague the trainability of neural networks. in addition, empirical evidence shows that these models easily encounter the over-confidence issue, i.e., the confidence of a predicted entity is much higher than its correctness probability. this is a manifestation of miscalibration @xcite . inspired by label smoothing @xcite , we propose boundary smoothing as a regularization technique for span-based neural ner models. by explicitly reallocating entity probabilities from annotated spans to the surrounding ones, boundary smoothing can effectively mitigate over-confidence, and result in consistently better performance. specifically, our baseline employs the contextualized embeddings from a pretrained transformer of a base size (768 hidden size, 12 layers), and the biaffine decoder proposed by @xcite . with boundary smoothing, our model outperforms previous sota on four english ner datasets @xcite @xcite and two chinese datasets (weibo ner and resume ner), and achieves competitive results on other two chinese datasets (ontonotes 4 and msra). such extensive experiments support the effectiveness and robustness of our proposed technique. in addition, we show that boundary smoothing can help the trained ner models to preserve calibration, such that the produced confidences can better represent the precision rate of a predicted entity. this corresponds to the effect of label smoothing on the image classification task @xcite . further, visualization results qualitatively suggest that boundary smoothing can lead to flatter solutions and more smoothed loss landscapes, which are typically associated with better generalization and trainability @xcite . ## related work named entity recognition the mainstream ner systems are designed to recognize flat entities and based on a sequence tagging framework. @xcite introduced the linear-chain conditional random field (crf) into neural networkbased sequence tagging models, which can explicitly encode the transition likelihoods between adjacent tags. many researchers followed this work, and employed lstm as the encoder. in addition, character-level representations are typically used for english tasks @xcite @xcite , whereas lexicon information is helpful for chinese ner @xcite @xcite . nested ner allows a token to belong to multiple entities, which conflicts with the plain sequence tagging framework. @xcite proposed to use stacked lstm-crfs to predict from inner to outer entities. @xcite concatenated the bilou tags for each token inside the nested entities, which allows the lstm-crf to work as for flat entities. @xcite reformulated nested ner as a machine reading comprehension task. @xcite proposed to recognize nested entities by the two-stage object detection method widely used in computer vision. recent years, a body of literature emerged on span-based models, which were compatible with both flat and nested entities, and achieved sota performance @xcite @xcite . these models typically enumerate all possible candidate text spans and then classify each span into entity types. in this work, the biaffine model @xcite ) is chosen and re-implemented with slight modifications as our baseline, because of its high performance and compatibility with boundary smoothing. in addition, pretrained language models, also known as contextualized embeddings, were also widely introduced to ner models, and significantly boosted the model performance @xcite . they are used in our baseline by default. @xcite proposed the label smoothing as a regularization technique to improve the accuracy of the inception networks on imagenet. by explicitly assigning a small probability to non-ground-truth labels, label smoothing can prevent the models from becoming too confident about the predictions, and thus improve generalization. it turned out to be a useful alternative to the standard cross entropy loss, and has been widely adopted to fight against the over-confidence @xcite @xcite , improve the model calibration @xcite , and denoise incorrect labels @xcite . ## conclusion in this study, we propose boundary smoothing as a regularization technique for span-based neural ner models. boundary smoothing re-assigns entity probabilities from annotated spans to the surrounding ones. it can be easily integrated into any span-based neural ner systems, but consistently bring improved performance. built on a simple but strong baseline (a base-sized pretrained language model followed by a bilstm layer, and the biaffine decoder), our model achieves sota results on eight well-known ner benchmarks, covering english and chinese, flat and nested ner tasks. in addition, experimental results show that boundary smoothing leads to less over-confidence, better model calibration, flatter neural minima and more smoothed loss landscapes. these properties plausibly explain the performance improvement. our findings shed light on the effects of smoothing regularization technique in the ner task. as discussed, boundary smoothing typically increases the overall f 1 score at the risk of a slight drop in the recall rate; hence, one may be careful to use it for recall-sensitive applications. future work will apply boundary smoothing to more variants of span-based ner models, and investigate its effect in a broader range of information extraction tasks.