GNN - SL : Sequence Labeling Based on Nearest Examples via GNN (2023)

Abstract: To better handle long-tail cases in the sequence labeling (SL) task, in this work, we introduce graph neural networks sequence labeling (GNN-SL), which augments the vanilla SL model output with similar tagging examples retrieved from the whole training set. Since not all the retrieved tagging examples benefit the model prediction, we construct a heterogeneous graph, and leverage graph neural networks (GNNs) to transfer information between the retrieved tagging examples and the input word sequence. The augmented node which aggregates information from neighbors is used to do prediction. This strategy enables the model to directly acquire similar tagging examples and improves the general quality of predictions. We conduct a variety of experiments on three typical sequence labeling tasks: Named Entity Recognition (NER), Part of Speech Tagging (POS), and Chinese Word Segmentation (CWS) to show the significant performance of our GNN-SL. Notably, GNN-SL achieves SOTA results of 96.9 (+0.2) on PKU, 98.3 (+0.4) on CITYU, 98.5 (+0.2) on MSR, and 96.9 (+0.2) on AS for the CWS task, and resultscomparable to SOTA performances on NER datasets, and POS datasets.

## introduction sequence labeling (sl) is a fundamental problem in nlp, which encompasses a variety of tasks e.g., named entity recognition (ner), part of speech tagging (pos), and chinese word segmentation (cws). most existing sequence labeling algorithms @xcite @xcite @xcite can be decomposed into two parts: (1) representation learning: mapping each input word to a higherdimensional contextual vector using neural network models such as lstms @xcite , cnns @xcite , or pretrained language models @xcite ; and (2) classification: fitting the vector representation of each word to a softmax layer to obtain the classification label. because the protocol described above relies on the model's ability to memorize the characteristics of training examples, its performance plummets when handling long-tail cases or minority categories. intuitively, it's easier for a model to make predictions on long-term cases at test time when it is able to refer to similar training examples. for example, in figure 1 , the model can more easily label the word "phoenix" in the given sentence as an "organization" entity when referring to a similar example. benefiting from the success of augmented models in nlp @xcite @xcite @xcite , a simple yet effective method to mitigate the above issues is to apply the k nearest neighbors (knn) strategy: the knn model retrieves k similar tagging examples from a large cached datastore for each input word and augments the prediction with the probability computed by the cosine similarity between the input word and each of the retrieved nearest neighbors. unfortunately, there is a significant shortcoming of this strategy. retrieved neighbors are related to the input word in different ways: some are related in semantics while others in syntactic, some are close to the original input word while others are just noise. a more sophisticated model is required to model the relationships between retrieved examples and the input word. in this work, inspired by recent progress in combining graph neural networks (gnns) with augmented models @xcite , we propose gnn-sl to provide a general sequence-labeling model with the ability of effectively referring to training examples at test time. the core idea of gnn-sl is to build a graph between the retrieved nearest training examples and the input word, and use graph neural networks (gnns) to model their relationships. to this end, we construct an undirected graph, where nodes represent both the input words and retrieved training examples, and edges represent the relationship between each node. the message is passed between the input words and retrieved training examples. in this way, we are able to more effectively harness evidence from the retrieved neighbors in the training set and by aggregating information from them, better token-level representations are obtained for final predictions. to evaluate the effectiveness of gnn-sl, we conduct experiments over three widely-used sequence labeling tasks: named entity recognition (ner), part of speech tagging (pos), and chinese word segmentation (cws), and choose both english and chinese datasets as benchmarks. notably, applying the gnn-sl to the chinesebert @xcite , a chinese robust pre-training language model, we achieve sota results of 96.9 (+0.2) on pku, 98.3 (+0.4) on cityu, 98.5 (+0.2) on msr, and 96.9 (+0.2) on as for the cws task. we also achieve performances comparable to current sota results on conll, ontonotes5.0, ontonotes4.0 and msra for ner, and ctb5, ctb6, ud1.4, wsj and tweets for pos. we also conduct comprehensive ablation experiments to better understand the working mechanism of gnn-sl. ## related work retrieval augmented model retrieval augmented models additionally use the input to retrieve information from the constructed datastore to the model performance. as described in @xcite , this process can be understood as "an open-book exam is easier than a close-book exam". the retrieval augmented model is more familiar in the question answering task, in which the model generates related answers from a constructed datastore @xcite @xcite . recently other nlp tasks have introduced this approach and achieved a good performance, such as language modeling (lm) @xcite , dialog generation @xcite , neural machine translation (nmt) @xcite @xcite . ## knn-sl sequence labeling (sl) is a typical nlp task, which assigns a label @xmath0, . . . , w n }, where n denotes the length of the given sentence. we assume that {x , y} = {(x 1 , y 1 ), . . . , (x n , y n )} denotes the training set, where (x i , y i ), ∀1 ≤ @xmath1. let n be the size of the training set. ## experiments we conduct experiments on three widely-used subtasks of sequence labeling: named entity recognition (ner), part of speech tagging (pos), and chinese word segmentation (cws). due to the limit of pages, we put our training details that including the vanilla sl model and the knn retrieval in appendix a. ## conclusion in this work, we propose gnn-sl, which augments the vanilla sl model output with similar tagging examples retrieved from the whole training set. since not all the retrieved tagging examples benefit the model prediction, we construct a heterogeneous graph, and leverage graph neural networks (gnns) to transfer information from the retrieved nearest examples to the input word. this strategy enables the model to directly acquire similar tagging examples and improves the effectiveness in handling long-tail cases. we conduct multi experiments and analyses on three sequence labeling tasks: ner, pos, and cws. notably, gnn-sl achieves sota 96.9 (+0.2) on pku, 98.3 (+0.4) on cityu, 98.5 (+0.2) on msr, and 96.9 (+0.2) on as for the cws task.