RGAT at S em E val-2023 Task 2: Named Entity Recognition Using Graph Attention Network (2023)

Abstract: In this paper, we (team RGAT) describe our approach for the SemEval 2023 Task 2: Multilingual Complex Named Entity Recognition (MultiCoNER II). The goal of this task is to locate and classify named entities in unstructured short complex texts in 12 different languages and one multilingual setup. We use the dependency tree of the input query as additional feature in a Graph Attention Network along with the token and part-of-speech features. We also experiment with additional layers like BiLSTM and Transformer in addition to the CRF layer. However, we have not included any external Knowledge base like Wikipedia to enrich our inputs. We evaluated our proposed approach on the English NER dataset that resulted in a clean-subset F1 of 61.29\% and overall F1 of 56.91\%. However, other approaches that used external knowledge base performed significantly better.

## introduction named entity recognition (ner) is an important natural language processing (nlp) task that deals with extracting relevant information from various types of queries by classifying each token/word of the query into one of the pre-defined classes called named entities. this has wide range of applications in industries like retail, healthcare and manufacturing. for example, in retail domain, we would like to extract information from customer feedback that is instrumental in identifying current drawbacks of a product and direction for further improvement. similarly, factory logs in a plant containing information about a specific machine and sensor with domain specific terminologies need to be understood to take necessary actions in time. however, the task becomes challenging as the set of entity classes can vary greatly from one domain to another domain. most of the ner datasets that are routinely used for benchmarking ner algorithms contain complete or almost complete sentences with relatively few entity classes (e.g., conll-2003 has four classes and wnut-2017 has six classes). however, in real world applications there could be many granular entities with a very short sentence that may not have any context information at all. as a result, performance of the models may significantly suffer and that is what we have also observed in our current study. the present task, multiconer ii @xcite is a continuation of the previous year's semeval-2022 task @xcite that was created for detecting semantically ambiguous and complex entities in short and lowcontextual settings for 11 languages (i.e., english, spanish, dutch, russian, turkish, korean, farsi, german, chinese, hindi, and bengali). multi-coner ii similarly has created tasks for 12 languages, namely, english, spanish, ukranian, hindi, bangla, chinese, swedish, farsi, french, italian, portugese and german with 66 fine-grained entity classes spanning 6 broad groups of product, medical, location, creative works, groups and person. the number of classes is significantly higher compared to last year (only 12 classes) that increases the difficulty of this task. another challenging aspect of this task is added simulated errors to the test set during evaluation to make it more realistic and difficult. our approach lies in encoding the graph associated with the dependency parsing of the input sentence and create token (node) level representations that take care of both the neighborhood (connected edges) and dependency type (edge type) of each token. we add this feature with the other standard features like token embedding from a large language model like bert or roberta and partof-speech embeddings. based on our overall finegrained macro-averaged f1-score of 56.91, we secured 23rd rank in the english track of the competition. it is to be noted that our approach heavily relies on the availability of dependency structure of the input query and thus cannot be extended easily to other languages. it is also another shortcoming of this approach especially where the queries are not well-formed english sentences. the organization of the paper is as follows. in the next section we provide a detailed literature survey on the techniques employed for ner. next, we present the details of the proposed approach. subsequently, the model predictions and comparisons with other baseline methods are discussed. finally, conclusions are drawn and scope for future works is outlined. ## related work ever since it's introduction in 1996 @xcite , ner has been an active field of research that has evolved considerably with the emergence of new techniques. initial approaches to ner were limited to rule based methods followed by svm, hmm and crf @xcite . with the advent of deep learning based techniques, there are rnn @xcite @xcite @xcite and cnn @xcite @xcite based models that tried to find the best representation of each token. other than the token embeddings, additional features created are in terms of spelling, context and gazetteer @xcite and it was found that incorporation of prior knowledge substantially boosted the performance. similarly, the bilstm-cnn model by @xcite uses additional word-level (capitalization, lexicons) and character-level features (4-dimensional vector representing the type of a character: upper case, lower case, punctuation, other) in addition to the token embeddings. recently, @xcite proposed modularized interaction network (min) that generated the best result for conll2003 @xcite , wnut2017 @xcite and jnlpba datasets. however, none of these approaches created features based on the dependency relations amongst different tokens. as the transformer architecture @xcite started replacing many rnn/cnn based encoding, bert @xcite based embeddings found applications for ner @xcite @xcite . all these approaches are based on large pre-trained language models that have seen large monlingual or multilin-gual corpus during training and later fine-tuned for ner task. similar studies are carried out by @xcite where entity-switched datasets are created where different national origins play a role. @xcite studied transfer learning in a multilingual setup by creating invertible adapters. while most of the ner tasks involve wellformed sentences as inputs the real-world datasets can be very different with short texts having low context, emerging domain-specific entities with long-tails and complex entities like noun phrases, infinitives or gerunds. these aspects are discussed in great details by @xcite where solutions are proposed to fuse gazetteer knowledge with learnable weights. @xcite extended this approach to multilingual and code-mixed settings along with a mixture-of-experts model on multilingual transformer. continuing on this approach, @xcite created the multi-coner task @xcite ) that further enhanced the task with expanded language base. there are not many studies on the application of graph neural network for ner. @xcite investigated the utility of dependency parser and applied stacked graph convolution network (gcn) on ontonotes 5.0 dataset and obtained an improvement of 2% over a similar bidirectional lstm (bi-lstm) model. @xcite and @xcite further extended this approach for nested ner using a bi-affine model. @xcite introduced a bipartite graph to link the outer and inner named entities along with a bi-lstm and gcn model. @xcite proposed a combination of contextual features from xlnet and global features from gcn for improving model performance. gcn based features are also explored with transformer and bi-lstm with crf layer for biomedical domain @xcite and chinese ner @xcite . carley (2019) and wang et al. (2020) where the latter modified the original dependency tree to create an aspect-oriented dependency tree that was used further in a relational gat (rgat) where different relations contributed differently in the computation of nodal representations. ## methodology we have modified the rgat approach @xcite which was originally targeted for aspect polarity prediction. the approach utilizes the dependency structure of the input sentence which captures the grammatical relations by connecting the words with the corresponding dependency type. to bring out the relations of different words with the aspect word(s) an aspect oriented dependency tree (aodt) was proposed by @xcite where the root of the original dependency tree was shifted to the target aspect word followed by pruning some of the unnecessary relations. however, this approach is not exactly applicable for the current study since we do not know the entity words, a priori. however, knowing that these words are generally nouns (ignoring gerunds) we have randomly chosen one of the nouns present in the sentence as a surrogate entity word and modified the dependency tree based on this word. ## results in this section first we describe the performance on the test data of multiconer-ii followed by performance of our approach on other ner datasets. the best performance is obtained by a rgat-bert-bilstm-crf model where the fine-grained macro average precision, recall and f1-score on the test data are 0.625, 0.615 and 0.613, respectively. the corresponding class-wise performance is shown in fig. 1 , where the top-3 classes are "humansettlement", "sportsgrp" and "artist" and the bottom-3 classes are "privatecorp", "symptom" and "scientist". it is to be noted that "artist" and "humansettlement" are amongst the top-3 classes in terms of the number of labels, whereas, "symptom" and "privatecorp" are amongst the bottom-5 classes. thus, it is tempting to conclude that the availability of data plays a significant role on the performance. however, as we can see on the right subfigure of fig. 1 , there is no definitive trend with respect to the number of available data points. we also show the coarse-grained performance on the validation and test dataset in tab. 2. for the validation split, the macro average precision, recall and f1-score are 0.58, 0.57 and 0.57, respectively. we can see that overall the per-formance is better on the test dataset compared to the validation dataset. we investigated if there is any correlation between the sentence length (ℓ) and model's performance and we can clearly see that the performance improves with increasing sentence length. in particular, we created 4 buckets with (1) 11 > ℓ ≥ 1, (2) 14 > ℓ ≥ 11, (3) 19 > ℓ ≥ 14 and (4) 54 > ℓ ≥ 19 where 1, 11, 14, 19 and 53 are the minimum, first quartile, second quartile, third quartile and the maximum sentence length, respectively. the corresponding macro average f1 scores for the four buckets are 0.57, 0.62, 0.66 and 0.68, respectively. we have also compared the performance of our model on other standard @xcite and wnut'17 @xcite . the results are shown in tab. 3 where we have compared our model's performance with min @xcite . it can be seen that our approach is quite competitive compared to min and multiconer-ii dataset is very different in terms of both complexity and context length that resulted in a dramatically inferior performance. ## conclusion in this work we have explored the capability of graph attention network based model for ner task on multiconer-ii dataset. in this approach we create features from the graph structure obtained by a dependency parser that are combined with other standard features like word embeddings and pos tags. however, we have not included any external knowledge base (typically wikimedia data) and as a result the model's performance is clean-subset f1 of 61.29% and overall f1 of 56.91%. the exercise clearly brings out the importance of having external knowledge base and the next step would be to explore ways to optimally integrate external data with the rest of the architecture.