SAB at S em E val-2023 Task 2: Does Linguistic Information Aid in Named Entity Recognition? (2023)

Abstract: This paper describes the submission to SemEval-2023 Task 2: Multilingual Complex Named Entity Recognition (MultiCoNER II) by team SAB. This task aims to encourage growth in the field of Named Entity Recognition (NER) by focusing on complex and difficult categories of entities, in 12 different language tracks. The task of NER has historically shown the best results when a model incorporates an external knowledge base or gazetteer, however, less research has been applied to examining the effects of incorporating linguistic information into the model. In this task, we explored combining NER, part-of-speech (POS), and dependency relation labels into a multi-task model and report on the findings. We determine that the addition of POS and dependency relation information in this manner does not improve results.

## introduction named entity recognition (ner, @xcite is a well established natural language processing (nlp) task which entails extracting named entities (nes), or specific types of proper nouns, from a text and classifying the type of entity. in the original task from 1996, entities were considered either a person, organization, or geographic location, but since then the classes have expanded to include more complex entity types such as creative works (book or film titles, etc.), groups, and products. while it can be considered a task on its own, ner has many applications including information extraction (ie) and question answering (qa) @xcite . complex named entities (such as titles of creative works, products, or groups) are especially difficult for ner systems, because unlike traditional nes which are generally noun phrases, complex named entities may take various other forms @xcite . in addition, the domain of complex named entities is constantly expanding as new movies, books, and products are released. as a result, ner models must be adaptable in order to successfully identify and classify such entities. semeval-2023 task 2: multilingual complex named entity recognition ii @xcite is a shared task with the aim of investigating methods of identifying complex named entities in 11 individual language tracks, as well as a multilingual track. this task is a sequel to semeval-2022 task 11 @xcite , but with different language tracks and a much more in-depth set of entity classes. apart from having 33 fine-grained entity classes, the task also included simulated errors added to some of the test set language tracks to make the task more realistic and challenging. a large portion of nlp research focuses on english and neglects lower-resource languages @xcite . the language tracks defined in the task are english (en), spanish (es), hindi (hi), bangla (bn), chinese (zh), swedish (sv), farsi (fa), french (fr), italian (it), portuguese (pt), ukrainian (uk), and german (de), as well as a multilingual track (multi), to include a mix of highand low-resource languages. it is interesting to note that each language either has approximately 9,700 training samples, or approximately 16,500 training samples. the 33 fine-grain entity classes can be generalized to 6 coarse parent classes: location, creative work, group, person, product, and medical. results from last year's multiconer challenge indicated that transformers alone could not achieve high scores on complex named entities without the use of external knowledge bases or ensembles @xcite . for this task, rather than attempting to achieve the top score, we explore how much can be gained by incorporating linguistic data (part-of-speech and dependency relation information) into a multi-task setup built on xlm-roberta (xlm-r, @xcite . ## related work the notion of training ner and pos tasks in a multi-task setup was first proposed by @xcite . they utilized a time-delay neural network @xcite to jointly train a network on six nlp tasks including pos and ner, and demonstrated that multi-task learning improved the generalization of the tasks. transformer architectures have shown promising results for ner on the common benchmark datasets conll03 and ontonotes @xcite , however, bilstm architectures with a crf layer have also achieved high performance @xcite . most ner systems use some form of additional information, from pos tags to gazetteers or knowledge bases @xcite . @xcite analyze the effect of adding morphological features to long short-term memory (lstm) and bert models for ner, among other tasks. these morphological features include pos tags and universal feature embeddings. as with the experiments described in this paper, the additional linguistic information was obtained using stanza. the paper experiments on 11 different languages, although there is minimal overlap with the languages examined in this task. they use bio tagging for the ner task, but only include location, person, and organization as named entity classes. unlike the xlm-r model used for the submitted system described in this paper, they use the cased multilingual bert base model for their experiments. they conclude that the additional information does not make a practical difference in the bertbased model, although many of the languages do display a slight improvement when the additional information is applied to the lstm model. this suggests that bert already captures this information. it is unclear, however, if this should also hold for the multi-task setup on xlm-r at the center of the experiments in this paper. nguyen and nguyen (2021) introduce phonlp, a multi-task learning model for joint part-of-speech tagging, named entity recognition and dependency parsing in vietnamese. the model has a bertbased encoding layer followed by three decoding layers of pos tagging, ner and dependency parsing. they employ phobert base (nguyen and tuan nguyen, 2020), a pre-trained vietnamese language model, to encode the sentences. then, the three tasks' decoding layers are applied. they conclude from their experiments that phonlp's joint multi-task learning model performs better than single-task training on phobert in vietnamese, demonstrating that multi-task learning with pos and dependency tags applied with a transformers encoder can potentially be beneficial to this task. ## data the dataset used for this exploration is multi-coner v2 @xcite . it is the second iteration of the original multiconer dataset @xcite . this dataset was assembled with the specific goal of addressing the contemporary challenges in ner with a focus on complex named entity recognition. the data is uncased, and often the text is very short, providing minimal context to the model. additionally, it includes a large number of syntactically complex entities like movie titles, and long-tail entity distributions. the test set that systems were evaluated on is far larger than the training and dev splits. the reasoning for this is to assess the generalizability of the models. named entities are categorized into a total of 33 classes, within 6 coarse parent classes: location (loc), creative work (cw), group (grp), person (per), product (prod), and medical (med). the label distributions of the training data are not even across classes, and only vaguely even across languages. the per coarse-grain class on average represents about a third of the entities. loc, grp, and cw all represent about 15-17% of entity labels each. the least represented coarse classes are med and prod, which at worst represent under 5% of entities (fr), and at best only represent 11% (hi). hindi has the most even class distribution, with the greatest disparity being 17 percentage points between per and cw. italian has the worst disparity of the language tracks, with 34 percentage points between the per and med classes. noise was added to 30% of the instances on the test sets of (en, es, sv, pt, fr, zh, and it). the noise was added to either the context or entity tokens and is meant to represent typing errors based on the keyboard layout of the respective languages. the goal of this is to help determine whether the noise in entity tokens has an impact on ner prediction. ## methodology in this section, we discuss the setup and methods used in the final model. ## results systems in this task were evaluated by f1 score on all 33 fine-grained classes as well as the 6 coarsegrained classes. the official ranking metric is the overall fine-grained macro-averaged f1 score. the test set included simulated errors in 7 of the language tracks (en, es, sv, pt, fr, zh, and it) in we submitted across all language tracks, achieving our best results on hindi. we can also see from the scores on the noisy test data in from the coarse-grained f1 scores on the test set shown in the full table of fine-grained f1 scores on the test set for the submitted models can be found in appendix b. identifying artwork seems to be particularly challenging for the models across all languages. even the best performing model, hindi, obtained an f1 score of 0.043 on artwork. software seems like a comparatively easier form of creative work for the models, although still somewhat difficult. as expected, labels in the loc and per categories are the easiest for the models to identify. as previously mentioned, our best performing language track was hindi. it was interesting that english did not perform particularly well, considering that english is a high-resource language and hindi is less so. upon investigating why hindi might have performed so well, one hypothesis is that it may be due to the proportion of unseen entities in the test data. while english has approximately 72% unseen entities in the development set and 62% in the test set, the test set for hindi has a mere 28.7% unseen entities. this drastic difference may be why hindi achieved such higher scores than english. the goal of building a multi-task setup was to hopefully show improvement on the task of ner when including additional information from pos and dependency relation tags. however, as previously discussed, this unfortunately was not the case. worse than adding nothing useful to the model, the additional information actually showed a markedly detrimental impact on the task. @xcite showed that bert-based models did not benefit from extra pos information, but it did not harm the models. there are many speculations which might explain these results. it may be that with the multitask setup, the ner task was not seen frequently enough within the 10 epoch training span. this could be explored in future work either by extending the number of training epochs, or altering the ner task's learning rate, hopefully making the model favor the ner task. another speculation is that the additional information may have hurt results because the pos and dependency relation labels were generated using stanza, and are not necessarily all correct. perhaps they are adding more noise than useful information. ## conclusion this paper describes the multi-task system built on xlm-r submitted to the multiconer ii shared task at semeval 2023 by sab. our system explores utilizing part-of-speech and dependency relation information, in addition to the named entity recognition labels provided in the dataset. we obtain the strongest results on the hindi language track, which we attribute to the data's far lower percentage of unseen entities in the test set than other languages. the simulated noise in the test set also consistently lowers scores by approximately 10 percentage points, indicating that these models do not generalize very well. we also show that our multi-task model using pos and dependency relation information does more harm than good for this ner task. in future experiments we will explore separating out the labels for each model in the multi-task setup, to observe if this had an effect on the loss function. we would also potentially explore other means of obtaining the part-of-speech and dependency relation information besides stanza, as every tagger produces different results of varying quality. additionally it would be interesting to see if a different type of tokenizer (i.e. character-based instead of sub-word) would perform better or worse when tested on the simulated errors in the test dataset. task 11: multilingual complex named entity recognition (multiconer). in proceedings of the 16th international workshop on semantic evaluation (semeval-2022). association for computational linguistics. dat quoc nguyen and anh tuan nguyen. 2020. phobert: pre-trained language models for vietnamese. in findings of the association for computational linguistics: emnlp 2020, pages 1037-1042, online. association for computational linguistics. linh the nguyen and dat quoc nguyen. 2021. phonlp: a joint multi-task learning model for vietnamese part-of-speech tagging, named entity recognition and dependency parsing. in north american chapter of the association for computational linguistics. peng qi, yuhao zhang, yuhui zhang, jason bolton, and christopher d manning. 2020. stanza: a python natural language processing toolkit for many human languages. arxiv preprint arxiv:2003.07082. lev ratinov and dan roth. 2009. design challenges and misconceptions in named entity recognition. in proceedings of the thirteenth conference on computational natural language learning (conll-2009), pages 147-155, boulder, colorado. association for computational linguistics. peng sun, xuezhen yang, xiaobing zhao, and zhijuan wang. 2018. an overview of named entity recognition. in 2018 international conference on asian language processing (ialp), pages 273-278. ieee. alexander waibel, toshiyuki hanazawa, geoffrey hinton, kiyohiro shikano, and kevin j lang. 1989. phoneme recognition using time-delay neural networks. ieee transactions on acoustics, speech, and signal processing, 37(3):328-339.