M ar S an at S em E val-2022 Task 11: Multilingual complex named entity recognition using T5 and transformer encoder (2022)

Abstract: The multilingual complex named entity recognition task of SemEval2020 required participants to detect semantically ambiguous and complex entities in 11 languages. In order to participate in this competition, a deep learning model is being used with the T5 text-to-text language model and its multilingual version, MT5, along with the transformer’s encoder module. The subtoken check has also been introduced, resulting in a 4% increase in the model F1-score in English. We also examined the use of the BPEmb model for converting input tokens to representation vectors in this research. A performance evaluation of the proposed entity detection model is presented at the end of this paper. Six different scenarios were defined, and the proposed model was evaluated in each scenario within the English development set. Our model is also evaluated in other languages.

## introduction named entity recognition (ner) is a key component of natural language processing (nlp) assigned to identify regions of text that contain references to entities. it is the process of identifying the informative part of data or applicable labels from unstructured data. in ner, data is gathered from unstructured data such as emails, blogs, newspapers, tweets, etc., to extract meaningful information. to put it another way, the term ner refers to identifying token spans of entities mentioned in the text and classifying them into a set of predetermined categories. the system finds entities from unstructured data and organizes them into multiple categories. as an extension of nlp, the field of ner can be considered as information extraction (ie). for nlp, ies are among the trending fields and play an essential role in the following tasks: find and understand limited relevant parts of texts, gather information from many pieces of text, and produce the unified representation of all the relevant information. the ner problem falls into a general class of nlp problems called sequence tagging. part of speech (pos) tagging and chunking are sequence tagging nlp tasks in addition to ner. it is possible to detect ner in three ways: flat ner, nested ner, and discontinuous ner. nested ner has overlapping in the span of text @xcite . most approaches only target flat entities, ignoring nested structures common in many scenarios. it is challenging to identify spans as well as types of named entities in text @xcite . so to find the best architecture, we implement a transformer-based language model in this research. nlp has significantly benefited from transfer learning in recent years. transfer learning gains power and effectiveness from pre-training on large, unlabeled text datasets. the model can then be fine-tuned to a smaller labeled dataset, resulting in better performance. many models have achieved success in this field, including the text-to-text transfer transformer (t5) @xcite . the t5 is a pre-trained encoder-decoder language model that employs the "text-to-text" format to accomplish all types of nlp work, including generation, translation, and summarization tasks. in semeval-2022 task 11 @xcite , a multilingual complex ner is provided. languages presented in @xcite are bangla, german, english, spanish, farsi, hindi, korean, dutch, russian, turkish, and chinese. since this dataset contains words from different languages, it is challenging to choose an appropriate word representation for converting them into their corresponding vectors. the multilingual nature of this task necessitated the selection of the multilingual variant of google's t5 model named @xcite that had already been trained on a database of more than 101 languages and con-tained up to 13 billion parameters. in this paper, mt5 is used as the main embedding. we evaluated the proposed model using the english test set and achieved the f1-score of 71.45% as part of this competition. in our next step, we considered this model in other subtasks, and our rank varied from 9 to 21 depending on which subtask we evaluated. our code is available at github foot_0 for researchers. the contributions of this paper are summarized as follows. section 2 introduces previous attempts in the ner. in section 3, information about the task and datasets is presented. we then offer a deep learning framework for recognizing named entities in section 4. section 5 details the experimental setup, while section 6 presents the results of the experiments. section 7 presents both quantitative and qualitative error analysis. we conclude our paper in section 8. ## background the ner field has undergone enormous changes in recent years @xcite . as mentioned before, ner is the process of identifying relevant objects such as persons, products, genes, place, organization, etc., that are mentioned in the string of the text, sentence, or paragraph. ner typically forms the basis of other tasks such as event detection from news, online shopping customer service, knowledge graph construction, and biological analysis @xcite . @xcite al. (2020) word representations and character representations are used in @xcite , an end-toend system designed using lstm, cnn, and crf. this idea has been implemented on the penn treebank wsj corpus @xcite @xcite for ner datasets. a cnn has been used in this research to extract the character-level representation of words, with its output then being input into lstms, followed by a crf layer. there is another extension to crf known as hybrid semi-markov conditional random fields (hscrfs) is explained in @xcite by contributing word-level labels in the building of scrfs lafferty et al. @xcite . the purpose of using word-level tags to derive segment scores is to obtain segment scores. to solve different sequence tagging models with a crf inference, @xcite implemented ncrf++ with three steps: a character sequence layer, a word sequence layer, and an inference layer. rodrigoagerri et al. presented a multilingual ner system that combines many features to cluster based on local information @xcite . results from standard task evaluation data such as conll for english, spanish, and dutch were reposted. according to @xcite , contextualized language models can produce better results when different embeddings are combined. this paper presents a framework for generating and scoring the output of embedded combinations using rein-forcement learning, which achieves the best accuracy in 6 different fields and 21 large datasets. according to @xcite , contextualized language models can produce better results when different embeddings are combined. by rewarding model scores for better concatenations of embeddings, can propose automated concatenation of embeddings to find better concatenations of embeddings for structured prediction tasks. @xcite considers the issue of informal data, whose unstructured and incomplete nature makes the process more challenging. to solve the mentioned challenge, bi-lstm based architecture for informal tweets in hindi and english was implemented. there is more than one way to do it in this field. finding the part of the text containing entity information, classifying and identifying the right entity, and applying that entity to the appropriate part of the text are just some ways. generating tags is one other way to implement them. as mentioned in @xcite , hang yan et al. propose that they use a novel and simple bidirectional auto-regressive transformer(bart) sequence-to-sequence (seq2seq) framework that uses a pointer mechanism @xcite to generate the entity sequence directly. another approach in ner, which is mentioned in @xcite , consists of using an attention mechanism to minimize the problem of detecting redundant and inessential data and ignoring them entirely. combining semantic, glyph, and phonetic features to improve the expression ability of chinese character embedding, @xcite proposes an architecture based on fusion embedding for the chinese language. there is also a paper for the chinese language entitled jia et al. (2020) that identifies entities from chinese social media texts, using uncertain information from word segmentation. researchers have proposed that interactions between spans of tokens can help determine discontinuous mentions and have developed a transition-based model with a generic neural encoding to be able to detect discontinuous mentions @xcite . a model of bidirectional transformers is presented in @xcite , which produces a contextualized representation of words and tokens. bert's masked language model is used as pretrained word embeddings. as a result, yamada et al. present advancement in attention known as entityaware self-attention mechanisms and achieve state-of-the-art in five benchmarks that include: open entity (entity typing), tacred (relation classification), conll-2003 (ner), record (cloze-style question answering), and squad 1.1 (extractive question answering). since carelessness or a lack of background knowledge of annotators might lead to model performance errors, some research has been conducted to identify and solve these issues. @xcite provides a framework for finding human errors in ner annotations. following the correction of labels in the test set, they re-evaluated state models in ner, claiming that the results were more accurate than the original test set. this paper's main idea is cross weighs, which accommodates label mistakes during training and then trains a more robust ner model. @xcite finds the external context of input sentences by retrieving relevant sentences. the process of selecting the top similar text involves re-ranking retrieved samples according to their semantic relevance to the input sentence. as a result, the inputs are the concatenation of input sentences and external contexts. both input types are used to implement cooperative learning (cl), and different representations are encouraged to produce similar contextual representations or output label distributions. results on eight other ner datasets achieve state-of-the-art results. but one drawback of this method is that there are no document-level contexts in practice. ## task description our investigation aims to comparatively study multiconer-2022 datasets that have been considered individually for complex ner systems in 11 languages. under short and low-context settings, the task detects semantically ambiguous and complex entities. languages presented in multiconer-2022 are: english, spanish, dutch, russian, turkish, korean, farsi, german, chinese, hindi, and bangla. the number of instances for each language varies between 150k to 500k. there are 15300 train data and 800 validation data for each language. this dataset is labeled with the following tags: per: person, loc: location, grp: group, corp: corporation, prod: product, cw: creative work. there is detailed information of datasets illustrated in the datasets used in this task include sentences labeled with the iob format. using this format, ## system overview in this section, we will introduce our proposed ner framework. the proposed framework consists of three parts: ## experimental setup we implemented the model in pytorch and trained it on nvidia v100 gpus. the adamw optimizer with a learning rate of 2e-5 is used to train the network. our training method includes early stopping, which ensures the validation loss reduction with patience of 10 epochs. the training batch size is set to 32, and the dropout rate is 0.2. transformer encoders have eight attention heads, and positionwise feed-forward layers have 2048 hidden sizes. in both t5 and mt5 tokenizers, the max length varies between 100 and 250 characters according to the evaluated language. the hyper-parameters of each subtask were tuned with the dev set. all other parameters are initialized randomly. ## results several experiments have been conducted to develop the most appropriate model for ner. experiments with the english dataset can be found in we have attempted to improve the language model results by adding deep learning architectures and textual features @xcite . we evaluated lstm and transformer architecture on top of the t5 and found that using the transformer improved the f1-score further than lstm. after various experiments, it was found that the use of bpemb could not improve the f1-score of the model. the results of proposed model for the different entities are shown in ## error analysis several scenarios could occur when comparing the golden standard annotation with the output of a scenario 2, complete false positive: an entity is predicted by the ner model but is not annotated in the hand-labeled text. scenario 3, complete false negative: the model does not predict a hand-labeled entity. scenario 4, wrong label, right span: a hand-labeled entity and a predicted one have the same span but different tags. scenario 5, right label, overlapping spans: a hand-labeled entity and a predicted one have overlapping spans and the same tags. scenario 6, wrong label, overlapping spans: a hand-labeled entity and a predicted one have overlapping spans but different tags. the output of the proposed model on the english dev dataset has been evaluated on six scenarios in scenario 4 includes 6.21% of "corp" entities. scenario 5 has the most significant number of entities compared to other scenarios, having 6.12 percent of the "prod" entities. the "corp" also has the highest number of entities in scenario 6. ## conclusion this paper proposes a model that uses an encoder module of transformers in the top of the hidden state of the t5 to process the extracted features to obtain the most important feature representations. many experiments were conducted to evaluate the model's performance and find the best language model. the experiments prove that our architecture is most compatible with the t5 language model and does cover a reasonable range of results. since this dataset is multilingual, the mt5 embedding