Raccoons at S em E val-2022 Task 11: Leveraging Concatenated Word Embeddings for Named Entity Recognition (2022)

Abstract: Named Entity Recognition (NER), an essential subtask in NLP that identifies text belonging to predefined semantics such as a person, location, organization, drug, time, clinical procedure, biological protein, etc. NER plays a vital role in various fields such as informationextraction, question answering, and machine translation. This paper describes our participating system run to the Named entity recognitionand classification shared task SemEval-2022. The task is motivated towards detecting semantically ambiguous and complex entities in shortand low-context settings. Our team focused on improving entity recognition by improving the word embeddings. We concatenated the word representations from State-of-the-art language models and passed them to find the best representation through a reinforcement trainer. Our results highlight the improvements achieved by various embedding concatenations.

## introduction named entity recognition and classification is an essential subtask in natural language processing (nlp). the task of identifying named entities like a person, location, organization, drug, time, clinical procedure, biological protein, etc. in a text document is key to many applications, including information extraction and retrieval, machine translation @xcite , topic modeling @xcite ), text summarization @xcite , and question answering. it can also be used in domainspecific entity types such as disease, symptoms, and treatments. recently many researchers have identified the name identification issue in a variety of languages and have made efforts to apply named entity recognition. complex and ambiguous named entities, like the title of creative works such as books, songs, and movies, are not simple nouns and are thus harder to recognize @xcite . unlike the traditional nes these works take the form of linguistic constituents such as "remember me when we are parted," which is an imperative clause and does not look like the traditional nes @xcite . this ambiguity in syntax makes it difficult to identify them based on their context. there can also occur instances where these titles are semantically ambiguous; for example, "the girl on the train" can be a preposition or the name of a book. such entities are growing at a faster rate as compared to the traditional categories. therefore, processing these nes has always been a challenging task in nlp in practical and opendomain settings. however, it has received enough engagement from the research community. despite the high score produced by neural models on benchmark datasets like conll03/ontonotes, it has been noticed @xcite that these models perform significantly low on complex or unseen data. this happens because the scores were driven by the presence of easy entities, well-formed text, and memorization due to entity overlap between train/test sets. various researchers have noted that the majority of the errors in their downstream tasks have occurred due to the failure of ner systems to recognize complex entities. various researchers using ner on downstream tasks have noted that a significant proportion of their errors are due to ner systems failing to recognize complex entities. this paper presents our system for shared task on "multiconer multilingual complex named entity recognition @ semeval 2022" @xcite , @xcite , @xcite . the task focused on detecting complex and ambiguous entities in short and low-context settings. our team focused on improving entity recognition by improving the word representations, following @xcite . we concatenated the word embeddings from stateof-the-art language models and passed them to find the best representation through a reinforcement trainer. ## related work name entity recognition has always been a challenging task that requires massive prior knowledge resources for good performance @xcite . research in various domains using a variety of approaches has been done. early methods included heuristics and handcrafted rules, lexicons, orthographic features, and ontologies. these systems involved recognizing entities based on pattern matching with input documents @xcite . a higher degree of accuracy may be achieved using the rule-based system. however, it is timeconsuming and challenging to port the developed rules from one domain to another. later machine learning methods were used @xcite @xcite . this learning technique involved supervised and unsupervised approaches. the supervised being a dominant technique for named entity recognition @xcite . however, this method required massive high-quality annotated data. there also exist some hybrid models that make use of different rulebased and/or learning methods for enhanced performance @xcite . these systems make the best use of good features and methods for improved performance. recently neural network systems @xcite have also become a popular method. these systems with minimal feature engineering have become appealing because they do not require sources such as lexicons or ontologies, thus making them domain independent. most of the neural architectures proposed are based on recurrent neural networks (rnn). ## dataset description the english dataset provided by semeval 2022 @xcite comprises 233,917 instances of data. the training sample included 15300 annotated statements, the dev set comprised 800 annotated sentences, and the test set included 217817 test data (as shown in the table). the sentences in the data are allocated a respective id. each sentence is further divided into tokens which are separated by a newline. each token is assigned a label according to the standard bio tagging followed by the named entity tag in the specified format. ## methodology the method used focuses on automating the process of finding better concatenation of embeddings for improved performance. in this approach, a task model and controller module repeatedly interact. to achieve high accuracy, the controller searches for a better embedding concatenation from the given set of pre-trained embeddings. the task model, on the other hand, produces a task output. the architecture works on a reward basis. the controller is rewarded every time the task model is trained over the task data after an embedding is generated. the controller receives a reward for updating its parameter and sampling new embedding concatenations. the general architecture of our approach is shown in figure 1 . ## experimental setup data was provided in form of sentences which were broken down into individual words, or tokens, each lying in a newline and their corresponding tags in front of them. all the tokens are first passed through a function to encode them into embeddings, which are initial representations for the tokens. these tokens are pushed through the rnn language model which forms the task model and iteratively returns a reward to the controller. this model first contains a dropout layer, then an encoder layer is added which contains the embedding. the next layer is an lstm layer with a hidden size of 800. finally, the representations are fed into a linear-chain crf layer to predict the final label sequence, where a linear layer is applied for the representations to score each entity label. the above deep learning model has been built using pytorch 1 along with the transformer embeddings that have been used. the learning rate was set at 0.1 at the beginning with a patience level set to 5, i.e. the learning rate was halved if there was no improvement in monitored metrics for the patience, that is the validation loss. the batch size was set to 64 and a maximum of 150 epochs were allowed for 20 episodes. ## results the table below shows the micro and macro f1 scores of prediction on the development set. as the tags of the test set are unavailable to us, the model's performance was judged on dev set only and the results of the best 3 models were ensembled in the end, for submission. the best performing model was achieved in a model, where the following embeddings were concatenated: finetuned bert-large-uncased, finetuned robertalarge, elmo original, fastcharacterembeddings, glove, fastwordembeddings-english, flair newsen embedding. the model was trained on with the following hyperparameters setting 150 epochs, 20 episodes, sgd optimizer, 800 hidden units of bilstm-crf, starting with a learning rate of 0.1 and a batch size of 64. these results were ensembled with two other models which were developed as follows: ## result analysis experiments were conducted to analyze the performance of systems by using different combinations of embeddings. the best concatenation of embeddings can be easily seen in the model containing bert-large-uncase transformer embedding and elmo original, fastcharacterembeddings, glove fastword: english, and flair: news-forward with a macro-f1 score of 0.8745 on the dev set. this model generated the best results after ensembling the results with the models containing bert-baseuncased embeddings and bert-large-uncased finetuned on the current dataset, which generated a macro-f1 score of 0.7418 2 on the test dataset. one clear distinction in the result can be seen among the bert uncased and cased models. in the 2 this is the final macro-f1 score on the test set 3 micro-f1 score was the only metric available for baseline bert-uncased models, the text is set to lowercase before the wordpiece tokenization step, hence case is irrelevant in it, while in bert-cased models, the case of words is also considered. in our dataset, the tokens were already lowercase. for comparison's sake, we have experimented with roberta embeddings as well and they have produced slightly better results than bert embeddings alone. one difference in the functioning of bert and roberta was noticed during experimenting with their tokenizers. the bert tokenizer sometimes separated hyphen-separated entities and treated them as individual entities while roberta tokenizer treated them as a combined single entity and generated a representation for it. for comparison, we have also experimented with xlmr embeddings during result compilation to show the difference (or similarity) between results. after removing a few of the other embeddings like elmo, flair, and glove, we have also collected results from the above model. all the results can be referred to in ## conclusion and future work in this paper, we present our approach to semeval-2022 task 11:multiconer multilingual complex named entity recognition. our best submission gave us an f 1 score of 0.7418, placing us 10 th on the evaluation phase leaderboard. future work includes experimenting with multilingual data and embeddings and different optimizers.