Multilinguals at S em E val-2022 Task 11: Transformer Based Architecture for Complex NER (2022)

Abstract: We investigate the task of complex NER for the English language. The task is non-trivial due to the semantic ambiguity of the textual structure and the rarity of occurrence of such entities in the prevalent literature. Using pre-trained language models such as BERT, we obtain a competitive performance on this task. We qualitatively analyze the performance of multiple architectures for this task. All our models are able to outperform the baseline by a significant margin. Our best performing model beats the baseline F1-score by over 9%.

## introduction the named entity recognition (ner) task aims to detect entities from unstructured text and classify them into predefined categories. although the task of ner has been investigated adequately by previous research work @xcite @xcite @xcite , the detection of named entities in open-domain settings is non-trivial. moreover, the introduction of additional layers of complexity, in the form of semantic ambiguity and a lower amount of contextual availability, poses further challenges. for example, in a low-context and semantically ambiguous sentence such as let us play among us, the token sequence among us, can refer to a common phrase or a popular video game, and hence be categorized as a creative work (cw). recently, deep learning models have gained popularity for ner @xcite @xcite . however, these approaches are data-intensive and become ineffective when there is a lack of labeled data. to foster research in this area, @xcite has introduced the semeval multiconer shared task that deals with multilingual complex named en-* equal contribution tity recognition. this task in based on the complex ner, search query and code-mixing ner challenges introduced by @xcite and @xcite . the baseline introduced by the organizers for this multiconer challenge is a pre-trained xlm-roberta model @xcite that was further fine-tuned on the taskspecific training dataset. this paper describes our approach to tackle complex ner task for the english language using stateof-the-art deep learning models and introduces a simple neural network architecture that builds on top of pre-trained language models. we compare multiple architectures on the validation and test set of the shared task. all our models outperform the baseline by a significant margin. through our experiments, we discover that leveraging transformer models based on attention mechanism @xcite results in better performance even in low context and ambiguous settings. the code is available at https://github.com/ amitpandey-research/complex_ner we describe the prior research work done with respect to both general and low-resource ner tasks in section 2. we provide the formal task description in section 3, the dataset details in section 4, the method and the model architecture in section 5. we provide details about the experimental implementation in section 6. we discuss the results obtained and error analysis in sections 7 and 8 respectively, and finally, we conclude the paper in section 9. ## related work a widely used benchmark for ner was the conll 2003 shared task. it contained annotated newswire text from the reuters rcv1 corpus. previous researchers @xcite had used bilstm models with attention to predict named entities on this dataset. @xcite used a bilstm-cnn-crf to predict the named entities. sequence labeling for named entity recognition: recent approaches have aimed at utilizing deep learning techniques for training ner models. however, these techniques require a large amount of token-level labeled data for ner tasks. annotation for such kinds of labeled datasets can be expensive, time-consuming, and laborious. the datasets introduced in this task encompass a large number of low-resource and complex ner entities. recent work on ner in scientific documents has been concentrated around detecting biomedical named entities (kocaman and talby, 2020) or scientific entities like tasks, methods and datasets @xcite @xcite . ner has been traditonally modelled as a sequence labelling task, using crf @xcite to classify the labels. recent approaches have used deep learning based models @xcite . these approaches are data intensive in nature. to tackle the label scarcity problem, methods like distant supervision @xcite @xcite , active learning @xcite , reinforcement learning-based distant supervision @xcite have been proposed. ## task description the objective of this shared task is to build complex named entity recognition systems. the task presents a unique challenge in the form of detecting the entities in semantically ambiguous and low-context settings. moreover, the shared task also tests the generalization capability and domain adaptability of the proposed systems by testing the system over additional (low-context) datasets containing questions and short search queries, such as google search queries. for this task, given an input sentence (an arbitrary sequence of tokens), the systems have to identify the b-i-o format (ramshaw and marcus, 1999) (short for beginning, inside, outside) tags for 6 ner entity classes: person, product, location, group, corporation, and creative work. the description attributed to each class label is described in ## dataset the multiconer dataset @xcite consists of labeled complex named entities (ne). for the monolingual track, the participants have to train a model that works for a single language. for training and validation purposes, train and dev sets are provided with labeled entities. the monolingual model trained needs to be used for the prediction of named entities in the test set that consists of more than 150k instances. the labels from the test set are not provided directly. in this system description for the monolingual track, we have considered the english ner dataset for our task. the dataset follows a bio tagging scheme, and there are six entity types in the label space. the statistics for the english dataset in the monolingual track for the train and dev set are provided in ## system overview this section describes our approach to designing a system to solve the problem of classifying the tokens (words) of a given sentence into one of the six ne categories. we also briefly describe features of the bert (bidirectional encoder representations from transformers) @xcite model employed in our system. we designed three architectures based on pretrained language model bert: 1) bert+linear, 2) bert+crf, and 3) bert+bilstm+crf. a detailed explanation of these architectures is as follows: ## implementation details we implement all our transformer-based models using pytorch and huggingface library. we implement 3 models: 1) bert+linear, 2) bert+crf, and 3) bert+bilstm+crf. we also experiment with feature engineering by concatenating label encoded part-of-speech (pos) tags to the token embeddings. we use a dropout from 0.2 to 0.5 in all models and find that a dropout probability of 0.3 gives the best results throughout. in the bert+linear model, we use two fully connected dense linear layers as a classifier on top of the bert embedding layer. we add a softmax layer to obtain the probability distribution across all the labels. for the bert+linear model, we run our experiments across 1-20 epochs. we find that the model starts to overfit after 10 epochs, and the best results are obtained after 5 epochs of training. we further experiment with bert-base (12 attention heads) and bert-large (16 attention heads). for bert+crf and bert+bilstm+crf, we experiment across 1-100 epochs. we find that the models give the most optimal result at the 20th epoch, after which they start to overfit. we use a learning rate of 1e -6 for all the models. we validate the results of all models using our dev set and then use the best performing model for final evaluation on the blind test set. ## results we compare the performance of our models in the validation set against the baseline. we use the best performing model for the final submission in the evaluation phase. we provide details of the performance of the best performing model over the blind test dataset provided in the evaluation phase. we provide a detailed comparison of the performance of our models across all the class labels in the validation dataset in also, it can be observed from ## error analysis we perform error analysis for all 3 different model performances on the validation dataset. we find that for all 3 models, each model has the greatest difficulty in accurately predicting the cw (creative work) label. this can be attributed to the higher degree of ambiguity when it comes to cw named entities, as these often share a similar type of textual structure as regular non-named entity text tokens. it can be inferred that all 3 models are memorizing entity names from the training data to some extent. it is most prevalent in bert+bilstm+crf model, as we can see that it has the least amount of prediction accuracy among other models. this is consistent with our reasoning that heavier models tend to overfit the dataset faster. hence, we deduce that named entity memorization can be attributed to a type of overfitting behavior by the model in question when the training data is scarce. the bert+linear model, which is the lightest model with the least amount of trainable parameters among all 3, is found to be significantly less prone to memorize entity names. furthermore, upon qualitative analysis, we find that our models often have difficulty in recognizing longer named entities (entities comprising of 5 or more tokens). this can be attributed to the lack of occurrence of such entities in the training dataset. the models are majorly exposed to shorter length entity spans across the training set. due to the lack of exposure of the models to adequate training instances of longer spans, the models are often unable to predict such longer entity spans. it is also worth noting that an increase in the number of attention heads in the bert layer helps in substantial improvement in the accuracy. as discussed, this can be attributed to better learning of the context with the help of attention mechanism. we conclude that the larger number of attention heads are able to classify longer entity spans with greater accuracy. ## conclusion and future work we experiment with 3 model architectures for a novel dataset introduced for the shared task of detection of complex ner. our best performing model comprises of a simple linear classifier on top of fine-tuned bert-based language model. we find that this simple approach performs competitively as compared to its heavier counterparts. upon analysis, we attribute this observation to the scarcity of labeled training data. bert+linear model is able to optimally avoid overfitting to a larger extent and hence performs better than other heavier models. we find that our simpler model ranks in the top 10 in the validation phase and outperforms numerous teams in the final evaluation phase. for future work, we aim to utilize other data augmentation techniques and distant supervision to create clean silver labels in order to increase our training instances. we believe that this would help us leverage larger models for training purposes.