TEAM -Atreides at S em E val-2022 Task 11: On leveraging data augmentation and ensemble to recognize complex Named Entities in B angla (2022)

Abstract: Many areas, such as the biological and healthcare domain, artistic works, and organization names, have nested, overlapping, discontinuous entity mentions that may even be syntactically or semantically ambiguous in practice. Traditional sequence tagging algorithms are unable to recognize these complex mentions because they may violate the assumptions upon which sequence tagging schemes are founded. In this paper, we describe our contribution to SemEval 2022 Task 11 on identifying such complex Named Entities. We have leveraged the ensemble of multiple ELECTRA-based models that were exclusively pretrained on the Bangla language with the performance of ELECTRA-based models pretrained on English to achieve competitive performance on the Track-11. Besides providing a system description, we will also present the outcomes of our experiments on architectural decisions, dataset augmentations, and post-competition findings.

## introduction and related works the task of identifying and classifying entities in text is known as named entity recognition (ner). some named entities are easy to distinguish in english since each of their words is capitalized; e.g. "the capital of bangladesh is dhaka". in this sentence, both "bangladesh" and "dhaka" are capitalized named entities. but there are other entity mentions that are not simple nouns and are more difficult to recognize. in the semeval task 11: multiconer multilingual complex named entity recognition @xcite , the organizers concentrated on the more unusual named entities, which can be difficult to identify accurately from the text. *these authors contributed equally ner tasks have received much attention from the research community due to its crucial role in different nlp problems like information retrieval @xcite , question answering @xcite , relation extraction, entity linking @xcite and searching @xcite . however, there is such a conceptual difference between an ordinary named entity and a complex named entity that traditional tagging strategies cannot be used to recognize these mentions @xcite . complex ners can be any language element (single word, abbreviations, imperative clauses, questions) of ambiguous (multi-type or overlapping) and non-regular forms (nested or discontinuous or overlapping) @xcite . what makes the task more challenging is, complex ner is part of the open-domain with ever expanding and emerging entity sets and categories. in recent days, transformer-based models @xcite @xcite have been performing as the state-of-the-art @xcite models in different ner benchmark datasets. although, augenstein and colleagues, demonstrate in their paper that these powerful models are only good at picking up the conventional ners from well formed texts @xcite , while for complex ners we still need to integrate external knowledge sources. a recent paper on integrating external sources or gazetteer features in combination with contextual information, has shown that this can indeed improve performance on complex ner tasks @xcite . gazetteer-based solutions also show good performance improvements in extracting ners from both normal and code-mixed webqueries @xcite . in tasks like ner, bangla nlp has not made significant progress. many linguistic issues arise while training models on bangla because it is a rich language in terms of both usability and vocabulary @xcite . in bangla, there are few markers for tags, such as capitalization @xcite . the same words can have a variety of meanings and types of entities. in addition, because bangla is a somewhat free word order language, words can exist in any place inside a phrase without changing their meaning @xcite . affixes that are added to the root word to cause complex inflections can modify the meaning and type of the word as well @xcite . despite these issues, transfomer models have been used with considerable success for ner tasks in bangla @xcite . in this work, we demonstrate our approaches in tackling the concerns raised in the semeval task 11, as well as the obstacles posed by the bangla language's intrinsic complexity. in our proposed architecture, we used a variety of methodologies, primarily focusing on transfer-learning with stateof-the-art deep learning architectures. in particular, we submitted the results obtained from monolingual electra models, while we also ran experiments with non-contextual word embeddings and multilingual language models. ## dataset description according to the organizers, the data were gathered from wikipedia and microsoft orcas, which included both statements and queries @xcite . the train set contains about 100 domain adaption instances, whereas the test set has significantly more out-of-domain data to measure out-of-domain performance. the test dataset is a large file of 130k+ sentences, with a preset training dataset of 15300 bangla sentences and a development dataset of 800 sentences. other important statistics about the dataset is presented in ??. the distribution of ner classes in the training set is shown in figure 1 . to perform the experiments, we augmented our datasets in several stages. at first we token-wise translated a portion of our non-bangla dataset to bangla using google translate api 1 . in the first stage, we combined translated hindi and farsi dataset with our bangla dataset, as all three lantype frequency train 15300 dev 800 test 133119 single word tokens 4824 multi word tokens 10481 ## system description the system we proposed for complex bangla named entity recognition is an ensemble of electra based models trained on the augmented datasets mentioned in ## experimental setup as we have previously discussed in section 2, we augmented our training data in multiple steps which extended the dataset several times compared to original. we split each version of these dataset into a 70%-30% ratio during training. the default dev set containing 800 sentences is used for the final validation, in choosing the best performing model during test phase. we employed accuracy, precision, recall, and f1 score as evaluation metrics, with the macro averaged f1 score as the primary and official metric, as per the benchmark of sem-eval 2022 task 11: multiconer @xcite . we defined each of our best performing model configurations in ## results we made 4 submissions during the test phase, by applying majority voting scheme on various combinations of model predictions. the performance of the final ensemble outputs are presented in 6. as we can observe, the final ensembles of all models performs the highest and it is ranked 8th among all model versions m1 s1 + d1 + mha m2 s1 + d2 m3 s1 + d4 m4 s2 + d1 + crf m5 s2 + d2 + crf m6 s2 + d4 + crf + mha m7 s2 + d4 + character cnn m8 s1 + d6 from section 5 we see that, there's hardly any difference among the variations of the s2 models, while major fluctuations can be observed among the variations of s1 models. furthermore, separately grouped ensembles of s1 and s2 performs almost identically, with the combined ensemble of s1 and s2. however, the performance improves upon including the predictions from s1.a models, which are trained on english translated datasets. despite this, the final best model is clearly overfitting because it had over 80% score on the development dataset, while performing significantly worse (approximately 60%) during the test phase of the competition. this outcome may be attributed to several factors, including the choice of hyperparameters, dataset augmentations and splitting process, early stopping criteria etc. as per the rules of the competition, we only experimented with mono-lingual models to obtain our results. however, we ran the baseline xlm-roberta model which achieves an f1-score of approximately 68% on the development dataset. there are many scopes of expanding this work. for starters, we would like to refine our data augmentation pipeline to generate more well-formed instances. we would explore and compare the performance of cross-lingual and mono-lingual models. we also believe that, the dataset requires further analysis and should receive both quantitative and qualitative error analysis. in addition, we want to do elaborate ablation studies on the components of our systems. in this paper, we have majorly focused on transfer learning and so, in the future, we want to compare the performance of simpler statistical and shallow models with these deep models. another thing we don't mention empirically in this paper is the class-wise performance of each of our models. from general observation, we find that all the models perform the worst in identifying cw (creative works) tags, while simpler tags like per (person) and loc (location) was the easiest to tag. in future, we look forward to investigate more into the reasons behind this behaviors. finally, we only exploited a simple majority voting based ensemble scheme during this competition. for our future directions, we would also experiment on fusioning the layers of our models to develop a more sophisticated and informed ensembling scheme.