Multilinguals at S em E val-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languages (2022)

Abstract: We leverage pre-trained language models to solve the task of complex NER for two low-resource languages: Chinese and Spanish. We use the technique of Whole Word Masking (WWM) to boost the performance of masked language modeling objective on large and unsupervised corpora. We experiment with multiple neural network architectures, incorporating CRF, BiLSTMs, and Linear Classifiers on top of a fine-tuned BERT layer. All our models outperform the baseline by a significant margin and our best performing model obtains a competitive position on the evaluation leaderboard for the blind test set.

## introduction the named entity recognition (ner) task aims to identify the named entities in an input sequence and categorize them into certain predefined categories or class labels. the task of ner can be further broken down into two subtasks: 1) identification of the entity span and 2) classification of the identified entity span into predefined class labels. for example, in the sentence: new york city is the most densely populated major city in the united states., new york city is a named entity of type location with an entity span of 3 tokens. the most popular ner task in the english language is conll @xcite , which is widely used as a benchmark for most ner models. multiple models have been able to obtain sufficiently high performances in this task setting @xcite @xcite @xcite @xcite . the conll training set consists of 14,987 train sentences, which comprise 203,621 tokens in total for english data. the entity space consists of 4 different types of entity type labels @xcite * equal contribution to classify each named token. the english data was taken from the reuters corpus, which comprises of reuters news stories for one year. the training data source, and by extension the labeled named entities, comprises of majorly popular entities found in the general english textual content prevalent in the media. hence, these entities were easier to classify into the correct classes due to the large prevalence of training data. with the use of pre-trained transformer-based language models, which are already trained on a large unlabelled corpus of english text, this task became even less challenging, as the nature of textual structure in these corpora largely overlap with that of conll. however, this task becomes challenging in practical settings where a multitude of varieties of named entities is possible. many of these entities, like creative works (cw) and products (prod) have complex and ambiguous textual structural content. such complex named entities rarely appear even in the large training data sets, and the length and structure of the named entities keep changing. we investigate the task of complex, semantically ambiguous, and low-resource ner @xcite . this task is based on the complex ner, search query and code-mixing ner challenges introduced by @xcite and @xcite . the shared task of multiconer (stands for multilingual complex ner) adds additional challenge by introducing rarer label types (like creative work, product, etc.). another way to increase the difficulty of ner task is to perform it for low-resource languages. there is a significant dearth of both labeled and unlabelled data for such languages. the complexity is further enhanced by using rarer entity types in such languages. therefore, the scarcity of training data, along with the rarity of entity types, makes it difficult for the models to perform better in the lowresource setting. the shared task of multiconer introduces datasets in multiple low-resource lan-guages. we leverage large pre-trained language models trained on low-resource language corpora to obtain competitive performances in the low-resource and complex ner setting. we show that simpler architectures successfully outperform other heavier counterparts. we use standard bert+crfbased models to obtain high performances in the evaluation set. we experiment on two lowresource datasets: spanish and chinese. the code is available at https://github.com/ amitpandey-research/complex_ner we compare multiple architectures on the test and validation set of the shared task. the organisers provide a baseline of xlm-roberta model @xcite finetuned on the training dataset of the specified language for the given task. in this paper, we treat the finetuned xlm-roberta as baseline (two separate models are trained for spanish and chinese language) and compare the performances against our models. all our models beat the baseline by a significant margin. we describe the prior research work done with respect to both general and low-resource ner tasks in section 2. we provide the formal task description in section 3, the dataset details in section 4, the method and the model architecture in section 5. we provide details about the experimental implementation in section 6. we discuss the results obtained and error analysis in sections 7 and 8 respectively, and finally, we conclude the paper in section 9. ## related work the task of low-resource ner has been investigated before by multiple researchers. this line of research focuses mainly on leveraging the crosslingual contextual information obtained from lowresource languages. @xcite use crosslingual knowledge transfer to train the ner model for the low-resource target language. @xcite use bilingual dictionaries to tackle the task of low-resource ner. @xcite proposes a bayesian graphical model approach to improve performance on ner tasks. ner models often use gazetteers (list of named entities) to improve performance in ner tasks. @xcite ) creates soft-gazetteers for low-resource languages, leveraging english knowledge bases. @xcite focuses on an unsupervised approach for ner for to circumvent the label scarcity problem in low-resource lan-guages. @xcite leverages multilingual transfer learning from multiple languages for low-resource ner tasks. @xcite uses distant supervision in the low-resource setting for ner. there are multiple approaches that have been undertaken in the recent past to improve the stateof-the-art in ner tasks. @xcite uses concatenation of embeddings to outperform the state-of-the-art in ner tasks, as they infer that concatenation of embeddings leads to a better word representation. their method automates the process of finding meaningful embeddings to concatenate for improved performance. @xcite propose a co-regularization framework for entity extraction comprising of multiple models with different architectures but different parameter initializations. this helps to tackle overfitting of large neural network-based models on low-resource training data for ner. @xcite use document-level features to improve information extraction on entity-centric tasks. ner and relation extraction are the core information extraction tasks in nlp. @xcite models this as a span-pair classification problem, and they further improve the pair representations by considering the dependencies between the spans (pairs) by strategically packing the markers in the encoder. @xcite proposes a novel entity-aware self-attention framework for transformer based models for ner. @xcite extracts document-level context for sentences for which document information is absent. they treat the sentence as a query and use a search engine to extract the document level contextual information. @xcite uses multiple neighbouring sentences as the contextual information for ner. pre-trained language models for ner : ever since the introduction of bert @xcite , transformer based pre-trained language models have effectively utilized transfer learning for downstream nlp tasks. ner has been traditionally modeled as a sequence labeling problem. @xcite proposed a bidirectional lstm with a crf layer on top for classifying tokens as entities. (jadhav, 2020) use a pretrained bert model with a crf layer on top for performing ner on the dailyhunt news dataset. we use a bert-based model with a crf layer on top and achieve competitive performance on low-resource ## task description in this task, we attempt complex ner for two lowresource languages: spanish and chinese. this task presents additional challenges in the form of test instances consisting of short search queries and low-context sentences. for this task, the systems had to identify the b-i-o format (ramshaw and marcus, 1999) (short for beginning, inside, outside) tags for six ner entity type labels: 1) person, 2) product, 3) location, 4) group, 5) corporation, and 6) creative work. the description of these labels is shown in ## dataset the multiconer dataset @xcite ) consists of multiple low-resource languages. we consider chinese and spanish languages in this paper. for the monolingual track, the participants have to train a model that works for a single language. we fine-tune the language model on the train set to obtain predictions on dev and test set. the labels from the blind test set are not disclosed. the dataset follows a bio tagging scheme and there are 6 entity types in the label space. the statistics for the chinese and spanish datasets in the monolingual track for the train and dev set are provided in ## system overview at first, we pre-train the bert language model on unlabelled corpora for the target low resource language. for chinese, we use the strategy outlined by @xcite . bert uses the wordpiece tokenizer @xcite to split tokens into smaller fragments. it is easier for the masked language model to predict these masked fragments. however, for the chinese textual texture, the chinese characters are not formed by alphabet-like symbols, so the wordpiece tokenizer is unable to split the words into small fragments. hence, we use the chinese word segmentation (cws) tool to split the text into separate words and then use whole word masking (wwm) strategy for the masked language model objective. in comparison to masking small fragments, this whole word masking strategy makes it harder for the model to predict whole masked words, leading to more robustness. for the spanish variant, we adopt the strategy outlined by @xcite . similar to @xcite , they use the strategy of whole word masking for pre-training bert language model on unlabelled spanish corpus. we adopt the strategy of finetuning these pretrained bert models on the downstream ner task for each language. ## implementation details we also experiment with a cyclic learning rate between 1e -4 to 1e -6 to avoid getting stuck in local minima. the size of each of the fc layers in the bert+linear model is 512. we validate the results of all models using our validation dataset. the hidden layer size of bilstm used in the bert+bilstm+crf model is 256. ## results we compare the performances of all models in the low-resource setting for both chinese and spanish languages. from tables 3 and 4 we observe that the bert+crf model performs the best across both languages on validation set. we choose the best performing model to evaluate our results on the blind test set. the baseline model chosen by the organisers of this task is xlm-roberta (conneau et al., 2019)(base model). it is pre-trained on 2.5tb of filtered commoncrawl data containing 100 languages @xcite . our approach beats the baseline by a significant margin and outperforms multiple models in the competition. we present the precision, recall and f1 scores for all 3 models in the tables 3 and 4 for spanish and chinese language respectively. we also compare the results between the baseline and our models for the validation dataset in the tables 5 and 6 . for the spanish language, we observe that the bert+crf (0.8117 f1 score) beats bert+linear (0.8096 f1 score) by a slender margin. this can be attributed to the addition of the crf layer, which exploits inter-token dependencies. the bert+bilstm+crf model is much heavier with a larger number of parameters and overfits the training dataset due to the smaller number of training instances. for the chinese language, we observe that both the bert+crf (0.8564 f1 score) and bert+bilstm+crf (0.8558 f1 score) beat bert+linear (0.846 f1 score). our models outperform the baseline for both the languages. our best performing model bert+crf beats the baseline f1 score by around 5% for spanish and by around 10% for chinese. bert+crf model in the evaluation phase are shown in ## error analysis we perform error analysis on all 3 different models. we qualitatively analyze the predictions on the validation dataset for both languages. as the final evaluation test set in blind, we are unable to perform analysis on the same. we find that the labels grp (group), prod (product), and cw (creative work) are the most inaccurately predicted labels for the spanish models. this conforms to our hypothesis that the long-tailed nature of these entities (which means the frequency of occurrence of such entity types in the general literature of the target language is rare). hence, the model has the most difficulty in recognizing these entities from the contextual sentences. the other label types are more common and were present in the conll dataset as well. we also notice that the bert+linear does marginally better than bert+crf on predicting such labels (for the spanish language), despite it not being the best performing model overall. this can be attributed to it being a lighter model, imparting it the capability of generalizing better while training on a relatively lower amount of training instances. bert+crf benefits from having crf along with the lower number of parameters compared to the bert+bilstm+crf model. this results in it having a better performance compared to both the other models. the drop in the performance of the model on the blind test dataset can be attributed to the model not generalizing well to handle instances of questions and short search queries in the additional test set. ## conclusion and future work we have introduced strong improvements over the baseline for the shared task of complex ner for low resource languages. we leverage the whole word masking objective to obtain a better performance in this low-resource setting. we perform extensive experiments and find that simple bert-crf based models perform strongly against other heavier models even in such low resource semantically ambiguous setting as evident by the final evaluation rankings. we find this approach to give a higher performance as it is able to utilize the contextual information from a sequence of tokens and learn inter-token dependencies to accurately predict the named entity labels. we also conduct qualitative error analysis and describe our findings. for future work, we aim to leverage these findings to circumvent the label scarcity problem in other low-resource languages and code mixed data.