A lex U - AIC at W ojood NER shared task: Sequence Labeling vs MRC and SWA for A rabic Named Entity Recognition (2023)

Abstract: Named entity recognition (NER) is one of many challenging tasks in Arabic Natural Language Processing. It is also the base of many critical downstream tasks to help understand the source of major trends and public opinion. In this paper, we will describe our submission in the NER Shared Task of ArabicNLP 2023. We used a simple machine reading comprehension-based technique in the Flat NER Subtask ranking eighth on the leaderboard, while we fine-tuned a language model for the Nested NER Subtask ranking third on the leaderboard.

## introduction arabic internet content has witnessed a leap in the past years which encourages the community to explore a large spectrum of tasks. named entity recognition (ner) is one of the fundamental tasks that can be included in many applications. it uses semantic text features to identify names, organizations, locations, and many other mentions in a given text. this information can be used to identify social media trends @xcite , summarize articles @xcite or as a component in question answering @xcite and machine translation @xcite . many techniques to solve the ner problem have emerged and can be classified into three categories: sequence labeling, span-based classification, and sequence-to-sequence generation. sequence labeling mainly classifies the entity type of each word or token. this category has been investigated widely in high and low-resource languages @xcite . for the span-based models, they depend on generating all possible spans in the input and classifying each span @xcite . for the sequenceto-sequence models, a decoder is required to start generating the tag for each token @xcite , or generate all found tags with their span indices @xcite . apart from the mentioned categories, other methods have been proposed that include contrastive learning as in @xcite @xcite . lots of challenges exist for the arabic ner problem, i.e. the lack of a large well-annotated dataset or language-dependent problems @xcite . these issues may have restricted the exploration of all the mentioned techniques. the sequence labeling technique has been the most investigated @xcite . many encoders have been deployed starting from recurrent neural networks till the transfer learning from pre-trained language models like arabert (antoun et al., 2020) and its variants. recently, there were attempts to explore the multitasking track as in @xcite . in this paper, we are trying to explore the machine reading comprehension method (mrc) @xcite and compare it to the sequence labeling technique with a pre-trained language model as a baseline. mrc injects a prompt alongside the input text to help the model better exploit the features that will aid it in answering the prompt. the model is guided not just to perform sequence labeling but to understand the meaning behind it and maybe better generalize to uncommon cases. we describe our submission, to the wojood ner shared task @xcite , which covers using the pre-trained model jaber @xcite in a sequence labeling technique, and formulating the arabic ner task as a machine reading comprehension task following @xcite ). further, we followed @xcite on averaging the best checkpoints of the flat ner model producing our best result. it contains three splits: train, validation, and test with 16,817, 3,133, and 5,990 sentences respectively. the train entities distribution can be found in figure 1 . to use this dataset in an mrc model, it needed some preparation. we created a new dataset where each sample includes the following fields: the context (the input text), the query (the entity type), and the start and the end positions of the answer to the queried type. these positions are indicated using the index of the start and end entity word in the context respectively. the sizes of the dataset splits went up to 353,157, 65,793, and 125,790 since there are 21 new data samples for each sentence. we tried two different types of queries: the arabic translation of the labels (keywords) using google translate 1 and the annotation guidelines of each entity as mentioned in @xcite . examples of the dataset using the annotation guidelines queries can be shown in ## system this section will describe our two approaches: the sequence labeling technique and the problem formulation as a machine reading comprehension problem. ## results we started with the sequence labeling technique in both tasks. the results with jaber on the validation set are higher in both flat and nested tasks hence we used them as our first test submission. we tried to enhance the results by employing the mrc technique. we tried the two backbones arabert and jaber for both tasks. in the flat ner task, the results improved, unlike the nested task. to further improve the results we tried performing the swa technique which gave us the best results on the flat ner task. a table of the conducted experiments and results can be shown in ## discussion by inspecting the model performance on the validation set. we found that the flat and nested models perform poorly in the quantity, website and product classes. this is due to the insufficient number of data samples as well as the inconsistency in the annotations. an example for the inconsistency: 'vodafone cash and orange cash', these are two equivalent entities but the ground truth label for 'vodafone cash' is organisation while the label for 'orange' is product. for the flat ner task, the two best-performing models are mrc (arabert) and mrc(jaber) with stochastic weighted averaging. we analyzed the output to find the cases mentioned in section 3.4. we found 100 words with different b-tag and i-tag labels amongst them 51 words with different i-tag-only labels and 12 words with different b-tag-only labels in the arabert-based model. an example of the b-tag confusion is the word 'google' where it is assigned the labels b-org and b-website. the jaber-based model has 163 words with conflicting b-tag and i-tag labels, amongst them 68 with conflicting i-tags only and 38 with conflicting b-tags only. we wanted to analyze the efficiency of our priority-based selection scheme. we compared it with choosing randomly the b-tag label amongst the conflicting ones. we conduct 5 runs, calculate the validation f1-score at each time, and average them. for the arabert-based model, we find the priority scheme to score 0.90642 and the random scheme to score 0.90675. for the jaber-based model, the priority scheme produces 0.90173 while the random scheme scores 0.90155. we notice that the more confusion in the model output, the more the random scheme fails. the first model had 12 conflicting b-tag words while the second had 38. hence, to ensure determinism and reproducibility, we decided to follow the priority scheme. as a plan, we can choose a better scheme that would keep the model confidence scores for all 21 inferences for the sentence and compare conflicting ones to choose the b-tag with the highest score. the flat ner results show that the effect of adding swa to the arabert-based mrc model is greater than adding it to the jaber-based model. we investigated the f1 score of each class for all the checkpoints involved in swa. for the jaberbased models, no checkpoint could have enhanced greatly the scores of the best checkpoint. on the other hand, other checkpoints included in the arabert-swa model perform better in the cardinal, gpe, money, time, and website classes which corrected the labels on 32 samples. meanwhile, there was a slight degradation in language, law, location, occupation, product, and quantity classes which yielded the mislabeling of 9 samples. the degradation is not effective though due to the sparsity of these classes in the dataset. in total, there was an improvement in the performance over the best checkpoint. ## conclusion arabic ner has been an underexplored problem, the lack of a large dataset can be one of the reasons. in this work, we investigate the effect of applying the machine reading comprehension technique to the arabic ner problem. we tried two different types of prompts and concluded that the label description is more beneficial than inserting keywords as queries. we compared mrc and the sequence labeling technique. we also investigated the effectiveness of applying the stochastic weighted averaging technique. we found that the results are comparable between the sequence labeling and mrc and either of them can be used in ner. many other methods still exist and can be tackled and finetuned for arabic usage.