Context-Gloss Augmentation for Improving A rabic Target Sense Verification (2023)

Abstract: Arabic language lacks semantic datasets and sense inventories. The most common semantically-labeled dataset for Arabic is the ArabGlossBERT, a relatively small dataset that consists of 167K context-gloss pairs (about 60K positive and 107K negative pairs), collected from Arabic dictionaries. This paper presents an enrichment to the ArabGlossBERT dataset, by augmenting it using (Arabic-English-Arabic) machine back-translation. Augmentation increased the dataset size to 352K pairs (149K positive and 203K negative pairs). We measure the impact of augmentation using different data configurations to fine-tune BERT on target sense verification (TSV) task. Overall, the accuracy ranges between 78% to 84% for different data configurations. Although our approach performed at par with the baseline, we did observe some improvements for some POS tags in some experiments. Furthermore, our fine-tuned models are trained on a larger dataset covering larger vocabulary and contexts. We provide an in-depth analysis of the accuracy for each part-of-speech (POS).

## introduction there are three tasks in the literature that are related to semantic understanding of natural language: (i) word sense disambiguation (wsd), (ii) target sense verification (tsv), and (iii) word-in-context (wic). wsd is the most common task, which aims to disambiguate word's semantics. given a context (i.e., sentence), a target word in the context, and a set of candidate senses (i.e., glosses, meaning definitions @xcite ) for the target word, the goal of the wsd task is to determine which of these senses is the intended meaning for the target word @xcite . for example, the word ( ǧdāwl ) has two senses in arabic: tables ( ) and creek ( such semantic understanding tasks have been challenging for many years, but recently gained attention due to the advances in contextualized word embedding models @xcite . language models, specially @xcite , have made significant advancements in down-streaming nlp tasks. bert is a transformer-based model pre-trained on huge corpora @xcite . it can be fine-tuned on domain/task-specific data (e.g., pos tagging, wsd, tsv, and wic) to update its contextualized embeddings. the tsv task has been addressed by fine-tuning bert on context-gloss pairs as a sentence pair binary classification problem @xcite @xcite @xcite @xcite . however, the tsv task, similar to most nlp tasks, suffers from the knowledge-gain bottleneck, i.e., the lack of available quality datasets to train machine learning models. arabic is a low-resourced language @xcite and the only available context-gloss pairs dataset is arabglossbert @xcite . it consists of 167k context-gloss pairs, a relatively small dataset for fine-tuning bert on a tsv task. the positive pairs (60k) were collected from multiple arabic dictionaries @xcite @xcite as well as from the arabic ontology @xcite . the pairs were cross-related to generate 106.8k negative pairs and used to fine-tune a bert model, which achieved 84% accuracy. this paper aims to enrich the arabglossbert dataset by augmenting it using the back-translation technique, similar to the work done for english by @xcite . with data augmentation, we generate new arabic paraphrased contexts and glosses by translating the original data into english and back to arabic, using google translate api. the new augmented dataset consists of 352k context-gloss pairs. to answer the question of whether the back-translation enrichment improves the tsv accuracy, we conduct 13 experiments that compare the accuracy obtained using the original dataset with the accuracy obtained using different combinations of the augmented datasets. we, also, provide an in-depth analysis of the tsv accuracy for each part-of-speech, which was not provided in @xcite . the main contributions of this work are: the rest of the paper is organized as follows: section 2 reviews the related literature, section 3 describes the data augmentation, section 4 presents the experiments, section 5 presents the results and we conclude in section 6 with limitations and future work. ## related work tsv has proven to be an effective solution for the wsd in many state-of-the-art efforts. although some researchers did not use the term tsv, this notion was also referred to as glossbert or context-gloss binary classification @xcite . a tsv training dataset is typically a set of context-gloss pairs, each labeled with positive or negative. a pretrained language model can then be fine-tuned for sentence pair binary classification. this idea was first proposed for english as glossbert @xcite , where the training pairs were generated from a known semcor dataset @xcite with the target word, in context, marked up by a bert-specific signal to emphasize it in the learning phase. a similar effort in @xcite followed the glossbert technique. their addition is the use of back-translation for improving the english wsd. they used back-translation from english to german and back to english in order to bridge the knowledge-gain gap and provide more context-gloss pairs. they also used a mark-up signal to surround the target word with double quotations. only 2% improvement was achieved using back-translation. this paper aims to evaluate this idea for arabic. another idea was proposed in @xcite , in which a learning signal (special token [tgt]) was used, and bert was fine-tuned on sequence-pair ranking, the model selects the most related gloss given a context sentence and a list of candidate glosses. botha et al. (2020) used different mark-up signals in the form of open and close tags to emphasize the target word [e]target[/e] within the context sentence. for arabic, the tsv task was addressed in @xcite , which presents the arab-glossbert, a dataset of 167k context-gloss pairs labeled with positive or negative. first, 60k positive pairs were extracted from different arabic lexicons, then 106k negative pairs were generated automatically by cross-relating the positive pairs. the target word was marked-up with different learning signals. different arabic pre-trained models were fine-tuned, and the best model using arabert-v2 (antoun et al., 2020) achieved 84% accuracy. similar work for arabic was proposed in @xcite using a smaller dataset (15k positive and 15k negative) in which they used arabert-v2 and reported 89% f1-score but this performance was criticized in @xcite . an idea or question is progressing to research -officials sat discussing project proposals back-translated (english to arabic) - ## data augmentation nlp tasks, including tsv, typically suffer from knowledge acquisition. the importance of knowledge acquisition is increasing especially because most nlp tasks are currently tackled using pretrained neural models such as bert, which generally requires large data to fine-tune. if the training data is not sufficient, the model will encounter the problem of unseen vocabulary and contexts, which harms model accuracy @xcite . the linguistic resources that can be utilized for semantic understanding tasks are limited in arabic language. our assumption, for the tsv task, is that the more context-gloss pairs can be used during the training phase, the more vocabulary and more contexts will be covered, thus the better tsv accuracy. this is why researchers started to try new techniques for data augmentation in order to enrich the available dataset with more knowledge @xcite . for arabic, and in order to enrich existing arabic datasets, we propose to use the arabic-english-arabic back-translation, as illustrated in we only back-translated the arabglossbert training dataset (152,035 pairs). the testing dataset (15,172 pairs) was not back-translated, because it is used as an evaluation benchmark to compare the performance improvement between the original and augmented datasets. the original training dataset is not balanced with 55,585 positive pairs (36.6%) and 96,450 negative pairs (63.4%). to produce a more balanced dataset, we generated an additional 32,839 positive pairs by matching the original glosses with the new back-translated glosses increasing the number of positive pairs to 144,009. the 144,009 include 55,585 pairs from the original data, 55,585 pairs from back-translation and the added 32,839 pairs, resulting in a new dataset with 42.7% positive and 57.3% negative pairs. observations on google translate: first, although the quality of google translations was generally acceptable, there are wrong translations. however, we did not make any improvements or revisions to these translations, as the goal of this paper is to measure whether automated backtranslations can improve the accuracy of the trained models. second, the output of the google translation api was not always complete. in some cases it translates part of the input sentence. to overcome this challenge we used two techniques: 1) add special characters (.#) at the end of each sentence, if the special characters were translated back, then we know the translation reached the end of the sentence, 2) compare the length of the original and back-translated sentences and if the difference is significant, then this is an indication of incomplete translations. partial translations are reviewed manually. ## experiments this section presents 13 experiments to measure the impact of data augmentation using backtranslation on tsv model accuracy. the first experiment uses the original arabglossbert dataset, d 1 , (as a baseline), while the other experiments are conducted with different dataset configurations. in all experiments, we used the original test dataset 15,172 pairs (4,738 positive and 10,434 negative). in all experiments we fine-tuned arabertv2 (aubmindlab/bert-base-arabertv02, cc-by-sa) using the following hyperparameters: η = 2e -5 , batch size @xmath0, max sequence length of 512, warm-up steps 1,412 and number of epochs 4. the results of the 13 experiments are presented in ## discussion and conclusions we presented an approach to improve arabic tsv using automatic back-translation. we augmented an existing arabic tsv dataset, arabglossbert, by doubling its size with back-translated data using google translate api. to measure the impact of the data augmentation, we presented 13 experiments with different data configurations. although we did not outperform the overall performance of the baseline model, we did observe that some experiments such as d 6 outperformed the baseline on noun positive pair and verb negative pair classification. overall, our results are close to the results presented in @xcite , which used back-translation augmentation for english tsv and achieved only 2% f1-score improvement. nevertheless, we would like to note the following findings: