A rab G loss BERT : Fine-Tuning BERT on Context-Gloss Pairs for WSD (2021)

Abstract: Using pre-trained transformer models such as BERT has proven to be effective in many NLP tasks. This paper presents our work to fine-tune BERT models for Arabic Word Sense Disambiguation (WSD). We treated the WSD task as a sentence-pair binary classification task. First, we constructed a dataset of labeled Arabic context-gloss pairs (~167k pairs) we extracted from the Arabic Ontology and the large lexicographic database available at Birzeit University. Each pair was labeled as True or False and target words in each context were identified and annotated. Second, we used this dataset for fine-tuning three pre-trained Arabic BERT models. Third, we experimented the use of different supervised signals used to emphasize target words in context. Our experiments achieved promising results (accuracy of 84%) although we used a large set of senses in the experiment.

## introduction word sense disambiguation (wsd) aims to determine which sense (i.e. meaning) a word may denote in a given context. this is a challenging task due to the semantic ambiguity of words. for example, the word "book" as a noun has ten different senses in princeton wordnet such as "a written work or composition that has been published" and "a number of pages bound together". wsd has been a challenging task for many years but has gained recent attention due to the advances in contextualized word embedding models such as bert @xcite , elmo @xcite and gpt-2 @xcite . such language models require less labeled training data since they are initially pre-trained on large corpora using self-supervised learning. the pre-trained language models can then be fine-tuned on various downstream nlp tasks such as sentiment analysis, social media mining, named-entity recognition, word sense disambiguation, topic classification and summarization, among others. a gloss is a short dictionary definition describing one sense of a lemma or lexical entry @xcite . a context is an example sentence in which the lemma or one of its inflections (i.e. the target word) appears. in this paper, we aim to fine-tune arabic models for arabic wsd. given a target word in a context and a set of glosses, we will finetune bert models to decide which gloss is the correct sense of the target word. to do that, we converted the wsd task into a bert sentence-pair binary classification task similar to @xcite @xcite . thus, bert is fine-tuned on a set of contextgloss pairs, where each pair is labeled as t rue or f alse to specify whether or not the gloss is the sense of the target word. in this way, the wsd task is converted into a sentence-pair classification task. one of the main challenges for fine-tuning bert for arabic wsd is that arabic is a low-resourced language and that there are no proper labeled context-gloss datasets available. to overcome this challenge, we collected a relatively large set of definitions from the arabic ontology @xcite and multiple arabic dictionaries available at birzeit university's lexicographic database @xcite then we extracted glosses and contexts from lexicon definitions. another challenge was to identify, locate and tag target words in context. tagging target words with special markers is important in the fine-tuning phase because they act as supervised signals to highlight these words in their contexts, as will be explained in section 5. identifying target words is not straightforward as they are typically inflections of lemmas, i.e. with different spellings. moreover, locating them is another challenge as the same word may appear multiple times in the same context with different senses. for example, the word ( ) appears two times in this context ( ) with two different meanings: went and gold. we used several heuristics and techniques (as described in subsection 3.3) to identify and locate target words in context in order to tag them with special markers. as a result, the dataset we constructed consists of about 167k context-gloss pair instances, 60k labeled as t rue and 107k labeled as f alse. the dataset covers about 26k unique lemmas (undiacritized), 32k glosses and 60k contexts. we used this dataset to fine-tune three pretrained arabic bert models: arabert (antoun et al., 2020), qarib @xcite and camelbert @xcite foot_0 . each of the three models was fine-tuned for context-gloss binary classification. furthermore, we investigated the use of different supervised signals used to highlight target words in context-gloss pairs. the contributions of this paper can be summarized as follows: the remainder of this paper is organized as follows. section 2 presents related work. section 3 describes the constructed dataset and our methodology to extract and label context-gloss pairs, and splitting the dataset into training and testing sets. section 4 outlines the task we resolved in this paper and section 5 presents the fine-tuning methodology. the experiments and the obtained results are presented in sections 6 and 7 respectively. finally, section 8 presents conclusions and future work. ## related work recent experiments in fine-tuning pre-trained language models for wsd and related tasks have shown promising results, especially those that use context-gloss pairs in fine-tuning such as @xcite @xcite . @xcite proposed to fine-tune bert on context-gloss pairs (@xmath0, no}) for wsd, such that the gloss corresponding to the context-gloss pair candidate, with the highest output score for yes, is selected. yap et al. (2020) proposed to group context-gloss pairs with the same context but different candidate glosses as 1 training instance (groups of 4 and 6 instances). then, they proposed to fine-tune bert model on group instances with 1 neuron in the output layer. after that, they formulated wsd as a ranking/selection problem where the most probable sense is ranked first. others also suggested to emphasize target words in context-gloss training instances. huang et al. (2019); botha et al. (2020); lei et al. (2017); yap et al. (2020) proposed to use different special signals in the training instance, which makes the target word "special" in it. as such, @xcite proposed to use quotation marks around target words in context. in addition, they proposed to add the target word followed by a colon at the beginning of each gloss, which contributes to emphasizing the target word in the training instance. @xcite proposed to surround the target word in context with two special [tgt] tokens. in contrast, @xcite ; @xcite proposed to surround the target word in context with two different special tokens as marks of opening and closing. in this paper, we investigate the use of different types of signals to emphasize target words in context for arabic wsd. @xcite fine-tuned two bert models on a small dataset of context-gloss pairs, consisting of about 5k lemmas, about 15k positive and 15k negative context-gloss pairs. they claimed an f1-score of 89%. however, this result is not reliable. after repeating the same experiment, we found that the majority of the context sentences used in the tests were already used for training. in this paper, we carefully selected the test set such that no contexts are used in both the training and the test sets. additionally, we used a much larger sense repository (26k lemmas, 33k concepts and 167k context-gloss positive and negative pairs), which makes the task more challenging. other works related to arabic wsd includes the use of static embeddings such as context and sense vectors @xcite , stem2vec and sense2vec @xcite , lemma2vec @xcite , word sense induction (alian and awajan, 2020), or using fasttext @xcite . @xcite reviewed arabic wsd approaches until 2018. ## dataset construction this section describes how we constructed a dataset of labeled arabic context-gloss pairs (see examples of pairs in figure 1 ). we extracted the contextgloss pairs from the arabic ontology and multiple lexicons in the birzeit university's lexicographic database. the extracted pairs are labeled as t rue, and based on these t rue pairs, we generated the f alse pairs. additionally, we identified the target word in each context and tagged it with different types of markers. ## methodology to address the binary context-gloss classification task, we experimented with four variations of the context-gloss pairs. the idea is to investigate using different supervised signals around target words to give them special attention during the fine-tuning. figure 2 illustrates these four variations. in variation 1, context-gloss pairs were left intact, without any signal. in the other three we fine-tuned three arabic pre-trained models: arabert (antoun et al., 2020), qarib @xcite and camelbert @xcite using our training dataset described in section 3. before fine-tuning arabert, we used the pre-processing method used in @xcite to pre-train version 2 of their model. before finetuning camelbert and qarib models, we used the pre-processing method used in @xcite to pre-train the camelbert which consists in the normalization of alif maksura ( ), teh marbuta ( ), alif ( ) and undiacritization. since bert has a max length limit of tokens equal to 512, we limit the length of each training instance (i.e. context-gloss pair) with a maximum of 512 tokens. given, for example, the tokenizer used in arabertv02, only 216 pairs are larger than 512 tokens out of the 167,207 pairs in our dataset. instances shorter than 512 were padded to the max length limit. the bertforsequenceclassification model architecture is used in fine-tuning the three arabic bert models. the last hidden state of the token [cls] is used for the classification task. the linear layer in the output consists of two neurons for the t rue and f alse classes. ## experiment setup we selected the base configuration of arabertv02, qarib, and camelbert models due to computational constraints and as larger models do not necessitate better performance @xcite . we used the huggingface "trainer" class in the fine-tuning. we performed a limited grid search to find a good hyperparameters combination then we fine-tuned each of the three models using the optimal configuration: initial learning rate of 2e-5, warmup steps of 1412 with a batch size of 16 over 4 training epochs. all other hyperparameters were kept at their default values. we used a single tesla p100-pcie-16gb in fine-tuning models. ## results and discussion this section presents the results of two experiments. as arabertv02 outperformed other models in the first experiment, it has been chosen for conducting a second experiment in which we fine-tuned on variation 1 (intact context-gloss pairs), variation 3 (two [unused0] tokens around the target word in context-gloss pairs) and variation 4 ([unused0] and [unused1] tokens around the target word in context-gloss pairs). reported results in ## conclusion and future work we presented a large dataset of context-gloss pairs (167,207 pairs) that we carefully extracted from the arabic ontology and diverse lexicon definitions. each pair was labeled as t rue and f alse and each target word in each context was annotated and tagged. we used this dataset to fine-tune three arabic bert models on binary context-gloss