SALMA : A rabic Sense-Annotated Corpus and WSD Benchmarks (2023)

Abstract: SALMA, the first Arabic sense-annotated corpus, consists of ~34K tokens, which are all sense-annotated. The corpus is annotated using two different sense inventories simultaneously (Modern and Ghani). SALMA novelty lies in how tokens and senses are associated. Instead of linking a token to only one intended sense, SALMA links a token to multiple senses and provides a score to each sense. A smart web-based annotation tool was developed to support scoring multiple senses against a given word. In addition to sense annotations, we also annotated the corpus using six types of named entities. The quality of our annotations was assessed using various metrics (Kappa, Linear Weighted Kappa, Quadratic Weighted Kappa, Mean Average Error, and Root Mean Square Error), which show very high inter-annotator agreement. To establish a Word Sense Disambiguation baseline using our SALMA corpus, we developed an end-to-end Word Sense Disambiguation system using Target Sense Verification. We used this system to evaluate three Target Sense Verification models available in the literature. Our best model achieved an accuracy with 84.2% using Modern and 78.7% using Ghani. The full corpus and the annotation tool are open-source and publicly available at https://sina.birzeit.edu/salma/.

## introduction wsd aims to determine a word's intended meaning (sense) in a given context. wsd is underdeveloped in arabic due to the lack of sense-annotated datasets. this is in addition to the challenging nature of the wsd task due to the semantic polysemy of the words @xcite . for instance, the arabic word ( ↩ayn ) has sixteen meanings in the contemporary arabic dictionary @xcite . in the context ( r aytuh r ay āl↩ayn ), word ( ↩ayn ) refers to eye, while in ( šribt min ↩ayn ālmā ), it refers to water spring. similarly, the english word book as a noun has ten different senses in princeton wordnet @xcite , such as (a written work or composition that has been published), or (number of pages bound together). wsd has been considered a challenging task for many years @xcite , but it has recently gained more attention due to the advances in learning contextualized word representations from language models, such as bert @xcite and gpt @xcite . as glosses are short descriptions of senses @xcite , recent research has demonstrated promising results in wsd task by framing the problem as a sentence-pair (context-gloss) binary classification task, referred to as target sense verification (tsv), where the context is a sentence containing the ambiguous word @xcite @xcite . @xcite proposed an approach for arabic wsd (using tsv) based on contextgloss pairs extracted from the arabic ontology and lexicons and they achieved 84% accuracy, but this evaluation was done on a tsv dataset rather than a wsd evaluation using a sense-annotated corpus. additionally, @xcite presented an attempt for arabic word-in-context (wic) disambiguation using the dataset provided by the semeval shared task @xcite . this article presents salma, the first senseannotated arabic corpus consisting of about 34k tokens, which are manually annotated with senses. since there are no available sense inventories for arabic, we used two arabic lexicons as sense inventories: contemporary arabic dictionary ( āllġh āl↩rbyh ālm↩ās . rh ), hereafter we refer to as modern @xcite , and al-ghani al-zaher ( ālġny ālzāhr), hereafter we refer to as ghani (abul-azm, 2014). these two lexicons are part of the lexicon digitization project and lexicographic database at sinalab foot_0 @xcite @xcite @xcite . we introduce a novel sense-annotation framework (section 3), in which all candidate senses, from both lexicons, are scored to indicate their semantic relatedness to a token appearing within a context. the higher the score, the more semantically related the sense is. for better coverage, we annotated each token in our corpus using both lexicons independently and in parallel. the scores assigned to senses of the modern do not influence the scoring of the ghani senses. in addition, we also annotated our corpus using six types of named entities: person (pers), organization (org), geopolitical entity (gpe), location (loc), facility (fac), and currency (curr). the corpus was annotated by three linguists and we assessed the inter-annotator agreement (iaa) using 2.6% of the annotated words in the corpus. to establish a baseline for wsd in arabic, we developed an end-to-end wsd system, in which we benchmarked three available tsv models, with different settings. the best model resulted in 84.2% accuracy using modern and 78.7% using ghani. the main contributions of this paper are: the remainder of the article is organized as follows: section 2 highlights the related work, section 3 presents the corpus, section 4 describes the interannotator agreement, sections 5 and 6 present how the baselines are produced, we conclude in section 7 and outline the limitations and future work in section 8. ## related work we will first review related sense-annotated corpora, then we will review related sense inventories. one of the known english sense-annotated corpora is semcor @xcite , which is annotated using the princeton wordnet @xcite . it contains about 200k sense annotations for around 700k words, but not all words are senseannotated in the semcor corpus, especially multiword expressions, articles, and prepositions. the ancora corpus for spanish and catalan languages @xcite was collected from newspapers and consists of 500k words, but only 200k noun words are semantically annotated using the spanish wordnet. ancora also includes morphological, semantic, and syntactic annotations. tuba-d/z is a german annotated corpus, manually collected from newspapers and annotated using the german-net senses @xcite . tuba-d/z was later used as a gold standard for the wsd task by @xcite . the italian syntactic-semantic treebank (isst) is a corpus built for the italian language with 89,941 senseannotated words @xcite . the isst annotations cover five levels that are related to lexico-semantics such as orthographic, morpho-syntactic, semantic, and syntactic aspects. the ntu-mc corpus @xcite covers eight languages including thai, vietnamese, arabic, korean, indonesian, japanese, mandarin chinese, and english. however, the arabic version is not publicly available. this corpus was collected from short stories, essays, and tourism articles resulting in a total of 116k words, but only 63k words are annotated. kpwr, a polish corpus, contains text from multiple domains including science, law, religion, and press @xcite with a total of 438,327 words, but only 9,157 words are annotated using the polish wordnet @xcite . for arabic, the focus of research has been primarily on developing corpora for morphological and syntactic tagging @xcite rather than semantic and sense annotation, as noted by @xcite and @xcite . for instance, part of the ontonotes corpus @xcite covers limited semantic annotations for arabic using a small sense inventory of size 261 senses (150 verbs and 111 nouns). additionally, aqmar corpus @xcite is annotated with 25 super-sense labels representing broad semantic fields such as artifact and person, which can be considered as general types of named entities, rather than word-sense annotations. they annotated ~22k nouns out of 65k tokens corpus. in addition to the lack of sense-annotated corpora, arabic lacks reliable sense inventories. although there are some available semantic resources, they are not mature enough to be used as sense inventories. for example, the arabic wordnet @xcite ) contains about 10k senses, and the arabic ontology @xcite ) contains about 18k synsets. however, both resources cannot be used as sense inventories as they do not provide a complete set of senses for a given lemma (i.e., lexicon entry). the lexicographic database developed at birzeit university contains about 150 arabic lexicons @xcite , but these lexicons are not well-structured or suitable to be used as sense inventories @xcite . due to the lack of dependable arabic sense inventory, we decided to obtain a license to digitize and use two arabic lexicons as sense inventories, namely, modern @xcite and ghani (abul-azm, 2014). ## inter-annotation agreement (iaa) to evaluate our annotations, we selected 250 annotated words from each annotator @xmath0, a 2 , a 3 }, and assigned them to a different annotator to perform double annotations. this yielded a total of 750 words (2.6% of the annotated words) divided among three pairs of annotators, {(a 1 , a 2 ), (a 1 , a 3 ), (a 2 , a 3 )}. because our sense annotations contain scores (i.e., not discrete values), computing iaa is not straightforward. we chose to use various evaluation metrics especially those that take ranking into consideration. the iaa metrics used are: (i) kappa, (ii) linear weighted kappa (lwk), (ii) quadratic weighted kappa (qwk), (iv) mean average error (mae), and (v) root mean square error (rmse). kappa is usually used when the data is nominal (eugenio and glass, 2004), so we set a threshold on the score (≥60%) in the six categories to be able to calculate cohen's kappa. the senses with scores above or equal this threshold carry the intended meanings that map with the context of the targeted word (see section 3.2). nonetheless, a more suitable metric for ranked labels is either the lwk or qwk, as specified in the following equations, which we adopt from @xcite : both lwk and qwk take the distance between categories into consideration, where the distance is defined as the number of categories separating the two annotators' selection. the difference is that lwk calculates the distance linearly while qwk calculates it quadratically. for measuring the ranking error deviation among annotators we used mae and rmse. ## conclusion we presented salma, the first sense-annotated arabic corpus. the novelty of salma lies in utilizing two sense inventories and named entity annotations. in addition, instead of linking a word to one intended sense, we scored all semantically related senses of each token in the corpus. the quality of the annotations was assessed using various inter-annotator agreement metrics (kappa, lwk, qwk, mae, and rsme). to compute a wsd baseline using our corpus, we proposed to build an end-to-end wsd system using tsv, and evaluated this system using three different tsv models. the full corpus, annotations, and the tool, are open source and publicly available on github.