The Extended A rabic W ord N et: a Case Study and an Evaluation using a Word Sense Disambiguation System (2019)

Abstract: Arabic WordNet (AWN) represents one of the best-known lexical resources for the Arabic language. However, it contains various issues that affect its use in different Natural Language Processing (NLP) applications. Due to resources deficiency, the update of Arabic WordNet requires much effort. There have only been only two updates it was first published in 2006. The most significant of those being in 2013, which represented a significant development in the usability and coverage of Arabic WordNet. This paper provides a study case on the updates of the Arabic WordNet and the development of its contents. More precisely, we present the new content in terms of relations that have been added to the extended version of Arabic WordNet. We also validate and evaluate its contents at different levels. We use its different versions in a Word Sense Disambiguation system. Finally, we compare the results and evaluate them. Results show that newly added semantic relations can improve the performance of a Word Sense Disambiguation system.

## introduction natural language processing (nlp) is part of computer linguistics, which is also part of artificial intelligence. there are many disciplines in nlp. information extraction is one of them. it can be text mining, information retrieval, named entity recognition. . . all these disciplines require lexical and semantic resources to proceed and generate satisfactory results. the more inclusive the resource, the more accurate the results will be. lack of resources, especially for less-resourced language such as arabic, has always been a persistent problem. one of the reliable resources for the arabic language is arabic wordnet (awn) @xcite . princeton wordnet (pwn) @xcite , english wordnet or simply wordnet is the original and most developed of all wordnets. from its first publication, it proved its reliability with various nlp tasks. many researchers were inspired by its usability and made a wordnet for their own languages. now we have more than 77 wordnet foot_0 , which awn is one. researches now are aiming either to create new wordnets for other languages (or dialects) or improve existing ones. creating new wordnets can be done by gathering an exhaustive repository of meanings and senses, e.g. dictionary or corpora, and assigning all words for each sense. this approach is called the merge approach @xcite . more common is the 'expansion' approach. it consists of translating the core of pwn foot_1 and extending it through more concepts related to the language. this is called the top-down approach. awn has followed this approach. generally speaking, a wordnet is a group of synsets interconnected with different relations. a synset is a set of synonyms. in other words, it is a group of words that share the same meaning. relations can be synonymy, antonymy, hyponymy, meronymy. . . the enrichment of a wordnet can follow the axe of synsets or relations. besides, the coverage in terms of synsets with diverse relations can be very useful in many nlp applications, especially question answering (qa) and word sense disambiguation (wsd). numerous approaches present themselves to construct and extend wordnets, from statistics to word embeddingbased approaches @xcite . even without enrichment, awn showed great results with several nlp applications like infor-mation retrieval @xcite and query expansion @xcite even for e-learning applications @xcite . but, awn has seen many attempts to enrich its content with different approaches, either by adding new synsets or new entities or even new specificity of the arabic language like broken plurals foot_2 @xcite @xcite @xcite . despite these efforts, awn remains inadequate to the needs of complex modern systems. there remains a huge gap between the contents of awn and the arabic language itself, and also between awn and other wordnets like pwn. this paper cites several significant programmes that have been undertaken to improve the contents of awn. this paper also seeks to shine a light on the semantic relations of awn and their importance for improving the performance of nlp applications. finally, the paper provides an overview of tests we have undertaken with three versions of awn in a concurrent nlp application. the paper is structured as follows. the next section is an overview of the various updates and extensions of the awn along a detailed discussion about its content. section 3 summarises most of the significant research undertaken to enrich the semantic relations in awn. section 4 discusses the procedures that we follow to validate the newly added relations. section 5 presents the conducted tests to show much the enriched awn can affect a wsd system. finally, section 6 will be our conclusion with some future works. ## versions of arabic wordnet the awn project started in 2006. the goal was to build a freely open source lexical database for the modern standard arabic available for the nlp community @xcite . by that time, it has 9,698 synsets, corresponding to 21,813 words. synsets were linked by 6 different types of semantic relations (hyponymy, meronymy, etc.), in a total of 143,715 relations @xcite . entities are distinguished by their part of speech pos: noun, verb, adverb, or adjective. synsets are linked to their counterpart in pwn and the suggested upper merged ontology (sumo) via the so-called interlingual index (ili) @xcite . in 2010, a second version has been published by rodriguez et al. @xcite . it has 11,269 synsets corresponding to 23,481 words with 22 types of semantic relationships in a total of 161,705 relations. this version has a browser written with java that has an update and search functions @xcite . this version is rich with more specific concepts related to the arabic cultures like named entities and the arabic language like broken plurals @xcite . several researchers have taken advantage of this version in most of their work in different areas of nlp to improve the performance of their systems. recently, an extended version has been published in 2015 by @xcite . this version is seen as an improvement of the coverage and usability of the previous version of awn @xcite . it includes 8,550 synsets which correspond with 60,157 words, among which we find 37,342 lemmas, 2,650 broken plurals, and 14,683 verbal roots. regragui et al. @xcite changed the structure of the database to the lexical markup framework (lmf) @xcite , the iso standard for nlp abd machine-readable dictionary (mrd) lexicons. they made it publicly available and ready to use from the open multilingual wordnet foot_3 . first of all, we notice that the number of entities and synsets in pwn is very high compared to all the versions of awn. in versions 1 and 2 (v1 and v2), we find that the number of entities is proportional to the number of synsets which is approximately two to three times the number of entities, which is not the case in the extended version (ex.v). on the one hand, v2 contains more synsets and fewer entities than the ex.v. on the other hand, v2 has 11,269 synsets connected with 161,705 relations and ex.v has only 8,550 synsets connected with only 41,136 relations. by comparing the number of relations in pwn with v2, we note that v2 is nearly rich in terms of connections between synsets. as a result, we can say that ex.v is more affluent than the other versions of awn in terms of synsets but impoverished in terms of relations. abouenour et al. @xcite put a focus on the entities, in this paper, we focus on the relations between them. ## related works until now, there are several attempts to enrich the awn using different methods and approaches. most of the works focused on the improvement of the number of entities and synsets @xcite @xcite @xcite @xcite . the main reason behind those works is the richness of the arabic language. one study on both arabic and english gigaword corpus has shown that to deal with the same linguistic content of 100,000 words in english, it takes approximately 175,600 words in arabic @xcite . in other words, one english word can be processed with approximately two arabic words. thus, resource-based applications expect more coverage of the arabic language. in contrast, the work on the relations of awn is much less. boudabous et al. @xcite proposed a linguistic method based on two phases. the first one defines morpho-lexical patterns using a corpus developed from arabic wikipedia. the second one uses the patterns to extract new semantic relations from the entities in awn. a linguistic expert has validated the obtained relations. while some of the new relations were good others were not -for various reasons, including the size of the corpus and the patterns applications. in our first work on the awn (batita and zrigui, 2017) we focused on the enrichment of antonym relations. as many studies have shown that the antonym relation is universal, but, it has been noted that there are different perspectives towards this lexical relation in different cultures @xcite . antonyms detection, in general, is a tough task for the nlp community. after a deep study, we have found that the extended version of awn has only four types of relations. one of them is the antonym relations with only 14 pairs. this work has been concentrated on the extended version of awn because it has been proved by abouenour et al. @xcite that it has given excellent results when testing in a q/a system. we proposed a pattern-based approach to extract new antonym relations from the entities of awn. for that, they extract patterns from an arabic corpus and used a corpus analysis tool to recognize automatically the antonym pairs from other pairs. the analysis tool is the sketch engine @xcite . it has many useful metrics like the logdice which gives a higher score to most likely related pairs. the results were filtered using the logdice and the validation was manual. after that our next step was the derivational relations in @xcite . by that, we tackled another matter of the arabic language which is the morphological aspect. the derivational and morphological problem has been a subject in different wordnet from other languages @xcite @xcite . generally speaking, and when it comes to studying a language aspect, rule-based approaches seem the more promoting one because they rely on linguistic rules verified by an expert or by a native speaker. based on that, wz relied on that kind of approach to add new derivational relations between entities in awn. we studied the derivational aspect of the arabic language to make a set of transformation rules. those rules are based on the pos switch, for example between the verb kataba foot_4 (write) and the noun kaātibun (writer) there is a has-derivedverb relation. rules are made by an expert and validated carefully to guaranty the precision of the results. for more information on the transformation rules see @xcite . in the end, we got 8 different relations with different frequencies. the validation of the rules and the finale results has been made by a lexicographer. the knowledge-based systems in general and wordnet-based systems specifically shown good results when they used a rich wordnet with as many relations as possible @xcite @xcite . yet, the use of a wordnet, in general, has shown a great result in different areas of nlp such as humor detection @xcite and human feelings @xcite even in the cybercrime investigation @xcite . given a sufficiently large database with many words and connections between them, many applications are quite capable of performing sophisticated semantic tasks. that is why work on the relations in awn has to increase because richer resource can achieve significant results in a real-world nlp application. evaluation and validation of the relations need to be considered as essential and continuous steps to guaranty the credibility of a resource. basically, validation can be done either manually by verifying each relations individually or automatically using different approaches. in the next, we will describe how we validated the newly added relations in the previous updates. ## validation of the new relations in arabic wordnet the previously cited works on the enrichment of the relations in awn confronted different parts of the arabic language, in general, using different methods and approaches. we will focus on the extended version published by @xcite and the new relations that we already added @xcite . since many relations need to be validated (12), we initially used an automatic approach, which we developed. while the majority of the new relations are specific to the arabic language (8 derivational relations), with the developed approach we will be working only on the three general relations: hyponyms, hypernyms, hasinstance, isinstance, and synonyms. we were inspired by the aspect of the dictionary and the construction of wordnets since they are based on the synonyms and the is-a relations (hyponym/hypernym). our automatic approach says that 'if a word w has a dictionary definition and belongs to a synset s with other words w 1 , . . . , w k then there is a strong probability that w mentions one or more of w k in her definition and/or other words (w k ) from the synonym/hypernymy/instance of s'. an example will simplify the point of the view: as we can see, @xmath0. if so, then the relation is validated, otherwise it should be reviewed. we collect all the definition of the words that have one of the three relations from different dictionary foot_5 . all definitions are stored in one file. the file is structered as a table and each line contains one definition per word. stop words are eliminated and remaining words have been lemmatized 7 . finally, we applied our idea and we got the results of each relation as described in as a start-up, the first approach yields to promoting accuracy. to guaranty efficiency and high confidentiality, a second validation is done manually by native speakers and a linguistic expert. the relations accuracy (%) hasinstance/isinstance 89,1 hypernym/hyponym 86,2 synonym 96,7 as an example, the two words wt . n (prepare to do) and nz . m (organize) are connected by the hyponym relation. native speakers suggested that it should be eliminated but the expert said otherwise. so, the expert takes the final decisions. if a relation is obvious and does not exist, the expert can add it, as well as he can eliminate it otherwise. besides his knowledge, the decisions of the expert are based on the following conditions: in the end, we got 81% correct relations, 5% wrong relations, 12% partially wrong relations (one of the pair of the words is wrong), and 2% of the words with no relations at all. most of the wrong relations were found in the relations that are specifics to the arabic language, like instrument and relatedness because they are based on transformation rules. sometimes, words (irregular ones) that share this kind of relations do not follow any transformation rules. some changes have been made by the linguistic expert regarding the 12% of the relations that are partially wrong by either changing one of the two words or replacing if the word does not exist in awn. finally, we could not do anything for the 2% of the words that have no relations at all. ## evaluation with a word sense disambiguation system in literature, we find different approaches to evaluate any lexical resources and the choice between them depending mainly on the kind of the resource itself and for what purpose @xcite . since awn is a lexical database in the first place, then its evaluation should follow one of the following strategy: as for the first approach of evaluation, many researchers have faced difficulties with it. abouenour et al. @xcite compared the content of awn with the content of pwn and the spanish wordnet. they found that the number of synsets in awn is around 8% (too low) of those of pwn, while the spanish wordnet represents 49%. taghizadeh et al. @xcite , also, compared their newly constructed persian wordnet with farsnet and they found a precision of 19%, which is too low to consider their resource as a reliable one. basically, one can tell if a wordnet is a reliable resource or not by how far it can help a system to achieve better results. this kind of evaluation seems to be a better way to test the extended awn. as mentioned above (section 2), many researchers used the awn in their applications and it helped achieve great results. as we are concentrated on the relations of awn, we looked into some nlp applications to see how the relations between the entities in awn can affect the precision of an nlp application. word sense disambiguation wsd seemed the most successful system to show the effectiveness of the relations between the words. the choice of the wsd system was made following a study of different systems that profit from the relations in awn. the aspect of the disambiguation is based on the similarity between words, which is exactly what the relations in awn are made for in the first place. besides, many wsd systems have been based on the relationship between words @xcite @xcite @xcite and other applications, like information retrieval and q/a system, rely more on the words themselves rather than the relations between them. all of this gives the wsd the advantage to be our best candidate. since our aim is to evaluate the impact of the relations in awn on a wsd system, the choice of the wsd algorithm is not the main task. we implement the very simple algorithm of galley et al. @xcite with a slight difference. the algorithm proceeds as follows: the algorithm takes a text as an input and proceeds all of the possible combinations between the current word and all the previous words. after that, a weighted edge takes the place if one of the senses of the current word has a semantic relation with any senses of the previous words. at the end of the text, a disambiguation graph is built with the nodes represent the senses of each word of the text and the edges representing the semantic relations between the senses of the words since awn links the senses and not the words. finally, the weights of each edge are summed up to represent a final score to each sense for each word in the text. the correct sense of the target word have the highest score. one thing to mention here is that this algorithm works with only 4 semantic relations (synonym, hypernym/hyponym, and sibling) and the weight of each edge is assigned according to the type of relations and the distance between the two words. we use the khaleej-2004 corpus @xcite . it contains 5690 documents divided to 4 categories; international and local news, economy, and sports. it has a total of nearly 3 millions words. we did not work on optimizing the weight nor the distance between the words. the only difference that we made is the number of relations. we implemented this algorithm to work with more relations. all relations in the extended version of awn are token into consideration. we tested the algorithm with three versions of awn; the version 2, the extended version with and without the new relations. as we can see from as an example, ↩zf is a polysemous verb. two of his senses are completely different. one could be 'playing music' and the other 'strike.' in the extended awn and without the enrichment of the relations, it has only two relations, hyponym with the verb šġl (fill) and hypernym with the verb ah ˘rǧ (get it out). when we run the test in the wsd system, we could get the appropriate sense. after the test with the new relations, we got the instrument relation a with a higher score. the obtained results with the enriched awn showed the importance of the resource and the relations between its words, even in a simple knowledge-based wsd algorithm like the one we used. ## conclusion in this paper, we presented the different versions of the awn along with a study case on the newly added relations to its extended version. next, we described the content of different versions of awn with some remarkable works done to enrich its relations. then, we cited many evaluation approaches in general and how we evaluated awn specifically. we provided an automatic method to validate some of the relations in awn. in the end, we found the most reliable approach is the human evalua-tion, despite the fact that it does not take advantage of computer programs and relies heavily on timeconsuming work. to make the new content more accurate, we tested different versions of awn with a real-life nlp application (wsd system). we attended interesting and promising results with the extended version of awn. before making it online and ready for the nlp community, we are still working on improving and refining the semantic relations in awn to get more accuracy and we are running some test in different nlp applications.