A Benchmark and Scoring Algorithm for Enriching A rabic Synonyms (2023)

Abstract: This paper addresses the task of extending a given synset with additional synonyms taking into account synonymy strength as a fuzzy value. Given a mono/multilingual synset and a threshold (a fuzzy value [0−1]), our goal is to extract new synonyms above this threshold from existing lexicons. We present twofold contributions: an algorithm and a benchmark dataset. The dataset consists of 3K candidate synonyms for 500 synsets. Each candidate synonym is annotated with a fuzzy value by four linguists. The dataset is important for (i) understanding how much linguists (dis/)agree on synonymy, in addition to (ii) using the dataset as a baseline to evaluate our algorithm. Our proposed algorithm extracts synonyms from existing lexicons and computes a fuzzy value for each candidate. Our evaluations show that the algorithm behaves like a linguist and its fuzzy values are close to those proposed by linguists (using RMSE and MAE). The dataset and a demo page are publicly available at https://portal.sina.birzeit.edu/synonyms.

## introduction and motivation synonymy relationships are used in many nlp tasks and knowledge organization systems. however, automatic synonym extraction is a challenging task, especially for low-resourced and highly ambiguous languages such as arabic @xcite . there are some arabic resources representing synonymy, such as @xcite and the arabic ontology @xcite ; however, these resources are limited in terms of size and coverage @xcite , especially if compared with the english princeton wordnet @xcite . building such resources is expensive and challenging @xcite @xcite . in addition, the notion of synonymy itself is problematic, as it can vary from near (i.e., semantically related) to strict synonymy @xcite . strict and formal synonymy is used in ontology engineering as an equivalence relation, thus its reflexive, symmetric, and transitive @xcite . a less formal synonymy is used in the construction of synsets in princeton wordnet, which relies on the substitutionability of words in a sentence: "two expressions are synonymous in a linguistic context c if the substitution of one for the other in c does not alter the truth value" @xcite . for example, ( /road) and ( /street) are substitutionable in many contexts in arabic, thus they can be synonyms. as will be reviewed in section 2, different approaches have been proposed for extracting synonyms automatically. nevertheless, one of the major challenges in extracting synonyms is that it is hard to evaluate them @xcite and there are no common evaluation datasets. moreover, the substitutionability criteria are subjective, because humans do not necessarily agree on synonymy. as will be illustrated later in this paper, if different linguists are given the same words to judge whether they are synonyms, it is unlikely that they will agree on all cases. thus, instead of relying on "the substitutionability of words in a sentence" as a criterion to judge whether two words are synonyms or not, we propose to model it with a fuzzy value. for example, let {confederacy, confederation} be two synonyms in the context of "a union of political organizations", and let "alliance" and "federation" be candidate additional synonyms, our goal is to assign a fuzzy value (e.g., 0.6 and 0.9) to each candidate synonym to indicate how much it is substitutionable, i.e., acceptable to be an additional third synonym. using such a fuzzy value is helpful for different application scenarios. for example, when constructing wordnet synsets, synonyms can be extracted with a high fuzzy value, but in the case of less sensitive information retrieval applications, a lower value might be more suitable. in a quality control scenario, one may evaluate a thesaurus by masking each synonym in a synset and assessing if its fuzzy value passes a threshold. nevertheless, assigning a meaningful fuzzy value to each synonym in a synset is challenging. thesauri are typically constructed based on linguists' intuition and without assigning a strength, or a fuzzy, value explicitly. to overcome this challenge, we developed a dataset of 3k synonyms, each assigned with a fuzzy value by four different linguists. we used this dataset to measure how much linguists (dis/)agree on synonymy. the dataset is also used to train our proposed algorithm (i.e., tune its fuzzy model) for extracting synonyms from dictionaries. task definition: the task we aim to address is defined as the following: let s be a set of synonyms, c is a candidate synonym to s, and a dictionary d, our goal is to compute a fuzzy value f to indicate how much c is acceptable to be an addition to s. as will be elaborated in section 4, we assume d to be a set of sets of synonyms, and that s can be mono or multilingual synonyms. our main contributions in this paper are a dataset and an algorithm. the dataset was constructed by employing four linguists and giving them 3,000 candidate synonyms and 500 synsets from the arabic wordnet @xcite . each linguist was asked to score each candidate synonym in a given synset. our proposed algorithm aims at discovering new candidate synonyms from existing linguistic resources. given a set of synonyms, the algorithm builds a directed graph, at level k for all words in this set. cyclic paths in this graph are then detected, and all words participating in these cyclic paths are considered candidate synonyms for the given synset. each of these candidate synonyms is assigned a fuzzy value, which is calculated based on a fuzzy model that we learned from the dataset and that takes into account the connectivity of the candidate synonym in the graph. the novelty of our algorithm and our dataset is that we treat synonyms as a fuzzy relation. we evaluated the algorithm's fuzzy values by comparing them with the average of the linguists' scores (i.e., as a baseline). the root mean squared error (rmse) between the scores of the algorithm and the average of the linguists' scores is 0.32 and the mean average error (mae) is 0.27. this means that the algorithm was behaving closely to a linguist. to evaluate the accuracy of our algorithm, we used the 10k synsets in arabic wordnet. we masked the word with (highest, lowest, average, and random) frequency in each synset and used the algorithm to see if it could discover it again with top rank. the achieved accuracy was indeed high. for example, with the average frequency we achieved an accuracy of 98.7% at level 3 and 92% at level 4. this paper is organized as follows: section 2 presents related works in the field of synonym extraction. section 3 overviews the algorithm. section 4 summarizes and discusses the experimental results. section 5 concludes the paper and proposes some perspectives. ## related work in what follows, we overview several approaches have been proposed to extract synonyms or build synsets. we refer to @xcite for a recent survey on this topic. ## dataset construction this section presents a benchmarking dataset annotated with fuzzy values foot_0 . the dataset can be used for training and evaluating (i.e., a baseline) synonym extraction algorithms. additionally, the construction of this dataset can also be used as an experiment to measure how much linguists (dis/)agree on synonymy. a union of political organizations ‫ﻣ‬ ُ ‫ﺣ‬ َ ‫ﺎ‬ ‫ﻟ‬ ‫ﻔَ‬ َ ‫ﺔ‬ 60 ‫ﻧ‬ ‫ﻔ‬ ‫س‬ ‫ا‬ ‫ﻟ‬ ‫د‬ ‫ﻻ‬ ‫ﻟ‬ ‫ﺔ‬ ، ‫ا‬ ‫ﻷ‬ ‫ﺳ‬ ‫ﻠ‬ ‫و‬ ‫ب‬ ‫ﺿ‬ ‫ﻌ‬ ‫ﯾ‬ ‫ف‬ ، ‫ﻏ‬ ‫ﯾ‬ ‫ر‬ ‫ﺷ‬ ‫ﺎ‬ ‫ﺋ‬ ‫ﻌ‬ ‫ﺔ‬ ‫ا‬ ‫ﺋِ‬ ‫ﺗْ‬ ِ ‫ﻼ‬ ‫ف‬ 80 ‫ﻧ‬ ‫ﻔ‬ ‫س‬ ‫ا‬ ‫ﻟ‬ ‫د‬ ‫ﻻ‬ ‫ﻟ‬ ‫ﺔ‬ ، ‫ا‬ ‫ﻷ‬ ‫ﺳ‬ ‫ﻠ‬ ‫و‬ ‫ب‬ ‫ﺻ‬ ‫ﺣ‬ ‫ﯾ‬ ‫ﺢ‬ ، ‫ﺷ‬ ‫ﺎ‬ ‫ﺋ‬ ‫ﻌ‬ ‫ﺔ‬ ‫ا‬ ‫ﻟ‬ ‫ﻰ‬ ‫ﺣ‬ ‫د‬ ‫ﻗ‬ ‫ﻠ‬ ‫ﯾ‬ ‫ل‬ ‫ا‬ ‫ﺗِ‬ ّ ِ ‫ﺣ‬ َ ‫ﺎ‬ ‫د‬ 100 ‫ﻧ‬ ‫ﻔ‬ ‫س‬ ‫ا‬ ‫ﻟ‬ ‫د‬ ‫ﻻ‬ ‫ﻟ‬ ‫ﺔ‬ ‫و‬ ‫ا‬ ‫ﻻ‬ ‫ﺳ‬ ‫ﻠ‬ ‫و‬ ‫ب‬ ‫و‬ ‫ا‬ ‫ﻟ‬ ‫ﺷ‬ ‫ﯾ‬ ‫و‬ ‫ع‬ ‫ﺟ‬ ‫ﺎ‬ ‫ِﻣ‬ ‫ﻌ‬ َ ‫ﺔ‬ 60 ‫ﻧ‬ ‫ﻔ‬ ‫س‬ ‫ا‬ ‫ﻟ‬ ‫د‬ ‫ﻻ‬ ‫ﻟ‬ ‫ﺔ‬ ، ‫ا‬ ‫ﻷ‬ ‫ﺳ‬ ‫ﻠ‬ ‫و‬ ‫ب‬ ‫ﺿ‬ ‫ﻌ‬ ‫ﯾ‬ ‫ف‬ ، ‫ﻏ‬ ‫ﯾ‬ ‫ر‬ ‫ﺷ‬ ‫ﺎ‬ ‫ﺋ‬ ‫ﻌ‬ ‫ﺔ‬ ## algorithm overview the algorithm takes two inputs: a dictionary d, and a synset s. the output is a set of candidate synonyms c, each synonym c i is assigned a fuzzy value f i . the dictionary d itself is assumed to consist of set of synsets, s @xmath0. each synset is a @xmath1, .., t @xmath2. in this way, we can benefit from mono and multiple dictionaries and thesauri. in the first step, the algorithm extracts the candidate synonyms c, then it computes the fuzzy value f i for each synonym c i . ## algorithm evaluation this section presents two experiments to evaluate the performance of our algorithm. the first experiment compares the results obtained by the algorithm with linguists' scores. the second experiment measures the accuracy of the algorithm. ## conclusion we presented a benchmark dataset and an algorithm to extract synonyms and fuzzy values. the benchmark dataset consists of 3k candidate synonyms for 500 synsets, each candidate synonym was annotated with a fuzzy value by four linguists. the dataset is important for measuring how much linguists disagree on synonymy, which ranged between 0.16 -0.22 for rmse and 0.12 -0.16 for mae. these measures were also used as a baseline to evaluate our algorithm. the algorithm presented in this paper aims to enrich a given mono/multilingual synset with more synonyms. our evaluation shows that our algorithm behaves as linguists in producing fuzzy values, and the fuzzy scores are also close to those of the linguists. the accuracy evaluation illustrates that it is highly accurate.