Is this Enough?-Evaluation of M alayalam W ordnet (2021)

Abstract: Quality of a product is the degree to which a product meets the customer’s expectation, which must also be valid for the case of lexical semantic resources. Conducting a periodic evaluation of resources is essential to ensure if the resources meet a native speaker’s expectations and free from errors. This paper defines the possible mistakes in a lexical semantic resource and explains the steps applied to quantify Malayalam wordnet quality. Malayalam is one of the classical languages of India. We hope to subset the less quality part of the wordnet and perform crowdsourcing to make it better.

## introduction internet is composed of machine-readable lexical or lexical semantic resources @xcite . however, only a small percentage of languages have these resources developed in full scale @xcite . hence the majority of languages are classified as under-resourced @xcite . in our study, we consider a language to be under-resourced if it has the following characteristics: lack of computerreadable resources, lack of linguist experts, and limited usage on the internet @xcite . the different large scale availability of language resources results in a digital language divide @xcite . figure 1 shows the visual representation of the study's findings on language-wise internet users in 2021 and was conducted by kpmg india. we notice that usage of hindi is greater than malayalam. bringing more languages online may ultimately be an exercise in cultural preservation, rather than utility. the probability of retrieving a relevant result using an under-resourced language is comparatively lower than with languages falling in the opposite category @xcite . to diminish the divide, we either need new methodologies that allow for the creation of language resources or the creation of a platform supporting the continuous development of such resources @xcite @xcite . a person communicates their need for knowledge through words with a precise meaning, however from a machine point of view meaning( as well as specific acceptations) do not automatically come in conjunction with the word. lexical-semantic resources come into play to bridge this gap, associating meanings to words. the most commonly used lexicalsemantic resource is the princeton word-net(pwn) @xcite . pwn is a lexical database for english that is organized around synsets. the popularity of pwn has spread in the language community and has lead to the generation of similar resources. one such attempt was made in india, a country both rich in culture and languages. indowordnet(iwn) is first multilingual wordnet for indian languages developed through the joint efforts of different reputed universities across the country. iwn developed intending to sketch india's cultures in length and breadth by including 18 languages out of 22 official languages @xcite . all the languages in the iwn does not have the same synset ids as pwn. for each synset in iwn, there is a corresponding synset in english that represent the same concept. in the paper, @xcite authors extended this idea to align the iwn with pwn with the help of a large-scale multilingual resource called the universal knowledge core(ukc) @xcite . the ukc has a similar structure of pwn and is designed as a multilayered ontology that has a language-independent semantic layer called the concept core (cc) and a language-specific lexico-semantic layer called the language core (lc). hence, in the ukc, the meaning of the words are represented not only by the synsets but also using lexical concepts @xcite . many works have been conducted under the ukc. recent one is @xcite , where authors explores the unique grammatical properties of natural language text and perform experiments on tokenisation, part of speech tagging and named entity recognition over real-world structured data. another important one is @xcite , which introduces a large-scale lexical database that provides words of common origin and meaning across languages. before enriching an existing resource, ensuring the quality and finding hidden errors is recommended in order to avoid producing a low-quality extensive resource. inadequate quality resources are not suitable to train artificial intelligence tools @xcite . one of the methods to ensure the quality of a resource is getting it validated using linguistic experts. hiring linguists are expensive, or for some languages it is difficult. and another drawback is that contribution from one expert could be biased @xcite . hence we need faster and cheaper ways to estimate the quality of a wordnet. also, the main question here is what are the factors that define the quality of a wordnet. as per the literature, a wordnet development team usually does structural check and whether the wordnet can perform applications like information retrieval @xcite and question-answering @xcite . in this paper, we classify possible errors that affect the quality of the wordnet: schema errors and semantic errors. we list five schema errors from the iwn. we estimate semantic error by computing cosine similarity @xcite between two wordnets and within the wordnet using a pre-trained machine learning model @xcite for indian languages. using our approach, we generate a candidate set of synsets with very high semantic error that will be verified by native speakers in a crowdsourcing application @xcite . this paper is structured as follows: section 2 covers details about the language malayalam and malayam wordnet (mwn), section 3 defines the quality of the wordnet and lists out the possible errors in wordnet, section 4 covers the procedure we used to find and establish the quality of the mwn, section 5 lists results and finally section 6 provides conclusions and direction for further work. ## malayalam language malayalam @xcite is a dravidian language spoken in the indian state of kerala and union territories of lakshadweep and puducherry. malayalam evolved from tamil language after 16 century ce by father of malayalam thunchaththu ramanujan ezhuthachan. in the early 21st century, malayalam was spoken by around 38 million people all over the world @xcite . in the past, media like television, newspaper, short stories, novels, etc. helped with the diffusion and prospering of malayalam language and literature @xcite . nowadays smartphones, laptops, and internet usage have become an important part of people's lives. the use of malayalam language through these platforms has helped people to communicate, learn and express. introduction of 'unicode' technology in the malayalam language helped correct spelling mistakes, grammatical errors, etc @xcite , which was initially possible only in the english language. the stories and articles were in the regional language, though they used english words for dif-ferent technical terms @xcite . for example, the malayalam word for mobile phones is "ദൂരഭാഷണശര് വണസഹായി" (dhoora bashana sravana sahayi), which was never used in any of the articles/blogs/news videos. the word 'switch' in malayalam was "ൈവദുയ് ത ആഗമന നിയന്തര് ണ യന്തര് ം" (vaidhyutha agamana niyanthrana yanthram). but due to its complexity, these words were never used. the malayalam language lost lots of words due to this reason @xcite . the malayalam vocabulary is rich in terms from specific domains, with the notable example of terms reflecting family relations @xcite . in kerala, where malayalam is mother tongue, family relations are essential and there are particular names for each type of relation. a notable example of this richness is represented by the english word cousin: "the child of your aunt or uncle". in malayalam the term used to denote a cousin will change depending on age, gender, whether the relation is from the paternal or maternal side. sometimes the background of the family is also a factor influencing the selection of a different term. there is no such thing as a direct translation of the word cousin in malayalam. the indian university, amrita vishwa vidyapeetham, took part in the development of mwn @xcite in 2011 as part of the project entitled "development of dravidian wordnet: an integrated wordnet for telugu, tamil, kannada and malayalam" which later integrated into iwn along with other indian languages. the development of mwn is motivated by pwn. the initial idea of pwn was to aid the search in dictionaries, conceptually rather than alphabetically. based on inspiration coming from current psycholinguistics @xcite theories of human memory and its working, pwn organizes english nouns, verbs,adverbs, and adjectives into synonym sets or synset, each representing one underlying lexical concept. synonyms are defined as words holding the same meaning in the same language @xcite . a set of synonymous words is called synset. for instance, "head" and "caput" are the synsets with the meaning "the front or the upper part of the body of animals bearing the face and brains". in pwn, synsets are associated with a gloss. a gloss defines the meaning of a synset, the concept associated with the synset. mwn uses the same structure and lists the following entries for each concept: synset id, synsets, gloss and example sentence. synset ids used in mwn are different from pwn as the expand approach was used to develop mwn. in the expand approach, hindi synsets are translated into malayalam, this ensures concepts which are culturally relevant to india are retained as part of mwn. in our study, we are using source files of mwn we received from amrita vishwa vidyapeetham. there are 30139 synsets available in mwn. in which, 20071 synsets are nouns, 3311 synsets are verbs, 501 synsets are adverbs, and 6256 synsets are adjectives. ## defining quality within the translation industry, three terms are used somewhat interchangeably to refer to quality-related initiatives: quality assessment @xcite , quality assurance @xcite and quality control @xcite . based on the definition, quality control requires specifications @xcite which can then be used to assess whether a resource meets quality thresholds. in our work, we are performing quality control of the mwn. we are defining the minimum specifications in order to evaluate the wordnet and assess whether it is a good quality resource. multidimensional quality metrics (mqm) @xcite provides a framework for describing and defining quality metrics used to assess the quality of translated texts and to identify specific issues in those texts. the scope of this framework is limited to quality assessment of translated content. it does not apply to the evaluation of the translation process or projects. mqm defines the quality as adherence of the text to appropriate specifications. as per mqm, specifications are a description of requirements for the translation. in our work, we have been inspired to adapt ideas used as part of mqm for defining quality specifications of the wordnet. in mqm, a target document quality score is defined as t @xmath0, tp the penalties for target content, and sp the penalty tied to source content. we then define a resource as being of good quality if the resource is free from errors. we define two categories of errors in word-net: schema errors and semantic errors. ## evaluating mwn in this section, we evaluate schema errors and semantic errors for the mwn. we estimated the schema errors of the resource as listed in the we sampled synsets common to english, hindi and malayalam wordnets; this yielded 18938 synsets. we then computed cosine similarity in order to identify synsets tied to poor semantic alignment and through this be able to identify semantic errors. we propose a cosine similarity score based classification parsed into three classes: low alignment for scores lower than 0.4; moderate for scores between 0.4 and 0.7; strong for scores greater than 0.7. we have used the threshold of minimum 0.4 inspired by the work by @xcite . in @xcite authors automatically generate wordnet data by using a machine-readable dictionary. based on our classification, we will consider the sets of synsets having cosine similarity scores below 0.4 as requiring attention. we used two approaches to measure semantic errors: we computed cosine similarity(semantic alignment) between two languages; we then calculated semantic alignment within the language. algorithm 1 shows the steps followed in computing the semantic similarity between english gloss and malayalam gloss. we have used the same steps to calculate the semantic similarity between english word sense and malayalam word sense. and repeat the above steps also for language hindi. algorithm 2 type of schema error no.synsetids empty example sentence 181 empty synsets 1789 poorly defined gloss 2374 duplicate gloss 925 a pre-trained machine learning model was used in order to compute semantic alignment and cosine similarity. there are a number of pre-trained models available for similarity checking. multilingual bert(mbert) @xcite , xlm-roberta @xcite and sentence transformers @xcite are only few of them. multilingual bert(mbert) and xlm-roberta have been known to produce less than ideal sentence representations when deployed out-ofbox. they would additionally pose the problem of coming with non-aligned vector spaces across languages. as a result, sentences with the same semantic content, expressed with different languages, would be mapped to different vector spaces. as part of our work we used sentence transformers: a python framework for state of the art sentence and text embeddings that can be used to compute sentence/text embeddings for more than 100 languages. these embeddings can be compared through cosine similarity, allowing for the identification of sentences with similar meaning. the framework is based on pytorch @xcite , transformers and offers a large collection of pre-trained models tuned for various tasks. we have used the pre-trained multilingual model stsb-xlm-r-multilingual and aligned vector spaces allowing for similar inputs across languages to be mapped close within the same vector space. xlm-r supports 100 languages including 13 indian languages, and is as such able to handle linguistic inputs without the need to specify what the input language is upfront. the model produces similar embeddings as the bert-base-nli-stsb-mean-token model. algorithm 1: semantic similarity between english and malayalam input: a csv file contains gloss and seed term of synset for languages english and malayalam output: a csv file contains id and semantic similarity score between english and malayalam data: common concepts in english and malayalam languages foreach row in csv file do compute embeddings of the english and malayalam gloss calculate cosine-similarity between the embeddings write the id and semantic similarity score into a csv file algorithm 2: semantic similarity between malayalam gloss and each word in the synset input: a csv file contains gloss and a word for malayalam language output: a csv file contains id and semantic similarity score of malayalam gloss and word data: common concepts in english and malayalam languages foreach row in csv file do compute embeddings of the malayalam gloss and word calculate cosine-similarity between the embeddings write the id and semantic similarity score into a csv file 5 results there are 30139 synsets available in the malayalam language. using our approach, we estimated that 14% of the synsets have schema errors. figure 8 summarizes cosine similarity score distribution. 30% of gloss-wise similarity scores for pwn-mwn synsets fall in the low alignment category, the score goes down to 17% if looking at word sense level alignment for pwn-mwn entries. interestingly, when looking at hwn-mwn data, synsets falling into the low alignment category decrease to 4% when looking at gloss-wise alignment; 2% when looking at word sense-wise alignment. looking at within language data, reflects that 20% of mwn word senses are tied to low scores(below 0.4). figure 9 shows the distribution of similarity agreement within the malayalam language and figure 10 shows the similarity distribution of malayalam language between english and hindi. computing semantic similarity within the language helps us understand and isolate poorly aligned word senses and better word senses. results aligned how malayalam language entries are more semantically aligned with hindi than english, which is expected. ## conclusion and future work this paper proposes specifications to be considered for a good quality lexical-semantic resource. we believe a resource without schema and semantic errors to be a useful resource. we defined schema errors and semantic errors for wordnet and estimated the presence of both for mwn. 14% of the malayalam resource has schema errors. an average of 23.5% meanings representing the same concepts across english and malayalam are tied to low similarity scores( poor semantic alignment). only 3% figure 10: distribution of similarity of mwn between pwn and hwn we are not taking these values as the final deciding factor. we will be using this low score synsets as a candidate set for our crowdsourcing application. this application will have different tasks like define gloss, provide synset, validate the gloss, and so on.