P ersian S em C or: A Bag of Word Sense Annotated Corpus for the P ersian Language (2021)

Abstract: Supervised approaches usually achieve the best performance in the Word Sense Disambiguation problem. However, the unavailability of large sense annotated corpora for many low-resource languages make these approaches inapplicable for them in practice. In this paper, we mitigate this issue for the Persian language by proposing a fully automatic approach for obtaining Persian SemCor (PerSemCor), as a Persian Bag-of-Word (BoW) sense-annotated corpus. We evaluated PerSemCor both intrinsically and extrinsically and showed that it can be effectively used as training sets for Persian supervised WSD systems. To encourage future research on Persian Word Sense Disambiguation, we release the PerSemCor in http://nlp.sbu.ac.ir .

## introduction word sense disambiguation (wsd) is the task of associating ambiguous context words with their most suitable meanings in a pre-defined sense inventory. wsd can be mentioned as a key area in natural language processing (nlp) since it plays a crucial rule in multiple down-stream tasks such as machine translation @xcite . the main approaches of wsd can be grouped into two categories i.e., knowledge-based and supervised wsd @xcite . knowledge-based wsd systems tend to exploit information from structure or content of lexical resources such as wordnet @xcite and @xcite . on the other hand, the latter approach utilizes machine learning techniques to train a model for automatic sense annotation @xcite , @xcite , @xcite . thanks to the training phase, supervised systems usually outperform the knowledge-based alternatives @xcite . in fact, the main reason for the high performance of the supervised systems is the utilization of large manually sense annotated corpus through the training process. unfortunately, obtaining manually sense annotated corpora such as semcor @xcite (i.e. the largest and the most predominant manually sense annotated corpus developed for english) is extremely hard and time-consuming and as a result only a limited number of languages can perform supervised wsd. to tackle this issue, in recent years, a line of research has focused on developing automatic or semi-automatic methodologies capable of producing annotated corpora @xcite ) @xcite @xcite @xcite . although the developed annotated corpora are multi-lingual and lead the supervised systems to achieve a big improvement in wsd, as mentioned in @xcite , they suffer from some limitations such as (1): strict dependency on the structure of the knowledge graph, (2): requiring huge parallel corpora. in addition, almost all developed corpora are only limited to nouns and provide no annotated instances for other parts-of-speech (pos) i.e., verbs, adjectives, and adverbs. in this paper, we focus on developing a fully automatic approach for creating a sense annotated corpus for the persian language. a key part of the former developed approaches for construction of automatic sense-annotated corpora is the use of high performance pre-processing tools i.e., lemmatizer, tokenizer and pos tagger. however, to the best of our knowledge, developed persian pre-processing tools can not perform as well as their counterparts for english or other european languages. it could be problematic especially when we need to tokenize multi-words and obtain their lemma for exploiting sense candidates from farsnet (the persian wordnet) synsets. to deal with this, we designed our method in such a way that it requires no auto-matic lemmatizer, tokenizer or pos tagger. our proposed system takes english sense annotated corpora (semcor, for instance) as input and utilizes inter-language semantic relations between english and persian to obtain a bag-of-words (bow) sense annotated corpus for persian. it can be a step forward towards the development of sentence-level sense-annotated corpora for the persian language. the main contributions of our proposed system are as follows: ## construction of persemcor in this section, we present our proposed approach which aims at automatic construction of persem-cor, a bow sense-annotated corpus for the persian language. the main idea of our proposed approach is inspired by the assumption presented in @xcite , i.e, sense annotations of a source language can be transferred to a target language. given a sense annotated corpus (semcor, in our case) as input, our proposed system utilizes inter-language semantic relations between english and persian lexical graphs (wordnet and farsnet) to obtain a bag-of-words(bow) sense annotated corpus for persian. in the following, we first introduce a set of notations which have been used in our proposed approach and then provide details on the way we used the relations between wordnet and farsnet to create persemcor. ## evaluation we carried out a number of experiments on persem-cor to evaluate it both intrinsically and extrinsically. more in detail, in our intrinsic evaluations, we assessed the quality of sense annotations of persemcor. in addition, we utilized persemcor for training a set of supervised wsd baselines to extrinsically evaluate it. ## conclusion in this paper, we presented persemcor, a fullyautomatic constructed sense-annotated corpus for the persian language. our approach for building persemcor includes no human intervention as it uses semantic inter-language relations to annotate the persian words. moreover, we eliminated the burden of high-performance pre-processing tools, i.e. tokenizer and lemmatizer, as they can be a source of error in constructing training data sets for the persian language. we evaluated the built corpus, persemcor, both intrinsically and extrinsically, and proved that it can count as a high-quality sense-annotated corpus for training supervised persian wsd models. as the future work, we plan to create a persian sentence-level sense-annotated corpus by employing a 'bow2seq' approach, i.e. an approach which takes a set of shuffled words of a sentence as input and reorder them like a real sentence.