Part-of-Speech Annotation Challenges in M arathi (2020)

Abstract: Part of Speech (POS) annotation is a significant challenge in natural language processing. The paper discusses issues and challenges faced in the process of POS annotation of the Marathi data from four domains viz., tourism, health, entertainment and agriculture. During POS annotation, a lot of issues were encountered. Some of the major ones are discussed in detail in this paper. Also, the two approaches viz., the lexical (L approach) and the functional (F approach) of POS tagging have been discussed and presented with examples. Further, some ambiguous cases in POS annotation are presented in the paper.

## introduction in any natural language, part of speech (pos) such as noun, pronoun, adjective, verb, adverb, demonstrative, etc., forms an integral building block of a sentence structure. pos tagging 1 is one of the major activities in natural language processing (nlp). in corpus linguistics, pos tagging is the process of marking/annotating a word in a text/corpus which corresponds to a particular pos. the annotation is done based on its definition and its context i.e., its relationship with adjacent and related words in a phrase, sentence, or paragraph. a simplified form of this is commonly taught to school-age children, in the identification of words as nouns, verbs, adjectives, adverbs, etc. the term 'part-of-speech tagging' is also known as pos tagging, post, pos annotation, grammatical tagging or word-category disambiguation. in this paper, the challenges and issues in pos tagging with special reference to marathi 2 have been presented. the marathi language is one of the major languages of india. it belongs to the indo-aryan language family with about 71,936,894 users 3 . it is predominantly spoken in the state of maharashtra in western india @xcite . in recent years many research institutions and organizations are involved in developing the lexical resources for marathi for nlp activities. marathi wordnet is one such lexical resource developed at iit bombay @xcite . the paper is organized as follows: section 2 introduces pos annotation; section 3 provides information on marathi annotated corpora; section 4 describes marathi tag set; section 5 explains tagging approaches, section 6 presents ambiguous behaviors of the marathi words, section 7 presents a discussion on special cases, and section 8 concludes the paper with future work. 1 http://language.worldofcomputing.net/pos-tagging/parts-ofspeech-tagging.html 2 http://www.indianmirror.com/languages/marathi-language.html 3 http://www.censusindia.gov.in/ ## parts-of-speech annotation in nlp pipeline pos tagging is an important activity which forms the base of various language processing applications. annotating a text with pos tags is a standard low-level text pre-processing step before moving to higher levels in the pipeline like chunking, dependency parsing, etc. @xcite . identification of the parts of speech such as nouns, verbs, adjectives, adverbs for each word (token) of the sentence helps in analyzing the role of each word in a sentence @xcite . it represents a token level annotation wherein it assigns a token with pos category. ## marathi annotated corpora aim of pos tagging is to create a large annotated corpora for natural language processing, speech recognition and other related applications. annotated corpora serve as an important resource in nlp activities. it proves to be a basic building block for constructing statistical models for the automatic processing of natural languages. the significance of large annotated corpora is widely appreciated by researchers and application developers. various research institutes in india viz., iit bombay 4 , iiit hyderabad 5 , jnu new delhi 6 , and other institutes have developed a large corpus of pos tagged data. in marathi, there is around 100k annotated data developed as a part of indian languages corpora initiative (ilci) 7 project funded by meity 8 , new delhi. this ilci corpus consists of four domains viz., tourism, health, agriculture, and entertainment. this tagged data (tourism -25k, health -25k, agriculture -10k, entertainment -10k, general -30k) is used for various applications like chunking, dependency tree banking, word sense disambiguation, etc. this ilci annotated data forms a baseline for marathi pos tagging and is available for download at tdil portal foot_5 . ## the marathi pos tag-set the bureau of indian standards (bis) foot_6 has come up with a standard set of tags for annotating data for indian languages. this tag-set is prepared for hindi under the guidance of bis. the bis tag-set aims to ensure standardization in the pos tagging across the indian languages. the tag sets of all indian languages have been drafted by dept. of information technology, meity and presented as unified pos standard in indian languages foot_7 . marathi pos tag-set has been prepared at iit bombay referring to the standard bis pos tag-set, iiit hyderabad guideline document @xcite and konkani tag-set @xcite . this marathi pos tag-set can be seen in appendix a. ## pos ambiguity: challenges and discussions ambiguity is a major open problem in nlp. several pos level ambiguity issues were faced by annotators while annotating the marathi corpus. following are some pos specific ambiguity problems encountered while annotating. ## conclusion and future work marathi pos tagging is an important activity for nlp tasks. while tagging, several challenges and issues were encountered. in this paper, marathi bis tag-set has been discussed. lexical and functional tagging approaches were discussed with examples. further, various challenges, experiences, and special cases have been presented. the issues discussed here will be helpful for annotators, researchers, language learners, etc. of marathi and other languages. in future, more issues such as tagging for words having multiple senses; words having multiple functional tags will be discussed. also, tagset comparison of close languages will be done. further, the evaluation of lexical and functional tagging using statistical analysis will be done.