Development of POS tagger for E nglish- B engali Code-Mixed data (2019)

Abstract: Code-mixed texts are widespread nowadays due to the advent of social media. Since these texts combine two languages to formulate a sentence, it gives rise to various research problems related to Natural Language Processing. In this paper, we try to excavate one such problem, namely, Parts of Speech tagging of code-mixed texts. We have built a system that can POS tag English-Bengali code-mixed data where the Bengali words were written in Roman script. Our approach initially involves the collection and cleaning of English-Bengali code-mixed tweets. These tweets were used as a development dataset for building our system. The proposed system is a modular approach that starts by tagging individual tokens with their respective languages and then passes them to different POS taggers, designed for different languages (English and Bengali, in our case). Tags given by the two systems are later joined together and the final result is then mapped to a universal POS tag set. Our system was checked using 100 manually POS tagged code-mixed sentences and it returned an accuracy of 75.29%.

## introduction a parts-of-speech (pos) tagger is a piece of software that reads the text in some language and assigns parts of speech tags, such as noun, verb, adjective, etc., to each word/token. pos tags are useful for building parse trees, which may be used to build textbfnamed entity recognizers (ner) or dependency parsers. pos tagging is also useful for building lemmatizers, which are used to reduce a word to its root form. pos taggers for widely spoken languages have been developed in abundance. but such resources are very scarce for low resourced languages. on the other hand, code-mixing is simply a mix of two or more languages in communication. due to the emergence of social media, a lavish amount of digital code-mixed data is generated. this is because people nowadays are very comfortable with multilingualism. this phenomenon has produced a section of researchers, who contemplate codemixed texts as being a new language. as mentioned earlier, since pos tagging systems for low resourced languages are hard to come by, developing one that will cater to code-mixed text is trivial. pos tagging systems, if developed for code-mixed data, can lead to deciphering many complex natural language processing (nlp) tasks and hence, we attempt to develop the same in this reported work. we try to focus on creating a pos tagger for english-bengali codemixed data, as languages such as bengali are morphologically rich in nature. our method includes scraping of code-mixed english-bengali tweets on twitter and cleaning them. the bengali words in these tweets were in roman script. these cleaned tweets were used as a development dataset for building our system. our system starts with tagging individual tokens of a tweet with their respective languages, either english, bengali or unknown. this step will give rise to segments/sub-sequences of the tweet, written in the same language. it is to be noted that tokens tagged as unknown were discarded. the segments will then be passed to two pos taggers, one designed for english and the other designed for bengali. the output from the pos taggers will then be joined together to get the final pos tagged, code-mixed tweet. since the pos tagging modules of english and bengali use different tag sets, we further map the tags to a manually defined universal pos tag set. this step produces a final pos tagged tweet with uniform tags. the architecture of the proposed model is shown in figure 1 . the remainder of the paper is organized as follows. section 2 documents a brief state-of-art on language tagger segments english segments english pos tagger bengali pos tagger join mapping to universal pos tag set output bengali segments 5,148 tweets 5,012 language tagged tweets tweet scraping and cleaning data preparation module pos tagging module this domain. section 3 defines the data preparation steps. section 4 defines the pipeline which helps us in pos tagging the code-mixed tweets. this will be followed by the results in section 5 and concluding remarks in section 6. ## related work in the past few years, a lot of significant work has been done in the field of parts of speech tagging. the first significant pos tagger came in the early nineties which was a rule-based tagger @xcite . one of the english rulebased taggers had an accuracy of 99.5% @xcite . pos taggers based on statistical approaches were also used during this time, which was based on statistical models like bi-gram,tri-gram and markov models @xcite @xcite @xcite . subsequently, pos tagger based on both statistical methods and a rule-based approach was proposed by @xcite . use of conditional random fields for the development of pos taggers was proposed by @xcite , @xcite and @xcite . @xcite used neural networks for pos tagging for the first time. pos taggers for the bengali language was also built by @xcite . this postagger was built on the analysis of the bengali morphemes. other works have been done in bengali pos tagging by @xcite and @xcite which were rule-based and semisupervised. pimpale and patel (2016) attempted to tag codemixed data using stanford pos tagger. he trained the pos tagger on constrained data of hindi, bengali, and telugu, mixed with english. they garnered accuracy figures of 71%. similarly, sarkar (2016) used the hmm model on constrained codemixed data and achieved an accuracy figure of 75.60%. pipeline architecture for pos tagging of codemixed data was first used by @xcite . the training data was very low in their case and the lid (language identification) and transliteration models used were based on support vector machines (svm) and manual transliteration. our approach also used pipeline architecture similar to theirs, but our model does not require any annotated data to train the system. also, the lid and transliteration modules, in our case, have been fully trained with much larger data, using deep learning architecture. ## data preparation we decided to use a development dataset for building our system. it is to be noted that this data was used to build the proposed system and not to train it. since code-mixed data consisting of english and bengali language are difficult to find, we decided to scrape such data from twitter. the collected tweets contained multiple degrees of noise and hence, it needed to be cleaned before using it to develop our future systems. after cleaning the tweets, they were subjected to a language tagger module that tagged every token of the tweet with their corresponding language (english, bengali, and unknown, in this case). ## parts of speech tagging after the data preparation step, the language tagged segments are passed to the corresponding for the bengali segments, we used a tagger developed by @xcite . they trained the tagger on 10,000 bengali (devanagari) pos tagged sentences and tested it on 2,000 bengali (devanagari) sentences. their model returned 92% accuracy. to use their model, we had to transliterate the bengali segments into its corresponding devanagari script. the model developed to do the same is described in section 4.1. ## conclusion in this work, we have devised a modular system that can pos tag english-bengali code-mixed sentences. the system uses sub-modules to perform the same. owing to the fact, that the submodules can be trained for any given language, the proposed approach can be used to tag a variety of code-mixed data involving any two language pairs. the system can be enhanced further if the submodules can be trained using more annotated data. e.g., if the pos tagger for the bengali language could have been trained using more data, the problem of tagging untrained tokens with 'un' tags could have been solved. also, the problem of wrongly tagging tokens, e.g., tagging noun as adj, verb and tagging pron as noun, verb, etc., could have been solved. this would have made the bengali pos tagging module more robust. the same applies to the transliteration module as well. in the future, we would like to develop an end-to-end system, so that the errors of one submodule do not propagate to the other sub-modules.