A Pre-trained Transformer and CNN Model with Joint Language ID and Part-of-Speech Tagging for Code-Mixed Social-Media Text (2021)

Abstract: Code-mixing (CM) is a frequently observed phenomenon that uses multiple languages in an utterance or sentence. There are no strict grammatical constraints observed in code-mixing, and it consists of non-standard variations of spelling. The linguistic complexity resulting from the above factors made the computational analysis of the code-mixed language a challenging task. Language identification (LI) and part of speech (POS) tagging are the fundamental steps that help analyze the structure of the code-mixed text. Often, the LI and POS tagging tasks are interdependent in the code-mixing scenario. We project the problem of dealing with multilingualism and grammatical structure while analyzing the code-mixed sentence as a joint learning task. In this paper, we jointly train and optimize language detection and part of speech tagging models in the code-mixed scenario. We used a Transformer with convolutional neural network architecture. We train a joint learning method by combining POS tagging and LI models on code-mixed social media text obtained from the ICON shared task.

## introduction in bilingual and multilingual communities, codemixing or code-switching occurs when a person alternates languages below the phrase level inside a sentence or an utterance. code-mixing(cm) is generally observed in informal settings such as casual conversations or social media text. code-mixing is defined as mixing phrases, words, and morphemes of one language into another language @xcite . language identification (li) and part of speech (pos) tagging are the fundamental steps in processing any code-mixed sentence. li deals with resolving the language ambiguity of each word in a code-mixed text. pos tagging involves assigning a part of the speech label for each word in a sentence based on its syntactic and semantic information. it helps analyze the grammatical structure of the sentence. both language identification and pos tagging are sequence labeling tasks as they tag each word in a sentence by its corresponding language and pos tags, respectively. while processing a monolingual text, the primary step to understand the sentence's grammatical structure would be pos tagging. however, in the code-mixed scenario, we must consider multilingual phenomena, i.e., each word's language while pos tagging. similarly, pos tagging helps capture a better grammatical structure of the text. moreover, identifying the grammatical structure can improve language identification. in the codemixing scenario, two tasks go hand in hand. thus, a joint learning model on pos tagging and li will considerably enhance the code-mixed text analysis. recently the transformer models with transfer learning such as bert @xcite achieved state-of-the-art accuracy in sequence classification tasks. an evaluation benchmark on codemixed datasets -gluecos @xcite stated that a modified version of bert @xcite called mod-mbert that was finetuned on synthetically generated code-switched data outperformed on all the code-mixed datasets. recently, bert with ensemble models has shown improved performance on text classification tasks @xcite . to improve our tasks' performance, we have used a convolutional model. the convolutional approach learns a compositional structure in the sequences more efficiently since the representations are built on hierarchy. in the code-mixed social media text, the native words are often written in the roman script. the introduction of such non-standard transliterations will add complications while processing the text. it is necessary to deal with these complications as they might introduce errors in further processing steps. to tackle such problems, we backtransliterated the roman words into their native script. this paper presents a pre-trained transformer encoder with convolutional neural network architecture for pos tagging and language identification of code-mixed social-media text. the model uses sub-word level input representation to handle morphologically rich words. our contributions are as follows: the paper is organized as follows. section 2 presents a survey of related works on language identification and pos tagging in a code-mixed social media text. section 3 introduces the proposed approach of jointly modeling the language identification and the part-of-speech tagging task for code-mixed social media text. in section 4, we present the experimental setup and performance of our joint model. section 5 concludes the work. ## related work since the last decade, li and pos tagging for codemixed text has been a topic of interest in the field of natural language processing (nlp). cm language identification :(aguilar and solorio, 2019) used a large pre-trained model elmo, and adapted it to code-switching settings to obtain contextually rich embeddings. the paper used a bi-lstm crf for language identification. @xcite compared different word-level language identification systems for code-switched hindi-english data and a standard spanish-english dataset and found that the crf model works the best on the given datasets. @xcite performed li using crf on telugu-english cm data. @xcite used svm and crf for language identification on cm data. @xcite used hmm for language identification on seven languages. cm pos tagging: @xcite addressed the problem of prediction of pos tags for oov words in low resource languages using character-based word embedding as input features to a bi-lstm and crf network. @xcite ) used a meta embedding approach for the part of speech tagging where the word is represented in both code-mixed languages. thus, it maintains embeddings for each language and is processed appropriately at inference time without committing to one or the other language. it used a bi-lstm model for pos tagging. @xcite presented pos tagging results on english-hindi social media text using various ml and deterministic approaches, among which crf gave the best results. joint learning: the joint learning models have been studied for the code-switched li and pos scenarios. @xcite proposed a joint learning approach for pos tagging and li using recurrent neural networks. @xcite used a factorial crf for joint modeling of pos tagging and language identification. to the best of our knowledge, a joint learning model with bert and cnn architecture for language identification and pos tagging on codemixed data is not yet analyzed. in this section, we briefly describe the bert and convolutional models. we then introduce the proposed joint model for pos tagging and li of cm social media text. the architecture of the proposed model is given in figure 1 . ## experiments this section evaluates the above method with the individual pos tagging and language identification models on the cm dataset. ## conclusion in this paper, we presented the joint bert with the cnn model for pos tagging and li. the joint learning model allowed the pos tagging and li to be conditioned on each other to achieve better processing of code-mixed text. we tested our model with individual tasks. the results prove that our model achieves better metrics when compared to individual models. such relations can effectively achieve better sentence-level semantic representa-tion due to such diverse learning scope. recently meta embedding representations that include both the pre-trained embeddings and domain-specific fine-tuned embeddings are achieving great results in the field of nlp. the joint learning with meta embeddings is left as future scope.