A Language-aware Approach to Code-switched Morphological Tagging (2021)

Abstract: Morphological tagging of code-switching (CS) data becomes more challenging especially when language pairs composing the CS data have different morphological representations. In this paper, we explore a number of ways of implementing a language-aware morphological tagging method and present our approach for integrating language IDs into a transformer-based framework for CS morphological tagging. We perform our set of experiments on the Turkish-German SAGT Treebank. Experimental results show that including language IDs to the learning model significantly improves accuracy over other approaches.

## introduction morphological tagging is a well known sequence labelling task in natural language processing (nlp). it is the task of finding the correct morphological analysis for a given word form. the analysis is usually represented with a set of morphological features. tagging these features is beneficial in solving most nlp tasks since having knowledge about the morphological analysis of natural language words gives clues about their syntactic nature and their roles in context @xcite . morphological tagging becomes more important when the language in question is a morphologically rich one and the part-of-speech (pos) information about word forms is not sufficient to syntactically classify them @xcite . morphological tagging is challenging in itself 1 and it becomes more challenging when the processed language is code-switched, a phenomenon that occurs when bilingual speakers frequently switch between languages and produce utterances 1 for instance, in the conll 2018 shared task of multilingual parsing from raw text to universal dependencies, morphological tagging has the lowest range of scores among sentence segmentation, word segmentation, tokenisation, lemmatisation, and pos tagging. universaldependencies. org/conll18/results.html that include word forms and phrases from both languages. the challenge amplifies as the linguistic difference between the composing languages increases. this is because unlike pos annotation that can be made common across languages (e.g. universal dependencies @xcite ), morphological annotation is more language-specific. the example in figure 1 shows this difference explicitly. even though both autos in german and arabalarda in turkish share the same pos tag as noun, they have different morphological analyses. this difference stems from inherent properties of these languages. german employs grammatical gender while turkish does not. additionally in the example, the turkish locative case corresponds to german dative. such structural differences, combined with the rich morphology of individual languages taking part in cs data, make cs morphological tagging even more challenging with respect to cs pos tagging, a task that is a more common and more studied nlp task (cf. section 2). in fact, there has not been any research focused on cs morphological tagging before. we hypothesise that the language-dependent nature of morphological tagging can be solved more successfully for the case of cs data when the learning model has the knowledge of which language a word form belongs to. starting from this hypothesis, we search ways of including the language id (lid) information to tagging and present a language-aware approach. the proposed approach integrates lids to the dense representation of input tokens in a transformer-based learning model. we conducted experiments on the only cs dataset with complete morphological annotation (turkish-german sagt treebank @xcite ). 2 results show that the proposed approach outperforms all of the baselines significantly and the use of lids is beneficial in tagging morphology for cs data. our contributions are twofold: we present the first study on cs morphological tagging, and our data-driven method of integrating lids is applicable to any cs dataset and task that can exploit language ids. ## related work although there does not exist any prior study on cs morphological tagging, utilising language ids in other cs tasks has been quite common. we divide how lid is utilised into three methods: as part of a pipeline, as part of joint processing, and as machine learning (ml) features. while one or more of these techniques have been applied to many cs tasks, e.g. parsing @xcite , sentiment analysis @xcite , and normalisation (van der goot and çetinoglu, 2021), we focus here mainly on pos tagging, as it is a sequence labelling task and the closest one to morphological tagging. one of the most commonly used pipeline approach is processing the data as monolingual fragments @xcite @xcite @xcite . for each language in the mixed data, a monolingual model is trained. during prediction, the input is split into fragments according to their language ids and each fragment is processed by the respective monolingual model. the output is then merged into its original form. the advantage of this approach is to eliminate the need of cs data for training. however, context information is lost. the other common pipeline approach is using lids in decision-making after getting predictions from monolingual models. in this setup the mixed input is given to both monolingual models. the predicted lid is then used to select the model output of the corresponding language. @xcite is the first to use this approach on english-spanish pos tagging. later @xcite and @xcite used this setup for english-bengali-hindi, and for english-spanish and modern standard arabic-egyptian arabic, as well as the first pipeline technique. while in @xcite case using the second pipeline method slightly outperforms the first one, @xcite show the first pipeline outperforms by a large margin. thus we opted for the first architecture as one of our baselines. another model of @xcite was jointly trained lid and pos taggers that achieve a quite large improvement over their pipeline models. soto and hirschberg (2018) in many work from pre-neural era, lids are given as one of the features to ml models. while @xcite did not observe any significant improvement in doing so, @xcite shows that adding the lid of a token improves its pos tagging for english-hindi. @xcite and @xcite also inserted lid as a feature into their ml models. as a neural approach, @xcite represented the six lid labels existing in their data as boolean features and concatenated them with word vectors in a bilstm along with other features they used. different from the previous approaches, aguilar and solorio (2020) use language identification to create a code-switching elmo from english elmo @xcite . later they show the effectiveness of their cs-elmo by achieving state-of-theart pos tagging results on a hindi-english dataset @xcite . they also employ multi-task learning where their auxiliary task is language identification with a simplified lid tag set for lid, pos, and ner tagging. ## methodology for morphological tagging of cs data, we chose to use steps foot_1 @xcite as our framework. steps is an nlp tool for tagging and syntactic parsing in universal dependencies (ud) style @xcite . our motivation behind deciding on steps as our framework is based on two reasons. first, for token representation it utilises transformer-based language models, which have recently become famous for their outstanding success in various nlp tasks @xcite . second, steps is an open-source system with a minimum use of black-box modules that make the modification of the source codes very challenging, if not impossible. moreover, steps is a current state-of-the-art nlp tool that outperformed other state-of-the-art tools udify @xcite and ud-pipe 2.0 @xcite in tagging and parsing of several languages @xcite . section 3.1 gives a brief description about steps. sections 3.2 and 3.3 describe the baseline methods and our proposed approach for integrating lids to cs morphological tagging, respectively. ## analysis on lid integration lid representation and initialisation in section 3.3, we mention two more ways in addition to our preferred approach for the representation of lids as vectors. the first way is representing lids as one-hot vectors. we define each lid label as a one-hot vector and concatenate these vectors with token embeddings provided by the lan-guage model as in lidvec. we experimented with this approach on the development set. however, this method showed poorer performance than standalone which does not utilise lids in any way. we believe that one-hot vector representation might be too rigid to be used together with token embeddings due to the fact that the range of the values in these two representations greatly vary. the second method for the lid vector representation includes the initialisation of lid embeddings by averaging the embeddings of same-lid tokens in the training set. in the initial experiments we see that when we use the average initialisation instead of a random initialisation, the training phase progresses faster and the learning stops early when the training accuracy is around 85%, in contrast to the random initialisation in which the training phase ends after a higher number of epochs and with a higher training accuracy. so, we extended the training time by changing the early stop criteria from 15 epochs to 50 epochs to give the average initialisation an opportunity to show its true capacity. figure 3 compares the performance of these two initialisation methods for two different early stop criteria on the development set. we see that the underfitting in the average initialisation method is eliminated as the number of epochs increases. overall, the performance of both initialisation methods is the same when they are trained sufficiently. we conclude that random initialisation can be preferred if there are time restrictions. the impact of lid prediction we proposed three different approaches for lid integration. in terms of resources needed, mtl does not need an external lid prediction by definition, since it predicts lids and morphology jointly. however, it is also the worst performing one among the three approaches. dsplit and lidvec both outperform mtl, but require predicted lids to function. to test how sensitive these models to the lid prediction accuracy, we evaluated dsplit and lidvec with marmot, a crf-based sequence tagger @xcite which has ~96% accuracy in lid prediction instead of the steps lid model with ~99% accuracy (cf. dsplit also requires pre-and post-processing of the input and output, respectively. considering its superior performance, and the robustness and compactness of its architecture, we suggest lidvec as the best approach to cs morphological tagging among the models discussed in this paper. ## qualitative analysis most common improvements we observe that integrating language ids contributes to a 10% in-crease in predicting the presence of possessive markers in turkish nouns, which are not a feature of german nouns. this is something expected since providing lids enables the model to differentiate between the different sets of morphological features of two languages better. similarly, the lid knowledge makes a 4% enhancement in predicting the existence of the gender feature that is present in german nouns but absent in turkish ones (cf. figure 1 ). to understand this better, we compared lidvec and standalone in terms of their feature-based success. in this feature-based performance measurement, partial matches are also given scores in contrast to the evaluation metric we adopted, which counts a predicted morphological tag as correct only if it is an exact match -i.e., all the features that constitute the morphological tag are predicted correctly. we measure the featurebased performance of the models by dividing each morphological tag into features and counting each feature match as a point. when we look at what categories benefit most from including lids, we see that for turkish they are verbs and nouns with an improvement of 11% and 10%, respectively. for german they are pronouns and nouns with 9% improvement. the success of morphology prediction for german verbs is already high for all models. hence, there is not much improvement in german verbs. we observe that all the nouns and pronouns in both languages and also the verbal nouns in turkish which are derived from verbs have the case feature in their morphological analyses. confusion in case feature values although all models easily predicted the existence of the case feature, they had the most trouble in deciding the value of it. hence, we created confusion matrices of standalone and lidvec for different values of the case feature on the development set as given in figure 4 . there are only four case markers in german: nominative, accusative, dative, and genitive. in turkish, there are three additional case markers, namely ablative, instrumental, and locative. albeit having a german lemma, mixed tokens in the sagt treebank are annotated according to turkish morphological annotation style due to the presence of turkish suffixes in them. we observe that the most confusion occurs between nominative and accusative cases for all three token types. this confusion in tr and mixed tokens results from the fact that the accusative suffix which makes the case of a word accusative and the possessive suffix in nominative nouns sometimes correspond to the same form in turkish. in de tokens, the situation is similar in the sense that nominative and accusative forms of german articles are different only for masculine, whereas they have the same form when their gender is feminine or neutral, or when they are in plural. lidvec consistently reduces this confusion and predicts correct cases that plays an important role in its overall performance. improvement on mixed tokens when observing the results in tables 3 and 4 , the notable success of lidvec on predicting morphological analyses of mixed tokens caught our attention. even when predicted lids are used, lidvec outperforms standalone by a large margin in the development and test sets. we observe that mixed tokens in the sagt treebank are mostly nouns. therefore mixed tokens get their share from overall case improvements. when proportioned to the total number of cases in each category, the success of lidvec is most visible in mixed tokens. ## conclusion in this paper, we tackle the morphological tagging problem for cs data. we present some challenging aspects of the task and suggest the use of token-wise lid information. we experience with different ways of using lids on a transformerbased model and propose the lid vectors approach. our proposed model outperforms all the baselines significantly and proves to be a robust and compact way of lid integration. being first on focusing morphological tagging on cs data, our study shows that utilising lids is an effective method in this task. we also give the first results on lid, pos, and morphological tagging on the turkish-german sagt dataset. an implementation of our model is available at https: //github.com/sb-b/steps-parser .