Improving Code-Switching Dependency Parsing with Semi-Supervised Auxiliary Tasks (2022)

Abstract: Code-switching dependency parsing stands as a challenging task due to both the scarcity of necessary resources and the structural difficulties embedded in code-switched languages. In this study, we introduce novel sequence labeling models to be used as auxiliary tasks for dependency parsing of code-switched text in a semi-supervised scheme. We show that using auxiliary tasks enhances the performance of an LSTM-based dependency parsing model and leads to better results compared to an XLM-R-based model with significantly less computational and time complexity. As the first study that focuses on multiple code-switching language pairs for dependency parsing, we acquire state-of-the-art scores on all of the studied languages. Our best models outperform the previous work by 7.4 LAS points on average.

## introduction code-switching (cs) is the producing of utterances by combining phrases and word forms from multiple languages. this is a phenomenon observed frequently in utterances of bilingual speakers @xcite . figure 1 shows an example to this type of utterance formation. although much work has been done on the syntactic parsing of monolingual languages, cs language pairs are quite understudied in this regard. there have been only a few studies on cs dependency parsing @xcite @xcite , each focusing only on a single cs language pair. although cs dependency parsing also benefited from the recent rise of multilingual and cross-lingual natural language processing (nlp) models as shown by @xcite , these models, which are usually trained on monolingual corpora, are insufficient on cs parsing. the poor performance on cs language pairs is not only due to the lack or scarcity of the training data but also because of the shortage on resources required by deep neural models such as pretrained embeddings, language models, or even raw data. in addition, each language composing a cs language pair inherits its own structural difficulties which contributes a good deal to the problem. recently, a small number of cs treebanks were manually annotated within universal dependencies (ud) @xcite . even though these treebanks have little to no training data, their existence provides an opportunity to study dependency parsing also on cs language pairs. in such low-resource scenarios, utilizing raw data can be helpful in boosting the performance. a common method to benefit from raw data is selftraining @xcite , a semi-supervised approach where a small number of labeled data is used to train a model that is later used to predict labels for unlabeled data. this pseudo-labeled data is then combined with the initial data to re-train the model. this method is usually found successful in low-resource scenarios @xcite , although error propagation is a known problem when pseudo-labels are noisy. with very restricted resources, we hypothesize that cs dependency parsing can also benefit from unlabeled data. based on this hypothesis, we form our first research question: is using pseudo-labeled data directly beneficial for cs dependency pars-ing or can we find better ways of integrating the knowledge from pseudo-labeled data? starting from this question, we follow a deep contextualized self-training approach @xcite and integrate semi-supervised auxiliary tasks to the parsing architecture to enhance cs dependency parsing. our method enhances a widely-used bilstm-based parser @xcite by training parsing-related auxiliary sequence labeling tasks on automatically labeled data and combining these trained auxiliary task models with the base parser through a gating mechanism. we introduce new sequence labeling tasks that are shown to be beneficial in improving the parsing performance. seeing the success of our semi-supervised enhancement method on the bilstm-based parser, we form our second research question: can we reach even better parsing scores if we combine this enhancement method with xlm-r @xcite , a state-of-theart (sota) transformer-based language model that shows superior performance on many nlp tasks? our experimental results demonstrate notable success of our proposed models over the previous stateof-the-art on these treebanks. our contributions are as follows: ## related work code-switching dependency parsing is a newlystudied research area. the first cs ud treebank was created by @xcite which included only a test set of hindi-english sentences. in the absence of cs training data, the test set was split to monolingual fragments and existing hindi and english monolingual treebanks in ud were used to parse these fragments. @xcite extended this dataset with a cs training set. they trained a bilstm architecture on this additional training data by also integrating syntactic knowledge extracted from monolingual treebanks. @xcite laid the first foundations of a komi-russian ud treebank with 25 cs sentences. they adopted a multilingual parsing approach @xcite and used russian and komi monolingual training data with bilingual komi-russian word embeddings. later, this treebank expanded into the komi-zyrian ikdp treebank @xcite . @xcite created a turkish-german ud treebank from a turkish-german spoken corpus. @xcite introduced the maghrebi arabic-french treebank and performed parsing experiments on the treebank using udpipe @xcite . this treebank is yet to be included in the ud. a frisian-dutch ud treebank which includes only test data was introduced by @xcite . the authors performed data selection from eight related monolingual treebanks using latent dirichlet allocation @xcite to create a training set. their experiments performed using a deep biaffine parser @xcite demonstrated no significant performance difference between training the parser on the selected training set and only on a dutch monolingual treebank. lately, multilingual and cross-lingual parsing studies have begun to include cs treebanks in their experimental setups. @xcite presented a multi-task learning tool that utilizes multilingual bert @xcite to perform several nlp tasks, including dependency parsing. evaluation was done on all available ud treebanks which include cs ud treebanks mentioned above. the model was fine-tuned on training set of each treebank, which is also the case for hindi-english and turkish-german cs treebanks. for frisian-dutch and komi-russian cs treebanks with no training data, they used dutch alpino and russian syntagrus treebanks, respectively. müller our study on cs dependency parsing differs from the previous work in the sense that none of the previous work utilized raw cs data to improve parsing in a semi-supervised scheme. ## results and discussion comparison to baselines on all treebanks, the auxiliary task enhancement methods improve the scores when compared to base lstm by 4.94 points in uas and 3.86 points in las on average. the best performing enhancement model differs across treebanks. we observe the same pattern for the traditional self-training method. self-training fails to surpass the proposed approach on any of the treebanks. its parsing performance even falls below that of base lstm on kpv-ru and tr-de. it shows the highest improvement with respect to base lstm on fy-nl. yet, the best one of the auxiliary task enhancement methods significantly outperforms self-training on each treebank. new individual tasks the +lih model which employs lids performs best on kpv-ru, and second best on hi-en. its performance on tr-de and fy-nl is comparable with the other models. it is also in the best combination ensemble for all treebanks. this indicates the importance of language ids in cs dependency parsing. the +smh model which is only applied to kpv-ru and tr-de is the best performing one on tr-de. however, all other tasks outperform +smh on kpv-ru. this might be due to the quality difference in morphological taggers trained on these treebanks. the morphological tagger we trained on the cs training set of tr-de has an accuracy of 82% on its test set. however, to train a tagger for kpv-ru we used monolingual komi data only. accuracy of this tagger on kpv-ru test set is 66%. it seems the kpv-ru parser suffers from error propagation. the simplest enhancement model +pc performs comparable to others, even outperforming +noc and +dtr on kpv-ru and tr-de. since it only needs the root position in the sentence to perform, this model can be an alternative to other models when gold/predicted pos or morphological tags are hard to acquire. it can also be preferred when the error propagated to the auxiliary tasks from the base parser through predicted trees is high, damaging accuracy of the tasks that rely on these parses. individual tasks vs ensembles ensembling multiple tasks improves uas and las on hi-en and fy-nl and las on tr-de when compared with the best performing single task. the +best combination ensemble works better on fy-nl and tr-de than the +noc,+dtr,+rpe ensemble proposed by @xcite . looking at the overall results, we observe that including +rpe and +lih together has a favorable effect on improving cs parsing performance. who benefits most and least? fy-nl is the most benefited treebank from the proposed model. the best performing enhancement model +best combination on fy-nl achieves almost 10/7 points increase in uas/las when compared with base lstm . the least benefited treebank is kpv-ru with 2.5/1.1 points increase in uas/las. having similar amount of unlabeled data and no cs training data, these treebanks differ in their training data amounts. the dutch alpino treebank used to train fy-nl models has 13,603 sentences whereas the komi-zyrian lattice treebank for kpv-ru models includes 662 sentences. so, automatic parsing of unlabeled data of kpv-ru by a model trained on 662 sentences can be much noisier than that of fy-nl. in appendix a.2, we show that the performance ranking of the systems does not change by the amount of gold training data. from the table, we observe that there is a tradeoff between performance and resource consumption for the three models. the training time of the base lstm model is the shortest. yet, our best model improves the performance significantly at the expense of a slight increase in training time. base xlmr has the longest training time by a large margin. in terms of memory usage, there is a similar pattern to that of training time. base lstm needs approximately 50% less memory than our best model, yet there is on average 3.86 points gap between las of the two models. base xlmr is again the least preferable model here due to its highest memory consumption and low performance on parsing the treebanks with the exception of tr-de. only for tr-de it outperforms the other two models and can be the model of choice for the parsing of tr-de data. considering the long training time and high resource consumption of the xlm-r-based parser and the success of our lstm-based enhancement models, we suggest lstm-based auxiliary task enhancement for low-resource dependency parsing of cs data. ## conclusion in this paper, we focus on cs dependency parsing. we present a semi-supervised auxiliary task enhancement to a graph-based neural parser and create novel sequence labeling tasks that are shown as useful in improving the parser's success. experimental results show that our enhancement technique achieves sota performance on all cs ud treebanks and helps better utilization of unlabeled data for cs dependency parsing. we combine our enhancement models with xlm-r to see their performance on a multilingual transformerbased model. results demonstrate that the powerful xlm-r shows limited performance and fails to surpass our semi-supervised auxiliary task enhancement models. our implementation of the proposed sequence labeling tasks and the xlm-r-based enhancement are publicly available for research purposes at https://github.com/ sb-b/ss-cs-depparser .