Supervised and Unsupervised Evaluation of Synthetic Code-Switching (2022)

Abstract: Code-switching (CS) is a phenomenon of mixing words and phrases from multiple languages within a single sentence or conversation. The ever-growing amount of CS communication among multilingual speakers in social media has highlighted the need to adapt existing NLP products for CS speakers and lead to a rising interest in solving CS NLP tasks. A large number of contemporary approaches use synthetic CS data for training. As previous work has shown the positive effect of pretraining on high-quality CS data, the task of evaluating synthetic CS becomes crucial. In this paper, we address the task of evaluating synthetic CS in two settings. In supervised setting, we apply Hinglish finetuned models to solve the quality rating prediction task of HinglishEval competition and establish a new SOTA. In unsupervised setting, we employ the method of acceptability measures with the same models. We find that in both settings, models finetuned on CS data consistently outperform their original counterparts.

## introduction code-switching (cs) is a phenomenon of mixing words and phrases from multiple languages within a single sentence or conversation 1 . it is common for multilingual speakers and happens across various language pairs across the globe, such as spanish-english (spanglish) and hindi-english (hinglish). various studies @xcite have predicted the high growth in the number of cs speakers, which would surpass the number of native speakers in various globally popular languages (e.g., english). the advent of social media has highlighted the amount of cs communication and lead to a further increase of the number of multilingual speakers 1 some works make a distinction and refer to intrasentential (within a single sentence) code alternation as "code-mixing" (cm) and intersentential (at or above the sentence level) as "code-switching" (cs). it is also common, however, to use the term "cs" for both cases. intrasentential code alternation is the focus of this paper and we refer to it as "cs". various studies have shown that cs data may pose a challenge for contemporary multilingual models @xcite . finetuning on cs data can alleviate this problem (e.g. @xcite . as social media can be noisy and not readily available to build a large scale corpus, various techniques of generating synthetic cs have been proposed (see section 2). however, it was shown that the performance of the models crucially depends on the quality of cs text used for pretraining @xcite . this creates the task of synthetic cs evaluation which is the main focus of current paper. cs evaluation methods range from computing intrinsic text metrics to measuring downstream task performance depending on the cs data used for pretraining and human evaluation (see section 2). @xcite show that most cs evaluation metrics fail to capture the linguistic diversity which leads to poorly estimating the quality of cs text. thus, human evaluation remains as a reliable method. @xcite propose hinge, a dataset of hinglish sentences with human quality ratings and organise hinglishe-val shared task based on it @xcite . in our paper, we address hinglisheval quality rating prediction task with hinglish models proposed in @xcite . moreover, we add an unsupervised setting of the task. our main contributions are: evaluating synthetic cs despite the practical need of synthetic cs datasets, the task of evaluating synthetic cs remains relatively understudied. some evaluation techniques involve estimating intrinsic text properties, such as code-switching ratio and length distribution. one of the most popular metrics is code-mixing index (cmi) @xcite , which accounts for code-switching ratio and the number of switches in a sentence. we defer to @xcite for a detailed overview of other metrics used for evaluating cs nlg. further, extrinsic measures can be used, like the perplexity of external language model. for example, @xcite propose a finetuned hinglish gpt model and suggest using it for evaluation. also, downstream task performance can be measured, depending on the cs data used for augmentation @xcite . downstream tasks are organised into benchmarks such as gluecos @xcite and lince @xcite which comprise data for popular language pairs like english-hindi and english-spanish. finally, human evaluators can be employed to assess the quality of the generated cs. there are examples of such evaluation studies in the literature which are usually performed to prove the quality of the proposed cs generation method @xcite . however, these studies are of low scale and do not result into substantial datasets which can be used in further research. in this context, hinge dataset @xcite is unique being the largest collection of synthetic cs with human ratings to date. it is described in detail in section 3.1. based on hinge, hinglisheval competition was organised (srivastava and singh, 2021c; see section 3.1.1), where the task is to model the annotators' opinion on cs sentences. language models for cs along with the development of language models (lms), work has been done to adapt them for cs data. @xcite compare different n-gram lms, @xcite suggest to improve language modeling by generating artificial cs text. a number of works propose lms that incorporate a syntactic constraint @xcite @xcite . another line of papers introduce lms where the output layer is factorized into languages, and pos tags are added to the input @xcite @xcite . with the advent of transformers @xcite , work has shifted to applying popular archi-tectures to cs data. @xcite show that m-bert can achieve promising results in hinglish downstream tasks when hindi parts are written in devanagari even in a zero-shot setup. the same, however, does not apply to romanized hinglish, as m-bert was pretrained on devanagari hindi. both gluecos @xcite and lince @xcite benchmarks provide m-bert baselines for their leaderboards. @xcite show that bert models produce better results in cs lid when pretrained on cs sentences rather than on multiple monolingual corpora. @xcite find that finetuning m-bert on natural cs data gives the best performance improvement compared to any synthetic cs. nayak and joshi (2022) length up to three tokens are aligned between the parallel sentences. the aligned hindi phrase is replaced with the corresponding english phrase. for both algorithms, the hindi parts are then transliterated into the roman script. third, an average of two human quality ratings on a scale of 1-10 is assigned to each synthetic hinglish sentence. refer to fourth, annotators' disagreement is given, which is calculated as the absolute difference between the human quality ratings and ranges 0-9. finally, for each pair of parallel sentences, at least two humangenerated hinglish sentences are provided. figure 1 demonstrates an example of the described fields of the dataset. overall, hinge contains 1976 parallel hindi-english, 3952 synthetic cs and 4803 humangenerated cs sentences. all synthetic cs sentences have human scores assigned to them, and hinge is the largest such dataset to date. we refer to the synthetic part of the dataset as hinge all. ## discussion our experiments show that both in unsupervised and supervised setups, models pretrained on hinglish data consistently outperform their original counterparts. this goes in line with previous studies which have shown that pretraining on cs data yields better results than monolingual pretraining @xcite . on hinglisheval 10-class classification, our hingmbert-mixed establishes new sota, surpassing the m-bert baseline from @xcite which was trained solely on hinglish sentences from hinge. moreover, our hinglish models trained solely on hinglish sentences produce scores competitive with the participants of hinglisheval shared task which use all available information from hinge (original hindi and english sentences and annotators' disagreement; @xcite @xcite . ## conclusion and further work in this paper, we address the task of evaluating synthetic cs in supervised and unsupervised approaches. in supervised setting, we solve hingli-sheval quality rating prediction task with a line of finetuned hinglish transformer models and establish a new sota. in unsupervised setting, we apply the method of acceptablity measures to evaluate the synthetic cs sentences in hinge dataset. we find that hinglish finetuned models consistently outperform their original versions. several further work directions open up based on this work. first, it is promising to directly compare the unsupervised and supervised approaches presented in this paper, possibly applying the semisupervised method of @xcite for acceptability measures. second, it is of interest to continue the analysis presented in section 6.1 with various cs metrics, thus repeating the study of @xcite on a larger scale.