Intrinsic evaluation of language models for code-switching (2021)

Abstract: Language models used in speech recognition are often either evaluated intrinsically using perplexity on test data, or extrinsically with an automatic speech recognition (ASR) system. The former evaluation does not always correlate well with ASR performance, while the latter could be specific to particular ASR systems. Recent work proposed to evaluate language models by using them to classify ground truth sentences among alternative phonetically similar sentences generated by a fine state transducer. Underlying such an evaluation is the assumption that the generated sentences are linguistically incorrect. In this paper, we first put this assumption into question, and observe that alternatively generated sentences could often be linguistically correct when they differ from the ground truth by only one edit. Secondly, we showed that by using multi-lingual BERT, we can achieve better performance than previous work on two code-switching data sets. Our implementation is publicly available on Github at https://github.com/sikfeng/language-modelling-for-code-switching .

## introduction code-switching (cs) is the phenomenon where a multilingual speaker alternates between the use of multiple languages in the same utterance @xcite . this practice is common among bilingual speakers, e.g., chinese @xcite , spanish or hindi-speakers @xcite who are bilingual in english. to build automatic speech recognition (asr) systems for cs utterances, a good language model is important, especially when cs training data is often scarce. language models (lms) are an important part of text generation systems in applications such as machine translation and asr systems. intrinsic evaluations of lms are often performed using perplexity, a measurement of how well a lm predicts a test sample. however, previous work has noted that perplexity does not correlate well with asr performance (e.g. @xcite . extrinsic evaluations of lms with asr performance, on the other hand, might be over-fitting to the asr used in the evaluation. to address this problem, @xcite proposed a new evaluation metric based on a classification task. first, given a ground truth sentence, they used finite state transducers (fst) to generate a set of phonetically similar sentences. they evaluate lms based on the task of identifying the ground truth sentence from this set of sentences. lms can thus be evaluated based on the accuracy of this task, independent of any specific asr. for this evaluation task, @xcite proposed a discriminative model that outperformed generative baselines on a english-spanish (en-es) cs test set. however, such an evaluation criteria assumes that all generated sentences are linguistically incorrect. in this paper, we make two contributions 1. we found that generated sentences with only one modification from the ground truth could be linguistically correct, violating the assumption that such sentences are negative examples. this poses a problem both during training and evaluation. to alleviate this problem, we propose to generate sentences with at least two edits from the ground truth, and 2. we fine-tuned multi-lingual bert (mbert) to improve their published results on their data set. we also experimented with a second english and chinese (en-zh) cs data set. ## related work the phenomenon of cs is well studied in linguistics (e.g. @xcite . earlier work often focus on en-es code-switching, studying minorities who claim spanish as their mother tongue in the english-speaking america. today, there is also much interest in the indian languages codeswitched with english, due to the large number of indians using social media (e.g. @xcite . recently, researchers in singapore have collected a cs data set of en and zh @xcite , reflecting the en-zh bilingual culture in singapore and malaysia. research on cs often focuses on downstream nlp tasks: khanuja et al. (2020b) built a data set called gluecos to evaluate nlp performance on cs data; winata et al. (2021) examined the effectiveness of multi-lingual embedding such as mbert on downstream tasks such as part-ofspeech tagging. the first en-zh lm was designed by vu et al. (2012) using statistical machine translation techniques. with deep learning, lms were also trained directly on cs data (kumar et al., 2020). luo et al. (2018) trained an endto-end en-zh cs speech recognition models, using a connectionist temporal classification and attention based model. qin et al. (2020) used dataaugmentation with cs data as an approach to facilitate cross-lingual transfer learning. winata et al. ( 2019 ) used a sequence-to-sequence model to generate cs data for data augmentation. @xcite pre-trained a bilstm (bidirectional long short term memory) on monolingual data before fine-tuning on cs data. in this paper, we propose to replace this approach by fine-tuning mbert @xcite . we show that our model outperformed bilstm @xcite @xcite generated alternative sentences with similar phonemes, and assumed that these generated sentences are linguistically incorrect, using them as negative examples in training and evaluation. however, @xcite have found that word form similarity increases with semantic similarity across a number of languages. hence, making a single phoneti--她/他/它去 airport (she/he/it goes to the airport) -no i remember seeing the same/sign -so we brought her some/son ribs so -i was talking to your mom/ma'am/man figure 2 : examples of sentences with alternatives of a single word separated with /. these generated replacements could be correct without more context. the translation for the first cs sentence is provided in brackets. ## generating evaluation data sets we experimented on two data sets: the en-es data used in @xcite , created from the bangor miami corpus, and the en-zh seame corpus @xcite , collected from residents of singapore and malaysia who are bilingual in en and zh. to generate sets of alternative monolingual and code-switched sentences, we took the same approach as @xcite . first, we convert a sequence of language tagged words (either cs or monolingual) into a sequence of the matching phonemes (using pronunciation dictionaries); then we decode the sequence of phonemes into new sentences, which include words from either language (possibly both); during decoding, we allow minor changes in the sequence of phonemes to facilitate the differences between the languages. these steps can be easily implemented using composition of finite-state transducers (fsts). we used the fst provided by @xcite for the en-es data. to build the fst for en-zh, we obtained word probabilities for chinese words from the mandarin chinese news text corpus, while we used the english word probabilities provided by @xcite . we used the jieba tokenizer foot_0 to segment chinese words, and mapped the chinese pinyin provided by cc-cedict. 3 we observed that if we substitute only one word, the altered sentence could sometimes still be linguistically correct. for example, in the cs sen-tence ''她/他/它去 airport (translated: he/she/it goes to the airport)", the three alternative words are different pronouns that have identical pronounciation. we give a few other examples in figure 2 . hence, we generate alternative sentences with at least two differences from the original sentence. this increased the probability that the generated sentences are linguistically incorrect, so that they serve their purpose as negative sentences both for training and for evaluation. for each sentence, we generated an alternative set that contains monolingual (for both languages) and cs sentences. the fst is not always able to produce sufficient alternatives for a given sentence: if the fst generates less than 5 alternatives for any type, we discard that set. we capped the number of alternatives of each type to 10, and hence each set has at most 30 sentences. we did a train-validation-test set split such that both the validation and test set contain 1,000 alternative sets each. for en-es, the training set consists of 1,021 cs sets, 7,171 en sets, 3,489 es sets, while for en-zh, it contains 20,689 cs sets, 6,515 en, and 7,050 zh sets. ## discriminative training language models are usually trained generatively to generate positive sentences. @xcite proposed a discriminative bilstm model by training the bilstm to discriminate between ground truth positive sentences with synthetically generated negative sentences. they used as their base model a 2-layer bilstm using dynet @xcite . the objective of their discriminative training was to assign a higher score to ground truth sentences compared to synthetically generated ones. we denote s 0 as the original ground truth sentence, and alt(s 0 ) to be the set of alternative sentences generated for s 0 . each sentence is scored using score(s) = w • bilstm(s), where @xmath0, with n the dimension of the embedding. ## experimental results in this section, we describe the empirical comparison between bilstm and mbert on the two data sets, and analyze the results. we experimented with different versions of bil-stm on the en-zh data. for the monolingual data sets, we experimented with using (1) only the alternative sets generated from seame, (2) only the alternative sets from mandarin chinese news text corpus and opensubtitles, and (3) all of the above. we obtained the best results with (3). using a 3-layer bilstm did not improve results, which could be due to insufficient training data. we compare the different versions of mbert in and goldberg (2019) on our generated data sets where alternative sentences are generated with at least 2 differences from the ground truth. we arrive at the same conclusions as before: mbert does better than bilstm, and the different variants of our mbert models achieved similar results. ## conclusion in this work, we proposed to modify the evaluation of language models proposed by @xcite . while the evaluation proposed by @xcite provided for a simple approach to intrinsic evaluation of language models for asr, we highlighted the problem of using linguistically correct sentences as negative examples, introducing noise both in the training and in the evaluation process. we proposed a simple solution to this problem by requiring a minimum edit distance between the generated samples and the ground truth. in addition, we showed that mbert outperforms bilstm on two code-switching data sets using this evaluation criteria.