Modeling Code-Switch Languages Using Bilingual Parallel Corpus (2020)

Abstract: Language modeling is the technique to estimate the probability of a sequence of words. A bilingual language model is expected to model the sequential dependency for words across languages, which is difficult due to the inherent lack of suitable training data as well as diverse syntactic structure across languages. We propose a bilingual attention language model (BALM) that simultaneously performs language modeling objective with a quasi-translation objective to model both the monolingual as well as the cross-lingual sequential dependency. The attention mechanism learns the bilingual context from a parallel corpus. BALM achieves state-of-the-art performance on the SEAME code-switch database by reducing the perplexity of 20.5% over the best-reported result. We also apply BALM in bilingual lexicon induction, and language normalization tasks to validate the idea.

## introduction monolingual language modeling has enabled many nlp tasks @xcite @xcite . however, the bilingual language model was not well studied. the recent advances in cross-lingual word embedding (clwe) @xcite , which projects word of different languages into a shared embedding space for cross-lingual representations @xcite , make possible some cross-lingual applications. unfortunately, they are not optimized to model the sequential dependency for word prediction in a bilingual text. in this paper, we would like to propose a bilingual language model that can learn word embeddings to represent the equivalent words between two languages, and more importantly, to model the sequential dependency for words across languages at the same time. for instance, the model should be able to predict the appropriate word to fill in the blank, given the bilingual context: it is tempting to think that, given enough of codeswitching text data, bilingual language modeling could be approached in the same way as that for monolingual data. the main challenge is the lack of such cs data. we note that cs mainly occurs in the spoken form, and cs does not occur in every sentence. therefore, collecting enough pure cs data is just not practical or even feasible @xcite . the problem is further exacerbated by the syntactic constraints of the two diverse languages, such as chinese and english. three dominant theories seek to explain the syntactic formation of cs sentences. they are the matrix language frame theory @xcite , which shows that individual monolingual sentences will conform to the grammar of the matrix language. the equivalence constraint theory @xcite , which further constrains the intra-sentential cs points to the syntactic boundaries shared by both languages, and the functional head constraint theory @xcite that imposes constraints on the functional head and its complements. a bilingual language model should be able to predict a word, either in the matrix language or otherwise, given either a bilingual or monolingual context. therefore, it has to respect the respective monolingual word sequential dependency, the cross-lingual word correspondence, as well as the switching rules between languages. the contributions of this paper are summarized as follows: ## related work several prior studies related to bilingual language modeling are the inspiration for this work. cross-lingual correspondence: several studies are focused on projecting words of different languages onto the common embedding space to establish cross-lingual correspondence. one idea is to train a model using bilingual information from corpora aligned at the sentence level @xcite @xcite and document level @xcite . another is to exploit the isomorphic structure @xcite , dictionary @xcite @xcite , shared cognate, vocab @xcite , numeral @xcite through ad-hoc projection. as the above approaches do not explicitly consider the sequential dependency of words, the embedding doesn't encode the word ordering information. the multilingual techniques, such as m-bert @xcite and xlm @xcite , do not explicitly model the syntactic constraints for cs as formulated in the equivalence constraint theory, thus not making full use of the information which could potentially improve their performance. code-switching modeling: another school of thoughts is to extend the monolingual language modeling technique to accommodate code-switch content. adel et al. (2013b, 2014 ) use factored language models and recurrent neural network (rnn) language model to improve the bilingual language model for cs asr rescoring. they include additional linguistic information such as partof-speech, language identifier to improve model generalization. inversion constraints @xcite and functional head constraints @xcite are also used in language models for the asr decoding process. @xcite use cross-lingual embedding to tie the input and output layer, and incorporate classes in the rnn language model. while these models are effective, they rely on the availability of cs training data. therefore, they are not easily scalable. to address this, we propose a way to make use of the existing abundant parallel corpora. the method will be explained in section 3.3. code-switching text generation: closer to our line of research, @xcite propose to use synthetic data following the equivalence constraint theory, while @xcite apply the matrix language frame theory. in their works, a parser or an aligner is required to process the parallel corpus, which is followed by the standard monolingual language modeling process. such techniques suffer from inaccurate alignment or parsing errors. these errors will be carried forward when training the language model. more recently, @xcite propose a technique to generate neuralbased synthetic data using parallel sentences, in which a point-gen network is used to synthesize cs data without external aligner or parser. in this paper, we propose to learn the bilingual context and the cs language model jointly by attending to the parallel sentences directly without the need for an external aligner, parser or explicitly generating the synthetic data. ## bilingual attention language model next, we discuss the motivation and the theoretical formulation of the proposed bilingual attention language model (balm). in a bilingual text, we could encounter a sequence of word, @xmath0, w l 2 2 , . . . w l 2 t , . . . , w l 1 t , code mixed between languages l 1 and l 2 . however, such code mixed training data are not easily available. let us assume that only parallel corpus at sentence level between l 1 and l 2 languages is available to us. assuming the validity of the matrix frame theory, and equivalence constraint theory, the above code-switch sentence, w, can be constructed from two parallel sentences, for a monolingual case, the language model maximizes the log-likelihood of p(w t |@xmath1. for a cs case, we would like to maximize p(w t |@xmath2, whereby the bilingual context, @xmath3, is non-existent during training. in the subsequent section, we will explain the idea to encode the bilingual context using an attention mechanism. ## conclusion we note that balm is an implementation of l 1 l 2 → l 1 l 2 . the experiments show that it outperforms all state-of-the-art models in the literature for similar tasks. the results validate the idea of bilingual attention. the same balm can be used in l 1 l 2 → l 1 or l 2 for language normalization. it can be further extended for l 1 → l 1 l 2 , or l 2 → l 1 l 2 for code switch sentence generation, and l 1 → l 2 , or l 2 → l 1 for machine translation.