Aligning Multilingual Embeddings for Improved Code-switched Natural Language Understanding (2022)

Abstract: Multilingual pretrained models, while effective on monolingual data, need additional training to work well with code-switched text. In this work, we present a novel idea of training multilingual models with alignment objectives using parallel text so as to explicitly align word representations with the same underlying semantics across languages. Such an explicit alignment step has a positive downstream effect and improves performance on multiple code-switched NLP tasks. We explore two alignment strategies and report improvements of up to 7.32%, 0.76% and 1.9% on Hindi-English Sentiment Analysis, Named Entity Recognition and Question Answering tasks compared to a competitive baseline model.

## introduction large pretrained multilingual models have enabled cross-lingual transfer on a number of downstream natural language understanding (nlu) tasks. apart from serving as a good starting point to train models for tasks in low-resource languages, multilingual models @xcite have also been used to achieve zero-shot cross-lingual transfer on target languages with no task-specific labeled data. however, compared to monolingual inputs, the effectiveness of multilingual models on code-switched inputs-i.e., inputs with two or more languages appearing within or across sentences in a conversation-has not been explored enough. in this work, we aim at explicitly modifying representations from pretrained multilingual models to be more amenable to code-switched inputs. we do this with the help of parallel text in the two component languages and alignment objectives that explicitly encourage representations to be better aligned across the two languages. we conjecture that modifying multilingual embeddings to be better aligned across the two languages will help the model deal better with tokens switching languages within a code-switched sentence. we start with a pretrained multilingual bert (mbert) baseline model @xcite and design two alignment objectives to be used with parallel text to align the multilingual embeddings. this "aligned" mbert model is then further fine-tuned with small amounts of code-switched labeled data in the target task. we find such an aligned model to be more accurate on multiple downstream tasks involving code-switched inputs. the two main highlights of this work can be summarized as follows: we also present visualizations that clearly show that the alignment objective helps bring representations for aligned words in hindi and english closer together. ## methodology we explore two different objectives to encourage cross-lingual contextual alignment in the mbert model. for this, we need access to parallel text in the component languages corresponding to the code-switched language of interest. we propose both a sequence-level alignment objective that is contrastive in nature, and a word-level alignment objective that is based on minimizing distances between aligned word embeddings. ## conclusion in this work, we propose aligning multilingual embeddings using sentence-level (contrastive) and word-level (non-contrastive) objectives. such an explicit alignment leads to improved performance on three code-switched hindi-english nlp tasks: sa, ner and qa. future work will explore the use of alignment objectives in a multi-task framework with the target tasks.