On Utilizing Constituent Language Resources to Improve Downstream Tasks in H inglish (2022)

Abstract: Performance of downstream NLP tasks on code-switched Hindi-English (aka ) continues to remain a significant challenge. Intuitively, Hindi and English corpora should aid improve task performance on Hinglish. We show that meta-learning framework can effectively utilize the the labelled resources of the downstream tasks in the constituent languages. The proposed approach improves the performance on downstream tasks on code-switched language. We experiment with code-switching benchmark GLUECoS and report significant improvements.

## introduction in parts of the world where people speak more than one language in day-to-day affairs, the mixing of the languages often occurs naturally. this phenomenon, termed as code-switching or codemixing, is observed for various language pairs, e.g. hindi-english, spanish-english. natural language processing tasks such as sentiment analysis, named entity recognition, question-answering are interesting research challenges for hinglish @xcite . with the rise of multilingual language models (lm), pretrain-and-finetune approach has been widely used for various downstream nlp tasks; it is observed that when lms are pretrained with large corpora, they can be easily transferred to downstream tasks with limited fine-tuning data. most of the publicly available multilingual lms are pretrained using combination of various monolingual corpora (such as @xcite , but not using code-switched data (e.g. hinglish). this leads to sub-optimal lms for the downstream tasks in code-switched language. moreover, the downstream tasks on codeswitched language (e.g. hinglish) usually suffers from low-resource problem. usually, the same task on the constituent languages (e.g. hindi and english) is not low-resource -which we see as an opportunity. we explore to utilize the constituent language resources to fine-tune the lm for the downstream task in code-switched language. based on these intuitions, we propose a meta-learning based approach to address the code-switching challenge, as illustrated in figure . 1. the meta-learner helps leverage hindi and english resources to improve the performance on code-switched data. during the fine-tuning of a pre-trained language model, its weights adjust for the downstream task. it has been shown that if the base pre-trained model is multilingual, the fine-tuning with one language helps achieve impressive performance on another language in a zero-shot setting too @xcite . it shows that fine-tuning teaches a multilingual model how to perform the downstream task in somewhat language-agnostic fashion. however, such transfers are better when source and target languages are closely related; obviously, the constituent languages are the closest source languages to the target code-switched language. however, the models learned/fine-tuned with constituent languages may become specific to the source languages. meta-learning approaches are good at solving this problem by learning model parameters suitable for multiple downstream tasks. therefore, we aim to use meta-learning with constituent language resources, to obtain a model state that serves as better initialization point for fine-tuning with limited code-switched samples. overall, we focus on the following inquiry: for various downstream tasks, can we effectively use hindi and english resources to improve the performance on codeswitched data? and explore utility of meta-learning for it. ## related work varieties of downstream tasks in code-switched languages have given rise to standard benchmarks and need for synthesizing cs data. token-level language identification (lid) is one of the earliest explored primary tasks in code-switched nlp for dialectal arabic-modern standard arab @xcite , spanish-english, nepalese-english, mandarin-english @xcite , and english-indic languages @xcite . part-of-speech (pos) tagging for hindi-english code-switching has been explored either via lid route @xcite that identified language of text chunks and applied pos tagger of the respective language; or by utilizing language-specific word representations @xcite . named entity recognition (ner) on codeswitched language pairs of spanish-english, nepalese-english, mandarin-english, modern standard arabic-egyptian @xcite @xcite have been explored. use of meta-embeddings @xcite has also been explored for the task of code-switched ner. gluecos and lince benchmarks are created by @xcite al. (2020) and khanuja et al. (2020) that include lid, pos-tagging, ner, questionanswering (qa), sentiment analysis (sa), natural language inference (nli), and machine translation (mt) for evaluating various models on several language pairs across several tasks. code-switched text generation is explored in recent literature to address the data scarcity for code-switched scenarios. @xcite proposed to create synthetic code-switched texts from parallel corpus by replacing named entities, noun phrases, adjectives in the hindi sentence with corresponding english translation obtained from align-ments. they train a deep learning model on this synthetic data to generate more code-switched texts. @xcite have released a toolkit for generating synthetic code-switched texts. they implement two linguistic theories; the equivalence constraint theory and the matrix language theory to constraint the synthetic code-switched sentences generated. @xcite use existing unsupervised neural machine translation techniques to generate code-switched sentences. however, in our (limited) initial experimentation with generation approaches, we observe that generated cs text suffers from unnatural switching and grammar violations. meta-learning frameworks are task-agnostic representation learning that is widely used for fast-adaptation to downstream tasks of interest. maml @xcite and reptile @xcite are arguably two of the most popular meta-learning approaches. maml's computation and space complexity makes it somewhat impractical -a problem that reptile addresses with heuristics without compromising on performance. meta learning is highly effective and shown to be beneficial for a variety of nlp tasks such as low resource machine translation, persona consistent dialogues, low resource sales prediction, speech recognition etc. we believe that its applications can be extended to code-switched nlp too. ## our approach in this work we incorporate a technique to improve the downstream task performance in hinglish: by utilize downstream task data of hindi and english in meta-learning framework. overall, we aim to utilize high resource unsupervised text corpora and task corpora of constituent language (english and/or hindi) to initialize the model that can be transferred to cs task effectively. the idea behind meta learning algorithms such as reptile is to learn a better initialization for the target task using a set of auxiliary source tasks. in cs setting, this amounts to using constituent languages to learn a better initialization parameters and then fine-tuning on small amount of available code-switched data starting from the initialized parameters. our specific methodology is inspired from @xcite . to represent it more formally, lets say the code-switched language has english (en) and hindi (hi) as the constituent lan-algorithm 1 our meta-learning approach for cs input: task data t en and t hi for english and hindi, and t hi-en for hinglish output: the model for hinglish downstream task 1: initialize model weights θ 2: while not converged do 3: perform reptile updates 4: draw total of m random batches from t en and t hi 5: θout ← θ 6: for i th batch, 0 < @xmath0: train and update model for k steps 8: fine-tune the model on target t hi-en 17: end while guages, lets represent tasks in theses languages as t en and t hi and the task in target code-mixed language as t hi-en . the proposed meta-learning approach is described in algorithm 1. first, the model weights θ are initialized. the model consists of a randomly initialized classifier head on top of a pre-trained language model (lm). the outer loop of the meta-learner executes for n e number of epochs over t hi and t en task datasets; this is our convergence criteria. within each meta-step, we sample m batches from the collective pool of batches of english and hindi. with a batch, the model is trained for k steps, and the updated model weights are persisted as θ i batch , and the model is reset to the θ out state prior to the meta-step. θ i batch represents the the model state had it been trained with the i th batch alone. since each of these states are too specific, their average θ represents somewhat generalized state of model for the downstream task. we obtain the measure of the model deviation as θ var that represents the direction of generalized model state θ relative to the current state θ out . finally, the model weights are updated in that direction, with step size of β. at convergence, the model state serves as a good initialization to fine-tune the same network on the higlish task data t hi-en . ## experiments and results we utilize gluecos @xcite benchmark as our test-bed for evaluation on hindi-english code-switching. the official benchmark involves pos tagging (pos) (universal dependency), named entity recognition (ner), sen-timent analysis (sa), question answering (qa) and natural language inference (nli). we submit test set predictions to the official benchmark portal to obtain the f1-score metrics. as part of meta-training and its ablations, we utilize the publicly available datasets of each downstream task for hindi and english languages. the details of the monolingual task datasets used for meta-training are described in ## conclusion we studied the usefulness of corpus in the constituent languages, hindi and english, for improving the performance of the nlp tasks in codeswitched language, hinglish. we propose a reptile based meta-learning framework which learns better initialization using task-specific labelled datasets in the constituent languages and improves performance for code-switched language. our results indicate meta-trained models outperform other strong baselines for all gluecos tasks.