PRO - CS : An Instance-Based Prompt Composition Technique for Code-Switched Tasks (2022)

Abstract: Code-switched (CS) data is ubiquitous in today’s globalized world, but the dearth of annotated datasets in code-switching poses a significant challenge for learning diverse tasks across different language pairs. Parameter-efficient prompt-tuning approaches conditioned on frozen language models have shown promise for transfer learning in limited-resource setups. In this paper, we propose a novel instance-based prompt composition technique, PRO-CS, for CS tasks that combine language and task knowledge. We compare our approach with prompt-tuning and fine-tuning for code-switched tasks on 10 datasets across 4 language pairs. Our model outperforms the prompt-tuning approach by significant margins across all datasets and outperforms or remains at par with fine-tuning by using just 0.18% of total parameters. We also achieve competitive results when compared with the fine-tuned model in the low-resource cross-lingual and cross-task setting, indicating the effectiveness of our approach to incorporate new code-switched tasks.

## introduction code-switching (cs) is the phenomenon of shifting from one language to another in the same context, usually used in an informal way of speaking or writing @xcite . within the scope of this paper, we have focused on the most commonly used form of code-switching which happens at the intra-sentential level. with the advent of social media platforms and the global rise in multilingual speakers, people generally converse in more than one language and often switch from one language to another @xcite . code-switching can go beyond the mere insertion of borrowed words, fillers, and phrases, and include morphological and grammatical mixing. it is * equal contribution frequently used to convey stronger emotions or to express one's ideas precisely @xcite . code-switching has gained attention in the nlp community due to a plethora of unstructured text data available that require an understanding of intermixing of two languages in the same sentence. understanding the meaning of such text relies on subtasks such as part-of-speech (pos) tagging, sentiment analysis (sa), and hate speech (hs), and the characteristic of cs languages. recent work on cs tasks focuses on the identification of sentiments in online texts @xcite , filtering of hate posts written in cs language @xcite , part-of-speech tagging in cs text @xcite . @xcite ; @xcite have created benchmarks in a variety of code-switched language pairs and tasks. one of the main challenges in the code-switched setting is the lack of high-quality labeled datasets across different language pairs and tasks. recent works @xcite @xcite do not discuss much about the transferability of models to unseen combinations, which is essential for this lowresource domain. parameter efficient approaches such as adapter-based models @xcite , prompt-tuning @xcite show promise for cross-lingual and cross-task transfer in multilingual few-shot and zero-shot settings @xcite . we take a step forward towards solving codeswitching tasks through parameter efficient prompttuning with multilingual pre-trained language models @xcite . specifically, we wish to explore the effectiveness of the prompt-tuning approach for code switching by training prompts with multilingual lms. since code-switching comprises of information from two different languages along with the task information, we also investigate whether prompt-composition of task and language specific prompts can be applied for downstream cs tasks and how efficiently this method transfers to other code-switched language pairs or tasks. essentially in this paper, we want to address three research questions. although current parameter-efficient techniques show promise for nlp, we find a big gap between the prompt-tuning and fine-tuning paradigms for code-switched nlp tasks. we feel this disparity is due to language and task complexities, as it involves switching between two languages. in this paper, we propose a novel pro-cs promptcomposition technique that leverages language and task specific knowledge captured through language and task-based prompts. task knowledge in prompts is learned through monolingual task corpora and language prompts are learned through masked token prediction via discriminative training on unlabeled language data. all these prompts (referred as source prompts) serve as a pool of stored knowledge that can be reused for different language pairs and tasks. unlike prompt-tuning, which learns prompts from scratch, we use an attention module to learn instance-based prompts by weighing the contributions of frozen source prompts and trainable prompts (referred as target prompts). we also use prompts trained on language identification, a characteristic task for code-switched language modeling to capture mixing between the two languages to initialize target language prompt. the frozen multilingual lm is then conditioned on the resulting prompt prepended with input for downstream tasks (see section 3 for more details). we hypothesize that both monolingual tasks and language understanding expose different facets of knowledge required for the code-switch setting. our results support this claim, showing that our proposed approach outperforms prompt-tuning across all datasets. we test the generalizability of our approach by transferring across tasks and languages in low-resource setting (see figure 1 ). for crosslingual transfer, we substitute task prompt of a language (bn-pos) with related language (hi-pos). similarly, for cross-task transfer, we replace task prompts (x-hs) with related tasks (x-sa). in summary, the main contribution of our work are as follows (i) to the best of our knowledge, this is the first work investigating prompt-composition for code switching tasks. (ii) we leverage language and task-related prompts for pro-cs and achieve consistent improvement over prompt-tuning. further, we outperform or remain at par with the fine-tuning performance in most datasets with just 0.18% of the total parameters. (iii) we also show the effectiveness of pro-cs for crosslingual/cross-task transfer settings. 10244 2 background & related work fine-tuning pre-trained language models @xcite is arguably the most prevalent paradigm in nlp research for transfer learning @xcite . in multilingual setting, fine-tuning language models trained on data from multiple languages has shown to be effective for downstream tasks and cross-lingual transfer @xcite . recent work in code-switched nlp @xcite fine-tune large multilingual models for different downstream tasks. as the size of the pre-trained multilingual models @xcite ) such as multilingual bert (178m), xlm-r (470m), increases, it becomes computationally expensive to fine-tune them for ever evolving new tasks and new code-switching languages. in low-resource scenarios (such as code switching), fine-tuning these large pre-trained models @xcite is often susceptible to overfitting. @xcite attempts to mitigate this by generating synthetic cs data, which in most cases is not able to generate diverse examples to mimic real-world scenarios. @xcite shows that training multilingual lms on real code-switched data is more helpful than training on synthetic data. code-switching datasets have very few annotated examples that present an ideal low-resource setting for parameter-efficient transfer learning. recent study @xcite shows promise in exploring parameter-efficient training (pet) for code switching using meta-embeddings that achieve similar performance to fine-tuned multilingual models with fewer parameters. @xcite presents a unified view of different parameter-efficient training (pet) approaches like adapter @xcite , prompt-tuning @xcite , bitfit @xcite , lora @xcite , and sparse fine-tuning @xcite that have shown promise as an alternative to fine-tuning. these techniques are closely connected and share design elements that are essential for their effectiveness. we choose prompt-tuning in this study as it is being thoroughly explored in the nlp community and can be used as plug-ins for different tasks and language pairs with a frozen lm. recently, many studies have focused on pet approaches for multilingual settings. hyper-x (ustun et al., 2022) uses a hypernetwork to generate weights for adapter modules conditioned on both tasks and language embeddings. mad-g @xcite ) learns a single model to generate a language adapter for an arbitrary target language using the generation of contextual parameters. @xcite show that discrete and soft prompting outperform fine-tuning in cross-lingual transfer. polyglot prompt @xcite proposes a two-tower encoder-based approach for languageindependent prompt generation. although pet techniques have been shown to be effective in monolingual and multilingual settings, this has not been explored in the code-switching domain. compositions of parameter-efficient methods for tasks and languages for better downstream transfer have been explored in the multilingual domain. mad-x learns modular task and language-based adapters for cross-lingual transfer @xcite . @xcite composes language and task information by selecting a subset of parameters that change the most during fine-tuning. it uses this composition to show zero-shot crosslingual transfer. @xcite explores promptcomposition where the target prompt is learned by attending over multiple source task prompts. motivated by this idea of prompt-composition and modular learning of task and language-based parameters for multilingual tasks, we explore the direction of prompt-composition for code-switching. ## method prompts can learn different facets of knowledge from both tasks and languages. in this paper, we hypothesize that monolingual task-based and language prompts can be used to compose instancelevel code-switched task prompts. such compositions are also compatible with heterogeneous supervision, that is, for different task and language combinations, and can be useful for initializing models in a low-resource setting. ## datasets we evaluate the performance of our model on a diverse set of code-switching datasets that span a variety of tasks and language pairs. ## experiment we describe the baselines used in this paper and the experimental setup of our pro-cs approach in the following subsections. we also discuss the transfer learning settings on which we evaluate our model. ## result we achieve competitive results compared to finetuning setting and even outperform it for en-es sa, en-es pos and en-hi sa with just 0.18% of mbert model parameters (178m). there is a performance gap between the fine-tuning and pro-cs approach on the ner dataset. we believe it is because the ner corpus is highly imbalanced across label classes. for example en-hi ner corpus consists of 7 labels where 4 labels combined have less than 5% of total examples. due to this skewed distribution of labels and limited model capacity for pro-cs or pt, there is a drop in performance compared to ft. we expect this drop to reduce as we increase model capacity through number of prompt tokens. also, for en-es ner, we observed a lot less code switching in the dataset, which also contributed to the performance drop. ## analysis in this section, we perform a detailed analysis of our model's performance with increasing number of examples for cross-lingual and cross-task transfer. further, we conduct an ablation study to show the effectiveness of each component in our model. ## limitation datasets in code-switching are usually very unbalanced making the task harder. our improvements in low-resource setting show that with a limited number of good examples, prompt-composition methods can achieve good results. our models suffer when there is a skewed distribution of data. we also restrict our models to only classification tasks as we find that they are more ubiquitous in the cs domain. with the recent rise in translation between english and cs data, it will be interesting to apply our techniques for generation tasks like translation, open domain question answering, dialogue, etc. we have also made an assumption of existing monolingual corpora for training languagebased prompts, which may not be true for various low-resource languages where code switching is prominent. ## conclusion in this paper, we perform a detailed analysis of prompt-tuning techniques to show that they are effective in code-switching settings as well. however we find a significant gap between finetuning and widely used prompt-tuning. we address this gap by proposing a novel technique of promptcomposition pro-cs for code-switching tasks. our approach outperforms prompt-tuning technique across all 6 datasets in full-data setting with an average improvement of 3 and 3.5 points on sequence tagging and classification tasks respectively. this shows that composing prompts from source task and language prompt is more effective than training target-task only prompts. it also achieves competitive results to fine-tuning, even in low-resource cross-lingual and cross-task setting for both classification and sequence tagging tasks. for future work, we want to investigate these compositions for encoder-decoder t5 models for generation-based tasks. it would also be interesting to see if multi-task training helps in the downstream code-switch prompt-composition.