Leveraging Large Language Models for Code-Mixed Data Augmentation in Sentiment Analysis (2024)

Abstract: Code-mixing (CM), where speakers blend languages within a single expression, is prevalent in multilingual societies but poses challenges for natural language processing due to its complexity and limited data. We propose using a large language model to generate synthetic CM data, which is then used to enhance the performance of task-specific models for CM sentiment analysis. Our results show that in Spanish-English, synthetic data improved the F1 score by 9.32%, outperforming previous augmentation techniques. However, in Malayalam-English, synthetic data only helped when the baseline was low; with strong natural data, additional synthetic data offered little benefit. Human evaluation confirmed that this approach is a simple, cost-effective way to generate natural-sounding CM sentences, particularly beneficial for low baselines. Our findings suggest that few-shot prompting of large language models is a promising method for CM data augmentation and has significant impact on improving sentiment analysis, an important element in the development of social influence systems.

## introduction code-mixing (cm), or code-switching, is the practice of switching between languages within a conversation or utterance. this practice is integral to multilingual societies, particularly in mexico and urban india @xcite , and is also significant in computer-mediated communication and social media, where multilingual users are predominant @xcite . despite its ubiquity, cm is mostly spoken and found in personal messages, making training data scarce and leading to poorer natural language processing (nlp) model performance compared to monolingual text @xcite . social influence (si) refers to the changes in thoughts, feelings, attitudes, or behaviors resulting from interactions with others. in multilingual societies, cm reflects an important aspect of these interactions, reflecting social dynamics and identity. sentiment analysis (sa) is crucial for understanding these dynamics, as it captures the emotional nuances embedded in multilingual interactions. furthermore, sa has become a primary cm task due to its need for complex semantic understanding and its implications for social media @xcite , where cm is commonly present @xcite . by accurately analyzing sentiment in code-mixed text, si systems enhance their ability to interpret user intent and emotional states, enabling more meaningful interactions addressing the more diverse environments in which si occurs. since multilingual speakers bridge information on social media @xcite , machines must also accurately analyze cm text to capture public opinion and disseminate news. however, current approaches fall short in handling code-mixed settings @xcite due to data scarcity. beyond the cm domain, few-shot learning has shown promise in overcoming data scarcity, as large language models (llms) trained on diverse tasks generalize to new ones with minimal training @xcite @xcite . llms are used for data augmentation @xcite @xcite , training data generation @xcite , and knowledge distillation @xcite , particularly in low-resource settings @xcite . however, this approach remains underexplored in the cm domain, which presents unique challenges @xcite . in this work, we bring llm-powered data augmentation to the task of code-mixed sentiment analysis. we use few-shot prompting to generate labeled cm sa data in spanish-english and lowresource malayalam-english. following @xcite ; @xcite ; @xcite , we quantify the performance gains by fine-tuning multilingual pre-trained language models (plms) on the llm-generated data. we investigate if these synthetic data samples can reflect natural code-mixing patterns and nuances compared to other data augmentation techniques and verify the synthetic data quality through human evaluation. figure 1 displays our overall system workflow with examples of natural and synthetic data. we summarize our contributions as follows: ## methods in this section, we introduce our data, the synthetic generation process, and our fine-tuning methods. ## results this section evaluates overall model performance and then quantifies relative percent improvements contributed by data augmentation. ## analysis this section details results from human evaluation, subsequent empirical data analysis, and discussion about the trade-offs of generating synthetic data. ## conclusion and future work to address cm data scarcity, we propose using fewshot prompting with llms to generate synthetic, labeled cm data for sa. we tested this approach by training mbert and xlm-t on natural, synthetic, and combined datasets for spanish-english and malayalam-english. in spanish-english, our method improved sentiment classification by 9.32% for the 3k training setup and achieved third place on the lince benchmark for the 12k training setup. human evaluations confirmed that our synthetic data closely mimic natural data and are indistinguishable from human-labeled examples. for malayalam-english, our baseline system exceeded the highest published benchmark with an f1 score of 0.847, though further improvements with additional data were limited. our findings indicate that llm-generated synthetic data are most effective for enhancing models with low baseline performance, particularly when the languages are evenly represented as well as for resource-constrained scenarios. overall, llm-powered data augmentation offers a cost-effective alternative to human annotation, producing high-quality, natural-sounding sentences with minimal label ambiguity. to improve performance in malayalam-english, we intend to apply our observations of synthetic data to refine our llm prompt and regenerate data. in addition, we aim to extend our research to encompass a broader range of llms and dialects, including those without english as a base and those primarily written in non-latin scripts. ultimately, our findings offer a promising avenue for cm data augmentation, and we encourage further exploration with llms in cm, an area which presents technical challenge and valuable social impact.