Crosslingual Retrieval Augmented In-context Learning for B angla (2023)

Abstract: The promise of Large Language Models (LLMs) in Natural Language Processing has often been overshadowed by their limited performance in low-resource languages such as Bangla. To address this, our paper presents a pioneering approach that utilizes cross-lingual retrieval augmented in-context learning. By strategically sourcing semantically similar prompts from high-resource language, we enable multilingual pretrained language models (MPLMs), especially the generative model BLOOMZ, to successfully boost performance on Bangla tasks. Our extensive evaluation highlights that the cross-lingual retrieval augmented prompts bring steady improvements to MPLMs over the zero-shot performance.

## introduction in recent years, the field of natural language processing (nlp) has witnessed transformative advancements, especially with the advent of deep transformer techniques @xcite . the introduction of large language models (llms), such as gpt-3 @xcite and @xcite , has further revolutionized the landscape. these models showcase unparalleled prowess in tasks like text classification and generation, unified under the umbrella of in-context learning, and cater to a plethora of applications across diverse languages @xcite . while comprehensive benchmarks like xtreme @xcite and buffet @xcite underscore their capabilities, languages such as english remain the primary beneficiaries. in stark contrast, several low-resource languages, bangla being a prime example, grapple with challenges, notably the scarcity of pretraining corpora (artetxe † corresponding author. and schwenk, @xcite . despite having a significant number of native speakers, bangla remains underrepresented in the nlp arena due to linguistic intricacies, limited labeled datasets, and prevalent issues like data duplication @xcite @xcite . although there have been commendable strides using conventional machine learning techniques in bangla nlp tasks, the untapped potential of the latest llms is evident @xcite . in the evolving landscape of in-context learning with llms, the concept of retrieval augmentation, which emphasizes sourcing semantically rich prompts, has gained traction @xcite . however, when it comes to multilingual in-context learning, previous works like mega @xcite often limit their scope to task instructions and lack deeper semantic insights due to their approach of random prompt selection. in contrast, strategies like parc @xcite pave the way for a more comprehensive methodology, fetching semantically aligned prompts from highresource languages. our work draws inspiration from these methodologies but introduces novel perspectives. while mega offers task-level instructions, we infuse semantic understanding into our approach. similar to parc, our approach is cross-lingual, ensuring a broader application spectrum. diverging from parc's focus on masked language models like mbert and xlmr, as shown in figure 1 , we venture into uncharted territories by employing larger, decoder-only multilingual pretrained language models (mplms) -bloom and bloomz -to tackle bangla nlp tasks in a generative style @xcite . in this paper, we explore the application of crosslingual retrieval augmented in-context learning to bangla text classification and summarization tasks. our main contributions encompass: ## related work bangla natural language processing bangla is a morphologically rich language with various dialects that belongs to the indo-aryan branch of the indo-european language family. with roughly 270 million speakers concentrating in bangladesh and some regions of india, bangla is ranked as the 7th most widely spoken language in the world foot_0 . however, bangla is still considered as a lowresource language in the nlp research due to the scarcity of digital text resources and annotated corpora. research on bangla nlp has covered a variety of common nlp subfields since 1990s, such as pos tagging @xcite , stemming and lemmatization @xcite , named entity recognition @xcite , sentiment analysis @xcite , news categorization @xcite , etc. however, the research in different areas of bangla nlp still remains sparse. in the era of deep learning, further progress has been made in bangla nlp, particularly in terms of the development datasets (rahman and kumar dey, @xcite and models @xcite . pretrained language models have achieved decent performance in a large variety of nlp downstream tasks through the finetuning. under this background, @xcite pretrained the banglabert model, a bert-based language understanding model pretrained on bangla language corpora. with the advent of the large language models (llms), zeroand few-shot prompting methods have gradually gained prominence. @xcite compared the zero-and few-shot prompting performance of llms with the finetuned models for the bangla sentiment analysis task. our work explores the application of the retrieval-augmented prompting method in bangla violence detection and sentiment analysis tasks. @xcite demonstrated that llms like gpt-3 can acquire task-solving abilities by incorporating input-output pairs as context. the in-context learning approach involves concatenating input with randomly selected examples from the training dataset, which is also called the prompting method. recent research @xcite has expanded on this idea by enhancing prompts for pretrained models through the inclusion of semantically similar examples. the effectiveness of prompting methods for english models extends to multilingual models in cross-lingual transfer learning as well. @xcite and @xcite al. (2022) information but also additional task information. however, previous studies mainly concentrated on the multilingual encoder or encode-decoder models, while our work extend the parc pipeline to the decoder-only multilingual llms. ## methodology our research extends the work of @xcite by focusing on improving multilingual pre-trained language models (mplms) for low-resource lan-guages in a zero-shot setting, specifically using retrieved content from high-resource languages such as english. the backbone of our research approach is a two-stage pipeline consisting of a cross-lingual retriever and a prompt engineering process as shown in figure 2 . this pipeline aims to build on the strengths of mplms while mitigating their limitations, especially when dealing with low-resource languages. the first stage of the pipeline uses a cross-lingual retriever that maps the input bangla text q to a vector q embed in a shared embedding space and uses it as a query. using semantic similarities with q embed , the retriever returns the most similar k examples from high-resource languages either with or without their labels: the second stage of the pipeline is the prompt engineering. the input bangla text and the retrieved pattern are subjected to this process. a prefix prompt template p is used to reformulate the input to facilitate the model's prediction y: @xmath0, r)) depending on the architecture of the chosen mplm, for decoder-only models, the answer is generated by the model directly. for encoder models, the answer is obtained by first mapping each label to its predefined word using the verbalizer and then deducing the label word using mask token prediction. by integrating cross-lingual content retrieval with prompt-guided prediction, we aim to improve the ability of mplms to handle low-resource languages. this synergy not only extracts rich linguistic insights from high-resource languages, but also uses them to improve performance on lowresource language tasks. ## experiments in this study, we focused on the tasks of classification and summarization. we refer to our research approach, which uses k retrieved samples for crosslingual augmented in-context learning methods, as the main method in the following sections. ## conclusion in this paper, we have introduced a novel approach to address the challenges of applying large language models to low-resource languages, with a focus on bangla. our methodology employs cross-lingual retrieval-augmented in-context learning, thereby enriching the capabilities of mplms, specifically bloom and bloomz. we have extensively tested our approach on two classification tasks and one summarization task. our experimental results demonstrate the effectiveness of our approach in achieve superior f1 scores for classification tasks. upon further analysis, the cross-lingual retrieval mechanism contributes significantly to the model's performance. this work lays the foundation for further studies on the application of cross-lingual retrieval and in-context learning methods in low-resource languages. future work could extend this approach to even more underrepresented languages and potentially adapt it to more complex nlp tasks such as question answering or machine translation.