Low-Resource Cross-Lingual Summarization through Few-Shot Learning with Large Language Models (2024)

Abstract: Cross-lingual summarization (XLS) aims to generate a summary in a target language different from the source language document. While large language models (LLMs) have shown promising zero-shot XLS performance, their few-shot capabilities on this task remain unexplored, especially for low-resource languages with limited parallel data. In this paper, we investigate the few-shot XLS performance of various models, including Mistral-7B-Instruct-v0.2, GPT-3.5, and GPT-4.Our experiments demonstrate that few-shot learning significantly improves the XLS performance of LLMs, particularly GPT-3.5 and GPT-4, in low-resource settings. However, the open-source model Mistral-7B-Instruct-v0.2 struggles to adapt effectively to the XLS task with limited examples. Our findings highlight the potential of few-shot learning for improving XLS performance and the need for further research in designing LLM architectures and pre-training objectives tailored for this task. We provide a future work direction to explore more effective few-shot learning strategies and to investigate the transfer learning capabilities of LLMs for cross-lingual summarization.

## introduction cross-lingual summarization (xls) is a task that involves generating a summary in a target language different from the source document's language. it is a complex natural language processing task that requires performing both summarization and machine translation simultaneously. this task is much more challenging than singlelanguage summarization, as it involves overcoming the differences between languages while effectively extracting and compressing key information. generally, there are two types of pipelines for xls systems @xcite : summarize-then-translate and † corresponding author. translate-then-summarize. the former summarizes the source text first and then translates it, while the latter translates the source text first and then summarizes it. however, these pipelines have the disadvantage that errors occurring at each stage can propagate and accumulate, potentially degrading the final performance. to resolve the issues of these pipelines, there has been a lot of research on the end-to-end approach @xcite , known as the direct method, which generates the target language summary directly from the source document in a single step. the direct method can mitigate the error propagation problem compared to traditional pipelines and enable more efficient learning. in parallel with the development of end-to-end approaches, large language models (llms) such as @xcite have demonstrated strong performance in various natural language processing tasks. it is known that these models can significantly improve performance through few-shot learning with only a small number of examples. (brown et al., 2020) however, directly applying these models to the xls task is still challenging. particularly for low-resource languages, where it is difficult to build large-scale parallel corpora, the lack of data makes it difficult to fully utilize the performance of pre-trained language models, acting as a major obstacle in xls research. @xcite hence, this study aims to explore the xls performance of various llms using a few-shot learning approach through in-context learning, focusing on the direct method. our findings demonstrate that few-shot learning enables llms, particularly gpt-3.5 and gpt-4, to achieve competitive performance in cross-lingual summarization tasks for low-resource language settings. the results also highlight open-source models' challenges in adapting to the xls task with limited parallel data. these findings emphasize the potential of few-shot learning in enhancing crosslingual summarization capabilities and the need for further research in developing effective few-shot strategies and architectures for low-resource languages. ## related work cross-lingual summarization (xls) has undergone significant evolution, shifting from early pipeline approaches like summarize-then-translate @xcite and translate-then-summarize @xcite to more sophisticated methods utilizing multilingual pre-trained models. these pipelines were initially dominant due to their simplicity but were plagued by error propagation and the limitations inherent to sequential processing tasks. @xcite the advent of multilingual pre-trained models such as mbart @xcite and mt5 @xcite marked a transformative shift towards endto-end approaches, directly generating summaries in the target language and substantially mitigating error propagation issues. in recent years, the emergence of large language models (llms) has revolutionized the field of natural language processing, including xls. especially, widely used llms like gpt-3.5 and gpt-4 have demonstrated remarkable zero-shot learning capabilities across various tasks. @xcite @xcite however, the exploration of llms in the context of xls is still in its early stages, with limited research on their zero-shot learning capabilities and even fewer studies focusing on their few-shot learning potential. @xcite recent studies have shown promising results in using llms for various nlp tasks. @xcite @xcite however, the specific exploration of these models in xls scenarios, particularly in the few-shot setting, remains largely unexplored. while some studies have investigated the zero-shot xls performance of llms @xcite , there is a notable lack of research on the few-shot learning capabilities of models such as gpt-3.5, gpt-4, and multilingual llms in the xls domain. moreover, the disparity in performance between proprietary models like gpt-4 and open-source alternatives in zero-shot settings underscores the necessity for further investigation into the few-shot capabilities of llms. this is particularly critical to ensure that advancements in xls are equitable and accessible across various linguistic and resource settings. in this paper, we aim to bridge this gap by exploring the few-shot learning capabilities of llms in the context of xls. we focus on the direct method, leveraging the nuanced capabilities of llms like gpt-3.5, gpt-4, and open-source models such as mistral-7b-instruct-v0.2. the mistral-7b-instruct-v0.2 is a 7.3b parameter model that outperforms llama-2 13b @xcite across all benchmarks and even surpasses llama-1 34b @xcite on many tasks and particularly noted for its ability to process up to 32k tokens, significantly enhancing its capability for few-shot learning by providing richer context management. @xcite this open-source model has demonstrated strong performance in various natural language processing tasks and offers robust multilingual support. our aim is to provide comprehensive insights into the practical applications and limitations of these models in low-resource languages, setting the stage for future advancements in the field. ## methods the main objective of our research is to compare and analyze the performance of pre-trained mt5 and few-shot prompt-based gpt-3.5 and gpt-4. then, we aim to experimentally confirm the impact of their few-shot learning on low-resource language xls tasks. additionally, we conduct comprehensive comparison with one of the open-source multilingual llms, such as the mistral-7b-instruct-v0.2 (la plateforme; @xcite , to provide a broader perspective on the performance of different llms in the xls task to provide a broader perspective on the performance of different llms in the xls task and assess the effectiveness of fewshot learning in mitigating the challenges posed by low-resource settings. ## conclusion this study empirically analyzed the few-shot performance of llms in cross-lingual summarization tasks, focusing on low-resource languages using a direct prompting approach. we observed that llms demonstrated competitive performance improvements through few-shot learning compared to zero-shot setups particularly in the many-to-one xls. but, we also demonstrated that there was no significant gain to llms in the one-to-many xls. these findings underscore the need for further research in developing more effective few-shot learning strategies and architectures tailored to lowresource languages.