Zero-Shot Cross-Lingual Summarization via Large Language Models (2023)

Abstract: Given a document in a source language, cross-lingual summarization (CLS) aims to generate a summary in a different target language. Recently, the emergence of Large Language Models (LLMs), such as GPT-3.5, ChatGPT and GPT-4, has attracted wide attention from the computational linguistics community. However, it is not yet known the performance of LLMs on CLS. In this report, we empirically use various prompts to guide LLMs to perform zero-shot CLS from different paradigms (i.e., end-to-end and pipeline), and provide a preliminary evaluation on the generated summaries. We find that ChatGPT and GPT-4 originally prefer to produce lengthy summaries with detailed information. These two LLMs can further balance informativeness and conciseness with the help of an interactive prompt, significantly improving their CLS performance. Experimental results on three widely-used CLS datasets show that GPT-4 achieves state-of-the-art zero-shot CLS performance, and performs competitively compared with the fine-tuned mBART-50. Moreover, we also find some multi-lingual and bilingual LLMs (i.e., BLOOMZ, ChatGLM-6B, Vicuna-13B and ChatYuan) have limited zero-shot CLS ability. Due to the composite nature of CLS, which requires models to perform summarization and translation simultaneously, accomplishing this task in a zero-shot manner is even a challenge for LLMs. Therefore, we sincerely hope and recommend future LLM research could use CLS as a testbed.

## introduction cross-lingual summarization (cls) aims to provide a target-language (e.g., chinese) summary for a lengthy document in a different source language (e.g., english) @xcite ; figure 1 : an example of zero-shot cls via chatgpt. @xcite @xcite @xcite @xcite @xcite @xcite @xcite . this task could help people quickly capture their interests from foreign documents. in recent years, a number of powerful multilingual pre-trained generative models have been proposed one after another, such as mbart @xcite , mbart-50 @xcite , mt5 @xcite and bloom @xcite . the parameters in these models have gradually increased from million levels (e.g., 580m in mt5-base and 610m in mbart-large) to billion levels (e.g., 3.7b in mt5-xl, 13b in mt5-xxl and 176b in bloom), facilitating various research topics (e.g., machine translation and cls) in the multi-lingual world. besides, large language mod-els (llms) have been key to strong performance when transferring to new tasks by simply conditioning on a few input-label pairs (in-context learning) @xcite or short sentences describing crucial reasoning steps (chainof-thoughts) @xcite . more recently, chatgpt and gpt-4 (openai, 2023) have attracted great attention from both the research communities and industries. similar to instructgpt @xcite , chatgpt is created by fine-tuning a gpt-3.5 series model via reinforcement learning from human feedback (rlhf) @xcite . gpt-4, as a multimodal llm that can accept image and text inputs and produce text outputs, exhibits human-level performance on various benchmark datasets @xcite . with the emergence of chatgpt and gpt-4, there is growing interest in leveraging llms for various nlp tasks @xcite @xcite @xcite @xcite @xcite @xcite . nevertheless, the exploration of llms on cls is still lacking. in this report, we present a preliminary evaluation of llms' zero-shot cls performance, including gpt-3.5, chatgpt, gpt-4, bloomz, chatglm-6b, vicuna-13b and chatyuan. in detail, we design various prompts to guide llms to perform cls in an end-to-end manner with or without chain-of-thoughts (cot). figure 1 gives an example of prompting chatgpt to perform zero-shot cls. to further exploit the interaction capability of conversational llms (e.g., chatgpt and gpt-4), we leverage an interactive prompt to let them produce more concise summaries. moreover, to provide a deeper analysis of llms' zero-shot cls performance, we compare them with fine-tuned mbart-50 @xcite which has shown its superiority in many previous cls works @xcite @xcite . experimental results on three cls datasets, covering three domains (news, howto guide and dialogue) and two cross-lingual directions (en⇒zh and en⇒de) 1 , show that gpt-4 achieves the best zero-shot performance but is still worse than the fine-tuned mbart-50 model in terms of rouge scores and bertscore. we 1 since a cls dataset might contain multiple source and target languages, we use "x⇒y" to indicate the source language and target language are x and y, respectively. en: english; zh: chinese; de: german. also conduct case studies to show that chatgpt and gpt-4 could absorb the core idea of the given source-language documents and generate fluent and concise target-language summaries. in addition, we find that the current open-source llms (i.e., bloomz, chatglm-6b, vicuna-13b and chatyuan) achieve limited zero-shot cls performance, which is significantly worse than that of gpt-4. this leads us to conclude that the composite end-to-end cls prompts are difficult for them to follow, and there is still a challenge for llms to perform zero-shot cls in an end-to-end manner which requires simultaneously carrying out translation and summarization. based on the finding, we suggest that future multi-lingual or bilingual llm research uses cls as a testbed to evaluate llms' capabilities to follow composite instructions as well as combine their different abilities. our main contributions are concluded as follows: • to the best of our knowledge, we are the first to explore the zero-shot cls performance of llms. to achieve that, we design various prompts to guide llms to perform cls in an end-to-end manner with or without chain-of-thoughts. ## conclusion and future work in this technical report, we evaluate the zero-shot performance of mainstream bilingual and multilingual llms on cross-lingual summarization. we find that davinci-003, chatgpt and gpt-4 can combine the ability to summarize and translate to perform zero-shot cls, and achieve competitive results with the fine-tuned baseline (i.e., mbart-50). in addition, the current open-source llms (i.e., bloomz, chatglm-6b, vicuna-13b and chatyuan) generally show their limited ability to perform cls in an end-to-end manner, showing the challenge of performing zero-shot cls still exists. in the future, we would like to unleash the potentiality of llms and leverage llms to perform cls in few-shot learning manners.