A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks (2023)

Abstract: “Recent efforts have evaluated large language models (LLMs) in areas such as com-monsense reasoning, mathematical reasoning, and code generation. However, to thebest of our knowledge, no work has specifically investigated the performance of LLMsin natural language generation (NLG) tasks, a pivotal criterion for determining modelexcellence. Thus, this paper conducts a comprehensive evaluation of well-known andhigh-performing LLMs, namely ChatGPT, ChatGLM, T5-based models, LLaMA-basedmodels, and Pythia-based models, in the context of NLG tasks. We select English andChinese datasets encompassing Dialogue Generation and Text Summarization. More-over, we propose a common evaluation setting that incorporates input templates andpost-processing strategies. Our study reports both automatic results, accompanied by adetailed analysis.”

## introduction recent studies have emphasized the importance of scaling large language models (llms), referring to both the dimensions of the model size themselves and the amount of data used, resulting in enhanced capability of the models for tasks downstream @xcite . numerous investigations have been conducted to explore the limits of performance by training increasingly larger pre-trained language models, such as gpt-3 175b @xcite and palm 540b @xcite . although scaling primarily involves increasing the model size while maintaining similar architectures and pre-training tasks, these large-sized plms exhibit distinct behaviors from their smaller counterparts and demonstrate surprising emergent abilities in solving complex tasks @xcite @xcite . an example of this is the contrasting performance of gpt-3 and gpt-2 when it comes to solving few-shot tasks. gpt-3 demonstrates effective problem-solving abilities by utilizing in-context learning, whereas gpt-2 faces difficulties in this aspect. as a result, these large-scale language models (llms) has become a huge research topic in current nlp area. in existing literature, remarkable llms such as chatgpt 0 , chatglm 1 , have been widely adopted as powerful ai assistants, benefiting from their exceptional generation capabilities. we hypothesis that a language model's performance in executing natural language generation (nlg) tasks is a crucial factor in determining its excellence @xcite . nlg tasks involve llms that are capable of accepting diverse types of input, such as texts and tables, and generating coherent and appropriate output text. we intuitively think that generate fluent, coherent, and consistent texts is the foundation of a language model, so as to large language models @xcite . when some research institutions release their large language models, they tend to evaluate these models first. community workers are also interested in testing well-known large language models. however, most of these evaluations focus on checking llms' ability of commonsense reasoning @xcite , mathematical reasoning @xcite , code completion @xcite , etc., but ignore the basic nlg tasks, such as dialogue generation @xcite ), text summarization @xcite , and story generation @xcite . besides, some researchers pointed out that the performance of a large model is determined not only by its size and architecture, but more by the quality and quantity of training data. based on this point of view, researchers open source and propose that some smaller-scale models trained on more and higher-quality data sets can achieve the same performance as models with more parameters than them. for example, llama-13b @xcite outperforms gpt-3 on most benchmarks, despite being 10 times smaller. this notable discovery makes us curious about the performance of models with different architecture, data size, and mode size, trying to figure out which factor is more important. therefore, we aim to address this gap by conducting a comparative analysis of llm performance on nlg tasks, considering different architectures and scales throughout the evaluation process. in this paper, we present a systematic evaluation of existing llms for nlg tasks. the main objective is to enhance our understanding of instruction and prompt design by conducting a comparative analysis of these models. initially, we provide an overview of classic nlg tasks, including their definitions and associated english and chinese datasets. subsequently, we devise a model input template that includes instructions for each dataset. following that, we introduce various llms, considering factors such as model size and architecture. finally, we present the results of both automatic and manual evaluation of llms on nlg datasets, and discuss the strengths and weaknesses of their performance across different models. ## natural language generation in this section, we will introduce the definition of nlg, and its sub-tasks with some corresponding datasets that we will use to evaluate llms.