Benchmarking Large Language Model Capabilities for Conditional Generation (2023)

Abstract: Pre-trained large language models (PLMs) underly most new developments in natural language processing. They have shifted the field from application-specific model pipelines to a single model that is adapted to a wide range of tasks. Autoregressive PLMs like GPT-3 or PaLM and associated techniques like fewshot learning, have additionally shifted the output modality to generation instead of classification or regression. Despite their ubiquitous use, the generation quality of language models is rarely evaluated when these models are introduced. Additionally, it is unclear how existing generation tasks–while they can be used to compare systems at a high level–relate to the real world use cases for which people have been adopting them. In this work, we discuss how to adapt existing application-specific generation benchmarks to PLMs and provide an in-depth, empirical study of the limitations and capabilities of PLMs in natural language generation tasks along dimensions such as scale, architecture, input and output language. Our results show that PLMs differ in their applicability to different data regimes and their generalization to multiple languages. They further inform practitioners as to which PLMs to use for a given generation task setup. We share best practices to be taken into consideration when benchmarking generation capabilities during the development of upcoming PLMs.

## introduction natural language generation tasks require generating understandable text given textual or nonlinguistic information as input, such as documents, tables, or other structured forms. these texts seek to achieve a communicative goal (e.g., summarize a document). the standard approach to tackle these problems over the last years has been to start with a pretrained encoder-decoder model like t5 @xcite or bart @xcite and finetune it on a corpus that captures the downstream task. the recent much larger pretrained language models use a decoder-only architecture and upended this paradigm. these models enabled few-shot or in-context learning approaches in which a model is presented with one or more examples and tasked to continue generating without any finetuning. we refer to both kinds of pretrained models as plms. due to the lack of grounding in the specific task setup, few-shot learning in generation settings leads to a model approaching the communicative goal from very different angles. these diverse range of outputs make the typical reference-based automatic evaluation strategies largely incompatible. while human evaluation can be used to overcome this shortcoming, it is infeasible to monitor the performance of an actively training model this way or to re-run all evaluations every time a new model is introduced. this leads to the question how one should reliably monitor generation capabilities, a question that is only growing in importance as more tasks are approached by casting them into generation setups. in this work, we evaluate 8 models in few-shot and finetuning settings on 27 generation tasks covering 14 languages via automatic evaluation, presenting the first large-scale benchmark of plms in conditional nlg settings. we discuss design choices and challenges to ensure a fair comparison between the different systems, including suitable methods, tasks, and metrics. based on our empirical results, we derive recommendations that could be used for future benchmarks during the development of plms. to combat the need for repeating computationally expensive explorations, we investigate how many evaluation examples are necessary to identify differences between models and find that, in many cases, fewer than 500 examples are sufficient, which opens the path for future evaluation-only task developments. ## background and related work the shift from specialized pipelines toward pretrained language models has led to significant changes in how models are evaluated. we now focus more on questions such as "how good are the learned representations?" instead of user-facing measures of utility. the changes manifested in leaderboards and standard benchmarks that aim to characterize a wide range of model capabilities @xcite ). an additional recent shift is that from finetuning toward few-shot learning. models like t5 @xcite , bart @xcite , and mt5 @xcite were finetuned on supervised datasets covering tasks including translation and summarization, and their outputs are compared to "ground truth" outputs via widely used metrics like rouge @xcite which provide a noisy indication of the "quality" of the output and which can be used to determine whether a model is better than others. 1 in contrast, large plms with autoregressive language modeling pretraining objectives are more capable to produce results without explicit finetuning and are thus typically evaluated via few-shot and in-context approaches, where the model is given the task description and exemplars showing how the task should be completed. @xcite and models that followed such as glam @xcite , gopher @xcite , and lamda @xcite , have achieved few-shot state-of-the-art results on a large number of tasks at their time of publication. however, few-shot approaches work best for tasks with a clear answer such as classification or span-based question-answering. 2 generation metrics penalize systems when their writing style differs from how the references are written @xcite @xcite . without finetuning, there is no guarantee that plms produce outputs that look like the ground truth, both in style and content. recent work found that these differences leads to sharp differences in how humans and automatic metrics rate the generation quality @xcite . due to this uncertainty, most evaluations of new plms are limited to nlu benchmarks such as super-glue @xcite . for example, lamda @xcite did not evaluate on nlg tasks, glam @xcite limited its generation evaluation to short span question answering tasks, and gpt-3 (brown et al., 2020) evaluated only on machine translation. a first autoregressive plm with broad nlg evaluation, palm @xcite , benchmarked summarization and data-to-text tasks in multiple languages. the recent holistic evaluation of language models project (helm, @xcite aims to standardize evaluation of language models. with the explicit goal to broaden the task and metric coverage, helm has established an impressive few-shot benchmark for many natural language tasks. corroborating the prior findings, they also conclude that human evaluation is necessary for nlg. this distinction means that the referencebased approach for generated text that the field has used since the advent of deep learning may no longer sufficient and that we need clear evaluation protocols that continue to allow us to answer broad questions about "generation quality" of a model. complementing this work, we take a deeper look at a wider set of nlg tasks and explore llms in finetuning and few-shot setups to identify whether reference-based automatic evaluation can still be used to produce system rankings. ## deriving evaluation practices figure 1 summarizes the recommendations we developed from challenges we faced and our observed empirical results. these recommendations are best understood in the context of monitoring few-shot results lag behind finetuned results and the gap increases as tasks become more complex. the non-english performance mostly follows the trend that higher percentages of non-english pretraining data leads to better performance. despite their much smaller size, encoder-decoder model frequently much larger decoder-only models after finetuning. comparable few-shot learning evaluation as mentioned in section 3, our design choices were made to ensure that results are comparable across plms. primarily, prompts were deliberately kept extremely simple and all few-shot exemplars were randomly sampled. while highly curated prompts or methods like chain-of-thought prompting can increase the performance considerably @xcite , it can also lead to overfitting to the particular model the prompt was developed on, in turn making a comparison to other models unfair and producing unrealistic expectations when people have single interactions with it. overlap-based metrics are not calibrated to evaluate few-shot learning few-shot generation suffers from not being able to predict output length properly given the few exemplars provided. while encoder-decoder models utilize endof-string tokens, these are not always learned during decoder-only pretraining. to circumvent this issue, researchers rely on plms match to the fewshot format provided e.g. line-breaks that separate exemplars. we observed plms fail to follow the format a significant number of times, producing the largest allowed length on occasion. in our experiments, we tried to avoid very long outputs by trimming outputs to the 95-percentile length seen in the targets. 10 still, few-shot output lengths are on average 2-3 times the average target length while finetuned model's output average 80% the average target length, across all tasks. overlap metrics used in generation are sensitive to length (sun et al., 2019) making a natural disadvantage for few-shot learners. we do not recommend using overlap-based metrics to compare few-shot results without length normalization. computational costs can be decreased without sacrificing relative model performance the computational cost of evaluating large datasets, some with more than 10k examples, are prohibitive and perhaps unnecessary. to that end, we investigate if a model ranking can be produced, with a high degree of certainty, while only considering a random subset of the test set, saving compute cost to possibly evaluate on more tasks instead. to investigate this effect, we ran the following experiment: (1) sample n datapoints from a dataset and all corresponding model scores. ## discussion and reality check in line with our goal to provide a "reality check" via empirical and theoretical research, and to reflect on the ways in which reported performance improvements are meaningful, we want to situate our findings in the context of the broader nlp community. openly accessible apis and publicly available large models have led to increased attention on large pretrained models, but they have also led to a "release-then-test" philosophy where models are released without extensive evaluations. while the findings we present in this paper do not solve this issue, agreeing on a shared evaluation process could lead to more realistic claims about model performance (and shortcomings), and allow for a more accurate monitoring of models during training. what claims can we not make? empirical findings demonstrate that incorporating generation into nlu tasks via chain-of-thought leads to better model performance @xcite . providing additional grounding via finetuning on instructions and aligning a model to human feedback leads to better task-specific performance without supervision @xcite . however, we lack the scientific methods to quantify these advances. while benchmarks provide an indication whether a model is performing better than a previous iteration, and projects like big-bench @xcite and helm @xcite enable evaluation on a very wide range of possible tasks, they are also inherently limited. when benchmarking models in few-shot settings, especially models for which little information about their training data is available, it is hard to disambiguate model performance from memorization, i.e. if the examples were seen during pretraining. instruction tuning further blur the line between finetuning and few-shot, which can lead to very different outputs and are not fully comparable. it is thus near impossible to make claims about why a model is succeeding at one particular task without having access to its training data. as mentioned earlier, the target of this work is to derive best practices for comparing models in generation settings with constrained computational budgets, for example when monitoring a training model or when trying to compare on many different tasks. our findings are grounded in much prior work that finds that metrics have a very high agreement with human judgments on the systemlevel (e.g., @xcite , but are essentially meaningless on the segment-level. for that reason, we cannot derive claims beyond these rankings about utility of a model or whether a particular model would actually produce useful outputs for a task. to derive such insights, we point to work on extrinsic evaluation which requires comprehensive human evaluations (e.g., @xcite . how can our findings be applied to improve the status quo? since the generation capabilities of plms are currently not extensively monitored or evaluated, we set out to derive best practices for how these evaluations can look. we found that many of the "easy" tasks, on which finetuned models saturate the metrics, still yield insights for fewshot approaches. we further identified the tension between doing a computationally expensive full evaluation on a dataset and adding more evaluation sets for different tasks or languages. our findings suggest that evaluation on small subsets of more tasks can be beneficial to the overall results. to further motivate this suggestion, consider the following thought experiment: we have two tasks, a and b. at 500 examples, they have a risk of producing a "wrong" ranking of 10%. at 1,000 examples, they have a risk of producing a wrong ranking of 5%. these risks are not correlated, i.e., their covariance is 0. given a computational budget of evaluating on 1,000 examples, the risk of only evaluating on one dataset is 5%, and the risk of producing two wrong ratings after evaluating on a and b is only 1%. while additional datasets introduce a larger risk of one individual dataset producing misleading results (18% in this case), one can easily expand this argument to a whole portfolio of tasks to hedge against individual dataset risk @xcite . many existing nlu benchmarks like big bench @xcite already follow such a strategy and we believe that generation evaluation, especially considering the additional risk due to metrics, should follow this approach for the use cases discussed in this work. to further minimize the individual dataset risk, they can be switched out once they saturate or their sample sizes increased. ## conclusion in this work, we produced an extensive evaluation of a diverse set of state-of-the-art pre-trained language models (plms) for 27 different multilingual generation tasks under few-shot learning and finetuning settings. we discuss empirical results that help inform practitioners which tasks, methods and metrics are suitable. we provide recommendations on how best to monitor conditional generation capabilities of plms, including how to fairly benchmark few-shot learning, automated metrics and their utility, and how to efficiently utilize computational resources. we hope that such findings and recommendations could positively influence natural language evaluation in future work.