Multi-Dimensional Evaluation of Text Summarization with In-Context Learning (2023)

Abstract: Evaluation of natural language generation (NLG) is complex and multi-dimensional. Generated text can be evaluated for fluency, coherence, factuality, or any other dimensions of interest. Most frameworks that perform such multi-dimensional evaluation require training on large manually or synthetically generated datasets. In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning, obviating the need for large training datasets. Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization, establishing state-of-the-art on dimensions such as relevance and factual consistency. We then analyze the effects of factors such as the selection and number of in-context examples on performance. Finally, we study the efficacy of in-context learning-based evaluators in evaluating zero-shot summaries written by large language models such as GPT-3.

## introduction developing comprehensive evaluation frameworks @xcite @xcite that can evaluate multiple humaninterpretable dimensions, such as factual consistency @xcite and coherence @xcite , is important for the advancement of natural language generation (nlg). however, similaritybased metrics @xcite @xcite @xcite still dominate nlg evaluation in practice. compared to them, desired multi-dimensional evaluators do not require reference texts for evaluation; and they can easily extend to new explainable evaluation dimensions. @xcite developed a unified evaluation framework that can generalize to multiple dimensions and text generation tasks. however, it relies on the construction of synthetic and auxiliary data for the finetuning of a pre-trained language model, requiring in-depth knowledge and significant engineering effort for each dimension. furthermore, the inclusion of new dimensions requires (continued) training of the model, and might affect the performance on other dimensions in unforeseen ways. in this work, we propose to use in-context learning @xcite with large language models (llms) -a commonly used method to perform many tasks by utilizing only a few input-output examples -to perform multi-dimensional text evaluation in a unified fashion. compared to pre-trained evaluators that need specialized supervised training for each dimension, our in-context learning-based evaluator (ice) framework is: in this paper, using text summarization as a test bed, we show that with a simple prompt design, ice is competitive with state-of-the-art trained evaluators on multi-dimensional evaluation of modelproduced summaries, establishing a new state-ofthe-art on dimensions such as relevance and factual consistency. to study the robustness of the evaluator to the selection of in-context examples, we analyze the factors that affect the performance of ice, such as the number of in-context examples and sampling procedures when picking in-context examples from a set of candidates. we find ice to be robust to the selection of in-context examples and observe a slight improvement in performance as the number of examples is increased. finally, in light of the recent work @xcite that points to the misalignment of existing evaluation metrics with human preference in evaluating zeroshot summaries generated by llms such as @xcite , we study the effectiveness of ice in evaluating zero-shot summaries generated by gpt-3. we find that ice evaluations agree closely with human judgments on such summaries. ## analysis in this section, we analyse the effects of our prompt engineering choices. the comparison between sampling procedures in section 4.1 is performed on the entire test set but the experiments in sections 4.2 and 4.3 are performed on a test set sample of size 200 to control costs. the analyses in sections 4.1 and 4.2 use four in-context examples. ## using ice to evaluate zero-shot prompting models recent work by @xcite showed that standard reference-based and reference-free metrics are not reliable in evaluating zero-shot summaries written by models such as gpt-3. through a human study comparing summaries from three systems-gpt-3, brio, and t0-they observed that while humans prefer gpt-3 summaries, automatic evaluators consistently score gpt-3 summaries lower than summaries from other models. we study the efficacy of ice in evaluating zeroshot summaries written by gpt-3 at a dimension level. we use the set of 500 cnn articles from @xcite , with summaries from gpt-3, brio, and t0 for each article. we sample 100 of these articles and have three annotators rate summaries for each of the dimensions defined in section 2.2 on a scale of {1, 2, 3, 4, 5}. we use ice, rouge, and bartscore (all of which do not require training data) to evaluate the summaries and present system-level results in we observe that ice agrees with human judgments for each dimension and overall preferences while existing reference-based and reference-free metrics such as rouge and bartscore foot_0 consistently rate gpt-3 summaries low. @xcite suggest that most existing evaluation metrics reward summaries that imitate references, while gpt-3 summaries are zero-shot and not trained to imitate human-written references, which is likely why they are penalized by most existing evaluators. however, since ice is not based on reference similarity (except when evaluating relevance) and is also not trained with reference summaries, it is able to better evaluate gpt-3 summaries and agrees with human preferences. ## conclusion we show that in-context learning can be used for nlg evaluation as an alternative to fine-tuned evaluation metrics. using a small number of examples, in-context learning evaluators can reach or exceed the state-of-the-art on multi-dimensional evaluation and that this is robust to the choice of in-context examples. finally, we show that in-context learning evaluators align well with human judgements when evaluating summaries written by gpt-3.