B atch E val: Towards Human-like Text Evaluation (2024)

Abstract: Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigm suffers from the following issues: (1) Sensitive to prompt design; (2) Poor resistance to noise; (3) Inferior ensemble performance with static reference. Inspired by the fact that humans treat both criterion definition and inter sample comparison as references for evaluation, we propose BatchEval, a paradigm that conducts batch-wise evaluation iteratively to alleviate the above problems. We explore variants under this paradigm and confirm the optimal settings are two stage procedure with heterogeneous batch composition strategy and decimal scoring format. Comprehensive experiments across 3 LLMs on 4 text evaluation tasks demonstrate that BatchEval outperforms state-of-the-art methods by 10.5% on Pearson correlations with only 64% API cost on average. Further analyses have been conducted to verify the robustness, generalization, and working mechanism of BatchEval.

## introduction accurately evaluating the text quality in specific criterion (e.g., coherence) can facilitate better understanding, application, and development of large language models (llms), which becomes more crucial with their recent rapid progress in text generation capabilities @xcite . due to the labor-intensive and time-consuming nature of human evaluation, early works have explored automatic evaluation methods, which can be categorized into rule-based @xcite , embedding-based @xcite , and learning-based @xcite approaches. continuous progress has been achieved through these methods, but there remains a significant gap in their consistency with human judgments @xcite . recently, the revolutionary power of llms has been applied across various fields, demonstrating performance that is even on par with humans (ope-nai, 2023; @xcite . in text evaluation filed, llm-based evaluators (chiang and lee, 2023a; @xcite @xcite have also made significant progress compared to traditional methods, but they still lag behind human evaluators. we carefully compare their working procedures and find that the difference in evaluation references might be the reason for the performance disparity (figure 1 ). human evaluators analyze samples based on the criterion definition and provide discriminative scores through comparison between samples. however, llm-based evaluators assess each sample individually, thus only having criterion as a reference. we analyze that current sample-wise evaluation paradigm will face problems on three aspects: (1) robustness against prompt design? since criterion is the sole reference for evaluation, minor changes of prompt may significantly affect the evaluation results (see §4.4 for empirical validation). (2) robustness against noise? due to the absence of comparison between samples, the evaluation scores lack discrimination and exhibit a non-uniform distribution (see figure 3 ), which can lead to reduced robustness against noise like random deletion or synonym substitution on samples (see theorem 1). to address the aforementioned problems, we propose batcheval, a new llm-based text evaluation paradigm that assesses samples batch-wise, akin to the way of humans. overall, batcheval iterates an allocation process where all samples are first split into batches, and then each batch is compiled into a prompt as the input of llms. by introducing in-batch samples as an additional reference apart from criterion, the orthogonal and complementary references can not only reduce the dependency on prompt design but also enhance the discrimination of scores between samples through in-batch comparison, leading to improved robustness against noise. furthermore, the iteratively changing batch composition can provide llms with varying evaluation references, thereby enhancing diversity and the ensemble performance. while the idea of batcheval is simple, there are many ways it can be realized. we explored variants in evaluation procedure, format of scoring and composition of batch. some of them work surprisingly well while some do not meet expectations. experiments and analyses confirm that separate analyzing and scoring evaluation procedure, decimal scoring format, and quality-heterogeneous batch composition strategy yield the optimal results. we conduct extensive experiments on 4 text evaluation tasks primarily with gpt-4: turn-level response, dialogue, text summarization, and story generation. by allowing in-batch samples to share single prompt and applying a small iteration rounds, batcheval outperforms best performing llmbased evaluators by a significant margin (10.5%) in terms of correlation with human evaluations, while incurring only 64% of api costs. we also validate the generalization of batcheval on more llms, robustness to prompt design and noise, and analyze the choice of hyperparameters through further experiments. finally, we probe into the working mechanism of batcheval through attention anal-ysis on llama-2-70b-chat-hf. our contributions are summarized as follows: ## experiments centered around batcheval, we will empirically explore the optimal variants in §4.2, demonstrate its performance on different llms and tasks in §4.3, validate the robustness in §4.4, and delve into its working mechanism in §4.5. we also investigate the choice of hyperparameters in appendix §b. ## conclusions in this paper, we propose batcheval, a new text evaluation paradigm that evaluate samples batch-wise to alleviate the limitations of samplewise evaluation paradigm. we explore variants of batcheval on multiple dimensions and figure out the optimal settings. following the human evaluation method, batcheval treats in-batch samples and criterion as complementary references and optimizes the batch composition through iteration to eliminate batch bias. comprehensive experiments have confirmed that batcheval can achieve higher consistency with humans at a lower cost, while also demonstrating better robustness to prompt design and noise. we further analyze and reveal the working mechanism of batcheval, shedding lights on future work. the combination of in-batch samples more diverse, thereby increasing scoring diversity, we have the following conjecture about figure 7 : when the batchsize starts to increase from 1, due to the effect of in-batch comparison and the increase in diversity, the performance of both 1-round score and ensemble score increase a lot. however, as the batchsize continues to increase, llm finds it difficult to handle too many samples simultaneously, resulting in a decrease in 1-round score performance. when the rate of decrease in 1-round score performance gets greater than the rate of increase in diversity, ensemble score performance also begins to decrease according to theorem 2. therefore, the batchsize should not be too large or too small. we found that setting the batchsize to 10 can achieve superior performance on different tasks. we also believe that for llms with weaker ability to handle longer context, the batchsize should be set to be smaller. fortunately, we have noticed that current llms are continually improving in processing long contextual texts, which illuminates further development prospects for batcheval in the future.