P roxy QA : An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models (2024)

Abstract: Large Language Models (LLMs) have succeeded remarkably in understanding long-form contents. However, exploring their capability for generating long-form contents, such as reports and articles, has been relatively unexplored and inadequately assessed by existing benchmarks. The prevalent evaluation methods, which predominantly rely on crowdsourcing, are recognized for their labor-intensive nature and lack of efficiency, whereas automated metrics, such as the ROUGE score, demonstrate discordance with human judgment criteria. In this paper, we propose ProxyQA, an innovative framework dedicated to assessing long-text generation. ProxyQA comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers. LLMs are tasked to generate extensive content in response to these meta-questions, by engaging an evaluator and incorporating the generated texts as contextual background, ProxyQA assesses the generated content’s quality through the evaluator’s accuracy in addressing the proxy-questions. We examine multiple LLMs, emphasizing ProxyQA’s demanding nature as a high-quality assessment tool. Human evaluation demonstrates that the proxy-question method is notably self-consistent and aligns closely with human evaluative standards. The dataset and leaderboard is available at https://proxy-qa.com .

## introduction recent large language models (llms) have made significant advancements @xcite @xcite . gpu technology innovations and memory-efficient attention mechanisms @xcite have further enabled llms to model context sequences figure 1 : prior efforts assess generated content by matching it with references through human evaluation or automated metrics. proxyqa evaluates the knowledge coverage and informativeness by checking if generated contents contain sufficient information to answer a set of proxy questions. one primary roadblock to understanding llms' capability to generate long-form texts is the lack of competent evaluation methods. current methods, often involving a combination of automated metrics and crowdsourced annotations, leave much to be desired @xcite . for instance, automated metrics that use word-level string @xcite or meaning representation matching @xcite rely on gold references, which unfortunately do not exist for many generation tasks, such as reports or essay writing. furthermore, these automated metrics are inadequate for reliably assessing long-form content due to the considerable and unstructured space of potential outputs @xcite . human evaluation has its own set of issues too. crowdsourced workers may lack the necessary expertise for evaluating knowledge-rich content, and domain experts' subjective preferences could result in inconsistent evaluations @xcite . while recent studies explored using llms for evaluation @xcite , llms have been found to lack the most current information required for precise verification. moreover, their assessments have been observed to be inconsistent @xcite . there is a clear need for more robust and precise evaluation methods. to address this issue, we introduce proxyqa, a benchmark comprising human-curated metaquestions covering a wide range of subjects, from computer science to history. these meta-questions require domain expertise and up-to-date knowledge, prompting llms to generate detailed and comprehensive responses. to assess knowledge coverage and informativeness, we pair each metaquestion with a series of proxy-questions and answers that capture its essential points. as illustrated in figure 1 , proxyqa uses an evaluator to answer proxy-questions based on the long-form content produced by llms, rather than comparing the output to a reference. if the generated content is sufficiently detailed and accurate, it should equip the evaluator with enough information to thoroughly answer all associated proxy-questions. proxyqa offers several benefits. by employing proxy questions and an evaluator, it eliminates the need for direct comparison against a single gold reference, enabling a more accessible and subjective evaluation. using this approach allows evaluators without specific domain knowledge to assess content. additionally, unlike previous datasets compiled from online sources @xcite ) that potentially leading to data contamination @xcite , all the proxy-questions and answers are invisible to public, thereby preventing data leakage. we apply proxyqa to extensively test different llms @xcite @xcite @xcite , including the llms that enhanced iterative reasoning @xcite and retrieval augmentation @xcite . a systematic human evaluation demonstrates that proxyqa offers a highly consistent evaluation scope, surpassing inter-human agreement rates while maintaining strong correlations with the majority preferences of humans. in-depth analysis shows that proxyqa effectively eliminates the bias that llm-as-a-judge @xcite prefer content generated by gpt models, and the evaluators' accuracy on the proxyquestions accurately reflects the quality of the generated contents. ## a long-form generation benchmark an alternative framework for evaluating long-form text generation is created. 100 meta-questions are annotated to prompt llms to generate detailed and informative responses, which cover subjects in artificial intelligence (ai) research, historical event investigations, sports and entertainment event analysis. the topic distribution is shown in figure 2 . each meta-question accompanies various proxyquestions with annotated answers, which are invisible to the llms to be tested. to proxy the objective evaluation of the generated contents to subjective metrics, we adopt an evaluator that takes the generated contents as contextual background and answers the proxy-questions. we assume that only if the contextual background is informative and comprehensive, can the proxy-questions be well-addressed by the evaluator. therefore, the quality of the generated contents is reflected in the evaluator's accuracy on the proxy-questions. ## conclusion in this work, we introduce proxyqa, a framework designed to evaluate llms' ability to generate long-form text. unlike traditional methods that rely on a direct comparison with a reference text, by employing an evaluator to use the information provided in the llm-generated text to answer proxy-questions, the framework assesses the llms' ability without fixed references or crowdsource workers. by mitigating concerns over data contamination and ensuring the relevance and freshness of evaluation content, proxyqa enhances our understanding of llms and drives innovation towards developing long-form generation methods with llms.