T^5Score : A Methodology for Automatically Assessing the Quality of LLM Generated Multi-Document Topic Sets (2025)

Abstract: Using LLMs for Multi-Document Topic Extraction has recently gained popularity due to their apparent high-quality outputs, expressiveness, and ease of use. However, most existing evaluation practices are not designed for LLM-generated topics and result in low inter-annotator agreement scores, hindering the reliable use of LLMs for the task. To address this, we introduce T^5Score , an evaluation methodology that decomposes the quality of a topic set into quantifiable aspects, measurable through easy-to-perform annotation tasks. This framing enables a convenient, manual or automatic, evaluation procedure resulting in a strong inter-annotator agreement score.To substantiate our methodology and claims, we perform extensive experimentation on multiple datasets and report the results.

## introduction topic extraction plays a key role in computational text analysis, enabling the unsupervised discovery of salient information while presenting it in a summarized and structured manner. for this reason, there are numerous methods for addressing it @xcite . recently, alongside the rise of llms, solutions are shifting towards using generative models (e.g., @xcite @xcite . this trend is motivated by llms' ability to overcome some of the drawbacks of existing methods, such as clear topic naming, contextualization, and operating without dedicated training @xcite . still, to validate such a use, llms must be assessed carefully. unfortunately, collecting manually annotated topic sets for direct assessment is impractical @xcite due to annotators' bias @xcite , 2020), and the cognitive burden of processing large amounts of information at once @xcite . for this reason, existing evaluation procedures rely on indirect approximations (e.g., @xcite , rather than direct assessments of quality. moreover, these practices result in poor inter-annotator agreement (iaa) @xcite . in the absence of standard evaluation practices, scientific works that employ topic sets often do not include a thorough evaluation of the "correctness" of the extracted topics @xcite @xcite . while these practices are common, stipulating the correctness of the topic sets without evidence is unwarranted, and equally in the case of llm-generated topic sets. in this paper, we develop a methodology for evaluating llm-based topic sets extracted from a collection of documents. while the term topic extraction (or topic modeling) commonly refers to methods like lda @xcite , which output topics as word distributions, in this work, we focus on llm-based topics that are free-text in nature and therefore are better portrayed as theme-describing titles (e.g., "experiences of discrimination"). for clarity, we will distinguish between the two cases by referring to them as word-distribution (wd) topics and free-text (ft) topics respectively, formally defined in §3.1. to overcome the challenges in direct assessment of topics, we introduce t 5 score, a dedicated evaluation methodology for indirectly assessing ft-topic sets, overcoming the annotator agreement barrier. acknowledging the drawbacks of using aggregate metrics @xcite , we propose to decompose the quality of an ft-topics set into three simple annotation tasks defining five scores along different aspects of quality. see fig. 1 for a schematized view of the methodology. the remainder of this paper is structured as follows. first, we motivate the formulation of t 5 score and formally define it. next, we conduct a human-oriented case study showing that using the methodology for manual evaluation results in high iaa scores. then, we claim that the slow and expensive manual process can be automated via llm-based labeling and support this by reporting a strong correlation with human annotators. finally, we show that the methodology validly reflects the expected ordering between generation systems by applying it to ft-topic sets generated by different systems. throughout this work, alongside others, we employ a novel dataset of holocaust survivor testimonies collected by usc shoah foundation (sf), 2 providing collections of common yet unique experiences ideal for our work. 3 2 related work ## iaa study in the following, we demonstrate that using t 5 score in evaluation procedures performed manually by human annotators results in high interannotator agreement (iaa). we conduct a humanoriented case study designed according to the methodology and report the observed iaa score. ## llms as automatic evaluators t 5 score decomposes the evaluation process into small and simple annotation tasks. this framing enables the use of off-the-shelf llms as autonomous annotators, making this methodology applicable independently of slow and expensive human annotation procedures. we experiment with different popular llms, reporting comparable performance to human annotators. ## applying t 5 score on ft-topic sets thus far, we motivated the design of t 5 score ( §3) and proposed an automatic implementation achieving human-like performance ( §6). yet, it remains to show whether this framing is valid, namely whether it reflects the expected ordering between different systems. often, such a study is performed by comparing the resulting relative order of systems to an ordering based on human preferences (e.g., @xcite . however, since human annotation of ft-topic sets is unreliable ( §2), this option is unfeasible. indeed, such analysis is often omitted from other topic set evaluation works (e.g., @xcite , where only the ability to perform the proxy task (e.g., recognizing an intruder in intrusion tests) is compared to humans (as in §6). instead, we devise case studies to empirically support the validity of our methodology. we carefully design generation systems where the relative ordering between at least a subset of them is clear and show that this ordering is reflected by t 5 score. since it could be argued that some of these systems are too simplistic, we also synthetically compile a set of human-written topics and study t 5 score's performance on it too. ## conclusion we presented t 5 score, an evaluation methodology that decomposes the quality of an ft-topics set into aspects quantifiable by easy-to-perform measurements. we justified using our methodology by conducting extensive experimentation on a novel dataset of holocaust survivor testimonies as well as a dataset of news reports. the results showed that our methodology achieves a high level of iaa on manual evaluation procedures. they also demonstrated that it can be automatically performed by judge models that simulate human annotations reliably. furthermore, the results confirmed that our methodology validly reflects the expected relative order between the tested systems in cases where the order is known beforehand. given the centrality of the task and the great difficulty in evaluating it reliably, we hope that the methodology proposed here will assist in the development of demonstrably stronger topic set generation systems.