QUARTZ : QA -based Unsupervised Abstractive Refinement for Task-oriented Dialogue Summarization (2025)

Abstract: Dialogue summarization aims to distill the core meaning of a conversation into a concise text. This is crucial for reducing the complexity and noise inherent in dialogue-heavy applications. While recent approaches typically train language models to mimic human-written summaries, such supervision is costly and often results in outputs that lack task-specific focus limiting their effectiveness in downstream applications, such as medical tasks. In this paper, we propose QUARTZ, a framework for task-oriented utility-based dialogue summarization. QUARTZ starts by generating multiple summaries and task-oriented question-answer pairs from a dialogue in a zero-shot manner using a pool of large language models (LLMs). The quality of the generated summaries is evaluated by having LLMs answer task-related questions before (i) selecting the best candidate answers and (ii) identifying the most informative summary based on these answers. Finally, we fine-tune the best LLM on the selected summaries. When validated on multiple datasets, QUARTZ demonstrates its effectiveness by achieving competitive results in various zero-shot settings, rivaling fully-supervised State-of-the-Art (SotA) methods. Code will be released publicly.

## introduction automatic text summarization (ats) is the task of generating a concise summary from a lengthy text. it can be (extractive) -selecting and concatenating key sentences -or (abstractive) -generating new condensed formulation that resemble humanwritten summaries @xcite . while early ats methods relied on graph-based @xcite and frequency-based techniques @xcite , the emergence of large language models (llms) has transformed the field, achieving breakthroughs across domains, including healthcare @xcite . dialogue summarization, a subfield of ats, targets key information distillation from conversations. it has gained prominence due to its applicability in realworld settings such as customer service @xcite , business meetings @xcite , and healthcare @xcite , where the input is often less structured than typical written texts like news articles. summarizing such dialogues helps simplify complex interactions and supports downstream tasks like decision-making and automation. dialogues introduces challenges absent in standard ats: disfluencies, speaker turns, redundancy, verbosity, and loosely structured content. these characteristics make traditional ats methods, designed for more structured text, less effective, often resulting in low-quality summaries @xcite . recently, the capability of llms to perform dialogue summarization has been assessed by framing it as a sequenceto-sequence task @xcite . fine-tuning llms for task-oriented summarization has demonstrated promising results, even surpassing human performance @xcite . however, this approach has limited practicality due to the reliance on costly human-written summaries. moreover, while the vast amount of text seen during training enables llms to be powerful "multitask learners", their lack of robust mechanisms for maintaining topic focus and factual consistency makes them prone to topic-deviations, often resulting in incoherent summaries @xcite . in this paper we present quartz, a framework for unsupervised, task-oriented abstractive dialogue summarization. its task-oriented design enables better control over llm outputs by maintaining topic focus and factual consistency, while its unsupervised nature eliminates the need for reference summaries or domain-specific knowledge beyond that used to design task-specific llm prompts. our framework encompasses the following novelties and contributions: ## our approach the overall workflow of our approach is illustrated in figure 1 . inspired by recent findings in summarization, showing that llm summaries are significantly preferred by human evaluators @xcite , quartz starts by @xcite prompting a pool of llms -rather than a single model -helps mitigate model-specific tendencies and enables the generation of varied task-oriented summaries along with gold question-answer pairs from the input dialogue. tailored prompts (appendix d) are used to ensure focus on task-relevant information. we also utilize llms as responders to answer the generated questions using the summaries, rather than using the original dialogues. we then (second step) judge the quality of the generated summaries by assessing the candidate answers using a twostage evaluation process. in the first stage, llms are employed as rankers to select the best candidate answer for each question and summary. in the second stage, llm ranking reveals the best summary for each dialogue. finally, (third step) the best llm is fine-tuned on the selected summaries to further improve its task-oriented summarization performance. we detail the different stages of quartz and its evaluation in the following. ## experimental settings in the following, we explain our experimental evaluation protocol, including the datasets we used and the implementation details. ## llm as a judge motivated by recent findings that llms can serve as reliable proxies for human annotators @xcite , we adopt the g-eval framework @xcite ## human evaluation given the high cost of human evaluation, we adopted a two-phase protocol with four expert annotators: all computer scientists, including one with a medical degree to ensure informed assessment, particularly for clinical dialogues. phase one. using the potato annotation tool @xcite , annotators reviewed each dialogue along with two anonymized summaries, one generated by quartz and the other being the reference. without knowing the source of either summary, they were asked to choose the one they preferred based on overall quality. notably, in 48% of cases, the annotators favored quartz summaries over the reference with a fleiss' kappa @xcite agreement of 0.14 (slight agreement). although no "equal quality" option was provided, some annotators reported difficulty in choosing a winner, stating that both summaries were often equally informative. phase two. annotators evaluated the final summaries produced by quartz according to the four qualitative dimensions as in section 6. the average scores are presented in ## limitations although we focused on minimizing data annotation costs while maintaining high-quality summaries, some limitations remain. for the sake of simplicity, we did not act on the generated gold question-answer pairs. while we can explicitly incorporate questions relevant to the task, we think that it would be beneficial to establish some control over the questions based on their contribution to summary evaluation, as not all questionanswer pairs carry the same level of importance. while quartz is designed for unsupervised summarization, the lack of human oversight during training and its reliance on llm performance may lead to discrepancies from human-preferred summaries or overlook subtle errors. although human evaluations provided high scores across all dimensions (average ratings over 4 on a 1-5 likert scale), inter-annotator agreement was low, indicating that even expert judgments might differ substantially. the employment of supplementary evaluation techniques is encouraged by this, which highlights the difficulties in assessing subjective components of summary quality. ## ethical considerations while this work demonstrates remarkable advancements toward reference-free dialogue summarization, it is important to be aware of the limitations and risks of relying on such approaches in particularly sensitive and high-stakes areas, especially in healthcare. generative models present serious challenges, such as implicit assumptions embedded in their training data and difficulties in generalizing across wide and complex domains. these limitations will need drastic scrutiny when applying such models in high-stakes environments. for our triage-oriented summarization task, each decision made by the model was checked by a medical expert to ensure its accuracy and relevance. however, the integration of automated systems in healthcare raises ethical and legal concerns regarding accountability and privacy. these considerations further validate our decision to operate on open-source models. unlike closed-source models, which may involve transferring sensitive private data to third-party companies and expose it to potential misuse, open-source approaches allow for greater transparency and control over the underlying data and processes. thus, while llms have shown promising capabilities, their application in critical fields must be approached with caution to ensure that ethical standards are maintained, and that these systems do not inadvertently compromise patient safety or privacy. yicheng zou, bolin zhu, xingwu hu, tao gui, and qi zhang. 2021. low-resource dialogue summarization with domain-agnostic multi-source pretraining. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 80-91.