Can GPT models Follow Human Summarization Guidelines? A Study for Targeted Communication Goals (2025)

Abstract: This study investigates the ability of GPT models (ChatGPT, GPT-4 and GPT-4o) to generate dialogue summaries that adhere to human guidelines. Our evaluation involved experimenting with various prompts to guide the models in complying with guidelines on two datasets: DialogSum (English social conversations) and DECODA (French call center interactions). Human evaluation, based on summarization guidelines, served as the primary assessment method, complemented by extensive quantitative and qualitative analyses. Our findings reveal a preference for GPT-generated summaries over those from task-specific pre-trained models and reference summaries, highlighting GPT models’ ability to follow human guidelines despite occasionally producing longer outputs and exhibiting divergent lexical and structural alignment with references. The discrepancy between ROUGE, BERTScore, and human evaluation underscores the need for more reliable automatic evaluation metrics.

## introduction although instruction-tuned large language models (llms) have shown impressive performance on several benchmark datasets, they still struggle with various challenging tasks @xcite . on automatic metrics like rouge, general-purpose models (e.g., chatgpt) consistently trail behind task-specific, fine-tuned models @xcite @xcite . in dialogue summarization specifically, chatgpt demonstrates poorer performance than state-of-the-art fine-tuned models on datasets including samsum @xcite and di-alogsum @xcite . however, human evaluation studies reveal a more nuanced picture: instruction-tuned llms can demonstrate humanaligned summarization capabilities that automatic metrics may fail to capture @xcite . the underlying causes of gpt models' underperformance on automatic summarization metrics remain poorly understood. human summarization is a complex cognitive process that involves understanding, condensing, and conveying essential information while adhering to specific guidelines. given that human annotators have to follow these guidelines when creating reference summaries, we speculate that the performance discrepancy could be attributed to the lack of explicit summarization instructions tailored to specific communication goals during model prompting. this is particularly critical for dialogue summarization, where targeted communication goals play a significant role. even though dialogue summarization is a wellestablished task, it remains challenging due to the absence of a consensus for what constitutes an ideal dialogue summary @xcite . the approach to dialogue summarization is heavily influenced by specific communication objectives, which vary across different contexts such as meetings, or customer service interactions. a review of guidelines from major dialogue summarization datasets reveals that each corpus defines its own set of objectives for creating reference summaries, applying different summary criteria to meet specific needs @xcite . for instance, in the context of customer service, while tweet-summ @xcite offers both extractive and abstractive summaries, csds @xcite provides three distinct summaries for each dialogue: an overall summary and two role-oriented summaries (user and agent). in this work, we evaluated the ability of gpt models to generate dialogue summaries that adhere to human guidelines, which provides the following contributions: ## experimental setup datasets. we used two datasets covering different communication goals, languages and contexts: dialogsum @xcite and @xcite . dialogsum consists of social conversations in english and includes 12,460, 1,500 and 500 samples in its training, validation and test splits, respectively. it was used in a challenge @xcite , which favors comparison of results. decoda @xcite is a call-center human-human spoken conversation corpus in french, collected from the ratp (paris public transport authority) and mostly deals with customer inquiries and agent responses. the corpus was proposed as a pilot task at multiling 2015 @xcite . the test set used in this study contains 100 conversations with 212 synopses, i.e. references. ## discussion and conclusion human evaluation results show that gpt models (chatgpt, gpt-4 and gpt-4o) can follow human summarization guidelines to some extent. their summaries were preferred over task-specific pretrained models and even outperformed reference summaries. the longer outputs of gpt models, compared to barthez, likely contribute to more comprehensive coverage of issues and resolutions, enhancing faithfulness to dialogues. however, gpt models underperform task-specific pretrained models on both rouge and bertscores metrics. this discrepancy between automatic metrics and human judgments aligns with previous findings, reinforcing (1) the continued necessity of human evaluation for text generation tasks, and the results also indicate that gpt models demonstrate some proficiency in adhering to human guidelines. in the overall evaluation using human guidelines, the @xmath0. interestingly, gpt-4 does not consistently outperform chatgpt; for instance, 4-wl only surpasses 3.5-wl on the 10 shortest dialogues. nevertheless, gpt-4 exhibits better adherence to the specified word length constraints. our findings also highlight the inherent subjectivity of summary quality assessment, as evidenced by the human evaluators' preference for gpt-generated summaries over reference texts, which is partly attributable to stylistic differences. by analyzing data points where gpt-generated summaries (4-wl and 4-@xmath1. however, they sometimes neglect specific rules: in dialogsum, they miss named entities, intent identification, and personal tags; in decoda, some summaries focus on individual locutors' perspectives and fail to present resolutions. the intermediate step of using hgr helped the model better adhere to the specific rules of human summarization guidelines, such as the use of named entities and personal tags in dialogsum. for future work, we plan to develop llmbased metrics that evaluate summaries against taskspecific guidelines, with a focus on improving response stability and alignment with human judgment. furthermore, we will investigate how llms follow different summarization guidelines, including those with lexical perturbations, to interpret their decision-making processes and operational mechanisms.