Chain-of-Interactions: Multi-step Iterative ICL Framework for Abstractive Task-Oriented Dialogue Summarization of Conversational AI Interactions (2025)

Abstract: Large Language Models (LLMs) have introduced paradigm-shifting approaches in natural language processing. Yet, their transformative in-context learning (ICL) capabilities remain underutilized, especially in customer service dialogue summarization—a domain plagued by generative hallucinations, detail omission, and inconsistencies. We present Chain-of-Interactions (CoI), a novel single-instance, multi-step framework that orchestrates information extraction, self-correction, and evaluation through sequential interactive generation chains. By strategically leveraging LLMs’ ICL capabilities through precisely engineered prompts, CoI dramatically enhances abstractive task-oriented dialogue summarization (ATODS) quality and usefulness. Our comprehensive evaluation on real-world and benchmark human-agent interaction datasets demonstrates CoI’s effectiveness through rigorous testing across 11 models and 7 prompting approaches, with 9 standard automatic evaluation metrics, 3 LLM-based evaluations, and human studies involving 480 evaluators across 9 quality dimensions. Results reveal CoI’s decisive superiority, outperforming all single-step approaches and achieving 6× better entity preservation, 49% higher quality scores, and 322% improvement in accuracy compared to state-of-the-art multi-step Chain-of-Density (CoD). This research addresses critical gaps in task-oriented dialogue summarization for customer service applications and establishes new standards for harnessing LLMs’ reasoning capabilities in practical, industry-relevant contexts.

## introduction large language models (llms) have transformed natural language processing through in-context learning (icl) capabilities @xcite ; tang 1 dataset, code, and materials are available: https:// github.com/jsl5710/coi score concise fedility coverage ## method we propose chain-of-interactions (coi) for generating progressively refined dialogue summaries through structured multi-step reasoning chains. coi implements a sequential eight-chain process to produce summaries that balance accuracy, relevance, and utility for customer support agents while addressing information loss, repetition, hallucinations, and inconsistencies. we validate coi through extensive evaluation encompassing llms, prompting methods, standard automatic metrics, llmbased evaluations, and human assessment, ensuring robust evidence across diverse architectures and evaluation paradigms. ## datasets customer service summaries emphasize entities and factual details (e.g., reservation numbers, contact information) that differ from general conversation summaries. after reviewing open-source dialogue benchmarks @xcite , @xcite , icsi @xcite , dialogsum @xcite , csds @xcite ), only todsum @xcite suited our tods focus. we evaluate on: (1) todsum, an open-source benchmark with structured goal-directed conversations, and (2) crm, a propri- given an original dialogue (x), summarize this customer service dialogue. think step by step. given an original dialogue (x), extract user intent, issues/concerns, agent-response, and entities from customer service dialogue, and create an informative summary using this information. given 2 selected dialogue-summary pair examples [(a,b), (a,b)], learn from them to create a summary for a given original dialogue (x) using our extract-informcot prompt instruction. examplars' retrieval approach niave: random lexical-based: bm25 semantic: s_bert-cosine zero-shot extract-inform icl sumcot generic semantic reasoning few-shot ei-icl cod starting with an initial summary, 5 iterative steps adding 1-3 new entities without increasing length, achieved through rewriting techniques like compression and fusion. ## models our study spans 3 large llms and 9 small instruction-tuned models (7-8b parameters), balancing performance with computational efficiency for industry deployment ( ## experiment our experiment framework spans 11 models, 7 prompting approaches, 9 automatic metrics, 3 llm-based evaluations, and 480 human evaluators across 9 quality dimensions, providing comprehensive validation of coi's effectiveness. ## conclusion we introduced chain of interactions (coi), a novel single-instance abstractive summarization technique that progressively refines task oriented customer service dialogues summaries through structured multi-step reasoning chains. our evaluation demonstrates that coi significantly outperforms state-of-the-art approaches, excelling in information retention (6× more critical entities than cod) and quality dimensions (49% higher overall quality scores). human evaluations confirm chain 7 as optimal for customer support applications. this work advances dialogue summarization for practical customer service implementations while establishing a robust evaluation framework balancing nine critical dimensions.