Evaluating Robustness of Open Dialogue Summarization Models in the Presence of Naturally Occurring Variations (2024)

Abstract: Dialogue summarization involves summarizing long conversations while preserving the most salient information. Real-life dialogues often involve naturally occurring variations (e.g., repetitions, hesitations). In this study, we systematically investigate the impact of such variations on state-of-the-art open dialogue summarization models whose details are publicly known (e.g., architectures, weights, and training corpora). To simulate real-life variations, we introduce two types of perturbations: utterance-level perturbations that modify individual utterances with errors and language variations, and dialogue-level perturbations that add non-informative exchanges (e.g., repetitions, greetings). We perform our analysis along three dimensions of robustness: consistency, saliency, and faithfulness, which aim to capture different aspects of performance of a summarization model. We find that both fine-tuned and instruction-tuned models are affected by input variations, with the latter being more susceptible, particularly to dialogue-level perturbations. We also validate our findings via human evaluation. Finally, we investigate whether the robustness of fine-tuned models can be improved by training them with a fraction of perturbed data. We find that this approach does not yield consistent performance gains, warranting further research. Overall, our work highlights robustness challenges in current open encoder-decoder summarization models and provides insights for future research.

## introduction real-life conversations often exhibit a wide range of language variations, including typographical errors, grammatical mistakes, and certain exchanges such as repetitions and speaker interruptions, which are unrelated to the primary purpose of the conversation @xcite . however, existing dialogue summarization datasets, which are used figure 1 : an example dialogue drawn from the tweet-sum dataset, with a repeated utterance introduced as a perturbation. while the reference summary for the original dialogue includes the agent's explanation about the train delay, the summary of the perturbed dialogue includes information from the repeated utterance. to train current summarization models, do not adequately capture these variations, as they are typically constructed by annotators simulating specific scenarios @xcite or extracted from english-speaking practice websites @xcite . even the datasets consisting of real-life conversations @xcite can exhibit only a limited range of variations owing to practical limitations posed by the data collection process (e.g., high or low prevalence of conversations from different social demographics). consequently, dialogue summarization models deployed in business scenarios encounter diverse variations not observed during training. this raises a crucial question: can current dialogue summarization models effectively handle conversations with naturally occurring variations that are legitimate inputs but not observed in the training data? in this work, we study the impact of naturally occurring variations on the performance of the state-of-the-art open dialogue summarization models (with publicly known architecture, weights, and training corpus) using three publicly available datasets. we examine the performance of encoder-decoder transformer models in two setups a) fine-tuned on specific dialogue summarization datasets @xcite @xcite , and b) instruction-tuned models which have shown impressive zero-shot performance more recently @xcite . such models are often preferred in high-stakes business settings (e.g., medical, legal, and customer support) over proprietary models (e.g., chatgpt), owing to user privacy concerns. to simulate variations we design two kinds of perturbations: (a) utterance-level perturbations, and (b) dialogue-level perturbations (defined in section 3), which are inspired by common real-life interaction patterns from the natural conversation framework @xcite . we evaluate the performance of summarization models along three conceptually distinct robustness dimensionsconsistency, saliency, and faithfulness-and elaborate on their empirical relationship. our analysis reveals that both fine-tuned and instruction-tuned models are impacted by utterance and dialogue-level perturbations. instruction-tuned models are impacted more than fine-tuned models and are also more susceptible to dialogue-level perturbations than utterance-level perturbations. both types of models show a preference for information from repeated, long, and leading utterances in the dialogue. figure 1 shows an example where the model includes repeated utterances in the summary, whereas the non-repeated original utterance wasn't included in the summary before perturbation. we also validate our findings via human evaluation. finally, we investigate whether fine-tuned models improve by training with perturbed data. we find that this approach does not consistently enhance performance, and different perturbations require varying amounts of training examples for gains. thus, further research is needed to address these robustness challenges. ## related work prior work has investigated the robustness of language understanding models mainly focusing on classification tasks @xcite . some dialogue-related classification tasks have also been explored, including dialogue act prediction @xcite , intent detection and slot tagging @xcite , state tracking and dialogue modeling @xcite @xcite @xcite . some studies have also investigated the robustness of neural language generation models, including neural machine translation @xcite @xcite , question answering @xcite , and open domain multi-document summarization @xcite . however, some of these studies consider perturbations that are of extreme nature (e.g., random shuffling and deletion of words) and may occur rarely in the real world. @xcite investigated the impact of natural variations on response prediction tasks in goal-oriented dialogues. for summarization task in particular, previous studies focused on summarizing news articles and documents @xcite @xcite . however, the nature of noise in a multi-party dialogue differs significantly from noise in documents. while some types of noise (e.g., spelling mistakes, grammatical errors) could occur in both, the patterns such as repetitions, reconfirmations, hesitations, and speaker interruptions @xcite @xcite are peculiar to dialogues, posing unique challenges for accurate and robust summarization. the focus of this work is to assess the robustness of dialogue summarization models in the presence of naturally occurring variations, which has been understudied in the prior literature. ## simulating naturally occurring variations to introduce naturally occurring variations in conversations, we consider two kinds of simulated perturbations, utterance-level and dialogue-level. we apply each perturbation individually to a dialogue to study its impact systematically. our perturbations are inspired by the natural conversation framework @xcite , created after analyzing real-world conversations across various use cases and provides common interactive patterns that occur in real life. foot_0 appendix a.2 lists examples for each perturbation. ## quantifying robustness for tasks involving text generation, such as dialogue summarization, measuring robustness involves determining the relationship between different pairs of natural language texts. as a result, the robustness of generative tasks is less well-defined, compared to a classification task @xcite and can manifest in several ways. we consider three dimensions for measuring robustness issues that can arise in dialogue summarization. let x denote the original dialogue, y r be the reference summary of the original dialogue, f be the summarization model trained on (x, y r ) ∼ d, and f (x) be its prediction over x. let x ′ = x + δ denote the perturbed dialogue and f (x ′ ) be its predicted summary. consistency a model is consistent (and hence robust) under a perturbation (δ) if the two summaries, f (x) and f (x ′ = x + δ), are semantically similar, resulting in minimal change. we quantify the change in model-generated output as follows, saliency assuming that the reference summary includes the most salient information conveyed in the input dialogue, we compute the change in salient information captured by the modelgenerated summaries (before and after perturbation) w.r.t the reference summary as follows: faithfulness faithfulness refers to the extent to which the generated summary is supported by the content of the input dialogue, thus accurately reflecting the information without introducing spurious or fabricated details, commonly termed as hallucinations. we compute the change in faithfulness as follows: ## evaluating robustness we present our key observations on how various perturbations impact the model performance. ## improving robustness one solution to address robustness issues could be to employ reverse heuristics to remove perturbations from dialogues. however, not all perturbations can be easily discovered and removed. for example, in repetition or time delay perturbations, the repeated utterance may include less information or be paraphrased compared to the original. while greetings and closing remarks might be simpler to remove, we include these perturbations as they offer a systematic approach to investigating model behavior, such as potential lead and recency biases. another potential solution to address robustness issues can be to use recent large language models to pre-process dialogues by removing errors and repetitions. however, this approach suffers from two challenges: (1) during deployment, additional preprocessing could increase latency, and (2) language models may hallucinate content, posing the risk of introducing factual errors in the input dialogue. finally, we examine if training with perturbations can help to mitigate robustness issues. we fine-tune bart on the training data augmented with perturbations and re-evaluate its performance. we create multiple training datasets, each modified by a specific kind of perturbation (typographical errors and language use variations for utterance level; repetitions, split utterances, and greetings for dialogue level), using tweetsum's training split. these modified datasets, with 5-50% of dialogues perturbed, are used to fine-tune bart, which we then test on a similarly altered tweetsum's test split. 8 we hypothesize that training with more perturbed dialogues will initially improve performance until a threshold, after which overfitting may reduce effectiveness. figure 3 shows the change in model consistency when fine-tuned with perturbations. the lower the change in consistency, the higher the model robustness to the perturbations. one takeaway is that different perturbations necessitate varying amounts of perturbed examples in the training set to achieve maximum performance gain. for example, typographical errors and language use variations yield the largest drop in ∆z c when approximately 40% and 45% of the dialogues are perturbed during training. in contrast, dialogue-level perturbations require significantly less perturbed data during training, with approximately 30% split-utterances, 15% greetings, and only 5% repetitions being sufficient. overall, the results demonstrate that fine-tuning 8 we experimented with training and evaluating a single model on data with all perturbations. however, since different perturbations can have varied impacts on model performance, we found perturbation-wise analysis more interpretable. with perturbed data does not yield consistent performance improvements, warranting more detailed exploration as part of future work. ## conclusion we investigate the impact of naturally occurring variations on state-of-the-art dialogue summarization models using three publicly available datasets. to simulate variations, we introduce utterance-level and dialogue-level perturbations. we conduct our analysis using three dimensions of robustness: consistency, saliency, and faithfulness, which capture different aspects of the summarization model's performance. our results show that both fine-tuned and instruction-tuned models are affected by perturbations, with instruction-tuned models being more susceptible, particularly to dialogue-level perturbations, spurring the need for future research.