Data Augmentation for Low-Resource Dialogue Summarization (2022)

Abstract: We present DADS, a novel Data Augmentation technique for low-resource Dialogue Summarization. Our method generates synthetic examples by replacing sections of text from both the input dialogue and summary while preserving the augmented summary to correspond to a viable summary for the augmented dialogue. We utilize pretrained language models that produce highly likely dialogue alternatives while still being free to generate diverse alternatives. We applied our data augmentation method to the SAMSum dataset in low resource scenarios, mimicking real world problems such as chat, thread, and meeting summarization where large scale supervised datasets with human-written summaries are scarce. Through both automatic and human evaluations, we show that DADS shows strong improvements for low resource scenarios while generating topically diverse summaries without introducing additional hallucinations to the summaries.

## introduction as many more language generation tasks are being explored, an outstanding issue is the lack of data available to train generation models. a question that follows is whether it is better to collect and annotate additional data in a particular domain or to generate synthetic data similar to the available data. considering the elevated cost of collecting data, expertise needed or the difficulty of finding the data, research on data augmentation is warranted. data augmentation (da) encompasses methods used to inject additional knowledge into learning systems without explicitly collecting new data; the knowledge injected comes in the form of additional training examples assumed to be silver standard than the collected gold data. in this paper, we propose an approach for data augmentation for dialogue summarization, aka dads, that creates semantically diverse synthetic examples from a low-resource dataset. our method modifies both the input dialogue and the target summary while preserving the augmented summary to correspond to a viable summary for the augmented dialogue. first, dads aligns pairs of utterances from the original dialogue to semantically similar sections in the summary; a large dialogue pretrained model, similar to @xcite , finetuned for dialogue reconstruction, is then used to replace the aligned utterances in the dialogue fabricating new dialogue. a new summary is then synthesized for the newly generated dialogue and the original summary, replacing the aligned sections in the summary using a stateof-the-art pretrained summarization model @xcite . models trained with dads augmented data produce significant performance gains in automated quality metrics for the samsum @xcite dialogue summarization dataset in lowresource settings, displaying 25% improvement in rouge when only 10 training examples are available. gains in performance are present in other low-resource settings, such as 50 and 100 examples, but decrease as one would expect as more data is available. as the data augmentation process is inherently noisy, we further investigate whether generation models augmented with dads are less faithful and analyze other aspects of language generation models such as diversity. our main contributions are as follows: (i) we introduce dads, a novel approach for data augmentation for dialogue summarization for low-resource scenarios. (ii) we demonstrate that models trained with dads augmented data are as faithful as models trained with the original data via human and automated faithfulness metrics. (iii) we found that the outputs generated by dads augmented models are more diverse than the strong baselines we compare against. ## related work there is extensive literature that explores da for machine learning systems in computer vision @xcite , natural language processing @xcite and other areas. in nlp approaches vary from general-purpose techniques that generate slightly modified copies of existing data; @xcite augment examples with noise directly in feature space rather than input space, to domain-specific transformations to create synthetic data, whereas @xcite use back-translation to augment text sequences. many methods aim to incorporate external knowledge or harness systems and domains where more data is available, e.g., large language models. recently, @xcite propose example extrapolation by training pretrained language models to extrapolate examples as a few-shot task. even though limited, research on data augmentation for language generation has had various approaches to data synthetization, such as corrupting the input text @xcite , the output text @xcite or both @xcite . notably, schick and schütze (2021) use pretrained language models and a diverse set of instructions to augment generation datasets in low-resource settings, rather than creating training examples. more recently, @xcite proposes an approach to augment dialogue systems when large amounts of in-domain data are available by training in-domain conversation generators. ## data augmentation we synthesize new training examples by augmenting both the dialogue and summary while ensuring that the generated summary is a good abstractive representation of the corresponding dialogue. the augmentation process is done in three steps: utterances-to-summary alignment, dialogue utterance replacement, and summary fillup. our workflow is shown in figure 1 and described below. utterances-to-summary alignment with the goal of transforming the (dialogue d, summary s) example pairs into a new training example (dialogue d , summary s ), great care has to be taken to avoid them diverging and losing the 'summaryof' relation between the pair. to accomplish this, dads keeps modifications limited to the aligned sections in the dialogue and summary. firstly, we align summary spans with utterances in the input. for samsum dataset, summaries are comprised of 1 to 2 sentences. we expanded the granularity of augmentations to a sub-sentence level by splitting each sentence into clauses using an off-the-shelf nlp pipeline annotator spacy @xcite . next, given the set of all summary clauses and dialogue utterances, we encode them into a shared space using the universal sentence encoder @xcite and computed their cosine similarity. for each clause in the summary, we select the top 20% utterances with the highest similarity scores as our input pairs for augmentation. one (utterances, clause) pair will generate one augmented example. ## results compared with the non-augmented baseline, which we call noaug, we find that models trained with data augmentation generate better quality summaries in terms of rouge (see ## conclusion we introduced dads, a new augmentation approach for dialogue summarization tasks. under 100 annotated examples, the improvement brought from augmentation is roughly equivalent to 10 more annotated examples. furthermore, we showed that dads generates semantically diverse synthetic examples. finally, through automatic and human evaluation, we showed that our augmentation approach does not introduce additional hallucinations to the summarization model. the methods described here are not particular to a type of dialogue summarization task and we leave for future research the application of similar methods to other dialogue summarization domains.