FREDS um: A Dialogue Summarization Corpus for F rench Political Debates (2023)

Abstract: Recent advances in deep learning, and especially the invention of encoder-decoder architectures, have significantly improved the performance of abstractive summarization systems. While the majority of research has focused on written documents, we have observed an increasing interest in the summarization of dialogues and multi-party conversations over the past few years. In this paper, we present a dataset of French political debates for the purpose of enhancing resources for multi-lingual dialogue summarization. Our dataset consists of manually transcribed and annotated political debates, covering a range of topics and perspectives. We highlight the importance of high-quality transcription and annotations for training accurate and effective dialogue summarization models, and emphasize the need for multilingual resources to support dialogue summarization in non-English languages. We also provide baseline experiments using state-of-the-art methods, and encourage further research in this area to advance the field of dialogue summarization. Our dataset will be made publicly available for use by the research community, enabling further advances in multilingual dialogue summarization.

## introduction summarization aims at condensing text into a shorter version that optimally covers the main points of a document or conversation. extant models adopt one of two approaches: while extractive summarization aims at creating an optimal subset of the original source document by simply selecting the most important sentences without changing them, abstractive summarization aims at reformulating the most important points with novel sentences and words that may not have been seen before in the text. as such, abstractive summarization is a natural language generation task that relies more deeply on the meaning of the source text. a significant research effort has been dedicated to traditional document summarization @xcite , due in part to the fact that high quality data is readily available online, leading to the creation of even multi-lingual datasets for news summarization @xcite @xcite , and scientific articles @xcite . such data is much harder to come by for dialogue, although there are a few corpora of multiparty conversation and chat that have been painstakingly created with the aim of furthering research on abstractive meeting and conversation summarization. some examples include the ami @xcite , icsi @xcite and elitr @xcite corpora for meeting summarization @xcite and the samsum corpus @xcite for chat summarization (cf. @xcite , for more discussion.) however, due to the particular nature of dialogical interactions and even different types of dialogue, the relatively small size of conversation data sets compared to those for document summarization and other difficulties posed by spontaneous speech @xcite , dialogue summarization remains an open task. another significant problem for the task of dialogue summarization is the lack of multi-lingual data. while elitr @xcite contains some meetings in czech, and vcsum @xcite , meetings in chinese, there are currently no datasets available for multiparty conversation in other non-english languages. this imposes a significant limitation on the development of solutions for dialogue summarization in multilingual situations. there is thus a significant need for multilingual dialogue summarization resources and research. in this paper, we present fredsum, a new dataset for french political debate summarization and the first large-scale french multi-party summarization resource. fredsum is designed to serve as a valuable resource for furthering research on dialogue summarization in french and multilingual dialogue summarization more generally. each debate in the corpus is manually transcribed and annotated with not only abstractive and extractive summaries but also important topics which can be used as training and evaluation data for dialogue summarization models. the debates were selected with an eye to covering a range of political topics such as immigration, healthcare, foreign policy, security and education, and involve speakers with different perspectives, so as to not over-represent a party or an ideology. we highlight the importance of high quality transcription and annotations for training accurate and effective dialogue summarization models. we also present baseline experiments using state-of-the-art methods and encourage further research in this area to advance the field of dialogue summarization. our goal is to promote the development of multilingual dialogue summarization techniques that can be applied not only to political debates but also to other domains, enabling a broader audience to engage with important discussions in their native language. ## fredsum : french debate summarization dataset due to the heterogeneous nature of conversations, different dialogue summarization datasets serve different purposes. the uses and challenges inherent to chat data, film or meeting transcripts, twitter threads, and other kinds of daily communication are heterogeneous. differences may stem from transcript length, the level of disfluencies, the specific speech style of, say, chatrooms and twitter threads, the goal oriented manner in which a meet-ing is conducted, or even the downstream tasks for which a dataset is constructed. the data and the models used to treat can indeed be very diverse. debates are multi-party discussions that are generally long and fairly well-behaved both in the cleanness of the language used and their tight organization. most debates follow the same structure, an initial topic is given, followed by an assertion from a speaker, refutal by others with an alternative, and a further refutal by the original speaker. this systematic organization lends itself well to the study of discourse structure and dialogue dynamics. in this section, we aim at explaining the elements of annotations provided in fredsum as well as the instructions followed by the annotators. ## statistics fredsum is comprised of 18 debates segmented into 138 sub-debates by topic. each of the subtopics have two extractive summaries as well as three abstractive summaries. extractive summaries are naturally longer than their abstractive counterparts; extractive summaries contain an average of 684 tokens while abstractive summaries have an average of 239. ## human evaluation while rouge is still considered as the go-to metric for evaluating summaries, it has inherent limitations that makes it unreliable when used as the sole measure for the assessment of a summary's quality. indeed, rouge focuses on measuring lexical overlaps between reference summaries and generated summaries, lacking the ability to evaluate semantic similarity, or hallucinations. in this paper, we propose to conduct human analysis to be able to evaluate how well both rouge and bertscore scale with human judgment in the case of dialogue summarization, as well as to see how well summaries produced by newer llms score with respect to human evaluation. we asked the annotators to rate 50 debates from 1-5 in informativity, which requires the human to assess how well the summary captures the essence and context of the original debate; readability, which will further explore the preference of abstractive summaries people tend to have for dialogues; and faithfulness, which aims at evaluating how much of the generated summary is comprised of made up facts and hallucinations. human evaluation provides a complementary assessment to identify the strengths and weaknesses of these advanced models and compare their performance in terms of faithfulness, readability, and informativity. multiple points are brought out by the human evaluations. while human annotated summaries consistently rank the highest in all metrics, chat-gpt produces remarkably competitive summaries, particularly in terms of informativity and readability, while maintaining a high level of faithfulness. it is worth noting that, in the case of chatgpt, although some of the annotated summaries contained hallucinations, these hallucinations differed significantly from those generated by openassistant or barthez and were more similar to what human annotators would produce. these hallucinations typically resulted from either ambiguous speech in the original transcript or an attempt to capture the original tone of a speaker, such as mentioning irony when it does not exist. furthermore, although barthez appeared relatively competitive in terms of rouge and bertscore, its summaries were considered significantly worse when compared to human judgment. lastly, while extractive summaries may rank lower in readability due to the nature of dialogue speech, on average, they outperform abstractive summaries in both informativity and faithfulness. additionally, it is challenging to include hallucinated facts in extractive summaries, as doing so would require omitting negation clauses entirely. for tasks where faithfulness is crucial, extractive summaries continue to rank higher than abstractive ones. the higher informativity of extractive summaries is likely a result of their more extensive content. ## meta-evaluation of automatic metrics as we have mentioned previously, the automatic evaluation metrics used in summarization have limitations that prevent them from perfectly assessing the quality of a given model. in this subsection, we aim to determine the correlation between widely used metrics such as rouge-1, rouge-2, rouge-l, and bertscore, and human judgment. for the sake of comprehensiveness, we employ the grading scheme discussed in section 5, encompassing readability, informativity, and faithfulness. the results show that, as expected, rouge-1 performs worse of all available metrics across the board. on the other hand, both bertscore and rouge-l demonstrate noteworthy pearson correlations, particularly in terms of informativity and faithfulness. notably, bertscore consistently exhibits stronger correlations across all three aspects (readability, informativity, and faithfulness). ## usage of national assembly and senate data all the data provided in this paper follows the terms and conditions of their respective sources; national assembly, senate, following article l. 122-5 of the french intellectual property code, stipulating that there are no copyright for any of the articles, allowing people to copy them freely. the caveat being that the usage has to be either personal, associative or for professional uses, forbidding reproduction for commercial and advertisement purposes in this paper, we present the first version of fred-sum, a french dataset for abstractive debate summarization. the dataset consists of manually transcribed political debates, covering a range of topics and perspectives and annotated with topic segmentation, extractive and abstractive summaries and abstractive communities. we conduct extensive analyses on all the annotations and provide baseline results on the dataset by appealing to both automatic and human evaluation.