D ialog S um Challenge: Results of the Dialogue Summarization Shared Task (2022)

Abstract: We report the results of DialogSum Challenge, the shared task on summarizing real-life sce- nario dialogues at INLG 2022. Four teams participate in this shared task and three submit their system reports, exploring different meth- ods to improve the performance of dialogue summarization. Although there is a great im- provement over the baseline models regarding automatic evaluation metrics, such as ROUGE scores, we find that there is a salient gap be- tween model generated outputs and human an- notated summaries by human evaluation from multiple aspects. These findings demonstrate the difficulty of dialogue summarization and suggest that more fine-grained evaluatuion met- rics are in need.

## introduction with the power of pretrained language models (plms), research on text summarization has made great progresses @xcite . however, previous research focuses on monologue summarization, such as news articles @xcite @xcite , patents @xcite and academic papers @xcite . however, as an important communicative channel @xcite , dialogues receive less attention from the community. to this end, we propose dialogsum challenge to encourage researchers to investigate different solutuons for real-life summarization @xcite . different from previous dialogue summarization tasks @xcite , dialogsum challenge focuses on diverse real-life scenarios such as schooling, work, medication, shopping, leisure, travel with large scale data. the challenges of dialogsum can be stated from three perspectives. first, dialogsum include a variety of topics, requiring models to process text with different real-life scenarios. second, compared with well-structured monologues, dialogues have unique discourse structures and language styles @xcite . the structures and use of languages differ from monologues, for instance, the key information spans over the context @xcite , which makes dialogues more difficult to encode. third, compared with monologue summarization, dialogue summaries are written from a different perspective, usually including speakers' intents and actions @xcite . therefore, dialogue summarization is abstractive in nature and requires a high level understanding beyond text semantics @xcite . figure 1 shows an example in dialogsum. apart from the research challenges, dialogsum challenge incentivizes summarization systems that can help end users. summarizing daily spoken dialogues can help archive the important information in business and personal communication. this presumably lifts the burden of manually taking notes, liberating human beings from the tedious work. responding to our calls, four teams participate in the shared task, and three of them submit their system reports. the submitted systems typically employ plms, such as bart @xcite and pegasus @xcite . in addition, they explore diverse methods to improve the performance, including integrating additional features, modifying the decoding process for better summary generation, multi-task tuning the model with auxiliary tasks and using data from other sources. to evaluate the performance, we construct a hidden test set that contains 100 manually annotated samples, and evaluate models on hidden and public test sets using both automatic and manual evaluations. for automatic evaluation, we use rouge scores and bertscore. for manual evaluation, we follow dialogsum challenge @xcite and evaluate model outputs from multiple aspects. results show that tuning models on cnn/daily news corpus @xcite or ami dataset @xcite , and incorporating topics in summary generation process can improve the model performance. however, there are still rooms for models to improve the metric scores, as well as the quality of the generated summaries in terms of identifying the interlocutors' intents, capturing the discourse relation, etc. besides, we observe the mismatch between bertscore and human scores, which aligns with the findings by @xcite . full details of the shared task description and logistics, as well as the dataset can be found at https://cylnlp.github.io/ dialogsum-challenge/ 2 task given a piece of dialogue text as input, the task is to ask a model to generate a summary that conveys the key information of the dialogue. the output summary should be concise, coherent, consistent and be written from a listener's perspective. manual evaluation furthermore, we conduct manual evaluation from various aspects, including standard summarization metrics @xcite , coreference information, intent identification, discourse relation following @xcite , as well as objectiveness (whether the summary is insusceptible to subjectivity such as subjective assumptions in the dialogues). besides, annotators give an overview score for the predicted summary. ## results in general, scores of the submissions are higher than the baseline models in the original dialog-sum paper, demonstrating the effort from the participants. both submissions from uot and iitp-cuni tune their models on other dataset then on dialogsum. in particular, uot tunes their model on cnn/daily news corpus, while iitp-cuni tunes their model on ami dataset. the reported results show that the model by uot outperforms the model by iitp-cuni. this might be attributed to the different in training size, where ami @xcite has 137 meetings, while cnn/daily news corpus @xcite has 312,000 articles. thus, the model tuned on cnn/daily news corpus might have better generalization ability. the model by tcs_witm which adopts such a method achieves 50.32 in rouge-1 score for the hidden test case, showing that generating the summary with the given topic can also help the performance. however, even the best-performed model underperforms humans by a margin larger than 5.0 in terms of all the rouge scores. this indicates that dialogsum is challenging and there is still a large room for future improvement. although existing works on summarization adopt bertscore @xcite , we observe that the bertscore deviates from the human scores. for instance, though goodbai achieves the best bertscore on the public test set, it is tcs_witm with a lower bertscore that achieves the best human scores (overview score in ## human analysis we randomly sample 50 examples for the public test set and 20 examples for the hidden test set to conduct manual analysis. as discussed in our proposal, we include the metrics of coreference information, discourse relation, objective description, intent identification, standard summarization metrics and overview scores. we annotate -1, 0, 1 for the metrics of coreference information, discourse relation, objective description, intent identification where 1 means all correct, 0 means partially correct and -1 means all incorrect. we annotate from ## conclusion we host dialogsum challenge of summarizing daily dialogue conversation at inlg 2021. our dataset possesses characteristics distinguished from the existing datasets and poses new challenge to the summarization community. there are 4 teams who submit their models during the challenges. an overview of their methods is provided in this report. we evaluate their predictions by automatic metrics and human analysis. results show that there are still rooms for models to improve the rouge scores, as well as the quality of the generated summaries in terms of identifying interlocutors' intents, capturing the discourse relation, etc. besides, we observe the mismatch between bertscore and human scores, which aligns with the findings by @xcite . therefore, we advocate to our community to explore automatic metrics that can better align with human scores. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020b. bertscore: evaluating text generation with bert. in 8th international conference on learning representations, iclr 2020, addis ababa, ethiopia, april 26-30, 2020. openreview.net. dialogue: #person1#: how long does it take to get to downtown from here? #person2#: it is 15 minutes ' drive. #person1#: what companies do we have in our neighborhood? #person2#: mitsubishi, hp, ibm and many other famous corporations. #person1#: does the 7th floor belong to our company too? #person2#: yes, our company has two floors, the 6th and 7th floor in this building. #person1#: it ' s such a large firm. do we have our own staff restaurant? #person2#: yes, at the end of the hall. gold: summary: #person2# tells #person1# information about their company and its surroundings. uot: prediction: #person2# asks #person2# about the distance to downtown and the companies in their neighborhood.