A Reproduction Study of the Human Evaluation of Role-Oriented Dialogue Summarization Models (2023)

Abstract: This paper reports a reproduction study of the human evaluation of role-oriented dialogue summarization models, as part of the ReproNLP Shared Task 2023 on Reproducibility of Evaluations in NLP. We outline the disparities between the original study’s experimental design and our reproduction study, along with the outcomes obtained. The inter-annotator agreement within the reproduction study is observed to be lower, measuring 0.40 as compared to the original study’s 0.48. Among the six conclusions drawn in the original study, four are validated in our reproduction study. We confirm the effectiveness of the proposed approach on the overall metric, albeit with slightly poorer relative performance compared to the original study. Furthermore, we raise an open-ended inquiry: how can subjective practices in the original study be identified and addressed when conducting reproduction studies?

## introduction reproducibility has gained significant attention within the field of natural language processing (nlp) in recent years. this paper presents a reproduction study focused on the human evaluation of role-oriented dialogue summarization models. the study was conducted as part of the repronlp shared task 2023, which aims to foster reproducibility in nlp evaluations. our participation in track c involved conducting a reproduction study specifically targeting the human evaluation component described in the work by @xcite , which is one of the five papers included in this track. the shared dataset used in this track originates from the reprohum project foot_0 , which employs a multi-lab paradigm to assess reproducibility in nlp. role-oriented dialogue summarization aims to generate summaries tailored to various roles within a conversation. for instance, in the context of a customer service chat, distinct summaries can be generated for the user's and the agent's utterances. the original research paper introduced an approach that leverages role interaction to effectively integrate the content of other roles into the summary pertaining to a specific role @xcite . the aforementioned study empirically demonstrated the effectiveness of the proposed approach in comparison to baseline methods through both automatic and human evaluations. in this study, we specifically concentrate on the human evaluation aspect. selection of evaluation samples. from the test set of csds, 100 dialogues were randomly chosen as evaluation samples. each dialogue is associated with two reference summaries, one for the user and one for the agent. a model also generated summaries for both the user and the agent. for each reference summary, four model-generated summaries (pgn-multi, bert-multi, pgn-both, and bert-both) were evaluated by human annotators. notably, the source dialogues were excluded from the human evaluation process. ## conclusion we present a reproduction study focused on evaluating dialogue summarization models through human evaluation. the successful execution of our reproduction experiment was facilitated by the collaboration with repronlp organizers and the utilization of materials provided by the original authors. as a result, we have drawn the following conclusions: one intriguing query that arises in the context of our reproduction study pertains to the identification and handling of subjective practices that may have been employed in the original study. specifically, we explore the different post-processing methods of annotation results from the initial 10 samples in this experiment. despite the limited impact of varying treatments on the ultimate outcome, the underlying concern persists. notably, if subjective practices are embedded within the core of the original experiment, the potential simulation of multiple possibilities can significantly amplify the scale of the experiment. this matter merits further investigation and remains an avenue for future research.