A reproduction study of methods for evaluating dialogue system output: Replicating Santhanam and Shaikh (2019) (2022)

Abstract: In this paper, we describe our reproduction ef- fort of the paper: Towards Best Experiment Design for Evaluating Dialogue System Output by Santhanam and Shaikh (2019) for the 2022 ReproGen shared task. We aim to produce the same results, using different human evaluators, and a different implementation of the automatic metrics used in the original paper. Although overall the study posed some challenges to re- produce (e.g. difficulties with reproduction of automatic metrics and statistics), in the end we did find that the results generally replicate the findings of Santhanam and Shaikh (2019) and seem to follow similar trends.

## introduction currently, a lot of attention is given to the reproducibility of nlp research. in this paper, we report our contributions to the 2022 reprogen shared task (belz et al., 2020). @xcite we aim at an exact reproduction of the work by santhanam and shaikh (2019) on experiment design for evaluating dialogue system output. no other reproductions of this paper have been published presently. we will first give a brief summary of the paper we aimed to reproduce ( §2), and explain how we replicated this research as closely to the original as possible ( §3). next, we will discuss our results and examine how these relate to the original study ( §4). lastly, we will discuss some difficulties we faced during our reproduction efforts ( §5). all of our code and data can be found on github. @xcite ## summary of the original study the study by santhanam and shaikh (2019) focuses on the design of the human evaluation task of evaluating dialogue system output. the purpose of the task is to see which task design yields the most consistent and highest-quality responses. the original study compared likert scale judgments, rank-based magnitude estimation (rme), biased magnitude estimation (bme) and best-worst scaling (bws) using the two metrics of readability and coherence. participants. the authors examined four different experimental conditions with 40 participants and 50 items each, yielding a total number of 160 participants. task. for each trial, participants were provided with a conversational context consisting of two turns. for each context, participants were asked to either rank or rate four different responses. three of these responses were generated by three models trained on the reddit conversation corpus (dziri et al., 2019a). the other response was humangenerated (i.e. the ground truth). in case of the likert scale, people rate a generated response on a 6-point scale (1 being lowest and 6 highest). for both rme and bme (magnitude estimation) they rate the responses with respect to a given standard value (bard et al., 1996). in the case of rme this value is always 100 while the value for bme is set by the automatic metrics of santhanam and shaikh (2019). in the last experiment design, bws, participants have to rank the responses from best to worst. reported values. santhanam and shaikh (2019) report on inter-rater consistency and agreement (intraclass correlations) and also examine if prior experience of rating dialogue system output or engaging with a conversational agent is of any influence. lastly, spearman correlations are reported between the human ratings and automatic metrics and between the ratings of readability and coherence on the four designs. results. overall, santhanam and shaikh (2019) find that the likert scale performs worst on intraclass correlation, show that participants without prior experience are more consistent in their ratings, and report low correlation between the automatic metrics and the human ratings.