Original: Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents
Revised: Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: At the heart of improving conversational AI isthe open problem of how to evaluate conversations.
Revised: At the heart of improving conversational AI is the open problem of how to evaluate conversations.

Original: Issues with automatic metrics are wellknown (Liu et al., 2016), with human evaluations still considered the gold standard.
Revised: Issues with automatic metrics are well known (Liu et al., 2016), with human evaluations still considered the gold standard.

Original: Unfortunately, how to perform human evaluationsis also an open problem: differing data collection methods have varying levels of humanagreement and statistical sensitivity, resultingin differing amounts of human annotation hoursand labor costs.
Revised: Unfortunately, how to perform human evaluations is also an open problem: differing data collection methods have varying levels of human agreement and statistical sensitivity, resulting in differing amounts of human annotation hours and labor costs.

Original: In this work we compare fivedifferent crowdworker-based human evaluationmethods and find that different methods arebest depending on the types of models compared, with no clear winner across the board.
Revised: In this work we compare five different crowdworker-based human evaluation methods and find that different methods are best depending on the types of models compared, with no clear winner across the board.

Original: While this highlights the open problems in thearea, our analysis leads to advice of when touse which one, and possible future directions.
Revised: While this highlights the open problems in the area, our analysis leads to advice of when to use which one, and possible future directions.

Original:  Introduction
Revised:  Introduction

Original: Any comprehensive analysis of the performanceof an open-domain conversational model must include human evaluations: automatic metrics cancapture certain aspects of model performance butare no replacement for having human raters judgehow adept models are at realistic and interestingconversation (Deriu et al., 2021;
Revised: Any comprehensive analysis of the performance of an open-domain conversational model must include human evaluations: automatic metrics can capture certain aspects of model performance but are no replacement for having human raters judge how adept models are at realistic and interesting conversation (Deriu et al., 2021;

Original: Liu et al., 2016;
Revised: Liu et al., 2016;

Original: Dinan et al., 2019b).
Revised: Dinan et al., 2019b).

Original: Unfortunately, human evaluations themselves must be carefully constructed inorder to capture all the aspects desired of a goodconversationalist.
Revised: Unfortunately, human evaluations themselves must be carefully constructed in order to capture all the aspects desired of a good conversationalist.

Original: Any evaluation technique mustevaluate over many turns of a conversation in order to detect emergent faults such as repetitivenessor contradiction, while techniques that rely solelyon a single evaluation at the end of a conversationmay fail to take into account changes in model performance over its span.
Revised: Any evaluation technique must evaluate over many turns of a conversation in order to detect emergent faults such as repetitiveness or contradiction, while techniques that rely solely on a single evaluation at the end of a conversation may fail to take into account changes in model performance over its span.

Original: Further, techniques thatrate model performance on a Likert scale may suffer from inconsistencies in subjective numericalratings across evaluations of different models (Liet al., 2019).
Revised: Further, techniques that rate model performance on a Likert scale may suffer from inconsistencies in subjective numerical ratings across evaluations of different models (Li et al., 2019).

Original: When comparing various human evaluation methods to assess which works best, we findthat each has success and failure cases, leading us to conclude that human evaluation is still an open problem.
Revised: When comparing various human evaluation methods to assess which works best, we find that each has success and failure cases, leading us to conclude that human evaluation is still an open problem.

Original: In this work, we analyze a representative set of human evaluation techniques.
Revised: In this work, we analyze a representative set of human evaluation techniques.

Original: First, we compare per-turn evaluations, where ratings are given after every model response, and per-dialogue evaluations, where ratings are collected solely at the end of the conversation.
Revised: First, we compare per-turn evaluations, where ratings are given after every model response, and per-dialogue evaluations, where ratings are collected solely at the end of the conversation.

Original: Per-turn evaluations have the advantage of being more fine-grained, encouraging annotators to focus on small differences; however, the quality of a conversation is more than the sum of its parts, and global per-dialogue evaluations can capture this better.
Revised: Per-turn evaluations have the advantage of being more fine-grained, encouraging annotators to focus on small differences; however, the quality of a conversation is more than the sum of its parts, and global per-dialogue evaluations can capture this better.

Original: Second, we consider pairwise methods , where two models are compared directly by an annotator, to single-model methods, where the annotator sees and rates only one model at a time.
Revised: Second, we consider pairwise methods , where two models are compared directly by an annotator, to single-model methods, where the annotator sees and rates only one model at a time.

Original: Both approaches can be either per-turn or per-dialogue.
Revised: Both approaches can be either per-turn or per-dialogue.

Original: For example, in Pairwise Per-Turn evaluation, a crowdworker chats with a dialogue agent, and after each of the worker’s messages, they must choose between two possible responses from the agent, one from each of two different models.
Revised: For example, in Pairwise Per-Turn evaluation, a crowdworker chats with a dialogue agent, and after each of the worker’s messages, they must choose between two possible responses from the agent, one from each of two different models.

Original: The pairwise approach can spot subtle differences apparent when comparing responses, and it can mitigate problems with distribution shift that occur in absolute scoring.
Revised: The pairwise approach can spot subtle differences apparent when comparing responses, and it can mitigate problems with distribution shift that occur in absolute scoring.

Original: Single-model approaches, however, can work well when direct comparison is not paramount.
Revised: Single-model approaches, however, can work well when direct comparison is not paramount.

Original: We compare all of these different techniques for evaluating dialogue models in three different settings, and we contrast their individual strengths.
Revised: We compare all of these different techniques for evaluating dialogue models in three different settings, and we contrast their individual strengths.

Original: We find that:
Revised: We find that:

Original: • Pairwise per-turn evaluations are adept at mea- suring changes in model performance through- out a conversation.
Revised: • Pairwise per-turn evaluations are adept at measuring changes in model performance throughout a conversation.

Original: This technique tends to work well when pairs of models clearly differ in how appropriate their responses are in the context of the previous lines of dialogue, for example, when comparing two models that are trained on different datasets.• Pairwise per-dialogue evaluations tend to per- form best when differences between models only emerge after several conversation turns,
Revised: This technique tends to work well when pairs of models clearly differ in how appropriate their responses are in the context of the previous lines of dialogue, for example, when comparing two models that are trained on different datasets.

Original:  such as when these differences are very sub- tle, or when noticing patterns in responses thatemerge globally across the entire conversation, for example the average length of responses.• Single-model evaluations, performed both per
Revised: • Pairwise per-dialogue evaluations tend to perform best when differences between models only emerge after several conversation turns, such as when these differences are very subtle, or when noticing patterns in responses that emerge globally across the entire conversation, for example the average length of responses.

Original: conversation turn and at the end of a con-

Original: versation, tend to not do as well in the two

Original:  previously described settings, but do perform well when comparing models that differ only slightly in quality but are otherwise similar, for example two models with different num-
Revised: • Single-model evaluations, performed both per conversation turn and at the end of a conversation, tend to not do as well in the two previously described settings, but do perform well when comparing models that differ only slightly in quality but are otherwise similar, for example two models with different numbers of parameters.

Original: bers of parameters.

Original: These findings, while highlighting the difficultyof human evaluation, also provide guidance onwhich method might be best to use in these different circumstances, as well as possible futurework.
Revised: These findings, while highlighting the difficulty of human evaluation, also provide guidance on which method might be best to use in these different circumstances, as well as possible future work.

Original: In particular, investigating the best way tomerge pairwise and single-model, per-turn and perdialogue benefits into a single method could be afruitful direction.
Revised: In particular, investigating the best way to merge pairwise and single-model, per-turn and per-dialogue benefits into a single method could be a fruitful direction.

Original: We also analyze the interpretability of these approaches when collecting humanwritten explanations
Revised: We also analyze the interpretability of these approaches when collecting human written explanations.

Original: We have released code for these evaluation techniques.
Revised: We have released code for these evaluation techniques in the ParlAI framework.

Original:  Existing work
Revised:  Existing work

Original: This work concerns itself with evaluation of opendomain dialogue, which, unlike more restricteddomains such as question-answering and goaloriented conversations, may not have a precise goal,and no widely accepted evaluation technique forit currently exists (Deriu et al., 2021; Huang et al. ,2020; Roller et al., 2020).
Revised: This work concerns itself with evaluation of opendomain dialogue, which, unlike more restricted domains such as question-answering and goaloriented conversations, may not have a precise goal, and no widely accepted evaluation technique for it currently exists (Deriu et al., 2021; Huang et al., 2020; Roller et al., 2020).

Original: Automatic metrics arerelatively fast, efficient, and reproducible, but manyof them have been shown to “correlate very weaklywith human judgement” (Liu et al., 2016, see also
Revised: Automatic metrics are relatively fast, efficient, and reproducible, but many of them have been shown to “correlate very weakly with human judgement” (Liu et al., 2016, see also

Original: Dinan et al.
Revised: Dinan et al.

Original: (2019b)), and the best way to create areliable automatic metric is still up for debate (Deriu et al., 2021).
Revised: (2019b)), and the best way to create a reliable automatic metric is still up for debate (Deriu et al., 2021).

Original: In this work we focus on humanevaluation, and in particular on employing crowdworkers, which has an advantage over utilizingtrained experts (Deriu et al., 2021) or deployment(Gabriel et al., 2020; Shuster et al., 2020) of allowing for a larger pool of evaluators and for ensuringalignment with research goals, respectively.
Revised: In this work we focus on human evaluation, and in particular on employing crowdworkers, which has an advantage over utilizing trained experts (Deriu et al., 2021) or deployment (Gabriel et al., 2020; Shuster et al., 2020) of allowing for a larger pool of evaluators and for ensuring alignment with research goals, respectively.

Original: However, the use of crowdworkers itself has a numberof pitfalls to avoid as well (Huynh et al., 2021).
Revised: However, the use of crowdworkers itself has a number of pitfalls to avoid as well (Huynh et al., 2021).

Original: Particular instruction wording choices to crowdworkers have a large effect on the quality of conversations and resulting evaluations Huynh et al.
Revised: Particular instruction wording choices to crowdworkers have a large effect on the quality of conversations and resulting evaluations (Huynh et al., 2021).

Original: Wording can direct workers to evaluate specific facets, such as general “get to know each other” chitchat (Zhang et al., 2018), getting a bot to generate unsafe utterances (Xu et al., 2020), and instructing crowdworkers to be adversarial vs. not (Dinan et al., 2019a).
Revised: Wording can direct workers to evaluate specific facets, such as general “get to know each other” chitchat (Zhang et al., 2018), getting a bot to generate unsafe utterances (Xu et al., 2020), and instructing crowdworkers to be adversarial vs. not (Dinan et al., 2019a).

Original: One can also pick from a variety of specific questions when asking crowdworkers to rate conversations, including asking about interestingness, making sense, fluency (See et al., 2019), sensibleness, specificity (Adiwardana et al., 2020), toxicity, and bias (Xu et al., 2020), and the exact phrasing of these questions can have a large impact on sensitivity (Li et al., 2019).
Revised: One can also pick from a variety of specific questions when asking crowdworkers to rate conversations, including asking about interestingness, making sense, fluency (See et al., 2019), sensibleness, specificity (Adiwardana et al., 2020), toxicity, and bias (Xu et al., 2020), and the exact phrasing of these questions can have a large impact on sensitivity (Li et al., 2019).

Original: Standard evaluation protocols have a single human both converse with a model and rate that conversation in the same task, but other methods have a rater rate pre-existing conversations between a human and a model or between a pair of models (Li et al., 2019; Deriu et al., 2020).
Revised: Standard evaluation protocols have a single human both converse with a model and rate that conversation in the same task, but other methods have a rater rate pre-existing conversations between a human and a model or between a pair of models (Li et al., 2019; Deriu et al., 2020).

Original: These latter techniques allow for efficient reuse of existing conversational data, and have shown to be useful experimentally (Li et al., 2019; Roller et al., 2021), but it may be harder for evaluators to rate conversations that they have not been involved in.
Revised: These latter techniques allow for efficient reuse of existing conversational data, and have shown to be useful experimentally (Li et al., 2019; Roller et al., 2021), but it may be harder for evaluators to rate conversations that they have not been involved in.

Original: Another choice when designing evaluation protocols is whether conversations are rated individually, e.g., with Likert-score ratings (Ashwin et al., 2017;
Revised: Another choice when designing evaluation protocols is whether conversations are rated individually, e.g., with Likert-score ratings (Ashwin et al., 2017;

Original: Venkatesh et al., 2018, see more in Appendix A), or pairwise by comparing models (Li et al., 2019;
Revised: Venkatesh et al., 2018, see more in Appendix A), or pairwise by comparing models (Li et al., 2019;

Original: Liang et al., 2020, etc.).
Revised: Liang et al., 2020, etc.).

Original: Likert scoring suffers from weaknesses such as potential per-annotator bias (Kulikov et al., 2019) and drift in the distribution of errors over time (See et al., 2019), but is more efficient than pairwise comparisons in that new models’ ratings can be compared to those of older models without having to re-collect those older models’ ratings.
Revised: Likert scoring suffers from weaknesses such as potential per-annotator bias (Kulikov et al., 2019) and drift in the distribution of errors over time (See et al., 2019), but is more efficient than pairwise comparisons in that new models’ ratings can be compared to those of older models without having to re-collect those older models’ ratings.

Original: Lastly, evaluation techniques differ in whether they collect ratings on each turn of the conversation (Adiwardana et al., 2020; Komeili et al., 2021) or only at the end of the conversation, as in AcuteEval (Li et al., 2019).
Revised: Lastly, evaluation techniques differ in whether they collect ratings on each turn of the conversation (Adiwardana et al., 2020; Komeili et al., 2021) or only at the end of the conversation, as in AcuteEval (Li et al., 2019).

Original: Whole-conversation techniques can work well if the quality of a conversation is assumed to be more than just the sum of its parts, but could perhaps suffer due to the primacy effect and recency effect that appear when more weight is given to information presented at the start and end of the rating session, respectively (Asch,
Revised: Whole-conversation techniques can work well if the quality of a conversation is assumed to be more than just the sum of its parts, but could perhaps suffer due to the primacy effect and recency effect that appear when more weight is given to information presented at the start and end of the rating session, respectively (Asch,

Original: 1946; Anderson, 1965; Murdock Jr, 1962; Postmanand Phillips, 1965).
Revised: 1946; Anderson, 1965; Murdock Jr, 1962; Postman and Phillips, 1965).

Original: See Appendix A for a more thorough assessmentof related works.
Revised: See Appendix A for a more thorough assessment of related works.

Original:  Methods 
Revised:  Methods 3. Evaluation techniques

Original: 3. Evaluation techniques

Original: We investigate several human evaluation techniques, spanning a cross-section of the differentmethods discussed in existing work.
Revised: We investigate several human evaluation techniques, spanning a cross-section of the different methods discussed in existing work.

Original: Specifically:
Revised: Specifically:

Original: • Single-model per-turn evaluations • Single-model per-dialogue evaluations • Pairwise per-turn evaluations • Pairwise per-dialogue evaluations 
Revised: • Single-model per-turn evaluations • Single-model per-dialogue evaluations • Pairwise per-turn evaluations • Pairwise per-dialogue evaluations • Pairwise per-dialogue self-chat evaluations

Original: • Pairwise per-dialogue self-chat evaluations

Original: We thus compare the spectrum of single vs. pairwise and per-turn vs. per-dialogue variations, aswell as trying a self-chat method compared to conventional human-bot conversation ratings.
Revised: We thus compare the spectrum of single vs. pairwise and per-turn vs. per-dialogue variations, as well as trying a self-chat method compared to conventional human-bot conversation ratings.

Original: Figure1 summarizes the methods.
Revised: Figure 1 summarizes the methods.

Original: In the following, wewill describe our exact methodology for each.
Revised: In the following, we will describe our exact methodology for each.

Original: See
Revised: See

Original: Appendix B.3 for details on quality checks usedwhen performing these evaluations.
Revised: Appendix B.3 for details on quality checks used when performing these evaluations.

Original: 3.1. Conversational setting
Revised: 3.1. Conversational setting

Original: Our human-bot evaluations consist of a set of conversations.
Revised: Our human-bot evaluations consist of a set of conversations.

Original: Each conversation consists of a crowdworker (the “Human Speaker”) paired with a conversational model (the “Bot Speaker”).
Revised: Each conversation consists of a human worker crowdsourced from Amazon Mechanical Turk 2 (the “Human Speaker”) paired with a conversational model (the “Bot Speaker”).

Original: The Human
Revised: The Human

Original: Speaker will speak naturally in the conversation,and they will be role-playing as a certain personawith the help of two provided persona sentencesgiven to them at the start of the conversation: see
Revised: Speaker will speak naturally in the conversation, and they will be role-playing as a certain persona with the help of two provided persona sentences given to them at the start of the conversation: see

Original: Figure 2 (left) for an example.
Revised: Figure 2 (left) for an example.

Original: The Human Speaker’s first message in the conversation is fixed to “Hi!”
Revised: The Human Speaker’s first message in the conversation is fixed to “Hi!”

Original: , following the convention of Adiwardana et al.
Revised: , following the convention of Adiwardana et al.

Original: The conversationends after the Human Speaker and Bot Speakerhave both spoken for 6 turns each.
Revised: The conversation ends after the Human Speaker and Bot Speaker have both spoken for 6 turns each, to roughly match the conversation lengths used for BlenderBot evaluations in Roller et al.

Original: We test three different evaluation metrics, preference , humannessand interestingness , with exact wordings describedin the following subsections.
Revised: We test three different evaluation metrics, preference , humanness and interestingness , with exact wordings described in the following subsections.

Original: 3.1. Pairwise per-turn evaluations
Revised: Pairwise per-turn evaluations

Original: The Pairwise Per-Turn evaluation (PW-Turn) technique provides annotations for every turn of conversation by asking for the crowdworker to choosefrom a pair of model responses after every sentmessage.
Revised: The Pairwise Per-Turn evaluation (PW-Turn) technique provides annotations for every turn of conversation by asking for the crowdworker to choose from a pair of model responses after every sent message.

Original: Hence, in this setting the Human Speaker speaks to a Bot Speaker, the latter of which actually represents the two models to be compared.
Revised: Hence, in this setting the Human Speaker speaks to a Bot Speaker, the latter of which actually represents the two models to be compared.

Original: The Human Speaker will speak naturally in the conversation.
Revised: The Human Speaker will speak naturally in the conversation.

Original: Every time that it is the Bot Speaker’s turn to speak, the crowdworker will first be presented with two options as possible responses: each response will come from one of the two models being compared, similarly to Clark and Smith (2021).
Revised: Every time that it is the Bot Speaker’s turn to speak, the crowdworker will first be presented with two options as possible responses: each response will come from one of the two models being compared, similarly to Clark and Smith (2021).

Original: We randomize the ordering of these model responses.
Revised: We randomize the ordering of these model responses.

Original: The worker must choose the better response for the given evaluation metric.
Revised: The worker must choose the better response for the given evaluation metric.

Original: The wordings we use for the three metrics are adapted from Li et al.
Revised: The wordings we use for the three metrics are adapted from Li et al.

Original: • Preference : “Which next response from your partner would you prefer in a long conversa-
Revised: • Preference : “Which next response from your partner would you prefer in a long conversation?”

Original: tion?” • Humanness : “Which next response from your partner sounds more human?”
Revised: • Humanness : “Which next response from your partner sounds more human?”

Original: • Interestingness : “If you had to say one of these responses is interesting and one is bor- ing, which would you say is more interest- ing?”
Revised: • Interestingness : “If you had to say one of these responses is interesting and one is boring, which would you say is more interesting?”

Original: The worker must give a free-text justification for their choice of response.
Revised: The worker must give a free-text justification for their choice of response.

Original: The response that they choose is set to be the actual response given by the
Revised: The response that they choose is set to be the actual response given by the

Original: Bot Speaker, and the conversation continues from there.
Revised: Bot Speaker, and the conversation continues from there.

Original: Figure 2 provides a screenshot example of the UI.
Revised: Figure 2 provides a screenshot example of the UI.

Original: A description of quality checks performed when onboarding workers for this evaluation technique is given in Appendix B.2.
Revised: A description of quality checks performed when onboarding workers for this evaluation technique is given in Appendix B.2.

Original: In our experiments we consider win rates based on simply averaging over turns, as well as nonlinear combinations of per-turn results over entire dialogues (e.g., winnertakes-all voting) in order to measure their impact.
Revised: In our experiments we consider win rates based on simply averaging over turns, as well as nonlinear combinations of per-turn results over entire dialogues (e.g., winnertakes-all voting) in order to measure their impact.

Original: 3.1. Pairwise per-dialogue evaluations
Revised: 3.1. Pairwise per-dialogue evaluations

Original: The Pairwise Per-Dialogue evaluation (PW-Dialog) technique we introduce asks evaluators to choose between two models by presenting a pair of conversations.
Revised: The Pairwise Per-Dialogue evaluation (PW-Dialog) technique we introduce asks evaluators to choose between two models by presenting a pair of conversations.

Original: The technique we employ is identical to the Acute-Eval method (Li et al., 2019), but for consistency with the names of other techniques, we refer to it here as PW-Dialog evaluations. 
Revised: The technique we employ is identical to the Acute-Eval method (Li et al., 2019), but for consistency with the names of other techniques, we refer to it here as PW-Dialog evaluations. For each of the model pairs and evaluation metrics used, we OR

Original: For each ofthe model pairs and evaluation metrics used, we collect evaluations on (1) conversations conducted between a crowdworker and a model agent; and (2) self-chat conversations conducted between two conversational agents of the same model (the selfchat variant).
Revised: OR OR OR or or Out of collect evaluations on (1) conversations conducted between a crowdworker and a model agent; and (2) self-chat conversations conducted between two conversational agents of the same model (the selfchat variant).

Original: The wordings we use (from Li et al.
Revised: The wordings we use (from Li et al.

Original: OR

Original: OR OR OR or or Out of (2019)) are almost identical to the PW-Turn versions than per 
Revised: (2019)) are almost identical to the PW-Turn versions, but phrased for the per-dialogue, rather than per case

Original: • Preference : “Who would you prefer to talk to for a long conversation?”
Revised: • Preference : “Who would you prefer to talk to for a long conversation?”

Original: • Humanness : “Which speaker sounds more human?”
Revised: • Humanness : “Which speaker sounds more human?”

Original: • Interestingness : “If you had to say one of these speakers is interesting and one is boring, who would you say is more interesting?”
Revised: • Interestingness : “If you had to say one of these speakers is interesting and one is boring, who would you say is more interesting?”

Original: Figure 3 provides a screenshot example of the
Revised: Figure 3 provides a screenshot example of the

Original: UI.
Revised: UI.

Original: 3.1. Single-model evaluations
Revised: 3.1. Single-model evaluations

Original: In our single-model evaluation experiments, wecombine per-turn and per-dialogue into the same
Revised: In our single-model evaluation experiments, we combine per-turn and per-dialogue into the same

Original: UI (see Figure 4 for a screenshot).
Revised: UI (see Figure 4 for a screenshot).

Original: 2 A crowdworkerchats with a conversational agent backed by a single model, and for each response of that modelthe worker must annotate whether it is engaging,human-like, and/or interesting, with wording provided in the screenshot.
Revised: A crowdworker chats with a conversational agent backed by a single model, and for each response of that model the worker must annotate whether it is engaging, human-like, and/or interesting, with wording provided in the screenshot.

Original: At the end of the conversation, again consisting of 6 messages per speaker,the worker must rate their partner on a Likert scale of 1 to 5 for each of the three evaluation metrics listed in Section 3.1.2.
Revised: At the end of the conversation, again consisting of 6 messages per speaker, the worker must rate their partner on a Likert scale of 1 to 5 for each of the three evaluation metrics listed in Section 3.1.2.

Original: We refer to the per-turn annotations of model responses from this task as
Revised: We refer to the per-turn annotations of model responses from this task as

Original: Single-Model Per-Turn evaluations (SM-Turn) and the end-of-conversation Likert scores as SingleModel Per-Dialogue evaluations (SM-Dialog).
Revised: Single-Model Per-Turn evaluations (SM-Turn) and the end-of-conversation Likert scores as SingleModel Per-Dialogue evaluations (SM-Dialog).

Original: Empirically, we find that SM-Turn success rates and SM-Dialog Likert scores are highly dependent on the particular day that the evaluations are collected: this is perhaps due to day-to-day variability in the pool of crowdworkers.
Revised: Empirically, we find that SM-Turn success rates and SM-Dialog Likert scores are highly dependent on the particular day that the evaluations are collected: this is perhaps due to day-to-day variability in the pool of crowdworkers.

Original: To counteract this, we run these evaluations on all four of the models discussed in this work (Section 3.2) simultaneously.
Revised: To counteract this, we run these evaluations on all four of the models discussed in this work (Section 3.2) simultaneously.

Original: 3. Models
Revised: 3. Models

Original: We analyze the relative performance of these five human evaluation techniques, SM-Turn, SMDialog, PW-Turn, PW-Dialog, and PW-Dialog selfchat , on four different well-performing but relatively similar dialogue models from Roller et al.
Revised: We analyze the relative performance of these five human evaluation techniques, SM-Turn, SMDialog, PW-Turn, PW-Dialog, and PW-Dialog selfchat , on four different well-performing but relatively similar dialogue models from Roller et al.

Original: • BlenderBot3B :The version of BlenderBot
Revised: • BlenderBot3B :

Original:  with 2.7 billion parameters, pretrained on a previously existing Reddit dataset (extracted and obtained by a third party and made available on pushshift.io (Baumgartner et al., 
Revised: The version of BlenderBot with 2.7 billion parameters, pretrained on a previously existing Reddit dataset (extracted and obtained by a third party and made available on pushshift.io (Baumgartner et al., 2020)) and then fine-tuned on several purposebuilt dialogue datasets.

Original: and then fine-tuned on several purposebuilt dialogue datasets.

Original: • BlenderBot3B-M0 : BlenderBot3B uses aminimum generation length of 20 tokens to ensure relatively long, interesting responses.
Revised: • BlenderBot3B-M0 : BlenderBot3B uses a minimum generation length of 20 tokens to ensure relatively long, interesting responses.

Original: We also compare to exactly the same modelbut without a minimum generation length, referring to it with -M0 postfix.
Revised: We also compare to exactly the same model but without a minimum generation length, referring to it with -M0 postfix.

Original: • BlenderBot90M : The variant of BlenderBotwith 90 million parameters, trained on thesame datasets as BlenderBot3B. • Reddit3B : BlenderBot3B, but only pretrainedon the third-party Reddit dump and not finetuned on dialogue datasets.
Revised: • BlenderBot90M : The variant of BlenderBot with 90 million parameters, trained on the same datasets as BlenderBot3B. • Reddit3B : BlenderBot3B, but only pretrained on the third-party Reddit dump and not finetuned on dialogue datasets.

Original: For all models, we use the same generation settings as in Roller et al.
Revised: For all models, we use the same generation settings as in Roller et al.

Original: (2021), apart from the -M0adaptation.
Revised: (2021), apart from theM0 adaptation.

Original: We choose these relatively similar models in our experiments as a difficult challenge forevaluation techniques to tell which one is best.
Revised: We choose these relatively similar models in our experiments as a difficult challenge for evaluation techniques to tell which one is best, but we a priori expect from previous Acute-Eval (PW-Dialog) self-chat measurements in Roller et al.

Revised: (2021) that BlenderBot3B may perform as well as or better than the other three models.

Original: For the three pairwise evaluation techniques, wespecifically perform comparisons between threepairs of models, each of which differ in a characteristic way:
Revised: For the three pairwise evaluation techniques, we specifically perform comparisons between three pairs of models, each of which differ in a characteristic way:

Original: • Length comparison : Comparing Blender- Bot3B to BlenderBot3B-M0: these models differ only in the length of their generations.• Size comparison : Comparing two models
Revised: • Length comparison : Comparing BlenderBot3B to BlenderBot3B-M0: these models differ only in the length of their generations.

Original: with different numbers of parameters, Blender-

Original: Bot3B and BlenderBot90M. • Fine-tuning comparison : Comparing the fine-tuned BlenderBot3B to the pretrained- only Reddit3B (both with the same number of 
Revised: • Size comparison : Comparing two models with different numbers of parameters, BlenderBot3B and BlenderBot90M. • Fine-tuning comparison : Comparing the fine-tuned BlenderBot3B to the pretrainedonly Reddit3B (both with the same number of parameters)

Original: parameters).  Results  Model win rates from pairwise per- turn
Revised:  Results 4. Model win rates from pairwise per-turn evaluations

Original: evaluations We compute the win rates of BlenderBot3B overother models in Table 1 for the pairwise evaluation technique PW-Turn.
Revised: We compute the win rates of BlenderBot3B over other models in Table 1 for the pairwise evalu- [Table] ation technique PW-Turn.

Original: We expect BlenderBot3B to be better, hence values closer to 100% aredeemed more preferable.
Revised: We expect BlenderBot3B to be better, hence values closer to 100% are deemed more preferable.

Original: We display the win ratesof four different variants: including all 6 conversation turns from the Bot Speaker, excluding the Bot
Revised: We display the win rates of four different variants: including all 6 conversation turns from the Bot Speaker, excluding the Bot

Original: [Table]

Original: Speaker’s first turn from the evaluations, and computing a nonlinear function of the turns: either calculating squared or winner-takes-all win rates for each conversation and then averaging those scores across all conversations.
Revised: Speaker’s first turn from the evaluations, and computing a nonlinear function of the turns: either calculating squared or winner-takes-all win rates for each conversation and then averaging those scores across all conversations.

Original: We generally find that PWTurn win rates are higher when dropping the first turn of the Bot Speaker, as discussed further in Appendix C.2.
Revised: We generally find that PWTurn win rates are higher when dropping the first turn of the Bot Speaker, as discussed further in Appendix C.2.

Original: Win rates are typically even higher by aggregating over conversations in a winner-takesall fashion, which has the effect of reducing the turn-by-turn variability of which model’s response is chosen by the crowdworker.
Revised: Win rates are typically even higher by aggregating over conversations in a winner-takesall fashion, which has the effect of reducing the turn-by-turn variability of which model’s response is chosen by the crowdworker.

Original: We find that, in general, win rates of BlenderBot3B do not vary much as a function of the evaluation question used when asking workers to choose one model response over the other.
Revised: We find that, in general, win rates of BlenderBot3B do not vary much as a function of the evaluation question used when asking workers to choose one model response over the other.

Original: It is unclear a priori whether this results from an ambiguity in the precise definitions of these questions/metrics when interpreted by the workers, correlations in how well models perform on some metrics vs. others, or some other reason.[Table]
Revised: It is unclear a priori whether this results from an ambiguity in the precise definitions of these questions/metrics when interpreted by the workers, correlations in how well models perform on some metrics vs. others, or some other reason.

Original: 4. Model scores from single-model 
Revised: 4. Model scores from single-model evaluations

Original: evaluations

Original:  Table 2 provides the per-turn success rates (SMTurn) and end-of-conversation Likert scores (SMDialog) over all models.
Revised: [Table] Table 2 provides the per-turn success rates (SMTurn) and end-of-conversation Likert scores (SMDialog) over all models.

Original: As with the pairwise evaluations of Section 4.1 and Roller et al.
Revised: As with the pairwise evaluations of Section 4.1 and Roller et al.

Original: BlenderBot3B generally outperforms the othermodels using the SM-Turn and SM-Dialog methods as well.
Revised: BlenderBot3B generally outperforms the other models using the SM-Turn and SM-Dialog methods as well.

Original: Table 5 (in the Appendix) shows success rates from the SM-Turn technique as a function of conversation turn (rather than aggregated).
Revised: Table 5 (in the Appendix) shows success rates from the SM-Turn technique as a function of conversation turn (rather than aggregated).

Original: BlenderBot3B scores are generally stable acrossconversation turn but are slightly lower on the firsttwo turns of the Bot Speaker, echoing similar findings with PW-Turn in Appendix C.2.
Revised: BlenderBot3B scores are generally stable across conversation turn but are slightly lower on the first two turns of the Bot Speaker, echoing similar findings with PW-Turn in Appendix C.2.

Original: We thus alsoconsider removing SM-Turn scores from the firsttwo turns in order to maximize the performance of
Revised: We thus also consider removing SM-Turn scores from the first two turns in order to maximize the performance of

Original: BlenderBot3B relative to the other models.
Revised: BlenderBot3B relative to the other models.

Original: As with
Revised: As with

Original: PW-Turn, we find that calculating the winner-takesall score per conversation allows for an even biggerseparation in performance between BlenderBot3Band the other models.
Revised: PW-Turn, we find that calculating the winner-takesall score per conversation allows for an even bigger separation in performance between BlenderBot3B and the other models.

Original: Unlike PW-Turn for which win rates are similaracross all three evaluation metrics (Section 4.1),single-model success rates on the Interestingness metric are generally lower than those on the other two, especially for SM-Turn.
Revised: Unlike PW-Turn for which win rates are similar across all three evaluation metrics (Section 4.1), single-model success rates on the Interestingness metric are generally lower than those on the other two, especially for SM-Turn.

Original: We hypothesize that the juxtaposition of all three evaluation questions side-by-side in the UI of the SM-Turn and SMDialog crowdworker task (Figure 4) may aid workers in distinguishing among these three metrics and rating models differently on them.
Revised: We hypothesize that the juxtaposition of all three evaluation questions side-by-side in the UI of the SM-Turn and SMDialog crowdworker task (Figure 4) may aid workers in distinguishing among these three metrics and rating models differently on them.

Original: See Appendix C.4 for an exploration of which turns of the conversation contribute most strongly to workers’ final Likert-scale ratings.
Revised: See Appendix C.4 for an exploration of which turns of the conversation contribute most strongly to workers’ final Likert-scale ratings.

Original: 4. Direct comparison of all evaluation 
Revised: 4. Direct comparison of all evaluation techniques

Original: techniques

Original: In this section we directly compare all the pairwise and single-model evaluation techniques to each other to discern their relative strengths.
Revised: In this section we directly compare all the pairwise and single-model evaluation techniques to each other to discern their relative strengths.

Original: See Appendix C.1 for details on the number of evaluations performed and number of crowdworker hours spent per technique.
Revised: See Appendix C.1 for details on the number of evaluations performed and number of crowdworker hours spent per technique.

Original: 4.3. Computing win rates across all 
Revised: 4.3. Computing win rates across all techniques

Original: techniques

Original: In order to directly compare the performance of
Revised: In order to directly compare the performance of

Original: SM-Turn and SM-Dialog with that of the pairwise techniques, we calculate effective win rates for the two single-model techniques by bootstrapping samples of ratings from different models and then calculating how often SM-Turn success rates and SMDialog Likert scores from one model are higher than those of another.
Revised: SM-Turn and SM-Dialog with that of the pairwise techniques, we calculate effective win rates for the two single-model techniques by bootstrapping samples of ratings from different models and then calculating how often SM-Turn success rates and SMDialog Likert scores from one model are higher than those of another.

Original: Following the analysis of best performing methods from Sections 4.1 and 4.2, we consider only Bot Speaker turns 2 through 6 for
Revised: Following the analysis of best performing methods from Sections 4.1 and 4.2, we consider only Bot Speaker turns 2 through 6 for

Original: PW-Turn and turns 3 through 6 for SM-Turn in winner-takes-all (WTA) mode, in order to maximize the ability of these techniques to distinguish different models’ performances.
Revised: PW-Turn and turns 3 through 6 for SM-Turn in winner-takes-all (WTA) mode, in order to maximize the ability of these techniques to distinguish different models’ performances.

Original: Table 3 compares the win rates produced by all evaluation techniques.
Revised: Table 3 compares the win rates produced by all evaluation techniques.

Original: Overall, we find that a different technique performs best for each of the three model comparison types:
Revised: Overall, we find that a different technique performs best for each of the three model comparison types:

Original: [Table]
Revised: Length comparison

Original: The pairwise evaluation techniques PW-Dialog and PW-Turn perform much better than the single-model ones.
Revised: The pairwise evaluation techniques PW-Dialog and PW-Turn perform much better than the single-model ones.

Original: BlenderBot3B responses tend to contain many more words on average than those of BlenderBot3B-M0, and so we hypothesize that this difference in sensitivity among the techniques may be due to the fact thatviewing responses from both models side-by-side makes the length differences between them much more evident, especially when comparing two en-
Revised: BlenderBot3B responses tend to contain many more words on average than those of BlenderBot3B-M0, and so we hypothesize that this difference in sensitivity among the techniques may be due to the fact that

Original: [Table]
Revised: [Table]

Original: tire conversations as in PW-Dialog.
Revised: viewing responses from both models side-by-side makes the length differences between them much more evident, especially when comparing two entire conversations as in PW-Dialog.

Original: Thus, if crowdworkers tend to prefer longer responses on average, the side-by-side comparison of model responses might aid in their ability to choose BlenderBot3B responses over those of BlenderBot3B-M0.
Revised: Thus, if crowdworkers tend to prefer longer responses on average, the side-by-side comparison of model responses might aid in their ability to choose BlenderBot3B responses over those of BlenderBot3B-M0.

Original: Size comparison
Revised: Size comparison

Original: The differences among thetechniques here are smaller than for the Lengthcomparison, with the full-dialogue techniques PWDialog and SM-Dialog slightly outperforming theper-turn ones.
Revised: The differences among the techniques here are smaller than for the Length comparison, with the full-dialogue techniques PWDialog and SM-Dialog slightly outperforming the per-turn ones.

Original: As shown by Roller et al.
Revised: As shown by Roller et al.

Original: BlenderBot3B and BlenderBot90M do not performstatistically significantly differently on Acute-Evals(i.e. PW-Dialog) on self-chat conversations.
Revised: BlenderBot3B and BlenderBot90M do not perform statistically significantly differently on Acute-Evals (i.e. PW-Dialog) on self-chat conversations.

Original: Thus,it may make sense that any small differences in performance between these models are more evidenton the level of whole conversations.
Revised: Thus, it may make sense that any small differences in performance between these models are more evident on the level of whole conversations.

Original: Fine-tuning comparison
Revised: Fine-tuning comparison

Original: In this comparison,
Revised: In this comparison,

Original: PW-Turn performs best out of all techniques.
Revised: PW-Turn performs best out of all techniques.

Original: Because the Reddit3B model was not fine-tuned onconversational dialogue datasets, its responses toits partner generally make less sense in context thanthose of BlenderBot3B. We hypothesize that thesemore nonsensical responses may be very obviousto workers who are in the middle of having a conversation with the Bot Speaker during the PW-Turnevaluation.
Revised: Because the Reddit3B model was not fine-tuned on conversational dialogue datasets, its responses to its partner generally make less sense in context than those of BlenderBot3B. We hypothesize that these more nonsensical responses may be very obvious to workers who are in the middle of having a conversation with the Bot Speaker during the PW-Turn evaluation.

Original: However, these responses may be lessobvious to workers reading whole conversations inthe PW-Dialog evaluation who have not interactedwith the models directly, as well as to workers in
Revised: However, these responses may be less obvious to workers reading whole conversations in the PW-Dialog evaluation who have not interacted with the models directly, as well as to workers in

Original: SM-Turn and SM-Dialog evaluations who cannotdirectly compare Reddit3B responses to those of amodel that has been fine-tuned on dialogue.
Revised: SM-Turn and SM-Dialog evaluations who cannot directly compare Reddit3B responses to those of a model that has been fine-tuned on dialogue.

Original: Explainability in experiments : analysis of 
Revised: Explainability in experiments: analysis of crowdworker reasons

Original: During the crowdworker evaluation tasks, we also ask for reasons for the crowdworker’s judgments.
Revised: During the crowdworker evaluation tasks, we also ask for reasons for the crowdworker’s judgments.

Original: These reasons can give interpretability to the results.
Revised: These reasons can give interpretability to the results.

Original: A full analysis is given in Appendix C.5.
Revised: A full analysis is given in Appendix C.5.

Original: Overall, we find justifications that make sense in each of the three model comparisons, e.g. in the Length comparison we see keywords like “information” and “detailed” appearing often.
Revised: Overall, we find justifications that make sense in each of the three model comparisons, e.g. in the Length comparison we see keywords like “information” and “detailed” appearing often.

Original: For the Fine-tuning comparison, we often find keywords like “flows” , “personal” and “contradicts” , which implies that the fine-tuning conversational datasets like Persona-Chat provide for more personal, less contradictory, and flowing conversations.
Revised: For the Fine-tuning comparison, we often find keywords like “flows” , “personal” and “contradicts” , which implies that the fine-tuning conversational datasets like Persona-Chat provide for more personal, less contradictory, and flowing conversations.

Original: Repeatability of experiments
Revised: Repeatability of experiments

Original: We provide an analysis in Appendix C.6 of the variability of model win rates over time for each of the evaluation techniques.
Revised: We provide an analysis in Appendix C.6 of the variability of model win rates over time for each of the evaluation techniques.

Original: Overall, we find that PW-Turn, PW-Dialog, and SM-Turn vary least across chunked experiments, with SM-Dialog having more variability.
Revised: Overall, we find that PW-Turn, PW-Dialog, and SM-Turn vary least across chunked experiments, with SM-Dialog having more variability.

Original: This makes the use of SM-Dialog less compelling.
Revised: This makes the use of SM-Dialog less compelling.

Original: 4.3. Overall findings
Revised: 4.3. Overall findings

Original: The results of these three model comparisons hint that perhaps a per-turn evaluation technique may be more suitable for pairs of models that differ in their ability to reply sensibly in a way that is easily detectable by their partner (i.e. BlenderBot3B vs.
Revised: The results of these three model comparisons hint that perhaps a per-turn evaluation technique may be more suitable for pairs of models that differ in their ability to reply sensibly in a way that is easily detectable by their partner (i.e. BlenderBot3B vs.

Original: Reddit3B), but that a whole-conversation technique may be preferable when differences between models are more sensitive.
Revised: Reddit3B), but that a whole-conversation technique may be preferable when differences between models are more sensitive.

Original: However, evaluations onmany more pairs of models would be needed to sufficiently support such a broad hypothesis.
Revised: However, evaluations on many more pairs of models would be needed to sufficiently support such a broad hypothesis.

Original: We alsofind that single-model techniques perform competitively to pairwise ones except for when modelgenerations differ by average length: in this case,comparing the responses of both models side-byside may make the differences between them moreapparent than just viewing them separately.
Revised: We also find that single-model techniques perform competitively to pairwise ones except for when model generations differ by average length: in this case, comparing the responses of both models side-byside may make the differences between them more apparent than just viewing them separately.

Original: Combining techniques
Revised: Combining techniques

Original: Given how much therelative sensitivities of different evaluation techniques vary across different pairs of models, wealso explore whether combining results from multiple techniques together may allow for a compromise technique that performs reasonably well inall cases.
Revised: Given how much the relative sensitivities of different evaluation techniques vary across different pairs of models, we also explore whether combining results from multiple techniques together may allow for a compromise technique that performs reasonably well in all cases.

Original: We thus include in Table 3 the win rate(“PW combo”) when sampling ratings from the
Revised: We thus include in Table 3 the win rate (“PW combo”) when sampling ratings from the

Original: PW-Turn and PW-Dialog techniques together ata ratio of 1:5.
Revised: PW-Turn and PW-Dialog techniques together at a ratio of 1:5.

Original: This sampling retains most of theability of PW-Dialog to quickly compare BlenderBot3B to BlenderBot3B-M0 and BlenderBot90M(the Length and Size comparisons), and it alsogains some of PW-Turn’s superior strength at measuring the performance of BlenderBot3B over Reddit3B (the Fine-tuning comparison).
Revised: This sampling retains most of the ability of PW-Dialog to quickly compare BlenderBot3B to BlenderBot3B-M0 and BlenderBot90M (the Length and Size comparisons), and it also gains some of PW-Turn’s superior strength at measuring the performance of BlenderBot3B over Reddit3B (the Fine-tuning comparison).

Original: By contrast, since ratings for the two singlemodel techniques SM-Turn and SM-Dialog arecollected simultaneously, ratings from both techniques on a given conversation can be averagedtogether to achieve slightly finer sensitivity than either technique individually.
Revised: By contrast, since ratings for the two singlemodel techniques SM-Turn and SM-Dialog are collected simultaneously, ratings from both techniques on a given conversation can be averaged together to achieve slightly finer sensitivity than either technique individually.

Original: Figures 11, 12, and 13show that, with the proper weighting, such averaging can produce a statistically significant difference between models a bit faster than with only
Revised: Figures 11, 12, andshow that, with the proper weighting, such averaging can produce a statistically significant difference between models a bit faster than with only

Original: SM-Dialog and dramatically faster than with only
Revised: SM-Dialog and dramatically faster than with only

Original: SM-Turn (Appendix C.8).
Revised: SM-Turn (Appendix C.8).

Original: Beyond win rates, another way to directly compare the relative usefulness of our various evaluation techniques is to estimate the amount of personhours that must be spent on evaluations by crowdworkers in order to achieve a statistically significant result.
Revised: Beyond win rates, another way to directly compare the relative usefulness of our various evaluation techniques is to estimate the amount of personhours that must be spent on evaluations by crowdworkers in order to achieve a statistically significant result.

Original: These results (Figures 8, 9, and 10)roughly follow the patterns found by win rates (Section 4.3.1).
Revised: These results (Figures 8, 9, and 10) roughly follow the patterns found by win rates (Section 4.3.1).

Original: See Appendix C.7 for a discussion ofthe assumptions made when producing these timeestimates.
Revised: See Appendix C.7 for a discussion of the assumptions made when producing these time estimates.

Original:  Conclusion
Revised:  Conclusion

Original: In this work we compare the extent to which different evaluation techniques are able to measureperformance differences between dialogue models, and we show instances in which the performance varies between per-turn techniques and perdialogue techniques, and between pairwise techniques and single-model techniques.
Revised: In this work we compare the extent to which different evaluation techniques are able to measure performance differences between dialogue models, and we show instances in which the performance varies between per-turn techniques and perdialogue techniques, and between pairwise techniques and single-model techniques.

Original: A completely exhaustive analysis of the cases in which each technique is most appropriate would require measurement on many more pairs of models than the three studied here, and would likely require a dramatic scaling-up of labor for crowdworkers.
Revised: A completely exhaustive analysis of the cases in which each technique is most appropriate would require measurement on many more pairs of models than the three studied here, and would likely require a dramatic scaling-up of labor for crowdworkers.

Original: Nevertheless, the results shown here demonstrate the difficulty in anointing one evaluation technique as superior to all others regardless of the models being compared, and they suggest that a combination of techniques, or else a different technique entirely, may be necessary for optimal measurement of differences among models.
Revised: Nevertheless, the results shown here demonstrate the difficulty in anointing one evaluation technique as superior to all others regardless of the models being compared, and they suggest that a combination of techniques, or else a different technique entirely, may be necessary for optimal measurement of differences among models.

Original: A more universally ideal technique would likely need to investigate model performance per-turn but still be able to give an overall judgment of model quality across a conversation in order to capture elements of performance that manifest clearest in a single response vs. in aggregate.
Revised: A more universally ideal technique would likely need to investigate model performance per-turn but still be able to give an overall judgment of model quality across a conversation in order to capture elements of performance that manifest clearest in a single response vs. in aggregate.

Original: We demonstrate that combining evaluation scores from per-turn and per-dialogue techniques can bridge the gap in the performance differences between the two, but that this does not outperform either individual technique in all cases, at least in the way that we combined them.
Revised: We demonstrate that combining evaluation scores from per-turn and per-dialogue techniques can bridge the gap in the performance differences between the two, but that this does not outperform either individual technique in all cases, at least in the way that we combined them.

Original: Future improvements may also come from exploring other ways to amplify the weak signal from models with only slight performance differences such as BlenderBot3B and BlenderBot90M, perhaps by training workers to select responses based on general measures of conversational quality, as opposed to content that appeals to their personal interests.
Revised: Future improvements may also come from exploring other ways to amplify the weak signal from models with only slight performance differences such as BlenderBot3B and BlenderBot90M, perhaps by training workers to select responses based on general measures of conversational quality, as opposed to content that appeals to their personal interests.

Original: Improving sensitivity to roughly equivalent pairs of models such as these should in turn enable the comparison of models whose performance differences are smaller still.
Revised: Improving sensitivity to roughly equivalent pairs of models such as these should in turn enable the comparison of models whose performance differences are smaller still.

Original: While this work has concentrated on evaluating techniques that enable differentiability (one can differentiate between models) with efficiency (with less annotator hours), there are other desirable qualities as well.
Revised: While this work has concentrated on evaluating techniques that enable differentiability (one can differentiate between models) with efficiency (with less annotator hours), there are other desirable qualities as well.

Original: Some of these in particular are diversity of conversations (Hashimoto et al., 2019), repeatability of experiments, and explainability of results (Deriu et al., 2021).
Revised: Some of these in particular are diversity of conversations (Hashimoto et al., 2019), repeatability of experiments, and explainability of results (Deriu et al., 2021).

Original: While there is some discussion of the latter two topics in our experiments, these topics are fully deserving of a more thorough analysis than is provided here.
Revised: While there is some discussion of the latter two topics in our experiments, these topics are fully deserving of a more thorough analysis than is provided here

Original: A Additional existing work 
Revised: A Additional existing work Open-domain versus specific domain

Original: Open-domain versus specific domain

Original: Our workconcentrates on the open-domain setting.
Revised: Our work concentrates on the open-domain setting.

Original: In specific conversational domains, such as question answering (QA), evaluation can be simpler and isoften reduced to measuring overlap or exact matchwith the correct answer (Chen et al., 2019).
Revised: In specific conversational domains, such as question answering (QA), evaluation can be simpler and is often reduced to measuring overlap or exact match with the correct answer (Chen et al., 2019).

Original: However, this no longer as easily suffices for free-form,conversational and long-form QA where answersare more open-ended (Fan et al., 2019; Adolphset al., 2021).
Revised: However, this no longer as easily suffices for free-form, conversational and long-form QA where answers are more open-ended (Fan et al., 2019; Adolphs et al., 2021).

Original: Similarly, for certain types of goaloriented conversations more targeted evaluationscan take place, for example evaluation of statetracking (Williams et al., 2016) and task completion (Hastie, 2012; Henderson et al., 2014; Bordeset al., 2017; El Asri et al., 2017; Wen et al., 2017).
Revised: Similarly, for certain types of goaloriented conversations more targeted evaluations can take place, for example evaluation of state tracking (Williams et al., 2016), interaction quality (Schmitt and Ultes, 2015), and task completion (Hastie, 2012; Henderson et al., 2014; Bordes et al., 2017; El Asri et al., 2017; Wen et al., 2017).

Original: Open-domain dialogue potentially covers all these other cases as special cases, but also covers conversations that are more free-form or do not have a precise goal.
Revised: Opendomain dialogue potentially covers all these other cases as special cases, but also covers conversations that are more free-form or do not have a precise goal.

Original: Hence, finding a reliable evaluation technique is more difficult, and there is currently no single standard method that is agreed upon (Deriu et al., 2021; Huang et al., 2020; Roller et al., 2020).
Revised: Hence, finding a reliable evaluation technique is more difficult, and there is currently no single standard method that is agreed upon (Deriu et al., 2021; Huang et al., 2020; Roller et al., 2020).

Original: Different techniques that have been proposed will be described in the following paragraphs.
Revised: Different techniques that have been proposed will be described in the following paragraphs.

Original: Automatic metrics
Revised: Automatic metrics

Original: Automatic metrics are the most convenient for fast, efficient and reproducible research with a quick turn-around and development cycle, hence they are frequently used.
Revised: Automatic metrics are the most convenient for fast, efficient and reproducible research with a quick turn-around and development cycle, hence they are frequently used.

Original: Unfortunately, many of them, such as BLEU, METEOR and ROUGE, have been shown to only “correlate very weakly with human judgement” (Liu et al., 2016).
Revised: Unfortunately, many of them, such as BLEU, METEOR and ROUGE, have been shown to only “correlate very weakly with human judgement” (Liu et al., 2016).

Original: A central problem is that, due to the openended nature of conversations, there are many possible responses in a given dialogue, and, while having multiple references can help, there is typically only one gold label available (Gupta et al., 2019).
Revised: A central problem is that, due to the openended nature of conversations, there are many possible responses in a given dialogue, and, while having multiple references can help, there is typically only one gold label available (Gupta et al., 2019).

Original: Perplexity (computing the predicted probability of the given gold utterances) has been argued to correlate with human judgments (Adiwardana et al., 2020), however this has also been shown to not always be the case (Dinan et al., 2019b), and moreover does not actually evaluate the generations themselves produced by a decoder architecture.
Revised: Perplexity (computing the predicted probability of the given gold utterances) has been argued to correlate with human judgments (Adiwardana et al., 2020), however this has also been shown to not always be the case (Dinan et al., 2019b), and moreover does not actually evaluate the generations themselves produced by a decoder architecture.

Original: Hence, changing the behavior of the generation method can dramatically change human evaluations, while maintaining identical or nearidentical perplexity (See et al., 2019; Welleck et al., 2020; Adiwardana et al., 2020; Roller et al., 2021).
Revised: Hence, changing the behavior of the generation method can dramatically change human evaluations, while maintaining identical or nearidentical perplexity (See et al., 2019; Welleck et al., 2020; Adiwardana et al., 2020; Roller et al., 2021).

Original: An alternative recent trend is to employ trainable metrics, whereby a neural network model is used to score the conversational model (typically also another neural network), see e.g. Lowe et al.
Revised: An alternative recent trend is to employ trainable metrics, whereby a neural network model is used to score the conversational model (typically also another neural network), see e.g. Lowe et al.

Original: Ghandeharioun et al.
Revised: Ghandeharioun et al.

Original: Such systems provide a promise of improved speed of research and development of dialogue agents, but so far have not been met with wide adoption.
Revised: Such systems provide a promise of improved speed of research and development of dialogue agents, but so far have not been met with wide adoption.

Original: Some issues are that they may not generalize as well to data beyond that which they are trained (overfit) and also may be biased and gameable (Wu et al., 2019; Albrecht and Hwa, 2007).
Revised: Some issues are that they may not generalize as well to data beyond that which they are trained (overfit) and also may be biased and gameable (Wu et al., 2019; Albrecht and Hwa, 2007).

Original: For a comprehensive comparison of automatic metrics – both standard and learned metrics – see Yeh et al.
Revised: For a comprehensive comparison of automatic metrics – both standard and learned metrics – see Yeh et al.

Original: In general, creating a reliable automatic metric is still considered an open problem (Deriu et al., 2021).
Revised: In general, creating a reliable automatic metric is still considered an open problem (Deriu et al., 2021).

Original: Crowdworkers versus experts versus organic users
Revised: Crowdworkers versus experts versus organic users

Original: While utilizing human evaluations in research is the current standard, we contend thatchoosing exactly which kind of human evaluationis also still an open question.
Revised: While utilizing human evaluations in research is the current standard, we contend that choosing exactly which kind of human evaluation is also still an open question.

Original: In this work weconcentrate on the study of crowdworker humanevaluations, however there are several alternativeparadigms.
Revised: In this work we concentrate on the study of crowdworker human evaluations, however there are several alternative paradigms.

Original: Utilizing trained experts, such as agroup of researchers in the same institution, is onealternative (Deriu et al., 2021).
Revised: Utilizing trained experts, such as a group of researchers in the same institution, is one alternative (Deriu et al., 2021).

Original: Compared to employing crowdworkers, while model comparisonresults can agree between the two types of annotators, there can be vastly different sensitivity andwin rates (Welleck et al., 2020), with the expertshaving more agreement and higher resulting sensitivity.
Revised: Compared to employing crowdworkers, while model comparison results can agree between the two types of annotators, there can be vastly different sensitivity and win rates (Welleck et al., 2020), with the experts having more agreement and higher resulting sensitivity.

Original: On the other hand, it is harder to recruitand employ experts, limiting reproducibility.
Revised: On the other hand, it is harder to recruit and employ experts, limiting reproducibility.

Original: Inboth the crowdworker and expert annotator cases,neither of those groups are necessarily the intendedtarget audience of a given system.
Revised: In both the crowdworker and expert annotator cases, neither of those groups are necessarily the intended target audience of a given system.

Original: If it is possibleto deploy a model to people who genuinely wantto talk to it (e.g., without being paid), conversations may be more natural and evaluations will bein line with genuine interests.
Revised: If it is possible to deploy a model to people who genuinely want to talk to it (e.g., without being paid), conversations may be more natural and evaluations will be in line with genuine interests.

Original: Evaluation by deployment can be successful (Gabriel et al., 2020;
Revised: Evaluation by deployment can be successful (Gabriel et al., 2020;

Original: Shuster et al., 2020), where behavioral metrics suchas the amount of conversation time per user or retention rate can serve as a proxy for interestingnessand engagingness metrics.
Revised: Shuster et al., 2020), where behavioral metrics such as the amount of conversation time per user or retention rate can serve as a proxy for interestingness and engagingness metrics.

Original: Model deployment however also has its issues.
Revised: Model deployment however also has its issues.

Original: First, user desires maynot necessarily be aligned with the goals of theresearch itself, meaning researchers may have todevelop features and improvements towards thegoals of the product rather than towards long-termresearch.
Revised: First, user desires may not necessarily be aligned with the goals of the research itself, meaning researchers may have to develop features and improvements towards the goals of the product rather than towards long-term research.

Original: Further, experiments are difficult to set upand may be difficult to reproduce by other groups.
Revised: Further, experiments are difficult to set up and may be difficult to reproduce by other groups.

Original: Crowdworker tasks can be more reproducible especially when code is made available to reproduceexperiments, but there are also many pitfalls whenconstructing the tasks, see e.g. Huynh et al.
Revised: Crowdworker tasks can be more reproducible especially when code is made available to reproduce experiments, but there are also many pitfalls when constructing the tasks, see e.g. Huynh et al.

Original: Conversation instructions to raters
Revised: Conversation instructions to raters

Original: When utilizing evaluators in a evaluator-model conversational setup, the precise instructions on how to goabout the conversation will clearly have large effects.
Revised: When utilizing evaluators in a evaluator-model conversational setup, the precise instructions on how to go about the conversation will clearly have large effects.

Original: Such instructions can control the topic, e.g.“get to know each other” as in the Persona-Chattask (Zhang et al., 2018), versus “have a knowledgeable conversation” in Wizard of Wikipedia(Dinan et al., 2018).
Revised: Such instructions can control the topic, e.g. “get to know each other” as in the Persona-Chat task (Zhang et al., 2018), versus “have a knowledgeable conversation” in Wizard of Wikipedia (Dinan et al., 2018).

Original: Instructions can also orientworkers towards a more fruitful strategy for a desired dataset, for example orienting them towardsopen questions on sensitive topics rather than profanity to get a bot to generate unsafe utterances (Xu et al., 2020).
Revised: Instructions can also orient workers towards a more fruitful strategy for a desired dataset, for example orienting them towards open questions on sensitive topics rather than profanity to get a bot to generate unsafe utterances (Xu et al., 2020).

Original: The length of the conversation will also play a role in the performance of models, for example, short conversations do not test the ability of models to retain knowledge in the long-term (Xu et al., 2021).
Revised: The length of the conversation will also play a role in the performance of models, for example, short conversations do not test the ability of models to retain knowledge in the long-term (Xu et al., 2021).

Original: Overall, the style of conversation has large effects (even if the topic is unchanged) for example when instructing crowdworkers to be adversarial vs. non-adversarial (Dinan et al., 2019a), which relates to the classic Turing Test (Turing and Haugeland, 1950).
Revised: Overall, the style of conversation has large effects (even if the topic is unchanged) for example when instructing crowdworkers to be adversarial vs. non-adversarial (Dinan et al., 2019a), which relates to the classic Turing Test (Turing and Haugeland, 1950).

Original: Further, particular instruction wording choices will change the quality of conversations, as they will change how well crowdworkers understand the task (Huynh et al., 2021).
Revised: Further, particular instruction wording choices will change the quality of conversations, as they will change how well crowdworkers understand the task (Huynh et al., 2021).

Original: Evaluation question phrasing for raters
Revised: Evaluation question phrasing for raters

Original: Besides how the conversation is carried out, one also needs to choose the precise question (or questions) being asked to crowdworkers in order for them to rate conversations.
Revised: Besides how the conversation is carried out, one also needs to choose the precise question (or questions) being asked to crowdworkers in order for them to rate conversations.

Original: In open-domain conversation there are a variety of qualities one could expect from a good conversationalist, and potentially one could ask about any of them individually, as well as asking for overall performance.
Revised: In open-domain conversation there are a variety of qualities one could expect from a good conversationalist, and potentially one could ask about any of them individually, as well as asking for overall performance.

Original: For example, See et al.
Revised: For example, See et al.

Original: (2019) asks evaluators for ratings of interestingness, making sense, fluency, avoiding repetition, listening ability and inquisitiveness as intermediate conversational aspects, and humanness and engagingness questions to measure overall quality.
Revised: (2019) asks evaluators for ratings of interestingness, making sense, fluency, avoiding repetition, listening ability and inquisitiveness as intermediate conversational aspects, and humanness and engagingness questions to measure overall quality.

Original: Adiwardana et al.
Revised: Adiwardana et al.

Original: (2020) asks questions based on sensibleness and specificity.
Revised: (2020) asks questions based on sensibleness and specificity.

Original: Responsibility, toxicity and bias can also be measured (Xu et al., 2020).
Revised: Responsibility, toxicity and bias can also be measured (Xu et al., 2020).

Original: Even after settling on the exact question(s) to be asked, their exact phrasing also has impact on sensitivity, as shown in Li et al.
Revised: Even after settling on the exact question(s) to be asked, their exact phrasing also has impact on sensitivity, as shown in Li et al.

Original: In that work, the authors optimized the question phrasing by running evaluations with alternative phrasings, and choosing the one with the highest agreement.
Revised: In that work, the authors optimized the question phrasing by running evaluations with alternative phrasings, and choosing the one with the highest agreement.

Original: Rating existing versus own conversations
Revised: Rating existing versus own conversations

Original: The standard setup is for a human to have a conversation with a model, and rate that conversation.
Revised: The standard setup is for a human to have a conversation with a model, and rate that conversation.

Original: Some evaluation protocols deviate from this setup, and ask evaluators to rate conversations they did not participate in.
Revised: Some evaluation protocols deviate from this setup, and ask evaluators to rate conversations they did not participate in.

Original: One simple approach of that kind is to present model completions of a dialogue from the fixed test set of a given task, and ask for their evaluation, with hence no human taking part in the actual conversation (Vinyals and Le, 2015; Li et al., 2016).
Revised: One simple approach of that kind is to present model completions of a dialogue from the fixed test set of a given task, and ask for their evaluation, with hence no human taking part in the actual conversation (Vinyals and Le, 2015; Li et al., 2016).

Original: In the Acute-Eval method (Li et al., 2019) raters are asked to compare two existing con-  versation logs, and the authors consider both thecase of human-model chat logs, and model-model(self-chat) logs, where the former are actually adifferent set of human conversationalists comparedto the final raters.
Revised: In the Acute-Eval method (Li et al., 2019) raters are asked to compare two existing conversation logs, and the authors consider both the case of human-model chat logs, and model-model (self-chat) logs, where the former are actually a different set of human conversationalists compared to the final raters.

Original: Deriu et al.
Revised: Deriu et al.

Original: (2020) considerschat logs between pairs of models, again with nohumans taking part in the conversations.
Revised: (2020) considers chat logs between pairs of models, again with no humans taking part in the conversations.

Original: Thesetechniques allow efficient reuse of existing conversational data and have some reproducibility gains:conversations collected in previous trials and byother systems can be directly compared with a newsystem, without having to recollect additional data.
Revised: These techniques allow efficient reuse of existing conversational data and have some reproducibility gains: conversations collected in previous trials and by other systems can be directly compared with a new system, without having to recollect additional data.

Original: This can significantly reduce the resources neededby a new evaluation, and ensure that multiple papers are comparing to prior work consistently.
Revised: This can significantly reduce the resources needed by a new evaluation, and ensure that multiple papers are comparing to prior work consistently.

Original: Onthe other hand, it may be harder for evaluators torate conversations that they have not been involvedin.
Revised: On the other hand, it may be harder for evaluators to rate conversations that they have not been involved in (Finch and Choi, 2020).

Original: Conversations that do not even involve humansshould be treated with some scepticism, as there isno human to guide conversation and hence evaluateinteractive quality.
Revised: Conversations that do not even involve humans should be treated with some scepticism, as there is no human to guide conversation and hence evaluate interactive quality.

Original: Nevertheless, such approachesdo appear to be useful experimentally (Li et al. ,2019; Roller et al., 2021).
Revised: Nevertheless, such approaches do appear to be useful experimentally (Li et al., 2019; Roller et al., 2021).

Original: Pairwise versus single-model ratings
Revised: Pairwise versus single-model ratings

Original: Conversations are often either rated individually, e.g.with Likert-score ratings (Ashwin et al., 2017;
Revised: Conversations are often either rated individually, e.g. with Likert-score ratings (Ashwin et al., 2017;

Original: Venkatesh et al., 2018; Zhang et al., 2018; Rashkinet al., 2019; See et al., 2019; Dinan et al., 2019b,2018), or pairwise by comparing models (Li et al.
Revised: Venkatesh et al., 2018; Zhang et al., 2018; Rashkin et al., 2019; See et al., 2019; Dinan et al., 2019b, 2018), or pairwise by comparing models (Li et al., 2019;

Original: Liang et al., 2020; Vinyals and Le, 2015;
Revised: Liang et al., 2020; Vinyals and Le, 2015;

Original: Li et al., 2016; Lee et al., 2020).
Revised: Li et al., 2016; Lee et al., 2020).

Original: Likert scoringrelies on absolute identification rather than relativediscrimination, which is less reliable in humans(Stewart et al., 2005), leading to different biasesper annotator (Kulikov et al., 2019).
Revised: Likert scoring relies on absolute identification rather than relative discrimination, which is less reliable in humans (Stewart et al., 2005), leading to different biases per annotator (Kulikov et al., 2019).

Original: It is thus oftennecessary to then re-evaluate existing models at thesame time as a new model, as the distribution ofhuman annotators can easily shift over time, causing measurement errors (See et al., 2019).
Revised: It is thus often necessary to then re-evaluate existing models at the same time as a new model, as the distribution of human annotators can easily shift over time, causing measurement errors (See et al., 2019).

Original: Anothercommon difficulty is related to sequential effects(Stewart et al., 2005), where the annotator can beinfluenced by the first model they evaluate, causing difficulties in using an absolute scale.
Revised: Another common difficulty is related to sequential effects (Stewart et al., 2005), where the annotator can be influenced by the first model they evaluate, causing difficulties in using an absolute scale.

Original: Pairwisecomparisons, on the other hand, make comparing aset of models less efficient, and also have the sameproblem that existing baseline models have to beessentially reassessed with respect to new ones.
Revised: Pairwise comparisons, on the other hand, make comparing a set of models less efficient, and also have the same problem that existing baseline models have to be essentially reassessed with respect to new ones.

Original: Per-turn versus per-dialogue evaluation
Revised: Per-turn versus per-dialogue evaluation

Original: Someresearch evaluates single-turn responses in conversations given gold dialogue contexts, withouttaking into account whole interactive conversations (Lee et al., 2020; Vinyals and Le, 2015; Li et al., 2016).
Revised: Some research evaluates single-turn responses in conversations given gold dialogue contexts, without taking into account whole interactive conversations (Lee et al., 2020; Vinyals and Le, 2015; Li et al., 2016).

Original: This fails to take into account multi-turn aspects of a conversation, for example a model repeating itself over multiple turns.
Revised: This fails to take into account multi-turn aspects of a conversation, for example a model repeating itself over multiple turns.

Original: Per-turn evaluation instead conducts an entire conversation, but raters are still asked to evaluate each turn (response by their partner)
Revised: Per-turn evaluation instead conducts an entire conversation, but raters are still asked to evaluate each turn (response by their partner)

Original: (Adiwardana et al., 2020; Komeili et al., 2021).
Revised: (Schmitt and Ultes, 2015; Adiwardana et al., 2020; Komeili et al., 2021).

Original: Collecting per-turn evaluation also allows for measuring learning effects where workers become more adept at interacting with the bot for certain specific tasks (e.g., see Xu et al.
Revised: Collecting per-turn evaluation also allows for measuring learning effects where workers become more adept at interacting with the bot for certain specific tasks (e.g., see Xu et al.

Original: In contrast, methods like multi-turn Likert or AcuteEval ask evaluators to assess the entire dialogue as a whole, rather than the individual turns, under the assumption that the quality of a conversation is not simply the sum of its parts.
Revised: In contrast, methods like multi-turn Likert or Acute-Eval ask evaluators to assess the entire dialogue as a whole, rather than the individual turns, under the assumption that the quality of a conversation is not simply the sum of its parts.

Original: Literature from psychology predicts several effects when considering how people combine their impressions from single conversational turns into an evaluation of an entire conversation.
Revised: Literature from psychology predicts several effects when considering how people combine their impressions from single conversational turns into an evaluation of an entire conversation.

Original: The primacy effect refers to how overall judgment is more shaped by characteristics presented earlier (Asch, 1946; Anderson, 1965).
Revised: The primacy effect refers to how overall judgment is more shaped by characteristics presented earlier (Asch, 1946; Anderson, 1965).

Original: Conversely, the recency effect appears when more weight is given to information presented the most recently, and both effects combine to give more weight to items at the beginning and end of a list (Murdock Jr, 1962; Postman and Phillips, 1965), with the recency effect being more prominent when judgment is elicited without any delay when the recent information is still fresh (Miller and Campbell, 1959; Hoch, 1984).
Revised: Conversely, the recency effect appears when more weight is given to information presented the most recently, and both effects combine to give more weight to items at the beginning and end of a list (Murdock Jr, 1962; Postman and Phillips, 1965), with the recency effect being more prominent when judgment is elicited without any delay when the recent information is still fresh (Miller and Campbell, 1959; Hoch, 1984).

Original: B Additional methods 
Revised: B Additional methods B. Screenshots of crowdsourced human evaluation tasks

Original: B. Screenshots of crowdsourced human evaluation tasks

Original: See Figure 2 for a screenshot of the PW-Turn evaluation technique, Figure 3 for a screenshot of the
Revised: See Figure 2 for a screenshot of the PW-Turn evaluation technique, Figure 3 for a screenshot of the

Original: PW-Dialog technique, and Figure 4 for a screenshot of the SM-Turn and SM-Dialog techniques.
Revised: PW-Dialog technique, and Figure 4 for a screenshot of the SM-Turn and SM-Dialog techniques.

Original: Figure 5 additionally displays the onboarding UI for PW-Turn
Revised: Figure 5 additionally displays the onboarding UI for PW-Turn.

Original: B. Pairwise per-turn evaluation onboarding
Revised: B. Pairwise per-turn evaluation onboarding

Original: In order to perform quality control on crowdworkers before the start of the conversation itself, we ask each worker to first annotate a conversation in which there are two possible responses for each turn of one of the speakers, one response of which is clearly better than the other (Figure 5).
Revised: In order to perform quality control on crowdworkers before the start of the conversation itself, we ask each worker to first annotate a conversation in which there are two possible responses for each turn of one of the speakers, one response of which is clearly better than the other (Figure 5).

Original: These pairs of responses vary slightly depending on which of the evaluation metrics is being tested.
Revised: These pairs of responses vary slightly depending on which of the evaluation metrics is being tested.

Original: Workersmust ultimately choose the correct response for allfour pairs of responses but have two tries in whichto do so.
Revised: Workers must ultimately choose the correct response for all four pairs of responses but have two tries in which to do so.

Original: B. Quality checks on crowdworkers
Revised: B. Quality checks on crowdworkers

Original: In order to ensure that our comparisons between evaluation techniques are not affected by variability in the pool of crowdworkers when running one technique vs. another, we adopt a consistent set of criteria across all techniques regarding which workers to exclude from our final set of data.
Revised: In order to ensure that our comparisons between evaluation techniques are not affected by variability in the pool of crowdworkers when running one technique vs. another, we adopt a consistent set of criteria across all techniques regarding which workers to exclude from our final set of data.

Original: If a worker fails one of the checks in Appendix B.4during one of the per-turn evaluations PW-Turnor SM-Turn, we retroactively exclude their ratingsfrom all of the evaluation techniques.
Revised: If a worker fails one of the checks in Appendix B. during one of the per-turn evaluations PW-Turn or SM-Turn, we retroactively exclude their ratings from all of the evaluation techniques.

Original: In order to prevent any worker from disproportionately contributing to the final results, eachworker is restricted to one conversation per modelpair and evaluation metric (for PW-Turn and PWDialog) or one conversation per model (for SMTurn and SM-Dialog).
Revised: In order to prevent any worker from disproportionately contributing to the final results, each worker is restricted to one conversation per model pair and evaluation metric (for PW-Turn and PWDialog) or one conversation per model (for SMTurn and SM-Dialog).

Original: All evaluations are collectedamong residents of the United States on weekdays,from roughly 9 AM to 6 PM in the U.S. Easterntime zone, following Li et al.
Revised: All evaluations are collected among residents of the United States on weekdays, from roughly 9 AM to 6 PM in the U.S. Eastern time zone, following Li et al.

Original: B. Checks used when filtering per-turn 
Revised: B. Checks used when filtering per-turn evaluations

Original: evaluations

Original: We check each conversation between a crowdworker and a Bot Speaker collected during PWTurn and SM-Turn evaluations against the criteriabelow to see if they have issues that warrant theirexclusion from the final filtered set of evaluations.
Revised: We check each conversation between a crowdworker and a Bot Speaker collected during PWTurn and SM-Turn evaluations against the criteria below to see if they have issues that warrant their exclusion from the final filtered set of evaluations.

Original: If at least one of the following problems is present,all evaluations from the crowdworker in questionare filtered out of the results shown in this work:
Revised: If at least one of the following problems is present, all evaluations from the crowdworker in question are filtered out of the results shown in this work:

Original: • The messages consist of less than three words on average • The first message inputted by the worker con- tains a greeting (redundant, since a dummy 
Revised: • The messages consist of less than three words on average • The first message inputted by the worker contains a greeting (redundant, since a dummy “Hi!”

Original: “Hi!” message is already fixed to be the worker’s first line of conversation)
Revised: message is already fixed to be the worker’s first line of conversation)

Original: • Several of the messages are written using all

Original: capital letters

Original: •  Later messages are duplicates of the first one (i.e. the worker is repeating their messages throughout the conversation) • One or more of the messages use offensive 
Revised: • Several of the messages are written using all capital letters • Later messages are duplicates of the first one (i.e. the worker is repeating their messages throughout the conversation) • One or more of the messages use offensive language

Original: language

Original: C Additional results 
Revised: C Additional results C. Evaluation data collection

Original: C. Evaluation data collection

Original: After filtering out workers with unacceptable messages following Appendix B.3, we are left with a minimum of 144 and a mean of 231 ratings (typically 6 per conversation) for each of the PW-Turn evaluations, a minimum of 191 and a mean of ratings for PW-Dialog, a minimum of 349 and a mean of 411 ratings (typically 6 per conversation) for SM-Turn, and a minimum of 58 and a mean of 68 ratings for SM-Dialog evaluations (for which there is only one rating per conversation).
Revised: After filtering out workers with unacceptable messages following Appendix B.3, we are left with a minimum of 144 and a mean of 231 ratings (typically 6 per conversation) for each of the PW-Turn evaluations, a minimum of 191 and a mean ofratings for PW-Dialog, a minimum of 349 and a mean of 411 ratings (typically 6 per conversation) for SM-Turn, and a minimum of 58 and a mean of 68 ratings for SM-Dialog evaluations (for which there is only one rating per conversation).

Original: On average, the collection of ratings after filtering represents 5.73 hours of worker labor for PW-Turn per model pair and evaluation metric, 6.03 hours for PW-Dialog per model pair and evaluation met-  ric, and 4.39 hours for joint SM-Turn/SM-Dialogevaluations per model.
Revised: On average, the collection of ratings after filtering represents 5.73 hours of worker labor for PW-Turn per model pair and evaluation metric, 6.03 hours for PW-Dialog per model pair and evaluation metric, and 4.39 hours for joint SM-Turn/SM-Dialog evaluations per model.

Original: C. PW-Turn model win rates as a function ofturn
Revised: C. PW-Turn model win rates as a function of turn

Original: Unlike PW-Dialog, the PW-Turn technique is ableto measure differences in the win rate of modelsas a function of the number of turns into the conversation.
Revised: Unlike PW-Dialog, the PW-Turn technique is able to measure differences in the win rate of models as a function of the number of turns into the conversation.

Original: In Figure 6 and Table 4, we see that
Revised: In Figure 6 and Table 4, we see that

Original: BlenderBot3B’s win rates tend to be closer to 50%in the first 1 or 2 turns of the Bot Speaker, andhigher later: this may be because the first few lines of the conversation typically consist of greetings ( “Hi, how are you?” ) or pleasantries, which may be harder to judge model performance on.
Revised: BlenderBot3B’s win rates tend to be closer to 50% in the first 1 or 2 turns of the Bot Speaker, and higher later: this may be because the first few lines of the conversation typically consist of greetings ( “Hi, how are you?” ) or pleasantries, which may be harder to judge model performance on.

Original: However, it may also be because improvements are accumulated and factored into evaluators’ decisionslater in the conversation.
Revised: However, it may also be because improvements are accumulated and factored into evaluators’ decisions

Original: Strikingly, BlenderBot3B performs very poorly vs. BlenderBot90M (the Size comparison) on the first Bot Speaker turn: empirically, this may be due to the fact that BlenderBot3B generally starts its first responses with the greet-

Original: [Table]
Revised: [Table]

Original: ings “Hi” or “Hello” much less frequently than
Revised: later in the conversation. Strikingly, BlenderBot3B performs very poorly vs. BlenderBot90M (the Size comparison) on the first Bot Speaker turn: empirically, this may be due to the fact that BlenderBot3B generally starts its first responses with the greetings “Hi” or “Hello” much less frequently than

Original: BlenderBot90M does.
Revised: BlenderBot90M does.

Original: C. SM-Turn success rates as a function of 
Revised: C. SM-Turn success rates as a function of conversation turn

Original: conversation turn

Original: See Table 5 for the success rates of model responsesusing the SM-Turn technique, as a function of Bot
Revised: See Table 5 for the success rates of model responses using the SM-Turn technique, as a function of Bot

Original: Speaker turn.

Original: [Table]
Revised: [Table]

Revised: -M

Revised: Pref     72 Human     66 Inter

Original: BB90M
Revised: BB90M

Original: Pref 64 Human     67 Inter
Revised: Pref [Table] [Table]

Original: Reddit3B
Revised: Reddit3B

Original: Pref     46 Human     58 Inter
Revised: Pref     46 Human     58 Inter

Revised: Speaker turn.

Original: C. Relationship between per-turn ratings 
Revised: C. Relationship between per-turn ratings and final ratings

Original: and final ratings

Original: Given that SM-Turn allows us to measure perturn ratings of model performance, it is worth exploring whether there are certain turns of the conversation that contribute more strongly to the workers’ final Likert-scale ratings of the conversation 
Revised: Given that SM-Turn allows us to measure per-turn ratings of model performance, it is worth exploring whether there are certain turns of the conversation that contribute more strongly to the workers’ final Likert-scale ratings of the conversation (SM-

Original: (SM-Dialog).
Revised: Dialog).

Original: Figure 7 plots the coefficients ofworkers’ per-turn SM-Turn ratings in an OLS regression, with the SM-Dialog Likert score at theend of the conversation as the dependent variable.(Here, we reduce variability by taking the meanover the three evaluation metrics for each turn’s SM-Turn ratings and SM-Dialog Likert scores.)
Revised: Figure 7 plots the coefficients of workers’ per-turn SM-Turn ratings in an OLS regression, with the SM-Dialog Likert score at the end of the conversation as the dependent variable. (Here, we reduce variability by taking the mean over the three evaluation metrics for each turn’s SM-Turn ratings and SM-Dialog Likert scores.)

Original: Generally, we see a higher positive coefficient ofthe SM-Turn ratings in later turns in the conversation, which implies that the workers may havea recency bias: they may remember the most recent turns of the conversation more strongly whendetermining how to rate the model’s performanceoverall.
Revised: Generally, we see a higher positive coefficient of the SM-Turn ratings in later turns in the conversation, which implies that the workers may have a recency bias: they may remember the most recent turns of the conversation more strongly when determining how to rate the model’s performance overall.

Original: C. Text justification for model response 
Revised: C. Text justification for model response selection

Original: selection

Original: For PW-Turn evaluations, we collect and analyzejustification texts for each turn, after the worker selects a model response.
Revised: For PW-Turn evaluations, we collect and analyze justification texts for each turn, after the worker selects a model response.

Original: We then group justificationtexts by model type and comparison.
Revised: We then group justification texts by model type and comparison.

Original: To measure lengths of justifications, we splittext strings into words (space-delimited), and wecalculate the mean number of words in each sample.
Revised: To measure lengths of justifications, we split text strings into words (space-delimited), and we calculate the mean number of words in each sample.

Original: Results are shown in Table 6.
Revised: Results are shown in Table 6.

Original: [Table]
Revised: [Table]

Original: For term importance, we use thescikit-learn TfidfVectorizer class tocompute TF-IDF scores for each term in eachmodel comparison.
Revised: For term importance, we use the scikit-learn TfidfVectorizer class to compute TF-IDF scores for each term in each model comparison.

Original: We use a list of English stopwords from the
Revised: We use a list of English stopwords from the

Original: NLTK library to filter out common terms.
Revised: NLTK library to filter out common terms.

Original: Additionally, we discard terms that have a higher documentfrequency than 0.8.
Revised: Additionally, we discard terms that have a higher document frequency than 0.8.

Original: The top 20 terms (descending order) for eachmodel pairing are shown in Table 7.
Revised: The top 20 terms (descending order) for each model pairing are shown in Table 7.

Original: Our analysis reveals the following:
Revised: Our analysis reveals the following:

Original: • Length comparison : While it appears that many crowdworkers prefer longer responses overall, at least in some conversational turns some crowdworkers may prefer shorter re-
Revised: • Length comparison : While it appears that many crowdworkers prefer longer responses overall, at least in some conversational turns some crowdworkers may prefer shorter responses.

Original: sponses. The top terms in justifications for BlenderBot3B-M0 responses include “sim-
Revised: The top terms in justifications for BlenderBot3B-M0 responses include “simple”, “short”

Original: ple”, “short” and “direct”, while top terms in reasons for choosing BlenderBot3B include “detailed” and “longer”.
Revised: and “direct”, while top terms in reasons for choosing BlenderBot3B include “detailed” and “longer”.

Original: This shows that PW- Turn evaluation does well in capturing sensi- tivity to length, and that workers’ selections are due to their own preferences at a given 
Revised: This shows that PWTurn evaluation does well in capturing sensitivity to length, and that workers’ selections are due to their own preferences at a given conversational turn.

Original: conversational turn.

Original: Interestingly, in PW-Turn we find that workers’ justifications for choosing the
Revised: Interestingly, in PW-Turn we find that workers’ justifications for choosing the

Original: BlenderBot3B-M0 responses are themselves on average shorter than for BlenderBot3B. Ta- ble 6 shows the mean justification lengths for different model pairings.
Revised: BlenderBot3B-M0 responses are themselves on average shorter than for BlenderBot3B. Table 6 shows the mean justification lengths for different model pairings.

Original: The mean justifica- tion length for BlenderBot3B is 8.85 words, compared to a mean length of BlenderBot3B- M0 justifications of 7.7 words.
Revised: The mean justification length for BlenderBot3B is 8.85 words, compared to a mean length of BlenderBot3BM0 justifications of 7.7 words.

Original: This suggests that workers choosing shorter, “simple” re- sponses may also be less detail-oriented.
Revised: This suggests that workers choosing shorter, “simple” re sponses may also be less detail-oriented.

Original: • Size comparison : Top TF-IDF weighted
Revised: • Size comparison : Top TF-IDF weighted

Original: terms from workers’ justifications for both models contain a mix of references to the con- versational content, such as “hiking”, “beach” or “dogs”, and conversational structure, such 
Revised: terms from workers’ justifications for both models contain a mix of references to the conversational content, such as “hiking”, “beach” or “dogs”, and conversational structure, such as “relates” or “engaging”.

Original: as “relates” or “engaging”.

Original: By inspection, there are no discernible differences between these terms.
Revised: By inspection, there are no discernible differences between these terms.

Original: • Fine-tuning comparison : High TF-IDF- weighted terms in justifications given by workers who choose the BlenderBot3B model are mostly related to conversational flow, such as “follows”, “responds”, and “acknowledges”.
Revised: • Fine-tuning comparison : High TF-IDFweighted terms in justifications given by workers who choose the BlenderBot3B model are mostly related to conversational flow, such as “follows”, “responds”, and “acknowledges”.

Original: In contrast, terms appearing in justifications for the Reddit3B model are specific and often refer to the topic instead of conversational style, such as “bath”, “robot”, and “paris”.
Revised: In contrast, terms appearing in justifications for the Reddit3B model are specific and often refer to the topic instead of conversational style, such as “bath”, “robot”, and “paris”.

Original: This suggests that workers who choose the Reddit3B model often favor less natural responses because they contain particular references.
Revised: This suggests that workers who choose the Reddit3B model often favor less natural responses because they contain particular references.

Original: These nuanced differences are clear when evaluating model responses per turn, but are difficult to capture when evaluating the conversation as a whole.
Revised: These nuanced differences are clear when evaluating model responses per turn, but are difficult to capture when evaluating the conversation as a whole.

Original: Analysis of worker justifications supports our hypothesis that differences in conversational quality are easier to identify in the PW-Turn evaluation.
Revised: Analysis of worker justifications supports our hypothesis that differences in conversational quality are easier to identify in the PW-Turn evaluation.

Original: [Table]

Original: C. Variability in win rate across evaluation 
Revised: C. Variability in win rate across evaluation techniques

Original: techniques

Original: [Table]

Original: of crowdworker time

Original: each, and then computing the standard deviation of the win rate across chunks.

Original: Standard deviations are averaged across the three evaluation metrics (Section 3.1.2).

Original: Win rates for PW-Turn were compiled over Bot Speaker turns 2 to 6 and for SM-Turn over turns 3 to 6, following

Original: Section 4.3.1.

Original: Table 8 shows the variability in the win rates of
Revised: Table 8 shows the variability in the win rates of

Original: BlenderBot3B per evaluation technique, as measured by splitting the ratings from each techniqueinto chunks of equal crowdworker time.
Revised: BlenderBot3B per evaluation technique, as measured by splitting the ratings from each technique into chunks of equal crowdworker time.

Original: The winrates from PW-Turn, PW-Dialog, and SM-Turnvary least across chunks, largely because the meantime per rating is small, leading to a larger number of ratings per chunk and thus a more preciseestimate obtainable within a given block of time.
Revised: The win rates from PW-Turn, PW-Dialog, and SM-Turn vary least across chunks, largely because the mean time per rating is small, leading to a larger number of ratings per chunk and thus a more precise estimate obtainable within a given block of time.

Original: This suggests that calculating the per-conversation winner-takes-all win rate for the per-turn methods
Revised: This suggests that calculating the per-conversation winner-takes-all win rate for the per-turn methods

Original: PW-Turn and SM-Turn may be disadvantageous if having a precise measurement of the win rate is more important than one that is statistically significant.
Revised: PW-Turn and SM-Turn may be disadvantageous if having a precise measurement of the win rate is more important than one that is statistically significant.

Original: technique
Revised: C.

Original: Figures 8, 9, and 10 show the time needed to achieve a statistically significant difference between a pair of models for each of the evaluation techniques studied.
Revised: Figures 8, 9, and 10 show the time needed to achieve a statistically significant difference between a pair of models for each of the evaluation [Table] [Table]

Revised: was mea sured by splitting each time-ordered set of into chunks representing 45 minutes of crowdworker time each, and then computing the standard deviation of the win rate across chunks.

Original: For these plots, we consider ratings for each turn in Bot Speaker turns 2 through 6for PW-Turn and Bot Speaker turns 3 through for SM-Turn, as in Section 4.3.1.
Revised: Standard deviations are averaged across the three evaluation metrics (Section 3.1.2). Win rates for PW-Turn were compiled over Bot Speaker turns 2 to 6 and for SM-Turn over turns 3 to 6, following Section 4.3.1.

Revised: techniques studied.

Revised: For these plots, we consider ratings for each turn in Bot Speaker turns 2 through 6 for PW-Turn and Bot Speaker turns 3 throughfor SM-Turn, as in Section 4.3.1.

Original:  We use a twosided binomial test for PW-Turn and PW-Dialog and a two-sided independent t -test for SM-Turn and SM-Dialog.
Revised: 6 We use a twosided binomial test for PW-Turn and PW-Dialog and a two-sided independent t -test for SM-Turn and SM-Dialog.

Original: Significance is measured at a pvalue of 5%.
Revised: Significance is measured at a pvalue of 5%.

Original: When estimating the crowdsourcing time needed for each evaluation technique, we include an estimate of each technique’s time to complete onboarding, which is mandatory before being approved to work on an evaluation.
Revised: When estimating the crowdsourcing time needed for each evaluation technique, we include an estimate of each technique’s time to complete onboarding, which is mandatory before being approved to work on an evaluation.

Original: For PW-Dialog evaluations (i.e. Acute-Evals)on conversations between a human and a model,the labor costs involve collecting both conversations and rating pairs.
Revised: For PW-Dialog evaluations (i.e. Acute-Evals) on conversations between a human and a model, the labor costs involve collecting both conversations and rating pairs.

Original: This gives us a parameterto tune in this method: how many conversationsto collect, and then how many times to reuse them when rating pairs of them.
Revised: This gives us a parameter to tune in this method: how many conversations to collect, and then how many times to reuse them when rating pairs of them.

Original: In our experiments, the number of conversations necessary is chosen such that each possible pairing of a conversation from one model and a conversation from another model should only be evaluated once at most: thus, if we have N conversations for each of the two models being compared, we will be able to perform a maximum of N 2 PW-Dialog evaluations on these conversations.
Revised: In our experiments, the number of conversations necessary is chosen such that each possible pairing of a conversation from one model and a conversation from another model should only be evaluated once at most: thus, if we have N conversations for each of the two models being compared, we will be able to perform a maximum of N 2 PW-Dialog evaluations on these conversations.

Original: C. Crowdsourcing time needed when 
Revised: C. Crowdsourcing time needed when combining single-model methods

Original: combining single-model methods

Original: Figures 11, 12, and 13 show the time needed to achieve a statistically significant difference between models when averaging together SM-Turn winner-takes-all success rates from Bot Speaker turns 3 to 6 (Section 4.2) with SM-Dialog Likert scores.
Revised: Figures 11, 12, and 13 show the time needed to achieve a statistically significant difference between models when averaging together SM-Turn winner-takes-all success rates from Bot Speaker turns 3 to 6 (Section 4.2) with SM-Dialog Likert scores.

Original: To perform the weighted average between
Revised: To perform the weighted average between

Original: SM-Turn and SM-Dialog on each conversation, we first shift and scale the originally 1-to-5 SM-Dialog
Revised: SM-Turn and SM-Dialog on each conversation, we first shift and scale the originally 1-to-5 SM-Dialog

Original: Likert scores to fall within the range [0 , 1] , matching the range of the individual binary SM-Turn success rates.
Revised: Likert scores to fall within the range [0 , 1] , matching the range of the individual binary SM-Turn success rates.

Original: We see that statistical significance is reached fastest when weighting SM-Turn much less heavily than SM-Dialog at a ratio of 1:5 or 1:10, which is to be expected given the alreadymuch stronger sensitivity of SM-Dialog.
Revised: We see that statistical significance is reached fastest when weighting SM-Turn much less heavily than SM-Dialog at a ratio of 1:5 or 1:10, which is to be expected given the already much stronger sensitivity of SM-Dialog.