Original: Abg-CoQA: Clarifying Ambiguity in Conversational Question Answering
Revised: Abg-CoQA: Clarifying Ambiguity in Conversational Question Answering

Original: Anonymous authors

Original: Abstract
Revised: Abstract

Original: Eﬀective communication is about the dissemination of properly worded meaningful ideas/messages that are comprehensible to both sender and receiver and which ultimately can attract the desired response or feedback. For machines to engage in a conversation, it is therefore essential to enable them to clarify ambiguity and achieve a common ground.
Revised: Eﬀective communication requires the ability to identify ambiguities and request clariﬁcation of utterances. For machines to engage in a conversation, they need to learn to generate diﬀerent forms of clariﬁcation questions. This paper aims at studying the extent to which the state of the art neural generation models can generate eﬀective clariﬁcation questions in conversational question answering.

Original: We introduce Abg-CoQA, a novel  dataset for clarifying ambiguity in Conversational Question Answering systems.
Revised: We introduce Abg-CoQA, a novel crowdsourced dataset for clarifying ambiguities in conversational question answering systems.

Original: Our dataset contains 9k questions with answers where 1k questions are ambiguous, obtained from 4k text passages from ﬁve diverse domains.
Revised: Our dataset contains 8,615 questions with answers where 994 questions are ambiguous. The conversational questions are about 3,968 text passages from ﬁve diverse domains which are pre-selected from the CoQA dataset.

Original: For ambiguous questions, a clariﬁcation conversational turn is collected.
Revised: For ambiguous turns, we have collected the clariﬁcation questions and their answers.

Original: We evaluate strong language generation models and conversational question answering models on Abg-CoQA.
Revised: We evaluate strong language generation models and conversational question answering models on Abg-CoQA.

Original: The bestperforming system achieves a BLEU-1 score of 12.9% on generating clariﬁcation question, which is 27.9 points behind human performance (40.8%); and a F1 score of 40.1% on question answering after clariﬁcation, which is 35.1 points behind human performance (75.2%), indicating there is ample room for improvement.
Revised: The best-performing system achieves a F1-score of 23.6% on ambiguity detection; an accuracy of 56.0% on generating clariﬁcation question in human evaluation; and a F1 score of 40.1% on question answering after clariﬁcation, which is 35.1 points behind human performance (75.2%), indicating there is ample room for improvement.

Original: 1. Introduction
Revised: 1. Introduction

Original: Ambiguity is an intrinsic characteristic of human conversations and is particularly challenging in natural language understanding.

Original: People naturally resolve ambiguities in conversation by asking context-dependent clariﬁcation questions
Revised: People naturally resolve ambiguities in conversation by asking context-dependent clariﬁcation questions

Original: [Clark and Brennan, 1991].
Revised: [Clark and Brennan, 1991].

Original: Although there has been a surge in datasets/tasks on conversational question answering [Choi et al., 2018, Reddy et al., 2019], few studies have explored ambiguity resolution and clariﬁcation.
Revised: Although there has been a surge in research on conversational question answering [Choi et al., 2018, Reddy et al., 2019], few studies have explored ambiguity resolution and clariﬁcation.

Revised: Question answering (QA) systems’ goal is to return the answer to a question by searching for a response in a collection of documents [Rajpurkar et al., 2016, 2018, Seo et al., 2017, Wang et al., 2017].

Revised: Often these systems may need additional information in order to answer an user’s query.

Revised: Similar to human-human conversation, QA systems needs to identify and clarify ambiguities in order to be able to return the correct response.

Original: In this paper we introduce Abg-CoQA, a dataset for clarifying the a m b i g uity in
Revised: In this paper, we introduce Abg-CoQA, a dataset for clarifying the a m b i g uity in

Original: Co nversational Q uestion A nswering.
Revised: Co nversational Q uestion A nswering.

Original: In Abg-CoQA, a machine has to answer an ambiguous question after resolving the ambiguity through a clariﬁcation dialog.
Revised: The speciﬁc problem we address in this paper is on detecting ambiguities and generating the right kinds of clariﬁcation questions for resolving them.

Original: As Figureshows, the model needs to ﬁrst detect whether a question ( Q i ) in a conversation is ambiguous or not; for ambiguous questions it needs to generate a clariﬁcation question ( CQ ) targeting the ambiguity; since there are in general several possibilities for answering CQ , it then needs to provide an answer ( A i ) based on each possible clariﬁcation reply ( R i ).
Revised: As Figure 1 shows, the model needs to ﬁrst detect whether a question ( Q i ) in a conversation is ambiguous or not. Then, for ambiguous questions it needs to generate a clariﬁcation question ( CQ ). Since there are in general several possibilities for answering CQ , it then needs to provide an answer ( A k ) based on each possible clariﬁcation reply ( R k ).

Original: We develop Abg-CoQA with three main goals in mind. The ﬁrst concerns the nature of ambiguity in a human conversation related to a text passage.
Revised: Our ﬁrst objective is to study how and when ambiguities arise in conversational question answering.

Original: Our dataset covers the two main types of ambiguity in human conversations: when the question focus is ambiguous ( e.g. , ambiguity in coreference resolution in Table 2); when there exist several possibilities to answer the question ( e.g. , ambiguity in answer types in Table 3, Figure 1) [Ginzburg, 1996, Larsson, 2002].The diversity of ambiguity types brings a challenge to models for generating appropriate clariﬁcation questions.
Revised: Our dataset covers four types of ambiguities (Table 2) in human conversations which are related to the two ambiguity types introduced in Ginzburg [1996]: 1) when the question’s focus is ambiguous; 2) when there exist several possibilities to answer the question [Larsson, 2002].

Original: The second goal of Abg-CoQA is to ensure the naturalness of clarifying ambiguity in a conversation. In case of ambiguity in human conversations, the answerer asks a clariﬁcation question for collecting more information from the questionner. Diﬀerent from a previous work on open-domain question [Min et al., 2020] which requires the model providing disambiguated rewrites of the ambiguous question, our dataset contains a natural clariﬁcation dialog for disambiguating the question. The third goal of Abg-CoQA is to enable building question answering systems that perform robustly on the same question according to diﬀerent clariﬁcation turns.
Revised: The second objective is to enable the design of robust question answering systems that can handle ambiguities in longer interactions.

Original: The current

Original: CoQA datasets test the question answering systems’ ability on understanding the passage and conversational history through answering a target question, which makes it hard to distinguish between a truly understanding of the context and a correct prediction based on superﬁcial features [Chen et al., 2016, Weissenborn et al., 2017].

Original: To summarize, Abg-CoQA has the following key characteristics: 1) it consists of 4k passages from ﬁve diﬀerent domains and 9k conversational questions where 1k of them are ambiguous; 2) it covers four diﬀerent ambiguity types and in most cases, the ambiguity is apparent after referring to the conversation history and researching all possible answers in the story; 3) each ambiguous question is followed by a clariﬁcationturn which consists of a question and several possible replies which lead to diﬀerent answers to the originally ambiguous question.
Revised: To summarize, Abg-CoQA has the following key characteristics: 1) it consists of 3, passages from ﬁve diﬀerent domains and 8,615 conversational questions where 994 of them are ambiguous; 2) it covers four diﬀerent ambiguity types and in most cases, the ambiguity is apparent after referring to the conversation history and researching all possible answers in the story; 3) each ambiguous question is clariﬁed by a clariﬁcation question and several possible replies which lead to diﬀerent answers to the originally ambiguous question.

Original: We benchmark several deep neural network models, building ontop of state-of-the-art conversational question answering and natural language generation models (Section 6). The best-performing system achieves an F1 score of 22.1% on predicting ambiguity, a BLEUscore of12.9% on generating clariﬁcation question and a F1 score of 40.1% on clariﬁcationbased question answering.
Revised: Building on state-of-the-art conversational question answering and neural language generation models, we present several benchmarks in ambiguity detection, generation of clariﬁcation questions and answering questions which are disambiguated by the clariﬁcation. We demonstrate the limitations of these end-to-end models and discuss possible future work.

Original: In contrast, humans achieve 40.8% BLEU-1 (27.9% higher) for clariﬁcation question generation and 75.2% F1 (35.1% higher) for clariﬁed question answering, indicating that there is a lot of headroom for improvement.
Revised: Our code and data is publicly available at : https://github.com/MeiqiGuo/ AKBC2021-Abg-CoQA .

Original: 2. Related Work
Revised: 2. Related Work

Original: Conversational question answering requires a system to understand a text passage and answer a series of questions that appear in a conversation [Reddy et al., 2019, Choi et al., 2018].
Revised: The ability to request clariﬁcation of utterance in dialogue has been extensively studied in dialogue research [Purver et al., 2003, Schlangen, 2004, Stoyanchev et al., 2014].

Original: It has potential applications on intelligent assistants and dialogue systems where ambiguity commonly exist.
Revised: To the best of our knowledge, however, most of the existing conversational question answering benchmarks have either underestimated or ignored the possibility of the presence of ambiguities in conversational question answering.

Original: Clarifying ambiguity is essential for grounding in communication [Clark and Brennan, 1991].
Revised: Khalid et al. [2020], Rodr´ıguez and Schlangen [2004] have combined cognitive modeling and discourse theories with reinforcement learning to choose the best clariﬁcation requests from a set of predeﬁned strategies.

Original: To the best of our knowledge, all existing conversational question answering benchmarks assume each question has a single clear answer and ignore the possibility to be ambiguous.
Revised: Our focus, however, will be on exploring the potential of large neural language generation models in automatically generating the optimal clariﬁcation questions in conversational question answering.

Original: A recent work investigate the ambiguity  in open-domain question answering [Min et al., 2020] and propose question rewriting for resolving ambiguity.
Revised: Min et al. [2020] investigated ambiguity resolution in open-domain question answering through question rewriting.

Original: However, question rewriting is not natural in conversation for clarifying ambiguity. Our work focuses on clarifying ambiguity in conversational question answeringand resolves the ambiguity by interactively asking clariﬁcation questions, which follows the naturalness of communication [Traum, 1994, Kato et al., 2013].
Revised: Our work, however, focuses on clarifying ambiguities in conversational question answering by interactively asking clariﬁcation questions [Traum, 1994, Kato et al., 2013].

Original: Clariﬁcation questions have been used to resolve question ambiguity in other areas.

Original: Prior work studies the types, subjects and eﬀectiveness of clariﬁcation questions that users ask on the Stack Exchange community question answering platform[Braslavski et al., 2017,
Revised: Prior work have studied the types, subjects and eﬀectiveness of clariﬁcation questions that users ask on the Stack Exchange community question answering platform

Original:  Rao and Daum´e III, 2018].
Revised: [Braslavski et al., 2017, Rao and Daum´e III, 2018].

Original: Our work diﬀers from these two studies in that conversational question answering is in multi-turn and the clariﬁcation has a direct impact on the answer.
Revised: Our work diﬀers from these two studies in that conversational question answering is in multi-turn and the clariﬁcation has a direct impact on the answer.

Original: Khalid et al.

Original: [2020] studies interactive communication with agent but focuses on communicative strategies for targeted, eﬀective feedback about the system’s understanding on reference tasks.

Original: Aliannejadi et al.
Revised: Aliannejadi et al.

Original: [2019], Zamani et al.
Revised: [2019], Zamani et al.

Original: [2020] study a sequence of clariﬁcation questions to reﬁne intents of simple searching query.
Revised: [2020] study a sequence of clariﬁcation questions to reﬁne intents of simple searching query.

Original: Saeidi et al.
Revised: Saeidi et al.

Original: [2018] worked onmachine comprehension of natural language rulesand considered clariﬁcation question for seeking missing information in  the question during the communication with a robot assistant.
Revised: [2018] worked on natural language rules in conversational machine reading and showed the success of crowdsourcing in answering the ambiguous questions that require access to world knowledge.

Original: Xu et al.
Revised: Xu et al.

Original: [2019] deﬁne similar tasks on clarifying ambiguityas ours but their work is for knowledge-based question answering and the ambiguity types are only limited to entity reference and pronoun reference.
Revised: [2019] deﬁne similar tasks on clarifying ambiguity for knowledge-based question answering where the ambiguity types are only limited to entity reference and pronoun reference.

Original: In our work, the ambiguity naturally comes from human communication with related to a text passage, thus covers a diversity of sources (Table 2).
Revised: We focus on resolving ambiguities that naturally arise in conversational question answering about a passage (Table 2).

Original: 3. Task Deﬁnition
Revised: 3. Task Deﬁnition

Original: Figure 1 depicts the Abg-CoQA task.
Revised: Figure 1 depicts the Abg-CoQA task.

Original: Given a passage P and a conversation { Q i − n , A i − n ,...,
Revised: Given a passage P and a conversation { Q i − n , A i − n ,...,

Original: Q i − 1 , A i − 1 } (where n is the number of the conversation turns), the task is to clarify the ambiguity in the next question Q i if it is ambiguous.
Revised: Q i − 1 , A i − 1 } (where n is the number of the conversation turns), the task is to clarify the ambiguity in the next question Q i if it is ambiguous.

Original: We consider three tasks.
Revised: We consider three tasks.

Original: Ambiguity Detection.
Revised: Ambiguity Detection.

Original: Given a passage P and a conversation Q i − n , A i − n ,..., Q i − 1 ,
Revised: Given a passage P and a conversation Q i − n , A i − n ,..., Q i − 1 ,

Original: A i − 1 , detect whether the next question Q i is ambiguous.
Revised: A i − 1 , detect whether the next question Q i is ambiguous.

Original: Clariﬁcation Question Generation.
Revised: Clariﬁcation Question Generation.

Original: Given a passage P , a conversation { Q i − n ,
Revised: Given a passage P , a conversation { Q i − n ,

Original: A i − n ,..., Q i − 1 , A i − 1 , Q i } where Q i is ambiguous, generate a clariﬁcation question CQ which is helpful for disambiguating Q i .
Revised: A i − n ,..., Q i − 1 , A i − 1 , Q i } where Q i is ambiguous, generate a clariﬁcation question CQ which is helpful for disambiguating Q i .

Original: Clariﬁcation-based Question Answering.
Revised: Clariﬁcation-based Question Answering.

Original: Given a passage P , a conversation { Q i − n ,
Revised: Given a passage P , a conversation { Q i − n ,

Original: A i − n ,..., Q i − 1 , A i − 1 , Q i , CQ , R k } where Q i is an ambiguous question, CQ is a clariﬁcation question for Q i , and R k is one possible answer to CQ , answer the question Q i which is no longer ambiguous based on the clariﬁcation.
Revised: A i − n ,..., Q i − 1 , A i − 1 , Q i , CQ , R k } where Q i is an ambiguous question, CQ is a clariﬁcation question for Q i , and R k is one possible reply to CQ , answer the question Q i which is no longer ambiguous based on the clariﬁcation.

Original: Note that there may exist several diﬀerent answers to the clariﬁcation question CQ , and the answer to Q i changes with the clariﬁcation answer.
Revised: There should be diﬀerent answers A R k for every R k .

Original: 4. Data Collection
Revised: 4. Data Collection

Original: We construct Abg-CoQA based on the CoQA dataset [Reddy et al., 2019].
Revised: We construct Abg-CoQA based on the CoQA dataset [Reddy et al., 2019].

Original: Since most questions in the CoQA dataset are not ambiguous, we increase the ambiguity rate in our annotated corpus by 1) considering a partial conversation (keeping several previous conversational turns) rather than the full conversation; 2) pre-select probably ambiguous questions by using question answering models which are trained on CoQA dataset.
Revised: Since most questions in the CoQA dataset are not ambiguous, we increase the ambiguity rate in our annotated corpus by 1) considering a partial conversation (keeping several previous conversational turns) rather than the full conversation; 2) pre-select probably ambiguous questions by using question answering models which are trained on CoQA dataset.

Original: We use Amazon
Revised: We use Amazon

Original: Mechanical Turk (AMT) for crowdsourcing.
Revised: Mechanical Turk (AMT) for crowdsourcing.

Original: 4.1 Collection Process
Revised: 4.1 Collection Process

Original: Given a story and a conversation (which is generally partial), annotators are asked to identify whether a question is ambiguous or not.
Revised: Given a story and a conversation (which is generally partial), annotators are asked to identify whether a question is ambiguous or not.

Original: If it is ambiguous, then provide a clariﬁcationturn. A clariﬁcation turn consists of a clariﬁcation question and all possible replies to it (could be one or several replies, see Figure 1).
Revised: If it is ambiguous, then provide a clariﬁcation question and all possible replies to it (could be one or several replies, see Figure 1).

Original: We also ask annotators to write all possible answers to the initial ambiguous question according to each clariﬁcation reply (refer to
Revised: We also ask annotators to write all possible answers to the initial ambiguous question according to each clariﬁcation reply 1 .

Original: Appendix A for examples of the annotation interface).

Original: In order to ensure the annotation quality on the crowd-sourcing platform, we ﬁlter AMT workers by location (US, CA, IN only for making sure  workers arenative English speaker), assignment approval rate ( > 97%) and customized qualiﬁcation tests (for making sure  workers understand the coding manual).
Revised: In order to ensure the annotation quality on the crowd-sourcing platform, we ﬁlter AMT workers by location (US, CA, IN only for making sure that workers are English speakers), assignment approval rate ( > 97%) and customized qualiﬁcation tests (for making sure that workers understand the guidelines).

Original: 4.2 Ambiguous Question Pre-Selection
Revised: 4.2 Ambiguous Question Pre-Selection

Original: CoQA is a large-scale dataset for conversational question answering [Reddy et al., 2019].

Original: We aim to annotate ambiguous questions in CoQA for our research purpose.
Revised: We pre-select samples which are likely to be ambiguous from CoQA.

Original: According to their experimental results with a varied number of previous turns used as conversation history, all models succeed at leveraging history but the gains are little beyond one previous turn.
Revised: According to Reddy et al., all models succeed at leveraging history but the gains quickly drop beyond one previous turn.

Original: They have  same observation with human performance: given two history turns, human performance reaches up to almost same as given the full history.
Revised: They have the same observation with human performance: given two history turns, human performance reaches up to almost same as given the full history.

Original: This suggests that most questions in a conversation have a limited dependency within a bound of two turns.
Revised: This suggests that most questions in a conversation have a limited dependency within a bound of two turns.

Original: Therefore, in our task, we only provide one or two history turns, which decreases the annotators’ work load (shorter conversation) and potentially increases the ambiguity rate of questions (some questions may have longer dependency than 2 turns).
Revised: Therefore, in our task, we only provide one or two history turns, which decreases the annotators’ work load (shorter conversation) and potentially increases the ambiguity rate of questions (some questions may have longer dependency than 2 turns).

Original: We pre-select questions which get a wrong answer given an incomplete  history but could have been answered correctly given the full history.
Revised: We pre-select questions which get a wrong answer given an incomplete conversation history but are answered correctly given the full history.

Original: The intuition is that those questions[Table] turn to be ambiguous because of the shorter conversation history rather than the inherent diﬃculty for answering the question itself.
Revised: The intuition is that those questions turn to be ambiguous because of the shorter conversation history rather than the inherent diﬃculty for answering the question itself.

Original: We ﬁrst train a baseline model which has a
Revised: We ﬁrst train a baseline model which has a

Original: BERT-based architecture with answer veriﬁcation on the CoQA training dataset given the full conversational history as input.
Revised: BERT-based architecture with answer veriﬁcation on the CoQA training dataset, given the full conversational history as input [Wu et al., 2019].

Original: Then we select a sample if the model prediction given an incomplete history as input is greatly worse than given the full history.
Revised: Then we select a sample if the model prediction given an incomplete history as input is greatly worse than given the full history.

Original: With this preselection process, we construct our corpus to be annotated.
Revised: With this pre-selection process, we construct our corpus to be annotated.

Original: Beside CoQA, we initially consider QuAC [Choi et al., 2018] as our data source since it is also a conversational question answering dataset based on context. We follow the same process for pre-selecting potentially ambiguous questions using a BERT-based model with history attention mechanism [Qu et al., 2019]. However, our pilot study on 50 samples of QuAC shows that the ambiguous rate is very low – 2%.
Revised: [Table]

Original: We thus don’t include QuAC in our work because of its low annotation eﬃciency.

Original: With respect to the data splitting, we follow the same way as CoQA [Reddy et al., 2019].
Revised: With respect to the data splitting, we follow the same way as CoQA.

Original: For each source dataset ( e.g. , Children’s Story, Litterature, etc.), we split the data such that there are 100 passages in the development set, 100 passages in the test set, and the rest in the training set.
Revised: For each source domain ( e.g. , Children’s Stories), we split the data such that there are 100 passages in the development set, 100 passages in the test set, and the rest in the training set.

Original: 5. Data Analysis
Revised: 5. Data Analysis

Original: The ﬁnal dataset contains 3,968 passages and 8,615 questions, where 994 questions are annotated as ambiguous.
Revised: The ﬁnal dataset contains 3,968 passages and 8,615 questions, where 994 questions are annotated as ambiguous.

Original: Domain Distribution.
Revised: Domain Distribution.

Original: Table 1 shows the distribution of passages and questions with respect to the source domains of CoQA.
Revised: Table 1 shows the distribution of passages and questions with respect to the source domains of CoQA.

Original: We observe that the domain of Literature has the lowest ambiguous rate and the domain of Children’s Story has the highest.
Revised: We observe that the domain of Literature has the lowest ambiguity rate and the domain of Children’s Stories has the highest.

Original: This meets our intuition that language uses in Literature are more precise therefore there is less ambiguity in the conversation; in contradictory, Children’s story is generally informal.
Revised: This meets our intuition that language uses in Literature are more precise therefore there is less ambiguity in the conversation; in contradictory, Children’s Stories is generally informal.

Original: Types of Ambiguity.
Revised: Types of Ambiguity.

Original: Table 2 shows a breakdown of the types of ambiguity in AbgCoQA.
Revised: Table 2 shows a breakdown of the types of ambiguity in AbgCoQA.

Original: We deﬁne a taxonomy with four categories, including ambiguity in coreference resolution, event references, time-dependency, and answer types.
Revised: We deﬁne a taxonomy with four categories, including ambiguity in coreference resolution, event references, time-dependency, and answer types.

Original: According to the two ambiguity types introduced in Ginzburg [1996], the ambiguity in coreference resolution is about the question focus and the ambiguity in answer types is about the answering possibilities; the ambiguity in event references and time-dependency cover the both ambiguity types.
Revised: According to the two ambiguity types introduced in Ginzburg [1996], the ambiguity in coreference resolution is about the question focus and the ambiguity in answer types is about the answering possibilities; the ambiguity in event references and time-dependency cover the both ambiguity types.

Original: In comparison to Min et al.
Revised: In comparison to Min et al.

Original: [2020], who studies ambiguity in open-domain questions, our [Table] everyone doing
Revised: [2020], who studies ambiguity in open-domain questions, our corpus contains one new ambiguity type – coreference resolution which is an inherent challenge in conversations.

Original: [Table] Riding a big wheel down [Table]
Revised: In addition, diﬀerent from open-domain questions where more than a tier have the ambiguity in event references, it is actually a minor class in conversational questions since requested events are under the scope of the given story.

Original: driveway into the street
Revised: In most cases, ambiguity is not apparent from the prompt question alone, but only after referring to the conversation history and researching all possible answers in the story.

Original: [Table]
Revised: Clariﬁcation Strategies.

Original: screaming [Table] yelling [Table]
Revised: We classify the clariﬁcation questions in three types:

Original: help
Revised: More

Original: [Table]
Revised: Information, Selection and Check.

Original: crying
Revised: Our taxonomy follows Kato et al. [2013]’s work which

Original: [Table]
Revised: [Table]

Original: His ninth-grade English class for boys centers on books. ”
Revised: His ninth-grade English class for boys centers on books. ”

Original: The novels they’re reading now, are very manly novels.
Revised: The novels they’re reading now, are very manly novels.

Original: They’re novels that deal with the arrogance of man and the pride of man.”
Revised: They’re novels that deal with the arrogance of man and the pride of man.”

Original: One of those books, for example, is ”The Call of the Wild”...
Revised: One of those books, for example, is ”The Call of the Wild”..

Original: [Table] Who does he teach?
Revised: Q i − 2 : Who does he teach?

Original: A i − 2 : Boys.
Revised: A i − 2 : Boys.

Original: Q i − 1 : What are his pupils doing?
Revised: Q i − 1 : What are his pupils doing?

Original: A i − 1 : They’re reading.
Revised: A i − 1 : They’re reading.

Original: Q i : What?
Revised: Q i : What?

Original: Clariﬁcation question : The type of book or an example of a book they’re reading?
Revised: Clariﬁcation question : The type of book or an example of a book they’re reading?

Original: Clariﬁcation reply #1 : The type of book.
Revised: Clariﬁcation reply #1 : The type of book.

Original: A i #1 : Novels that deal with the arrogance of man and the pride of man.
Revised: A i #1 : Novels that deal with the arrogance of man and the pride of man.

Original: Clariﬁcation reply #2 : An example of a book.
Revised: Clariﬁcation reply #2 : An example of a book.

Original: A i #2 : ”The Call of the Wild”.
Revised: A i #2 : ”The Call of the Wild”.

Original: [Table]
Revised: [Table]

Original: references
Revised: references

Original: [Table]
Revised: [Table]

Original: question
Revised: Was

Original: [Table]
Revised: [Table]

Original: Do you mean identiﬁed by whom
Revised: Do you [Table] whom

Original: [Table]
Revised: [Table]

Original: police.
Revised: police.

Original: [Table]
Revised: [Table]

Original: Yes.
Revised: Yes.

Revised: Clariﬁcation

Original: [Table]
Revised: [Table]

Original: I 
Revised: I mean

Original: [Table]
Revised: [Table]

Original: CNN.
Revised: CNN.

Original: [Table]
Revised: [Table]

Original: corpus contains one new ambiguity type – coreference resolution which is an inherent challenge in conversations. In addition, diﬀerent from open-domain questions where more than a tier have the ambiguity in event references, it is actually a minor class in conversational questions since requested events are under the scope of the given story. In most cases, ambiguity is not apparent from the prompt question alone, but only after referring to the conversation history and researching all possible answers in the story.
Revised: i #

Original: Clariﬁcation Strategies.
Revised: [Table]

Original: We classify the clariﬁcation questions in three types: More Information, Selection and Check. Our taxonomy follows Kato et al.
Revised: No.

Original: [2013]’s work which classiﬁes clariﬁcation requests of users in six categories, however, we only consider three types among them since we focus on the clariﬁcation strategy rather than the user intent.
Revised: classiﬁes clariﬁcation requests of users in six categories, however, we only consider three types among them since we focus on the clariﬁcation strategy rather than the user intent.

Original: Check aims to conﬁrm a hypothesis corresponding to the ambiguity ( e.g. , the second example in Table 2); Selection aims to request an answer from two or more possibilities about the ambiguity ( e.g. , the ﬁrst and third example in Table 2); More Information directly asks for further details for clarifying the ambiguity ( e.g. , the last example in Table 2).
Revised: Check aims to conﬁrm a hypothesis corresponding to the ambiguity ( e.g. , the second example in Table 2); Selection aims to request an answer from two or more possibilities about the ambiguity ( e.g. , the ﬁrst and third example in Table 2); More Information directly asks for further details for clarifying the ambiguity ( e.g. , the last example in Table 2).

Revised: We compute the distribution of clariﬁcation strategies in 50 random ambiguous items: Check , Selection and More Info respectively occupies 47%, 43% and 10% of samples.

Original: As shown in Figure 2(a), people prefer verifying their hypotheses ( e.g. , Check , Selection ) rather than asking open questions ( e.g. , More Info ) for clarifying the ambiguity.
Revised: It shows that people prefer verifying their hypotheses ( e.g. , Check , Selection ) rather than asking open questions ( e.g. , More Info ) for clarifying the ambiguity.

Original: With respect to the number ofreplies to the  clariﬁcation question, we report the distribution in Figure 2(b).
Revised: In addition, we compute the distribution of the number of clariﬁcation replies in all ambiguous items.

Original: Most clariﬁcation questions have two diﬀerent answers; 11% of them have only one reply; and 15% of ambiguous questions have more than two.
Revised: Most clariﬁcation questions have two diﬀerent replies (74%); 11% of them have only one reply 3 ; and 15% of them have more than two replies.

Original: 6. Models
Revised: 6. Models

Original: To set initial performance levels on Abg-CoQA, we present a baseline model for each task.
Revised: To set initial performance levels on Abg-CoQA, we present a baseline model for each task.

Original: These tasks cover both the conversational question answering and language generation.
Revised: These tasks cover both the conversational question answering and language generation.

Original: Ambiguity Detection.
Revised: Ambiguity Detection.

Original: We formulate this task as the traditional question answering task by adding ”ambiguous” as a possible prediction output.
Revised: We formulate this task as the traditional question answering task by adding “ambiguous” as a possible prediction output.

Original: We consider two extractionbased models which have shown promising results for generating conversational responses on the CoQA dataset as the baseline models for this task.
Revised: We consider two extractionbased models which have shown promising results for generating conversational responses on the CoQA dataset as the baseline models for this task.

Original: Our baseline models are respectively build upon BERT [Devlin et al., 2019] and XLNET [Yang et al., 2019] plus prediction heads for each answer type 1 (respectively called BERT-AnsType and XLNET-AnsType).
Revised: Our baseline models are respectively build upon BERT [Devlin et al., 2019] and XLNET [Yang et al., 2019] plus prediction heads for each answer type 4 (respectively called BERT+AnsType [Wu et al., 2019] and XLNET+AnsType 5 ).

Original: In order to take the ambiguity of questions into consideration, we append the “ambiguous” token at the end of the input passage.
Revised: In order to take the ambiguity of questions into consideration, we append the “ambiguous” token at the end of the input passage so that models are able to extract it as the prediction result 6 .

Original: Therefore, the input to the model is the passage appended by “ambiguous”, the conversation history and the question, and the expected output is the speciﬁc ”ambiguous” token when the question is ambiguous; or the original response to the question when it is not ambiguous.
Revised: Therefore, the input to the model is the passage appended by “ambiguous”, the conversation history and the question, and the expected output is the speciﬁc ”ambiguous” token when the question is ambiguous; or the original response to the question when it is not ambiguous.

Original: Clariﬁcation Question Generation.
Revised: Clariﬁcation Question Generation.

Original: We ﬁne tune a strong model for text generation – BART [Lewis et al., 2020] on our corpus as the baseline model for generating clariﬁcation questions.
Revised: We ﬁne tune a strong model for text generation – BART [Lewis et al., 2020] on our corpus as the baseline model for generating clariﬁcation questions.

Original: Prior work on BART demonstrates its eﬀectiveness when ﬁne tuned for news summarization.

Original: We append the given conversation history and the current question to the text passage and feed it into BART.
Revised: We append the given conversation history and the current question to the text passage and feed it into BART.

Original: The expected output is the clariﬁcation question.
Revised: The expected output is the clariﬁcation question.

Original: Since the ambiguous samples is in a small amount ( i.e. , 1k), we also consider adding an additional ﬁne-tuning prior to this clariﬁcation question generation task. In order to make the model learn to generate conversational questions, we ﬁrst ﬁne tune BART for generating the next question given the conversation history on the CoQA dataset (excluding the test set of Abg-CoQA). Then we ﬁne tune the model on ambiguous samples for generating clariﬁcation questions.
Revised: Since the ambiguous samples are in a small amount ( i.e. , 1k), we also consider adding an additional ﬁne-tuning prior to this clariﬁcation question generation task: generating the next question given the conversation history on the CoQA dataset (excluding the test set of Abg-CoQA) in order to make the model learn to generate common conversational questions.

Original: Clariﬁcation-based Question Answering.
Revised: Clariﬁcation-based Question Answering.

Original: We formulate this task as the original conversational question answering task by considering the clariﬁcationturn as the previous conversation history.
Revised: We formulate this task as the original conversational question answering task by considering the clariﬁcation as the previous conversation history.

Original: In this task, we append the clariﬁcationturn ( i.e. , a clariﬁcation question and one possible reply) to the passage and the conversation as the input sequence to the model.
Revised: In this task, we append the clariﬁcation question and one possible reply to the passage and the conversation as the input sequence to the model.

Original: The expected output is the answer to the originally ambiguous question based on the clariﬁcation.
Revised: The expected output is the answer to the originally ambiguous question based on the clariﬁcation.

Original: We consider three diﬀerent types of models as our baseline: the BERT-based model with answer veriﬁcation (BERT+AnsType) which is also used forprevious tasks, the XLNET model [Yang et al., 2019] with answer type prediction (XLNET+AnsType) which is a more powerful language model than BERT on question answering, and a generative model GPT-

Revised: We consider three diﬀerent types of models as our baseline: BERT+AnsType [Wu et al., 2019] and XLNET+AnsType which are also used for the task of ambiguity detection and a generative model GPT-

Original: [Radford et al.
Revised: [Radford et al.

Original: for zero-shot prediction.
Revised: for zero-shot prediction.

Original: For BERT+AnsType and XLNET+AnsTyp, we ﬁrst pre-train them on CoQA then ﬁne-tune on Abg-CoQA by adding a clariﬁcationturn into the conversation.
Revised: For BERT+AnsType and XLNET+AnsType, we ﬁrst pre-train them on CoQA then ﬁne-tune on Abg-CoQA by adding the clariﬁcation into the conversation.

Revised: 7. Evaluation

Revised: We ﬁrst evaluate the inter-rater agreement of annotations and discuss the subjectivity of clarifying ambiguity.

Revised: Then we report the performance of several state-of-the-art models on each task as well as the take-away observations.

Original: 7. Evaluation 7.1 Evaluation Metric
Revised: 7.1 Evaluation Metric

Original: For answer generation, we use the same metric as CoQA: macro-average F1 score of word overlap.
Revised: For answer generation, we use the same metric as CoQA: macro-average F1 score of word overlap.

Original: For computing a model’s performance, each individual prediction is compared against n human answers resulting in n F1 scores, the maximum of which is chosen as the prediction’s F1.
Revised: For computing a model’s performance, each individual prediction is compared against n human answers resulting in n F1 scores, the maximum of which is chosen as the prediction’s F1.

Original: For each question, we average out F1 across these n sets, both for humans and models.
Revised: For each question, we average out F1 across these n sets, both for humans and models.

Original: We follow the same way as CoQA for ﬁxing the bias when computing human performance.
Revised: We follow the same way as CoQA for ﬁxing the bias when computing human performance.

Original: In our evaluation on clariﬁcation-based question answering, n is equal to 3.
Revised: In our evaluation on clariﬁcation-based question answering, n is equal to 3.

Original: [Table]

Original: For detecting ambiguity, we compute the precision, recall and F1score as the evaluation metric on the two-class classiﬁcation.
Revised: For detecting ambiguity, we compute the precision, recall and F1 as the evaluation metric on the two-class classiﬁcation.

Original: We also report the macro-average F1score on answer generation ( n = 1) sincewe formulate predicting ”ambiguous”  as extracting an answer span from the passage.
Revised: We also report the macro-average F1 on answer generation ( n = 1) since predicting ”ambiguous” is formulated as extracting an answer span.

Original: For clariﬁcation question generation, we use BLEU scores as the main metric with a gold standard set of two human annotations.
Revised: For clariﬁcation question generation, we use BLEU scores as the main metric with the human-annotated clariﬁcation question as the gold standard.

Revised: In addition to measuring the similarity with BLEU, we manually evaluate whether the generated question is reasonable and helpful for clarifying the existing ambiguity.

Original: 7.2 Inter-rater Agreement
Revised: 7.2 Inter-rater Agreement

Original: For measuring the inter-rater agreement on whether a question is ambiguous or not, we compute the Cohen’s Kappa score on 68 randomly selected samples which are annotated by two Amazon Turk workers.
Revised: For measuring the inter-rater agreement on whether a question is ambiguous or not, we compute the Cohen’s Kappa score on 68 randomly selected samples which are annotated by two Amazon Turk workers.

Original: The Cohen’s Kappa score on the ambiguity detection is equal to 0.26, which shows a fair agreement. The key reason is that the ambiguity is subjective and personalized.
Revised: The Cohen’s Kappa score on the ambiguity detection is equal to 0.26 (a fair agreement), indicating the task is subjective and challenging.

Original: In order to make it more objective, we randomly select 100 ambiguous cases (based on  theprevious annotation) in developmentset and ask three annotators to write an answer to each ambiguous question.
Revised: For measuring the inter-rater agreement on clariﬁcation questions, we ask a second Amazon Turk worker for writing a clariﬁcation question (and all possible replies) on ambiguous samples of the development and test sets.

Original: Then we compute the macro-average F1 score of word overlap as a way to measuring the human agreement on answering ambiguous question.
Revised: We compute the macro-average F1 score of word overlap and the BLEU scores.

Original: The F1 score is 65.3%, which is 23.5 points behind 88.8% F1 reported by CoQA, which reveals that ambiguous questions identiﬁed by our annotators are indeed diﬃcult for human to provide consistent answers.
Revised: The F1 score is 45.3% and the BLEU-4 score is 21.9 on the test set (more BLEU scores are reported in Table 4). [Table] [Table]

Original: Formeasuring the inter-rater agreement on clariﬁcation questions, we ask a second
Revised: For the clariﬁcation-based question answering

Original: Amazon Turk worker for annotating the clariﬁcation turn on ambiguous samples of the development and test sets. The F1 score is 45.3% and the BLEU-4 score is 21.9% in the test set (more BLEU scores are shown in Table 4). It is not surprising since there are diﬀerent clariﬁcation strategies (Figure 2(a)) and annotators may have their own preference.
Revised: [Table]

Original: For the clariﬁcation-based question answering, we collect three annotations for each sample of the test set.
Revised: we ask three annotators to answer each ambiguous question in the test set after considering the clariﬁcation.

Original: The macro-average F1 score  is equal to 75.2% and it shows the human performance on this task.
Revised: The macro-average F1 score of these three annotations is equal to 75.2%. 7.3 Subjectivity of Clarifying Ambiguity

Original: Our study shows that  the clariﬁcation increases 10 points for theinter-rater agreement(measured by F1 score) on answering ambiguous questions.
Revised: The inter-rater agreement score on ambiguity detection indicates that it is a subjective task. In order to verify if this subjectivity does not hurt the quality of annotations [Table] we evaluate them through the agreement on answering ambiguous questions.

Original: We didn’t train workers to write answers in the same style ( e.g. , concise, short answers), so it  is normal that the  F1 score is lower than one reported in CoQA.
Revised: We randomly selectambiguous cases (based on the original annotation) in development set and ask three annotators to write an answer to each ambiguous question. We compute the macro-average F score of word overlap as a way to measuring the human agreement on answering ambiguous question, which is 65.3%. Then we compare it with the F1 score of the clariﬁcation-based question answering which is 75.2%. It shows that the clariﬁcation increases 10 points for the inter-rater agreement (measured by F1 score) on answering ambiguous questions, which demonstrates that those questions are indeed ambiguous and the clariﬁcation is eﬀective.

Revised: How to formulate a clariﬁcation question is also subjective since there are diﬀerent clariﬁcation strategies (Section 5) and annotators may use diﬀerent expressions for clarifying the same ambiguous doubt.

Revised: For verifying this hypothesis, we randomly select 50 samples and manually compare the diﬀerence between the two clariﬁcation questions for each sample.

Revised: We observe that more than a half (57%) of them ask additional information for the same ambiguous doubt by using diﬀerent expressions or clarifying strategies, which supports our hypothesis.

Revised: Besides, only 17% of them are exactly the same and 26% of them target diﬀerent ambiguity.

Original: 7.3 Results and Discussion
Revised: 7.4 Results and Discussion

Original: We report the experimental results of baseline models on our deﬁned three tasks.
Revised: We report the experimental results of baseline models on our deﬁned three tasks.

Original: The detailed experimental setting is introduced in Appendix B.
Revised: The detailed experimental setting is introduced in Appendix B.

Original:  Ambiguity Detection.
Revised: 7.4.1 Ambiguity Detection.

Original:  Since we consider ambiguity detection in the question answering setting, we report both the performance on answer prediction and on ambiguity detection in Table 3.
Revised: It is a challenging task for the state-of-the-art question answering models. [Table] [Table] Since we consider ambiguity detection in the question answering setting

Original: With respect to the performance on question answering, the model trained on Abg-CoQA acheives an F1 score equal to 31.4% for BERT-based and 45.5% for

Original: [Table]
Revised: [Table]

Revised: we report both the performance on answer prediction and on ambiguity detection in Table 3.

Revised: With respect to the performance on question answering

Original: [Table]
Revised: [Table]

Original:  XLNET-based.
Revised: the model trained on Abg-CoQA acheives an F1 score equal to 31.4% for BERT-based and 45.5% for XLNET-based.

Original: It is not surprising because the baseline model failed on these samples when trained on CoQA (samples were selected by the pre-selection process; original average F score on them is only 3.8%).
Revised: It is not surprising because the baseline model failed on these samples when trained on CoQA (samples were selected by the pre-selection process; original average F1 score on them is only 3.8%).

Original: The best ambiguity detection performance gets a F1 score of 23.6%, which reveals the challenging of this task.
Revised: The best ambiguity detection performance gets a F1 score of 23.6%, which reveals the challenging of this task.

Original: We observe that the XLNET+AnsType baseline has a higher precision while the BERT+AnsType model has a higher recall.
Revised: We observe that the XLNET+AnsType baseline has a higher precision while the BERT+AnsType model has a higher recall.

Original: Clariﬁcation Question Generation.
Revised: 7.4.2 Clarification Question Generation.

Revised: Pre-training on CoQA on the question generation task helps but doesn’t solve it.

Revised: Human evaluation is necessary besides automatic metrics.

Original: Results for the clariﬁcation question generation task is shown in Table 4.
Revised: Results for the clariﬁcation question generation task is shown in Table 4.

Revised: For determining whether the generated question is reasonable/helpful for clarifying the existing ambiguity, we randomly select 50 samples in the test set and manually evaluate them.

Revised: We report the accuracy in Table 5 for both the general performance and each ambiguity type.

Original: We ﬁnd that pre-training on CoQA on the question generation taskdoesn’t help improve the performance, even slightly worse than directly ﬁne-tuning BART on Abg-CoQA.
Revised: According to the BLEU metrics, pre-training on CoQA on the question generation task slightly improves the performance (by around 2 points) comparing to directly ﬁne-tuning BART on Abg-CoQA.

Original: We see a great gap between model and human performance.
Revised: However, human evaluation shows that the improvement is greatly signiﬁcant: gaining 26% points of accuracy. This observation reveals that though BLEU metrics are correlated with human evaluation, the scaling is not a straightforward signal for the comparison in this case. The best performing model gets an accuracy score of 56%, which shows that there is still a gap comparing to the human performance.

Original: The diﬃculty mainly comes from identifying the ambiguous point in the question, so that the system can correctly generate a clariﬁcation question targeting the ambiguity.
Revised: By looking into each ambiguity type, we observe that models are generally good at clarifying the ambiguity of event references, while the ambiguity of answer types is the most diﬃcult for models. Interestingly, pre-training BART on CoQA signiﬁcantly helps the model generating clariﬁcation questions when the ambiguity type is time-dependency (from 0.0% accuracy to 71.4%).

Original: Clariﬁcation-based Question Answering.
Revised: 7.4.3 Clarification-based Question Answering

Revised: It demands more reasoning for models than common question answering.

Revised: The model built upon XLNET achieves the best performance but still largely behind human.

Original: Results for the answer prediction after the clariﬁcation is shown in Table 5.
Revised: Results for the answer prediction after the clariﬁcation is shown in Table 6.

Original: The model built upon XLNET achieves the best performance, however, still 35.1% behind the human performance, which shows that our task brings a new challenge to the question answering community.
Revised: The model built upon XLNET achieves the best performance, however, still 35.1% behind the human performance, which shows that our task brings a new challenge to the question answering community.

Original: Even though samples in Abg-CoQA were pre-selected by the BERT-based baseline model, we don’t see a great diﬀerence of performance between BERT and XLNET on this task.
Revised: Even though samples in Abg-CoQA were pre-selected by the BERT-based baseline model, we don’t see a great diﬀerence of performance between BERT and XLNET on this task.

Original: It demonstrates that the task is not biased towards the pre-processing of the BERT-based model.
Revised: It demonstrates that the task is not biased towards the pre-processing of the BERT-based model.

Original: We also run GPT-2 as a representative of generative models on zero-shot prediction.
Revised: We also run GPT-2 as a representative of generative models on zeroshot prediction.

Original: Its performance decreases more than 40 points comparing to the reported F1 score (55%) on CoQA
Revised: Its performance decreases more than 40 points comparing to the reported F1 score (55%) on CoQA

Original: [Radford et al.].
Revised: [Radford et al.].

Original: We conduct an error analysis and ﬁnd that existing strong models on standard conversational question answering tasks actually can’t correctly answer the question based on diﬀerent clariﬁcation replies.
Revised: We conduct an error analysis and ﬁnd that existing strong models on standard conversational question answering tasks actually can’t correctly answer the question based on diﬀerent clariﬁcation replies.

Original: For example, a question asks ”what is the color of the book?”

Original: and the story mentions two books respectively in red and green, thus the question is ambiguous.

Original: A clariﬁcation question is asked ”Do you mean the ﬁrst book or the second?”. Models always predict ”green” no matter the clariﬁcation reply  is ”the ﬁrst” or ”the second”.
Revised: For example in Fig 1, models always predict ”green” no matter the clariﬁcation reply to ”Do you mean the ﬁrst book?” is ”Yes” or ”No, the second”.

Original: It reveals that current models may be saturated to the training distribution rather than truly understand the conversation.
Revised: It reveals that current models may be saturated to the training distribution rather than truly understand the conversation.

Original: 8. Conclusions
Revised: 8. Conclusions

Original:  We introduce Abg-CoQA, anovel dataset  for clarifying ambiguity in Conversational Question Answering systems.
Revised: Our empirical study and computational experiments show that identifying and resolving ambiguities in a conversation is rather a challenging task even for people. We discuss the success and limitations of the state-of-the-art end-to-end neural models and present a dataset that can be used for further investigating solutions for managing ambiguities in conversational question answering. A direct application of pre-trained large models, e.g.

Original: Our empirical study shows that it is challenging to identify ambiguity in a information-seeking conversation and generate clariﬁcation question.
Revised: BERT, BART, is not able to identify ambiguities or to generate clariﬁcation questions for resolving the ambiguity.

Original: We propose clariﬁcation-based question answering as a benchmark task for evaluating the robustness of existing conversational question answering systems. We compare the performance of various models on this task and conclude that more research in conversational modeling is needed even though the performance on certain existing datasets is saturated.
Revised: Future research developing on Abg-CoQA models may include modeling uncertainty as a signal of ambiguity and generating clariﬁcation questions in several steps, such as ﬁrst identifying the ambiguity type, then generating a contextual phrase which targets the ambiguity, ﬁnally formulating the complete question. 9. Ethics We have not worked with sensitive or personal data.

Original: Appendix A. Examples of Annotation Interface
Revised: The presented dataset is fully anonymized. Our work relies on pretrained models. These models are known to reproduce and magnify societal bias that is presented to them in the training process. Moreover, like other deep learning methods, the presented models are likely to perform better for content that is better represented in training, leading to further bias against marginalized groups.

Revised: We hope that general methods to mitigate harms from ML bias can address these issues.

Revised: Appendix A. Examples of Annotation Interface

Original: Figure 3 and Figure 4 show examples of the interface for AMT annotators.
Revised: Figure 2 and Figure 3 show examples of the interface for AMT annotators.

Original: Appendix B. Experimental Setup
Revised: Appendix B. Experimental Setup

Original: For the BERT-based model, we adapt the SogouMRCToolkit 2 to our dataset and use its setting: batch size of 6, number epoch of 10, warm-up proportion of 0.1.
Revised: For the BERT-based model, we adapt the SogouMRCToolkit 7 to our dataset and use its setting: batch size of 6, number epoch of 10, warm-up proportion of 0.1.

Original: We use the uncased-base BERT model as the backbone.
Revised: We use the uncased-base BERT model as the backbone.

Original: The models are optimized using Adamax, with a learning rate of 3e-5.
Revised: The models are optimized using Adamax, [Table] Was Andorra part of Roman Hispania? A i-1 :

Revised: yes

Original: For the XLNET-based model, we adapt the XLNET extension toolkit 3 .
Revised: After reading the above text, you need to answer the following questions. The next question Q i after the above conversation is "Name another area that was part of that region.". Is it an ambiguous question? Ambiguous (you are not sure what the question is asking.)

Revised: Non-ambiguous (the question is clear.)

Revised: Please write the answer to the question Q i according to the story and the conversation history.

Revised: submit with a learning rate of 3e-5.

Revised: For the XLNET-based model, we adapt the XLNET extension toolkit 8 .

Original: The batch size is 8, the number of training steps is 6000.
Revised: The batch size is 8, the number of training steps is 6000.

Original: The model is optimized using Adam, with a learning rate of 3e-5.
Revised: The model is optimized using Adam, with a learning rate of 3e-5.

Original: For the GPT-2, we follow its reported setting on the zero-shot CoQA task [Reddy et al., 2019]: add “Q” before each conversational question and clariﬁcation question and “A” before each answer as well as the end of the input sequence.
Revised: For the GPT-2, we follow its reported setting on the zero-shot CoQA task [Reddy et al., 2019]: add “Q” before each conversational question and clariﬁcation question and “A” before each answer as well as the end of the input sequence.

Original: For ﬁne-tuning the BART model, we use the Fairseq toolkit [Ott et al., 2019].
Revised: For ﬁne-tuning the BART model, we use the Fairseq toolkit [Ott et al., 2019].

Original: We use the pre-trained large model, with Adam optimizer and learning rate of 3e-05.
Revised: We use the pre-trained large model, with Adam optimizer and learning rate of 3e-05.

Original: The total number of training steps is 2000 and the number of warm-up steps is 50.
Revised: The total number of training steps is 2000 and the number of warm-up steps is 50.

Original:  For generative models, i.e. , BART and GPT-2, we only considerthe generated ﬁrst sentence for evaluation since the number of tokens in generated text is in general deﬁned larger than the ground truth. Before you accept this HIT
Revised: We follow the default decoding strategy by setting the beam size as 5 and removing duplicated trigrams in beam search. For generative models, i.e. , BART and GPT-2, we only consider

Original: [Table]
Revised: [Table]

Original: please read the following instructionscarefully [Table] If you have read our instructions and tutorial and would like to continue, please answer the following questions
Revised: instructions

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: submit Before [Table] accept this HIT
Revised: submit the generated ﬁrst sentence for evaluation since the number of tokens in generated text is in general deﬁned larger than the ground truth.

Original: [Table]

Original: please read the following instructions carefully

Original: [Table]

Original: [Figure]