Original: T OWARDS
Revised: T OWARDS

Original: D ATA
Revised: D ATA

Original: D ISTILLATION
Revised: D ISTILLATION

Original: FOR
Revised: FOR

Original: E ND - TO - END
Revised: E ND - TO - END

Original: S POKEN
Revised: S POKEN

Original: C ONVERSATIONAL Q UESTION A
Revised: C ONVERSATIONAL Q UESTION A

Original: NSWERING
Revised: NSWERING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In spoken question answering, QA systems are designed to answer questions from contiguous text spans within the related speech transcripts.
Revised: In spoken question answering, QA systems are designed to answer questions from contiguous text spans within the related speech transcripts.

Original: However, the most natural way that human seek or test their knowledge is via human conversations.
Revised: However, the most natural way that human seek or test their knowledge is via human conversations.

Original: Therefore, we propose a new S poken C onversational Q uestion A nswering task (SCQA), aiming at enabling QA systems to model complex dialogues ﬂow given the speech utterances and text corpora.
Revised: Therefore, we propose a new S poken C onversational Q uestion A nswering task (SCQA), aiming at enabling QA systems to model complex dialogues ﬂow given the speech utterances and text corpora.

Original: In this task, our main objective is to build a QA system to deal with conversational questions both in spoken and text forms, and to explore the plausibility of providing more cues in spoken documents with systems in information gathering.
Revised: In this task, our main objective is to build a QA system to deal with conversational questions both in spoken and text forms, and to explore the plausibility of providing more cues in spoken documents with systems in information gathering.

Original: To this end, instead of adopting automatically generated speech transcripts with highly noisy data, we propose a novel uniﬁed data distillation approach, DDNet , which directly fuse audio-text features to reduce the misalignment between automatic speech recognition hypotheses and the reference transcriptions.
Revised: To this end, instead of adopting automatically generated speech transcripts with highly noisy data, we propose a novel uniﬁed data distillation approach, DDNet , which directly fuse audio-text features to reduce the misalignment between automatic speech recognition hypotheses and the reference transcriptions.

Original: In addition, to evaluate the capacity of QA systems in a dialogue-style interaction, we assemble a Spoken Co nversational Q uestion
Revised: In addition, to evaluate the capacity of QA systems in a dialogue-style interaction, we assemble a Spoken Co nversational Q uestion

Original: A nswering (Spoken-CoQA) dataset with more than 120k question-answer pairs.
Revised: A nswering (Spoken-CoQA) dataset with more than 120k question-answer pairs.

Original: Experiments demonstrate that our proposed method achieves superior performance in spoken conversational question answering.
Revised: Experiments demonstrate that our proposed method achieves superior performance in spoken conversational question answering.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Conversational Machine Reading Comprehension (CMRC) has been studied extensively over the past few years within the natural language processing (NLP) communities (Zhu et al., 2018;
Revised: Conversational Machine Reading Comprehension (CMRC) has been studied extensively over the past few years within the natural language processing (NLP) communities (Zhu et al., 2018;

Original: Liu et al., 2019; Yang et al., 2019).
Revised: Liu et al., 2019; Yang et al., 2019).

Original: Different from traditional MRC tasks, CMRC aims to enable models to learn the representation of the context paragraph and multi-turn dialogues.
Revised: Different from traditional MRC tasks, CMRC aims to enable models to learn the representation of the context paragraph and multi-turn dialogues.

Original: Existing methods to the conversational question answering (QA) tasks (Huang et al., 2018a; Devlin et al., 2018; Xu et al., 2019;
Revised: Existing methods to the conversational question answering (QA) tasks (Huang et al., 2018a; Devlin et al., 2018; Xu et al., 2019;

Original: Gong et al., 2020) have achieved superior performances on several benchmark datasets, such as QuAC (Choi et al., 2018) and CoQA (Elgohary et al., 2018).
Revised: Gong et al., 2020) have achieved superior performances on several benchmark datasets, such as QuAC (Choi et al., 2018) and CoQA (Elgohary et al., 2018).

Original: However, few studies have investigated CMRC in both spoken content and text documents.
Revised: However, few studies have investigated CMRC in both spoken content and text documents.

Original: To incorporate spoken content into machine comprehension, there are few public datasets that evaluate the effectiveness of the model in spoken question answering (SQA) scenarios.
Revised: To incorporate spoken content into machine comprehension, there are few public datasets that evaluate the effectiveness of the model in spoken question answering (SQA) scenarios.

Original: TOEFL listening comprehension (Tseng et al., 2016) is one of the related corpus for this task, an English test designed to evaluate the English language proﬁciency of non-native speakers.
Revised: TOEFL listening comprehension (Tseng et al., 2016) is one of the related corpus for this task, an English test designed to evaluate the English language proﬁciency of non-native speakers.

Original: But the multi-choice question answering setting and its scale is limited to train for robust SCQA models.
Revised: But the multi-choice question answering setting and its scale is limited to train for robust SCQA models.

Original: The rest two spoken question answering datasets are Spoken-SQuAD (Li et al., 2018) and ODSQA (Lee et al., 2018), respectively.
Revised: The rest two spoken question answering datasets are Spoken-SQuAD (Li et al., 2018) and ODSQA (Lee et al., 2018), respectively.

Original: However, there is usually no connection between a series of questions and answers within the same spoken passage among these datasets.
Revised: However, there is usually no connection between a series of questions and answers within the same spoken passage among these datasets.

Original: More importantly, the most common way people seek or test their knowledge is via human conversations, which capture and maintain the common ground in spoken and text context from the dialogue ﬂow.
Revised: More importantly, the most common way people seek or test their knowledge is via human conversations, which capture and maintain the common ground in spoken and text context from the dialogue ﬂow.

Original: There are many real-world applications related to SCQA tasks, such as voice assistant and chat robot.
Revised: There are many real-world applications related to SCQA tasks, such as voice assistant and chat robot.

Original: In recent years, neural network based methods have achieved promising progress in speech processing domain.
Revised: In recent years, neural network based methods have achieved promising progress in speech processing domain.

Original: Most existing works ﬁrst select a feature extractor (Gao et al., 2019), and then enroll the feature embedding into the state-of-the-art learning framework, as used in single-turn spoken language processing tasks such as speech retrieval (Lee et al., 2015; Fan-Jiang et al., 2020;
Revised: Most existing works ﬁrst select a feature extractor (Gao et al., 2019), and then enroll the feature embedding into the state-of-the-art learning framework, as used in single-turn spoken language processing tasks such as speech retrieval (Lee et al., 2015; Fan-Jiang et al., 2020;

Original: Karakos et al., 2020), translation (B´erard et al., 2016; Di Gangi et al., 2020; Tu et al., 2020) and recognition (Zhang et al., 2017; Zhou et al., 2018; Bruguier et al., 2019; Siriwardhana et al., 2020).
Revised: Karakos et al., 2020), translation (B´erard et al., 2016; Di Gangi et al., 2020; Tu et al., 2020) and recognition (Zhang et al., 2017; Zhou et al., 2018; Bruguier et al., 2019; Siriwardhana et al., 2020).

Original: 𝑆 = { 𝑠 ! , 𝑠 " , ⋯ , 𝑠 # } … … … … … …
Revised: 𝑆 = { 𝑠 ! , 𝑠 " , ⋯ , 𝑠 # } … … … … … …

Original: Viterbi Decoding
Revised: Viterbi Decoding

Original: A c o u s t
Revised: A c o u s t

Original: i c m o d e l
Revised: i c m o d e l

Original: A S R
Revised: A S R

Original: S y s t
Revised: S y s t

Original: e m
Revised: e m

Original: Spoken Documents
Revised: Spoken Documents

Original: ASR Context:
Revised: ASR Context:

Original: Once upon a time in a bar near farmhouse, there lived a little like captain named cotton.
Revised: Once upon a time in a bar near farmhouse, there lived a little like captain named cotton.

Original: How to live tied up in a nice warm place above the bar and we're all ofthe farmers horses slapped…
Revised: How to live tied up in a nice warm place above the bar and we're all ofthe farmers horses slapped…

Original: Conversational MRC models 𝑨 𝟐 : white 𝐀𝐒𝐑−𝐐 𝟑 : What color was caught in?
Revised: Conversational MRC models 𝑨 𝟑 : white 𝐀𝐒𝐑−𝐐 𝟑 : What color was caught in?

Original: Dialogue History 𝐀𝐒𝐑−𝐐 𝟏 : What color was caught in?
Revised: Dialogue History 𝐀𝐒𝐑−𝐐 𝟏 : What color was caught in?

Original: Machine Reasoning 𝐀 𝟏 : white 𝐀 𝟐 : in a barn 𝐀𝐒𝐑−𝐐 𝟐 : What did she live?
Revised: Machine Reasoning 𝐀 𝟏 : white 𝐀 𝟐 : in a barn 𝐀𝐒𝐑−𝐐 𝟐 : What did she live?

Original: Viterbi Decoding
Revised: Viterbi Decoding

Original: A c o u s t
Revised: A c o u s t

Original: i c m o d e l
Revised: i c m o d e l

Original: A S R
Revised: A S R

Original: S y s t
Revised: S y s t

Original: e m
Revised: e m

Original: Spoken Questions 𝑄 = { 𝑞 ! , 𝑞 " , ⋯ , 𝑞 # }
Revised: Spoken Questions 𝑄 = { 𝑞 ! , 𝑞 " , ⋯ , 𝑞 # }

Original: [Table]
Revised: [Table]

Original: However, simply adopting existing methods to the SCQA tasks will cause several challenges.
Revised: However, simply adopting existing methods to the SCQA tasks will cause several challenges.

Original: First, transforming speech signals into ASR transcriptions is inevitably associated with ASR errors (See
Revised: First, transforming speech signals into ASR transcriptions is inevitably associated with ASR errors (See

Original: Table 2).
Revised: Table 2).

Original: Previous work (Lee et al., 2019) shows that directly feed the ASR output as the input for the following down-stream modules usually cause signiﬁcant performance loss, especially in SQA tasks.
Revised: Previous work (Lee et al., 2019) shows that directly feed the ASR output as the input for the following down-stream modules usually cause signiﬁcant performance loss, especially in SQA tasks.

Original: Second, speech corresponds to a multi-turn conversation (e.g. lectures, interview, meetings), thus the discourse structure will have more complex correlations between questions and answers than that of a monologue.
Revised: Second, speech corresponds to a multi-turn conversation (e.g. lectures, interview, meetings), thus the discourse structure will have more complex correlations between questions and answers than that of a monologue.

Original: Third, additional information, such as audio recordings, contains potentially valuable information in spoken form.
Revised: Third, additional information, such as audio recordings, contains potentially valuable information in spoken form.

Original: Many QA systems may leverage kind of orality to generate better representations.
Revised: Many QA systems may leverage kind of orality to generate better representations.

Original: Fourth, existing QA models are tailored for a speciﬁc (text) domain.
Revised: Fourth, existing QA models are tailored for a speciﬁc (text) domain.

Original: For our SCQA tasks, it is crucial to guide the system to learn kind of orality in documents.
Revised: For our SCQA tasks, it is crucial to guide the system to learn kind of orality in documents.

Original: In this work, we propose a new spoken conversational question answering task - SCQA, and introduce Spoken-CoQA, a spoken conversational question answering dataset to evaluate a QA system whether necessary to tackle the task of question answering on noisy speech transcripts and text document.
Revised: In this work, we propose a new spoken conversational question answering task - SCQA, and introduce Spoken-CoQA, a spoken conversational question answering dataset to evaluate a QA system whether necessary to tackle the task of question answering on noisy speech transcripts and text document.

Original: We compare Spoken-CoQA with existing SQA datasets (See Table 1).
Revised: We compare Spoken-CoQA with existing SQA datasets (See Table 1).

Original: Unlike existing
Revised: Unlike existing

Original: SQA datasets, Spoken-CoQA is a multi-turn conversational SQA dataset, which is more challenging than single-turn benchmarks.
Revised: SQA datasets, Spoken-CoQA is a multi-turn conversational SQA dataset, which is more challenging than single-turn benchmarks.

Original: First, every question is dependent on the conversation history in the
Revised: First, every question is dependent on the conversation history in the

Original: Spoken-CoQA dataset.
Revised: Spoken-CoQA dataset.

Original: It is thus difﬁcult for the machine to parse.
Revised: It is thus difﬁcult for the machine to parse.

Original: Second, errors in ASR modules also degrade the performance of machines in tackling contextual understanding with context paragraph.
Revised: Second, errors in ASR modules also degrade the performance of machines in tackling contextual understanding with context paragraph.

Original: To mitigate the effects of speech recognition errors, we then present a novel knowledge distillation (KD) method for spoken conversational question answering tasks.
Revised: To mitigate the effects of speech recognition errors, we then present a novel knowledge distillation (KD) method for spoken conversational question answering tasks.

Original: Our ﬁrst intuition is speech utterances and text contents share the dual nature property, and we can take advantage of this property to learn these two forms of the correspondences.
Revised: Our ﬁrst intuition is speech utterances and text contents share the dual nature property, and we can take advantage of this property to learn these two forms of the correspondences.

Original: We enroll this knowledge into the student model, and then guide the student to unveil the bottleneck in noisy ASR outputs to boost perfor-
Revised: We enroll this knowledge into the student model, and then guide the student to unveil the bottleneck in noisy ASR outputs to boost performance.

Revised: Empirical results show that our proposed DDNet achieves remarkable performance gains in

Revised: SCQA tasks.

Revised: To the best of our knowledge, we are the ﬁrst work in spoken conversational machine reading comprehension tasks.

Original: [Figure]
Revised: [Figure]

Original: Documents/Questions
Revised: Documents/Questions

Original: [Figure]
Revised: [Figure]

Original: 𝑳
Revised: 𝑳

Original: [Figure]
Revised: [Figure]

Original: mance.

Original: Empirical results show that our proposed DDNet achieves remarkable performance gains in

Original: SCQA tasks.

Original: To the best of our knowledge, we are the ﬁrst work in spoken conversational machine reading comprehension tasks.

Original: In summary, the main contributions of this work are as follows:
Revised: In summary, the main contributions of this work are as follows:

Original: • We propose a new task for machine comprehension of spoken question-answering style conversation to improve the network performance.
Revised: • We propose a new task for machine comprehension of spoken question-answering style conversation to improve the network performance.

Original: To the best of our knowledge, our Spoken-CoQA is the ﬁrst spoken conversational machine reading comprehension dataset.
Revised: To the best of our knowledge, our Spoken-CoQA is the ﬁrst spoken conversational machine reading comprehension dataset.

Original: • We develop a novel end-to-end method based on data distillation to learn both from speech and text domain.
Revised: • We develop a novel end-to-end method based on data distillation to learn both from speech and text domain.

Original: Speciﬁcally, we use the model trained on clear syntax and close-distance recording to guide the model trained on noisy ASR transcriptions to achieve substantial performance gains in prediction accuracy.
Revised: Speciﬁcally, we use the model trained on clear syntax and close-distance recording to guide the model trained on noisy ASR transcriptions to achieve substantial performance gains in prediction accuracy.

Original: • We demonstrate the robustness of our DDNet on Spoken-CoQA, and demonstrates that the model can effectively alleviate ASR errors in noisy conditions.
Revised: • We demonstrate the robustness of our DDNet on Spoken-CoQA, and demonstrates that the model can effectively alleviate ASR errors in noisy conditions.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Conversational Machine Reading Comprehension
Revised: Conversational Machine Reading Comprehension

Original: In recent years, the natural language processing research community has devoted substantial efforts to conversational machine reading comprehension tasks (Huang et al., 2018a; Zhu et al., 2018; Xu et al., 2019; Zhang et al., 2020;
Revised: In recent years, the natural language processing research community has devoted substantial efforts to conversational machine reading comprehension tasks (Huang et al., 2018a; Zhu et al., 2018; Xu et al., 2019; Zhang et al., 2020;

Original: Gong et al., 2020).
Revised: Gong et al., 2020).

Original: Within the growing body of work on conversational machine reading comprehension, two signature attributes have emerged: the availability of large benchmark datasets (Choi et al., 2018;
Revised: Within the growing body of work on conversational machine reading comprehension, two signature attributes have emerged: the availability of large benchmark datasets (Choi et al., 2018;

Original: Elgohary et al., 2018;
Revised: Elgohary et al., 2018;

Original: Reddy et al., 2019) and pre-trained language models (Devlin et al., 2018;
Revised: Reddy et al., 2019) and pre-trained language models (Devlin et al., 2018;

Original: Liu et al., 2019; Lan et al., 2020).
Revised: Liu et al., 2019; Lan et al., 2020).

Original: However, these existing works typically focus on modeling the complicated context dependency in text form.
Revised: However, these existing works typically focus on modeling the complicated context dependency in text form.

Original: In contrast, we focus on enabling the machine to build the capability of language recognition and dialogue modeling in both speech and text domains.
Revised: In contrast, we focus on enabling the machine to build the capability of language recognition and dialogue modeling in both speech and text domains.

Original: Spoken Question Answering
Revised: Spoken Question Answering

Original: In parallel to the recent works in natural language processing, these trends have also been pronounced in the speech processing (SP) ﬁeld, where spoken question answering, an extended form of Question Answering, have explored the prospect of machine comprehension in spoken form.
Revised: In parallel to the recent works in natural language processing, these trends have also been pronounced in the speech processing (SP) ﬁeld, where spoken question answering, an extended form of Question Answering, have explored the prospect of machine comprehension in spoken form.

Original: Previous work on SQA typically includes two separate modules: automatic speech recognition and text question answering.
Revised: Previous work on SQA typically includes two separate modules: automatic speech recognition and text question answering.

Original: It entails transferring spoken content to ASR transcriptions, and then employs natural language processing techniques to handle the speech language processing tasks.
Revised: It entails transferring spoken content to ASR transcriptions, and then employs natural language processing techniques to handle the speech language processing tasks.

Original: Prior to this point, the existing methods (Tseng et al., 2016; Serdyuk et al., 2018;
Revised: Prior to this point, the existing methods (Tseng et al., 2016; Serdyuk et al., 2018;

Original: Su & Fung, 2020) focus on optimizing each module in a two-stage manner, where errors in the ASR module would suffer from severe performance loss.
Revised: Su & Fung, 2020) focus on optimizing each module in a two-stage manner, where errors in the ASR module would suffer from severe performance loss.

Original: Concurrent with our research, Serdyuk et al.
Revised: Concurrent with our research, Serdyuk et al.

Original: (2018) proposes an end-to-end approach for natural language understanding (NLU) tasks.
Revised: (2018) proposes an end-to-end approach for natural language understanding (NLU) tasks.

Original: SpeechBERT (Chuang et al., 2019) cascades the BERT-based models as a uniﬁed model and then trains it in an audio-and-text jointly learned manner.
Revised: SpeechBERT (Chuang et al., 2019) cascades the BERT-based models as a uniﬁed model and then trains it in an audio-and-text jointly learned manner.

Original: However, the existing SQA methods aim at solving a single question given the related passage without building and maintaining the connections of different questions within the human conversations.
Revised: However, the existing SQA methods aim at solving a single question given the related passage without building and maintaining the connections of different questions within the human conversations.

Original: Knowledge Distillation Hinton et al.
Revised: Knowledge Distillation Hinton et al.

Original: (2015) introduces the idea of Knowledge Distillation (KD) in a teacher-student scenario.
Revised: (2015) introduces the idea of Knowledge Distillation (KD) in a teacher-student scenario.

Original: In other words, we can distill the knowledge from one model (massive or teacher model) to another (small or student model).
Revised: In other words, we can distill the knowledge from one model (massive or teacher model) to another (small or student model).

Original: Previous work has shown that KD can signiﬁcantly boost prediction accuracy in natural language processing and speech processing (Kim & Rush, 2016;
Revised: Previous work has shown that KD can signiﬁcantly boost prediction accuracy in natural language processing and speech processing (Kim & Rush, 2016;

Original: Hu et al., 2018; Huang et al., 2018b; Hahn & Choi, 2019), while adopting KD-based methods for SCQA tasks has been less explored.
Revised: Hu et al., 2018; Huang et al., 2018b; Hahn & Choi, 2019), while adopting KD-based methods for SCQA tasks has been less explored.

Original: Although we share the same research topic and application, our research direction and methods differ.
Revised: Although we share the same research topic and application, our research direction and methods differ.

Original: Previous methods design a uniﬁed model to model the single-turn speech-language task.
Revised: Previous methods design a uniﬁed model to model the single-turn speech-language task.

Original: In contrast, our model explores the prospect of handling
Revised: In contrast, our model explores the prospect of handling

Original: SQA tasks.
Revised: SQA tasks.

Original: More importantly, we focus the question of nature property in speech and text: do spoken conversational dialogues can further assist the model to boost the performance.
Revised: More importantly, we focus the question of nature property in speech and text: do spoken conversational dialogues can further assist the model to boost the performance.

Original: Finally, we incorporate the knowledge distillation framework to distill reliable dialogue ﬂow from the spoken contexts, and utilize the learned predictions to guide the student model to train well on the noisy input data.
Revised: Finally, we incorporate the knowledge distillation framework to distill reliable dialogue ﬂow from the spoken contexts, and utilize the learned predictions to guide the student model to train well on the noisy input data.

Original:  T ASK D EFINITION 3. D ATA F ORMAT
Revised:  T ASK D EFINITION 3. D ATA F ORMAT

Original: We introduce Spoken-CoQA, a new spoken conversational machine reading comprehension dataset where the documents are in the spoken and text form.
Revised: We introduce Spoken-CoQA, a new spoken conversational machine reading comprehension dataset where the documents are in the spoken and text form.

Original: Given the spoken multi-turn dialogues and spoken documents, the task is to answer questions in multi-party conversations.
Revised: Given the spoken multi-turn dialogues and spoken documents, the task is to answer questions in multi-party conversations.

Original: Each example in this dataset is deﬁned as follows: { D i , Q i , A i } N 1 , where Q i = { q i 1 , q i 2 , ..., q iL } and A i = { a i 1 , a i 2 , ..., a iL } represent a passage with L -turn queries and corresponding answers, respectively.
Revised: Each example in this dataset is deﬁned as follows: { D i , Q i , A i } N 1 , where Q i = { q i 1 , q i 2 , ..., q iL } and A i = { a i 1 , a i 2 , ..., a iL } represent a passage with L -turn queries and corresponding answers, respectively.

Original: Given a passage D i , multi-turn history questions { q i 1 , q i 2 , ..., q iL − 1 } and the reference answers { a i 1 , a i 2 , ..., a iL − 1 } , our goal is to generate a iL for the given current question q iL .
Revised: Given a passage D i , multi-turn history questions { q i 1 , q i 2 , ..., q iL − 1 } and the reference answers { a i 1 , a i 2 , ..., a iL − 1 } , our goal is to generate a iL for the given current question q iL .

Original: In this study, we use the spoken form of questions and documents as the network input for training.
Revised: In this study, we use the spoken form of questions and documents as the network input for training.

Original: Note that questions and documents (passages) in Spoken-CoQA are in both text and spoken forms, and answers are in the text form.
Revised: Note that questions and documents (passages) in Spoken-CoQA are in both text and spoken forms, and answers are in the text form.

Original: 3. D ATA C OLLECTION
Revised: 3. D ATA C OLLECTION

Original: We detail the procedures to build Spoken-CoQA as follows.
Revised: We detail the procedures to build Spoken-CoQA as follows.

Original: First, we select the conversational question-answering dataset CoQA (Reddy et al., 2019) since it is one of the largest public CMRC datasets.
Revised: First, we select the conversational question-answering dataset CoQA (Reddy et al., 2019) since it is one of the largest public CMRC datasets.

Original: CoQA contains around 8k stories (documents) and over 120k questions with answers.
Revised: CoQA contains around 8k stories (documents) and over 120k questions with answers.

Original: The average dialogue length of CoQA is about 15 turns, and the answer is in free-form text.
Revised: The average dialogue length of CoQA is about 15 turns, and the answer is in free-form text.

Original: In
Revised: In

Original: CoQA, the training set and the development set contain 7,199 and 500 conversations over the given stories, respectively.
Revised: CoQA, the training set and the development set contain 7,199 and 500 conversations over the given stories, respectively.

Original: Therefore, we use the CoQA training set as our reference text of the training set and the CoQA development set as the test set in Spoken-CoQA.
Revised: Therefore, we use the CoQA training set as our reference text of the training set and the CoQA development set as the test set in Spoken-CoQA.

Original: Then we employ the Google text-to-speech system to transform questions and documents in CoQA into the spoken form.
Revised: Then we employ the Google text-to-speech system to transform questions and documents in CoQA into the spoken form.

Original: Next, we adopt CMU Sphinx to transcribe the processed spoken content into ASR transcriptions.
Revised: Next, we adopt CMU Sphinx to transcribe the processed spoken content into ASR transcriptions.

Original: As such, we collect more than 40G audio data, and the data duration is around 300 hours.
Revised: As such, we collect more than 40G audio data, and the data duration is around 300 hours.

Original: It is worth to note that since the constructed dataset does not update the answer spans based on the noisy ASR text and continues to assume answer-spans as per the actual text, we perform data ﬁltering in our investigation by eliminating question-answer pairs from the corpus if answer spans to questions do not exist in the referenced ASR transcriptions.
Revised: It is worth to note that since the constructed dataset does not update the answer spans based on the noisy ASR text and continues to assume answer-spans as per the actual text, we perform data ﬁltering in our investigation by eliminating question-answer pairs from the corpus if answer spans to questions do not exist in the referenced ASR transcriptions.

Original: For clarity, we provide an example of our Spoken-CoQA dataset in Table 2.
Revised: For clarity, we provide an example of our Spoken-CoQA dataset in Table 2.

Original: Figure 4 compares spectrograms of samples from ASR modules.
Revised: Figure 4 compares spectrograms of samples from ASR modules.

Original: In this example, we observe that given the text document (ASR-document), the conversation starts with the question Q 1 (ASR- Q 1 ), and then the system requires to answer Q 1 (ASR- Q 1 ) with A 1 based on a contiguous text span R 1 .
Revised: In this example, we observe that given the text document (ASR-document), the conversation starts with the question Q 1 (ASR- Q 1 ), and then the system requires to answer Q 1 (ASR- Q 1 ) with A 1 based on a contiguous text span R 1 .

Original: Compared to the existing benchmark datasets, ASR transcripts (both the document and questions) are much more difﬁcult for the machine to comprehend questions, reason among the passage, and even predict the correct answer. 
Revised: Compared to the existing benchmark datasets, ASR transcripts (both the document and questions) are much more difﬁcult for the machine to comprehend questions, reason among the passage, and even predict the correct answer. Manual Transcript

Original: [Table]
Revised: [Table]

Original: Once upon a time, in a barn near a farm house, there lived a little white kitten named
Revised: Once upon a time, in a barn near a farm house, there lived a little white kitten named

Original: Cotton.
Revised: Cotton.

Original: Cotton lived high up in a nice warm place above the barn where all of the farmer’s horses slept.
Revised: Cotton lived high up in a nice warm place above the barn where all of the farmer’s horses slept.

Original: But Cotton wasn’t alone in her little home above the barn,
Revised: But Cotton wasn’t alone in her little home above the barn,

Original: oh no.
Revised: oh no.

Original: She shared her hay bed with her mommy and 5 other sisters...
Revised: She shared her hay bed with her mommy and 5 other sisters...

Original: Once upon a time in a bar near farm house, there lived a little like captain named cotton.
Revised: Once upon a time in a bar near farm house, there lived a little like captain named cotton.

Original: How to live tied up in a nice warm place above the bar
Revised: How to live tied up in a nice warm place above the bar

Original: and we’re all of the farmers horses slapped.
Revised: and we’re all of the farmers horses slapped.

Original: But happened was not alone in her little home above the bar in now .
Revised: But happened was not alone in her little home above the bar in now .

Original: She shared her hey bed with her mommy and ﬁve other sisters...
Revised: She shared her hey bed with her mommy and ﬁve other sisters...

Original: Q
Revised: Q

Original: [Table]
Revised: [Table]

Original: : What color was Cotton?A
Revised: : What color was Cotton?

Original: [Table]
Revised: [Table]

Original: : white R 1 : a little white kitten named Cotton.
Revised: white R 1 : a little white kitten named Cotton.

Original: [Table]
Revised: [Table]

Original: in
Revised: color [Table] named

Original: [Table]
Revised: [Table]

Original: Q 2 : Where did she live?
Revised: Q 2 : Where did she live?

Original: A 2 : in a barn R 2 : in a barn near a farm house, there lived a little white kitten.
Revised: A 2 : in a barn R 2 : in a barn near a farm house, there lived a little white kitten.

Original: [Table]
Revised: [Table]

Original: in a barn near a farm house [Table] there lived a little whitekitten. Q
Revised: little white

Original: [Table]
Revised: [Table]

Original:  : Did she live alone?
Revised: Q 3 : Did she live alone?

Original: [Table]

Original:  : no R 3 : Cotton wasn’t alone
Revised: A 3 : no R 3 : Cotton wasn’t alone

Original: [Table]
Revised: [Table]

Original: wasn’t
Revised: alone

Original: [Table]
Revised: [Table]

Revised: Cotton wasn’t alone

Original:  DDN ET
Revised:  DDN ET

Original: In this section, we detail our data distillation approach by leveraging the dual nature of speech and text domains to boost the prediction accuracy in a spoken dialogue system.
Revised: In this section, we detail our data distillation approach by leveraging the dual nature of speech and text domains to boost the prediction accuracy in a spoken dialogue system.

Original: An overview pipeline of this task is shown in Figure 1.
Revised: An overview pipeline of this task is shown in Figure 1.

Original: We ﬁrst introduce the multi-modality fusion mechanism.
Revised: We ﬁrst introduce the multi-modality fusion mechanism.

Original: Then we present the major components of the CRMC module.
Revised: Then we present the major components of the CRMC module.

Original: Finally, we describe a simple yet effective distillation strategy in the proposed DDNet to learn feature representation in the speech-text domain comprehensively.
Revised: Finally, we describe a simple yet effective distillation strategy in the proposed DDNet to learn feature representation in the speech-text domain comprehensively.

Original: Given spoken words S = { s 1 , s 2 , ..., s n } and corresponding text words
Revised: Given spoken words S = { s 1 , s 2 , ..., s n } and corresponding text words

Original: X = { x 1 , x 2 , ..., x n } , we utilize Speech-BERT and Text-BERT to generate speech feature embedding E s
Revised: X = { x 1 , x 2 , ..., x n } , we utilize Speech-BERT and Text-BERT to generate speech feature embedding E s

Original: = { E s 1 , E s 2 , ..., E sn } and context word embedding E x = { E x 1 , E x 2 , ..., E xn } , respectively.
Revised: = { E s 1 , E s 2 , ..., E sn } and context word embedding E x = { E x 1 , E x 2 , ..., E xn } , respectively.

Original: Concretely, we ﬁrst use vqwav2vec (Baevski et al., 2019) to transfer speech signals into a series of tokens, which is the standard tokenization procedure in natural language processing tasks, and then use Speech-BERT (Chuang et al., 2019), a variant of BERT-based models, to process the speech sequences for training.
Revised: Concretely, we ﬁrst use vqwav2vec (Baevski et al., 2019) to transfer speech signals into a series of tokens, which is the standard tokenization procedure in natural language processing tasks, and then use Speech-BERT (Chuang et al., 2019), a variant of BERT-based models, to process the speech sequences for training.

Original: We retrain Speech-BERT (Chuang et al., 2019) on our Spoken-CoQA dataset.
Revised: We retrain Speech-BERT (Chuang et al., 2019) on our Spoken-CoQA dataset.

Original: The scale of Speech-BERT is similar with BERT-base (Devlin et al., 2018) model that contains 12 transformer layers with the residual structure and
Revised: The scale of Speech-BERT is similar with BERT-base (Devlin et al., 2018) model that contains 12 transformer layers with the residual structure and

Original: the embedding dimension is with 768.
Revised: the embedding dimension is with 768.

Original: In parallel, we embed the text context into a sequence of vectors via our text encoder - Text-BERT.
Revised: In parallel, we embed the text context into a sequence of vectors via our text encoder - Text-BERT.

Original: We adopt the same architecture of
Revised: We adopt the same architecture of

Original: BERT-base (Devlin et al., 2018) in our Text-BERT due to its superior performance.
Revised: BERT-base (Devlin et al., 2018) in our Text-BERT due to its superior performance.

Original: Cross Attention Inspired by ViLBERT (Lu et al., 2019), we apply the co-attention transformer layer(Lu et al., 2019), a variant of Self-Attention (Vaswani et al., 2017), as the Cross Attention module for speech and text embedding fusion.
Revised: Cross Attention Inspired by ViLBERT (Lu et al., 2019), we apply the co-attention transformer layer(Lu et al., 2019), a variant of Self-Attention (Vaswani et al., 2017), as the Cross Attention module for speech and text embedding fusion.

Original: We pass query, key, and value matrices ( Q , K , V ) as input to the Cross Attention module.
Revised: We pass query, key, and value matrices ( Q , K , V ) as input to the Cross Attention module.

Original: We then compute the cross attention-pooled features by querying one modality with Q vector from another modality.
Revised: We then compute the cross attention-pooled features by querying one modality with Q vector from another modality.

Original: [Equation]
Revised: [Equation]

Original: Finally, we obtain the aligned cross attentionembedding E cross by concatenating ˆ E crosss and ˆ E crossx .
Revised: Finally, we obtain the aligned cross attention

Original: 4. K EY C OMPONENTS
Revised: 4. K EY C OMPONENTS

Original: We build our CMRC module, based on recent works (Zhu et al., 2018; Huang et al., 2017).
Revised: We build our CMRC module, based on recent works (Zhu et al., 2018; Huang et al., 2017).

Original: We divide our CMRC module into three key components: Encoding Layer, Attention Layer and Output
Revised: We divide our CMRC module into three key components: Encoding Layer, Attention Layer and Output

Original: Layer.
Revised: Layer.

Original: Encoding Layer We encode documents and conversations (questions and answers) into the corresponding feature embedding (e.g.,character embedding, word embedding, and contextual embedding), and then concatenate the output contextual embedding and the aligned cross attention embedding E cross , and pass it as input.
Revised: Encoding Layer We encode documents and conversations (questions and answers) into the corresponding feature embedding (e.g.,character embedding, word embedding, and contextual embedding), and then concatenate the output contextual embedding and the aligned cross attention embedding E cross , and pass it as input.

Original: [Equation]
Revised: [Equation]

Original: Attention Layer We compute the attention on the context representations of the documents and questions, and extensively exploit correlations between them.
Revised: Attention Layer We compute the attention on the context representations of the documents and questions, and extensively exploit correlations between them.

Original: Note that we adopt the default attention layers in four baseline models.
Revised: Note that we adopt the default attention layers in four baseline models.

Original: Output Layer After obtaining attention-pooled representations, the Output Layer computes the probability distribution of the start and end index within the entire documents and predicts an answer to the current question.
Revised: Output Layer After obtaining attention-pooled representations, the Output Layer computes the probability distribution of the start and end index within the entire documents and predicts an answer to the current question.

Original: 4. K NOWLEDGE D ISTILLATION
Revised: 4. K NOWLEDGE D ISTILLATION

Original: For prior speech-language models, the only guidance is the standard training objective to measure the difference between the prediction and the reference answer.
Revised: For prior speech-language models, the only guidance is the standard training objective to measure the difference between the prediction and the reference answer.

Original: However, such criteria makes no sense for noisy ASR transcriptions.
Revised: However, such criteria makes no sense for noisy ASR transcriptions.

Original: To tackle this issue, we distill the knowledge from our teacher model, and use them to guide the student model to learn contextual features in our spoken CMRC task.
Revised: To tackle this issue, we distill the knowledge from our teacher model, and use them to guide the student model to learn contextual features in our spoken CMRC task.

Original: Concretely, we set the model trained on the speech document and text corpus as the teacher model and trained on the ASR transcripts as the student model, respectively.
Revised: Concretely, we set the model trained on the speech document and text corpus as the teacher model and trained on the ASR transcripts as the student model, respectively.

Original: Thus, the student trained on low-quality data learn to imbibe the knowledge that the teacher has discovered.
Revised: Thus, the student trained on low-quality data learn to imbibe the knowledge that the teacher has discovered.

Original: Concretely, given the z S and z T are the prediction vectors by the student and teacher models, the objective is deﬁne as:
Revised: Concretely, given the z S and z T are the prediction vectors by the student and teacher models, the objective is deﬁne as:

Original: [Equation]
Revised: [Equation]

Original: where KL ( · ) and XE ( · ) denote the Kullback-Leibler divergence and cross entropy, respectively.
Revised: where KL ( · ) and XE ( · ) denote the Kullback-Leibler divergence and cross entropy, respectively.

Original: y represents the ground truth labels in the text training dataset X .
Revised: y represents the ground truth labels in the text training dataset X .

Original: p τ ( · ) refers the softmax function with temperature τ , and α is a balancing factor.
Revised: p τ ( · ) refers the softmax function with temperature τ , and α is a balancing factor.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: In this section, we ﬁrst introduce several state-of-the-art language models as our baselines, and then evaluate the robustness of these models on our proposed Spoken-CoQA dataset.
Revised: In this section, we ﬁrst introduce several state-of-the-art language models as our baselines, and then evaluate the robustness of these models on our proposed Spoken-CoQA dataset.

Original: Finally, we provide a thorough analysis of different components of our method.
Revised: Finally, we provide a thorough analysis of different components of our method.

Original: Note that we use the default setting in all the evaluated methods.
Revised: Note that we use the default setting in all the evaluated methods.

Original: 5. B ASELINES
Revised: 5. B ASELINES

Original: In principle, DDNet can utilize any backbone network for SCQA tasks.
Revised: In principle, DDNet can utilize any backbone network for SCQA tasks.

Original: We choose several state-ofthe-art language models (FlowQA (Huang et al., 2018a), SDNet (Zhu et al., 2018),
Revised: We choose several state-ofthe-art language models (FlowQA (Huang et al., 2018a), SDNet (Zhu et al., 2018),

Original: BERT-base (Devlin et al., 2018), ALBERT (Lan et al., 2020)) as our backbone network due to its superior performance.
Revised: BERT-base (Devlin et al., 2018), ALBERT (Lan et al., 2020)) as our backbone network due to its superior performance.

Original: To train the teacher-student pairs simultaneously, we ﬁrst train baselines on the CoQA training set and then compare the performances of testing baselines on CoQA dev set and SpokenCoQA dev set.
Revised: To train the teacher-student pairs simultaneously, we ﬁrst train baselines on the CoQA training set and then compare the performances of testing baselines on CoQA dev set and SpokenCoQA dev set.

Original: Finally, we train the baselines on the Spoken-CoQA training set and evaluate the baselines on the CoQA dev set and Spoken-CoQA test set.
Revised: Finally, we train the baselines on the Spoken-CoQA training set and evaluate the baselines on the CoQA dev set and Spoken-CoQA test set.

Original: We provide quantitative results in Table 3.
Revised: We provide quantitative results in Table 3.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5. E XPERIMENT S ETTINGS
Revised: 5. E XPERIMENT S ETTINGS

Original: We use the ofﬁcial BERT (Devlin et al., 2018) and ALBERT (Lan et al., 2020) as our starting point for training.
Revised: We use the ofﬁcial BERT (Devlin et al., 2018) and ALBERT (Lan et al., 2020) as our starting point for training.

Original: We use BERT-base (Devlin et al., 2018) and ALBERT-base (Lan et al., 2020), which both include 12 transformer encoders, and the hidden size of each word vector is 768.
Revised: We use BERT-base (Devlin et al., 2018) and ALBERT-base (Lan et al., 2020), which both include 12 transformer encoders, and the hidden size of each word vector is 768.

Original: BERT and
Revised: BERT and

Original: ALBERT utilize BPE as the tokenizer, but FlowQA and SDNet use SpaCy (Honnibal & Montani, 2017) for tokenization.
Revised: ALBERT utilize BPE as the tokenizer, but FlowQA and SDNet use SpaCy (Honnibal & Montani, 2017) for tokenization.

Original: Speciﬁcally, in the case of tokens in spaCy (Honnibal & Montani, 2017) correspond to more than one BPE sub-tokens, we average the BERT embeddings of these BPE sub-tokens as the embedding for each token.
Revised: Speciﬁcally, in the case of tokens in spaCy (Honnibal & Montani, 2017) correspond to more than one BPE sub-tokens, we average the BERT embeddings of these BPE sub-tokens as the embedding for each token.

Original: To maintain the integrity of all evaluated model performance, we use standard implementations and hyper-parameters of four baselines for training.
Revised: To maintain the integrity of all evaluated model performance, we use standard implementations and hyper-parameters of four baselines for training.

Original: The balancing factor α is set to 0.9, and the temperature τ is set to 2.
Revised: The balancing factor α is set to 0.9, and the temperature τ is set to 2.

Original: For evaluation, we use Exact
Revised: For evaluation, we use Exact

Original: Match (EM) and F 1 score to compare the model performance on the test set.
Revised: Match (EM) and F 1 score to compare the model performance on the test set.

Original: Note that in this work, each baseline is trained in the local computing environment, which may results in different results compared with the ones on the CoQA leader board.
Revised: Note that in this work, each baseline is trained in the local computing environment, which may results in different results compared with the ones on the CoQA leader board.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: We compare several teacher-student pairs on CoQA and Spoken-CoQA dataset.
Revised: We compare several teacher-student pairs on CoQA and Spoken-CoQA dataset.

Original: Quantitative results are shown in Table 3.
Revised: Quantitative results are shown in Table 3.

Original: We can observe that the average F1 scores are 77.6 % when training on CoQA (text document) and testing on the CoQA dev set.
Revised: We can observe that the average F1 scores are 77.6 % when training on CoQA (text document) and testing on the CoQA dev set.

Original: However, when training the models on SpokenCoQA (ASR transcriptions) and testing on the Spoken-CoQA test set, average F1 scores are dropped to 49.3 % .
Revised: However, when training the models on SpokenCoQA (ASR transcriptions) and testing on the Spoken-CoQA test set, average F1 scores are dropped to 49.3 % .

Original: For FlowQA, the performance even dropped by 40.4 % on F1 score.
Revised: For FlowQA, the performance even dropped by 40.4 % on F1 score.

Original: This conﬁrms the importance of mitigating ASR errors which severely degrade the model performance in our tasks.
Revised: This conﬁrms the importance of mitigating ASR errors which severely degrade the model performance in our tasks.

Original: As shown in Table 4, it demonstrates that our proposed Cross Attention block and knowledge distillation strategy consistently boost the remarkable performance on all baselines, respectively.
Revised: As shown in Table 4, it demonstrates that our proposed Cross Attention block and knowledge distillation strategy consistently boost the remarkable performance on all baselines, respectively.

Original: More importantly, our distillation strategy works particularly well.
Revised: More importantly, our distillation strategy works particularly well.

Original: For FlowQA, our method achieves 53.7 % (vs.51.6 % ) and 39.2 % (vs.34.7 % ) in terms of F1 score over text document and ASR transcriptions, respectively.
Revised: For FlowQA, our method achieves 53.7 % (vs.51.6 % ) and 39.2 % (vs.34.7 % ) in terms of F1 score over text document and ASR transcriptions, respectively.

Original: For SDNet, our method outperforms the baseline without distillation strat- egy, achieving 55.6 % (vs.52.5 % ) and 56.7 % (vs.53.1 % ) in terms of F1 score.
Revised: For SDNet, our method outperforms the baseline without distillation strat- egy, achieving 55.6 % (vs.52.5 % ) and 56.7 % (vs.53.1 % ) in terms of F1 score.

Original: For two BERT-like models (BRET-base and ALBERT-base), our methods also improve F1 scores to 58.8 % (vs.55.8 % ) and 57.7 % (vs.54.1 % ); 59.6 % (vs.56.0 % ) and 58.7 % (vs.55.2 % ), respectively.
Revised: For two BERT-like models (BRET-base and ALBERT-base), our methods also improve F1 scores to 58.8 % (vs.55.8 % ) and 57.7 % (vs.54.1 % ); 59.6 % (vs.56.0 % ) and 58.7 % (vs.55.2 % ), respectively.

Original: Such signiﬁcant improvements demonstrate the effectiveness of DDNet .
Revised: Such signiﬁcant improvements demonstrate the effectiveness of DDNet .

Original:  Q UANTITATIVE A NALYSIS
Revised:  Q UANTITATIVE A NALYSIS

Original: Speech Feature in ASR System To perform qualitative analysis of speech features, we visualize the log-mel spectrogram features and the mel-frequency cepstral coefﬁcients (MFCC) feature embedding learned by DDNet in Figure 4.
Revised: Speech Feature in ASR System To perform qualitative analysis of speech features, we visualize the log-mel spectrogram features and the mel-frequency cepstral coefﬁcients (MFCC) feature embedding learned by DDNet in Figure 4.

Original: We can observe how the spectrogram features respond to different sentence examples.
Revised: We can observe how the spectrogram features respond to different sentence examples.

Original: Temperature τ To study the effect of temperature τ (See Section 4.2), we conduct the additional experiments of four baselines with the standard choice of the temperature τ ∈ { 1 , 2 , 4 , 6 , 8 , 10 } .
Revised: Temperature τ To study the effect of temperature τ (See Section 4.2), we conduct the additional experiments of four baselines with the standard choice of the temperature τ ∈ { 1 , 2 , 4 , 6 , 8 , 10 } .

Original: All models are trained on Spoken-CoQA dataset, and validated on the CoQA dev and Spoken-CoQA test set, respectively.
Revised: All models are trained on Spoken-CoQA dataset, and validated on the CoQA dev and Spoken-CoQA test set, respectively.

Original: In Figure 3, when T is set to 2, four baselines all achieve their best performance in term of F1 and EM metrics.
Revised: In Figure 3, when T is set to 2, four baselines all achieve their best performance in term of F1 and EM metrics.

Original: Multi-Modality Fusion Mechanism To study the effect of different modality fusion mechanisms, we introduce a novel fusion mechanism Con Fusion : ﬁrst, we directly concatenate two output embedding from speech-BERT and text-BERT models, and then pass it to the encoding layer in the following CMRC module.
Revised: Multi-Modality Fusion Mechanism To study the effect of different modality fusion mechanisms, we introduce a novel fusion mechanism Con Fusion : ﬁrst, we directly concatenate two output embedding from speech-BERT and text-BERT models, and then pass it to the encoding layer in the following CMRC module.

Original: In Table 5, we observe that Cross Attention fusion mechanism outperform four baselines with Con Fusion in terms of EM and F1 scores.
Revised: In Table 5, we observe that Cross Attention fusion mechanism outperform four baselines with Con Fusion in terms of EM and F1 scores.

Original: We further investigate the effect of uni-model input.
Revised: We further investigate the effect of uni-model input.

Original: Table 5 shows that text-only performs better than speech-only .
Revised: Table 5 shows that text-only performs better than speech-only .

Original: One possible reason for this performance is that only using speech features can bring additional noise.
Revised: One possible reason for this performance is that only using speech features can bring additional noise.

Original: Note that speech-only (text-only) represents that we only feed the speech (text) embedding for speech-BERT (text-BERT) to the encoding layer in the CMRC module.
Revised: Note that speech-only (text-only) represents that we only feed the speech (text) embedding for speech-BERT (text-BERT) to the encoding layer in the CMRC module.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have presented a new spoken conversational question answering task - SpokenCoQA, for enabling human-machine communication.
Revised: In this paper, we have presented a new spoken conversational question answering task - SpokenCoQA, for enabling human-machine communication.

Original: Unlike the existing Spoken conversational machine reading comprehension datasets, Spoken-CoQA includes multi-turn conversations and passages in both text and speech form.
Revised: Unlike the existing Spoken conversational machine reading comprehension datasets, Spoken-CoQA includes multi-turn conversations and passages in both text and speech form.

Original: Furthermore, we propose a data distillation method, which leverages audio-text features to reduce the misalignment between ASR hypotheses and the reference transcriptions.
Revised: Furthermore, we propose a data distillation method, which leverages audio-text features to reduce the misalignment between ASR hypotheses and the reference transcriptions.

Original: Experimental results show that DDNet achieves superior performance in prediction accuracy.
Revised: Experimental results show that DDNet achieves superior performance in prediction accuracy.

Original: For future work, we will further investigate different mechanism of integrating speech and text content, and propose novel machine learning based networks to migrate ASR recognition errors to boost the performance of QA systems.
Revised: For future work, we will further investigate different mechanism of integrating speech and text content, and propose novel machine learning based networks to migrate ASR recognition errors to boost the performance of QA systems.

Original: A A PPENDIX A. S PEECH FEATURES IN ASR S YSTEM
Revised: A A PPENDIX A. S PEECH FEATURES IN ASR S YSTEM

Original: Due to the page limit, we present some examples of speech features here.
Revised: Due to the page limit, we present some examples of speech features here.

Original: Once upon a time, in a barn near a farm house, there lived a little white kitten named Cotton.
Revised: Once upon a time, in a barn near a farm house, there lived a little white kitten named Cotton.

Original: Cotton lived high up in a nice warm place above the barn where all of the farmer's horses slept.
Revised: Cotton lived high up in a nice warm place above the barn where all of the farmer's horses slept.

Original: But Cotton wasn't alone in her little home above the barn,
Revised: But Cotton wasn't alone in her little home above the barn,

Original: oh no.
Revised: oh no.

Original: She shared her hay bed with her mommy and 5 other sisters.
Revised: She shared her hay bed with her mommy and 5 other sisters.

Original: A. Q UANTITATIVE ANALYSIS RESULTS
Revised: A. Q UANTITATIVE ANALYSIS RESULTS

Original: [Table]
Revised: [Table]

Original: ., 2018)+ Cross Attention 42.
Revised: ., 2018)+ Cross Attention 42.

Original: ALBERT-base (Lan et al., 2020) 42.
Revised: ALBERT-base (Lan et al., 2020) 42.

Original: ALBERT-base (Lan et al., 2020)+ speech-only 41.
Revised: ALBERT-base (Lan et al., 2020)+ speech-only 41.

Original: ALBERT-base (Lan et al.
Revised: ALBERT-base (Lan et al.

Original: , 2020)+ text-only 42.
Revised: , 2020)+ text-only 42.

Original: ALBERT-base (Lan et al.
Revised: ALBERT-base (Lan et al.

Original: , 2020)+ Con Fusion 42.
Revised: , 2020)+ Con Fusion 42.

Original: ALBERT-base (Lan et al., 2020)+ Cross Attention 42.
Revised: ALBERT-base (Lan et al., 2020)+ Cross Attention 42.