Original: SC O R E : P RE -T RAINING FOR C ONTEXT R EPRESENTATION IN C ONVERSATIONAL S EMANTIC P ARSING
Revised: SC O R E : P RE -T RAINING FOR C ONTEXT R EPRESENTATION IN C ONVERSATIONAL S EMANTIC P ARSING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Conversational Semantic Parsing (CSP) is the task of converting a sequence of natural language queries to formal queries (e.g., SQL, SPARQL) to be executed against a structured ontology (e.g. databases, KBs).
Revised: Conversational Semantic Parsing (CSP) is the task of converting a sequence of natural language queries to formal queries (e.g., SQL, SPARQL) to be executed against a structured ontology (e.g., databases, KBs).

Original: A CSP system needs to model the alignment between the unstructured language utterance and the structured ontology in the context of multi-turn dialog dynamics.
Revised: A CSP system needs to model the alignment between the unstructured language utterance and the structured ontology in the context of multi-turn dialog dynamics.

Original: Pre-trained language models have limited ability to represent NL references to structural data.
Revised: Pre-trained language models have limited ability to represent NL references to structural data.

Original: We present SC O R E , a new pre-training approach for CSP tasks designed to induce representations that capture the alignment between the conversational ﬂow and the structural context.
Revised: We present SC O R E , a new pre-training approach for CSP tasks designed to induce representations that capture the alignment between the conversational ﬂow and the structural context.

Original: By combining SC O R E with strong base systems on four different tasks (SP AR C,
Revised: By combining SC O R E with strong base systems on four different tasks (SP AR C,

Original: C O SQL, MW O Z, and SQA), we improve the performance over all baselines by a signiﬁcant margin and achieve state-of-the-art results on three of them.
Revised: C O SQL, MW O Z, and SQA), we improve the performance over all baselines by a signiﬁcant margin and achieve state-of-the-art results on three of them.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Task-oriented dialog (TOD) systems (Tur & Mori, 2011) assist the user in completing a task by performing an action or retrieving information backed by a structured ontology (e.g. a database, KB, or API).
Revised: Task-oriented dialog (TOD) systems (Tur & Mori, 2011) assist users in completing a task by performing an action or retrieving information backed by a structured ontology (e.g. a database, KB, or API).

Original: A key component of TOD is Conversational Semantic Parsing (CSP), which converts each dialog utterance into a formal query (e.g., SQL, SPARQL) to be executed against the ontology.
Revised: A key component of TOD is Conversational Semantic Parsing (CSP), which converts each dialog utterance into a formal query (e.g., SQL, SPARQL) to be executed against the ontology.

Original: CSP emerges in dialog systems (e.g., dialog state tracking in MW O Z (Budzianowski et al., 2018)), context-dependent semantic parsing (e.g., SP AR C
Revised: CSP emerges in dialog systems (e.g., dialog state tracking in MW O Z (Budzianowski et al., 2018)), context-dependent semantic parsing (e.g., SP AR C

Original: (Yu et al., 2019b)), SQL-grounded state tracking (e.g., C O SQL (Yu et al., 2019a)), and sequential question answering (Iyyer et al., 2017).
Revised: (Yu et al., 2019b)), SQL-grounded state tracking (e.g., C O SQL (Yu et al., 2019a)), and sequential question answering (Iyyer et al., 2017).

Original: These settings differ, but all share the same objective and key challenge: how to jointly represent the NL utterances and underlying structured ontology in the context of multi-turn dynamics of the dialog .
Revised: These settings differ, but all share the same objective and key challenge: how to jointly represent the NL utterances and underlying structured ontology in the context of multi-turn dynamics of the dialog .

Original: CSP, like other NL tasks, beneﬁts from pre-trained language models (PLMs) such as BERT (Devlin et al., 2019).
Revised: CSP, like other NL tasks, beneﬁts from pre-trained language models (PLMs) such as BERT (Devlin et al., 2019).

Original: However, general-purpose PLMs are pre-trained on free-form text using language- [Table] [Table] [Figure]
Revised: However, general-purpose PLMs are pre-trained on free-form text using languagedriven model objectives.

Original: D i a l og s t a t e t r a ck i ng

Original: [Figure]

Original: i

Original: [Figure]

Original: t u r n t

Original: [Figure]

Original: x t - t o - S

Original: Q L

Original: [Figure]

Original: driven model objectives.

Original: This limits their ability in modeling the structured ontology context or the dialog dynamics.
Revised: This limits their ability in modeling the structured ontology context or the dialog dynamics.

Original: As a result, the emitted formal queries are often not grounded in the existing
Revised: As a result, the emitted formal queries are often not grounded in the existing

Original: KB facts (Wang et al., 2020).
Revised: KB facts (Wang et al., 2020).

Original: In this work, we introduce SC O R E ( S tructured & S equential
Revised: In this work, we introduce SC O R E ( S tructured & S equential

Original: Co ntext
Revised: Co ntext

Original: Re presentation), a language model pre-training approach for CSP.
Revised: Re presentation), a language model pre-training approach for CSP.

Original: SC O R E adapts general PLMs by introducing a second phase of pre-training on synthesized CSP data with novel context-driven training objectives, which aim to ground utterances in the underlying ontology schema and model the conversational ﬂow.
Revised: SC O R E adapts general PLMs by introducing a second phase of pre-training on synthesized CSP data with novel context-driven training objectives, which aim to ground utterances in the underlying ontology schema and model the conversational ﬂow.

Original: SC O R E effectively injects structural and conversational inductive biases in
Revised: SC O R E effectively injects structural and conversational inductive biases in

Original: PLMs that translate to many CSP tasks out-of-the-box.
Revised: PLMs that translate to many CSP tasks out-of-the-box.

Original: It does not require changes to the pre-trained model architecture and can be used as a drop-in replacement with any semantic parsing model.
Revised: It does not require changes to the pre-trained model architecture and can be used as a drop-in replacement with any semantic parsing model.

Original: We apply SC O R E to four different CSP tasks: (1) sequential text-to-SQL (SP AR C), (2) conversational text-to-SQL (C O SQL), (3) dialog state tracking (MW O Z), and (4) weakly-supervised sequential question answering (SQA).
Revised: We apply SC O R E to four different CSP tasks: (1) sequential text-to-SQL (SP AR C), (2) conversational text-to-SQL (C O SQL), (3) dialog state tracking (MW O Z), and (4) weakly-supervised sequential [Table] [Table] [Figure]

Revised: D i a l og s t a t e t r a ck i ng

Original: They represent different scenarios, ontologies, supervision signals, system responses, and domains (see Figure 1 for a detailed comparison and examples).
Revised: [Figure] Transformer Encoder [Figure] u l t i - t u r n t e x t - t o - S

Revised: Q L

Revised: [Figure]

Revised: question answering (SQA).

Revised: They represent different scenarios, ontologies, supervision signals, system responses, and domains (see Figure 1 for a detailed comparison and examples).

Original: We show that: (1) SC O R E training objectives effectively incorporate synthetic data, (2) a single pre-trained
Revised: We show that: (1) SC O R E training objectives effectively incorporate synthetic data, (2) a single pre-trained

Original: SC O R E model can be used for several CSP tasks and can be combined with different baseline systems and (3) SC O R E signiﬁcantly improves all baselines, achieves new state-of-the-art results on three benchmarks (SP AR C, SP AR C, and MW O Z) and comparable performance to state-of-the-art on SQA.
Revised: SC O R E model can be used for several CSP tasks and can be combined with different baseline systems and (3) SC O R E signiﬁcantly improves all baselines, achieves new state-of-the-art results on three benchmarks (SP AR C, SP AR C, and MW O Z) and comparable performance to state-of-the-art on SQA.

Original:  SC O R E
Revised:  SC O R E

Original: The key challenge of CSP is to align the NL utterance and the structured ontology in the multi-turn dialog context.
Revised: The key challenge of CSP is to align the NL utterance and the structured ontology in the multi-turn dialog context.

Original: To this end, we inject structural and conversational inductive biases in SC O R E using two novel objective functions: Column Contextual Semantics (CCS) and the Turn Contextual Switch (TCS) , in addition to established language-driven Masked Language Modeling (MLM) .
Revised: To this end, we inject structural and conversational inductive biases in SC O R E using two novel objective functions: Column Contextual Semantics (CCS) and the Turn Contextual Switch (TCS) , in addition to established language-driven Masked Language Modeling (MLM) .

Original: 2. P RELIMINARIES
Revised: 2. P RELIMINARIES

Original: Task Deﬁnition
Revised: Task Deﬁnition

Original: In CSP, at each turn t , we aim to produce a formal query q t given the current utterance u t , the interaction history h t =
Revised: In CSP, at each turn t , we aim to produce a formal query q t given the current utterance u t , the interaction history h t =

Original: [ u 1 , u 2 , . . .
Revised: [ u 1 , u 2 , . . .

Original: , u t − 1 ] , and the schema c (table and column names, slots, etc.) of the target database (ontology) d .
Revised: , u t − 1 ] , and the schema c (table and column names, slots, etc.) of the target database (ontology) d .

Original: The tasks we consider (Figure 1) differ in their target formal language and ontology:
Revised: The tasks we consider (Figure 1) differ in their target formal language and ontology:

Original: • The utterance u is the user question for SP AR C and SQA, while for C O SQL and MW O Z, u is the combination of a user query and a system response.
Revised: • The utterance u is the user question for SP AR C and SQA, while for C O SQL and MW O Z, u is the combination of a user query and a system response.

Original: • The database d is used verbatim (multi-table) for SP AR C and C O SQL; for MW O Z, the predeﬁned ontology d can be viewed as a database; for SQA, d is a single table.
Revised: • The database d is used verbatim (multi-table) for SP AR C and C O SQL; for MW O Z, the predeﬁned ontology d can be viewed as a database; for SQA, d is a single table.

Original: • The formal query q is SQL for SP AR C and C O SQL; for MW
Revised: • The formal query q is SQL for SP AR C and C O SQL; for MW

Original: O Z it is the slot-value pairs, viewed as simple SQL consisting of SELECT and WHERE ; and for SQA, q is the latent program.
Revised: O Z it is the slot-value pairs, viewed as simple SQL consisting of SELECT and WHERE ; and for SQA, q is the latent program.

Original: Base Architecture The base architecture of SC O R E takes as input a single turn of a CSP dialog
Revised: Base Architecture The base architecture of SC O R E takes as input a single turn of a CSP dialog

Original: C t = (cid:104) u t , h t , c (cid:105) , and encodes it into contextualized conversation representations (cid:126)S t for each token in C t .
Revised: C t = (cid:104) u t , h t , c (cid:105) , and encodes it into contextualized conversation representations (cid:126)S t for each token in C t .

Original: The encoder architecture follows RoBERTa
Revised: The encoder architecture follows RoBERTa

Original: (Liu et al., 2019).
Revised: (Liu et al., 2019).

Original: It is then followed by a linear layer and normalized (Ba et al., 2016) to produce ﬁnal representations (cid:126)h t for each token:
Revised: It is then followed by a linear layer and normalized (Ba et al., 2016) to produce ﬁnal representations (cid:126) h t for each token:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where GELU is an activation by Hendrycks & Gimpel (2016) and W 1 is a learned parameter matrix.
Revised: where GELU is an activation by Hendrycks & Gimpel (2016) and W 1 is a learned parameter matrix.

Original: The context representations (cid:126)h t are then used as input to a program decoder model to produce the formalquery q t =f dec ((cid:126)h t |C t) .
Revised: The context representations (cid:126)h t are then used as input to a program decoder model to produce the formal q t = dec ( t | t

Original: As § 3 details, we use different state-of-the-art models for f dec .

Original: 2. P RE - TRAINING O BJECTIVES
Revised: 2. P RE - TRAINING O BJECTIVES

Original: Column Contextual Semantics To address alignment between the utterance and the underlying database schema, we optimize SC O R E with the auxiliary objective of Column Contextual Semantics (CCS) .
Revised: Column Contextual Semantics To address alignment between the utterance and the underlying database schema, we optimize SC O R E with the auxiliary objective of Column Contextual Semantics (CCS) .

Original: For each column in the schema, CCS targets the operations that should be performed on it in a given dialog turn.
Revised: For each column in the schema, CCS targets the operations that should be performed on it in a given dialog turn.

Original: Each query q is decomposed into operations on columns and tables, e.g. GROUPBY and HAVING for SQL queries, or WHERE for slot-value pairs.
Revised: Each query q is decomposed into operations on columns and tables, e.g. GROUPBY and HAVING for SQL queries, or WHERE for slot-value pairs.

Original: The CCS loss is given by:
Revised: The CCS loss is given by:

Original: [Equation]
Revised: [Equation]

Original: where h ct,i is the contextualized representation of the i th column’s initial special token in C t .
Revised: where h ct,i is the contextualized representation of the i th column’s initial special token in C t .

Original: Turn Contextual Switch
Revised: Turn Contextual Switch

Original: The Turn Contextual Switch (TCS) objective targets predicting the difference in formal queries between dialog turns based on the utterance to capture the conversational context ﬂow.
Revised: The Turn Contextual Switch (TCS) objective targets predicting the difference in formal queries between dialog turns based on the utterance to capture the conversational context ﬂow.

Original: Using the context-free grammar of SQL, we identify 26 possible kinds of differences that a conversational turn could elicit.
Revised: Using the context-free grammar of SQL, we identify 26 possible kinds of differences that a conversational turn could elicit.

Original: For example, DEL(SELECT.agg) indicates removing an aggregate operation from a SELECT statement (e.g. when an utterance “Show all the ages instead.”
Revised: For example, DEL(SELECT.agg) indicates removing an aggregate operation from a SELECT statement (e.g. when an utterance “Show all the ages instead.”

Original: elicits a change SELECT MAX(age) ...
Revised: elicits a change SELECT MAX(age) ...

Original: → SELECT age ... ).
Revised: → SELECT age ... ).

Original: The TCS loss targets predicting the context switch label between each conversational turn and the previous history:
Revised: The TCS loss targets predicting the context switch label between each conversational turn and the previous history:

Original: [Equation]
Revised: [Equation]

Original: where H st ∈ R ( t − 1) × d is the contextualized representation of all previous turns in C t with hidden dimension d .
Revised: where H st ∈ R ( t − 1) × d is the contextualized representation of all previous turns in C t with hidden dimension d .

Original: TCS is not used to pre-train on MW O Z because its context switch labels are trivial.
Revised: TCS is not used to pre-train on MW O Z because its context switch labels are trivial.

Original: Masked Language Modeling As in prior work on large-scale language models (Devlin et al., 2019), we use the Masked Language Modeling (MLM) objective to facilitate contextual representation learning for natural language utterances.
Revised: Masked Language Modeling As in prior work on large-scale language models (Devlin et al., 2019), we use the Masked Language Modeling (MLM) objective to facilitate contextual representation learning for natural language utterances.

Original: Importantly for regularization, we only apply this loss on in-domain human-annotated NL data – namely, SP AR C, C O SQL, SQA, and nine TOD datasets by
Revised: Importantly for regularization, we only apply this loss on in-domain human-annotated NL data – namely, SP AR C, C O SQL, SQA, and nine TOD datasets by

Original: Wu et al.
Revised: Wu et al.

Original: Formally, on each pre-training step we mask 15% of tokens in C t and optimize
Revised: Formally, on each pre-training step we mask 15% of tokens in C t and optimize

Original: [Equation]
Revised: [Equation]

Revised: CrossEntropy

Revised: [Equation]

Original: where h mt are the contextualized representations of the masked tokens in C t .
Revised: where h mt are the contextualized representations of the masked tokens in C t .

Original: Pre-Training Setup As Table 1 indicates, the total number of labeled dialog turns in CSP datasets is ca.
Revised: Pre-Training Setup As Table 1 indicates, the total number of labeled dialog turns in CSP datasets is ca.

Original: Following established practice, we augment them with synthesized conversational data .
Revised: Following established practice, we augment them with synthesized conversational data .

Original: We incorporate it into pre-training, forming an auxiliary dataset D syn in addition to the naturally supervised dataset D sup .
Revised: We incorporate it into pre-training, forming an auxiliary dataset D syn in addition to the naturally supervised dataset D sup .

Original: The total pre-training loss is the sum of the three with CCS and TCS only applied to D syn and MLM only to D sup :
Revised: The total pre-training loss is the sum of the three with CCS and TCS only applied to D syn and MLM only to D sup :

Original: [Equation]
Revised: [Equation]

Original: We re-use the dataset of 120k synthetic task-oriented dialogues for MW O Z, introduced by Campagna et al.
Revised: We re-use the dataset of 120k synthetic task-oriented dialogues for MW O Z, introduced by Campagna et al.

Original: In this work, we introduce a complementary procedure to synthesize data for conversational text-to-SQL dialogues.
Revised: In this work, we introduce a complementary procedure to synthesize data for conversational text-to-SQL dialogues.

Original: We use about 400k tables in W IKI T ABLES (Bhagavatula et al., 2015),
Revised: We use about 400k tables in W IKI T ABLES (Bhagavatula et al., 2015),

Original: WikiSQL (Zhong et al., 2017), and Spider (Yu et al., 2018b) datasets as underlying databases d , over which we generate 435k text-to-SQL conversations.
Revised: WikiSQL (Zhong et al., 2017), and Spider (Yu et al., 2018b) datasets as underlying databases d , over which we generate 435k text-to-SQL conversations.

Original: To this end, we induce two context-free utterance-SQL generation grammars: (1) a single-turn grammar G s for generating context-independent question-SQL pairs, and (2) a follow-up grammar
Revised: To this end, we induce two context-free utterance-SQL generation grammars: (1) a single-turn grammar G s for generating context-independent question-SQL pairs, and (2) a follow-up grammar

Original: G c for conversational question-SQL pairs.
Revised: G c for conversational question-SQL pairs.

Original: The follow-up grammar G c contains context switch labels and lists of follow-up question templates, e.g. a label INS(SELECT.column0) could generate the question “How about show column0 too?”
Revised: The follow-up grammar G c contains context switch labels and lists of follow-up question templates, e.g. a label INS(SELECT.column0) could generate the question “How about show column0 too?”

Original: To ensure generalization, we only induce the grammars from the SP AR C training set.
Revised: To ensure generalization, we only induce the grammars from the SP AR C training set.

Original: Appendix B shows examples of the templates and synthetic utterances.
Revised: Appendix B shows examples of the templates and synthetic utterances.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Metrics & Base Models We evaluate SC O R E on four CSP tasks in Figure 1, using the ofﬁcial metrics for each.
Revised: Metrics & Base Models We evaluate SC O R E on four CSP tasks in Figure 1, using the ofﬁcial metrics for each.

Revised: For SP AR C and C O SQL, we report question match accuracy (QM) , the exact set

Revised: [Table]

Original: For SP AR C and C O SQL, we report question match accuracy (QM) , the exact set match accuracy over SQL templates (Yu et al., 2018b), and interaction match accuracy (IM) , the ratio of interactions with all questions predicted correctly.
Revised: [Table] [Table] match accuracy over SQL templates (Yu et al., 2018b), and interaction match accuracy (IM) , the ratio of interactions with all questions predicted correctly.

Original: For MW O Z, we report joint goal accuracy (JGA) – similar to the IM accuracy used in SP AR C and C O SQL.
Revised: For MW O Z, we report joint goal accuracy (JGA) – similar to the IM accuracy used in SP AR C and C O SQL.

Original: For SQA, we report denotation
Revised: For SQA, we report denotation

Original: QM and IM accuracies.
Revised: QM and IM accuracies.

Original: For SP AR C and C O SQL , we use RAT-SQL (Wang et al., 2020) as our base model.
Revised: For SP AR C and C O SQL , we use RAT-SQL (Wang et al., 2020) as our base model.

Original: Since it is originally developed for single-turn text-to-SQL, we extend it to a multi-turn setting by concatenating [Table] SQL [Table]
Revised: Since it is originally developed for single-turn text-to-SQL, we extend it to a multi-turn setting by concatenating current utterances and dialog history.

Original: [Table] [Table] current utterances and dialog history. For MW O Z , we use the state-of-the-art Trippy model (Heck et al., 2020), which uses BERT base to encode the utterances and dialog history.
Revised: For MW O Z , we use the state-of-the-art Trippy model (Heck et al., 2020), which uses BERT base to encode the utterances and dialog history.

Original: We report ∼ 2% higher results because we train it for more epochs (25 vs. 10).
Revised: We report ∼ 2% higher results because we train it for more epochs (25 vs. 10).

Original: For SQA , we use the semantic parser by Wang et al.
Revised: For SQA , we use the semantic parser by Wang et al.

Original: , extended to multi-turn similarly to RAT-SQL.
Revised: , extended to multi-turn similarly to RAT-SQL.

Original: It ﬁrst generates a program template and then instantiates it by searching for matches between template slots and question spans.
Revised: It ﬁrst generates a program template and then instantiates it by searching for matches between template slots and question spans.

Original: Overall Results
Revised: Overall Results

Original: The results of SP AR C and C O SQL, MW O Z, and SQA are in Table 1, 2, andrespectively.
Revised: The results of SP AR C and C O SQL, MW O Z, and SQA are in Table 1, 2, andrespectively.

Original: We run each main experiment three times with different random seeds and report the mean.
Revised: We run each main experiment three times with different random seeds and report the mean.

Original: Overall, SC O R E gains signiﬁcant improvements over BERT and R O BERT A on all tasks, achieving state-of-the-art performances on SP AR C, C O SQL, and MW O Z.
Revised: Overall, SC O R E gains signiﬁcant improvements over BERT and R O BERT A on all tasks, achieving state-of-the-art performances on SP AR C, C O SQL, and MW O Z.

Original: We also show that SC O R E outperforms R O BERT
Revised: We also show that SC O R E outperforms R O BERT

Original: A under a few-shot setting of SQA when only 10% of training data is available.
Revised: A under a few-shot setting of SQA when only 10% of training data is available.

Original: [Table]
Revised: [Table]

Revised: What is the effect of each pre-training objective?

Revised: Table 4 shows an ablation study on different pre-training objectives.

Revised: We ﬁnd that the performance drops for C O SQL and

Revised: MW O Z while increases for SP AR C when removing the MLM loss (CCS+TCS).

Revised: One possible reason is that SP AR C is created by decomposing complex questions in sequences of inner related simpler questions with more strict underlying patterns and less language diversity.

Revised: Also, the synthesized data used to pre-train SC O R E for SP AR C and C O SQL is generated by the grammar induced by SP AR C, which might overﬁt to

Revised: SP AR C.

Revised: In addition, SC O R E pre-trained with only MLM loss doesn’t help but even hurt the performance on SP AR C. For the other tasks, the MLM loss slightly improves the performance especially on C O SQL.

Revised: Finally, we test the effectiveness of TCS on SP AR C and C O SQL (CCS only vs. CCS +

Revised: TCS), SC O R E gains improvements of 1.5% on SP AR C and 0.5% on C O SQL.

Revised: [Table]

Original: What if we directly augment the training set with synthetic data?
Revised: What if we directly augment the training set with synthetic data?

Original: We compare the base models trained with or without synthetic data on C O SQL and MW O Z. As shown in Table 4, synthetic data augmentation in the training set does not signiﬁcantly improve performance, conﬁrming similar ﬁndings in recent works (Zhang et al., 2019; Herzig et al., 2020;
Revised: We compare the base models trained with or without synthetic data on C O SQL and MW O Z. As shown in Table 5, synthetic data augmentation in the training set does not signiﬁcantly improve performance, conﬁrming similar ﬁndings in recent works (Zhang et al., 2019; Herzig et al., 2020;

Original: Campagna et al., 2020; Zhong et al., 2020).
Revised: Campagna et al., 2020; Zhong et al., 2020).

Original: In contrast, pre-training on it using our objectives yields downstream improvements.
Revised: In contrast, pre-training on it using our objectives yields downstream improvements.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Revised: Conversational semantic parsing is one of the most important research topics in conversational AI and has been studied in different settings including task-oriented dialogue (Budzianowski et al., 2018), question answering (Iyyer et al., 2017), and text-to-SQL (Yu et al., 2019a;b).

Revised: We present SC O R E , a new pre-training approach for CSP.

Revised: The training objectives of SC O R E aim to induce natural language representations that capture the multi-turn dynamics, compositional semantic of the target language, and the references to the structural ontology appearing in the dialog.

Revised: We demonstrated SC O R E effectiveness by using it as a drop-in feature representation encoder with strong baseline models for four different CSP tasks and achieving state-of-the-art results on three of them.

Original: A D ETAILED R ESULTS
Revised: A D ETAILED R ESULTS

Original: [Table]
Revised: [Table]

Original: Does SC O R E improve question match accuracy on individual turns?
Revised: Does SC O R E improve question match accuracy on individual turns?

Original: Table 5 shows detailed results of SC O R E ’s question accuracy for individual conversation turns on the SP AR C dev set.
Revised: Table 6 shows detailed results of SC O R E ’s question accuracy for individual conversation turns on the SP AR C dev set.

Original: SC O R E provides a signiﬁcant improvement for every conversation turn except the ﬁrst (in which the task is more similar to single-turn semantic parsing).
Revised: SC O R E provides a signiﬁcant improvement for every conversation turn except the ﬁrst (in which the task is more similar to single-turn semantic parsing).

Original: C O SQL and SQA exhibit similar behavior, as shown in Table 6 and Table[Table]
Revised: C O SQL and SQA exhibit similar behavior, as shown in Table 7 and Table

Original: BERT

Original: [Table]
Revised: [Table]

Original: +RoBERTa

Original: [Table]

Original: [Table]

Original: Few-Shot (10% training data)

Original: Wang et al.

Original: +RoBERTa
Revised: +RoBERTa

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: +SC O R E
Revised: + SC O R E

Original: [Table] [Table]
Revised: [Table] [Table]

Original: What is the effect of each pre-training objective? Table 8 shows an ablation study on different pre-training objectives. We ﬁnd that the performance drops for C O SQL and MW O Z while increases for SP AR C when removing the MLM loss (CCS+TCS).
Revised: et

Original: One possible reason is that SP AR C is created by decomposing complex questions in sequences of inner related simpler questions with more strict underlying patterns and less language diversity. Also, the synthesized data used to pre-train SC O R E for SP AR C and C O SQL is generated by the grammar induced by SP AR C, which might overﬁt to SP AR C. In addition, SC O R E pre-trained with only MLM loss doesn’t help but even hurt the performance on SP AR C. For the other tasks, the MLM loss slightly improves the performance especially on C O SQL.
Revised: [Table]

Original: Finally, we test the effectiveness of TCS on SP AR C and C O SQL (CCS only vs. CCS +

Original: TCS), SC O R E gains improvements of 1.5% on SP AR C and 0.5% on C O SQL.

Original: [Table]
Revised: [Table]

Original: How general is SC O R E ?
Revised: How general is SC O R E ?

Original: In addition to generalization over different task settings , we demonstrate the effectiveness of SC O R E on different base models .
Revised: In addition to generalization over different task settings , we demonstrate the effectiveness of SC O R E on different base models .

Original: To this end, we experiment with a different base model SOM-DST for MW O Z. As shown in Table 9,
Revised: To this end, we experiment with a different base model SOM-DST for MW O Z. As shown in Table 9,

Original: SC O R E can still improve the performance with a different base model on MW O Z (SOM-DST+BERT vs. SOM-DST+SC O R E on syn.
Revised: SC O R E can still improve the performance with a different base model on MW O Z (SOM-DST+BERT vs. SOM-DST+SC O R E on syn.

Original: MW O Z).
Revised: MW O Z).

Original: For generalization in synthetic grammar and data, as shown in Table 1 and 3, SC O R E is pre-trained on the synthesized data of the grammar induced from SP AR C, and it still improves the performance on C O SQL and SQA.
Revised: For generalization in synthetic grammar and data, as shown in Table 1 and 3, SC O R E is pre-trained on the synthesized data of the grammar induced from SP AR C, and it still improves the performance on C O SQL and SQA.

Original: Moreover, in Table 9 we show that SC O R E pre-trained on the text-to-SQL synthesized data could also surprisingly improve the performance on MW O Z.
Revised: Moreover, in Table 9 we show that SC O R E pre-trained on the text-to-SQL synthesized data could also surprisingly improve the performance on MW O Z.

Original: We expect that higher performance could be achieved with SC O R E pre-trained on task-speciﬁc synthesized data [Table]
Revised: We expect that higher performance could be achieved with SC O R E pre-trained on task-speciﬁc synthesized data.

Original: B S YNTHESIZED E XAMPLES & T EMPLATES
Revised: B S YNTHESIZED E XAMPLES & T EMPLATES

Original: Table 10 shows an example of the synthesized question-SQL pairs and their corresponding templates in our grammars
Revised: Table 10 shows an example of the synthesized question-SQL pairs and their corresponding templates in our grammars.

Original: [Table]
Revised: [Table]

Original: C P RE -T RAINING C OST
Revised: C P RE -T RAINING C OST

Original: We test the performance of SC O R E with respect to the number of pre-training epochs.
Revised: We test the performance of SC O R E with respect to the number of pre-training epochs.

Original: Figureshows that the best performance of the downstream tasks is usually achieved in early epochs, more speciﬁcally 5 for SP AR C and C O SQL and 15 for MW O Z. Longer pre-training time does not improve or even hurts the performance.
Revised: Figureshows that the best performance of the downstream tasks is usually achieved in early epochs, more speciﬁcally 5 for SP AR C and C O SQL and 15 for MW O Z. Longer pre-training time does not improve or even hurts the performance.

Original: One possible reason is that longer pre-training makes
Revised: One possible reason is that longer pre-training makes

Original: SC O R E overﬁt to the synthesized data whose utterances are unnatural.
Revised: SC O R E overﬁt to the synthesized data whose utterances are unnatural.

Original: As for the data, as shown in Table 8, even if
Revised: As for the data, as shown in Table 4, even if

Original: SC O R E is pre-trained with only a relatively small amount of synthesized data (without the MLM loss), most of the tasks can achieve much higher performances.
Revised: SC O R E is pre-trained with only a relatively small amount of synthesized data (without the MLM loss), most of the tasks can achieve much higher performances.

Original: With a relatively smaller training corpus and shorter training time compared to other pre-trained language models, SC O R E is efﬁcient in time and data.
Revised: With a relatively smaller training corpus and shorter training time compared to other pre-trained language models, SC O R E is efﬁcient in time and data.

Original: D I MPLEMENTATION D ETAILS D. SC 
Revised: D I MPLEMENTATION D ETAILS D. SC O R E

Original: O R E

Original: For pre-training SC O R E on synthesized text-to-SQL data, we use R O BERT A large and pre-train it with batch size 12, gradient accumulation step 2, and maximum length 248.
Revised: For pre-training SC O R E on synthesized text-to-SQL data, we use R O BERT A large and pre-train it with batch size 12, gradient accumulation step 2, and maximum length 248.

Original: We use a learning rate 1 e -5 and gradually reduce the learning rate without a warm-up period using Adam (Kingma & Ba, 2014) with epsilon 1 e -8.
Revised: We use a learning rate 1 e -5 and gradually reduce the learning rate without a warm-up period using Adam (Kingma & Ba, 2014) with epsilon 1 e -8.

Original: BERT base is used in pre-training SC O R E on synthesized MW O Z data because it contains longer conversations.
Revised: BERT base is used in pre-training SC O R E on synthesized MW O Z data because it contains longer conversations.

Original: We set the maximum length to 512 and batch size 24.
Revised: We set the maximum length to 512 and batch size 24.

Original: All SC O R E are pre-trained for 30 epochs, which usually take less than half a day on 8 V100 GPUs.
Revised: All SC O R E are pre-trained for 30 epochs, which usually take less than half a day on 8 V100 GPUs.

Original: We experimented with SC O R E pre-trained for 5, 10, and 30 epochs and found that most of the best downstream performances occur when base systems incorporate with SC O R E pre-trained for less than 10 epochs.
Revised: We experimented with SC O R E pre-trained for 5, 10, and 30 epochs and found that most of the best downstream performances occur when base systems incorporate with SC O R E pre-trained for less than 10 epochs.

Original: Our implementation is based on the Transformers library (Wolf et al., 2019).
Revised: Our implementation is based on the Transformers library (Wolf et al., 2019).

Original: D. B  M 
Revised: D. B ASE M ODELS

Original: RAT-SQL:
Revised: RAT-SQL:

Original: For a fair comparison, all RAT-SQL experiments are trained for 40k steps.
Revised: For a fair comparison, all RAT-SQL experiments are trained for 40k steps.

Original: We adopt the same hyperparameters as Shaw et al.
Revised: We adopt the same hyperparameters as Shaw et al.

Original: except for learning rates.
Revised: except for learning rates.

Original: We ﬁnd that learning rates of 1 e -4 and 1 e -5 for RAT and BERT respectively produce more stable results.
Revised: We ﬁnd that learning rates of 1 e -4 and 1 e -5 for RAT and BERT respectively produce more stable results.

Original: TripPy:
Revised: TripPy:

Original: We use the same hyperparameters for training TripPy on MW O Z as in (Heck et al., 2020) except we train it for 25 epochs (as opposed to 10 epochs as reported in (Heck et al., 2020)).
Revised: We use the same hyperparameters for training TripPy on MW O Z as in (Heck et al., 2020) except we train it for 25 epochs (as opposed to 10 epochs as reported in (Heck et al., 2020)).

Original: When we train TripPy for 25 epochs, we get a new result that is higher (around 2%) than the one reported in (Heck et al., 2020).
Revised: When we train TripPy for 25 epochs, we get a new result that is higher (around 2%) than the one reported in (Heck et al., 2020).

Original: Similarly, when we train TripPy with SC O R E , we train it for 25 epochs.
Revised: Similarly, when we train TripPy with SC O R E , we train it for 25 epochs.

Original: SOM-DST: We use the same hyperparameters from Kim et al.
Revised: SOM-DST: We use the same hyperparameters from Kim et al.

Original: (2020) for all SOM-DST experiments on MW O Z.
Revised: (2020) for all SOM-DST experiments on MW O Z.

Original: Wang et al.
Revised: Wang et al.

Original: (2019): We use the same hyperparameters from Wang et al.
Revised: (2019): We use the same hyperparameters from Wang et al.

Original: for SQA experiments.
Revised: for SQA experiments.

Original: E T ASK -O RIENTED D IALOGUE D ATASETS
Revised: E T ASK -O RIENTED D IALOGUE D ATASETS

Original: [Table]

Original: (Budzianowski

Original: [Table]

Original: [Table]

Original: [Table]

Original: WOZ (Mrkˇsi´c

Original: [Table]

Original: [Table]

Original: [Table]

Original: on target dialogue domains, but their training and ﬁne-tuning code is not released.

Original: Peng et al.

Original: [Table]
Revised: [Table]

Original: 2020) focus on the natural language generation (NLG) task, which assumes dialogue acts and slot-tagging results are given to generate a natural language response.
Revised: focus on the natural language generation (NLG) task, which assumes dialogue acts and slot-tagging results are given to generate a natural language response.

Original: By pre-training on a set of annotated NLG corpora, it can improve conditional generation quality using a GPT-2 model.
Revised: By pre-training on a set of annotated NLG corpora, it can improve conditional generation qual ity using a GPT-2 model.

Original:  Method
Revised:  Method

Original: In this section, we ﬁrst discuss each dataset used for our task-oriented pre-training and how we process the data.
Revised: In this section, we ﬁrst discuss each dataset used for our task-oriented pre-training and how we process the data.

Original: Then we introduce the selected pretraining base model and its objective functions.
Revised: Then we introduce the selected pretraining base model and its objective functions.

Original: 3.1 Datasets
Revised: 3.1 Datasets

Original: We collect nine different task-oriented datasets which are English-based, human-human, multiturn and publicly available.
Revised: We collect nine different task-oriented datasets which are English-based, human-human, multiturn and publicly available.

Original: In total, there are 100,707 dialogues, which contain 1,388,152 utterances over 60 domains.
Revised: In total, there are 100,707 dialogues, which contain 1,388,152 utterances over 60 domains.

Original: Dataset statistics is shown in Table 1.
Revised: Dataset statistics is shown in Table 1.

Original: • MetaLWOZ (Lee et al., 2019):
Revised: • MetaLWOZ (Lee et al., 2019):

Original: Meta-Learning Wizard-of-Oz is a dataset designed to help develop models capable of predicting user responses in unseen domains.
Revised: Meta-Learning Wizard-of-Oz is a dataset designed to help develop models capable of predicting user responses in unseen domains.

Original: This large dataset was created by crowdsourcing 37,884 goaloriented dialogs, covering 227 tasks in 47 domains.
Revised: This large dataset was created by crowdsourcing 37,884 goaloriented dialogs, covering 227 tasks in 47 domains.

Original: The MetaLWOZ dataset is used as the fast adaptation task for DSTC8 (Kim et al., 2019) dialogue competition.
Revised: The MetaLWOZ dataset is used as the fast adaptation task for DSTC8 (Kim et al., 2019) dialogue competition.

Original: • Schema (Rastogi et al., 2019): Schema-guided dialogue has 22,825 dialogues and provides a challenging testbed for several tasks, in partic- ular, dialogue state tracking.
Revised: • Schema (Rastogi et al., 2019): Schema-guided dialogue has 22,825 dialogues and provides a challenging testbed for several tasks, in particular, dialogue state tracking.

Original: Each schema is a set of tracking slots and each domain could have multiple possible schemas.
Revised: Each schema is a set of tracking slots and each domain could have multiple possible schemas.

Original: This allows a single dialogue system to support a large number of services and facilitates the simple integration of new services without requiring much training data.
Revised: This allows a single dialogue system to support a large number of services and facilitates the simple integration of new services without requiring much training data.

Original: The Schema dataset is used as the dialogue state tracking task for DSTC8 (Kim et al., 2019) dialogue competition.
Revised: The Schema dataset is used as the dialogue state tracking task for DSTC8 (Kim et al., 2019) dialogue competition.

Original: Taskmaster (Byrne et al., 2019):
Revised: Taskmaster (Byrne et al., 2019):

Original: This dataset includes 13,215 dialogues comprising six domains, including 5,507 spoken and 7,708 written dialogs created with two distinct procedures.
Revised: This dataset includes 13,215 dialogues comprising six domains, including 5,507 spoken and 7,708 written dialogs created with two distinct procedures.

Original: One is a two-person Wizard of Oz approach that one person acts like a robot and the other is a self-dialogue approach in which crowdsourced workers wrote the entire dialog themselves.
Revised: One is a two-person Wizard of Oz approach that one person acts like a robot and the other is a self-dialogue approach in which crowdsourced workers wrote the entire dialog themselves.

Original: It has 22.9 average conversational turns in a single dialogue, which is the longest among all taskoriented datasets listed.
Revised: It has 22.9 average conversational turns in a single dialogue, which is the longest among all taskoriented datasets listed.

Original: • MWOZ (Budzianowski et al., 2018):
Revised: • MWOZ (Budzianowski et al., 2018):

Original: MultiDomain Wizard-of-Oz dataset contains 10, dialogues over seven domains, and it has multiple domains in a single dialogue.
Revised: MultiDomain Wizard-of-Oz dataset contains 10, dialogues over seven domains, and it has multiple domains in a single dialogue.

Original: It has a detailed description of the data collection procedure, and user goal, system act, and dialogue state labels.
Revised: It has a detailed description of the data collection procedure, and user goal, system act, and dialogue state labels.

Original: Different from most of the existing corpora, it also provides full database information.
Revised: Different from most of the existing corpora, it also provides full database information.

Original: • MSR-E2E
Revised: • MSR-E2E

Original: (Li et al., 2018): Microsoft end-toend dialogue challenge has 10,087 dialogues in three domains, movie-ticket booking, restaurant reservation, and taxi [Figure]
Revised: (Li et al., 2018): Microsoft end-toend dialogue challenge has 10,087 dialogues in three domains, movie-ticket booking, restaurant reservation, and taxi booking.

Original: It also includes 
Revised: It also includes an experiment platform with built-in simulators in each domain.

Original: [Figure]
Revised: SMD (Eric and Manning, 2017): Stanford multidomain dialogue is an in-car personal assistant dataset, comprising 3,301 dialogues and three domains: calendar scheduling, weather informa- tion retrieval, and point-of-interest navigation.