Revised: M ULTI -T ASK

Revised: L EARNING

Original:  S EMANTIC P ARSING VIA C ROSS -D OMAIN S CHEMA
Revised: FOR S EMANTIC P ARSING WITH C ROSS -D OMAIN S KETCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Semantic parsing which maps a natural language sentence into aformal machinereadable representation of its meaning, is highly constrained by the limited annotated training data.
Revised: Semantic parsing, which maps a natural language sentence into a machinereadable representation of its meaning, is highly constrained by the limited annotated training data.

Original: Inspired by the idea of coarse-to-ﬁne, we propose a generalto-detailed neural network(GDNN) by incorporating  cross-domain schema(CDS) among utterances and their logic forms.
Revised: Inspired by the idea of coarse-to-ﬁne, we propose a generalto-detailed neural network (GDNN) by incorporating middle coarse cross-domain sketch (CDS) among utterances and their logic forms.

Original: For utterances in different domains, the General Network will extract CDS using an encoder-decoder model in a multitask learning setup.
Revised: For utterances in different domains, the General Network will extract CDS using an encoder-decoder model in a multi-task learning setup.

Original: Then for some utterances in a speciﬁc domain, the Detailed Network will generate the detailed target parts using sequence-to-sequence architecture with advanced attention to both utterance and generatedCDS.
Revised: Then for some utterances in a speciﬁc domain, the Detailed Network will generate the detailed target parts using a sequenceto-sequence architecture with advanced attention to both utterance and generated

Original:  Our experiments show that compared to direct multi-task learning, CDS has improved the performance in semantic parsing task which converts users’ requests into meaning representation language(MRL).
Revised: CDS. Our experiments show that compared to direct multi-task learning, CDS has improved the performance in semantic parsing task which converts users’ requests into meaning representation language (MRL).

Original: We also use experiments to illustrate that CDS works by adding some constraints to the target decoding process, which further proves the effectiveness and rationality of CDS.
Revised: We also use experiments to illustrate that CDS works by adding some constraints to the target decoding process, which further proves the effectiveness and rationality of CDS.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently many natural language processing(NLP) tasks based on recurrent neural network have shown promising results and gained much attention, especially in neural machine translation(NMT).
Revised: Recently many natural language processing (NLP) tasks based on the neural network have shown promising results and gained much attention because these studies are purely data-driven without linguistic prior knowledge.

Original: Semantic parsing  which maps a natural language sentence into a machine-readable representation of its meaning(Fan et al.
Revised: Semantic parsing task which maps a natural language sentence into a machine-readable representation (Fan et al.

Original: (2017)), as a special translation task, can be treated as a sequence-to-sequence problem(Dong & Lapata (2016)).
Revised: as a particular translation task, can be treated as a sequence-to-sequence problem (Dong & Lapata (2016)).

Revised: Lately, a compositional graph-based semantic meaning representation language (MRL) has been introduced (Kollar et al.

Revised: (2018a)), which converts utterance into logic form (action-object-attribute), increasing the ability to represent complex requests.

Original: Since semantic parsing task highly depends on the amount of annotatedtraining data and it’s even harder to annotate the data insome logic form formats(lambdacalculas, SQL, Alexa MRL, etc), several researchers have focused on the area of multi-task learning and transfer learning(Hakkani-T ¨ ur et al.
Revised: This work is based on MRL format for semantic parsing task. Semantic parsing highly depends on the amount of annotated data and it is hard to annotate the data in logic forms such as Alexa MRL. Several researchers have focused on the area of multi-task learning and transfer learning (Hakkani-T ¨ ur et al.

Original: (2016),Fan et al.
Revised: (2016), Fan et al.

Original: (2017),Kollar et al. (2018b)).
Revised: (2017), Kollar et al. (2018b))

Original: Dong & 
Revised: with the observation that while these tasks differ in their domains and forms, the structure of language composition repeats across domains (Herzig & Berant (2017)).

Original: Lapata (2018) believe utterance understanding is coarse-to-ﬁne(high-level tolow-level) and they improve the  performance by employing sketch which represents high-level information. Lately, a compositional graph-based semantic meaning representation language(MRL) has been introduced, increasing the ability to represent more complex requests, which converts utterance into logic form(action-object-attribute)(Kollar et al.
Revised: Compared to the model trained on a single domain only, a multi-task model that shares information across domains can improve both performance and generalization. However, there is still a lack of interpretations why the multi-task learning setting works (Ruder (2017)) and what the tasks have shared. Some NLP studies around language modeling (Le & Mikolov (2014), Vaswani et al.

Original: (2018a)).
Revised: (2017), Devlin et al.

Original: To our knowledge, there exists much structural and phrasal similarity in human habitual expressions and our work is based on this fact.
Revised: (2018)) indicate that implicit commonalities of the sentences including syntax and morphology exist and can share among domains, but these commonalities have not been fully discussed and quantiﬁed.

Original: For example, Search weather in 10 days in domain Weather and Find schedule for ﬁlms at night in domain ScreeningEvent both have action SearchAction and Attribute time ,  they share the same MRL structure like SearchAction( Type( time@?)) , where Type indicates domain and ? indicates attribute value which is copying from  utterance. In this work, compared to multi-task learning which directly use neural networks to learn shared features implicitly, we try to deﬁne these cross-domain commonalities explicitly as cross-domain schema(CDS) and use them as a coarse layer, inspired by the research of coarse-to-ﬁne.
Revised: To address this problem, in this work, compared to multi-task learning mentioned above which directly use neural networks to learn shared features in an implicit way, we try to deﬁne these cross-domain commonalities explicitly as cross-domain sketch (CDS). E.g., Search weather indays in domain Weather and Find schedule for ﬁlms at night in domain ScreeningEvent both have action SearchAction and Attribute time , so that they share a same MRL structure like SearchAction(Type(time@?)) , where Type indicates domain and ? indicates attribute value which is copying from the original utterance. We extract this domain general MRL structure as CDS.

Original: Instead of a single encoder-decoder to represent the process of converting utterance into target logic form, we construct a two-level encoder-decoder.
Revised: Inspired by the research of coarse-to-ﬁne (Dong & Lapata (2018)), we construct a two-level encoder-decoder by using CDS as a middle coarse layer.

Original: We ﬁrstly use General Network to get the CDS for every utterance in all domains.
Revised: We ﬁrstly use General Network to get the CDS for every utterance in all domains.

Original: Then for a single speciﬁc domain, based on both utterance and extracted CDS, we decode the ﬁnal target with advanced attention while CDS can be seen as adding some constraints to this process.
Revised: Then for a single speciﬁc domain, based on both utterance and extracted CDS, we decode the ﬁnal target with advanced attention while CDS can be seen as adding some constraints to this process.

Original: The ﬁrst utterance-CDS process can be regarded as  multi-task learning setup, since it is suitable for all utterances across the domains.
Revised: The ﬁrst utterance-CDS process can be regarded as a multi-task learning setup since it is suitable for all utterances across the domains.

Original: This work mainly introducing CDS with multi-task learning has some contributions listed below:
Revised: This work mainly introducing CDS using multi-task learning has some contributions listed below:

Original: 1) We make a solid assumption that there exist cross-domain commonalities including structural and phrasal similarity for utterances and extract these commonalities as cross-domain schema(CDS) which for our knowledge is the ﬁrst time.
Revised: 1) We make an assumption that there exist cross-domain commonalities including syntactic and phrasal similarity for utterances and extract these commonalities as cross-domain sketch (CDS) which for our knowledge is the ﬁrst time.

Original: We then deﬁne CDS in two different levels(action-level and attribute-level) trying to seek the most appropriate deﬁnition of CDS. 
Revised: We then deﬁne CDS on two different levels (action-level and attribute-level) trying to seek the most appropriate deﬁnition of CDS. 2)

Original: We propose a general-to-detailed neural network so as to incorporate CDS into the neural network.
Revised: We propose a general-to-detailed neural network by incorporating CDS as a middle coarse layer.

Original: We use the CDS to affect the ﬁnal target decoding, through a two-level encoder-decoder with advanced attention with both utterance and CDS which can be seen as adding some constraints to the ﬁnal target decoding process.
Revised: CDS is not only a high-level extraction of commonalities across all the domains, but also a prior information for ﬁne process helping the ﬁnal decoding. 3)

Original: 3) Since CDS is cross-domain, our ﬁrst-level network General Network which encodes the utterance and decodes CDS, can be seen as  multi-task learning setup, capturing the commonalities among utterances expressions from different domains which is exactly the goal of multi-task learning.
Revised: Since CDS is cross-domain our ﬁrst-level network General Network which encodes the utterance and decodes CDS can be seen as a multi-task learning setup, capturing the commonalities among utterances expressions from different domains which is exactly the goal of multi-task learning.

Original:  R ELATED W ORK 2. SLU TO MRL
Revised:  R ELATED W ORK 2. S POKEN L ANGUAGE U NDERSTANDING

Original: Traditional spoken language understanding(SLU) factors language understanding into domain classiﬁcation, intent prediction and slot ﬁlling, which proves to be effective in some domains(Gupta et al.
Revised: Traditional spoken language understanding (SLU) factors language understanding into domain classiﬁcation, intent prediction, and slot ﬁlling, which proves to be effective in some domains (Gupta et al.

Original: Representations of SLU use pre-deﬁned ﬁxed and ﬂat structures, which limit its expression skills including that it is hard to capture the similarity among utterances when the utterances are from different domains(Kollar et al.
Revised: Representations of SLU use pre-deﬁned ﬁxed and ﬂat structures, which limit its expression skills like that it is hard to capture the similarity among utterances when the utterances are from different domains (Kollar et al.

Original: (2018b)).
Revised: (2018b)).

Original: Due to SLU’s limited representation skills, meaning representation language (MRL) has been introduced which is a compositional graph-based semantic representation, increasing the ability to represent more complex requests(Kollar et al.
Revised: Due to SLU’s limited representation skills, meaning representation language (MRL) has been introduced which is a compositional graph-based semantic representation, increasing the ability to represent more complex requests (Kollar et al.

Original: (2018a)).
Revised: (2018a)).

Original: There are several different logic forms including lambda-calculas expression(Kwiatkowski et al.
Revised: There are several different logic forms including lambda-calculus expression (Kwiatkowski et al.

Original: , SQL(Zhong et al.
Revised: , SQL (Zhong et al.

Original: (2018)), Alexa MRL(Kollar et al.
Revised: Alexa MRL (Kollar et al.

Original: (2018a)).
Revised: (2018a)).

Original: Compared to ﬁxed and ﬂat
Revised: Compared to ﬁxed and ﬂat

Original: SLU representations, MRL(Kollar et al.
Revised: SLU representations, MRL (Kollar et al.

Original: (2018a))
Revised: (2018a))

Original: based on a large-scale ontology, is much stronger in expression in several aspects like cross-domain utterances, complex utterances etc.
Revised: based on a large-scale ontology, is much stronger in expression in several aspects like cross-domain and complex utterances.

Original: 2. S EQUENCE - TO -S EQUENCE FOR S EMANTIC P ARSING
Revised: 2. S EQUENCE - TO -S EQUENCE FOR S EMANTIC P ARSING

Original: Mapping a natural language utterance into machine interpreted logic form(such as MRL) can be regarded as a special translation task, which is treated as a sequence-to-sequence problem(Sutskever et al.
Revised: Mapping a natural language utterance into machine interpreted logic form (such as MRL) can be regarded as a special translation task, which is treated as a sequence-to-sequence problem (Sutskever et al.

Original: Then Bahdanau et al.
Revised: Then Bahdanau et al.

Original: and Luong et al.
Revised: and Luong et al.

Original: (2015) advance the sequence-tosequence network with attention mechanism learning the alignments between target and input words, making great progress in the performance.
Revised: (2015) advance the sequence-tosequence network with attention mechanism learning the alignments between target and input words, making great progress in the performance.

Original: Malaviya et al.
Revised: Malaviya et al.

Original: (2018) explore the attention mechanism with some improvements by replacing attention function with attention sparsity.
Revised: (2018) explore the attention mechanism with some improvements by replacing attention function with attention sparsity.

Original: In addition, to deal with the rare words, Gu et al.
Revised: Besides, to deal with the rare words, Gu et al.

Original: (2016) incorporate the copy mechanism into the encoder-decoder model by directly copying words from inputs.
Revised: (2016) incorporate the copy mechanism into the encoder-decoder model by directly copying words from inputs.

Original: Lately, many researchers have been around improving sequence-to-sequence model itself, in interpreting the sentence structural information.
Revised: Lately, many researchers have been around improving sequence-to-sequence model itself, in interpreting the sentence syntax information.

Original: Eriguchi et al.
Revised: Eriguchi et al.

Original: (2016) encode the input sentence recursively in a bottom-up fashion.
Revised: (2016) encode the input sentence recursively in a bottom-up fashion.

Original: Wu et al.
Revised: Wu et al.

Original: (2017) generate the target sequence and syntax tree through actions simultaneously.
Revised: (2017) generate the target sequence and syntax tree through actions simultaneously.

Original: Another aspect which has caught much attention is constrained decoding.
Revised: Another aspect which has caught much attention is constrained decoding.

Original: Krishnamurthy et al.
Revised: Krishnamurthy et al.

Original: and Post & Vilar (2018) add some constraints into decoding process, making it more controllable.
Revised: and Post & Vilar (2018) add some constraints into decoding process, making it more controllable.

Original: Dong & Lapata (2016) use  recurrent network as encoder which proves effective in sequence representation, and respectively use  recurrent network as decoder and tree-decoder.
Revised: Dong & Lapata (2016) use the recurrent network as encoder which proves effective in sequence representation, and respectively use the recurrent network as decoder and tree-decoder.

Original: Krishnamurthy et al.
Revised: Krishnamurthy et al.

Original: (2017) employ the grammar to constrain the decoding process.
Revised: (2017) employ the grammar to constrain the decoding process.

Original: Dong & Lapata (2018), believe utterance understanding is from high-level to low-level and by employing sketch, improve the performance.
Revised: Dong & Lapata (2018), believe utterance understanding is from high-level to low-level and by employing sketch, improve the performance.

Original: 2. M ULTI - TASK LEARNING
Revised: 2. M ULTI - TASK LEARNING

Original: Forany semantic parsing task especially in MRL format, it isboth expensive and time-consuming to annotate the data, and it isalso challenging to train semantic parsing neural models in limited training data.
Revised: For semantic parsing task especially in MRL format, it is expensive and time-consuming to annotate the data, and it is challenging to train semantic parsing neural models.

Original: Multi-task learning’s aim is to use other related tasks to improve target task performance.
Revised: Multi-task learning aims to use other related tasks to improve target task performance.

Original: Liu & Lane (2016b) deal with traditional SLU piper-line network by jointly detecting intent and doing slot ﬁlling.
Revised: Liu & Lane (2016b) deal with traditional SLU piper-line network by jointly detecting intent and doing slot ﬁlling.

Original: Sogaard & Goldberg (2016) share parameters among various tasks, according to lowlevel and high-level difference.
Revised: Sogaard & Goldberg (2016) share parameters among various tasks, according to the low-level and high-level difference.

Original: Hershcovich et al.
Revised: Hershcovich et al.

Original: (2018) divide the representation network into task-speciﬁc and general which is shared during multi-task learning.
Revised: (2018) divide the representation network into task-speciﬁc and general which is shared during multi-task learning.

Original: Fan et al.
Revised: Fan et al.

Original: (2017)  directly share the encoder or decoder neural layers(model params) through different semantic parsing tasks.
Revised: (2017) and Herzig & Berant (2017) directly share the encoder or decoder neural layers (model params) through different semantic parsing tasks.

Original: In Kollar et al.
Revised: In Kollar et al.

Original: (2018b), multi-task learning also mainly acts sharing the params of  network.
Revised: (2018b), multi-task learning also mainly acts sharing the params of the network.

Original:  A PPROACH 
Revised:  A PPROACH 3. D EFINITION OF C ROSS -D OMAIN SKETCH

Revised: For human language expressions, especially task-oriented requests, there exist commonalities across sentences including linguistic syntax and phrase similarity. They can be seen with general sentence templates. Table 1 shows some examples.

Revised: [Table]

Revised: Since sentence templates are too many, we try to leverage these common regularities more abstractly.

Original: Figure 1 shows our network structure in a brief way, which contains a two-level encoder-decoder.
Revised: We extract these invariant commonalities which are implicit across domains, and call them as crossdomain sketch (CDS) in a canonical way. We deﬁne CDS in meaning representation language (MRL) format (action-object-attribute) and on two levels ( action-level and attribute-level ). Action-level CDS means to acquire the same action for utterances from different domains while the attribute-level CDS means to extract more detailed information. See examples in Table 1.

Original: The General Network encodes utterance and decodes cross-domain schema(CDS), Since this process is irrelevant to domain, it can be done to all domains, which can be seen as a multi-task setup.
Revised: Instead of extracting CDS from utterance directly, we try converting from logic form into CDS reversely, because it is easier to deal with structural logic form than utterance in natural language form. We analyze the dataset Snips and use a rule-based method to obtain CDS. We strip logic forms from domain-speciﬁc components and preserve domainindependent parts including general actions and attributes. We do some statistics on the dataset Snips (Goo et al.

Revised: (2018)) used in this paper.

Revised: We convert attributes [object type, movie type, restaurant type] into { object type } , [object name, restaurant name, movie name] into { object name } , [year, timeRange] into { time } , [location name, current location] into { object location } , [country, city] into { place } .

Revised: All those attributes account for 55% of all attributes which indicate the existence and feasibility of CDS.

Original:  The Detailed Networkﬁrstly encodesthe obtained CDS and the utterance then decodes the target result based on utterance and CDS both.
Revised: Figure 1 shows our overall network, which contains a two-level encoder-decoder. The General Network encodes utterance and decodes cross-domain sketch (CDS).

Original: This process is domain-relevant so it can be treated as ﬁne-tuning process in a speciﬁc domain. The details of model are shown in Figure [Figure] [Figure]
Revised: Since this process is domaingeneral, it can be done to all domains, which is a multi-task setup.

Original: Utterances: u n [Figure] u [Figure]
Revised: The Detailed Network ﬁrstly encodes the CDS and the utterance, then it decodes the target result based on both utterance and

Original: u [Figure] u
Revised: CDS.

Original: Find schedule for films at night Specific Utterance Encoding [Figure] Decoding encoding unit decoding [Figure]
Revised: This process is domain-dependent, so that it is a ﬁne-tuning process in a speciﬁc domain.

Original: 3. P ROBLEM D EFINITION
Revised: 3.2. P ROBLEM D EFINITION

Original: For an input utterance u = u 1 , u 2 , ...
Revised: For an input utterance u = u 1 , u 2 , ...

Original: u | u | , and its middle result cross-domain schema(CDS)
Revised: u | u | , and its middle result cross-domain sketch (CDS)

Original: c
Revised: c

Original: = c 1 , c 2 , ...c | c | , and its meaning representation logic form y = y 1 , y 2 , ...
Revised: = c 1 , c 2 , ...c | c | , and its ﬁnal result logic form y = y 1 , y 2 , ...

Original: y | y | , the conditional probability can be seen as:
Revised: y | y | , the conditional probability is: p ( y | u, c ) = | y | (cid:89) t = p ( y t | y <t , u, c ) p

Original: [Equation]
Revised: ( c | u ) =

Original: where y <t 
Revised: | c | (cid:89) t = p ( c t | c <t , u ) (1) d d d d d d8 d d <S> d d d d7 d <S> d5 d Weather in 10 days (Domain: Weather) Find schedule for films at night (Domain:ScreeningEvent)

Original: = y 1 , y 2 , ...
Revised: s <S> s s s s s

Original: y | t − 1 | , and c <t = c 1 , c 2 , ...
Revised: Multi-Task Learning

Original: c | t − 1 | .
Revised: CDS Decoding u attention … General Utterance Encoding SearchAction( Weather( time@days ) ) <E> SearchAction( Film( time@ night ) ) <

Revised: E>

Revised: SearchAction( Type( time@? ) )

Revised: <E> s1’ s2’ s3’ s4’ s5’ attention

Revised: CDS Encoding SearchAction( Type( time@? ) )

Revised: Utterances: u n u u u u u attention

Revised: Specific Utterance Encoding Weather in days u u u u attention u u

Revised: Find schedule for films at night Specific Utterance Encoding

Revised: Target Decoding

Revised: Target Decoding encoding unit decoding unit where

Original: 3. U TTERANCE E NCODER
Revised: U TTERANCE E NCODER

Original: The neural encoder of our model is similar to neural machine translation(NMT) model, which uses  bi-directional recurrent neural network.
Revised: The neural encoder of our model is similar to neural machine translation (NMT) model, which uses a bi-directional recurrent neural network.

Original: Firstly each word of utterance is mapped into a vector u t ∈ R d via embedding layer.
Revised: Firstly each word of utterance is mapped into a vector u t ∈ R d via embedding layer and we get a word sequence u

Original:  Then we use a bi-directional recurrent neural network with long short-term memory units(Hochreiter & Schmidhuber (1997)) to learn the representation of word sequence.
Revised: = ( u 1 , ..., u | u | ) . Then we use a bi-directional recurrent neural network with long short-term memory units (LSTM) (Hochreiter & Schmidhuber (1997)) to learn the representation of word sequence.

Revised: We generate forward hidden state −→ h ut = f LSTM ( −−→ h ut − 1 , u t ) and backward state ←− h ut =

Revised: f LSTM ( ←−− h ut − 1 , u t ) .

Original: The t -th word will be:
Revised: The t -th word will be h ut =

Original: [Equation] where u represents utterance encoder, f LSTM represents LSTM function, [ · , · ] denotes vector concatenation.
Revised: [ −→ h ut , ←− h ut ] .

Original: We construct two kinds of utterance encoders, general utterance encoder for General Network and speciﬁc utterance encoder for Detailed Network(see in Figure 2), so as to extract different information for different purposes.
Revised: We construct two kinds of utterance encoders, general utterance encoder for General Network and speciﬁc utterance encoder for Detailed Network (see in Figure 1), so as to extract different information for different purposes.

Original: The general utterance encoder, meant to pay more attention to cross-domain commonalities of utterances, is used by all utterances from all domains.
Revised: The general utterance encoder, meant to pay more attention to cross-domain commonalities of utterances, is used by all utterances from all domains.

Original: The speciﬁc utterance encoder, which is domain-relevant, belongs to one speciﬁc domain and is more sensible to details.
Revised: The speciﬁc utterance encoder, which is domain-dependent, belongs to one speciﬁc domain and is more sensitive to details.

Original: We call encoder outputs h ugt from general utterance encoder and h ust from speciﬁc utterance encoder.
Revised: We call encoder outputs h ugt from general utterance encoder and h ust from speciﬁc utterance encoder.

Original: When the two encoders share the same parameters that is h ugt =
Revised: When the two encoders share the same parameters that is h ugt =

Original: h ust , we call it identical encoding and when they are not, we call it separate encoding , inspired by (Sogaard & Goldberg (2016);Krishnamurthy et al.
Revised: h ust , we call it identical encoding and when they are not, we call it separate encoding , inspired by (Sogaard & Goldberg (2016);Krishnamurthy et al.

Original: (2017);Liu et al.
Revised: (2017);Liu et al.

Original: (2017);Abdou et al.
Revised: (2017);Abdou et al.

Original: (2018)) which explore the sharing mechanisms of multi-task learning and propose some improvements.
Revised: (2018)) which explore the sharing mechanisms of multi-task learning and propose some improvements.

Original: 3. CDS D ECODER & E NCODER
Revised: 3.2. CDS D ECODER & E NCODER

Original: The General Network is meant to obtain cross-domain schema(CDS) c conditioned on utterance u , using an encoder-decoder network.
Revised: The General Network is meant to obtain cross-domain sketch (CDS) c conditioned on utterance u , using an encoder-decoder network.

Original: After encoding utterance by general utterance encoder for all domains, we obtain h ugt (see Section 3.2).
Revised: After encoding utterance by general utterance encoder for all domains, we obtain h ugt (see Section 3.2.2).

Original: Then we start to decode CDS.
Revised: Then we start to decode CDS.

Original: The decoder is based on a unidirectional recurrent neural network and the output vector is used to predict the word.
Revised: The decoder is based on a unidirectional recurrent neural network, and the output vector is used to predict the word.

Original: The cd represents CDS decoder.
Revised: The cd represents CDS decoder.

Original: [Equation]
Revised: [Equation]

Original: where c t is the previously predicted word embedding.
Revised: where c t is the previously predicted word embedding.

Original: The LuongAttention(Luong et al.
Revised: The LuongAttention (Luong et al.

Original: to utterance h ugi ( i = 1 , 2 , ... | u | ) at time  t is computed as:
Revised: between decoding hidden state d t and encoding sequence e i ( i = 1 , 2 , ... | e | ) at time step t is computed as:

Original: [Equation]
Revised: [Equation]

Revised: Based on equations (3) and (4), we compute the attention a ut by setting d t = h cdt and e i

Revised: = h ugi ( i = 1 , 2 , ...

Revised: | u | ) .

Original: The t -th predicted output token will be:
Revised: The t -th predicted output token will be:

Original: [Equation]
Revised: [Equation]

Original: where W, b are parameters.
Revised: where W, b are parameters.

Original: After decoding CDS words, we use an encoder to represent its meaning and due to words’ relation with forward and backward contexts, we choose using a bi-directional LSTM.
Revised: After decoding CDS words c = ( c 1 , ..., c | c | ) , we use an encoder to represent its meaning and due to words’ relation with forward and backward contexts, we choose to use a bi-directional LSTM.

Revised: We generate forward hidden state −→ h cet and backward state ←− h cet .

Original: The ce represents
Revised: The t -th word will be h cet =

Original: CDS encoder. [Equation]
Revised: [ −→ h cet , ←− h cet ] .

Original: 3. T ARGET D ECODER
Revised: 3.2. T ARGET D ECODER

Original: Through speciﬁc utterance encoder and cross-domain schema(CDS) encoder, we acquired t -th word representation h ust and h cet .
Revised: Through speciﬁc utterance encoder and cross-domain sketch (CDS) encoder, we acquired t -th word representation h ust and h cet .

Original: Finally with advanced attention to both encoded utterance u and CDS c , we decode the ﬁnal target y .
Revised: Finally with advanced attention to both encoded utterance u and CDS c , we decode the ﬁnal target y .

Original: The decoder is based on a unidirectional recurrent neural network and the output vector is used to predict the word.
Revised: The decoder is based on a unidirectional recurrent neural network, and the output vector is used to predict the word.

Original: The y represents target decoder.
Revised: The y represents target decoder.

Original: [Equation]
Revised: [Equation]

Original: where y t is the previously predicted word embedding.
Revised: where y t is the previously predicted word embedding.

Original: During target decoding process and at time step t , we not only compute the attention to utterance encoding outputs h us but also compute the attention to CDS encoding outputs h ce .
Revised: During target decoding process and at time step t , we not only compute the attention to utterance encoding outputs h us but also compute the attention to CDS encoding outputs h ce .

Original: The attention between target hidden state and utterance is a ut . The attention between target hidden state and CDS is a ct .
Revised: The attention between target hidden state and utterance is a ut by computing attention between h yt and encoding sequence

Original: [Equation]
Revised: h usi ( i = 1 , 2 , ... | u | ) based on equations (3) and (4). The attention between target hidden state and CDS is a c t by also computing attention between h yt and h cei ( i = 1 , 2 , ... | c | ) in the same way.

Original: Then the t -th predicted output token will be based on the advanced two-aspect attention:
Revised: Then the t -th predicted output token will be based on the advanced two-aspect attention:

Original: [Equation]
Revised: [Equation]

Original: 3. M ODEL T RAINING AND I NFERENCE
Revised: 3. M ODEL T RAINING AND I NFERENCE

Original: For training, the objective is:
Revised: For training process, the objective is:

Original: [Equation]
Revised: [Equation]

Original: T is the training corpus.
Revised: T is the training corpus.

Original: For inference, we ﬁrst obtain cross-domain schema(CDS) via (cid:101) c
Revised: For inference process, we ﬁrstly obtain cross-domain sketch (CDS) via (cid:101) c

Original: = argmax p ( c | u )and then  get the ﬁnal target logic form via (cid:101) 
Revised: = argmax p ( c | u ) then we get the ﬁnal target logic form via (cid:101) y

Original: y

Original: = argmax p ( y | u, (cid:101) c ) .
Revised: = argmax p ( y | u, (cid:101) c ) .

Original: For both decoding processes, we use greedy search to generate words one by one.
Revised: For both decoding processes, we use greedy search to generate words one by one.

Original: E XPERIMENTS

Original: We mainly consider the semantic parsing task based on meaning representation language(MRL) format(action-object-attribute).

Original: In order to testify the rationality of CDS, we do several experiments with CDS and we deﬁne the CDS on two levels( action-level and attribute-level ).

Original: Action-level CDS means to acquire the same action for utterances from different domains, e.g. Search weather indays and Find schedule for ﬁlms at night both belong to action SearchAction .

Original: Attribute-level CDS means to extract more detailed information while the examples both do SearchAction with attribute time limit.

Original: More examples are shown in Table 1.

Original:   4. D ATASETS
Revised:  E XPERIMENTS 4. D ATASETS

Original: Existed semantic parsing datasets, e.g. GEO(Zettlemoyer & Collins (2012)), ATIS(Zettlemoyer &
Revised: Existed semantic parsing datasets, e.g., GEO (Zettlemoyer & Collins (2012)), ATIS (Zettlemoyer &

Original: Collins (2007)),generally collect data  from onespeciﬁc domain and have  limited amount, which can not fully interpret the effectiveness of cross-domain schema(CDS).
Revised: Collins (2007)), collect data only from one domain and have a very limited amount, which can not fully interpret the effectiveness of cross-domain sketch (CDS) since it needs large dataset among different domains.

Original: In this case, we mainly consider the semantic parsing task  based on MRL format(action-objectattribute).
Revised: In this case [Table] we mainly consider the semantic parsing task Snips (Goo et al. based on MRL format (action-object-attribute).

Original: We evaluate on dataset Snips(Goo et al. (2018)) which collects users’ requests from a personal voice assistant.
Revised: Snips collects users’ requests from a personal voice assistant.

Original: The original dataset is annotated in spoken language understanding(SLU) format(intent-slot).
Revised: The original dataset is annotated in spoken language understanding (SLU) format (intent-slot).

Original: The training set contains 13084 utterances and 700 test samples with another 700 samples as the development set.

Original: It has 7 intent types and 72 slot labels.
Revised: It has 7 intent types and 72 slot labels [Table]

Original: The statistics are shown in Table 2.
Revised: and more statistics are shown in Table 2.

Original: Based on intent and slot, we pre-process this dataset into MRL format(actionobject-attribute) by some pre-deﬁned rules. We then  regard the intent as domain/task and share CDS among them.
Revised: Based on the format (intent-slot), we pre-process this dataset into MRL format by some pre-deﬁned rules, then we regard the intent as domain/task and share CDS among them.

Original: The details are shown in Table1.
Revised: The details are shown in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. S ETTINGS
Revised: 4. S ETTINGS

Original: We use Tensorﬂow in all our experiments, with LuongAttention(Luong et al.
Revised: We use Tensorﬂow in all our experiments, with LuongAttention (Luong et al.

Original: and copy mechanism.
Revised: and copy mechanism.

Original: The embedding dimension is set to 100 and initialized with GloVe embeddings(Pennington et al.
Revised: The embedding dimension is set to 100 and initialized with GloVe embeddings (Pennington et al.

Original: The encoder and decoder both use one-layer LSTM with hidden size 50.
Revised: The encoder and decoder both use one-layer LSTM with hidden size 50.

Original: We apply the dropout selected in { 0.3,0.5 } .
Revised: We apply the dropout selected in { 0.3,0.5 } .

Original: Learning rate is initialized with 0.001 and is decaying during training.
Revised: Learning rate is initialized with 0.001 and is decaying during training.

Original: Early stopping is applied.
Revised: Early stopping is applied.

Original: The mini-batch size is set to 16.
Revised: The mini-batch size is set to 16.

Original: We use  accuracy as the evaluation metric.
Revised: We use the logic form accuracy as the evaluation metric.

Original: 4. R ESULTS AND A NALYSIS
Revised: 4. R ESULTS AND A NALYSIS

Original: Firstly, in order to prove the role of  cross-domain schema(CDS) in helping guiding decoding process with multi-tasking learning setup, we do several experiments and the results are shown in Table 3.
Revised: Firstly, in order to prove the role of the cross-domain sketch (CDS) in helping to guide decoding process with multi-tasking learning setup, we do several experiments, and the results are shown in Table 4.

Original: For joint learning, we apply several multi-task architectures from (Fan et al.
Revised: For joint learning, we apply several multi-task architectures from (Fan et al.

Original: (2017)), including oneto-one, one-to-many and one-to-shareMany.
Revised: (2017)), including one-to-one, one-to-many and one-to-shareMany.

Original: One-to-one architecture applies a single sequence-tosequence model across all the tasks.
Revised: One-to-one architecture applies a single sequence-to-sequence model across all the tasks.

Original: One-to-many only shares the encoder across all the tasks while the decoder including the attention parameters are not shared.
Revised: One-to-many only shares the encoder across all the tasks while the decoder including the attention parameters is not shared.

Original: In one-to-shareMany model, tasks share encoder and decoder(including attention), but the output layer of decoder is task-independent.
Revised: In one-to-shareMany model, tasks share encoder and decoder (including attention) params, but the output layer of decoder is task-independent.

Original: From the Table 3, in general, joint learning performs better than single task learning.
Revised: From the Table 4, in general, joint learning performs better than single task learning.

Original: In joint learning, one-to-one is the best and performs way better than one-to-many and one-to-shareMany, probably limited by the dataset’s size and similarity among tasks.
Revised: In joint learning, one-to-one is the best and performs way better than one-to-many and one-to-shareMany, probably limited by the dataset’s size and similarity among tasks.

Original: By incorporating CDS, our GDNN(general-to-detailed neural network) models have all improved the performance to different degree.
Revised: By incorporating CDS, our GDNN (general-to-detailed neural network) models have all improved the performance to different degrees.

Original: The CDS is deﬁned on two levels(action-level and attribute-level, see examples in Table 1) and attribute-level CDS improves greater than action-level CDS, which is in our expectation, since it offers more information for tasks to share.
Revised: The CDS is deﬁned on two levels (action-level and attribute-level, see examples in Table 3) and attribute-level CDS improves greater than action-level CDS, which is in our expectation since it offers more information for tasks to share.

Original: We also experiment on different utterance encoding setups with identical encoding and separate encoding(see Section 3.2).
Revised: We also experiment on different utterance encoding setups with identical encoding and separate encoding (see Section 3.2.2).

Original: The separate encoding setup performs better than sharing the same encoder for utterance, which integrates the fact that different encoders pay different attention to the utterances due to different purposes which means one is more general and the other is more speciﬁc detailed.
Revised: The separate encoding setup performs better than sharing the same encoder for utterance, which integrates the fact that different encoders pay different attention to the utterances due to different purposes which means one is more general and the other is more speciﬁc detailed.

Original: [Table]
Revised: [Table]

Original: Joint show We also list the full results of GDNN in Table 4 below, including CDS accuracy by General Network, target accuracy by Detailed Network when feeding with 
Revised: We also list the full results of GDNN in Table 5 below, including CDS accuracy by General Network, target accuracy by Detailed Network when feeding with right CDS [Table]

Original: CDS, and the ﬁnal , which further prove the effectiveness of CDS.
Revised: and the ﬁnal accuracy, which further proves the effectiveness of CDS.

Original: [Table]
Revised: [Table]

Original: -detailed neural network(GDNN) with different levels of CDS(action-level/attribute level) and different utterance encoding mechanisms(identical encoding/separate encoding).

Original: Moreover, we compare our experiments with traditional models which regard the task as intent classiﬁcation and slot ﬁlling(IC SF).
Revised: Moreover, we compare our experiments with traditional models which regard the task as intent classiﬁcation and slot ﬁlling (IC SF).

Original: The results are shown in Table 5 below.
Revised: The results are shown in Table 6 below.

Original: [Table] [Table]
Revised: From Table [Table]

Original: From Table 5, we can see compared to IC SF models(based on sequence labeling format), Seq2Seq perform worse(71.4% compared to 73.2%) due to its fewer assumptions and larger decode size as well as its difﬁculty to train, which is usual in comparing IC SF models and sequence-to-sequence models.
Revised: we can see compared to IC SF models (based on sequence labeling format), Seq2Seq perform worse (71.4% compared to 73.2%) due to its fewer assumptions and larger decode size as well as its difﬁculty of training, which is usual in comparing IC SF models and sequence-tosequence models.

Original: Through using CDS, the performance has signiﬁcantly improved. 
Revised: Through using CDS, the performance has signiﬁcantly improved. On the one

Original: On one hand, CDS extract the cross-domain commonalities among tasks helping making the multi-task learning more speciﬁc, which can be seen as an advance to multi-task learning. 
Revised: [Table] hand, CDS extract the cross-domain commonalities among tasks helping to make the multi-task learning more speciﬁc, which can be seen as an advance to multi-task learning. On the other hand

Original: On the other hand, CDS can be seen adding some constraints to the ﬁnal target decoding process which has offered more information for the decoding process, compared to direct joint Seq2Seq.
Revised: [Table] CDS can be seen adding some constraints to the ﬁnal target decoding process which has offered more information for the decoding process, compared to direct joint Seq2Seq.

Original: To better prove and explain this idea, we do some experiments according to constraint decoding aspect.
Revised: To better prove and explain this idea, we do some experiments according to constraint decoding aspect.

Original: We try to compare the sub-process of converting utterance to CDS through different models e.g.
Revised: We try to compare the sub-process of converting utterance to CDS through different models, e [Table] g.,

Original: IC SF, Seq2Seq.
Revised: IC SF, Seq2Seq.

Original: From the Table 6, we can see that Seq2Seq achieve the comparable results(87.7%) to IC SF model(84.9%) for generating CDS from utterance, which further explains that, the fact joint seq2seq performs worse(71.4%, see Table 5) than IC SF model(73.2%) is owing to the lack of guidance and constraints during the follow-up decoding process.
Revised: From the Table [Table] we can see that Seq2Seq achieve the comparable results (87.7%) to IC SF model (84.9%) for generating CDS from utterance, which further explains that, the fact joint seq2seq performs worse (71.4%, see Table 6) than IC SF model (73.2%) is owing to the lack of guidance and constraints during the follow-up decoding process.

Original: By incorporating CDS, we add some constraints to this decoding process thus obtaining better performance.
Revised: By incorporating CDS, we add some constraints to this decoding process thus obtaining better performance

Original: [Table]
Revised: [Table]

Revised: by two methods.

Revised: IC SF is using intent classiﬁcation and slot ﬁlling with evaluation metric (intent accuracy, slot labelling accuracy and ﬁnal accuracy).

Revised: Seq2Seq generates CDS based on utterance using an encoder-decoder.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: In this paper, we propose the concept of cross-domain schema(CDS) which extracts some shared information across domains, trying to fully utilize the cross-domain commonalities such as structural and phrasal similarity in human expressions. 
Revised: In this paper, we propose the concept of cross-domain sketch (CDS) which extracts some shared information across domains, trying to fully utilize the cross-domain commonalities such as syntactic and phrasal similarity in human expressions. We try to deﬁne CDS on two levels and give some examples to illustrate our idea.

Original: We also present a general-to-detailed neural network(GDNN) for converting an utterance into a logic form based on meaning representation language(MRL) form.
Revised: We also present a general-to-detailed neural network (GDNN) for converting an utterance into a logic form based on meaning representation language (MRL) form.

Original: The general network, which is meant to extract cross-domain commonalities, uses an encoder-decoder model to obtain CDS in a multi-task setup.
Revised: The general network, which is meant to extract cross-domain commonalities, uses an encoderdecoder model to obtain CDS in a multi-task setup.

Original: Then the detailed network, generates the ﬁnal domain-speciﬁc target by exploiting utterance and CDS simultaneously via attention mechanism.
Revised: Then the detailed network generates the ﬁnal domain-speciﬁc target by exploiting utterance and CDS simultaneously via attention mechanism.

Original: Our experiments demonstrate the effectiveness of CDS and multi-task learning.
Revised: Our experiments demonstrate the effectiveness of CDS and multi-task learning.

Original: Moreover, CDS can also apply to other tasks since it’s an extraction to language expressions.
Revised: CDS is able to generalize over a wide range of tasks since it is an extraction to language expressions.

Original: Therefore, in the future, we would like to perfect the CDS deﬁnition and  explore more  ways  to make it work better.
Revised: Therefore, in the future, we would like to perfect the CDS deﬁnition and extend its’ ontology to other domains and tasks. Besides, in this paper, we use attention mechanism to make use of CDS which is still a indirect way. We would like to explore more effective ways such as constraint decoding to further enhance the role of CDS.