Syntactic and Semantic Uniformity for Semantic Parsing and Task-Oriented Dialogue Systems (2022)

Abstract: This paper proposes a data representation framework for semantic parsing and task-oriented dialogue systems, aiming to achieve a uniform representation for syntactically and semantically diverse machine-readable formats.Current NLP systems heavily rely on adapting pre-trained language models to specific tasks, and this approach has been proven effective for modeling natural language texts.However, little attention has been paid to the representation of machine-readable formats, such as database queries and dialogue states.We present a method for converting original machine-readable formats of semantic parsing and task-oriented dialogue datasets into a syntactically and semantically uniform representation.We define a meta grammar for syntactically uniform representations and translate semantically equivalent functions into a uniform vocabulary.Empirical experiments on 13 datasets show that accuracy consistently improves over original formats, revealing the advantage of the proposed representation.Additionally, we show that the proposed representation allows for transfer learning across datasets.

## introduction the common practice in current nlp research is to encode or decode natural language texts using large-scale pre-trained language models @xcite @xcite , which have been proven effective in modeling diverse natural language texts. in contrast with natural language, however, little attention has been paid to modeling machine-readable formats, such as database queries and dialogue states, while such formats are widely accepted in nlp tasks and developed to have specific usage like semantic parsing and task-oriented dialogue systems. each machine-readable format is defined based on its corresponding task and dataset, but the syntactic and semantic gap among machine-readable formats is huge. this hinders the development of general methods that can be applied to diverse tasks and datasets. to address the above problem, we propose a framework for representing machine-readable formats of diverse datasets on semantic parsing and task-oriented dialogue systems, with increased syntactic and semantic uniformity (figure 1 ) to mitigate such syntactic and semantic gap among machine-readable formats. first, we define a meta grammar to build a syntactically uniform representation across various datasets. original machinereadable formats are converted to a representation defined by this meta grammar. further, we define mapping rules to translate semantically equivalent functions into a uniform vocabulary. this translation assigns the same form to frequently used functions such as equivalence and comparison. combining these two methods, we can incorporate multiple data formats into a shared data representation, achieving syntactic and semantic uniformity. it should be noted that, however, the target of this work is not to create a completely unified dataset. our goal is to achieve a high-level abstraction of diverse datasets that enables us to explore a universal model for machine-readable formats while preserving their original features. we evaluate the proposed representation on 13 datasets of semantic parsing and task-oriented dialogue systems involving diverse machine-readable outputs, including logical forms, sql, and dialogue states. the experiment shows that accuracy consistently improves by converting original data into the uniform representation, proving the effectiveness of the proposed framework. further, our representation enables an easy knowledge transfer between datasets. transfer experiments show that the model could benefit from other datasets, which improves accuracy and leads to quicker convergence in the training. 2 related work 2.1 semantic parsing and task-oriented dialogue systems semantic parsing @xcite is a task to transform a user's utterance into a machinereadable format. an illustrating application is database qa, in which a system translates a natural language utterance into a query language for a given database. a variety of datasets have been proposed, where the representation format of database queries is diverse; examples include sql, prolog, and mrl (machine readable language) @xcite @xcite . another application of semantic parsing is to convert a natural language text into some form of meaning representation. for example, scan (lake and baroni, 2018) is a dataset of robot command utterances paired with their meaning representations. semantic parsing datasets are developed under the shared goal, i.e., convert natural language texts into a machine-readable format, while data representation specifics are varied. task-oriented dialogue systems interact with a user in natural language to achieve a specific goal like buying tickets or booking a hotel. a system tracks the user's intent from the dialogue, queries a database, and generates a response according to the result. previous approaches use belief state representations, a.k.a. dialogue states, to represent the user's intent as slot-value pairs. most datasets on task-oriented dialogue systems adopt this representation, including multiwoz @xcite , dstc2 @xcite , atis @xcite , and m2m @xcite . a closely related task called conversational semantic parsing proposed recently represents user intent as a program. @xcite proposes treedst, in which user intent is represented as a hierarchical format program. smcalflow (andreas et al., 2020) further develops this idea, where the representation involves a mechanism to track a user's previous information and retrieve it when necessary to handle dialogue-specific phenomena such as revision. all the datasets listed above use machine-readable formats to track user intent, while data representations diverge considerably. the focus of the present research is to minimize the diversity of machine-readable formats of semantic parsing and dialogue systems discussed above. it should be noted that, the present paper does not work on complete dialogue systems involving response generation. however, previous research @xcite @xcite proved that the performance of dialogue state tracking is essential for the entire system performance because the model is likely to generate incorrect responses when the dialogue state is wrongly predicted. ## uniform representation figure 1 shows the overview of our proposal. datasets of semantic parsing and task-oriented dialogue systems consist of pairs of input and output instances. we call a natural language query in semantic parsing and a dialogue in task-oriented dialogue systems the human input. a machinereadable output, such as database queries and dialogue states, is called the machine output. as shown in this figure, machine outputs are given in heterogeneous representations in different datasets. our goal is to convert original representations of machine outputs into a syntactically and semantically uniform representation. ## experiment this section describes experiment results to evaluate the effectiveness of the proposed uniform representation. it should be noted that we do not propose any new models in this paper, while most sota methods for each dataset are based on specifically designed models or external knowledge. we use simple baseline models while changing the representation of machine outputs of each dataset. we describe datasets and evaluation metrics we use for the experiments, followed by results on in-domain experiments and transfer experiments. ## conclusion this research presents a syntactically and semantically uniform data representation framework for semantic parsing and task-oriented dialogue systems. using our framework, we can incorporate both human input and machine output in 13 datasets into a both syntactically and semantically uniform representation. the experiments show promising improvements in multiple datasets compared to original data representations and even outperform several sota performances showing the benefit brought by the proposed format. additionally, our representation framework enables a convenient knowledge transfer. transfer experiment results show that the knowledge learned from other tasks can be easily transferred to the target task, improving the results on the target task and the convergence speed.