S truct SP : Efficient Fine-tuning of Task-Oriented Dialog System by Using Structure-aware Boosting and Grammar Constraints (2023)

Abstract: We have investigated methods utilizing hierarchical structure information representation in the semantic parsing task and have devised a method that reinforces the semantic awareness of a pre-trained language model via a two-step fine-tuning mechanism: hierarchical structure information strengthening and a final specific task. The model used is better than existing ones at learning the contextual representations of utterances embedded within its hierarchical semantic structure and thereby improves system performance. In addition, we created a mechanism using inductive grammar to dynamically prune the unpromising directions in the semantic structure parsing process. Finally, through experimentsOur code will be published when this paper is accepted. on the TOP and TOPv2 (low-resource setting) datasets, we achieved state-of-the-art (SOTA) performance, confirming the effectiveness of our proposed model.

## introduction task-oriented dialog systems are computer systems designed to perform a specific task @xcite . these systems have a wide range of applications in modern business @xcite and daily lives @xcite , and so on. the semantic parsing model at the core of these systems plays an essential role in converting user utterances into machine-understandable representations for use in capturing the semantic meaning and returning appropriate responses. the introduction of hierarchical representation by @xcite demonstrated the importance of nested sublogic composition in a task-oriented dialog system. although this representation is flexible enough to capture the meaning of complicated queries, it also challenges models to identify labels and select the pre-trained language models (e.g., bert @xcite ) have recently achieved impressive results on many tasks, including semantic parsing @xcite @xcite . among them, the recursive insertion-based encoding (rine) model @xcite produces sota results by using recursive insertion-based mechanism that directly injects the decoded tree from the previous steps as input to the current step. although many semantic parsing systems have shown impressive results, there is still much room for improvement in their overall performance. a promising way to improve the performance of a task-oriented dialog system is to use a pretrained language model based on the characteristics of this task. the language models is pre-trained on large-scale unstructured text, which means that it does not explicitly learn the logical structure of any sentences. since information about sentence structures is crucial in the semantic parsing task. besides, the hierarchical semantic representation in a task-oriented dialog system usually follows a grammar based on a specific task or domain. for training data [in:get_event [sl:cate_event concerts ] by [sl:name_event chris rock ] ] phase 1: structureaware boosting lstr_mlm structure-aware boosted model optimize recursive insertionbased encoder (with conditional loss by grammar rules) semantic parsing model lcontrastive phase 2: grammarbased rine in:get_event sl:cate_event by sl:name_event concerts chris rock grammar structure-focused mlm [in:get_event [sl:cate_event concerts ... ] ] 0.3 0.3 0.15 ... text: masking prob.: masked text: [mask] [sl:cate_event concerts ... ] ] relative tree agreement ... example, the parent node of slot sl:name_event is always the intent in:get_event, etc. however, existing methods ignore this information @xcite . we thus focused on making two improvements: strengthening the hierarchical structure awareness of the pre-trained language model and using dynamically pruning the unpromising decoding directions by using inductive grammar (figure 2 ). in this paper, we introduce structsp, a framework for effectively embedding the structured logic information of utterances into a pre-trained language model and thereby enhancing the performance of the semantic parsing system. our method is generalized and can be adapted to any logical structure, such as a logical form, by using λcalculus or prolog syntax @xcite . in particular, we exploit the hierarchical representation, a deep fine-grained structure widely used in semantic parsing @xcite @xcite . our method consists of two phases: structure-aware boosting and grammar-based rine. the structure-aware boosting phase aims to enhance the structured information of the pre-trained language models and includes two subtasks. the first is structurefocused mask language modeling (mlm), which extends the standard mlm task @xcite by focusing more on the logical units in the linearized hierarchical representation. the second subtask is relative tree agreement where relative trees are the trees parsed in the middle steps of the parsing process (figure 1 ). this subtask enables the model to represent the relative trees with closer hidden vectors and learn the hidden relationship between them. the second phase, grammar-based rine uses the grammar extracted from annotated data to support node label prediction. incorporating structure information into the model enabled our structsp to achieve an exact match score on the top dataset 0.61 points better than with a sota method and demonstrated similar potential results on the topv2 datasets (low-resource setting). this work makes three contributions: (1) an effective fine-tuning approach is introduced for incorporating hierarchical semantic structured information into a pre-trained language model; (2) the use of grammar is introduced in the parsing process to reduce unpromising node label prediction; (3) the structsp framework is shown to outperform existing models in task-oriented semantic parsing on two datasets, top and topv2. ## related work overview task-oriented parsing (top) @xcite and its variant topv2 @xcite were created as benchmarks for assessing the performance of task-oriented semantic parsing models. various approaches have been proposed for tackling the semantic parsing task on these datasets. @xcite introduced a non-autoregressive sequence-to-sequence semantic parsing model, which is based on the insertion transformer model @xcite . @xcite introduced an extension of the hierarchical representation, "decoupled representation", and used sequence-to-sequence (s2s) models based on the pointer generator architecture to parse this representation. @xcite introduced a unified architecture based on s2s mod-els and a pointer-generator network for semantic parsing. @xcite introduced a shift-reduce parser based on recurrent neural network grammars (rnngs) with three improvements to the base rnng model: incorporating contextualized embeddings, ensembling, and pairwise reranking based on a language model. additionally, @xcite showed that compositional top could be formulated as abstractive question answering (qa), where the parse tree nodes are generated by posing queries to a qa model. the rine model @xcite splits the work of parsing an utterance into multiple steps, where the input of each step is the output of the previous step. however, even sota methods ignored the hierarchical structure information during the parsing process. we thus focused on utilizing the hierarchical structure information by using inductive grammar extracted from annotated data to guide the node label predictions. pre-trained language model adaptation several recent studies have demonstrated the value of adapting pre-trained language models to specific tasks with different training objectives, such as in summarization @xcite and knowledge inference @xcite . in the realm of semantic parsing, the score pretraining method @xcite focuses on inducing representations that capture the alignment between dialogue flow and structural context in conversational semantic parsing tasks. grappa @xcite ) is a pre-training approach designed for table semantic parsing and seeks to learn an inductive compositional bias through the joint representation of textual and tabular data. @xcite introduced a semantic-based pre-training approach that uses abstract meaning representation (amr) as explicit semantic knowledge to capture the core semantic information of utterances in dialogues. in comparison, to strengthening hierarchical structure information, we continue training the model by using only annotated data for the finetuning task instead of generating (or augmenting) a large artificial dataset to learn structure information @xcite . this enables the model to effectively performs the hierarchical semantic parsing task using fewer computational resources. grammar-constrained neural network models incorporating constraint grammar in text generation tasks using neural networks is a topic of interest to researchers. @xcite proposed an approach using a grammar model to generate an abstract syntax tree by a series of actions. @xcite presented a typeconstrained semantic parsing model, which ensures that the decoder only generates well-typed logical forms. @xcite demonstrated that the constrained decoding process with an intermediate sub-language mapped by a large language model can support parsing the target language. more recently, @xcite extracted context-free grammar from the target logical form of the semantic parsing task. these grammar constraints were then enforced with an lr parser to maintain syntactically valid sequences throughout decoding. in comparison, our approach utilizes grammar as additional structured information during training with a conditional loss function. this key difference sets our approach apart from previous works. ## experiment to verify the effectiveness of our proposed method, we conducted experiments on datasets top 2 and topv2 3 (low-resource setting). following mansimov and zhang (2022), we conducted three random seeds for each experimental setting and reported the average results with standard deviation. ## analysis 5.1 ablation study to evaluate the effect of using each component in our framework, we compared the model's performance for each combination of component settings with that of the rine baseline model on the validation set of the top dataset ( we found that using grammar in the second phase grammar-based rine led to improved performance on the top dataset (em score 0.12 points higher than with the baseline model). additionally, using the structure-aware boosting phase substantially improved the em score compared with the baseline (0.52 points higher). to further understand the contributions of each training subtask in the structure-aware boosting phase, we conducted two additional experiments. the results show that using the structure-focused mask language modeling subtask improves em performance by 0.36 points compared with the baseline while using the relative tree agreement subtask leads to only a 0.05 improvement. furthermore, we conducted a t-test @xcite with the null hypothesis that the expected values of our full-setting model (structsp) and the baseline model (rine) are identical ( ## conclusion we have presented a novel approach to improving the performance of sota semantic parsing models on hierarchical representation datasets. in this approach, a model is created that incorporates knowledge of the utterance structures into the semantic parsing process. this is achieved by learning contextual representations from the hierarchical representation of utterances with objective functions targeted at the semantic parsing task as well as by using grammar rules containing knowledge about the structure of the data for training and label prediction. in experiments on the top and topv2 datasets, our model outperforms previous sota approaches.