Syntax Matters: Towards Spoken Language Understanding via Syntax-Aware Attention (2023)

Abstract: Spoken Language Understanding (SLU), a crucial component of task-oriented dialogue systems, has consistently garnered attention from both academic and industrial communities. Although incorporating syntactic information into models has the potential to enhance the comprehension of user utterances and yield impressive results, its application in SLU systems remains largely unexplored. In this paper, we propose a carefully designed model termed Syntax-aware attention (SAT) to enhance SLU, where attention scopes are constrained based on relationships within the syntactic structure. Experimental results on three datasets show that our model achieves substantial improvements and excellent performance. Moreover, SAT can be integrated into other BERT-based language models to further boost their performance.

## introduction in recent years, there has been a surge of research interest in dialogue systems from academia and industry, where slu @xcite @xcite is a crucial component, responsible for identifying user intents based on their natural language utterances @xcite @xcite . many works have been proposed to enhance the understanding of utterance semantics, leading to remarkable successes. while language models (lms) have the ability to capture contextual semantics to some extent, their performance can be further improved by integrating external knowledge @xcite . the inclusion of additional knowledge in lms has become a growing trend, and researchers have extensively explored the integration of syntax information into pre-trained lms @xcite @xcite . incorporating syntax information through parser dependencies into plms improves performance compared to intent utterance num how many watts make a kilowatt ? abbr what is the abbreviation for micro ? desc what caused the death of bob marley ? enty what kind of creature is a coot ? loc where is the valley of the kings ? hum who was the second man to walk on the moon ? what asian city boasts the world 's biggest bowling alley ? training with utterances alone. however, relevant research on slu is lacking. to address the deficiency, we propose syntaxaware attention (sat) for slu. specifically, we first utilize a pre-trained dependency parse tree structure to generate the related nodes for each word in a sentence. these related nodes, known as syntax-aware attention mask, consist of all ancestor nodes and the word itself in the dependency parsing tree, treating the word as a child node. examples of intent detection and syntax parsing are shown in figure 1 . to effectively incorporate syntax-aware attention module (satm), we also design the dual context fusion module (dcfm) to merge the original transformer @xcite with the syntax-aware transformer. this enables sat to provide more linguistically inspired representations for slu. to summarize, our contributions are three-fold: (1) sat can capture the syntactic structure information from utterances for enhancing slu. it can be straightforward integrated into bert-based models. (2) in order to effectively merge syntactic information, various feature fusion methods were explored with the aim of enhancing linguistic representation. (3) experiments demonstrate the effectiveness on three slu benchmarks. ## methodology the main architecture mainly consisting of satm and dcfm is illustrated in figure 2 . we elaborate on each component below. ## conclusion in this paper, we propose a novel syntax-aware attention (sat) to improve slu by incorporating syntactic information, specifically dependency structures, into plm. experimental results demonstrate the superiority of our method over the backbone bert/roberta models on three benchmark datasets. moreover, we experiment with multiple parsers and find that parsers with stronger syntactic parsing abilities lead to more substantial improvements in slu systems. detailed analysis confirms the effectiveness and power of sat in enhancing slu systems.