Modeling Compositionality with Dependency Graph for Dialogue Generation (2022)

Abstract: Because of the compositionality of natural language, syntactic structure which contains the information about the relationship between words is a key factor for semantic understanding. However, the widely adopted Transformer is hard to learn the syntactic structure effectively in dialogue generation tasks. To explicitly model the compositionaity of language in Transformer Block, we restrict the information flow between words by constructing directed dependency graph and propose Dependency Relation Attention (DRA). Experimental results demonstrate that DRA can further improve the performance of state-of-the-art models for dialogue generation.

## introduction in natural language, complex semantics are often expressed by combining words with certain rules. for example, "room" can express higher-level semantics by fusing the information of "a" and "hotel", and the meaning of "reserve" will be clearer after fusing the information of "room". prior works have achieved great success in nlp tasks by leveraging syntactic structure knowledge, such as semantic relatedness @xcite , sentiment analysis @xcite , relation extraction @xcite , and named entity recognition @xcite . due to the strong ability to capture long-term dependencies @xcite , many recent works have adopted the transformer block @xcite to extract context features in dialogue generation tasks @xcite @xcite . however, it is hard for transformer block to implicitly learn the compositionality of language in the training process of dialog generation, since it simply uses position embeddings to represent the relationships between words, and it learns the local position information that can only be effective in masked language modeling @xcite . besides, the computation of attention weights on unrelated word pairs in transformer block is redundant and decreases performance. to obtain better distributed representations of context in dialogue generation tasks, we propose dependency relation attention to model the relationship between words as an alternative to position embeddings. specifically, we incorporate dependency relation knowledge that contains syntactic structure information into the transformer block. as shown in figure 1 , we use the dependency parser @xcite in the stanfordcorenlp toolkit @xcite to build dependency graphs of utterances. then, the dependency relation mask is generated to avoid performing attention on words without dependency relations, and the fusion of information among words depends on the direction specified by the dependency graph. our contributions can be summarized as follows: in the past few years, dependency graph has drawn attention from many researchers in the field of nlp. strubell et al. (2018) propose syntacticallyinformed self-attention and incorporate syntactic dependency knowledge into a attention head of specific transformer block. to make the attention learned by transformer more interpretable, wang et al. (2019) propose constituent attention which makes each position not attend to the position in different constituents. ahmad et al. (2021) explicitly fuse structural information to learn the dependency relations between words with different syntactic distances. in dialogue generation tasks, to improve the quality of generated responses, previous works focus on capturing the high-level relationships between contexts and responses @xcite or between utterances in context @xcite . how to effectively model the relationships between words in transformer has not been explored. inspired by @xcite , our method aim at modeling the compositionality in language, then the transformer block does not need to learn the relationships between words through position embeddings in the training process of dialog generation. the differences between dra and others dependency relation-aware attention mechanisms are: (1) dra incorporates the dependency arc directions into transformer block to model the relationships between words instead of position embeddings. (2) the position embeddings are excluded for the models with dra applied. ## method in dialogue generation tasks, given a piece of context containing m utterances @xmath0, ..., x m } as inputs, where x @xmath1,1 , ..., x i,n i }, @xmath2, m] indicates the i-th utterance containing n i words, dialogue generation models map it into feature vectors and estimate the generation probability of the corresponding response @xmath3, ..., y t }: to obtain a better representations of context, we incorporate dependency relation knowledge into the transformer block, which is widely used in recent works. ## conclusion and future work in this paper, we propose dependency relation attention (dra) to model the relationships between words instead of position embeddings in the transformer encoder. experimental results show that our method can further improve the performance of models that use transformer block to obtain the distributed representations of context in dialogue generation task. in the future, we will study the effect of the specific domains that parsers are usually trained in, as well as the possibility of improving the performance of pretrained language models with dra.