Syntax-Aware Graph-to-Graph Transformer for Semantic Role Labelling (2023)

Abstract: Recent models have shown that incorporating syntactic knowledge into the semantic role labelling (SRL) task leads to a significant improvement. In this paper, we propose Syntax-aware Graph-to-Graph Transformer (SynG2G-Tr) model, which encodes the syntactic structure using a novel way to input graph relations as embeddings, directly into the self-attention mechanism of Transformer. This approach adds a soft bias towards attention patterns that follow the syntactic structure but also allows the model to use this information to learn alternative patterns. We evaluate our model on both span-based and dependency-based SRL datasets, and outperform previous alternative methods in both in-domain and out-of-domain settings, on CoNLL 2005 and CoNLL 2009 datasets.

## introduction the task of semantic role labelling (srl) provides a shallow representation of the semantics in a sentence, and constructs event properties and relations among relevant words. traditionally, a syntactic structure was considered a prerequisite for srl models @xcite , but, newer models that leverage deep neural network architectures @xcite @xcite have outperformed syntax-aware architectures, without the need for explicit encoding of syntactic structure. however, recent studies @xcite @xcite have proposed that deep neural network models could benefit from using syntactic information, rather than disregarding it. these studies suggest that incorporating syntax into the model can improve srl prediction by jointly learning both syntactic and semantic structures @xcite , training a self-attention head in transformer @xcite to attend to each token's syntactic parent @xcite , or encoding the syntactic structure using graph convolutional networks @xcite . in this paper, we propose a novel method for encoding syntactic knowledge by introducing syntax-aware graph-to-graph transformer (syng2g-tr) architecture. the model conditions on the sentence's dependency structure and jointly predicts both span-based and dependency-based srl structures. inspired by @xcite , our model inputs graph relations as embeddings incorporated into the self-attention mechanism of transformer @xcite . different from the original graph-to-graph transformer, our self-attention function models the interaction of the graph relations with both the query and key vectors of self-attention mechanism, instead of just the query. we also find that excluding the interaction of graph structure with the value vectors of self-attention does not harm the performance. furthermore, compared to the previous work on graph-to-graph transformers @xcite , our architecture uses different types of graphs as the input and output. we show empirically that our model outperforms previous comparable models. • we evaluate our model on conll 2005 and conll 2009 datasets and outperform previous comparable models in most cases of both in-domain and out-of-domain sets. the architecture of the syng2g-tr model is illustrated in figure 1 . the input to the model is the tokenised text (@xmath0,w 2 ,...,w n )), which are the nodes of the input and output graphs, and n is the length of tokenised input. the outputs are the dependency-based (g dep ) and span-based (g span ) srl graphs. the syng2g-tr model can be formalised in terms of an encoder e sg2g and decoder d sg2g : initially, a syntactic parser predicts the dependency graph (g syn ), and part-of-speech (pos) tags (@xmath1, p 2 , ..., p n )). then the encoder of syng2g-tr (e sg2g ) encodes both sequences (w, p ) and the dependency graph (g syn ) into contextualised representations of graph nodes (z). this representation (z) is then used by the decoder (d sg2g ) to jointly predict srl graphs. for the decoder, we follow the same unified scorer and decoder as defined in @xcite . further explanation of srl scorer and decoding mechanism is provided in appendix a. the encoder employs an enhanced way of inputting graph relations into the self-attention mechanism of transformer @xcite . unlike the previously proposed version of graph-to-graph transformer @xcite , we modify the self-attention mechanism to have a more comprehensive interaction between graph relations, queries and keys. we also find that excluding the interaction of graph relations with value vectors retains good performance. ms haag plays elianti . syntactic parser root ms haag plays elianti nsubj obj root comp . punct root ms haag plays elianti . sep z1 z2 z3 z4 z5 z6 z7 syng2g transformer ... ... score function decoder root ms haag plays elianti a1 a0 play.02 dependency-based srl . root ms haag plays elianti a1 a0 play.02 span-based srl . nnp,nnp,vbz,nnp, . specifically, given the output of an intermediate embedding layer @xmath2,...,x n ), we define the attention mechanism of each head in each layer to take the dependency graph as input. these attention scores (α ij ) are calculated as a softmax function over e ij values: syntactic parser. the parser jointly predicts pos tags and the dependency graph. we apply the parser defined in @xcite , which uses a joint scorer and decoder for dependency and constituency graphs based on head-driven phrase structure grammar @xcite . this method has achieved state-of-the-art results in the dependency parsing task. several approaches have been proposed to use syntax for the srl task. roth and lapata (2016) embed dependency paths, while some researchers (fei et al., 2021; munir et al., 2021; marcheggiani and titov, 2017) use graph convolutional networks to encode the syntactic structure. strubell et al. (2018) incorporates a dependency graph by training one attention head of transformer to attend to syntactic parents for each token, in a multi-task setting. he et al. (2019, 2018b) use syntactic information to guide the argument pruning. xia et al. (2019) exploit different alternatives e.g. tree-structured gru and graph features of dependency tree to encode syntactic knowledge. kasai et al. (2019) apply bilstm to tag the text with supertags extracted from dependency parses and feed them into srl models. xia et al. (2020) showed that incorporating heterogeneous syntactic knowledge results in significant improvement. some other work focus on joint learning of both srl and syntax @xcite . additionally, some approaches discarded the syntax, but achieve impressive results @xcite @xcite @xcite @xcite . and conll 2009 @xcite ). 2 for predicate disambiguation, we follow previous work @xcite , and use an off-the-shelf disambiguator from @xcite . as in previous work, we evaluate in both end-to-end, and given predicate settings. for a more accurate comparison, we train syng2g-tr both with and without bert initialisation (syng2g-tr w/o bert). the discrepancy between bert tokenisation and the tokenisation used in the srl corpora is handled as in mohammadshahi and henderson (2020). 3 for the syntactic parser, we use the same hyper-parameters as defined in @xcite . the performance of the syntactic parser is shown in conll 2005 results. 4 the results for spanbased srl are shown in ## conclusion in this paper, we propose the syntax-aware graph-to-graph transformer architecture, which effectively incorporates syntactic information by inputting the syntactic dependency graph into the self-attention mechanism of transformer. our mechanism for inputting graph relation embeddings differs from the original graph-to-graph transformer in that it models the complete interaction between the dependency relation, query vector and key vector. it also excludes the graph interaction with value vectors while maintaining good performance. we have evaluated our model on conll 2005 and conll 2009 srl datasets and outperformed previous comparable models. future studies can apply our model to any nlp task which might benefit from conditioning on the syntactic structure or other graphs.