Multi-Layer Pseudo- S iamese Biaffine Model for Dependency Parsing (2022)

Abstract: Biaffine method is a strong and efficient method for graph-based dependency parsing. However, previous work only used the biaffine method at the end of the dependency parser as a scorer, and its application in multi-layer form is ignored. In this paper, we propose a multi-layer pseudo-Siamese biaffine model for neural dependency parsing. In this model, we modify the biaffine method so that it can be utilized in multi-layer form, and use pseudo-Siamese biaffine module to construct arc weight matrix for final prediction. In our proposed multi-layer architecture, the biaffine method plays important roles in both scorer and attention mechanism at the same time in each layer. We evaluate our model on PTB, CTB, and UD. The model achieves state-of-the-art results on these datasets. Further experiments show the benefits of introducing multi-layer form and pseudo-Siamese module into the biaffine method with low efficiency loss.

## introduction dependency parsing is a fundamental task in nlp. given a input sequence @xmath0...w n , the output is a dependency tree @xmath1, d, l), 0 ≤ @xmath2, 1 ≤ @xmath3, @xmath4, where w i (1 ≤ @xmath5, w 0 is a pseudo-word as root and (h, d, l) is an arc from w h to w d with label l in a relation set l. due to the simplicity and effectiveness of representing syntactic information by using the tree structure, many works involve dependency parsing, such as syntax-enhanced pre-trained model @xcite . there are two approaches to dependency parsing: transition-based and graph-based methods. graphbased dependency parsing scores the components of a sentence and selects the highest scoring tree. @xcite for the first time introduce biaffine method into the dependency parsing. this strong and efficient biaffine first-order graphbased parser consists of bilstm encoder and bi-affine scorer. after the biaffine parser is proposed, many works have focused on further improving the performance of this parser. since the biaffine parser consists of two parts: bilstm encoder and biaffine scorer, there are two main directions of improvement focusing on the two parts respectively. one direction of improvement is modifying the encoder. @xcite introduce pre-trained model in the embedding stage. @xcite fuse the hidden states of different layers of bilstm to construct the input of the biaffine scorer. @xcite add graph neural networks before the biaffine scorer to capture high-order information. @xcite apply the self-attention mechanism as the replacement of bilstm encoder. @xcite introduce the label attention layer, a new form of self-attention where attention heads represent labels, into the encoder. in these works, although the encoder changes, the biaffine method remains as a scorer at the end of dependency parser. another direction is extending the biaffine scorer for second-order parser. currently, there are relatively few works in this direction due to its difficulty. @xcite introduce efficient treecrf. wang and tu (2020) introduce mfvi. both works extend the biaffine scorer to the triaffine scorer for scoring second-order subtrees. the results show that introducing high-order information is beneficial to the dependency parser in many ways. in these works, the extended biaffine method still plays the role of a scorer for high-order modeling. we observe that biaffine method only plays the role of a scorer in almost all graph-based dependency parsers improved by the biaffine parser. the ways and effects of making the method play more roles remains to be explored. this leads us to a new direction of improvement: how to make biaffine method play more roles in dependency parser? we notice that biaffine method is essentially an attention mechanism, so it can be utilized in a multi-layer architecture like transformers @xcite to capture more information before the biaffine scorer. however, such an application of biaffine method is equivalent to adding transformers which uses biaffine attention upon bilstm encoder and keeping the biaffine scorer at the end of the model unchanged, that is, the biaffine scorer is independent of the multi-layer architecture, so it cannot obtain more information, which limits the performance gains. this raises a question: how to fuse the biaffine scorer into the multi-layer architecture, which allows the biaffine scorer to make use of more information? to address this question, we propose a multilayer pseudo-siamese biaffine model for graphbased dependency parsing. our multi-layer architecture consists of several connectable layers with the same structure, and each layer contains two biaffine modules that are identical but have separate parameters, namely pseudo-siamese module. one biaffine module is used as an attention mechanism to obtain the attention weight to construct the token representation for the next layer; the other is used as a scorer to obtain the arc weight matrix that contributes to final prediction. therefore, in our proposed multi-layer architecture, the biaffine method plays the role of a scorer and an attention mechanism at the same time in each layer. the biaffine scorer can obtain information from each layer before, instead of only seeing the output of the last layer, so it can make use of more information captured by the multi-layer architecture. we conduct experiments on ptb, ctb, and ud to verify the effectiveness of our model. since the usage of pre-trained models like bert @xcite and xlnet @xcite in the embedding stage is common for dependency parsing, we evaluate our model with different pretrained models. we conduct the experiments in three aspects: 1) compare our model with previous state-of-the-art dependency parsers using the same pre-trained model to show that our model achieves state-of-the-art performance; 2) implement the original biaffine model and compare it with our model when using the same pre-trained model to show that our modification based on biaffine model significantly improves the performance; 3) compare our multi-layer architecture with other multi-layer architectures for dependency parsing to justify the specific layers we propose. we also conduct detailed analysis to illustrate four aspects relevant to our model: 1) the impact of the choice of number of layers and attention function on our model; 2) the importance of fusing the biaffine scorer into the multi-layer architecture; 3) the benefit of introducing multi-layer form and pseudo-siamese module into the biaffine method in many ways, including the overall performance, the performance on short and long sentences respectively and the performance of label prediction; 4) the low efficiency loss of introducing multi-layer form and pseudo-siamese module into the biaffine method. in summary, the major contributions of our work are as follows: ## model our graph-based dependency parser contains three parts, i.e., encoder, multi-layer biaffine model, and pseudo-siamese module. ## related work transition-based method @xcite and graph-based method (mc-donald et al., 2005; mcdonald and pereira, 2006) are two main approaches to dependency parsing. before the deep learning era, almost all dependency parsers classify based on many sparse indicator features. titov and henderson (2007) for the first time introduce neural networks into transition-based dependency parsing. chen and manning (2014) propose a fast and accurate transition-based dependency parser using neural networks. dyer et al. (2015) introduce bilstm into transition-based dependency parsing. wang and chang (2016) introduce bilstm encoder into the graph-based dependency parser. kiperwasser and goldberg (2016) also introduce bilstm encoder into dependency parsing and test it with both transition-based and graph-based methods. based on these works, dozat and manning (2017) introduce biaffine method as a scorer into the graph-based dependency parsing and achieve significant performance improvement. multi-layer architecture is widely used to improve the performance of the original biaffine parser. since the original biaffine parser uses multilayer bilstm, @xcite propose to use hidden states of different layers of bilstm to construct role-dependent representations for each layer, which are aggregated as the input of the biaffine scorer. @xcite propose to use multi-layer self-attention at the encoder stage as the replacement of bilstm. @xcite propose to add multi-layer graph neural networks before the biaffine scorer and apply layer-wise loss to capture high-order information. @xcite propose a recursive non-autoregressive graph-to-graph transformer for dependency parsing. they introduce a multi-layer architecture similar to bert but changing the attention function. in these models, the biaffine method is used as a scorer after the multi-layer architecture. differently, our multi-layer architecture uses the biaffine method in each layer as the role of an attention mechanism and a scorer at the same time. pseudo-siamese network architecture, which means using two separate but identical networks to process a pair of inputs and fusing the information at a later stage, is widely used in the field of computer vision @xcite . in our model, we use two separate but identical biaffine modules in each layer to process the same input into two branches, which is a reversed pseudo-siamese network architecture. ## conclusions in this paper, we propose a multi-layer biaffine pseudo-siamese model for neural dependency parsing. in our proposed multi-layer architecture, biaffine method plays the role of a scorer and an attention mechanism at the same time in each layer. we conduct experiments and detailed analysis on ptb, ctb, and ud, showing the benefit of introducing multi-layer form and pseudo-siamese module into the biaffine method. compared with the original single-layer biaffine model, our model has a significant advantage in performance with low efficiency loss. our analysis shows that fusing the biaffine scorer into the multi-layer architecture is important for the performance gains of our model. compared with other multi-layer architectures for dependency parsing, our multi-layer architecture also has a performance advantage. our model achieves state-of-the-art results on ptb, ctb, and ud. our code is available at https://github. com/xzy-xzy/mlpsb-parser .