Bidirectional Masked Self-attention and N-gram Span Attention for Constituency Parsing (2023)

Abstract: Attention mechanisms have become a crucial aspect of deep learning, particularly in natural language processing (NLP) tasks. However, in tasks such as constituency parsing, attention mechanisms can lack the directional information needed to form sentence spans. To address this issue, we propose a Bidirectional masked and N-gram span Attention (BNA) model, which is designed by modifying the attention mechanisms to capture the explicit dependencies between each word and enhance the representation of the output span vectors. The proposed model achieves state-of-the-art performance on the Penn Treebank and Chinese Penn Treebank datasets, with F1 scores of 96.47 and 94.15, respectively. Ablation studies and analysis show that our proposed BNA model effectively captures sentence structure by contextualizing each word in a sentence through bidirectional dependencies and enhancing span representation.

## introduction the concept of attention has become a major aspect of deep learning, and improving attention is essential to enhance the model efficacy. in natural language processing (nlp), numerous studies that utilize the sequence-to-sequence model have achieved significant performance improvements by modifying the attention mechanisms to specific tasks. tasks such as summarization @xcite , translation @xcite , question answering @xcite , and multi-modal learning @xcite are examples of the efficacy of such mechanisms in improving model performance. in the constituency parsing task, which involves identifying constituent phrases and their relationships in a sentence, attention mechanisms, especially self-attention, improves the performance of a parser. many studies on constituency parsing have emphasized the importance of comprehending sentence spans to improve parser performance @xcite @xcite . recent studies that incorporate attention mechanisms train parsers to comprehend sentence spans by referring to the n-grams of a sentence as the span @xcite or by considering the directional and positional dependencies from splited word representation @xcite . however, because attention mechanisms compute the dependency of each element simultaneously, there can be a lack of the directional information that is needed to form sentence spans. this contrasts with long short-term memory (lstm) models that consider directional information. in attention mechanisms that use attention weights between the query and key vectors as relational information between each element, the weights are computed regardless of the element's relative position. previous studies @xcite acknowledged that this method could be problematic and made efforts to address it. however, such attempts were conducted under the assumption of ideal learning conditions, and the problem in the calculation process has persisted. the purpose of this paper is to modify the attention mechanism into two types of capability. the first one obtains explicit directional information for each word, similar to the approach used by bidirectional lstm (figure 1(b) ). the second one enhances the representation of each word by incorporating information from spans, which are suitable for constituency parsing. in this work, we propose a novel model called bna (bidirectional masked and ngram span attention). bna employs a variant of masked self-attention (msa) in which each element in a sequence is considered sequentially by its attention weights bidirectionally, rather than simultaneously. moreover, bna incorporates a novel span attention mechanism that represents a key-value matrix by subtracting the hidden states at the span boundaries. this approach enables the query (i.e., word sequence) to access the contextual information of n spans in a sentence. our parser achieves state-of-the-art performance with f1 scores of 96.47 and 94.15 for the penn treebank and chinese treebank datasets, respectively. in addition, through ablation study and analysis, we demonstrate that our proposed bna model effectively captures sentence structure by contextualizing each word in a sentence through bidirectional dependencies and enhancing span representation. ## related work in the field of constituency parsing, since the introduction of the span-based approach by @xcite , chart-based neural parsers have outperformed transition-based ones (zhang, 2020). the span-based approach involves labeling specific text spans instead of individual tokens or words, enabling the parsers to consider the context and relationships between different spans of the sentence. with the rise of the transformer model @xcite in nlp, attention mechanisms have become an attractive alternative to lstm networks. in constituency parsing, attention mechanisms have shown promising results, as demonstrated by @xcite , who used a self-attentive network applied to the span-based parser to improve performance. they split the input vector into content and position representations and performed self-attention on each component separately. building on this work, mrini et al. (2020) introduced label attention layers, a modified form of self-attention that enables the model to learn label-specific views of the input sentence. in this mechanism, the attention heads are split into half, forward and backward representations, which are then used to construct span vectors of the input sentence. more recently, @xcite proposed span attention, which assumes no strong dependency between each hidden vector in a transformerbased encoder. their method involves enhancing the span representation by summing the attention vector of n-grams consisting of embedded word vectors with the span vector, without using directional vectors. however, conventional attention mechanisms treat all elements simultaneously without considering directional dependencies, making it challenging to construct span vectors using an encoder based on the attention mechanism. furthermore, constructing arbitrary span vectors from embedded words that lack contextual information of the sentence could be improved. in this paper, we introduce two types of attention mechanisms that address the issue of directional dependencies and that strengthen span representation. ## background self-attention is a powerful mechanism that enables neural networks to capture dependencies between different parts of a sequence. the basic idea behind self-attention is to compute a representation of the entire sequence by weighting the importance of different elements in the sequence based on their similarity to each other. in a typical self-attention sub-layer, the sequence of input vectors @xmath0, ..., x n ] is transformed into three sequences of vectors: queries @xmath1, ..., q n ], keys @xmath2, ..., k n ], and values @xmath3, ..., v n ]. these sequences are computed using learned linear projections: attention weights α i,j are computed as the dot product of the query vector q at position i and the key vector k at position j, which is subsequently normalized using the softmax function as follows: this weighted sum h i can be seen as a hidden representation of the i-th vector that considers the importance of each of the other vectors in the sequence. ## approach our approach is motivated by the problem that self-attention mechanisms struggle to encode the relative positions and sequential order of elements within the context of a sequence @xcite . studies have been conducted to resolve this issue in tasks that require bidirectional information, such as relation extraction @xcite and machine translation @xcite . to address this issue, we propose the bidirectional masked self-attention (bimsa) and n-gram span attention (nsa) mechanisms. together, these two attention mechanisms comprise our bidirectional masked and n-gram span attention (bna) model. section 4.1 provides a brief overview of the constituency parsing process. section 4.2 provides a more detailed explanation of bimsa and nsa and how they are integrated into the bna model. ## conclusions the primary goal of this study was to design attention mechanisms to capture the explicit dependencies between each word and enhance the representation of the output span vectors. through our experiments, we demonstrated that our proposed bimsa more effectively contextualizes each word in a sentence by considering the bidirectional dependencies, while nsa improves the span representation by attending to arbitrary n-gram spans. our findings have major implications for span-based approaches in constituency parsing tasks. specifically, applying the span representation method to the attention mechanism leads to a significant performance improvement. in conclusion, constructing a span representation from words contextualized within a given sentence can lead to additional improvement in parsing. overall, our study contributes to the advancement of attention mechanisms in nlp. we hope that our findings will inspire further research in this area.