Improving Constituency Parsing with Span Attention (2020)

Abstract: Constituency parsing is a fundamental and important task for natural language understanding, where a good representation of contextual information can help this task. N-grams, which is a conventional type of feature for contextual information, have been demonstrated to be useful in many tasks, and thus could also be beneficial for constituency parsing if they are appropriately modeled. In this paper, we propose span attention for neural chart-based constituency parsing to leverage n-gram information. Considering that current chart-based parsers with Transformer-based encoder represent spans by subtraction of the hidden states at the span boundaries, which may cause information loss especially for long spans, we incorporate n-grams into span representations by weighting them according to their contributions to the parsing process. Moreover, we propose categorical span attention to further enhance the model by weighting n-grams within different length categories, and thus benefit long-sentence parsing. Experimental results on three widely used benchmark datasets demonstrate the effectiveness of our approach in parsing Arabic, Chinese, and English, where state-of-the-art performance is obtained by our approach on all of them.

## introduction constituency parsing, which aims to generate a structured syntactic parse tree for a given sentence, is one of the most fundamental tasks in natural language processing (nlp), and plays an important role in many downstream tasks such as relation extraction @xcite , natural language inference @xcite , and machine translation @xcite . recently, figure 1 : the treelet of an example of the form "v+np+pp", where the "pp" should attach to the "v" (in green) rather than the "np" (in red). in general, besides powerful encoders, other extra information (such as pre-trained embeddings and extra syntactic information) can also provide useful contextual information and thus enhance model performance in many nlp tasks @xcite @xcite @xcite . as one type of the extra information, n-grams are used as a simple yet effective source of contextual feature in many studies @xcite @xcite ) therefore, they could be potentially beneficial for parsing as well. however, recent chart-based parers @xcite @xcite @xcite ) make rare effort to leverage such n-gram information. another potential issue with current figure 2 : the architecture of the chart-based constituency parser with span attention, with an example partial input sentence and its output. the right part of the figure shows the categorical span attention, where extracted n-grams in span (i, j) are categorized by their length so that n-grams in different categories are weighted separately (different colors refer to different n-gram categories). note that for normal span attention, all n-grams are weighted together, where attention a i,j directly corresponds to e i,j,• in the figure . chart-based parsers is that they represent spans by subtraction of hidden states at the span boundaries, where the context information in-between may be lost and thus hurt parsing performance especially for long sentences. n-grams can be a simple yet useful source to fill the missing information. for instance, figure 1 illustrates the treelet of an example in the form of "v+np+pp". as a classic example of pp-attachment ambiguity, a parser may wrongly attach the "pp" to the "np" if it only focuses on the words at the boundaries of the text span "flag ... year" and in-between information is not represented properly. in this case, n-grams within that span (e.g., the uni-gram "telescope") can provide useful cues indicating that the "pp" should be attached to the "v". although there are traditional non-neural parsers using n-grams as features to improve parsing @xcite , they are limited in treating them euqally without learning their weights. therefore, unimportant n-grams may deliver misleading information and lead to wrong predictions. to address this problem, in this paper, we propose a span attention module to enhance chartbased neural constituency parsing by incorporating appropriate n-grams into span representations. specifically, for each text span we extract all its substrings that appear in an n-gram lexicon; the span attention uses the normal attention mechanism to weight them with respect to their contributions to predict the constituency label of the span. because in general short n-grams occur more frequently than long ones, they may dominate in the attention if all n-grams are globally weighted, we further enhance our approach with a categorical mechanism which first groups n-grams into different categories according to their length and then weights them within each category. thus, n-grams with different lengths are separately treated and the infrequent long ones carrying more contextual information can be better leveraged. the effectiveness of our approach is illustrated by experimental results on three benchmark datasets from different languages (i.e., arabic, chinese, and english), on all of which state-of-the-art performance is achieved. ## the approach our approach follows the chart-based paradigm for constituency parsing, where the parse tree t of an input sentence the right side of figure 2 shows the proposed span attention to enhance the backbone parser, where n-grams in x are extracted from a preconstructed lexicon n and are weighted through the attention module according to their contribution to the parsing process. therefore, the process of computing s(i, j, l) of the labeled spans through our approach is formalized by in this section, we start with a brief introduction of neural chart-based parsing, then describe our span attention, and end with an illustration of incorporating span attention into the parsing process. ## related work there are two main types of parsing methodologies. one is the transition-based approaches @xcite ; the other is the chart-based approaches @xcite . recently, neural methods start to play a dominant role in this task, where improvements mainly come from powerful encodings @xcite @xcite @xcite @xcite . moreover, there are studies that do not follow the aforementioned methodologies, which instead regard the task as a sequence-to-sequence generation task @xcite , a language modeling (choe and charniak, 2016) task or a sequence labeling task @xcite . to further improve the performance, some studies leverage extra resources (such as auto-parsed large corpus @xcite , pre-trained word embeddings @xcite ), hpsg information @xcite , or use model ensembles @xcite . compared to these studies, our approach offers an alternative way to enhance constituency parsing with effective leveraging of n-gram information. moreover, the proposed span attention addresses the limitation of previous studies @xcite that spans are represented by the subtraction of encoded vectors at span boundaries (i.e., the hidden states at initial and ending positions of the span) and thus reduces information loss accordingly. in addition, the categorical span attention provides a simple, yet effective, improvement over the normal attention to process n-grams in a more precise way, which could become a reference for leveraging similar resources in future research. ## conclusion in this paper, we proposed span attention to integrate n-gram into span representations to enhance chart-based neural constituency parsing. specifically, for each text span in an input sentence, we firstly extracted n-grams in that span from an ngram lexicon, and then fed them into the span attention to weight them according to their contribution to the parsing process. to better leverage n-grams, especially the long ones, categorical span attention was proposed to improve the normal attention by categorizing n-grams according to their length and weighting them separately within each category. such span attention not only leverages important contextual information from n-grams but also addresses the limitation of current transformer-based encoders using subtraction for span representations. to the best our knowledge, this is the first work using n-grams for neural constituency parsing. the effectiveness of our approach was demonstrated by experimental results on three benchmark datasets from arabic, chinese, and english, where state-ofthe-art performance is obtained on all of them. n-grams avg. attention said 0.0141 more than 0.0063 as well as 0.0167 from a year earlier 0.0267 in the past few years 0.0184 for arabic, chinese, and english on the entire atb, ctb5, and ptb datasets, respectively. then, for each n-gram, we compute its average attention weights according to its appearance in the entire dataset. afterwards, we group n-grams by their length and rank the n-grams according to their average attention weights within each group. the top 50 n-grams in each group as well as their attention weights for each language are reported in the supplemental material. as a demonstration,