Investigating Non-local Features for Neural Constituency Parsing (2022)

Abstract: Thanks to the strong representation power of neural encoders, neural chart-based parsers have achieved highly competitive performance by using local features. Recently, it has been shown that non-local features in CRF structures lead to improvements. In this paper, we investigate injecting non-local features into the training process of a local span-based parser, by predicting constituent n -gram non-local patterns and ensuring consistency between non-local patterns and local constituents. Results show that our simple method gives better results than the self-attentive parser on both PTB and CTB. Besides, our method achieves state-of-the-art BERT-based performance on PTB (95.92 F1) and strong performance on CTB (92.31 F1). Our parser also outperforms the self-attentive parser in multi-lingual and zero-shot cross-domain settings.

## introduction constituency parsing is a fundamental task in natural language processing, which provides useful information for downstream tasks such as machine translation @xcite , natural language inference @xcite , text summarization @xcite . over the recent years, with advance in deep learning and pre-training, neural chart-based constituency parsers @xcite have achieved highly competitive results on benchmarks like penn treebank (ptb) and penn chinese treebank (ctb) by solely using local span prediction. the above methods take the contextualized representation (e.g., bert) of a text span as input, and use a local classifier network to calculate the scores of the span being a syntactic constituent, together with its constituent label. for testing, the output layer uses a non-parametric dynamic programming algorithm (e.g., cky) to find the highest-scoring tree. without explicitly modeling structured dependencies between different constituents, the methods give competitive results compared to non-local discrete parsers @xcite . one possible explanation for their strong performance is that the powerful neural encoders are capable of capturing implicit output correlation of the tree structure @xcite @xcite . recent work has shown that modeling non-local output dependencies can benefit neural structured prediction tasks, such as ner @xcite , ccg supertagging @xcite and dependency parsing @xcite . thus, an interesting research question is whether injecting non-local tree structure features is also beneficial to neural chart-based constituency parsing. to this end, we introduce two auxiliary training objectives. the first is pattern prediction. as shown in figure 1 , we define pattern as the n-gram constituents sharing the same parent. 1 we ask the model to predict the pattern based on its span representation, which directly injects the non-local constituent tree structure to the encoder. to allow stronger interaction between non-local patterns and local constituents, we further propose a consistency loss, which regularizes the cooccurrence between constituents and patterns by collecting corpus-level statistics. in particular, we count whether the constituents can be a sub-tree of the pattern based on the training set. for instance, both nns and np are legal to occur as sub-trees of the 3-gram pattern {vbd np pp} in figure 1 , while s or adjp cannot be contained within this pattern based on grammar rules. similarly, for the 2-gram pattern {np pp} highlighted in figure 1 , both in and np are consistent constituents, but jj is not. the consistency loss can be considered as injecting prior linguistic knowledge to our model, which forces the encoder to understand the grammar rules. non-local dependencies among the constituents that share the same pattern are thus explicitly modeled. we denote our model as injecting non-local features for neural chart-based parsers (nfc). we conduct experiments on both ptb and ctb. equipped with bert, nfc achieves 95.92 f1 on ptb test set, which is the best reported performance for bert-based single-model parsers. for chinese constituency parsing, nfc achieves highly competitive results (92.31 f1) on ctb, outperforming the baseline self-attentive parser (91.98 f1) and a 0-th order neural crf parser (92.27 f1) @xcite . to further test the generalization ability, we annotate a multi-domain test set in english, including dialogue, forum, law, literature and review domains. experiments demonstrate that nfc is robust in zero-shot cross-domain settings. finally, nfc also performs competitively with other languages using the spmrl 2013/2014 shared tasks, establishing the best reported results on three rich resource languages. we release our code and models at https://github.com/ ringos/nfc-parser . ## related work constituency parsing. there are mainly two lines of approaches for constituency parsing. transition-based methods process the input words sequentially and construct the output constituency tree incrementally by predicting a series of local transition actions @xcite @xcite . for these methods, the sequence of transition actions make traversal over a constituent tree. although transition-based methods directly model partial tree structures, their local decision nature may lead to error propagation @xcite and worse performance compared with methods that model long-term dependencies @xcite . similar to transition-based methods, nfc also directly models partial tree structures. the difference is that we inject tree structure information using two additional loss functions. thus, our integration of nonlocal constituent features is implicit in the encoder, rather than explicit in the decoding process. while the relative effectiveness is empirical, it could potentially alleviate error propagation. chart-based methods score each span independently and perform global search over all possible trees to find the highest-score tree given a sentence. @xcite represented nonlinear features to a traditional crf parser computed with a feed-forward neural network. @xcite first used lstm to represent span features. kitaev and klein (2018) adopted a self-attentive encoder instead of the lstm encoder to boost parser performance. @xcite proposed label attention layers to replace self-attention layers. @xcite integrated constituency and dependency structures into head-driven phrase structure grammar. @xcite used span attention to produce span representation to replace the subtraction of the hidden states at the span boundaries. despite their success, above work mainly focuses on how to better encode features over the input sentence. in contrast, we take the encoder of @xcite intact, being the first to explore new ways to introduce non-local training signal into the local neural chart-based parsers. modeling label dependency. there is a line of work focusing on modeling non-local output dependencies. @xcite used a bayesian network to encode the label dependency in multilabel learning. for neural sequence labeling, @xcite and @xcite built a crf layer on top of neural encoders to capture label transition patterns. @xcite introduced a sentence-level constraint to encourage the model to generate coherent ner predictions. @xcite investigated label attention network to model the label dependency by producing label distribution in sequence labeling tasks. @xcite proposed a two-stage label decoding framework based on bayesian network to model long-term label dependencies. for syntactic parsing, @xcite demonstrated that structured tree crf can boost parsing performance over graph-based dependency parser. our work is in line with these in the sense that we consider non-local structure information for neural structure prediction. to our knowledge, we are the first to inject sub-tree structure into neural chart-based encoders for constituency parsing. ## baseline our baseline is adopted from the parsing model of @xcite and @xcite . given a sentence @xmath0, ..., x n }, its corresponding constituency parse tree t is composed by a set of labeled spans following @xcite , we use bert with a self-attentive encoder as the scoring function s(i, j, •), and a chart decoder to perform a globaloptimal search over all possible trees to find the highest-scoring tree given the sentence. in particular, given an input sentence @xmath1, ..., x n }, a list of hidden representations h n 1 = {h 1 , h 2 , . . . , h n } is produced by the encoder, where h i is a hidden representation of the input token x i . following previous work, the representation of a span (i, j) is constructed by: finally, v i,j is fed into an mlp to produce realvalued scores s(i, j, •) for all constituency labels: ## additional training objectives we propose two auxiliary training objectives to inject non-local features into the encoder, which rely only on the annotations in the constituency treebank, but not external resources. ## experiments we empirically compare nfc with the baseline parser in different settings, including in-domain, cross-domain and multilingual benchmarks. ## analysis 6.1 n-gram pattern level performance figure 4 shows the pattern-level f1 before and after introducing the two auxiliary training objectives. in particular, we calculate the pattern-level f1 by calculating the f1 score for patterns based on the constituency trees predicted by cky decoding. although our baseline parser with bert achieves 95.76% f1 scores on ptb, the patternlevel f1 is 80.28% measured by 3-gram. when testing on the dialogue domain, the result is reduced to only 57.47% f1, which indicates that even a strong neural encoder still has difficulties capturing constituent dependency from the input sequence alone. after introducing the pattern and consistency losses, nfc significantly outperforms the baseline parser measured by 3-gram pattern f1. though there is no direct supervision signal for 2-gram pattern, nfc also gives better results on pattern f1 of 2-gram, which are subsumed by 3-gram patterns. this suggests that nfc can effectively represent sub-tree structures. ## conclusion we investigated graph-based constituency parsing with non-local features -both in the sense that features are not restricted to one constituent, and in the sense that they are not restricted to each training instance. experimental results verify the effectiveness of injecting non-local features to neural chart-based constituency parsing. equipped with pre-trained bert, our method achieves 95.92% f1 on ptb and 92.31% f1 on ctb. we further demonstrated that the proposed method gives better or competitive results in multilingual and zero-shot cross-domain settings.