High-Accuracy Transition-Based Constituency Parsing (2025)

Abstract: Constituency parsers have improved markedly in recent years, with the F1 accuracy on the venerable Penn Treebank reaching 96.47, half of the error rate of the first transformer model in 2017. However, while dependency parsing frequently uses transition-based parsers, it is unclear whether transition-based parsing can still provide state-of-the-art results for constituency parsing. Despite promising work by Liu and Zhang in 2017 using an in-order transition-based parser, recent work uses other methods, mainly CKY charts built over LLM encoders. Starting from previous work, we implement self-training and a dynamic oracle to make a language-agnostic transition-based constituency parser. We test on seven languages; using Electra embeddings as the input layer on Penn Treebank, with a self-training dataset built from Wikipedia, our parser achieves a new SOTA F1 of 96.61.

## introduction this work examines whether it is still possible to build a state of the art constituency parser using transition-based parsing. recent years have seen strong progress in both dependency and constituency parsers. models for both of these tasks have progressed from using categorical features in pcfg models or shift-reduce parsers, to word embeddings, to transformers. upgrading to a transformer can have a dramatic effect on the quality of the results @xcite . while transition-based parsing is a widely used technique for dependencies, in recent years it has been less common for constituency parsing. there are still models built on transitions which are quite successful, such as @xcite , which uses an attention mechanism to predict how and where to attach the next word of a sentence. however, most state of the art models use a derivative of the cky algorithm @xcite . for example, the current state of the art uses bidirectional attention as part of the encoder for a chart parser @xcite . in this work, we revisit older transition schemes and show that they can be used to build a state-ofthe-art model. we improve on existing dynamic oracle methods @xcite , allowing the model to better learn the state space after an error. furthermore, we explore self-training using a method inspired by active learning @xcite , and present a simple way to ensemble transition models with the same structure. we present state of the art results on 6 languages. using electra embeddings with the penn treebank, our parser achieves a new sota f1 of 96.61. all models and code described are publicly released in the (anonymous) software package. ## experimental setup we experiment on penn treebank 3 for english @xcite and the chinese treebank 5.1 for simplified chinese @xcite . we also report results for german, indonesian, italian, japanese, and vietnamese @xcite @xcite @xcite . see see appendix b for the hyperparameters used when training the model, and appendix c for a more complete description of the datasets used. scores are reported as averages of 5 models. ## model improvements improvements to base model. the parser builds from a base of the lstm in-order transition-based parser of @xcite , which itself builds from @xcite . several specific improvements to this model improved scores slightly. the first is that the original lstm between the word embeddings and the classification layers was a unidirectional lstm but using a bi-lstm instead slightly improves performance. as we use self-training to boost performance, rather than reranking, we do not need an autoregressive parser. another minor improvement is to follow @xcite in building the encoding of a new constituent by using max over the embeddings of the children. this is simpler and more effective than the bi-lstm labeled with the subtree used in @xcite and @xcite . pretrained embeddings. in @xcite , the standard at the time was to use pretrained word embeddings such as those from word2vec @xcite , glove @xcite , or fasttext @xcite . later work incorporated character models @xcite . the well-known berkeley neural parser uses xlnet @xcite . we find that for english, electra-large @xcite works best of the currently available pretrained word embeddings. following best practices, we finetuned the embeddings and transformers for the parsing task. for release as foss, we found that peft, from @xcite , reduces download size for a single model and vram when used in an ensemble without sacrificing model quality. attention layer. multiple past works on cky constituency parsing add a specialized attention layer after the transformer, such as partitioned attention @xcite , label attention (mrini et al., 2020), and bi-directional attention @xcite . here, we combine ideas from bi-directional attention with streaming llm @xcite . bi-directional attention computes attention layers twice, alternatively masking forwards and backwards directions. streaming llm uses a windowed attention layer with attention sinks to improve nlg. we find that a single layer of windowed attention in both the forwards and backwards directions improves scores on lower resource constituency datasets, with insignificant effects on to explore how the resource size affects these scores, we reran the experiment with a randomly selected 5000 sentences for each language ( ensemble. we investigated the gains that are possible by ensembling multiple parsing models (at the cost of more computation). for chart parsing, a common technique is to use a high-accuracy language model to rerank many outputs of a single parser model, as in @xcite . for transition-based parsing, there is a simpler mechanism for combining the results of multiple models. instead of choosing a transition by taking the maximum scoring legal transition from one model, we first add the logits from several models, then use the transition with the maximum sum. it is necessary to build models with different results in order to successfully ensemble them. we explore three methods of achieving the necessary differentiation: (i) initializing each model's layers with a different random seed (varied seed), (ii) using different numbers of layers from the transformer for the input embedding (mixed layers), or (iii) randomly reweighting the training trees while keeping the structure the same, ensuring training proceeds differently (random reweighting). each method produced useful gains versus a single model, without much or consistent differentiation between them; see oracle. the standard oracle training for transition parsers is teacher forcing: backpropagate the errors from the prediction of a transition, apply the correct transition, and repeat. the limitation of this approach is, at runtime, the model will naturally make errors, resulting in a state which does not directly correspond to any training state. an improvement is to update the remaining transitions to build the best result still possible after an error. this technique is a dynamic oracle, introduced in @xcite for dependency parsers. this technique works for constituency parsing as well, as shown in the bottomup parser of @xcite or the discontinuous parser of @xcite . fernández-gonzález and gómez-rodríguez (2018) further explored this concept for top-down and in-order parsers. after each training error, the oracle used the first possible transition which minimized subsequent errors in the tree. however, this is a potentially ambiguous solution, as some transition sequences have multiple fixes which result in the same number of errors. an example of an ambiguous transition is the subtree in figure 1 from @xcite . in both the in-order and top-down schemes, the correct transition is to close the adjp after the word "rolled". if the close is missed, the dynamic oracle can correct the sequence equally well by closing after "sheet" or after "steel". we ran an experiment using multiple different types of ambiguous top-down dynamic oracle repairs (see appendix e). when run over multiple languages, we found that not using a dynamic oracle when the optimal repair is ambiguous works better than choosing either randomly or deterministically; see to produce silver parses for a given language, we use two ensembles (see 3), one of top-down parsers and the other of in-order parsers, and parse wikipedia for that language. by only keeping the trees where the two ensembles agree, although not guaranteed to filter only correct parses, we build a dataset of silver trees with higher accuracy than either ensemble can produce by itself. such a silver dataset does not use the idea of difficulty from the active learning work, though. we can extend this by noting that the individual parsers in those ensembles do not always agree, with less agreement occurring on "harder" parses. for each silver parse, we tally how many of those individual parsers return that exact parse. using only the parses with the least agreement in an ensemble produces a silver dataset which is both accurate and difficult, improving the overall results. as shown in figure 2 , using trees with full agreement produced little effect, and the dataset with zero or one agreeing parsers were too small to use by themselves. 12 ## results in this work sets a new sota score for a single model parser for ptb when using self-training, while performing close to sota for ctb. it also outperforms previously established best results for german, indonesian, and vietnamese. ## conclusion in this paper, we have shown the continuing viability of neural transition-based constituency parsing, once the basic technique is carefully combined with state-of-the-art neural text encoders, other successful parser techniques, and active learning.