Scaling Graph-Based Dependency Parsing with Arc Vectorization and Attention-Based Refinement (2025)

Abstract: We propose a novel architecture for graph-based dependency parsing that explicitly constructs vectors, from which both arcs and labels are scored. Our method addresses key limitations of the standard two-pipeline approach by unifying arc scoring and labeling into a single network, reducing scalability issues caused by the information bottleneck and lack of parameter sharing. Additionally, our architecture overcomes limited arc interactions with transformer layers to efficiently simulate higher-order dependencies. Experiments on PTB and UD show that our model outperforms state-of-the-art parsers in both accuracy and efficiency.

## introduction recent graph-based dependency parsers have adopted a standard architecture @xcite extended by @xcite . these models consist of two pipelines: one pipeline scores arcs while the other scores their labels. each pipeline uses independent models to generate specialized head and dependent representations from word embeddings, followed by a biaffine scoring model. we investigate the scalability of this widelyused architecture. our motivation stems from the observation that not all model architectures scale efficiently with increased parameters. for example, transformer-based language models exhibit predictable scaling laws, where performance consistently improves with more parameters @xcite . in contrast, other architectures, e.g. cnns, require careful scaling across multiple dimensions @xcite . similar observations have been made in computer vision @xcite . our empirical results show that simply increasing the number of parameters in the stan-dard parsing model does not improve performance. we hypothesize that the core issue lies in the indirect representation of arcs. the model encodes the entire space of possible arcs through word vectors and biaffine scoring, which limits its ability to handle increased complexity. furthermore, using two scoring networks restricts information flow between arc selection and labeling tasks. we propose a novel architecture foot_0 that explicitly constructs vector representations for each arc. by unifying arc scoring and labeling tasks within a single network, our approach allows more parameter sharing and enhances scalability. finally, we add transformer layers over a selection of arc representations to promote interactions, inspired by higher-order models. the selection is performed by a differential filtering mechanism. this design captures dependencies between arcs while maintaining computational and memory efficiency. ## model we review the standard biaffine parser (figure 1 , left) and then highlight the key differences of our arc-centric approach (figure 1 , right). prior to parsing, from an input sentence x 0 x 1 . . . x n , where x 0 is the dummy root and ∀1 ≤ @xmath0, x i corresponds to the i th token of the sentence, models start by computing contextual embeddings e 0 , e 1 , . . . , e n . this can be implemented in various ways, e.g. with averaged layers from pretrained dynamic word embeddings. these contextual embeddings are further specialized for head and modifier roles using two feed-forward (ffn) transformations. this results in two sets of word representations, h 0 , h 1 , . . . , h n for heads and m 1 , . . . , m n for modifiers. figure 1: illustration of both models. left: standard model with 2 (resp. 3) pipelines for loc (resp. crf2o) with shared word embeddings. right: our proposal with a single pipeline and optionally p transformers. ## related work our model, assigning vectors to arcs, i.e. the objects to be scored, draws inspiration from the autoregressive neural approach to parsing @xcite , as well as from span-based parsers such as @xcite and arc-hybrid parsing in @xcite . recently @xcite proposed arc vectorization for semantic higher-order dependency parsing based on gnns. refining initial arc representations has also been explored @xcite . our model with transformers bears a resemblance to earlier work on forest reranking for parsing @xcite , as we use transformers to promote or demote arcs before scoring and parsing, and to @xcite where the parse forest is exploited to recompute vectors for words, as opposed to our work where we recompute arc vectors. attention is widely utilized in parsing @xcite , possibly with ad-hoc constraints on attention @xcite . representing spans has been shown to be beneficial for nlp @xcite @xcite while in @xcite transformers have also been used to enhance span representations. our method uses standard softmax attention with a differentiable filter as opposed to rigid constrained masking @xcite model #param ( 10 and other forms of attention @xcite @xcite . our model is part of the literature on generalizing transformers to relational graph-structured data @xcite @xcite . ## conclusion we presented a change in the main graph-based dependency parsing architecture, where arcs have their own vector representation, from which scores are computed. our model improves parsing metrics and achieves state-of-the-art results on ptb and 11 ud corpora. we also demonstrated that transformer-based refinement simulates higherorder interactions and enhances parameter scalability. our model can be extended to many other tasks in nlp, such as constituent parsing or relation extraction.