Applying Occam’s Razor to Transformer-Based Dependency Parsing: What Works, What Doesn’t, and What is Really Necessary (2021)

Abstract: The introduction of pre-trained transformer-based contextualized word embeddings has led to considerable improvements in the accuracy of graph-based parsers for frameworks such as Universal Dependencies (UD). However, previous works differ in various dimensions, including their choice of pre-trained language models and whether they use LSTM layers. With the aims of disentangling the effects of these choices and identifying a simple yet widely applicable architecture, we introduce STEPS, a new modular graph-based dependency parser. Using STEPS, we perform a series of analyses on the UD corpora of a diverse set of languages. We find that the choice of pre-trained embeddings has by far the greatest impact on parser performance and identify XLM-R as a robust choice across the languages in our study. Adding LSTM layers provides no benefits when using transformer-based embeddings. A multi-task training setup outputting additional UD features may contort results. Taking these insights together, we propose a simple but widely applicable parser architecture and configuration, achieving new state-of-the-art results (in terms of LAS) for 10 out of 12 diverse languages.

## introduction recent years have seen considerable improvements in the performance of syntactic dependency parsers for frameworks such as universal dependencies (ud; de @xcite . for graph-based parsers, these improvements can in large part be attributed to two developments: (1) the introduction of deep biaffine classifiers @xcite , which now constitute the de-facto standard approach for graph-based dependency parsing, and (2) the rise of pre-trained distributed word representations, particularly transformer-based contextualized embeddings such as bert @xcite or roberta @xcite . both characteristics are present in recent top-performing systems @xcite @xcite @xcite . however, there remain a considerable number of implementation and configuration choices whose impact on parser performance is less well understood. this is evidenced by the many different model configurations (see the aim of this paper is to disentangle the effects of the above factors and determine their impact on parser performance. we appeal to the concept of occam's razor by ways of avoiding architectural elements that do not bring about a testable advantage. with this idea in mind, we introduce steps (the stuttgart transformer-based extensible parsing system), a modular graph-based dependency parser which implements commonly used modules such as biaffine scorers @xcite or lstm layers @xcite (see figure 1 ). using steps, we perform a series of experiments on the ud treebanks of a diverse set of languages. our setup facilitates estimating the impact of the various architectures and configuration decisions in a comparable way. our most important insight is that a relatively simple architecture using biaffine heads on top of fine-tuned xlm-r @xcite leads to the highest parsing accuracy for almost all languages in our study, outperforming prior systems on most languages. our analysis indicates that lstm layers do not lead to benefits. simplifying the architecture even further by using a single scorer for edge and label prediction results in similar performance but on average leads to longer training times. our contributions are as follows: parser pre-trained embed. lstm mtl stanfordnlp word2vec, fasttext yes no udpipe 2.0 word2vec, fasttext yes yes hit-scir fasttext, elmo yes no udify mbert no yes trankit xlm-r no yes to enhanced ud parsing, also resulting in stateof-the-art performance(in terms of elas) for 5 out of 7 evaluated languages. this paper is structured as follows. sec. 2 gives the necessary background on relevant state-of-theart neural graph-based dependency parsers as well as related work on analysing and comparing parsers. sec. 3 describes the architecture and configuration options for our new steps parser, at the same time introducing the various factors studied in our experiments (sec. 4). sec. 5 presents the adaption of our system to enhanced ud. finally, we discuss implications for parser choice and future parser design (sec. 6). ## related work this section provides a brief outline of the use of contextualized word embeddings in syntactic parsers, recently developed graph-based dependency parsers, and related work on dependency parser analysis. ## steps: a modular graph-based dependency parser in this section, we describe our modular dependency parser steps (stuttgart transformer-based extensible parsing system). each subsection focuses on a particular aspect of the parser setup, providing background on its usage and its potential impact on parser performance. ## experiments this section describes our experimental setup and reports the results of our experiments on pre-trained embeddings (sec. 4.3), factorized vs. unfactor- language transformer lm ud treebank arabic arabicbert-large (safaya et al., 2020) padt (smrž et al., 2008) chinese chinese bert (devlin et al., 2019) gsd czech slavic-bert (arkhipov et al., 2019) pdt (bejček et al., 2012) english roberta-large (liu et al., 2019) ewt (silveira et al., 2014) finnish finbert (virtanen et al., 2019) tdt german german bert (github.com/dbmdz/berts) gsd (mcdonald et al., 2013) hindi wikibert-hindi (pyysalo et al., 2020) hdtb (bhat et al.; palmer et al., 2009) italian italian bert-xxl (github.com/dbmdz/berts) isdt japanese wikibert-japanese (pyysalo et al., 2020) gsd korean kr-bert (lee et al., 2020) kaist (chun et al., 2018) latvian wikibert-latvian (pyysalo et al., 2020) lvtb russian rubert @xcite syntagrus @xcite multilingual mbert @xcite ) -multilingual xlm-r @xcite - ## enhanced ud parsing with steps in order to determine whether our conclusions also hold for the related graph parsing task of enhanced ud @xcite , we run an additional batch of experiments on 7 treebanks from the iwpt 2020 shared task @xcite . ## discussion and conclusion in this paper, we have performed a detailed and principled analysis on a variety of decisions arising during dependency parser design. what works? we have identified an architecture based on finetuned xlm-r embeddings and factorized scoring that lead to new state-of-the-art performance for 11 out of 12 diverse language in our study on basic ud parsing, and for 5 out of 7 lanugages for enhanced ud parsing. what doesn't? adding lstm layers on top of the transformer leads to a decrease in accuracy in most cases. we have also shown that multi-task setups predicting upos and ufeats often degrade parsing performance. what is really necessary? for current state-of-the-art ud parsers, we recommend making sure that the pretrained language model covers the intended domain well. in addition, keeping a factorized approach is a good idea for tree parsing, while in graph parsing, a single scorer module may suffice. in this paper, we have addressed a high-to medium-resource scenario, assuming that we know the application language of a parser and thus training a single parser per language. future work may address multilingual approaches such as the training setup used by udify or the recently proposed udapter @xcite , which aims at boosting performance of low-resource languages while keeping performance of high-resource languages high. furthermore, it would be interesting to see if our results about biaffine achitectures also hold for non-syntactic tasks that have recently been framed as dependency parsing tasks, such as named entity recognition @xcite , negation scope detection @xcite or semantic role labeling @xcite . to sum up, in this paper we have applied "occam's razor" to graph-based dependency parsing. we believe that the insights from our study will foster further research on dependency parsing and on framing other tasks as dependency parsing, taking our simplified but robustly performing steps parser as a starting point.