A Modest P areto Optimisation Analysis of Dependency Parsers in 2021 (2021)

Abstract: We evaluate three leading dependency parser systems from different paradigms on a small yet diverse subset of languages in terms of their accuracy-efficiency Pareto front. As we are interested in efficiency, we evaluate core parsers without pretrained language models (as these are typically huge networks and would constitute most of the compute time) or other augmentations that can be transversally applied to any of them. Biaffine parsing emerges as a well-balanced default choice, with sequence-labelling parsing being preferable if inference speed (but not training energy cost) is the priority.

## introduction the inefficiency of modern nlp systems has recently come under scrutiny, especially regarding their large energy consumption @xcite . this hasn't started a revolution, but there is some nlp work where efficiency is considered. @xcite studied different settings for neural machine translation systems, evaluating not only accuracy but also certain costs such as inference time, training time, and model size. @xcite analysed the fine-tuning and inference time for pretrained lms, and estimated the cost of pretraining. @xcite presented a pareto optimisation analysis for pos taggers, considering accuracy and model size. in parsing in particular, @xcite evaluated dependency parsing as sequence labelling specifically to increase inference efficiency and also undertook a pareto optimisation analysis. others used model compression via distillation to increase inference speed of neural parsers with a mixed bag of results @xcite . @xcite also took into consideration the training energy costs of distilling models, which highlighted the high energy cost of this technique. we present a pareto optimisation analysis on modern dependency parsing systems. we cover three systems which are broadly representative of current approaches. we analyse their efficiency with respect to inference speed and also their training cost, measured in energy consumption. contribution: a simple, modest analysis on the merits of different parser systems that cover three current paradigms. our goal is not to provide surprising results, but a realistic snapshot of the current state of affairs of a representative sample of modern parsing systems on linguistically diverse data. this analysis runs the systems in a consistent way with respect to software, hardware, and network settings. we also offer a brief overview of self-reported performance on ptb for systems that have a published speed. we add to this measurements for a subset of these systems which we ran locally for a more consistent comparison, i.e. something of a reproducibility effort. disclaimer we make a practical comparison for practitioners, so we focus on publicly available systems on typical hardware that doesn't require a huge budget. we are not making general claims that technique x is always more efficient than technique y in the abstract or that this will hold in any hardware. also, the extent to which an implementation has been engineered will impact performance, so we have referenced the original repositories used. foot_0 ## ptb performance for historical reasons, it is common practice for parsers to report performance results on the english penn treebank (ptb) @xcite . while such results at best provide a partial picture on a single language, they are by far the most comprehensive source of results provided in the literature under a consistent context (at least in terms of data and splits, although not hardware), so they are useful to see high-level trends and as a starting point to choose parsers for our experiment. in we report results for uuparser of smith et al. based on this, we opted to use the basic biaffine parser to represent graph-based parsers, the pointer-lr network as the representative of transition-based algorithms, foot_1 and the sequencelabelling parser to represent sl systems. they all have the added benefit of working under the same software and having code available. note that, as we make emphasis on efficiency, we focus on reasonably bare-bones versions of the parsers. the impact of pretrained language models, or other augmentations that are transversal to the parsing system, is outside the scope of this paper. ## pareto optimisation analysis here we detail the parsing systems, the data we used, and how model structures were altered. ## conclusion we have presented a simple pareto optimisation analysis for a representative sample of modern dependency parsers. we evaluated efficiency in two ways. we evaluated the trade-off between accuracy and parsing speed and the trade-off between accuracy and training energy consumption. the biaffine and seqlab occupied the speed pareto front with the former being slower and more accurate and the latter being faster and less accurate. we didn't observe any real trade-off with regards to training energy and performance, but it was clear that seqlab is not particularly efficient in this regard. typically training energy varied based on how long a model took to converge, with l2r being somewhat sensitive to the different treebanks. overall, for most scenarios, biaffine emerged as a well-balanced practical solution. for the sake of candour, we offer a brief discussion of the limitations of this analysis in appendix 4. we use the biaffine implementation described below and edit it to be a simple sequence-labelling system, i.e. an embedding layer, followed by a number of bilstm layers, and mlps one for predicting the bracket tags and one for predicting the edge labels. we use the same hyperparameters as used for the biaffine parser. deep biaffine (biaffine) is a graph-based parser that creates two representations of each token from the hidden representations from bil-stms, hypothesised to be a representation of each token as dependents and as heads @xcite . 9 an affine transformation is applied to the head representation and then this and the dependent one are then combined via a second affine transformation (hence biaffine) to give a matrix of scores, which gives a probability distribution for each node representing the probability any other node is that node's head. a well-formed tree is then enforced using the chu-liu/edmonds' algorithm @xcite . the edge labels are then predicted based on the predicted edges. we use the standard hyperparameters for this system except where we match them to better correspond to the l2r parser and then only alter the hyperparameters associated with the bilstms. sub-families and which represent different syntactic systems covering analytic, fusional, and agglutinative languages and all are written in different scripts. we offer a brief description of the treebanks used and some of the salient features of their respective languages. the treebanks were chosen to represent varying syntactic features, but also because of their high quality from being either manually annotated or manually corrected. we also chose relatively large treebanks. the statistics for each treebank are shown in ud hindi-hdtb (hindi) is a ud treebank for hindi based on manually annotated news data @xcite . hindi is a lightly fusional language with some degree of verbal inflection and noun declension but also makes extensive use of postpositions @xcite . it is a split-ergative language meaning in certain cases it uses a nominative-accusative structure but in others it uses an ablative-ergative syntax where the subject of an intransitive verb behaves like the object of a transitive one @xcite . it also exhibits tripartite behaviour in certain clauses, where the subject of intransitive verbs, the object of transitive verbs, and the subject of transitive verbs all have different case markings @xcite . it is a sov language, but it has a fairly free word order @xcite . it is indo-iranian and is written in the devanagari script. ud polish-pdb (polish) is a ud treebank manually annotated on fiction, non-fiction, and news data @xcite . polish is a highly fusional language with a high degree of verbal inflection @xcite and 7 case-markings @xcite . it is a null-subject language @xcite with a nominal svo order but has relatively free word order @xcite . like most slavic languages it doesn't make use of articles @xcite but it does have a complex system of numeral and quantifiers that result in agreement mistmatches @xcite . it is a balto-slavic language written in the latin script. ud korean kaist (korean) is a large treebank generated from a constituency treebank which was semi-automatically annotated with manual corrections based on academic, fiction, and news data @xcite . korean is a strongly suffixing agglutinative language @xcite . this results in a large number of cases and a high degree of verbal inflection @xcite 2000). it is technically a sov ordered languae but it has a highly flexible word order @xcite . korean also uses honorifics and speech levels, the former encoding the social relationship between the speaker and the referents in a discussion and the latter the speaker and the person/people being spoken to @xcite . it is a koreanic language written in the hangul script. chinese penn treebank (chinese) is large manually annotated treebank for mandarin based on news data @xcite . it is an analytic, isolating language with a svo dominant word order and is a pro-drop language @xcite . chinese has no grammatical tense markers so relies on context or temporal expressions, but aspect is expressed via the use of particles @xcite . classifiers and measure words must be used when a noun is preceded by a number, a demonstrative pronoun, or certain quantifiers which are particles that appear between these qualifiers and their respective nouns @xcite . chinese is said to be a verb stacking language, where more than one verb or verb phrases are stacked together in the same clause, but there is some disagreement if the way verbs are combined actually constitutes verb stacking @xcite . it is a sino-tibetan language written in simplified hanzi. we re-split the data because the standard split has tiny development and test sets. the resulting sizes are shown