Parsing linearizations appreciate P o S tags - but some are fussy about errors (2022)

Abstract: PoS tags, once taken for granted as a useful resource for syntactic parsing, have become more situational with the popularization of deep learning. Recent work on the impact of PoS tags on graph- and transition-based parsers suggests that they are only useful when tagging accuracy is prohibitively high, or in low-resource scenarios. However, such an analysis is lacking for the emerging sequence labeling parsing paradigm, where it is especially relevant as some models explicitly use PoS tags for encoding and decoding. We undertake a study and uncover some trends. Among them, PoS tags are generally more useful for sequence labeling parsers than for other paradigms, but the impact of their accuracy is highly encoding-dependent, with the PoS-based head-selection encoding being best only when both tagging accuracy and resource availability are high.

## introduction pos tags have long been considered a useful feature for parsers, especially prior to the prevalence of neural networks @xcite @xcite . for neural parsers, it is less clear if they are useful or not. work has shown that when using word and character embeddings, pos tags become much less useful @xcite . however, @xcite found using universal pos (upos) tags to be somewhat helpful, but improvements are typically quite small @xcite . similarly, for multi-task systems, small improvements have been observed for both upos and finer-grained tags @xcite . a limiting factor when using predicted pos tags is the apparent need for very high accuracy from taggers @xcite . this is particularly problematic in a low-resource setting where using gold tags gives unreasonably high performance @xcite and high accuracy taggers are difficult to obtain @xcite . however, some work has suggested that in a low-resource setting even low accuracy taggers can be beneficial for parsing performance, especially when there is more pos tag annotations than dependency tree annotations @xcite . these findings relate to transition-based (tb) and graph-based (gb) parsers, but recently several encodings have been proposed to frame dependency parsing as a sequence labeling task @xcite @xcite , providing an alternative to gb and tb models when efficiency is a priority @xcite . @xcite found that the amount of data required for different encodings varied and that some were impacted by predicted pos tag use more than others. here, we evaluate the impact of pos tagging accuracy on different encodings and also the interplay of this potential relation and the amount of available data (using low-, mid-, high-, and very-highresource treebanks). this is done by artificially controlling the accuracy of pos taggers by using the nature of errors generated by robust taggers. foot_0 ## sequence labeling parsing in dependency parsing as sequence labeling, the goal is to assign a single label of the form (x i , l i ) to every input token w i of a sequence, where x i encodes a subset of the arcs related to w i and l i is the dependency type. below, we review the existing families of linearizations used in this work. head-selection (spoustová and spousta, 2010), where x i encodes the head of w i using an absolute index or a relative offset, that can be based on some word property (usually pos tags, which is also the property we use in this work due to its strong performance in previous work). so for instance, if x @xmath0, x), this would indicate that the head of w i is the nth word to the right of w i with the word property x. some desirable properties of this encoding family are a direct correspondence between words and arcs and the capacity to encode any nonprojective tree. however, a major weakness is its dependency on the chosen property (in our case, pos tags) to decode trees. bracketing-based x i represents the dependency arcs using a string of brackets, with each arc represented by a bracket pair. its main advantage is that it is independent of external features, but regarding projectivity it cannot represent arcs that cross in the same direction. to alleviate this, we use the encoding proposed by @xcite , that adds a second independent plane of brackets (2p b ), inspired by multiplanarity @xcite . transition-based @xcite , where given a sequence of transitions generated by a left-to-right transition-based parser, it splits it in labels based on read transitions (e.g. shift), such that each word receives a label x i with a subset of transition actions. for this work, we consider mappings from a projective algorithm, arc-hybrid (ah tb ; @xcite and a non projective algorithm, covington (c tb ; covington, 2001). ## controlling pos tag accuracy we purposefully change the accuracy of the pos tags in a treebank, effectively treating this accuracy as the independent variable in a controlled experiment and las as the dependent variable, i.e. @xmath1. rather than randomly altering the gold label of pos tags, we alter them based on the actual errors that pos taggers make for a given treebank. this means pos tags that are more likely to be incorrect for a given treebank will be more likely to be altered when changing the overall pos accuracy of that treebank. we refer to this as the error rate for pos tags. the incorrect label is also based on the most likely incorrect label for the pos tag error for that treebank based on the incorrect labeling from the tagger. we refer to this as the error type, e.g. ## experiments we now present the experimental setup to determine how parsing scores evolve for the chosen linearizations when the tagging accuracy degrades. as evaluation metrics, we use labeled (las) and unlabeled attachment scores (uas). ## discussion the obtained results offer some valuable information about how pos tag quality affects performance for different encodings and quantities of data. in most situations using pos tags as features is better than not using them, in contrast with results for other parser architectures as described above. in addition, the less resources, the harder it is for rp h to beat brackets: cutoffs are at 97.5%, 95%, 90% for low-, mid-, and high-resource treebanks, respectively. however, for very high-resource treebanks, the cutoff is back at 95%. compounded with the low tagging accuracy expected in low-resource setups, this highlights that rp h is less suited for them. 2p b , which generally outperforms the other encodings below 90% tagging accuracy, is the best low-resource option. the more resources available, the harder it is for the models using pos tags to outperform base-no-tags, both for bracketing-and transition-based linearizations; i.e. experiments suggest that the benefits provided by the pos tags decline when more training data is available. for brackets, the cutoffs occur @xmath2, 80%, 85% and 90% for the low-, mid-, high-and very highresource set, and for transition encodings, they are @xmath3. for the very-high resource set, cutoff points are at 85% for c tb and 90% for ah tb . ## conclusion we connected the impact that the quality of pos tags and quantity of available data has on several dependency parsing linearizations. we tested this by controlling pos tagging performance on a range of ud treebanks, diverse in terms of both amount of resources and typology. the results showed that for sequence labeling parsing, which prioritizes efficiency, pos tags are still welcome, contrary to more mature parsing paradigms such as transitionbased and graph-based ones. the experiments also showed that parsing linearizations benefit from pos tagging accuracy differently, and in particular linearizations that represent arcs as bracket strings are a better choice for most realistic scenarios. meanwhile,