On the Frailty of Universal POS Tags for Neural UD Parsers (2020)

Abstract: We present an analysis on the effect UPOS accuracy has on parsing performance. Results suggest that leveraging UPOS tags as fea-tures for neural parsers requires a prohibitively high tagging accuracy and that the use of gold tags offers a non-linear increase in performance, suggesting some sort of exceptionality. We also investigate what aspects of predicted UPOS tags impact parsing accuracy the most, highlighting some potentially meaningful linguistic facets of the problem.

## introduction part-of-speech (pos) tags and dependency parsing have formed a long-standing union in nlp. but equally long-standing has been the question of its efficacy. prior to the prevalence of deep learning in nlp, they were shown to be useful for syntactic disambiguation in certain contexts @xcite @xcite . however, for neural network implementations, especially those which utilise character embeddings, pos tags have been shown to be much less useful @xcite . others have found that pos tags can still have a positive impact when using character representations given that the accuracy of the predicted pos tags used is sufficiently high @xcite . @xcite undertook a systematic study of the impact of features for universal dependency (ud) parsing and found that using universal pos (upos) tags does still offer a marginal improvement for their transition-based neural parser. the use of fine-grained pos tags still seems to garner noticeable improvements @xcite . latterly, pos tags have been commonly utilised implicitly for neural network parsers in multilearning frameworks where they can be leveraged without the cost of error-propagation @xcite @xcite @xcite . beyond multi-learning systems, @xcite introduced dependency parsing as sequence labelling by encoding dependencies using relative positions of upos tags, thus explicitly requiring them at runtime. we follow the work of @xcite and evaluate the interplay of word embeddings, character embeddings, and pos tags as features for two modern parsers, one a graph-based parser, biaffine, and the other a transition-based parser, uuparser @xcite . similar to @xcite , we focus on the contribution of pos tags but evaluate upos tags. ## experimental details we ran three experiments to measure the impact pos foot_0 tagging accuracy has on parsing performance when using pos tags as features. experiment 1 considered the pos tagging accuracy as a controlled variable, set by training taggers as described below and then using the output of these taggers as features for parsers. experiment 2 was similar, except the size of character embeddings were also changed. experiment 3 was an extension to test the impact of taggers in an optimal setting where they achieve very high accuracies. data we use the same subset of ud v2.4 treebanks @xcite as @xcite : ancient greek perseus, chinese gsd, english ewt, finnish tdt, hebrew htb, russian gsd, tamil ttb, uyghur udt, and wolof wtb. we used fasttext word embeddings for each language except for ancient greek and wolof @xcite . for ancient greek we use embeddings from @xcite and for wolof those from @xcite . when necessary, we reduced the dimensions to 100 using the algorithm of @xcite . ## results and analysis experiment 1 the results of experiment 1 are shown in figure 1 , where the average difference in attachment scores between the baseline parsers (without pos tags) and those with differing pos tag accuracies are shown. we show the differences in attachment scores rather than the absolute values, as averaging over treebanks obscures differences. there is an unsurprising relation between parsing score and tagging performance when training with gold pos tags. what is less expected is how little of an impact is observed when using predicted tags during training, with an almost consistent performance with respect to pos tag accuracy. the gold training trend for the graph-based parser suggests that it is less sensitive to pos tag accuracy than the transition-based parser. this is likely due to the transition-based parser being able to leverage pos tags more, so that it will see more of an impact when tagging accuracy is low. this is somewhat corroborated by the larger positive difference over the baseline when using gold tags at prediction time for uuparser compared to the increase seen for biaffine. another notable phenomenon is that the results for parsing texts annotated with gold pos (rightmost point in each plot) outperform what could be expected from extrapolating the general trends. this raises the question as to whether this is due to smooth nonlinear accuracy increases in the rightmost part of the curves (where we couldn't obtain taggers) or to a sudden jump at the very end in a hockey-stick shape, indicating an exceptionality of gold pos tags and inadequacy of even very accurate but imperfect pos tags (which is relevant under the assumption that tagging accuracy can be pushed further with future model and/or training data improvements). answering this question was the motivation for experiment 3. almost exclusively, using predicted pos tags does not outperform the parser trained without any pos tags. curiously, the only parsers that are marginally better are those trained with predicted pos tags from the least accurate pos taggers. figures 6 7 8 9 in the appendix show these results for each treebank separately, and almost all treebanks follow the general trend seen for both parsers. the only exception is tamil ttb for uuparser, which benefits from pos tags both when training with gold and predicted tags. tamil ttb is the smallest treebank, and it has the additional difficulty for parsing and tagging of being an agglutinative language, so possibly this combination of factors lends itself well to leveraging pos tags even in less than optimal circumstances. tamil is also the lowest performing language with respect to pos tagging and parsing accuracy, but compared to uyghur and ancient greek (the next two lowest performing languages) it outperforms both when using gold tags. in fact, tamil has the biggest difference when using gold tags over the baseline than any other language, suggesting that they might be particularly useful when there is a heightened probability of ambiguity coupled with a dearth of data. ## conclusion we have evaluated the impact pos tag accuracy has on parsing performance for leading graph-and transition-based parsers across a diverse range of ud treebanks, highlighting the stark difference between using predicted pos tags and gold pos tags at runtime. we observed a non-linear increase in performance when using gold tags, suggesting they are somehow exceptional, i.e., precisely the tag patterns that not even the most accurate taggers can correctly predict (the last 2-3 percentage points towards 100% accuracy) seem to be the most important for parsing. this could be due to the parsers implicitly learning pos tag information, in such a way that the taggers learn nothing new to contribute or not enough to avoid a loss in performance due to the errors disrupting what the parsers have learnt. our analysis also shows that practitioners should evaluate the efficacy of using predicted tags for a given system or language. we have also analysed what aspects of erroneous tagging predictions have the greatest impact and correlation to parsing performance. we observed some global trends, like the importance of cconj, but also languagespecific issues which highlight the need to evaluate the usefulness of pos tags per language. the results also suggest that using a subset of pos tags might be effective.