What Taggers Fail to Learn, Parsers Need the Most (2021)

Abstract: We present an error analysis of neural UPOS taggers to evaluate why using gold tags has such a large positive contribution to parsing performance while using predicted UPOS either harms performance or offers a negligible improvement. We also evaluate what neural dependency parsers implicitly learn about word types and how this relates to the errors taggers make, to explain the minimal impact using predicted tags has on parsers. We then mask UPOS tags based on errors made by taggers to tease away the contribution of UPOS tags that taggers succeed and fail to classify correctly and the impact of tagging errors.

## introduction part-of-speech (pos) tags have commonly been used as input features for dependency parsers. they were especially useful for non-neural implementations @xcite @xcite . however, the efficacy of pos tags for neural network dependency parsers is less apparent especially when utilising character embeddings @xcite . universal pos (upos) tags have still been seen to improve parsing performance but only if the predicted tags come from a sufficiently accurate tagger @xcite . typically using predicted pos tags has offered a nominal increase in performance or has had no impact at all. @xcite undertook a thorough systematic analysis of the interplay of upos tags, character embeddings, and pre-trained word embeddings for multi-lingual universal dependency (ud) parsing and found that tags offer a marginal improvement for their transition based parser. however, @xcite found that the only way to leverage pos tags (both coarse and fine-grained) for english and chinese dependency parsing was to utilise them as an auxiliary task in a multi-task framework. further, @xcite investigated the impact upos tagging accuracy has on graph-based and transition-based parsers and found that a prohibitively high tagging accuracy was needed to utilise predicted upos tags. here we investigate whether dependency parsers inherently learn similar word type information to taggers, and therefore can only benefit from the hard to predict tags that taggers fail to capture. we also investigate what makes them hard to predict. ## methodology we performed two experiments. the first was an attempt to compare what biaffine parsers learn about upos tags by fine-tuning them with tagging information and comparing their errors with those from normally trained upos taggers. the second experiment attempted to evaluate the impact tagging errors have by either masking errors or using the gold standard tags for erroneously predicted tags while masking all other tags. ## results and discussion experiment 1: error crossover clearly more training would likely improve the parsers fine-tuned for tagging, but it would be less clear if that would be extracting information the parser previously learnt or adding more information via mlp weights. for the closed class type tags, again the parser performs similarly to the tagger but obtains a few points less except for det, num, part, and pron with drops for parser scores of 7.9, 15.8, 13.6, and 23.9, respectively. however, of these 4 tags, only pron and det appear in the most common errors and only twice and once, respectively. the most common tag to appear in an error is noun occurring 41 times, but there is less than one point in difference between the tagger's performance and the parser's for noun. of these 41 appearances, 14 cooccur with adj and 15 with propn with a fairly even split of mis-tagging noun as either of these tags or the other way around. so generally noun tokens are fairly easy to tag, but the times where the tagger fails are typically where there is confusion with adj and propn tags. figure 2 shows statistical metrics of the taggers' errors. first we show the proportion of out-of-vocabulary (oov) word forms for all tokens and also the tokens where the tagger makes an error. consistently across all treebanks the oov proportion is considerably higher for tokens erroneously tagged. second we report the mean upos surprisal. for a given upos tag, θ n for token n, the surprisal of that upos tag in a given context, c k is given as: then the mean surprisal, i , over a sample of tokens is given as: the difference between the error sub-sample and the whole treebank is starker for the head-relation surprisal, suggesting that the tagger struggles more when the syntactic structure is uncommon. experiment 2: masked tags we then used gold standard tags but masking the tags that the taggers correctly predicted to test if the erroneous tags are particularly useful. we did this for the normal taggers (m¬e t ) and also for the fine-tuned parsers (m¬e p ). the average increase for both is about 2.5 over the no tag baseline and over 2 points better than using predicted tags. also, the improvement is universal with at least a small increase in performance over using predicted upos tags. interestingly the smaller set from the tagger outperforms the larger set from the parser by 0.15, suggesting that what both the taggers and the parsers fail to capture is more important than the errors unique to the parsers. we then masked the errors from the taggers (m∀e t ) to test if avoiding adding errors would still be beneficial. the performance is almost 2 points better than using the ## conclusion we have presented results which suggest that parsers do learn something of word types and that what taggers fail to learn is needed to augment that knowledge. we have evaluated the nature of typical tagging errors for a diverse subset of ud treebanks and highlighted consistent error types and also what statistical features they have compared to the average measurement across all tokens in a treebank. we have shown that it would be more beneficial to implement taggers to not only predict tags but also decide when to do so, as the errors undermine anything gained from using predicted tags for dependency parsers. note that while we only used one parser system, the original paper (anderson and gómez-rodríguez, 2020) which prompted this work observed similar behaviour with regard to predicted upos tags for both the system used here (graph-based) and a neural transition-based parser, suggesting that the results discussed here might extend to other parsing systems. and while it is true that we have only investigated one pos tagger system, we feel we have been careful in not making egregiously grand claims of the universality of our findings: it is merely one data point to be considered amongst many.