A Falta de Pan, Buenas Son Tortas: The Efficacy of Predicted UPOS Tags for Low Resource UD Parsing (2021)

Abstract: We evaluate the efficacy of predicted UPOS tags as input features for dependency parsers in lower resource settings to evaluate how treebank size affects the impact tagging accuracy has on parsing performance. We do this for real low resource universal dependency treebanks, artificially low resource data with varying treebank sizes, and for very small treebanks with varying amounts of augmented data. We find that predicted UPOS tags are somewhat helpful for low resource treebanks, especially when fewer fully-annotated trees are available. We also find that this positive impact diminishes as the amount of data increases.

## introduction low resource parsing is a long-standing problem in nlp and many techniques have been introduced to tackle it @xcite @xcite @xcite . for an extensive review and comparison of techniques see @xcite . here we focus on the utility of part-of-speech (pos) tags as features for low resource dependency parsers. pos tags are a common feature for dependency parsers. @xcite highlighted the unrealistic performance of low resource parsers when using gold pos tags in a simulated low resource setting. the performance difference was stark despite using fairly accurate taggers, which is not a reasonable assumption for low resource languages. tagging performance in low resource settings is still very weak even when utilising cross-lingual techniques and other forms of weak supervision @xcite . even when more annotated data is available, it isn't clear how useful pos tags are for neural dependency parsers, especially when utilising character embeddings @xcite . work investigating the utility of pos tags typically observe a small increase in performance or no impact when used as features for neural dependency parsers. @xcite found that universal pos (upos) tags offer a marginal improvement for their transition based parser for multi-lingual universal dependency (ud) parsing. @xcite also observed an improvement in parsing performance for graph-based parsers when the predicted upos tags came from sufficiently accurate taggers. zhang et al. (2020) only found pos tags to be useful for english and chinese when utilising them as an auxiliary task in a multi-task system. anderson and gómez-rodríguez (2020) found that a prohibitively high accuracy was needed to utilise predicted upos tags for both graph-and transitionbased parsers for ud parsing. they also obtained results that suggested smaller treebanks might be able to directly utilise less accurate upos tags. we evaluate this further by analysing the impact of tagging accuracy on ud parsing in low resource contexts, with regards to the amount of data available to train taggers and parsers. ## methodology we performed three experiments. the first is an evaluation of predicted tags as features for biaffine parsers for real low resource treebanks. it also includes parsers trained with upos tagging as an auxiliary task similar to the experiments in zhang et al. (2020). the second experiment evaluates the impact of different tagging accuracies on different dataset sizes using artificial low resource treebanks by sampling from high resource treebanks. the last experiment utilises a data augmentation technique to investigate the efficacy of predicted upos tags for very small treebanks (∼20 sentences) when augmented with varying amounts of data. low resource data we take all ud v2.6 treebanks @xcite with less than 750 sentences in both its training dataset and development dataset. we cluster these treebanks into two groups, very low with less than 50 sentences and low with less than 750. the very low resource treebanks consist of buryat bdt (bxr), kazakh ktb (kk), kurmanji mg (kmr), livvi kkpp (olo), and upper sorbian ufal (hsb). the low resource set is made up of belarusian hse (be), galician treegal (gl), lithuanian hse (lt), marathi ufal (mr), old russian rnc (orv), tamil ttb (ta), and welsh ccg (cy). we combined the training and development data (when available) to then split them 80|20. the statistics for the resulting splits are shown in artificial low resource data we use indonesian gsd (id), irish idt (ga), japanese gsd (ja), and wolof wtb (wo) to create artificially low resource treebanks. we take a sample of 100, 232, and 541 sentences from the training and development data. these are then split 80|20 for training and development data. we do this three times for each treebank size so we have multiple samples to verify our results. we use the original test data for analysis. augmented data for the experiment using augmented data we use a subset of the smallest treebanks, namely kazakh, kurmanji, and upper sorbian. we then generate data using the subtree swapping data augmentation technique of @xcite . we generate 10, 25, and 50 trees for each and we then split them 80|20. we do this three times for each number of generated trees. we use the original test data for analysis. ## results and discussion figure 1 shows the average las performance for the parsers trained with the artificial low resource data. when the parsers have sufficient data, using upos tags doesn't offer any improvement in performance. for the parsers trained with 232 samples, there is a slight upward trend when using tags predicted from taggers trained with 541 samples. the improvement increases with respect to upos tag accuracy and exceeds the performance of the parsers trained with no upos tags. the most no- figure 2 : impact of tagging accuracy for varying amounts of data for both taggers and parsers using augmented data (0, 10, 25, and 50 augmented trees) on top of the original gold data. the standard error of upos accuracy is not shown as it is very small (< 0.1% relative error for all bins). horizontal lines show the mean parsing performance for the baseline parsers trained without upos tags (standard error not shown due to too much overlap between augmented data sample sizes). figure 2 shows the impact of predicted upos accuracy when using data generated with subtree swapping augmentation. the first result worth noting is that the augmented data increases performance in this very low resource context. across the board, the best performing parsers using aug-mented data outperform the parsers trained only on gold data by 3-6 points which corroborates the findings in previous work. however, it appears that there is a limit to how much augmented data helps as the performance of the parsers which use 25 and 50 augmented instances is similar. it also appears that this upper limit is even lower for training taggers with the best performance coming when using predicted tags from taggers utilising only 10 augmented samples or none at all. using more invariably hurts performance no matter what accuracy the taggers obtained, as can be seen in the subplots showing the performance for parsers trained with predicted tags from taggers using 25 and 50 augmented samples. also, there is no clear trend showing the impact of upos accuracy in this very low resource context. ## conclusion we have presented results which suggest that lower accuracy taggers can still be beneficial when little data is available for training parsers, but this requires a high ratio of upos annotated data to tree annotated data. experiments using artificial low resource treebanks highlight that this utility diminishes if the number of samples reaches a fairly small amount. we have also shown that very small treebanks can benefit from augmented data and utilise predicted upos tags even when they come from taggers with very low accuracy. our experi-ments haven't considered pretrained multilingual language models (lms) which could potentially offset the small benefits of using pos tags. it would be interesting to develop this analysis further by testing whether the implicit information encoded in these lms are more useful than explicit but potentially erroneous pos tag information. finally, as one reviewer highlighted, the set of pos tags in the ud framework might just not be sufficiently informative in this setting. while this might be true, the greater contributing factor is surely the low accuracy of the taggers.