Another Dead End for Morphological Tags? Perturbed Inputs and Parsing (2023)

Abstract: The usefulness of part-of-speech tags for parsing has been heavily questioned due to the success of word-contextualized parsers. Yet, most studies are limited to coarse-grained tags and high quality written content; while we know little about their influence when it comes to models in production that face lexical errors. We expand these setups and design an adversarial attack to verify if the use of morphological information by parsers: (i) contributes to error propagation or (ii) if on the other hand it can play a role to correct mistakes that word-only neural parsers make. The results on 14 diverse UD treebanks show that under such attacks, for transition- and graph-based models their use contributes to degrade the performance even faster, while for the (lower-performing) sequence labeling parsers they are helpful. We also show that if morphological tags were utopically robust against lexical perturbations, they would be able to correct parsing mistakes.

## introduction the use of morphological tags was a core component of dependency parsers to improve performance @xcite . with the rise of neural models, feeding explicit morphological information is a practice that has greatly vanished, with (often) the exception of part-of-speech (pos) tags. in this line, @xcite already found that character-based word vectors helped improving performance over purely word-level models, specially for rich-resource languages, for which the use of morphological information is more relevant @xcite . related, @xcite showed that predicted pos tags still improved the performance of their graph-based parser, even when used together with character-based representations. @xcite and @xcite studied the impact that ignoring pos tag vectors had on the performance of a bilstm transition-based parser @xcite . they conclude that when considering pos tags, word-level, and character-level embedddings, any two of those vectors are enough to maximize a parser performance, i.e., pos tag vectors can be excluded when using both word-level and characterlevel vectors. @xcite showed the utility of pos tags when learned jointly with parsing. recently, @xcite and @xcite have explored the differences between using gold and predicted pos tags, showing that the former are helpful to improve the results, while the latter are often not, with the exception of low-resource languages, where they obtain small but consistent improvements. furthermore, @xcite showed that the efficacy of pos tags in the context of sequence labeling parsing is greatly influenced by the chosen linearization method. however, most of such work has focused on: (i) studying the effect of the universal pos tags @xcite , and (ii) its impact on nonperturbed inputs. yet, nlp models are very sensible and brittle against small attacks, and simple perturbations like misspellings can greatly reduce performance @xcite . this has been shown for tasks such as named-entity recognition, question answering, semantic similarity, and sentiment analysis @xcite . in parallel, defensive strategies have been tested to improve the robustness of nlp systems, e.g., placing a word recognition module before downstream classifiers @xcite , or using spelling checks and adversarial training @xcite . yet, as far as we know, no related work has been done on testing perturbed inputs for parsing and the effect, positive or negative, that using morphological information as explicit signals during inference might have in guiding the parsers. foot_0 ## adversarial framework perturbed inputs occur for several reasons, such as for instance on-purpose adversarial attacks @xcite or, more likely, unintended mistakes made by human writers. in any case, they have an undesirable effect on nlp tools, including parsers. our goal is to test if under such adversarial setups, coarse-and fine-grained morphological tags: (i) could help obtaining more robust and better results in comparison to word-only parsers (going against the current trend of removing any explicit linguistic input from parsers); or (ii) if on the contrary they contribute to degrade parsing performance. below, we describe both how we generate (i, §2.1) linguistically-inspired attacks at characterlevel, and (ii, §2.2) the tested parsers. ## experiments we now describe our experimental setup: data we selected 14 ud treebanks (zeman et al., 2021) that use the latin alphabet and are annotated with universal pos tags (upos), languagespecific pos tags (xpos), and morphological feats (feats). it is a diverse sample that considers different language families and amounts of data, whose details are shown in generating perturbed treebanks for each test set, we create several versions with increasing percentages of perturbed content words (from 0% to 100%, with steps of 10 percent points) to monitor 5 for the models that use morphological information we went for gold tags for training. the potential advantages of training with predicted pos tags vanish here, as the error distribution for pos tags would be different for non-perturbed (during training) versus perturbed inputs (during testing). 6 we exclude experiments with bert-based models for a few reasons: (i) to be homogeneous with previous setups (e.g. smith et al. (2018), @xcite ), (ii) because the chosen parsers already obtain competitive results without the need of these models, and (iii) for a better understanding of the results, since it is hard to interpret the performances of individual languages while not extracting conclusions biased on the language model used, instead of the parsing architecture. how the magnitude of the attacks affects the results. for each targeted word, one of the four proposed perturbations is applied randomly. to control for randomness, each model is tested against 10 perturbed test sets with the same level of perturbation. to check that the scores were similar across runs, we computed the average scores and the standard deviation (most of them exhibiting low values). setup for each parser we trained four models: a word-only (word) baseline where the input is just the concatenation of a pre-trained word vector and a character-level vector, and three extra models that use universal pos tags (word+upos), language-specific pos tags (word+xpos), or feats (word+feats). for parsing evaluation, we use labeled attachment scores (las). for the taggers, we report accuracy. we evaluate the models on two setups regarding the prediction of morphological tags: (i) tags predicted on the same perturbed inputs as the dependency tree, and (ii) tags predicted on non-perturbed inputs. specifically, the aim of setup ii is to simulate the impact of using a tagger that is very robust against lexical perturbations. ## conclusion this paper explored the utility of morphological information to create stronger dependency parsers when these face adversarial attacks at characterlevel. experiments over 14 diverse ud treebanks, with different percentages of perturbed inputs, show that using morphological signals help creating more robust sequence labeling parsers, but contribute to a faster degradation of the performance for transition-and graph-based parsers, in comparison to the corresponding word-only models.