Error Analysis and the Role of Morphology (2021)

Abstract: We evaluate two common conjectures in error analysis of NLP models: (i) Morphology is predictive of errors; and (ii) the importance of morphology increases with the morphological complexity of a language. We show across four different tasks and up to 57 languages that of these conjectures, somewhat surprisingly, only (i) is true. Using morphological features does improve error prediction across tasks; however, this effect is less pronounced with morphologically complex languages. We speculate this is because morphology is more discriminative in morphologically simple languages. Across all four tasks, case and gender are the morphological features most predictive of error.

## introduction in error analysis, we often blame morphology @xcite , i.e., the productive inflection and derivation of new word forms. morphology has been argued to be a major source of error in syntactic parsing @xcite , semantic parsing @xcite , machine translation @xcite and a range of other tasks, in particular in morphologically complex languages @xcite @xcite . this paper presents a large-scale study showing that morphology is, as commonly conjectured, an important source of error across tasks, but somewhat surprisingly, that morphology is less predictive of errors in morphologically complex languages. english is a morphologically simple language, showing very limited inflection and expressing most concepts through syntactic structure instead; it is also the most-represented language at major natural language processing (nlp) venues and that with the largest amount of language resources available @xcite . this w jego imieniu @xmath0. . . ## background morphology is frequently identified as a source of error during qualitative evaluations of nlp systems. @xcite observe that inflectional variants cause problems for statistical ccg tagging due to training data sparseness, and explicit morphological analysis helps, even for english. for dependency parsing, @xcite identify case syncretism as a source of error propagation in data from czech, german, and hungarian. @xcite give a broader overview of the challenges that rich morphological structure presents for dependency parsing, and şahin and steedman (2018) discuss the importance of morphology in semantic parsing. many observations of the effect of morphology come from evaluating machine translation (mt) systems. @xcite show that morphological errors are common for mt into arabic and russian and strongly affect human quality judgement. for english-romanian mt, @xcite find that tense and verb form on the target side are a common source of error. @xcite find that errors in english-croatian mt are more common for some morphological categories, such as case. in a similar vein, @xcite evaluate morphological competence of mt systems using contrast pairs and show that systems have different strengths and weaknesses for different morphological phenomena. beyond parsing and mt, morphology has also been shown to present a challenge for tasks such as arabic handwriting recognition @xcite or russian anaphora resolution @xcite . most of the studies cited above predate contextual embedding models such as bert @xcite , which are now considered state-of-the-art for many nlp tasks. so far, few studies have explicitly analysed bert with regard to morphology. @xcite analyses morphological content in bert-style models for five languages and finds that "[morphological] ambiguity is negatively correlated with performance on classification, and to a significant degree in many cases", suggesting that morphology is still a significant source of error in these models. we go significantly beyond this work by studying a much larger set of morphological variables, across several architectures and tasks, and across up to 57 languages. ## datasets we collect datasets from shared tasks that (i) publish system outputs along with their gold annotations, (ii) span a variety of languages, and (iii) cover different nlp tasks. based on these criteria, we pick datasets from the following shared tasks: dependencies @xcite , covering semantic role labeling for seven languages. additionally, we use the following dataset for its gold annotations: here, we are not interested in the system outputs from the shared task; instead, we use the gold annotations for the quality estimation, which give us token-level error labels for the underlying machine translation outputs. section 4.2 describes in detail how we assign error labels to these datasets. ## methodology we train a classifier to predict errors made by nlp systems based on morphological features of the input tokens, in order to then analyze which morphological features (if any) are most predictive of such errors. we first describe how we obtain these features (sec. 4.1) and how we classify when an nlp system has made an error (sec. 4.2), then describe the classifier itself (sec. 4.3). ## analysis for each shared task (sec. 3), we ran our classification pipeline (sec. 4) separately for each combination of (i) system submission and (ii) language evaluated on. since random forests are largely interpretable, our analysis focuses on the important features in our learned models. first, though, we look at the overall f 1 -score of the individual classifiers, which we evaluate via stratified 5-fold cross-validation on each data point (sec. 5.1). additionally, to better estimate the importance of morphology, we run our crossvalidation pipeline a second time without the mor-phological features, i.e., only providing the classifiers with the "control features" shown in tab. 1. we refer to these two feature sets as "full" and "control" settings, respectively, and analyze their differences in f 1 -score (sec. 5.2). 8 finally, we analyse the importance of individual morphological features (sec. 5.3). ## conclusion we presented a large-scale error analysis focusing on the role of morphology. our analysis spans a range of morphological variables, four nlp tasks, and up to 57 languages. we confirm the common conjecture that morphological variablesespecially case and gender-are predictive of errors across nlp tasks and languages. somewhat sur- pos fi (•10 -6 ) adv 39.5 intj 38.4 part 26.5 aux 15.0 propn 11.2 cconj 5.2 adp 4.9 sconj 3.4 pron 3.2 sym 3.1 x 2.0 det 0.6 verb -0.3 num -1.7 punct -13.8 adj -14.7 noun -15.9 (a) udp pos fi (•10 -6 ) adv 9.9 aux 6.8 x 4.3 adp 3.7 sconj 2.9 sym 1.7 noun -1.2 pron -1.2 part -2.6 intj -4.3 num -4.3 propn -5.0 verb -5.0 det -6.2 cconj -6.8 adj -8.1 punct -9.3 (b) sem gold: 1 1 sys: 1 1 (a) vmwe example from polish ersetzt die standardglyphen durch die glyphenglyphenglyphen . gold: ok ok ok ok ok bad ok ok ok ok ok bad ok ok ok (b) mt example from german