Tweaking UD Annotations to Investigate the Placement of Determiners, Quantifiers and Numerals in the Noun Phrase (2022)

Abstract: We describe a methodology to extract with finer accuracy word order patterns from texts automatically annotated with Universal Dependency (UD) trained parsers. We use the methodology to quantify the word order entropy of determiners, quantifiers and numerals in ten Indo-European languages, using UD-parsed texts from a parallel corpus of prosaic texts. Our results suggest that the combinations of different UD annotation layers, such as UD Relations, Universal Parts of Speech and lemma, and the introduction of language-specific lists of closed-category lemmata has the two-fold effect of improving the quality of analysis and unveiling hidden areas of variability in word order patterns.

## introduction most of the work on word order variation using universal dependencies (ud: de @xcite is based on curated dependency treebanks, with only a few works using dependency corpora derived from raw texts. although the accuracy rate of nlp systems trained on ud models is reportedly very high @xcite @xcite , a certain level of noise i.e., erroneous annotations is in fact present when working with automatically annotated texts @xcite ; furthermore, different layers of ud annotations such as universal parts of speech (upos) and ud relations are not always used consistently across languages, often resulting in the cross-linguistic comparison of different categories. we discuss a methodology to tweak the ud annotations in order to achieve a better representation of word order entropy; the methodology is exemplified on three categories that are particularly difficult to analyze with automatic methods and from a cross-linguistic perspective. determiners, quantifiers and numerals are often treated in descriptive grammars as heterogeneous categories; the lexical category of determiners includes articles and demonstratives, while the category of quantifiers often includes elements of other closed categories, such as pronouns and gradation markers, and sometimes members of open categories, such as adjectives and adverbs; finally, numerals are often not restricted to cardinal, ordinal and distributive numbers, but overlap with quantifiers. this heterogeneity is reflected by the ud implementation of these categories, both at the relation and the upos annotation layer. numerals are treated as a separate category and represented at the syntactic level by the nummod ud relation and at word category level by the num upos; by contrast, the ud framework conflates articles, demonstratives and quantifiers into one upos tag (det) and into one ud relation (det), resulting in the 'determiners & quantifiers' macro-category. at the language-specific level several individual pos tags and ud relation subtypes are used; for instance, in slavic languages quantifiers get two specific subtypes, det:numgov and det:numposs, and morpho-syntactic features of numerals can be specified using additional ud relation subtypes. our methodology combines two layers of ud annotations, upos and ud relations, with manuallycompiled and language-specific lists of lemmata. we test the methodology against a parallel corpus of fiction texts and their translations in 10 indo-european languages; given their particular genre, these texts are quite challenging for parsers that are mostly trained on non-fiction data such as wiki and news. following previous studies, we employ here shannon's entropy as a metric for word order variation. ## related work since its inception in 2015 @xcite , ud has been widely used in corpus-based studies on word order variability. however, as earlier mentioned, corpora used in previous studies are "dependency corpora of the hamledt 2.0 and universal dependencies 1.00" @xcite , "the universal dependencies treebank version 2.2" @xcite , "a selection of 55 treebanks from universal dependencies v2.4" @xcite , "surface-syntactic universal dependencies (sud) [treebanks]" @xcite and the "universal dependencies project, release 2.1" @xcite . ud treebanks can be considered de-facto gold standards, as large parts of them are manually compiled or at least semiautomatically checked for wrong annotations, allowing scholars to work with high quality of data. however, as ud treebanks wildly vary across languages with respect to size and text genres @xcite , results from most of the previous works are biased against these factors. exceptions are represented by works using the lisca parse assessment algorithm @xcite , whose models have been trained on ud-parsed wikipedia corpora @xcite and tested on the so-called 'reference corpora', which consist "of a monolingual corpus of texts from the news and wikipedia domains [...] morpho-syntactically annotated and dependency parsed by the udpipe pipeline trained on the universal dependency treebanks, version 2.2" @xcite ; automatically annotated texts are also partially employed in @xcite , who uses eleven ud-parsed corpora from the leipzig corpora collection for one of her case-studies on word order entropy. finally, talamo and verkerk (to appear) is a study on word order variation in the nominal phrase and is entirely based on parallel texts that are parsed by the udpipe pipeline trained on ud treebanks v.~2.5. to the best of our knowledge, @xcite and talamo and verkerk (to appear) are the only studies on word order variation combining ud relations with other annotation layers; although her methodology is not fully disclosed, @xcite applies the upos annotation layer to the head in her first case study, where word order variability is taken from a syntactic perspective, and the wordform annotation layer to the dependent in her second case-study, where word order variability is investigated with respect to the lexically specific level; talamo and verkerk (to appear) take a step further and operationalize this methodology by introducing several combinations of ud relation and upos annotation layer to restrict either the head, the dependent or both, and introducing language-specific list of lemmata to match modifiers at the lexical level. ## results as shown in the left panel of figure 1 , entropy values captured by the second combination, rel+upos, are significantly lower than values captured by the rel combination. this is particularly clear for numerals, which display a substantial drop in entropy in half of the languages; furthermore, in some languages the entropy of numerals is reduced to near-zero values (german: .02, french and irish: .04, polish: .06). as for determiners & quantifiers, the introduction of the upos layer is overall less significant; a significant reduction of entropy is observed only for three languages @xcite . this is partly due to the low value of entropy already captured by the rel combination, but it also reflects the biunivocal relation between the det upos tag det and the ud relation and its subtypes; by contrast, the nummod ud relation and its subtypes are not in biunivocal relation with the num upos tag, since, as already mentioned above, this ud relation is often used with quantifiers as well. the third combination, which adds language-specific lists of lemmata to the upos and ud relation layers and is plotted in the right panel of figure 1 , allows us to zoom in on the entropy of determiners & quantifiers, disentangling the category into articles, demonstratives and quantifiers; although the entropy of determiners & quantifiers is extremely low for all languages of the sample, two languages have moderate values of entropy for demonstratives or quantifiers. more specifically, irish has the highest value of entropy for quantifiers (.71), while greek the highest value for demonstratives (.59). according to @xcite , the position of quantifiers in irish is lexically determined; most irish quantifiers precede the noun, while few follow it; postnominal quantifiers include the high-frequency lexeme uilig 'all', which explains the high entropy of irish quantifiers. as for greek, the high value of entropy for demonstratives can be accounted on a pragmatic and semantic basis, as postnominal demonstratives have an emphatic reading (lascaratou, 1998, 164). furthermore, low-to-moderate values of entropy are observed for quantifiers in portuguese (.30) and spanish (.10) and for demonstratives in portuguese (.15). as for numerals, we use language-specific lists of quantifiers as negative matches against the lemma field; this approach is however of little use, as entropy values captured by the third combination are the same of the second combination. thanks to the introduction of language-specific list of lemmata, the third combination is suitable for closed categories such as articles, demonstratives and quantifiers, while the second combination is already effective for capturing open categories such as numerals. ## conclusion we have discussed a methodology to extract with better accuracy word order patterns from conll-u files obtained from the automatic parsing of raw texts; the methodology, which exploits different ud annotation layers and language-specific list of lemmata, is exemplified on the heterogenous lexical categories of determiners and numerals, whose word order patterns are analyzed in a parallel corpus of 10 indo-european languages. we have shown that the methodology is able to correct some of the errors introduced by the automatic parsing and inconsistent use of upos tags and ud relations, thus improving the quality of the analysis, as shown with the category of numerals. furthermore, the methodology sheds light on areas of variability, which were previously hidden by the ud lumping of articles, demonstratives and quantifiers into a unitary category. given the very high frequency of articles, whose variability is close to zero, this unitary category displays very low values of entropy across languages; once this unitary category is split into its three components, some languages show moderate-to-high levels of entropy with respect to demonstratives (greek) and quantifiers (irish and portuguese).