Comparing learnability of two dependency schemes: ‘semantic’ ( UD ) and ‘syntactic’ ( SUD ) (2021)

Abstract: This paper contributes to the thread of research on the learnability of different dependency annotation schemes: one (‘semantic’) favouring content words as heads of dependency relations and the other (‘syntactic’) favouring syntactic heads. Several studies have lent support to the idea that choosing syntactic criteria for assigning heads in dependency trees improves the performance of dependency parsers. This may be explained by postulating that syntactic approaches are generally more learnable. In this study, we test this hypothesis by comparing the performance of five parsing systems (both transition- and graph-based) on a selection of 21 treebanks, each in a ‘semantic’ variant, represented by standard UD (Universal Dependencies), and a ‘syntactic’ variant, represented by SUD (Surface-syntactic Universal Dependencies): unlike previously reported experiments, which considered learnability of ‘semantic’ and ‘syntactic’ annotations of particular constructions in vitro, the experiments reported here consider whole annotation schemes in vivo. Additionally, we compare these annotation schemes using a range of quantitative syntactic properties, which may also reflect their learnability. The results of the experiments show that SUD tends to be more learnable than UD, but the advantage of one or the other scheme depends on the parser and the corpus in question.

## introduction and background this paper compares the learnability of two approaches to dependency annotation. one, represented by universal dependencies (ud; http: //universaldependencies.org/ ; @xcite , favours content words over function words as dependency heads, as this increases crosslinguistic uniformity of the resulting scheme; here we will call this approach 'semantic'. 1 another, represented by surface-syntactic universal dependencies (sud; https://surfacesyntacticud . github.io; gerdes et al., 2018, 2019), uses purely syntactic criteria for determining headedness; hence the moniker 'syntactic'. the sud scheme was designed as minimally different from -'near-isomorphic to' -ud, and many ud treebanks have been converted to sud, so differences in learnability between the two approaches should be relatively easy to assess and interpret. as is clear from figure 1 , which juxtaposes the ud basic tree (at the top) and the sud tree (at the bottom), sud generally adopts the principle that function words such as auxiliaries (e.g., do), subordinating conjunctions (until), copula ('re), and prepositions (with) are heads of relevant constructions. on the other hand, sud representation of coordination is similar to that of ud, but where all non-initial conjuncts are attached to the head of the first one in ud, each conjunct is attached to the head of the previous conjunct in sud; when there are just two conjuncts, annotations are the same. previous results suggest that the syntactic scheme should be more learnable. for example, @xcite compared six constructions, four of which are coded differently in ud do n't use a joke until you 're completely comfortable with it root advmod aux obj det advcl mark nsubj cop advmod obl case root mod comp:aux mod comp:obj det comp:obj subj mod comp:pred udep comp:obj @xcite and sud: foot_1 preposition-noun (e.g., of rome)a class which also includes complementiser-clause constructions (e.g., after you go), to-infinitival (e.g., to eat), modal-verb (e.g., can come), and coordination. the experiments involved five different parsers (representing both transition-based and graph-based methodologies) and two different learnability measures (including one based on attachment scores). the results of these experiments favour sud-like representations in all four cases. in the case of constructions involving a preposition or a complementiser, having them as heads -as in sud, but unlike in ud -results in extremely strong ('unanimous') learnability improvements. the effect is weaker in the case of verb groups containing a modal and still weaker in the case of infinitivals introduced by to, but in both cases having the main lexical verb as the dependent -as in sud, but unlike in ud -gives generally better results. a similar range of constructions is inspected in @xcite . for each kind of construction, 3 different variants of conversion from semantic to syntactic headedness are considered, depending on how many of the dependents of the semantic head are moved to the syntactic head. the best variant gives significant improvements in the learnability of the syntactic scheme in the case of preposition-noun (but not complementiser-clause), auxiliary-verb (rather than the more general modal-verb, considered in @xcite and -and this is were the improvement was most clear -in the case of copula-predicate constructions. other papers that report better learn-ability of a more syntactic scheme converted automatically from a more semantic scheme include: @xcite (auxiliary-verb constructions in arabic, czech, dutch and slovene, small improvement observed in the case of the transition based maltparser, but not with the graphbased mstparser), @xcite (adposition-noun constructions in 30 languages), @xcite (various constructions involving function and content words in 19 typologically varied languages), and @xcite (15 languages, although the extent of the improvements varied considerably, and in the exceptional case of turkish regress was observed for all three parsers used in the experiments). 3 on the other hand, de lhoneux and nivre (2016) report on an experiment involving 24 languages, in which the original ud representation of verb groups (modal-verb constructions) turns out to be more learnable by maltparser than the converted representation with main verbs acting as dependents of modal verbs. in a similar vein, @xcite report that languages and particular constructions vary drastically in the extent to which the syntactic or the semantic approach to headedness is more or less learnable by their own transition-based parser. however, out of the seven constructions they consider (similar to those considered in @xcite , four differentiate ud and sud, and out of these four, two (copula-predicate and case-noun, but not mark-verb) are more learnable in the syntactic encoding in the majority of languages -copula-predicate con-structions by a wide margin (75% of languages). unfortunately, the paper does not present the full results of the experiments, so it is not clear whether there is a correlation between, say, language family and learnability of particular representations of particular constructions. the current paper is methodologically closest to @xcite and @xcite : it reports results of experiments performed on multiple corpora of typologically diverse languages, and it compares the learnability of different annotation schemes applied to the same underlying texts. however, the novelty of the current paper lies in comparing the learnability of two comprehensive linguistically-informed annotation schemes rather than a real scheme and an artificial scheme differing from it in the headedness of a single or a small number of constructions. that is, unlike previous experiments reported in the literature cited above, the experiments reported here were performed in vivo rather than in vitro. this matters, as any realistic annotation schema which employs a more 'syntactic' approach to headedness than ud will also differ from ud in the repertoire and distribution of dependency labels, and will also take into account the intrinsic linguistic interaction between various constructions. the co-existence of large and high-quality treebanks in their ud and sud variants presents the unique opportunity to compare the learnability of 'semantic' and 'syntactic' annotation schemes in a realistic setup. ## conclusions while some initial work suggested clear relation between learnability of dependency parsing and the 'semantic' or 'syntactic' approach to headedness, with 'syntactic' annotations usually reported as more learnable, the experiments often had a very limited scope: they concerned one language, or just one or a very small number of constructions, or just one or two parsers. more extensive experiments, performed on a number of languages, taking into account a handful of constructions and a few parsers, such as those reported in @xcite , showed that this relation between learnability and different approaches to headedness, even though imperfect, in general favours syntactic-like approaches, but also suggested a more stable correlation between learnability and other corpus characteristics (such as ade). all these experiments were performed in vitro, on the basis of dependency corpora with one or just a few constructions reanalysed for the purpose of the experiments. in contrast, the current paper presents the results of comparing two full-fledged annotation schemes in vivo: the 'semantic' ud and the 'syntactic' sud. the experiments confirm that it cannot be claimed that more 'syntactic' approaches to annotation uniformly lead to better learnability: this depends on particular languages (rather than on language families) and on particular parsers. however, corpora annotated according to the sud scheme tend to be more learnable, especially, by the graph-based parsers utilised in the experiments. as to correlations between corpus characteristics and learnability, the experiments show a clear correlation between label entropy and parsers' performance (especially, in terms of uas), which suggests that sud may take advantage of its smaller set of labels or lower order variability of labels, or both. also, correlation was found between the difference in the percentage of non-projective trees between both schemes and learnability in the case of udpipe (again, in terms of uas). this may suggest the inability of this parser to effectively deal with higher degrees of non-projectivity, even though some hyperparameter tuning was implemented to deal with this issue. on the other hand, the results do not confirm the recent hypothesis that learnability of the two kinds of annotations is negatively correlated with arc direction entropy. in the same vein, we have not found statistically significant correlations between parser performance, and average dependency length. future work should seek to dissociate the effect of more learnable dependency labels from that of different approaches to headedness; to this end experiments should be performed on corpora with trees typologically just like those in ud and sud, but with label schemes modified so that label entropy is not correlated with learnability. also, it