Extraction of Contrastive Rules from Syntactic Treebanks: A Case Study in R omance Languages (2025)

Abstract: In this paper, we develop a data-driven contrastive framework to extract common and distinctive linguistic descriptions from syntactic treebanks. The extracted contrastive rules are defined by a statistically significant difference in precision and classified as common and distinctive rules across the set of treebanks. We illustrate our method by working on object word order using Universal Dependencies (UD) treebanks in 6 Romance languages: Brazilian Portuguese, Catalan, French, Italian, Romanian and Spanish. We discuss the limitations faced due to inconsistent annotation and the feasibility of conducting contrasting studies using the UD collection.

## introduction cross-lingual corpus-based studies normally focus on finding common and distinctive structural features or tendencies among languages, language families, or typological balanced samples. word order tendencies and their correlation with other language formal properties are an example of typological high-level descriptions. however, one might be also interested in comparing fine-grained patterns that explain the variation or the similarity between the compared corpora. this is a common goal in translation, second language teaching, textual genre research, and more generally, in corpusbased contrastive linguistics. comparing languages becomes more challenging the closer the languages are to each other. for example, syntactic objects vary considerably among romance languages, even though they also exhibit some shared properties. nominal objects often follow their verb, and personal pronominal objects tend to be in preverbal position. both word order rules are common and dominant (in terms of frequency) in all romance languages. however, personal pronouns are enclitics of infinitives (or gerunds where they exist) in languages such as spanish, catalan or italian, among others, while this is not possible in french. and even if the exact syntactic configuration exists in all languages, the relative frequency may vary. in addition, finegrained differences within a language family are not always shared by the same group of languages. as mentioned, spanish and french do not have the same word order between infinitive verbs and personal pronominal objects, but they do when the verb is an imperative (see figure 1 ). merecia coisa melhor deserve thing.noun better obj amod lo consiguió 3sg.pron.clitic get obj inutile guardar -li useless to look.inf 3sg.pron.clitic amod obj mieux vaut les oublier better worth 3pl.pron.clitic to forget.inf amod csubj obj panoramele le-am cercetat views.obj 3pl.pron.clitic 1p.aux research aux expl obj figure 1: different nominal and pronominal object word orders in portuguese, spanish, italian, french and romanian. from this perspective, corpus-based contrastive analysis should be able to identify absolute differences across a set of languages, such as unique linguistic features, and be flexible enough to capture significant differences in frequency for finegrained phenomena. moreover, a contrastive approach should focus not only on distinctive patterns, but also on common ones. this approach allows a more detailed analysis of languages and corpora, focusing on variation or similarity in a specific lin-guistic feature rather than on language profiles. to address this issue, we develop a data-driven framework to extract fine-grained common and distinctive linguistic descriptions from syntactic treebanks. the extracted contrastive rules are defined by a statistically significant difference in precision and classified as common and distinctive rules across the set of treebanks. for each distinctive rule, we regroup languages having the same behavior. to test our method, we analyse the word order of objects in six romance languages using universal dependencies (ud) treebanks. object word order, especially object clitic order, varies considerably across romance languages @xcite . the present study is limited to this specific phenomenon. further studies are left for future exploration. our approach follows and contributes to the core principles of most recent descriptive grammar extraction works. we seek descriptive but systematic descriptions of linguistic phenomena @xcite , where extracted rules are overlapping rules, more or less fine-grained, and associated to quantitative data @xcite . the rules should be few and easy to interpret. the contributions of this study can be summarised as follows: ## related works corpus-based approaches to cross-linguistic analysis are widespread in both typological and contrastive linguistics, although in recent years they have become much more prominent in the latter. in either case, multilingual corpora have offered the possibility to capture and compare quantitative and gradient properties on a larger scale @xcite . ## task definition our main goal is to identify common and distinctive rules given a set of treebanks, focusing on object word order. following @xcite formalisation, a quantitative rule posits a predictive relationship within the data, where the presence of a pattern p , identified within a starting sample s, increases the likelihood, by an α extent, of a phenomenon of interest q. for example, to extract object word order rules for all objects in the sample, we look for linguistic patterns that favour the right or the left position with respect to its governor: given this formalisation, we consider a common grammatical rule to be evenly distributed across all languages. a distinctive rule, on the other hand, is unevenly distributed, though it may be shared equally by a subset of languages. if a corpus-based rule is quantitative in nature and captures gradient phenomena, a contrastive rule additionally posits whether the distribution of the predictive factor p is uniformly distributed across subsamples, in our case across languages. to explore word order, we extract contrastive rules for the following four questions: (1) word order between two nodes connected by a dependency, (2) object word order, (3) pronominal object word order, and (4) nominal object word order. analyzing object word order and its subtypes in two steps might be considered unnecessary, as the rules emerge when examining general object word order when the regularization parameter of the linear model is low enough. however, extracting rules for each type of object allows us to examine them in a more precise subspace. furthermore, it is reasonable to explore pronouns in more detail as they are selected as good predictors of object word order. overall, this multistep process yields fewer, more significant rules on average. before discussing the methods in detail, we first present the data, its preprocessing, and the sampling strategies employed. ## data this study focuses on romance languages for three main reasons. first, they are well represented in the ud treebank collection, providing us with a robust dataset. second, we are experts and have native proficiency or possess a comprehensive understanding in all of the chosen languages. finally, the use of these languages as a test case is advantageous due to their closeness. we hypothesize that the successful detection of differences and similarities between closely related languages will serve as an indication of the applicability of our method. when multiple treebanks were available in the ud collection for a single language, the largest one in terms of tokens was selected, except in cases where annotation quality or methodology warranted a different choice. in ## methodology our method can be divided into three separate steps. first, we extract and rank the most overall salient patterns for a given linguistic phenomenon across the set of treebanks using a linear classifier. secondly, in order to identify common and distinctive patterns, we assess if the distributions of selected patterns are statistically different from a proportionally expected distribution. finally, for each distinctive pattern we cluster languages to find those that share the same behaviour. ## results we present the raw results without postselection, even though some rules may be redundant. for the object order excluding the catalan treebank (refer to section 6.1 for the reason), we extracted 69 potential grammar rules or tendencies. of these, 39 are common, 14 are distinctive, 4 have lowfrequency occurrences, and 12 are present only in one treebank. for an overview of shared and distinctive rules after the clustering process refer to appendix b for all languages. 2 we evaluate the sparse linear models with the selected features on the treebank test sets, and they generalise well (refer to appendix c). however, such an evaluation provides only limited insight into the extracted rules, as we are not interested in classification scores but in the selected features. since it is not feasible to qualitatively evaluate all the extracted rules, we explore a few relevant rules to illustrate how to interpret them and what the limits are. ## takeaways we present a comprehensive framework to extract contrastive grammar rules and tendencies from syntactic treebanks. it allows us to induce a concise set of grammar rules that reflect statistical differences between closely related languages. a more exploratory and less conservative approach is possible by adjusting a few hyperparameters. indirectly, the method can be used to find annotation inconsistencies across treebanks. experiments also show the limitations of doing automatic grammar extraction and linguistic analysis with universal collections. for this reason, we encourage ud contributors maintaining related language treebanks to work together to harmonise annotation choices.