What quantifying word order freedom can tell us about dependency corpora (2023)

Abstract: Building upon existing work on word order freedom and syntactic annotation, this paper investigates whether we can differentiate between findings that reveal inherent properties of natural languages and their syntax, and features dependent on annotations used in computing the measures. An existing quantifiable and linguistically interpretable measure of word order freedom in language is applied to take a closer look at the robustness of the basic measure (word order entropy) to variations in dependency corpora used in the analysis. Measures are compared at three levels of generality, applied to corpora annotated according to the Universal Dependencies v1 and v2 annotation guidelines, selecting 31 languages for analysis. Preliminary results show that certain measures, such as subject-object relation order freedom, are sensitive to slight changes in annotation guidelines, while simpler measures are more robust, highlighting aspects of these metrics that should be taken into consideration when using dependency corpora for linguistic analysis and generalisation.

## introduction with the breadth of existing resources and research into developing dependency treebanks, cross-linguistic research has expanded to largescale comparative work, formalising and computing quantifiable properties of natural language. the use of morphological and syntactic annotations, to name a few, has enabled typological research to move from type-based-treating languages as individual data points with a categorical valueto token-based-making generalisations and comparative analyses by using corpora to observe linguistic units in language use and express their behaviour using aggregate measures @xcite . in this work, the focus is on word order freedom, a property of natural language syntax, extensively covered in previous work that makes use of dependency treebanks @xcite @xcite . the main point of interest is word order freedom expressed by the measure of word order entropy (woe), as defined by @xcite . the cited work expands on methodological issues, aiming to find a balance between linguistic interpretability, robustness independent of corpus size, and cross-lingual applicability. the defined measure also enables quantitative verification of hypotheses on the relation between case marking and word order freedom @xcite ; word order freedom and patterns across languages with respect to head direction; and the positions of subject and object in the main clause @xcite . however, in applying this measure to different corpus domains and sources, several issues arise and require further addressing-mainly, when expressing word order freedom with measures based on dependency annotations, does the measure reveal more about the language itself, or the annotation used as a layer between the raw text and the computable data? further, and in line with the question raised in the original study, is this measure consistent across corpus sizes, and different text samples? these questions are investigated through a replication of the methodology on the same set of languages covered by the original study (with minor exceptions). the aim is to compare two generations of universal dependency annotation styles @xcite , using the latest releases of universal dependencies v1 @xcite and v2 @xcite . the analysis is focused on three levels-(1) comparing scores obtained over the full corpus with multiple random samples, to verify whether the measure is robust to sample size; section 2 gives a brief summary of the key methodological points of @xcite (further also referred to as "the original study"). section 3 highlights the specifics of the experimental setup. results and findings are presented in section 4, and section 5 concludes the paper. the degree of word order freedom is quantified through the unordered dependency graph of a sentence, using conditional entropy: to avoid the issue of sparsity, entropy is computed only on local subtrees-consisting of a head and its immediate dependents. to avoid issues with misrepresented variability in certain word order phenomena, this means preferring annotation styles with content-head dependency. this requirement is satisfied in universal dependencies annotations. @xcite introduce three measures of word order entropy (woe): relation order entropy (roe) conditioning on the unordered local subtree structure (c being the set of dependency relations and part-of-speech (pos) tags of constituents), the dependent variable x is the linear order of relation types expressed in the local subtree. ## experimental setup this study follows the methodology of @xcite as closely as possible, with three exceptions: omitting three languages from the original study due to data limitations, adjusting entropy estimation due to technical limitations, and performing computations over multiple random subcorpora samples to perform a more robust evaluation of the effects of sampling and data sparsity. the experimental setup is further detailed in subsequent paragraphs. ## analysis the aim of the analysis is threefold: (1) comparing the scores obtained on the full corpora against the random samples, to evaluate the effects of sampling and data sparsity, as well as comparing the random samples to estimate variance; (2) comparing udv1 scores to udv2 scores, to evaluate the effect of annotation; and (3) comparing the results of the original study to the rankings obtained on udv1 and udv2. ## conclusion this paper has taken a deeper look into an existing methodology of quantifying word order freedom in dependency corpora. the study attempted to determine whether this methodology and measure allows for draw reliable conclusions about word order freedom, or whether it depends to a relevant extent on the underlying dependency annotationsboth in terms of annotation guidelines, and in the quality of annotation depending on annotator experience and consistency. the study identified diffi-culties in finding a definition of measure that would be robust enough to avoid noise and misrepresentation, yet fine-grained enough to give meaningful linguistic insight. the analysis shows that changes in annotation styles can alter the results of estimates and change the comparative presentation of word order freedom across languages. furthermore, it has shown that the observed measures may be susceptible to differences between samples, and that random sampling as defined by this methodology is selectively unreliable, depending on measure complexity. in conclusion, there is merit in cross-testing treebank-based metrics on different versions of treebanks, considering changes in annotation guidelines or even annotator teams, as well as on random subsamples of treebanks. future work may also investigate the optimal size for these samples-currently fixed on an arbitrary count. building on existing work on universal dependencies, the question that next arises concerns what potential levels of complexity using enhanced universal dependencies would introduce to this method of quantifying word order freedom. future work may also focus on similar comparisons between manually annotated (gold-standard) and automatically generated dependency annotations, as well as possible differences between domains (e.g., newswire vs. literary text; written vs. spoken corpora), as well as across different annotation styles.