A Computational Method for Analyzing Syntactic Profiles: The Case of the ELEXIS - WSD Parallel Sense-Annotated Corpus (2025)

Abstract: In the paper, we present an approach to comparing corpora annotated with dependency relations. The method relies on the compilation of syntactic profiles – numeric vectors representing the relative frequencies of different syntactic (sub)trees extracted automatically with the STARK 3.0 open-access dependency tree extraction tool. We perform the extraction on the ELEXIS-WSD Parallel Sense-Annotated Corpus, which has recently been published as version 1.2 with UD dependency relation annotations for 10 European languages. The corpus provides an additional resource for contrastive studies in quantitative syntax. In addition to presenting the corpus and conducting some proof-of-concept analyses, we discuss several other potential uses and improvements to the proposed approach.

## introduction the proliferation of corpus resources annotated with dependency relations in the last decade (such as universal dependencies treebanks; de @xcite has facilitated automatic syntactic analyses with different computational approaches. however, the field of quantitative syntax analysis is arguably still discovering its full potential, and methods that have been ubiquitous in other (sub)fields of computational linguistics are still to be implemented in quantitative syntax studies. the same is true for language resources, with new corpora being developed every year but not included in syntactic studies. the growing interest of the research community in quantitative syntax studies is emphasized by studies focusing on the benefits of quantitative methods (e.g. @xcite , as a counterweight to the prevalent methods of obtaining a judgment of the acceptability of a sentence pair by a handful of participants @xcite . in addition, data extraction for quantitative analyses has been facilitated by recently developed tools specialized for syntactic features @xcite @xcite . the goal of this paper is to make a contribution to the growing toolbox of quantitative syntax methods by (a) presenting a new approach to comparing syntactically annotated corpora with the use of syntactic profiles (numerical vectors of quantitative syntactic features; see section 4), and (b) introducing the elexis-wsd parallel sense-annotated corpus 1.2 (see section 3), a new multilayered and multilingual parallel corpus that can be used for syntactic analyses. the paper is structured as follows: we first provide a brief overview of related work in analyses and tools for syntactically annotated (parallel) corpora (section 2). we then describe the latest version of the elexis-wsd parallel sense-annotated corpus (section 3) and the method for extracting syntactic profiles from its subcorpora as well as individual sentences (section 4). we analyze the corpus-level and sentence-level syntactic profiles (section 5) with statistical tests to determine the most statistically significant differences in distributions of syntactic structures across different languages. in section 6, we focus on the analysis of individual syntactic structures. we conclude the paper (section 7) with several suggestions for future improvements to the method. ## related work many studies in quantitative syntax so far have focused on a restricted set of specific syntactic phenomena (see e.g. van craenenbroeck et al., 2019 for a study of word order in verb clusters in 186 dutch dialects; @xcite for an analysis of differences between regular transitive and experiencer-object verbs in german; or @xcite for an analysis of the properties of rare constructions such as it-clefts and topicalization in english) or tests of pre-determined language universals @xcite . instead of focusing on a specific syntactic phenomenon, our approach is designed in more bottom-up manner (see section 4). the study most similar to our approach was conducted by @xcite , who extracted syntactic profiles from the universal dependencies corpora by focusing on co-occurrences of words and syntactic relations, then crosscomparing the most frequent pairs to obtain similarity/correlation scores between languages. however, the method only provided results on the level of individual languages and their subcorpora, and the syntactic profiles used were limited to a limited set of the most frequent tuples. we build on this approach and focus not only on subcorpora, but individual sentences. in addition, we do not deconstruct syntactic (sub)trees into tuples of relations and focus on a much larger set of complete syntactic (sub)trees as features extracted from the elexis-wsd parallel sense-annotated corpus (see section 3). ## corpus the elexis-wsd parallel sense-annotated corpus @xcite is a dataset that in its current version (1.2; @xcite consists of subcorpora containing the same 2,024 sentences in 10 european languages: bulgarian, danish, english, spanish, estonian, hungarian, italian, portuguese, and slovene. an example of a sentence and some of its parallel equivalents is shown in the corpus was primarily designed within the elexis project foot_0 as a word-sense disambiguation dataset in which the content words (verbs, nouns, adjectives, and adverbs) in each subcorpus are annotated with their corresponding senses from an accompanying sense inventory (a collection of lexemes and their sense divisions with definitions). the sentences were extracted from wikimatrix @xcite , a collection of parallel sentences from wikipedia, and selected according to several mostly semantic criteria (e.g., the number of semantically ambiguous words). missing translations into other languages were automatically translated and manually validated by native speakers. the final versions were tokenized, lemmatized and morphosyntactically tagged using udpipe @xcite . 2 these annotation layers were also manually validated, and the corpus is available in the conll-u format under a creative commons by-sa 4.0 license. within the context of the unidive cost action (universality, diversity and idiosyncrasy in language technology; @xcite , which at the time of writing this paper is still underway, the elexis-wsd corpus is being extended with new languages on the one hand, and new annotation layers on the other. this includes universal dependencies parsing annotations @xcite , which were absent in previous versions. for the slovene and estonian subcorpora, the annotations have already been manually validated. for the other languages, the dependency relations were added using the udpipe 2.15 models. 3 the performance of the models on gold tokenization is shown in ## extraction of syntactic profiles we prepare the data for statistical analysis by extracting syntactic profiles of individual subcorpora as well as individual sentences from elexis-wsd. we define a syntactic profile of a unit as a numerical vector of relative frequencies of various syntactic features extracted from the unit. in this paper, we focus on features representing the relative frequencies of different syntactic trees and subtrees in different units. we extract the frequencies using stark 3.0 @xcite , an open-access dependency-tree extraction tool available under the apache 2.0 license. stark takes a conll-u file with syntactic annotations as input and, based on several customizable parameters, outputs a frequency list of syntactic structures (trees) represented with the simple dep_search query language. 5 an example is shown in figure 1 . depending on the settings, the frequency list contains absolute and relative frequencies of syntactic structures (normalized by the number of tokens in the extracted unit per million). instead of feeding entire subcorpora to stark, we first split the files into individual sentences and performed the extraction foot_5 on each sentence individually. from each sentence, we extracted complete syntactic (sub)trees encompassing the head and all its (in)direct dependants, as well as the order of the dependants. a sample of extracted (sub)trees is shown in after extracting syntactic (sub)trees from all sentences, we removed the structures occurring less than 3 times throughout the entire corpus and ended up with a set of 2,582 distinct (sub)trees. these were used as features for the numerical vectors representing the syntactic profile of each sentence. for each sentence s, its syntactic profile is compiled by concatenating the relative frequencies (within s) of each tree t from the set of n distinct (sub)trees: @xmath0, f r (t 2 ), f r (t 3 ), ..., f r (t n )]. in our case, this generated a 20,240 x 2,582 matrix that was used for statistical comparisons (see section 5 ). an additional 10 x 2,582 matrix of syntactic profiles was compiled for individual subcorpora, consisting of the means of relative frequencies of each syntactic tree. ## statistical analysis of selected features to determine in which specific languages the differences in frequencies of a given syntactic tree are statistically significant, we performed a series of pair-wise mann-whitney u tests @xcite with bonferroni correction (at @xmath1.001). 13 the effect sizes were measured with the rank-biserial correlation coefficient (r) @xcite ). 14 ## conclusion and future work we have presented the latest version of the elexis-wsd parallel corpus, which also contains ud dependency relations and can be used as an additional resource for studies in quantitative syntax alongside the many existing ud treebanks, including parallel ud treebanks. 15 we have also presented a method to observe the differences in the distribution of syntactic (sub)trees between corpora by using the stark 3.0 dependency-tree extraction tool. while we showcased the method on a parallel corpus, it can also be used to compare e.g. two corpora in the same language (e.g. a spoken and a written corpus; a learner vs. a general corpus) to determine the most salient differences in syntactic structures. in addition to contrastive syntactic comparisons, the method could also provide a basis for several other uses. first, by generating quantified syntactic profiles of sentences in a corpus, groups of syntactically similar sentences can be extracted by exporting clusters with high cosine similarity scores compared to a reference sentence. second, the method could be used to compare whether (and to what degree) a sampled corpus is syntactically representative of the whole. on the other hand, the method can help extract syntactically diverse samples to ensure as many syntactic structures are included as possible. however, there are potential challenges with the scalability of the method. in this paper, we have limited the extraction of syntactic profiles to only complete syntactic (sub)trees. extracting all parts of syntactic (sub)trees would help provide a more accurate profile, but would also be much more computationally expensive. during our tests, extracting partial and full syntactic profiles resulted in approximately 2kb vs. 10mb of data per sentence, respectively. more tests are required to compare which (additional) features are best at representing the syntactic characteristics of the remaining links not extracted when focusing solely on complete syntactic (sub)trees. in the future, we intend to publish new versions of the elexis-wsd corpus within the unidive cost action. on the one hand, the corpus will be extended with subcorpora for new languages, and the dependency relation annotations for more of the existing corpora will be manually validated. the corpora will eventually also contain several other annotations that can be cross-compared with syntax, such as sense-, named entity-, and multiword expression annotations. once the corpus is fully manually annotated, the parallel alignment of sentences will allow for an even more direct comparison of syntactic structures. exporting co-occurrences of syntactic (sub)trees between equivalent sentences from different languages will enable us to observe the most frequently or typically co-occurring structures (by calculating association measures such as pointwise mutual information @xcite ). the next step should also involve extending the method of extracting syntactic profiles by including e.g. combinations of syntactic structures and additional quantitative features, such as direction, frequency, and depth of individual dependency relations (or combinations thereof), which have been shown to be effective at representing certain aspects of syntactic complexity (see e.g. @xcite and can be easily extracted with recently developed tools such as comparatree @xcite and @xcite . these options will be explored in future studies, in which the method will also be tested on other corpora.