Analysing and Validating Language Complexity Metrics Across S outh A merican Indigenous Languages (2024)

Abstract: Language complexity is an emerging concept critical for NLP and for quantitative and cognitive approaches to linguistics. In this work, we evaluate the behavior of a set of compression-based language complexity metrics when applied to a large set of native South American languages. Our goal is to validate the desirable properties of such metrics against a more diverse set of languages, guaranteeing the universality of the techniques developed on the basis of this type of theoretical artifact. Our analysis confirmed with statistical confidence most propositions about the metrics studied, affirming their robustness, despite showing less stability than when the same metrics were applied to Indo-European languages. We also observed that the trade-off between morphological and syntactic complexities is strongly related to language phylogeny.

## introduction the development of means for quantifying linguistic properties is essential for cognitive approaches to computational linguistics, becoming simultaneously more challenging and useful as the property of interest is transversal to different languages and, therefore, an important clue for accessing cognitive processes behind human language. this is the case of language complexity. the concept of language complexity, whether of an utterance or of a language as a whole, is instinctive for us. people know how to recognize when a text is written in a difficult or elaborate way and they usually recognize that certain languages are less or more complicated to learn depending on their linguistic background. informally, we can say that: (i) the complexity of an utterance encompasses the quantity and sophistication of linguistic constructs necessary to form and understand the utterance and (ii) the complexity of a language as a whole refers to the quantity and sophistication of communicative strategies available for the formation of such utterances in that language. despite a relative consensus around these intuitions, we lack established formal and quantifiable definitions of language complexity. it is difficult to find a definition that encompasses the heterogeneous range of human language manifestations, both in terms of different languages and of different levels in which meaning can be conveyed within a language. even in light of these challenges, it is crucial to establish rigorous, theoretically and experimentally validated definitions of language complexity. both cognitive and non-cognitive approaches to linguistics can significantly enhance their expressive capacity and theoretical framework. in nlp, complexity measures can be used in automatic text simplifiers, translators, domain-sensitive correctors and completers @xcite , but can also be integrated into the of training machine learning models, to increase performance @xcite . another challenge for the construction of a robust theory for language complexity is that of inclusion: historically, the construction of tools and theories of human language has included indo-european languages, to the detriment of other linguistic manifestations, e.g. american native languages. for a concept that aims to be transversal to different languages and provide universal insights into them and their underlying cognitive processes, as is the case with language complexity, it is necessary to include the broadest possible range of languages in its development and validation. this inclusion is the focus of our work. here, we examine a set of language complexity metrics derived from information theory, proposed in @xcite @xcite , and @xcite . the authors ran several experiments with the proposed metrics, drawing on data from a sub-stantial set of languages, with a predominant focus on those belonging to the indo-european language family. here, we repeat these experiments with data from south american indigenous languages, attempting to ascertain whether the desirable properties of these metrics remain solid when incorporating frequently excluded languages. we seek to verify the robustness of the proposed metrics and include a more diverse set of linguistic manifestations in the construction of a quantifiable theory of human language complexity. our text is structured as follows: in section 2, we present our theoretical background and related works; section 3 outlines our methodology, complexity metrics and its properties, the data used and the experimental pipeline employed; in section 4 we exhibit our results and, in section 5, we present our conclusions. in contrast to the computational challenges of nichols (1998)'s metric, @xcite @xcite proposes a set of compression-based complexity metrics based on information theory. the author compares these metrics to alternatives, extend them to different linguistic tiers, and evaluate them on different parallel corpora. this family of metrics is the main object of study in this paper, with a focus on their behavior when applied to indigenous south american languages, not explored in the original works. @xcite suggests modifications to them, proposing improvements for eliminating potential spurious correlations, and experiment on semi-parallel and non-parallel data. several subsequent works draw directly or indirectly from the notion of compression-based complexity metrics @xcite @xcite @xcite @xcite @xcite , in particular, to quantify the difficulty of acquiring a second language @xcite ). an alternative approach, which characterizes complexity as a function of linguistic features, was explored in @xcite ; @xcite for english and in @xcite for the portuguese language. similar works study language complexity from the perspective of readability, instead of the typological approach adopted here, focusing on text simplification or elaboration @xcite @xcite @xcite @xcite . regarding the study of indigenous languages of south america, several classic works studied and documented the languages explored here, e.g. @xcite ; @xcite ; @xcite inter alia. the investigation into the computational complexity of indigenous languages remains much less explored and our work is completely original, to the best of our knowledge. @xcite , @xcite , oh and pellegrino (2023), @xcite , @xcite and @xcite are the works that closely resemble the work we present here, assessing various complexity metrics or associated measures on language sets that incorporate south american languages. nevertheless, these studies diverge from ours in terms of goals, methodology, and/or the quantity of included indigenous languages, typically covering a significantly smaller number compared to our assessment. ## methodology this paper aims to evaluate a collection of compression-based language complexity metrics m introduced in previous works @xcite @xcite . the evaluation is conducted on a dataset d encoded in a broad range of south american indigenous languages l. the objective is to determine the validity of the theoretical and experimental propositions p regarding m, as observed in the aforementioned studies, when m is applied to the languages in l. in this section, we present the methodology adopted to achieve this goal. in subsection 3.1, we define the set m of language complexity metrics evaluated; in subsection 3.2, we present the set l of south american languages tested and the data d used to represent them; in subsection 3.4 we present the propositions p about m, whose validity we wish to verify when m are applied to l through d; in subsection 3.5, we outline the experimental processing pipeline employed for conducting this verification. subsection 3.3 presents a brief interlude on the writing systems used to encode the languages in l. ## results and discussion using the complexity metrics computed from the described experimental pipeline, we conducted analyses to empirically validate each proposition outlined in section 3.4, obtaining the following results: ## conclusions and future steps the majority of propositions about the studied complexity metrics (h 2 , h 3 , o 1 , o 2 , and o 4 ) were successfully validated in our vast dataset of south american indigenous languages. these results confirm the robustness of such metrics and indicate the universality of the techniques proposed by @xcite to compute the different forms of linguistic complexity. as we used a greater variety of languages, we were also able to document that the trade-off between morphological and syntactic complexities strongly relates with language phylogeny. although we confirmed most of our propositions, we obtained inconclusive results for h 1 and o 3 , and even for the confirmed hypothesis, we found them to be weaker in south american languages compared to the sets of predominantly indo-european languages used in the original experiments. this highlights the need to validate and adjust these metrics for a wider range of human languages, a task we have initiated here. in future research, we aim to investigate the inconclusive propositions, particularly focusing on the impact of orthographic complexity on overall linguistic complexity, extending our results to a greater set of writing systems. our findings add to those of @xcite and @xcite , expanding the set of languages on which these family of language complexity metrics have been validated.