Vulgaris: Analysis of a Corpus for Middle-Age Varieties of I talian Language (2020)

Abstract: Italian is a Romance language that has its roots in Vulgar Latin. The birth of the modern Italian started in Tuscany around the 14th century, and it is mainly attributed to the works of Dante Alighieri, Francesco Petrarca and Giovanni Boccaccio, who are among the most acclaimed authors of the medieval age in Tuscany. However, Italy has been characterized by a high variety of dialects, which are often loosely related to each other, due to the past fragmentation of the territory. Italian has absorbed influences from many of these dialects, as also from other languages due to dominion of portions of the country by other nations, such as Spain and France. In this work we present Vulgaris, a project aimed at studying a corpus of Italian textual resources from authors of different regions, ranging in a time period between 1200 and 1600. Each composition is associated to its author, and authors are also grouped in families, i.e. sharing similar stylistic/chronological characteristics. Hence, the dataset is not only a valuable resource for studying the diachronic evolution of Italian and the differences between its dialects, but it is also useful to investigate stylistic aspects between single authors. We provide a detailed statistical analysis of the data, and a corpus-driven study in dialectology and diachronic varieties.

## introduction understanding the evolution of a language is a challenging problem. when a language originated? what are the influences coming from dialects and other languages? these are crucial questions that the study of language evolution aims to face. natural language processing techniques are powerful tools that can support researchers in the analysis of dialects and diachronic language varieties @xcite . there exists several lines of research that approach the problem of defining distances between languages or varieties. linguistic phylogenetics @xcite aim at determining a rooted tree to describe the evolution of a group of languages or varieties. trees are built based on the so called lexicostatistics technique, that takes into account words with common origin to determine a taxonomic organization of the languages. language distance approaches instead rely on the distributional hypothesis of words and require cross-lingual corpora. similarity is based on word co-occurrences @xcite , or using perplexity-based methods @xcite @xcite . perplexity is estimated from language models, typically n-grams lms of characters, trained on one corpus and evaluated on another variety. differently from previous work, we consider neural language models (nlms) @xcite , that are more robust estimators well known for their generalization capabilities and currently the state-of-the-art approaches in language modeling tasks. there is a vast literature on nlms. many works also address the problem of character language modeling @xcite or character-aware lms @xcite . in this work we focus on italian, a romance language derived from vulgar latin. the uniquely fragmented political situation that occurred in italy during the middle age makes italian an extremely variegate and complex case of study, rich of dialects that are still spoken nowadays. we consider a corpus of medieval text collections, with the purpose of easing the research activity on diachronic varieties. moreover, the dataset can be also a valid resource to study the problem text generation in low-data and variegate styles conditions. similar corpora have been already collected for other languages. colonia @xcite , is a portuguese diachronic dataset of about 5 milion tokens grouped by century in five sub-corpora. in (campos et al., 2020) they gathered three corpora for english, spanish and portuguese. in summary, the contributions of this paper are: (1) we present a project, vulgaris, that studied a text corpus consisting of vulgar italian language literary resources, organized in such a way to ease language research, (2) studying the historical and geographical background, the statistical properties of the collected data and its composition and (3) deepening our analysis through a corpus-driven study in dialectology and diachronic varieties exploiting perplexity-based distances. in particular, we introduce neural language models to estimate the perplexity and provide a new indicator, named as perplexitybased language ratio (plr), to analyse the historical evolution process of the varieties. the rest of the paper is organized as follows. in section 2, we describe in detail vulgaris, its composition and we report several statistics on it. then, in section 3, we introduce perplexity-based metrics combined with neural language models that are then used to carry out experiments on the diachronic varieties within vulgaris. ## vulgaris the main goal of project vulgaris foot_0 is the analysis of the diachronic evolution and variance of the vulgar italian language. in order to do so, we collected an heterogeneous literary text corpus, comprehensive of poetry, prose, epistles and correspondence by the most important italian authors ranging from the dawn of the vulgar language to the reinassance age. henceforth, for compactness, we refer to such data as vulgaris. the dataset represents a fundamental timeframe for the italian language, including the first steps and diachronic evolutions departing from the latin language. moreover, through vulgaris it is possible to gain evidence of the early language fragmentation deriving from the complex historical geo-political context of the middle age. ## analysis of language varieties in vulgaris vulgaris texts span over a time period of about four centuries. the diachronic varieties within the dataset are measured in terms of perplexity-based distances, taking into account the different centuries as a reference for the comparison.