D iatop I t: A Corpus of Social Media Posts for the Study of Diatopic Language Variation in I taly (2023)

Abstract: We introduce DiatopIt, the first corpus specifically focused on diatopic language variation in Italy for language varieties other than Standard Italian. DiatopIt comprises over 15K geolocated social media posts from Twitter over a period of two years, including regional Italian usage and content fully written in local language varieties or exhibiting code-switching with Standard Italian. We detail how we tackled key challenges in creating such a resource, including the absence of orthography standards for most local language varieties and the lack of reliable language identification tools. We assess the representativeness of DiatopIt across time and space, and show that the density of non-Standard Italian content across areas correlates with actual language use. We finally conduct computational experiments and find that modeling diatopic variation on highly multilingual areas such as Italy is a complex task even for recent language models.

## introduction italy is one of the most linguistically-diverse countries in europe. despite its relatively small geographical area, it exhibits a notable profusion of linguistic variation, "hold[ing] especial treasures for linguists" @xcite . therefore, the study of diatopic linguistic variation in italy has constantly been a focal point in linguistics @xcite . on the other hand, little attention has been given so far to this matter in the natural language processing community. indeed, most work in nlp still focuses on standard italian (ita; the official national language), considering it as a "monolithic language". however, a large number of local languages, dialects, and regional varieties of standard italian (i.e., regional italian) 2 shape the italian lin-1 repository: https://github.com/dhfbk/diatopit 2 geographical differentiation of standard italian due to influences by languages and dialects of italy @xcite . in this paper we introduce diatopit, the first corpus of geolocated social media posts from twitter with a focus on diatopic variation in italy for language varieties foot_0 other than standard italian. di-atopit comprises 15,039 posts with content fully written in local language varieties (cf. figure 1 , (a)), exhibiting code-switching with standard italian (cf. figure 1 , (b) ), or including regional italian (cf. figure 1 , (c )). compared to other existing datasets with geolocation information @xcite @xcite , diatopit is focused on italy and on nonstandard language use. we describe how we tackled challenges in the corpus creation process, such as the lack of reliable, variation-informed language identification tools and the absence of orthography standards for most local varieties (section 2), and provide detailed analyses over time and space, also highlighting the density and function of non-standard italian content across italian regions (section 3). finally, we show that modeling diatopic language variation is a difficult task even for stateof-the-art language models (section 4). the corpus is meant to encourage research on diatopic variation in italy, study code-switching and divergences in orthography for local language varieties, and serve as a basis for responsible development of annotated resources for italy's varieties. ## corpus creation building a corpus of social media posts written in language varieties of italy other than standard italian is a tough task, especially in the absence of reliable language identification tools. 4 most languages and dialects of italy -see @xcite for an overview -are primarily oral and have no established orthography, and standards that have been proposed for a fraction of them are rarely adopted by their speakers. indeed, when those language varieties are transposed into writing, speakers typically write "the way words sound" (ramponi, 2022). the language functions of those varieties -most of which are endangered @xcite -are increasingly restricted, resulting in frequent code-switching with standard italian, a sign of language replacement @xcite . in this section we describe how we tackle these challenges to build the diatopit corpus. we detail all stages, from data collection (section 2.1) and sampling for non-standard italian content (section 2.2), to content curation and data augmentation of under-represented speaking areas (section 2.3). data statements @xcite for diatopit are presented in appendix a. ## corpus analysis in this section we present detailed analyses on the diatopit corpus. we first provide summary statistics (section 3.1). then, we discuss the corpus distribution across time and space (section 3.2). lastly, we show that the density of non-standard italian tokens across regions correlates with the actual use of languages varieties in italy, and that language functions of the most indicative tokens per region are good indicators of vitality (section 3.3). ## experiments in this section we present our experiments on the diatopit corpus. our objective is to understand how difficult it is to model diatopic language variation in italy, i.e., by identifying coarse-and finegrained geographical areas of a post based solely on its textual content. 14 ultimately, this will help in building tools to reliably identify content for language varieties of italy from social media, and thus to better represent them in nlp. we first introduce the experimental setup (section 4.1) and the baselines we employed (section 4.2). then, we present the results and provide a discussion (section 4.3). ## conclusion we present diatopit, the first corpus focused on diatopic variation in italy for language varieties other than standard italian. our analyses and experiments show that diatopit is highly representative of actual use of italy's language varieties, and can thus be used to advance research in the area. we plan to study divergences in orthography and code-switching in future work, in order to further assess vitality across varieties. data and relevant materials (e.g., search terms) are available to the research community at https: //github.com/dhfbk/diatopit .