A New Dataset for Tonal and Segmental Dialectometry from the Y ue- and Pinghua-Speaking Area (2024)

Abstract: Traditional dialectology or dialect geography is the study of geographical variation of language. Originated in Europe and pioneered in Germany and France, this field has predominantly been focusing on sounds, more specifically, on segments. Similarly, quantitative approaches to language variation concerned with the phonetic level are in most cases focusing on segments as well. However, more than half of the world’s languages include lexical tones (Yip, 2002). Despite this, tones are still underexplored in quantitative language comparison, partly due to the low accessibility of the suitable data. This paper aims to introduce a newly digitised dataset which comes from the Yue- and Pinghua-speaking areas in Southern China, with over 100 dialects. This dataset consists of two parts: tones and segments. In this paper, we illustrate how we can computationaly model tones in order to explore linguistic variation. We have applied a tone distance metric on our data, and we have found that 1) dialects also form a continuum on the tonal level and 2) other than tonemic (inventory) and tonetic differences, dialects can also differ in the lexical distribution of tones. The availability of this dataset will hopefully enable further exploration of the role of tones in quantitative typology and NLP research.

## introduction traditional dialectology or dialect geography @xcite , is the study of geographical variation of language. this field originated in europe, pioneered in germany and france, and has always been focusing on sounds, more specifically, on segments. in the second half of the 20th century, the quantitative turn in dialectology, known as dialectometry, is no exception to this. while these methodologies have been widely used in europe and america, there are only limited regions in the rest of the world which employ these methodologies, although there is a sign of growth in recent years. for instance, yucatec mayan @xcite , bantu languages @xcite , japanese @xcite . tonal languages are defined as " @xcite in which an indication of pitch enters into the lexical realisation of at least some morphemes" @xcite , and more than half of the world' s languages include lexical tones @xcite . despite this, tones are still gaining little attention in quantitative models of language variation. this lack of attention on tones is not surprising, however, since most european languages mostly do not use pitch to differentiate lexical meaning. one other reason could be the fact that digital data is not accessible and freely available. these factors cause barriers to the development of computational methods for tonal languages. for example, it is unclear whether the existing methods (e.g. @xcite are suitable and adequate to deal with tones in tonal languages @xcite . take chinese dialectology as an example, there are numerous studies on dialects spoken in china, and it has a century-long tradition, but most studies on tonal variation are descriptive. traditional studies usually report the tonal inventory of a dialect after a fieldwork investigation, and/or tones are analysed in terms of how they correspond to historical tone categories (from the middle chinese period, based on the ancient rhyme dictionary descriptions). although there is a huge amount of dialect data available for chinese (in the form of ipa transcriptions, including tones), they are mostly printed on paper and are not digitised, ready to be used for quantitative analyses. until today, there is generally a very limited number of digital datasets which allows us to quantitatively model variation of tones, which is problematic given that the majority of the world's languages are tonal. furthermore, although there are tools which allow us to align southeast asian tone languages @xcite , and then visualize the correspondences (both tones and segments) in table form @xcite , these tools were developed for historical linguistics. in order to understand the synchronic dialect variation on the tonal level, alternative methods are needed in order to investigate how tones vary beyond correspondences. this paper aims to introduce a newly digitised dataset which comes from the yue-and pinghuaspeaking areas in southern china, with over 100 dialects. 1 this dataset consists of two parts: segments (section 3) and tones (section 4). the availability of this dataset will hopefully be an invitation to researchers around the world to initiate an exploration of tonal variation, which has long been neglected. in section 5 we present out preliminary research on tonal variation, followed by a conclusion. ## data sources the data presented in this paper consists of segments and tones. segments contain impressionistic transcriptions of consonants and vowels of the words. impressionistic tone transcriptions of pitch contours are represented using @xcite tone letters. the two sets of transcriptions are from the same sources; they were extracted from the same words (see below) and from the same dialects. there are two main sources for the dataset, namely word lists and homonymic syllabaries, which came from various dialect surveys and individual studies. both sources are based on impressionistic transcriptions from word elicitation, but they are presented differently. word lists are word-based, meaning words are organised in a tabular format @xcite , where the ipa transcriptions of each word are listed for each dialect all at once. on the other hand, homonymic syllabaries are pronunciation-based, meaning words with the same pronunciation are grouped together under one pronunciation (represented by the ipa transcriptions). ## modifications to the original segmental transcriptions dialect survey data are often found with transcribers' differences (or fieldworker isoglosses, @xcite ) when there are more than one fieldworker documenting dialects in the field. transcribers'differences are inconsistencies of impressionistic transcriptions due to the different uses of phonetic symbols to represent the same sound by different transcribers. in other words, the differences we see in the data might be due to the habitual difference of the fieldworker instead of 'real' linguistic difference. to reduce the effects from the transcribers'differences, we have made some modifications to the original data. ## tone data the second half of the dataset consists of the tonal data from the same words and the same dialects. up to date, there are no large-scale dialectometric studies on tones; the highest number of dialects involved are no more than 20 dialects (see @xcite ; @xcite ). in some dialectometric studies, tones were neglected (e.g. wichmann and ran (2019) ), others used a rather simplified method (e.g. @xcite ). in addition, there are studies on the correlation between phonetic distance and the perception of tones (e.g. @xcite ), which do not focus on the application of these measures on dialect classification. research questions regarding to the variation of tones in larger dialect areas, or if there is a correlation between tonal and segmental variation cannot be researched upon using these datasets. our tonal dataset is different from existing digital datasets, since it allows comparisons between tonal and segmental levels. the tones were transcribed in @xcite tone letters, which is a system for tone transcription consisting of 5 digits, 1, 2, 3, 4, 5, representing different (possible) contour levels in a tone. in this system, 1 represents the lowest contour level and 5 represents the highest. when combined together (with two digits or three digits), they can indicate a change in the contour, which represents the shape of the tone. for example, 53 is a falling tone, whereas 213 is a dipping tone (a falling contour followed by a rising contour). the tonal transcriptions cannot directly be used for the purpose of dialectometry, though. sung et al. (forthcoming) have found that directly applying levenshtein distance on the tone letters and comparing these tones categorically (the 'binary' method, sung et al. (forthcoming)) do not yield satisfying tone distances for the purpose of dialectometry. further conversion of these tones to another representation (e.g. onset-contour-offset, @xcite is required in order to get more meaningful tone distances. the availability of tones as tone letters allows users to apply any conversion of their choice. the question raised in @xcite shows that currently there is no existing satisfying tone distance metric for dialectometry. in the subsections below, we briefly introduce three quantitative models of tone representation, tested in @xcite , as well as our modified version of the existing rep-resentation proposed by @xcite 5 . ## preliminary results of tonal dialectometry of yue dialects in this section, we will present our preliminary analysis of the tonal data, using the moco tone representation. the first question that we will try to answer is whether dialects form a dialect contin-6 superscripted characters are counted as a difference of 0.5 in the levenshtein algorithm implemented in gabmap, if the last character of the offset of both tones (but not the length) are identical. this implies that the tone length is only differentiated if the final character of the offset in the moco representation is identical. please note that led-a.org does not have the same implementation of the superscript h . uum, which has previously been observed in different languages on the segmental level. a follow-up question we investigated is in what ways do tones differ gradually from one dialect to another. ## conclusion our yue dataset has provided new possibilities in the study of language variation. it consists of both tonal and segmental data for the same lexical items for over 100 dialects. to our knowledge, this is one of the biggest dialectal dataset for tones within one language area. our tonal dataset is digitised from dialects surveys which were transcribed in @xcite , which means that it can be converted to any existing tone representations for further dialectometric analyses. in this paper, we have briefly demonstrated how we can use one of these representations to investigate how tones vary across different dialects. by using the moco representation, which can differentiate almost 99% of the tones in our data, we have identified a dialect continuum as well as a dialect island, namely the siyi dialect group. through a comparison of tone inventories and tone correspondences of siyi dialects, we have further identified that dialects can differ on the tonal level tonemically, sub-tonemically and in terms of lexical distribution. tonal languages have been neglected in the study of linguistic variation for decades, partly due to the lack of available data. we hope this dataset will serve as the first step to remove the barrier for any scholars who are interested in variation of tones.