Letters From the Past: Modeling Historical Sound Change Through Diachronic Character Embeddings (2022)

Abstract: While a great deal of work has been done on NLP approaches to lexical semantic change detection, other aspects of language change have received less attention from the NLP community. In this paper, we address the detection of sound change through historical spelling. We propose that a sound change can be captured by comparing the relative distance through time between the distributions of the characters involved before and after the change has taken place. We model these distributions using PPMI character embeddings. We verify this hypothesis in synthetic data and then test the method’s ability to trace the well-known historical change of lenition of plosives in Danish historical sources. We show that the models are able to identify several of the changes under consideration and to uncover meaningful contexts in which they appeared. The methodology has the potential to contribute to the study of open questions such as the relative chronology of sound shifts and their geographical distribution.

## introduction the study of sound change goes back to the beginnings of modern linguistics in early nineteenth century, when scholars such as rask and grimm started making hypotheses about the way sound changes over time, which in turn lead to the discovery of regular sound correspondences between ancient languages and the identification of cognates in modern ones @xcite . since spoken language from the past is not available, sound change in ancient languages must be deduced from written records by considering development in spelling through time. in addition, while we may be able to see from the written records that a change did occur, less is known on the specific dynamics of the change. details of these dynamics include knowledge of when the change started to appear, how long it took for it to be complete, what was the relative chronology of individual sounds in a larger shift, what was the geographical distribution of a change and so forth. due to the sparsity of linguistic evidence, detailed empirical studies of chronological sound change are difficult to conduct. this is especially the case for older stages of languages, where little written text was produced, and much of what did exist has been lost in transmission. however, as we move forward in history to the rise of bureaucracy, for example in medieval europe, we see that an extensive amount of written records were made. text from this period of time is interesting in the context of a study of sound change because it shows great variability in spelling patterns. with the digitalization of such archives foot_0 , therefore, new opportunities arise to apply computational methods to the study of sound change through written text. considerable effort has already been devoted to the development of computational approaches aimed at discovering lexical semantic change (lsc) in historical corpora @xcite . however, change related to phonology, morphology and syntax has remained out of the spotlight in nlp research. in this study, we seek to bridge this gap as regards phonology. inspired by the work on lsc, we propose a method whereby sound change is traced via spelling change in historical text and modeled by training diachronic character embeddings over text from different time periods. we start by reviewing previous approaches to the automatic detection of semantic shifts and spelling modification due to sound change. then we formulate our hypothesis that a sound change can be traced using diachronic distributional embeddings. while sound change is not completely analogous to word meaning change, we argue that similar meth-ods can be used for both. to verify our hypothesis, we conduct three studies on simulated sound change. first, we test the methods on the phonological environment of a simple artificial language. then, we apply the same methods to a more complex scenario created by simulating sound change in a corpus of synchronic danish text. having established the suitability of the methods on these two datasets, we finally experiment with tracing a well-known sound change in real historical language data, again in danish. the implementation and datasets are available online foot_1 . ## related work the application of nlp methods to automatic lsc detection is already a rather well-developed subfield of nlp research @xcite . in particular, the emergence of word embeddings as a viable way to model the distributional hypothesis in semantics @xcite has paved the way for an application of word embeddings to lsc modeling @xcite @xcite . synchronically, the meaning of a word is characterized by word embeddings in terms of the contexts it appears in. lsc is captured by training word embeddings at different time points and comparing these distributions typically using cosine distance. recently, contextualized word embeddings have also been applied to the problem. while such models have the capability to capture change in distinct word usages, preliminary results suggest that traditional word embeddings are superior to the task of semantic change detection @xcite . one of the main issues in this comparison is the temporal alignment of dense embedding spaces. for example in the case of neural models, embeddings are initialized and trained stochastically, which means that separate runs -on even the same data -will yield different embedding spaces. thus, work has focused on the development of methods to perform alignments to make dense embedding spaces comparable across time (see @xcite for an overview). as an alternative to neural embeddings, scholars have also used purely count-based measures, which are naturally aligned across dimensions. normalization techniques are also applied, e.g. based on positive pointwise mutual information (ppmi) @xcite . most studies of lsc do not rely on a control dataset against which to validate their conclusions. in @xcite , on the contrary, it is argued that any claims about putative laws of semantic change in diachronic corpora must be evaluated against a relevant control condition. the authors propose a methodology in which a control condition is created artificially from the original diachronic text collection by reshuffling the data. no systematic lsc is expected in the artificially developed control dataset. the distributional hypothesis has also been proposed as an explanatory model within the domain of phonology suggesting that phonological classes are acquired through distributional information @xcite . driven by this hypothesis, recent work has focused on testing how distributional properties can be learned by phoneme embeddings (see mayer 2020 for an overview). @xcite investigated to what extent learned vector representations of phonemes align with their respective representations in a feature space in which dimensions are articulatory descriptors (e.g., ±plosive). recently, mayer (2020) has shown that phonological classes, such as long and short vowels, can be deduced from phoneme embeddings normalized using ppmi by iteratively performing pca on candidate classes. thus, while the distributional hypothesis for phonology is well-established, one notable issue is the fact that the empirical evidence to study sound change is relatively inaccessible since it requires recorded speech or phonologically transcribed data. simulation is therefore used as a tool for studying the underlying mechanisms of sound change by creating computational models based on linguistic theory @xcite . through simulation, questions pertaining to e.g., what factors influence the (in)stability of vowel systems across generations @xcite can be modeled by controlling the assumptions made by the model. work on simulation ranges from implementing theoretical approaches using mathematical models @xcite to iterated learning and neural networks @xcite . while the output of such models can be tested empirically on what we observe at a synchronic level, they are primarily theoretically driven. in this paper, we wish to take a data-driven approach and utilize some of the methods reviewed above to track historical sound change in writing. rather than using word embeddings as done to model lexical change, we will use character embeddings, that are better suited to the task of sound change modeling. ## modeling sound change within the field of lsc detection, change in word semantics is traditionally measured by computing pairwise similarity @xcite ) over a time series, (t, ..., t + δ), in which a shift in the meaning of a word, w i , can be measured by its relative distance to another word, w j . in this way, hypotheses about specific shifts may be tested. another measure is semantic displacement, in which semantic change for a given word is quantified by measuring its temporal displacement. for both measures, looking at consecutive time steps provides a measure to the rate of change of a wordin relation to another word, or independently. while lsc is about meaning shifts of unchanged word forms, sound change is a change of form, i.e., a given phoneme changes to another one within certain contexts. we denote such a change @xmath0, where 'c' stands for a given context. while changes of either a or b will be reflected in changes to their individual distributions (displacement), looking at them independently of one another will not tell us whether one of the phonemes is becoming similar to the other. therefore, we suggest to look at the pairwise similarity between a and b. more specifically, given a time series (t 1 , ..., t n ), in which t 1 denotes a time before a sound change was in effect and t n denotes a time where a sound change is completed, we expect b i to move towards a 1 as @xmath1, in other words to become similar to a 1 , since it will begin to appear in the same contexts. as was noted earlier, sound is not accessible in historical text, to which we would like to be able to apply our methodology. in historical text preceding spelling conventions, sound is assumed to be reflected in spelling. while detailed philological and linguistics analyses of written language can help to determine if a distinct spelling corresponds to a particular phoneme, or whether that spelling is rather a reflection of synchronic spelling variation @xcite , resources including such analyses are scarce. thus, we chose to use characters as a proxy for sound, and model sound change through changes in the distance between pairs of character distributions. in addition, before assuming that an observed decrease in the distance between two such distributions reflects a real change, we also want to see that the same decrease is not visible in a control corpus in which no such change has indeed taken place. ## experimental setup in order to verify the hypothesis that sound change can be traced using distributional information with the methodology proposed above, we test whether we are able to trace simulated change in synthetic data. as a first synthetic setting, we restrict ourselves to track change in a synthetic language with simple phonotactics. in this way, we get a sense of whether the proposed hypothesis works under perfect conditions, i.e., one in which characters correspond with phonemes one-to-one. in the second synthetic setting, we seek to imitate the condition of tracing change in an orthographic setting by simulating change in a corpus of synchronic text in which character distributions interact with the noise added by spelling and lexicon. in both synthetic settings, we compare the simulated change to a control setting where no change has occurred. finally, we will test the hypothesis on real data. our goal is to trace the lenition after vowels of voiceless plosives, p t k, to their voiced counterparts, b d g, in historical danish. while this change is believed to have initiated around the beginning of the 14 th century, details about the relative chronology of the series and geographical distribution of the change are difficult to account for @xcite . therefore, in an attempt to discover interesting patterns of this change, we train character embeddings on historical sources from the periods following the time when the change is believed to have started. as we did for the synthetic data, and again following @xcite , we also introduce a control setting to test the significance of the observed changes. ## results the 'bin' estimate shows by how much the distance is expected to change for every temporal interval. a negative effect means that the distance between the two sounds is becoming smaller. the 'control' effect shows the predicted change to the initial intercept in the control corpus (this corresponds to the effect of the corpus variable), and finally 'bin:control' shows the interaction between temporal bin and corpus type. in the r implementation, each of the models is expressed in terms of the following equation: in both corpora where change is simulated, there is a significant effect of temporal interval. this is expected given the fact that gradual change has been induced in the data. for both corpora, the effect of the control corpus on the initial sound distance is not significant. importantly, the interaction between the effect of the temporal bin and the control corpus is significant in both cases. the interaction supports the hypothesis that we see a pattern of change in the simulated corpora that is significantly different compared to the control data. the interactions are shown in the plots in figure 1 . turning to the results for the danish geographical names corpus, while the models show significant effects of bin, control and interaction between the two for the @xmath2, no significant effects are found for the @xmath3. when we look at the corresponding interaction plots in figure 2 , we see that the distance between p and b in the corpus decreases in the third bin to then increase and finally slightly decrease again in a non-linear way. the changes displayed in the plots in (b) and (c), on the contrary, follow the expected trend: the observed consonant is moving towards its voiced version in the real corpus but not in the control. ## discussion the results from the two simulation studies suggest that sound change can be traced with our proposed methodology of measuring the distance between pairs of character distributions over time. we showed this both in a simplified setting (parupa), and in the orthographically noisy environment pro-vided by synchronic danish data (ud danish). the main assumption in these simulation studies was that change could be modeled linearly. however, as discussed by scholars, change is often not linear, and can follow an s-shaped curve through a community @xcite . in a study of semantic lexical change based on synthetic data, @xcite experiment with the injection of changes the probabilities of which vary linearly or logarithmically, and find that regression in general provides reasonable results in spite of being sensitive to outliers and of producing a certain amount of false positive results. it also performs better than a non-parametric measure like kendall's τ . the results obtained in our study seem to confirm the usefulness of linear models to detect sound change even though one of the cases of lenition targeted in the danish geographical names corpus could not be modelled. focusing on our results on the tracing of lenition, then, we were able to identify a change from /t k/ → /d g/. however, an important thing to note in regards to the control setting for these results is how it diverges from the synthetic settings, which we initially used as a verification of the proposed hypothesis to trace sound change. there, the to get a sense of this, instead of looking at the euclidean distance for the full embedding, we ran linear regression on the target data looking at differences between character distributions for each dimension. we then extracted the patterns corresponding to the dimensions showing significant differences and considered those with the highest pearson's r coefficient (tables 2 3 4 ). starting with the resulting patterns for parupa and ud danish, in both cases we are able to identify the exact contexts where the change was simulated: in parupa before i/u and in the ud danish corpus, between vowels and in the frequent suffix -ig(t) (although the end-of-word is not captured due to n-gram size restrictions). for parupa, it is worth noting how the model captures patterns after vowel as well. this position is only implicitly involved in the conditioning of the simulated change, and the this latter example is related to an important issue in language evolution: when language changes 9 danish sig 'bog; mire' from old danish sik, compare norwegian and swedish (dialectal) sik (danmarks @xcite 10 danish ager 'field' from proto-germanic akra, compare english acre and swedish åker @xcite . in this respect, another important point to note about the retrieved patterns -both from the simulation of ud danish and the tracing of @xmath4, and are thus characterized by high frequency of occurrence across word forms. while the observation that frequent patterns are more easily captured may seem trivial, lack of sufficient evidence may nevertheless be the reason why we cannot model the @xmath5. germanic p descends from proto-indo-european (pie) *b, which, however, has a special place in the pie phoneme inventory and is considered a sort of black sheep that some scholars do not believe to have existed due to its few attestations. in fact, the attestations of germanic p most often come from loan words and are not seen in morphemes. thus the evidence for @xmath6. further investigation of this sound change could be carried out by means of additional simulations, or more detailed analysis of the obtained character distribution, and is left for the future. a final observation on the identified patterns is that the model fails to generalize across synchronic variation in spellings. for example, we see that a spelling with ii is treated alongside spelling with a single i. while this type of variation could to some extent be accounted for by treating it as an independent variable, such a solution would have consequences for our experiment design in that we use ppmi weighting on raw n-gram counts. this method enabled us to interpret the exact inner workings of the model and find the contexts in which a change has happened. if we had used neural models for example, in which characters are represented by dense embeddings, similar characters would have shared similar representations, thereby perhaps allowing the model to generalise e.g., to sound change occurring after a vowel. in this study, we wanted to privilege explainability, but dense representations should be explored in the future. ## conclusion and future work in this paper we presented a novel method for the modeling of sound change through the use of diachronic character embeddings. sound change is modeled in terms of increasing similarity between character distributions across time intervals. the proposed method was tested on synthetic data with promising results, and then applied to a real world scenario with the goal of tracing the lenition of /p t k/ → /b d g/ in danish by looking at spelling in historical sources. the method was able to detect the changes for two of the sound pairs, and also to point at specific contexts of occurrence that influenced the changes. however, our evaluation showed that the proposed models were sensitive to variation relating to vocabulary. to what extent such variation is responsible for the occurrence of false positives has yet to be investigated. for scholars interested in sound change, there are a number of important open questions, such as the relative chronology and geographical distribution of sound shifts. although we have not addressed these questions here, we believe our methodology can be further developed in ways that would allow to do so, e.g., by adding geographical location as an additional factor in the models. both issues would constitute interesting avenues for future research. in this paper we have used purely count-based methods. while this approach enables us to directly interpret the results of the models, it also suffers from its inability to generalise across contexts. this drawback motivates experimenting with neural methods that make use of dense character representations, to test whether they can make similar generalisations as done by historical linguists, particularly as regards infrequent patterns that could be captured across word forms.