Restoring the Sister: Reconstructing a Lexicon from Sister Languages using Neural Machine Translation (2021)

Abstract: The historical comparative method has a long history in historical linguists. It describes a process by which historical linguists aim to reverse-engineer the historical developments of language families in order to reconstruct proto-forms and familial relations between languages. In recent years, there have been multiple attempts to replicate this process through machine learning, especially in the realm of cognate detection (List et al., 2016; Ciobanu and Dinu, 2014; Rama et al., 2018). So far, most of these experiments aimed at actual reconstruction have attempted the prediction of a proto-form from the forms of the daughter languages (Ciobanu and Dinu, 2018; Meloni et al., 2019).. Here, we propose a reimplementation that uses modern related languages, or sisters, instead, to reconstruct the vocabulary of a target language. In particular, we show that we can reconstruct vocabulary of a target language by using a fairly small data set of parallel cognates from different sister languages, using a neural machine translation (NMT) architecture with a standard encoder-decoder setup. This effort is directly in furtherance of the goal to use machine learning tools to help under-served language communities in their efforts at reclaiming, preserving, or reconstructing their own languages.

## introduction historical linguistics has long employed the historical comparative method to establish familial connections between languages and to reconstruct proto-forms (cf. @xcite . more recently, the comparative method has been employed by revitalization projects for lexical reconstruction of lost lexical items (cf. @xcite . in the particular case of @xcite , lost lexical items of the target language are reconstructed by using equivalent cognates of stillspoken modern sister languages, i.e., languages in the same language family that share some established common ancestor language and a significant amount of cognates with the target language. by reverse-engineering the historical phonological processes that happened between the target language and the sister-languages, one can predict what the lexical item in the target language should be. this is essentially a twist on the comparative method, using the same principles, but to reconstruct a modern sister, as opposed to a proto-antecedent. while neural net systems have been used to emulate the historical comparative method 1 to reconstruct proto-forms @xcite and for cognate detection @xcite @xcite , there have not, to the best of our knowledge, been any attempts to use neural nets to predict/reconstruct lexical items of a sister language for revitalization/reconstruction purposes. @xcite report success for a similar task (reconstructing latin proto-forms) by using cognate pattern lists as a training input. instead of reconstructing latin proto-forms from only italian roots, they use italian, spanish, portuguese, romanian and french cognates of latin, i.e., mapping from many languages to one. as our intended usecase (see section 1.1) is one that suffers from data sparsity, we explicitly explore the degree to which expanding the list of sister-languages in the manyto-one mapping can compensate for fewer available data-points. since the long-term goal of this project is to aid language revitalization efforts, the question of available data is of utmost importance. machine learning often requires vast amounts of data, and languages which are undergoing revitalization usually have very sparse amounts of data available. hence, the goal for a machine learning approach 1 due to the nature of neural nets we do not know whether these systems actually emulate the historical comparative method or not. what is meant here is that they were used for the same tasks. our particular contributions are: ## the dataset the data set used for this experiment was provided by shauli rafvogel of @xcite . the initial set consisted of 5420 lines of cognate sextuples of the romance language family, specifically: ro-2 or, rather, it interprets. ## experimental setup this experiment was run using the opennmtpytorch neural machine translation @xcite ) framework, using the default settings (a 2layer lstm with 500 hidden units on both the encoder and decoder). the opennmt-py default setup was chosen intentionally; the envisioned use-case requires an easily reproducable approach for interested users or communities who might profit from using this method for their own purposes, but who don't necessarily have deep expertise in machine learning or tuning neural models. a publicly available toolkit, like opennmt, and a no-configuration setup helps lower the bar to entry for these parties. neural machine translation (nmt) frameworks are designed to translate sentences from one language to another, but they can be used for a number of sequential data tasks @xcite . one such task is the prediction of a cognate from a set of input words, as used here. these frameworks are typically an encoder-decoder setup, where both the encoder and decoder are often implemented as lstm (long short-term memory) networks @xcite , which have the advantage of effectively capturing long-distance dependencies @xcite . in an encoder-decoder setup, the encoder reads in the character based input representation and transforms it into a vector representation. the decoder takes this vector representation and transforms it into a character based output representation @xcite . nmt frameworks also employ a "vocabulary" set, which contains vocabulary of the language that is being translated from and vocabulary of the language that is being translated to. the size of this vocabulary is often an issue for the effectiveness of nmt models @xcite . in our case, the source vocabulary simply contains all of the characters that occur in all the input language examples and the target vocabulary contains the characters that occur in the target language example. to illustrate: if this task was about predicting english words, then the target vocabulary would contain all the letters of the english alphabet. ## evaluation measures machine translation is usually evaluated using the bleu @xcite score, but bleu is designed with sentence level translations in mind. we instead evaluate the output according to edit distance in the style of @xcite by calculating the percentage of the output which is within a given edit distance. in addition to this metric, we also use a custom evaluation metric designed to emphasize the usability of the output for the intended use-case, i.e., as predictions to be vetted by an expert to save time over doing the entire analysis manually. in order to calculate this score, we calculate the damerau-levenshtein edit distance to the target for each word and assign weights to them by their edit distance. that is: @xmath0.9 + c * .8 + d * .7 + e * .6)/t where a is number of predictions with distance 0, b is the number with distance 1, c is the number with distance 2, d is the number with distance 3, e the number with distance 4, and t is the total number of predictions. as an example, consider a scenario where there are three predicted cognates. if system 1 produces 3 output patterns within an edit distance of 2, it would receive a score of 0.8. if system 2 produces two output patterns with edit distance 0 and one within a distance of 5, this would result in a score of 0.67. the logic behind this metric is that any prediction with an edit distance larger than 4 edits is essentially useless for the proposed task. since such a large edit distance essentially constitutes an incorrigible mistake as mentioned in (section 1.1). the edit distance of 4 constitutes an arbitrary cut-off to a degree, but it allows us a simple and informative evaluation metric for our use case. this metric will rank a model that has a large number of items in a and a large number of items beyond 4 edits lower than a model with items mostly in the b-d range. presumably, the latter is more useful to the task, as small errors can be adjusted by linguists or language users. using this metric, we can rank different input combinations according to their assumed useful- in we can also observe that the performance successively drops as we remove languages, with the spanish only 6 performing worst. however, the way in which this performance drops is not entirely transparent. it appears that in terms of scoring, the spanish-french (spa-fre) sample actu- the noticeable difference between span-fre-port and span-port-ro is surprising and warrants some examination. the likely explanation is twofold. first, the romanian set is the one with the most empty patterns. the romanian training data only includes 930 filled patterns, in comparison, portuguese includes 1905 patterns, french includes 1790, and spanish has 2125. it may be the case that the romanian data is too small in comparison with the others to have a significant impact on the outcome. the other factor may be that romanian is phylogenetically the most distant from the target language (italian) (figure 1 ). this becomes even more apparent in figure 2 , which. shows the performance of different models over time. 7 here we can observe that there is hardly any difference between the performance of span-fre-port-ro and span-fre-port over time, and it is only at 10,000 steps that they start to diverge. this divergence at the 10,000 step mark is likely random, the graph suggest that their overall performance is almost identical in regards to scoring. another point in this direction are the seemingly convergent graphs of span-fre and span-port-ro, suggesting that there is no difference between using 2 or 3 languages as input if the third language is romanian. discounting the performance of the exclusion/inclusion of romanian, we can observe that performance overall tends to increase with each parallel language added. this is especially evident with the obvious drop-off in performance of the spanish only input. if we assume that romanian has no impact, then we can see that 3 languages (blue and orange) perform similarly and two languages (red and green) perform similarly, and there is an obvious drop-off between those two patterns. this suggests that using parallel language input can compensate smaller datasets. due to the small dataset, the scores plateau fairly early, around the 3000 epoch mark for most. this suggests that it would be sufficient to run these models at 3000 epochs, which would save some time on low-end hardware. however, with these small datasets, training time should rarely exceed 5 hours on consumer grade pcs. foot_5 ## warning labels for interested linguists there are some important aspects of this kind of approach that linguists, or community members who are interested in utilizing it for their purposes, should be aware of. there are certain things that this type of approach can and cannot do for a community or project. the model does not so much reconstruct a word for the community, but rather proposes what the word could be, according to the data it has been fed. the model will propose these recommendations on the basis of an abstract notion of what the historic phonological and morphological differences are between languages abc and language d. this does not necessarily mean that the model learns or understands the historical phonological and morphological processes that separate the input sister languages from the target languages. it has simply learned a way to generalize from the input to the output with some degree of accuracy. what is learned need not necessarily overlap with what linguists believe to have happened. therefore, this type of model will only ever generate cognates of the input. it cannot generate novel items. this is an important factor to consider for any community or linguist planning on using this approach. consider the following case: imagine we are trying to use this approach to reconstruct english from other germanic languages. a large part of the english lexicon is not of germanic ancestry. however, any lexicon we would try to reconstruct using this trained algorithm would give us approximations of a germanic derived lexeme for the word we are trying to reconstruct. this is a potentially undesirable effect of the way the model was trained. linguists and interested community members need to be aware of this and implement their own quality control. however, this approach can potentially be useful for any language project where a community and or linguists are working with an incomplete lexicon for a language. the prerequisite for this being a useful tool in such a scenario is the assumption that the sister languages to the target language are somewhat well documented and have at least dictionaries available from which data can be extracted. a final prerequisite is the presence of minimally a small dictionary of the target language. the model would then be trained using the sister languages as input, and the target language list as a target output. after training confirms a reasonable accuracy, the model can then be fed with other known words in the sister language to get a prediction of those words in the target language. after producing said output, the linguist, or language community, needs to subject the output to a quality control and decide on a series of questions: do the output patterns match what we know of the target language? can we assume that these words are cognates in the target language, or is there some evidence that other forms were present? finally, if this is used by a community to fill in empty patterns in their language, the community needs to decide whether the output is something that the community wants in their language. the algorithm is not infallible, and only proposes. ultimately, a language community using this tool must make a decision whether to accept or reject the algorithm's recommendations. ## conclusions in this paper, we have shown that nmt frameworks can be used to predict cognates of a target language from cognates of its sister languages. we have further shown that adding or removing input languages has interesting effects on the accuracy of the model. this indicates that we can use additional sister languages to compensate the lack of data in a given situation, though, as demonstrated in the case of romanian, we cannot blindly add sister languages, nor assume that all additions are equally useful. this might be a promising method for situations where not a lot of data is present, but there are multiple well-documented related languages of the target language. the next step for this line of research is to move from a proof of concept to an implementation in an actual language revitalization scenario. this is something we are currently working on. a further question that need to be addressed as well, is how well this approach performs with languages that exhibit a different morphology from the romance languages, such as agglutinative and polysynthetic languages. all code and data used for this project are opensource and can be found here, in order to reproduce these results. something we would like to address in this final paragraphs is that machine learning is a potential tool. like every tool, it has its uses and cases where it is not useful. the decision of using such a tool to expand the lexicon of a language is a decision of that language community, and not of a linguist.