Unstable Grounds for Beautiful Trees? Testing the Robustness of Concept Translations in the Compilation of Multilingual Wordlists (2025)

Abstract: Multilingual wordlists play a crucial role in comparative linguistics. While many studies have been carried out to test the power of computational methods for language subgrouping or divergence time estimation, few studies have put the data upon which these studies are based to a rigorous test. Here, we conduct a first experiment that tests the robustness of concept translation as an integral part of the compilation of multilingual wordlists. Investigating the variation in concept translations in independently compiled wordlists from 10 dataset pairs covering 9 different language families, we find that on average, only 83% of all translations yield the same word form, while identical forms in terms of phonetic transcriptions can only be found in 23% of all cases. Our findings can prove important when trying to assess the uncertainty of phylogenetic studies and the conclusions derived from them.

## introduction while the quantitative turn in historical linguistics has been met with a considerable amount of skepticism for a long time @xcite , phylogenetic methods -originally developed to infer phylogenies of biological species -have by now become the new state of the art in the field, replacing more traditional methods for subgrouping almost completely. given that most linguistic approaches to phylogenetic reconstruction make use of lexical data, multilingual wordlists play a crucial role in phylogenetic reconstruction in historical linguistics. the compilation of multilingual wordlists itself is quite tedious. starting from a list of concepts, scholars must translate the concepts into all target languages under investigation. the translation into the target languages, however, is no standardized procedure, but may require various steps, including the consultation of informants, the consultation of published resources, or the inspection of archived material. in all these cases, scholars who compile a wordlist must weight their evidence carefully, in order to avoid errors. given the complexity of this process, it is no surprise that errors can easily slip into the translations. a given concept may lack a direct translational equivalent in a given language, or there may be several good candidates from which scholars must select the most appropriate ones. as a result, there is a great risk that multilingual wordlists compiled for phylogenetic studies show a considerable amount of idiosyncrasies that might have an impact on the phylogenies scholars compute from them. studies that try to measure the amount of inconsistency in multilingual wordlists -introduced by the translation of concepts into target languages -are lacking so far. here, we present a first attempt to shed light on the robustness of the concept translation task, taking advantage of the fact that recent efforts have produced large-scale repositories of standardized multilingual wordlists @xcite . in the following, we will give a short overview on previous discussions and studies that focus on the translation of concepts in multilingual wordlist compilation ( § 2). after this, we introduce the materials and methods by which we try to evaluate the robustness of concept translation ( § 3). having presented the results ( § 4), we discuss them in more detail and share some ideas to improve the enterprise of wordlist compilation in historical linguistics ( § 5). ## background phylogenetic approaches rely on multilingual wordlists compiled through lexicostatistic methods, dating back to swadesh's foundational work @xcite @xcite . a multilingual wordlist in this context is a list of concepts translated into one or more target languages @xcite . although typically emphasizing their difference with respect to swadesh's lexicostatistics, modern phylogenetic approaches all build on this onomasiological (i.e. concept-based) approach that takes the concept as the major aspect by which languages are compared. building on @xcite , we can identify five major steps in the typical workflow applied in modern approaches to phylogenetic reconstruction. starting from the compilation of a concept list (1: concept list compilation), the concepts are translated into the target languages (2: concept translation) in order to create an initial comparative wordlist. this wordlist is then used as the basis for the identification of cognate words (3: cognate identification). having converted the information on cognate words into a numerical or computer-readable format (4: cognate coding), scholars then employ their phylogenetic method of choice in order to compute a phylogeny of the languages in question (5: phylogenetic reconstruction). up to now, most critics of lexicostatistics and its modern equivalents have concentrated on either the stage of cognate identification or the resulting phylogenetic methods. cognate identification is often criticized as being flawed due to undetected borrowings @xcite . when disputing over phylogenetic methods, there have been long-standing debates about the complexity of the models employed, as reflected in the debate about the age of indo-european, where models differing in complexity yielded quite different age estimates @xcite @xcite . what has much less often been discussed in the context of phylogenetic methods, however, are the first two stages of the workflow, that is, the stage of concept list compilation, and the stage of concept translation. while it has been clear for a long time that different concept lists often yield different phylogenies @xcite , a closer discussion regarding the impact of concept lists on the results of phylogenetic analyses has not been carried out so far. the same holds for the translation of concepts into target languages. while @xcite found that concept translation across romance languages in two independently compiled multilingual wordlists differs by about 10%, and @xcite and @xcite could show that selecting but one out of several translations for the same concept in the same language can have direct consequences on the re-sulting phylogenies, no closer investigation regarding the degree of variation in concept translation or the impact of concept translation on phylogenetic analyses has been conducted up to now. ## discussion and conclusion our results confirm the findings of @xcite , who emphasize that concept translation may cause larger discrepancies between wordlists compiled independently by different scholars. while geisler and list (2010) report differences of about 10% for the romance language partition in the two indo-european datasets they compared, our results, taking data from 9 different language families into account, show that the differences are typically even larger, with only 83% of concept translations reflecting the same underlying word forms on average. figure 2 also demonstrates the magnitude of variation which occurs in concept translation between individual language varieties, also within the same language family, with some varieties such as russian achieving the exceptionally high similarity score of 0.99 and others like jeju scoring 0.65. many factors of course impact this variation, not least the increased standardization present in certain language varieties we examined, but also perhaps sociolinguistic factors such as dialect contact, semantic extension and cultural salience, which is beyond the scope of our current study. it is also worth noting that certain language families such as tupian also contain many unknowns, it is therefore expected to have a certain level of disparity in these datasets. while we have not tested the impact that differences in concept translation may have on phylogenetic reconstruction, we consider our results robust enough to call for the attention of scholars who use phylogenetic methods to answer big picture questions about human prehistory. although we think it is far too early to discard phylogenetic methods, we think that it would be useful for phylogenetic approaches to take potential problems resulting from the concept translation stage during the wordlist compilation seriously and make sure to apply certain measures to increase the robustness of their inferences. these measures could include tests on sampling errors, as outlined in @xcite , involve additional tests on inter-annotator agreement @xcite during concept translation, or entail conducting robustness tests similar to the bootstrap in phylogenetic reconstruction. in any case, the declared goal of phylogenetic approaches should be the same as for traditional historical linguistics. @xcite suggests that an ideal test of the reliability of the traditional comparative method for linguistic reconstruction would be to "take two teams of researchers trained in the comparative method, put them in the libraries, keep them in isolation from each other and see what they come up with" @xcite . while it is clear that such experiments are still lacking in traditional historical linguistics, it seems important that scholars working in the field of historical language comparison, no matter if they work computationally or manually, maintain a mindset that does not take the reliability of their data for granted. as far as our own experiments are concerned, we are still in the early stages of this research. additional experiments -potentially even including additional data from language families that do not feature in the sample presented here -will be needed to get a better understanding of the potential impact of concept translation on phylogenetic analysis. it is possible that phylogenetic methods turn out to be robust enough to yield similar results in terms of subgrouping and divergence time estimates, even if the wordlists that they employed show a certain amount of differences in the translations. without detailed analyses, however, we cannot be sure, and should not exclude the possibility that concept translation has a direct impact on the results of phylogenetic reconstruction analyses. in addition to phylogenetic reconstruction, it would also be important to test to which degree concept translation might influence the results of other studies that make use of multilingual wordlists. among these, the most important candidates that we can identify are global studies on sound symbolism or similar phenomena @xcite , as well as studies that make use of cross-linguistic colexification data @xcite @xcite . recently introduced methods for language affiliation without phylogenetic reconstruction @xcite ) may also suffer from variation in concept translation. in any case, the last word on the robustness of multilingual wordlists has not yet been spoken, and more tests are needed to understand the full implications of the findings that we reported in this study.