Machine-translated texts from E nglish to P olish show a potential for typological explanations in Source Language Identification (2023)

Abstract: This work examines a case study that investigates (1) the achievability of extracting typological features from Polish texts, and (2) their contrastive power to discriminate between machine-translated texts from English. The findings indicate potential for a proposed method that deals with the explainable prediction of the source language of translated texts.

## introduction in the modern-day world of global interconnectedness, the act of translation has evolved into an indispensable part of daily life as a result of the growing availability of ever more advanced translation engines @xcite . this trend has been further amplified by the increasing accessibility of such tools; e.g., through their integration into messaging services or social media platforms @xcite ; @xcite ). the recently acquired prominent position of translation tools in human society brings attention to the value of comparatively un(der)explored areas in machine translation (mt) that are more ethical in nature. found within this space is the task of determining the source language of a machine-translated text, also referred to as source language identification foot_0 (sli). this task may not only contribute to the qualitative improvement of translation engines, it further has a practical application in the field of forensics-bad actors use translation tools too. the viability of sli relies on the premise that, despite considerable progress in mt as a result of 'the transformer revolution' @xcite , machine translations may still be imperfect or unnatural: they may contain artifacts that indicate that a text originated in a different language, even when being grammatically and semantically sound. @xcite coined the term 'translationese' to refer to this phenomenon that is now widely recognised in the literature. translationese may arise from various causes, with the characteristics of the source language (i.e., source language interference) playing a prominent role @xcite . an understanding of the features leading to such perceived linguistic unnaturalness might therefore allow the source language to be identified from translated text alone. we propose a methodology that exploits these symptoms of translationese to approach the task of sli. although research on translationese has mostly concerned human translation, an emerging line of work specifically focuses on mt, with the majority of effort in this area dealing with evaluating mt models (e.g., @xcite ; @xcite ). while some authors presuppose that translationese across humans and machines is similar @xcite , empirical evidence suggests that structural properties of the source language (i.e., grammar) 'shine through' more overtly in text translated by machines @xcite . as mt is ultimately trained on human translation, of particular interest is therefore how structural features of human translationese can be identified. a common strategy for this involves training a classifier to leverage surface features indicative of structural translationese: surface features comprise easily observable attributes of texts, such as parts of speech (pos). particularly the latter have demonstrated encouraging performance in the form of n-grams @xcite ; @xcite , @xcite ). drawing on these results, we employ such features in our method. while sli remains largely unexplored, many have studied its equivalent in (human) secondlanguage acquisition: native language identification (nli 2 ). this field was initially researched by @xcite and gained momentum through a shared task @xcite . the inherent reliance of both nli and sli on leveraging features of language interference makes the former a highly relevant field. a shortcoming of many nli approaches is their lack of explainability @xcite ; a quality that is naturally demanded by the field of forensics @xcite , and a quality that could provide useful insight into the limitations of current approaches in mt. as we exploit the structural properties of a source language that hint at the origin through its artifacts, correspondingly, explanations ought to be in terms of the structural differences between the source and target language of a translated text. we aim to achieve this by reformulating sli as a typological feature prediction task. such features are the products of the field of linguistic typology and serve to distinguish between the structural properties of languages @xcite . consequently, they have the capacity to provide human-interpretable explanations that are linguistically grounded. @xcite show that the typology of native languages are predictable within the context of nli, providing further ground to our approach. in a paper published in the field of law, @xcite similarly advocate the need for typology-based explanations in sli-like contexts, indicating a convergence of ideas. our paper contributes by presenting a practical implementation, while also adding to its theoretical foundation. to further underline the potential of a typological approach to achieve explainable sli, we present a case study that examined the feasibility of extracting typological features from polish texts, and their capacity to discriminate between translated texts from english. the slavic language family poses an interesting testbed for such analyses, as it exhibits unique features in contrast to english ( §2.1), while still being in relative linguistic proximity. our preliminary experiments indicate that structural features reminiscent of the origin language display significant promise for typology prediction to warrant further research that implements the methodology proposed in this work. we are currently examining the effectiveness of our method in practice. in the following section, we provide the aforementioned experiments. the paper then proceeds by explicating our proposed methodology. it concludes by discussing the findings and limitations. ## experimenting with polish and english to gauge the exploitability of features specific to the polish language, we conducted two experiments. the first experiment analyses polish word order to gain an intuition on the practical utility of the features. experiment 2 then compares the applicability of all features listed in the following subsection. all code relating to the experiments and the scraping of the data can be found on github. foot_2 ## a methodology for explainable sli we now put forth a two-part methodology that places an intermediary map to typological features in between the definitional map of sli from translated text to source language. this effectively elevates the problem of sli to a 'typology predictionlike' task that is unique in that it aims not to grasp the typology of the language in which a translated text is given, but rather to identify the typological features of the origin language of the text. the first mapping, from translated text to a prediction of the typological features of its source language, may be realised by an appeal to surface features. these lend themselves well for verifying whether the model exploiting these surface features has truly 'learned' to identify the artifacts of the typology characteristic to the source language. moreover, surface features show promise for typology prediction in the first place, as established in the previous sections. the incentive to identify source features instead of source languages is motivated by the fact that the second mapping, from predicted typological features to a set of possible languages carrying these features, subsequently becomes a more trivial component in the pipeline that can be addressed by traditionally interpretable models, such as svms. the typological features then become the 'building blocks' of the explanations for predictions. as brought up in the introduction, the choice for typological features is justified by their tendency to capture the structural elements of a source language, which are especially pronounced in machine translationese. as an additional consequence, they tend to be more robust than, e.g., lexical features, for language change manifests slowest in the core structural elements of a language (trudgill, 2020, p. 1)-precisely those grasped by highlevel typological descriptions. for example, word order features are "diachronically stable" @xcite . we therefore hypothesise that sli approaches based in typology perform more consistently across mt models, linguistic (sub)communities, and genres. moreover, typological features naturally reflect phylogenetic relationships between languages @xcite ; this paves the way for 'fuzzy' classifications that align with the historical development of languages along geographical lines, as predictions may not necessarily be restricted to a single source language, but (branches of) a language family instead. they have the additional capacity to transcend these genealogical boundaries where overlapping typology challenges traditional linguistic classification, as is, e.g., the case with ukrainian in relation to russian and polish @xcite . this and the previous implication may especially prove beneficial in forensic contexts. lastly, an approach that is rooted in a robust body of linguistic research offers a ground for verification of the internal reasoning of a resulting model. it moreover keeps open a dialogue between linguistics and ai: developments in linguistic typology may inform work in sli, and possibly vice-versa. the world atlas of language structures online (wals) appears to be a natural fit as a basis from which to draw the reasoning underlying the prediction of source languages and the corresponding explanations; it is a rich, freely available resource of typological features in a table-like format for over 2,000 languages @xcite . a diagram of the method is given in figure 2 . ## discussion and conclusion discussion a perceived limitation of the method stems from the presumption that typology prediction is more challenging than sli, as this could harm the performance of a model that implements the suggested methodology. however, we argue that this is only of secondary concern to a work that primarily focuses on explainability. an aim for trustworthy explanations requires the internal reasoning of a model to align with the reasoning conveyed in explanations for the model's behaviour. our method thus needs to incorporate human-understandable concepts (e.g., typological features) that are potentially less sophisticated than those developed by more 'naive', 'black-box' methods. furthermore, although introducing typology likely complicates the task in the general case, the complexity may be reduced in a multilingual setting, for the ability to predict a fixed set of typological features provides access to a wide prediction the experimental results in section 2 indicate that features specifically designed to accommodate known differences within a certain language pair may be fruitfully used for the methodology proposed in section 3. however, the method is likely to be limited by its reliance on wals, which contains much more generally described features than those introduced in our case study @xcite . for example, polish is classified as an 'adj-noun language' (feature 87a), placing english and polish in the same category, while, clearly, the latter language is more permitting in its word order for adjectives and nouns, as was also observed in our case study. the broad nature of wals may limit the ability of the method to exploit surface features in the way that was manually done in experiment 2. moreover, it may pose an additional challenge to define a subset of wals features that is relevant for pointing to the source language of (small) texts in the first place. @xcite come to a similar conclusion about the usefulness of wals for this task. their 2020 paper puts forward a framework for pro-viding different types of explanations for sli-like tasks, with those informed by typology comprising only one tier among the multiple levels elaborated on by the authors, which ultimately lessens the effect of this issue on the overall picture, in which our methodology takes on only a part of the solution. in conclusion, while it is impossible to make hard assertions, the experimental findings indicate promising potential for further development of the proposed methodology. a natural progression of this work is to implement the method and to qualitatively analyse its performance by evaluating it on language pairs including slavic languages. especially in light of the latter, future approaches may additionally consider more fine-grained differences between slavic languages, such as tendencies for nominal or verbal constructions between ukrainian, polish, and russian @xcite .