GTNC : A Many-To-One Dataset of G oogle Translations from N ews C rawl (2024)

Abstract: This paper lays the groundwork for initiating research into Source Language Identification; the task of identifying the original language of a machine-translated text. We contribute a dataset of translations from a typologically diverse spectrum of languages into English and use it to set initial baselines for this novel task.

## introduction in an era of globalisation, the world is becoming increasingly reliant on machine translation. but as translation tools find their way into people's daily routines, they spark curiosity about previously unexplored tasks, such as identifying the source language of a machine-translated text. this is an emerging challenge that has been referred to as source language identification @xcite . the task has a relevant application in forensics: knowledge of an individual's native language can offer crucial insights into their identity. the problem of classifying the original language of a machine-translated text inherently relies on finding markers in the translation that hint at the source (i.e., traces of 'source language interference'). in a first exploration of the field, reijnaers and herrewijnen (2023) indicated that such markers can be related to typological differences between the languages involved in the translation process, aligning with theory on human translation @xcite . typological features contribute to the explainability of sli models @xcite , a quality essential in forensic contexts @xcite . however, owing to the novelty of the task, research on sli is hindered by a lack of sufficiently sized datasets that contain machine translations from a large number of languages into a single language. this work aims to fill this gap to propel this emerging area of research forward. we introduce google translations from newscrawl (gtnc): a unique dataset of state-of-the-art machine translations from a diverse set of languages into english, offering a rich typological diversity to facilitate experiments with a wide range of source languages. the dataset spans 50 languages (listed below), contains 7,500 sentences per language, and is representative of real-world data given its domain (news articles) and the translation engine used (google translate). in addition, we offer initial baselines for future work on sli and thereby confirm the feasibility of the task. the next section of this paper will discuss existing datasets that may be used for sli. in addressing their limitations, we propose a novel dataset in section 3, which we will then use in a series of experiments in the section that follows. the findings reiterate the value of a typological approach in sli. included languages amharic, arabic, bengali, bulgarian, chinese, croatian, czech, dutch, english (untranslated), estonian, finnish, french, german, greek, gujarati, hausa, hindi, hungarian, icelandic, igbo, indonesian, italian, japanese, kannada, korean, kyrgyz, latvian, lithuanian, macedonian, malayalam, marathi, odia, oromo, pashto, persian, polish, portuguese, punjabi, romanian, russian, shona, spanish, swahili, tagalog, tamil, telugu, tigrinya, turkish, ukrainian, and yoruba. ## existing datasets in the realm of human translation, several corpora exist that contain translations from multiple languages into a single language, among which the most popular is a collection of proceedings of the european parliament (europarl, @xcite . numerous studies have leveraged this corpus to provide empirical evidence for distinctions between original and translated texts @xcite @xcite , while some have explicitly aimed to identify the european source language of these documents @xcite . however, machine translations are divergent from human translations in a systematic @xcite and measurable @xcite way: machine translations often exhibit less morphological and lexical diversity @xcite and adhere more closely to the structure of the source text @xcite . moreover, machine translations are more susceptible to source language interference (toral, 2019, p. 279), particularly concerning the structural properties of the source language @xcite . as such, a dataset of purely machine translations is desirable. a handful of datasets exist that contain machine translations from multiple languages into one. an example is demetr @xcite , consisting of translations from ten, predominantly indo-european languages 1 into english. the dataset was constructed to aid models in detecting errors in machine translation output. as a result, a downside is that the authors post-edited the translations to ensure their correctness, thereby potentially eliminating valuable hints that pointed to the source language of these texts. demetr is also modest in size, comprising only 100 sentences per language. another example is mlqe-pe @xcite , containing the translations of 9,000 sentences for each of five, diverse indo-european languages 2 into english. apart from the small number of classes, a drawback of this dataset is that the samples vary widely in length across languages (figure 2a ) and are often noisy (e.g., containing urls or html tags). this could potentially bias sli models towards relying on spurious features instead of learning linguistic patterns purely governed by typology. a comparison of the key features mentioned above can be found in in the subsections below, we will describe the steps taken to build gtnc and will provide analyses into its diversity and characteristics. the data and all code used to generate them is available on github. ## preliminary experiments in this section, we will use the english translations from gtnc to predict the source language of both individual samples and combinations of sentences. ## conclusion and discussion we showcased gtnc: a thoughtfully designed dataset of google-translated news articles from diverse languages into english. our experiments provide compelling evidence attesting to the feasibility of sli and emphasise the dataset's suitability for typological approaches-a quality that holds significant promise on the path to explainable sli. as our goal was to introduce a dataset, we deliberately avoided a lengthy discussion on the underlying phenomena that enable the identification of a source language in the first place; i.e., a more in-depth analysis of how 'artifacts' of the typology of the original language are left behind in a translation. an exploration of the involved scientific concepts, particularly 'translationese' @xcite and 'interlanguage' @xcite , and how they relate to machine translation, would demand a thorough examination that is beyond the scope of this short paper. while gtnc encompasses a wide array of languages, the number of samples per language re-mains limited. we encourage the community to improve our dataset using the tools that we have made available. beyond sli, the data may also help other applications, such as evaluating google translate's performance across languages. we hope that gtnc will additionally foster exploration in new directions. larry selinker. 1972. interlanguage. 10(1-4):209-232. serva, m. and petroni, f. 2008. indo-european languages tree by levenshtein distance. epl, 81(6):68005. elke teich. 2003. cross-linguistic variation in system and text. de gruyter mouton, berlin, boston. antonio toral. 2019. post-editese: an exacerbated translationese. in proceedings of machine translation summit xvii: research track, pages 273-281, dublin, ireland. european association for machine translation. tobias van der werff, rik van noord, and antonio toral. 2022. automatic discrimination of human and neural machine translation: a study with multiple pretrained models and longer context. in proceedings of the 23rd annual conference of the european association for machine translation, pages 161-170, ghent, belgium. european association for machine translation. hans van halteren. 2008. source language markers in europarl translations. in proceedings of the 22nd international conference on computational linguistics (coling 2008), pages 937-944, manchester, uk. coling 2008 organizing committee. eva vanmassenhove, dimitar shterionov, and matthew gwilliam. 2021. machine translationese: effects of algorithmic bias on linguistic complexity in machine translation. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 2203-2213, online. association for computational linguistics. vered volansky, noam ordan, and shuly wintner. 2013. on the features of translationese. digital scholarship in the humanities, 30(1):98-118. joe h. ward. 1963. hierarchical grouping to optimize an objective function. journal of the american statistical association, 58(301):236-244.