Understanding Translationese in Multi-view Embedding Spaces (2020)

Abstract: Recent studies use a combination of lexical and syntactic features to show that footprints of the source language remain visible in translations, to the extent that it is possible to predict the original source language from the translation. In this paper, we focus on embedding-based semantic spaces, exploiting departures from isomorphism between spaces built from original target language and translations into this target language to predict relations between languages in an unsupervised way. We use different views of the data — words, parts of speech, semantic tags and synsets — to track translationese. Our analysis shows that (i) semantic distances between original target language and translations into this target language can be detected using the notion of isomorphism, (ii) language family ties with characteristics similar to linguistically motivated phylogenetic trees can be inferred from the distances and (iii) with delexicalised embeddings exhibiting source-language interference most significantly, other levels of abstraction display the same tendency, indicating the lexicalised results to be not “just” due to possible topic differences between original and translated texts. To the best of our knowledge, this is the first time departures from isomorphism between embedding spaces are used to track translationese.

## introduction the term "translationese" refers to systematic differences between translations and text originally authored in the target language of the translation, in the same genre and style @xcite @xcite . characteristics such as simplification, over-adherence to conventions of the target language, and explicitation can occur as a communicative process itself. this is contrasted with "interference" or "shining-through" @xcite , described as "phenomena pertaining to the make-up of the source text tend to be transferred to the target text" @xcite . prominent evidence for shining-through as a translationese effect is found in the work of @xcite , who show that footprints of the source language remain visible in translations, to the extent that it is possible to predict the original source language from the translation. in the similar vein, a significant amount of work has gone into training classifiers to distinguish between translations and originally authored text and then investigating the contributions of individual features to the result of the classification @xcite @xcite . features that contribute strongly to classification are interpreted as indicating important dimensions of translationese. in contrast, in this work, we leverage departures from isomorphism between embedding-based semantic spaces to detect translationese. we construct embedding spaces from original english (o) data and translations into english (t ) from comparable data in a number of languages. we hypothesize that the closer the source language is to english, the more isomorphic the embedding spaces are. in other words, departure from isomorphism is an indicator of language distance. we use eigenvector similarity @xcite to quantify departure from isomorphism. if our hypothesis is correct, we should be able to reconstruct phylogentic trees from measures of departure from isomorphism. we show that this is indeed the case and compare our embedding-based results with previous results of reconstructing phylogenetic trees @xcite . in order to show that our outcomes are not the result of topic divergences between o and t data, we explore delexicalised views foot_1 of our data, using embeddings based on parts of speech (pos), semantic tags (st), and synsets (ss), rather than word tokens (raw). we show that our results are robust under delexicalisation. our paper is structured as follows: section 2 discusses related work and inference of phylogenetic trees. section 3 introduces our experimental setup. the distance measure is described in section 4. we present our results and analysis in section 5, followed by conclusions in section 6. ## phylogenetics and shining-through historical comparative linguistics determines genetic relationships between languages using concept lists of words that share a common origin, similar meaning and pronunciation across multiple languages @xcite . by contrast, computational analysis methods aim to reconstruct language phylogeny based on measurable linguistic patterns @xcite . @xcite showed that source language interference is visible in translation. specifically, they leverage interference (pos trigrams and function words) and translation universal features (cohesive markers) to construct phylogenetic trees. agglomerative clustering with variance minimisation @xcite is used as linkage procedure to cluster the data. the result is compared to the pruned gold tree of @xcite (henceforth referred to as sp08) used as the linguistic phylogenetic gold standard tree. their comparison metric, which is based on the l2 norm, is basically the sum of squared deviations between each pair's gold-tree distance g and computed distance p : sp08 was constructed by computing the levenshtein (edit) distance between words from an open cross-lingual list @xcite to compare linguistic divergence through time and thus partially encodes lexical similarity of languages @xcite . @xcite also acknowledges that sp08 has been disputed and researchers have not yet agreed on a commonly accepted tree of the indo-european languages @xcite . more recently, @xcite built on this work and compared different languages based on distance metrics computed on phrase structure trees and dependency relations. they claimed that such language representations correlate better with structural family distances between languages than genetic similarities. these examples show that phylogenetic reconstruction approaches and in particular, the evaluation of generated trees remains a highly debated topic in the history of linguistics and is beyond the scope of this study. ## experimental settings data. we use the comparable portion of europarl @xcite with translations from 21 european union languages into english to minimise the impact of domain difference. the amount of tokens per language varies, ranging from 67 k tokens for maltese to 7.2 m for german. we refer to the multiple translations into english as l j 's, where @xmath0, 2, ..., n; and to originally written text in english as l e . we select the subset of translations from 16 languages covering three language families: romance (french, italian, spanish, romanian, portuguese), germanic (dutch, german, swedish, danish) and balto-slavic (latvian, lithuanian, czech, slovak, slovenian, polish and bulgarian) into english and english original text. abstractions. in addition to using raw word tokens, we create multiple views of the data at the morphological (pos), lexical semantic (st) and conceptual-semantic (ss) levels. we use the spacy tagger @xcite with the ontonotes 5 version @xcite of the penn treebank pos tag set. for st @xcite , we use the best model of brants for ss, we follow @xcite to retrieve synsets according to the pos of a token using the knowledge base of wordnet @xcite . we only select a subset of pos tags, namely nn, adv, adj and vb and consider the first synset for each word/tag combination. vector spaces. for each view, we induce a separate monolingual word embedding space (both l e and l j 's) by treating each token or tag as a word using fasttext @xcite . embeddings have 300 dimensions and only words with more than 5 occurrences are retained for training. we use skip-gram with negative sampling @xcite and standard hyper-parameters. ## measuring isomorphism an empirical measure of semantic proximity between two languages is often computed using the degree of isomorphism, that is, how similar the structures of two languages are in topological space @xcite . research in cross-lingual transfer tasks shows that linguistic differences across languages often make spaces depart from isomorphism @xcite @xcite . while this degrades the quality of bilingual embeddings, it is a desired characteristic in our case: since our task involves processing of (multi-view) representations of monolingual text, departures from isomorphism indicate diversity in the source that generates them. to quantify isomorphism, we compute embeddings on a corpus in language l. embeddings reflect distributional properties in the data: words in similar contexts have similar meanings @xcite and should be close in embedding space. we then view the points representing words or tags in the resulting hyperdimensional embedding space as a graph and compare different spaces (e.g. for data from different languages, or originals and translations into the language of the originals) in terms of how similar or dissimilar the corresponding graphs are. this is measured in terms of a well established metric, the eigenvector similarity. eigenvector similiarity (ev). @xcite proposed this spectral metric based on laplacian eigenvalues @xcite to estimate the extent to which nearest neighbor graphs are isomorphic. we use the same idea to model differences between two spaces: original x and translationese y for the single target language translations from different source languages. first, we compute the nearest neighbour graphs g i of the two embedding spaces for the most frequent overlapping vocabulary. 2 we then compute the eigenvector similarity of the laplacians of the nearest neighbor graphs, l(x ) and l(y) in original and translationese respectively. the degree of graph similarity is given by the distance among the eigenvectors λ of the laplacian of g: following @xcite , we find the smallest k 1 in equation 2 such that the sum of its k 1 largest eigenvalues k 1 @xmath1: correlations between similarities (spo8 and ev) on the different views of linguistic representations (the higher the better). and set @xmath2, k 2 ). the graph similarity metric returns a value in the half-open interval [0, ∞), where values (∆) closer to zero indicate more isomorphic embedding spaces. to control the impact of data size for different l j 's, we choose the size of common overlapping vocabulary list corresponding to the range of the most resource-poor language in each view (see last column of ## results and analysis to analyse the computed distances between original english and english target translations on different levels of linguistic analysis, in a first step, we calculate the sum of the normalised ev scores per language family (i.e., germanic, romance, balto-slavic) shown in footprints of the source language into the translation product allow us to construct phylogenetic trees. figure 1 shows the result of clustering the distance scores using agglomerative clustering with variance minimisation @xcite for four views considered in this study. consistent results across all the trees demonstrate strong presence of the shining-through. we identify some of the well known language-language relationships in all four trees, such as for example, english is grouped with germanic and romance languages while balto-slavic languages are always put together. some interesting divergences, with respect to balkan sprachbund (bs) are visible as well. the geographical location of romanian opens it to cross-pollination with the other languages of the bs area and the figures provide some evidence for that. ## conclusion we presented an investigation of translationese effects in a single language sourced from multiple translations based on the topology of their multi-view embedding spaces. we explored embedding spaces constructed from word level (raw), morphological (pos), lexical semantic (st) and conceptual-semantic (ss) views of the data. to the best of our knowledge, our study is the first to track translationese using isomorphism in semantic space. our translationese-based results can infer phylogenetic language family relations based on divergence from isomorphism between embedding spaces from translations and originally authored text. we find that language distances correlate with the divergence from isomorphism in embedding space. our analysis demonstrates that while all embedding views exhibit source-language interference, delexicalised embeddings do so most significantly. in turn, this allows us to conclude that the lexicalised results are not just due to possible topic differences between original and translated texts. unlike supervised lexicostatistic approaches relying on aligned multilingual cognate lists, our isomorphism analysis is unsupervised and still able to detect important language differences related to linguistic typology. in a sense, and compared to some previous approaches, departure from isomorphism in embedding spaces lets "the data speak more for itself". as future work, we intend to extend our experiments to capture geometric properties of the embedding features and work on isolated languages. since we see that spaces with less tags, i.e., a smaller vocabulary, are better predictor of genetic relationships, more thorough robustness tests on the quality of the embeddings that might have an effect in skewing the results will be applied as well.