Computational analysis of different translations: by professionals, students and machines (2023)

Abstract: In this work, we analyse different translated texts in terms of various text features. We compare two types of human translations, professional and students’, and machine translation outputs in terms of lexical and grammatical variety, sentence length,as well as frequencies of different POS tags and POS-trigrams. Our experimentsare carried out on parallel translations into three languages, Croatian, Finnish andRussian, all originating from the same source English texts. Our results indicatethat machine translations are closest to the source text, followed by student translations. Also, student translations are similar both to professional as well as to MT, sometimes even more to MT. Furthermore, we identify sets of features which are convenient for distinguishing machine from human translations.

## introduction it is well-known that there is generally more than one way to translate any given source text (segment) and that versions created by different human translators therefore can vary from each other. variation between different translators has been observed in terms of various linguistic features from lexis to syntax (see section 2). as there is usually no single correct translation, these diverging versions may be equally good despite their differences. on the other hand, it has also been observed that machine translations differ from human translations in ways that might contain errors. distinguishing genuine variation in choices of lexical or grammatical expressions from the kind of divergence that indicates actual errors or other quality issues would be important for example for machine translatio evaluation. separating these two would require clearer understanding of how diverging translation versions in fact differ from each other. so, we analyse differences between texts translated by mt systems and those translated by two groups of human translators: professionals and students. although previous studies (see section 2 below) already compared such translation variants, they focused on one language pair and different genres, and did not consider neural mt. we also want to compare translations with their sources, as close resemblance to the source text could indicate more literal translations, which may be less than optimal in terms of fluency and style, even if the meaning is correct. besides that, we investigate in which aspects in terms of linguistic features translations resemble each other. thus, the main goals of this work are: rg1 to re-examine linguistic features from previous work on a parallel data set and three target languages; rg3 to further explore linguistic features in terms of distinctiveness for every translation variant under analysis. ## related work from the existing studies on human translation @xcite @xcite , we know that translated texts differ from non-translated ones in terms of linguistic features called translationese @xcite and it is possible to tease apart translated and non-translated texts automatically. moreover, we know that translationese can be influenced by various factors driven by the variation in human translation @xcite @xcite , including translator's background @xcite @xcite . however, we also know that while texts translated by various translator groups may vary in terms of lexical choices @xcite or morphosyntactic constructions @xcite @xcite , they may also converge, as it was shown by corpas @xcite popović (2020) showed that the observed variation in human translation is important for mt evaluation, especially when machine-translated outputs are compared against the available human translations. translations by certain groups of translators seem to be more similar with machinetranslated outputs, which has an impact on the evaluation result: those machine-translated outputs are rated higher. thus, the main outcome of this study was that when evaluating machine translation, it is important to know which human translation variety is being used. however, the translation data used in the corpus had different sources and not all of them were originally written in english. besides that, there was more variation in the analysed translator groups. machine translations were compared to human translations in a number of studies to either automatically differentiate between humans and machines or to evaluate specific linguistic phenomena @xcite @xcite . in a few studies, machine-translated outputs were also compared to human language production by different user groups, e.g. student translators (see the study by @xcite ). however, the analysed automatic translations contained no neural machine translations. in our work, we will focus on the differences between machine translation outputs and two types of human translations, i.e. professional and student. we will compare them in terms of lexicogrammatical features following the previous work on human and machine translation. in contrast to popović (2020), we will use a balanced parallel data set consisting of the same source texts for all translations and the same groups of translators per language. our analysis will also include stateof-the-art neural machine translations, by contrast to studies by lapshinova-koltunski (2015) and @xcite . ## data we use the publicly available corpus di-hutra foot_0 @xcite which contains english source texts and their translations into three languages produced by two groups of translators: several professional translators and several students foot_1 . we select the subcorpus of the amazon product reviews, which contains 196 texts (balanced as fourteen texts per fourteen various topics). the corpus contains six translation variants for each source review -two (professional and student) translations per three languages -croatian, russian and finnish. we add machine-translated outputs to each language pair. for translations into croatian, we used the best ranked output by human evaluation from the wmt 2022 shared task 3 @xcite . for the other two target languages, there were no recent publicly available mt outputs. we used the open source system google translate 4 to produce machine translations into russian. the finnish mt versions were produced using opus-mt @xcite ) pre-trained model @xcite . all the parallel texts in the corpus were annotated with universal pos as well as universal dependencies with the help of the stanford nlp python library stanza (v1.2.1). 5 we use these an-notations for the extraction of the linguistic features described in section 4 below. ## linguistic features the choice of features is based on findings reported in popović (2020) -we selected those indicating differences between students and professional translations. although they are also motivated by the theoretical categories of simplification @xcite and interference @xcite , they do not represent any of these categories exclusively. punctuation marks were separated and counted as words. the features are defined and calculated as follows. sentence length number of words in each sentence of the text. some translators might tend to generate longer sentences in the target text than others. some translators might keep the number of words in the translated sentences closer to the number of source text words than others. mt outputs might have different sentence lengths than human translations. mt systems might keep the number of words in the translated text closer to the number of source text words than human translators. ## analysis using the previously described features, we performed the following experiments: ## summary we present the results of computational analyses on different types of translated texts: professional, student and machine translations. the experiments were carried out on three language pairs. the main contributions of the work are insights into the differences between texts translated by different translator groups including neural machine translation, as well as identifying the most distinctive features. our observations for the three language pairs under analysis are similar to the existing analyses of english-german translations @xcite , where the author stated that students translations seem to be more similar with statistical and rulebased machine-translated texts. however, in our study, we analyse neural machine translation and a different type of features. besides that, we compare all translations to the original sources and find out that machine translations seem to be the most literal ones in terms of structural patterns (pos trigrams and dependency features), they keep the structure of the source text more frequently than the other translation variants under analysis. also, the two mt outputs available for croatian are very similar, more than any other pair of texts. comparing professional and student translations, we find that student translations are more literal and therefore similar to the sources than the professional translations, being placed in between, sometimes even more similar to mt outputs than to professional translations. moreover, a set of distinctive features was identified for machine and for human translations. lexical variety is distinctive for human translations, while all other varieties and sentence length are distinctive for machine translations. interestingly, pos tags and pos-trigrams are also different for machine translations than for human translations. in addition, pos-trigrams are more convenient for detecting human translations, whereas pos tags suit better for identifying machine translations. future work is planned to better understand these differences in terms of more complex properties, such as sentiment, tone, etc. also, automatic mt scores using different human translations will be explored in detail.