Using C oll G ram to Compare Formulaic Language in Human and Machine Translation (2021)

Abstract: A comparison of formulaic sequences in human and neural machine translation of quality newspaper articles shows that neural machine translations contain less lower-frequency, but strongly-associated formulaic sequences (FSs), and more high-frequency FSs. These observations can be related to the differences between second language learners of various levels and between translated and untranslated texts. The comparison between the neural machine translation systems indicates that some systems produce more FSs of both types than other systems.

## introduction neural machine translation (nmt) systems are currently considered to bridge the gap between human and machine translation @xcite . however, little research has been done to determine whether nmt systems are also very effective in processing multiword units @xcite , whereas the importance of preformed units in language use is now well established, including in foreign language learning and translation @xcite @xcite . the present study addresses this issue by comparing formulaic language in human and neural machine translation. it focuses on a specific category of multiword units, the "habitually occurring lexical combinations" @xcite , such as dramatic increase, depend on, out of, which are not necessarily semantically non-compositional, but are considered statistically typical of the language because they occur "with markedly high frequency, relative to the component words or alternative phrasings of the same expression" @xcite . these formulaic sequences (fss) are analyzed by means of a technique proposed by @xcite , improved by @xcite and automated by @xcite under the name collgram 1 . the author is a research associate of the fonds de la recherche scientifique -fnrs (fédération wallonie bruxelles de belgique). he would like to warmly thank sylviane granger and maïté dupont for access to the pleci corpus. 1 as one reviewer pointed out, this automation is not an "easily available plug-andplay implementation". however, there is a freely available system that implements it collgram rests on two lexical association indices that measure the strength of attraction between the words that compose a bigram, mutual information (mi) and t-score @xcite , calculated on the basis of the frequencies of occurrences in a reference corpus @xcite . these two indices are complementary, mi favoring lower-frequency, but strongly-associated, fss such as self-fulfilling prophecy, sparsely populated or sunnier climes while the t-score favors high-frequency bigrams such as you know, out of or more than. a series of studies have shown that, compared to native speakers, english as a foreign language learners tend to underuse collocations with high mi scores, while overusing those with high t-scores and that exactly the same differences are observed between advanced learners and intermediate learners @xcite . these observations are in agreement with usage-based models of language learning which "hold that a major determining force in the acquisition of formulas is the frequency of occurrence and co-occurrence of linguistic forms in the input" @xcite . it is worth noting that the same differences were observed between translated and untranslated texts, but the proposed explanation relies on a tendency towards normalization in translation @xcite . since neural models also seem to be affected by frequency of use @xcite , the hypothesis tested in the present study is that the same effects could be observed when comparing human translations (hts) and nmts, namely that nmts will underuse high mi fss and overuse high t-score fss. ## analyses and results as shown in these tables, both hypotheses are verified. compared to hts, texts translated by the three neural systems contain a significantly smaller percentage of highly collocational bigrams for the mi and a larger percentage of highly collocational bigrams for the t-score. cohen's ds are almost always medium or large and the percentages of texts for which differences are observed are greater than 70% except in one case. an analysis of the passages in which the differences between ht and nmt are the largest suggests that the origin lies at least partially in the less literal nature of human translations (see ## discussion and conclusion the reported analyses confirm the hypotheses and thus suggest that, compared to hts, nmts more closely resemble texts written by intermediate learners than by advanced learners of english as a foreign language, a result that could be interpreted in the context of a usage-based model of language learning @xcite . the nmts also resemble translated texts more than untranslated texts, but it is not clear that this can be explained by a normalization process. statistically significant differences, but smaller in terms of effect size, were also observed between the three nmt systems. it is important to keep in mind that the present study only considers global quantitative properties of mwus. at no point is the appropriateness in context of the mwus assessed. it is therefore a very partial approach. however, it has the advantage of not requiring a human qualitative evaluation that is often complicated and cumbersome to set up. moreover, it is likely that the appropriateness of a mwu is much more important for non-compositional expressions than for the habitually occurring lexical combinations studied here @xcite . another important feature of the approach is that it relies on a native reference corpus to identify highly collocational bigrams for both indices. as already mentioned, research on foreign language learning, but also on the comparison of translated and untranslated texts, has shown that the use of other large reference corpora such as coca or wacky @xcite did not change the results @xcite . one can also wonder whether the use of a comparable reference corpus, rather than a generic one, would have returned different results. in the case of the comparison of translated and untranslated texts, @xcite observed that the use of a journalistic corpus, the corpus est républicain (115 million words) made available by the centre national de ressources textuelles et lexicales, produced differences similar to those obtained with the wacky corpus. before considering taking advantage of these observations to try to improve nmt systems, a series of complementary analyses must be conducted. indeed, this study has many limitations, such as focusing only on a subcategory of mwus @xcite , on a single language pair, and on a single genre of texts. moreover, a thorough qualitative analysis is essential to better understand the results and evaluate the proposed explanations. as it has been shown in foreign language learning @xcite , it would also be interesting to verify that the observed effects are not explained by differences in single-word lexical richness. finally, the differences between the three nmt systems also require further analysis.