Do GPT s Produce Less Literal Translations? (2023)

Abstract: Large Language Models (LLMs) such as GPT-3 have emerged as general-purpose language models capable of addressing many natural language generation or understanding tasks. On the task of Machine Translation (MT), multiple works have investigated few-shot prompting mechanisms to elicit better translations from LLMs. However, there has been relatively little investigation on how such translations differ qualitatively from the translations generated by standard Neural Machine Translation (NMT) models. In this work, we investigate these differences in terms of the literalness of translations produced by the two systems. Using literalness measures involving word alignment and monotonicity, we find that translations out of English (E-X) from GPTs tend to be less literal, while exhibiting similar or better scores on MT quality metrics. We demonstrate that this finding is borne out in human evaluations as well. We then show that these differences are especially pronounced when translating sentences that contain idiomatic expressions.

## introduction despite training only on a language-modeling objective, with no explicit supervision on aligned parallel data @xcite , llms such as gpt-3 or palm @xcite achieve close to state-of-the-art translation performance under few-shot prompting @xcite . work investigating the output of these models has noted that the gains in performance are not visible when using older surface-based metrics such as bleu @xcite , which typically show large losses against nmt systems. this raises a question: how do these llm translations differ qualitatively from those of traditional nmt systems? we explore this question using the property of translation literalness. machine translation systems have long been noted for their tendency to produce source he survived by the skin of his teeth . ## quantifying translation literalness we compare the state-of-the-art nmt systems against the most capable publicly-accessible gpt models (at the time of writing) across measures designed to capture differences in translation literalness. we conduct both automatic metric-based as well as human evaluations. we explain the evaluation and experimental details below. ## wmt20-comet-qe-da this can also be interpreted as (normalized) alignment crossings, which has been shown to correlate with translation non-literalness @xcite . we use the multilingual-bert-based awesomealigner @xcite to obtain the word to word alignments between the source and the translation. ## effects on figurative compositionality in this section, we explore whether the less literal nature of @xmath0. we posit the phenomenon of composing the non-compositional meanings of idioms @xcite with the meanings of the compositional constituents within a sentence as figurative compositionality. thereby, a model exhibiting greater figurative compositionality would be able to abstract the meaning of the idiomatic expression in the source sentence and express it in the target language non-literally, either through a non-literal (paraphrased) expression of the idiom's meaning or through an equivalent idiom in the target language. note that greater nonliteralness does not imply better figurative compositionality. non-literalness in a translation could potentially be generated by variations in translation different from the desired figurative translation. ## discussion in our experiments conducted across different nmt systems and gpt models, we find evidence that gpts produce translations with greater nonliteralness for @xmath1. there could be a number of potential causes for this; we list two plausible hypotheses below: parallel data bias nmt models are trained on parallel data, which often contains very literal webcollected outputs. some of this may even be the output of previous-generation mt systems, which is highly adopted and hard to detect. in addition, even high quality target text in parallel data always contains artifacts that distinguishes it from text originally written in that language, i.e. the 'translationese' effect @xcite . these factors could likely contribute to making nmt translations comparatively more literal. language modeling bias translation capability in gpts arises in the absence of any explicit supervision for the task during the pre-training stage. therefore, the computational mechanism that gpts leverage for producing translations might be different from nmt models, imparting them greater abstractive abilities. this could have some measurable manifestation in the translations produced, e.g., in the literalness of the translations. differences in @xmath2, we consistently find that gpt translations of similar quality are less literal and in the @xmath3, we observe a few anomalies. for @xmath4, in figure 1 , in all but one comparison (wmt-21-sota vs gpts for de-en) gpts obtain higher measures for non-literalness. on the other hand, we did not see anomalies in the trend for @xmath5. ## summary and conclusion we investigated how the translations obtained through llms from the gpt family are qualitatively different by quantifying the property of translation literalness. we find that for @xmath6, there is a greater tendency towards nonliteralness in gpt translations. in particular, this tendency becomes evident in gpt systems' ability to figuratively translate idioms.