Machine Translation Believability (2021)

Abstract: Successful Machine Translation (MT) deployment requires understanding not only the intrinsic qualities of MT output, such as fluency and adequacy, but also user perceptions. Users who do not understand the source language respond to MT output based on their perception of the likelihood that the meaning of the MT output matches the meaning of the source text. We refer to this as believability. Output that is not believable may be off-putting to users, but believable MT output with incorrect meaning may mislead them. In this work, we study the relationship of believability to fluency and adequacy by applying traditional MT direct assessment protocols to annotate all three features on the output of neural MT systems. Quantitative analysis of these annotations shows that believability is closely related to but distinct from fluency, and initial qualitative analysis suggests that semantic features may account for the difference.

## introduction past work on evaluating machine translation (mt) has focused on the intrinsic quality of the translation product without taking into account how translations are perceived by their users. yet, some translation errors are more obvious than others, and have different consequences depending on what the translations are used for. in this work, we take a user-centered view of mt evaluation, exploring one aspect of users' perception of mt: believability of the output, defined as a monolingual user's perception of the likelihood that the meaning of the mt output matches the meaning of the input, without understanding the source. assessing the degree to which mt is believable acknowledges that users play an active role in interpreting its output, informed by their linguistic competence, their common sense reasoning abilities, and their knowledge of the world. what we learn from assessing believability can complement traditional evaluation methods to inform the deployment and even development of mt systems, particularly for gisting and communication use cases. we first define believability of mt and contextualize it within prior work on credibility and mt evaluation. we apply mt direct assessment (da) protocols to obtain human judgments of believability, annotating the output of neural machine translation (nmt) systems for three challenging language pairs (arabic-, farsi-, and korean-to-english) with varying translation quality. these annotations show that believability is closely related to, but distinct from fluency. preliminary qualitative analysis suggests that in addition to fluency features, believability is also influenced by semantic features. we define believability as a user's perception of the likelihood that the meaning of a given mt output matches the meaning of the input, without understanding the input. whether the user accepts the output unquestioningly or finds it unbelievable, their judgment will affect how they act on it, regardless of the true accuracy of the translation. for example, a facebook user might be dubious of a translation from chinese with the phrase "blowing a little more cow" and ask the author for clarification and learn that it is a literal translation of an idiom, "吹牛" (meaning to brag). users may take more consequential action, such as the israeli police officers who chose not to consult an arabic speaker before arresting a man based on a believable mistranslation of his "good morning" facebook post as "attack them" @xcite . to illustrate how believability can be independent of adequacy, because these judgments are based on perception, they may be more subjective than traditional mt da features. we control for some factors that may affect believability (section 3), resulting in annotations that are similarly reliable to the da features (section 4). ## related work although believability is an unexplored aspect of mt, there is prior work outside of mt on the general concept of credibility. common elements of credibility include source, media, and message credibility @xcite . for mt, we can think of the mt provider as the source, the interface through which the mt is viewed as the medium, and the output itself as the message or content. all of these aspects likely affect the credibility of deployed mt systems, but our investigation of mt believability is focused on the content (mt output). some intrinsic content features addressed in the credibility literature that may affect mt believability include reasonableness @xcite @xcite and grammatical errors @xcite @xcite @xcite . reasonableness is a semantic feature encompassing elements of plausibility, logic, and internal consistency. these elements are related to previously studied concepts in the computational linguistics literature: semantic plausibility, commonsense reasoning, and discourse coherence. semantic plau-sibility can be thought of as, "whether in an ordinary real-life situation (not "fairy-tale" circumstances) the sentence could be reasonably uttered" @xcite . if the source text is expected to reflect "ordinary real-life," the output should be plausible to be believable. mt output may also be unbelievable if it violates commonsense reasoning, a challenging element of natural language understanding @xcite . lack of discourse coherence might likewise signal unbelievable translations. for document generation, improving the consistency of generated documents makes it harder for human subjects to distinguish automatically generated text from real text @xcite . grammatical errors are related to fluency, a traditional mt quality evaluation feature. fluency has been defined as a judgment of "whether the translation reads like good english...without knowing the accuracy of the content," and is typically combined with adequacy, an assessment of "the degree to which the information in a professional translation can be found in an mt (or control) output of the same text" @xcite . a user who cannot understand the source cannot judge adequacy, but may use expectations based on features like fluency and reasonableness to guess. believability could thus be seen as predicted adequacy via a human cognitive process with inputs from surface features of the output such as fluency and semantic cues from context. two other computational linguistics concepts that relate to both reasonableness and grammatical errors may affect believability: acceptability and comprehensibility. in empirical linguistics, acceptability judgments measure users' linguistic competence @xcite . while acceptability is primarily used to observe grammatical knowledge, judgments are not limited to grammaticality in practice: "semantic plausibility, various types of processing difficulties, and so on, can individually or jointly cause grammaticality and acceptability to come apart" @xcite . these breakdowns can lead to issues with comprehensibility. popović (2020) cites comprehensibility as a key factor in misleadingness of mt output: if a user cannot understand the text, they cannot be misled by it. similarly, if they cannot understand it, the user is unlikely to believe that the translation is correct. ## annotating believability to understand the relationships between believability and traditional mt quality criteria (fluency and adequacy), we hired professionals to annotate mt output for these characteristics in tasks based on the fluency and adequacy da methods of @xcite and @xcite . the annotated data sets are available at: https://github.com/ mjmartindale/mt_believability . annotators our annotators were salaried translators with proficiency levels of at least advanced on the actfl scale (actfl, 2012) rather than mt researchers or crowd workers as in wmt @xcite . because they were not paid per item, they were willing to spend significant time on each item, averaging 15 items in 30 minutes. we believe this reflects more attention to detail as indicated by the correlation between annotators (see section 4). we note that factors such as foreign language proficiency may affect believability judgments. further work with a wider variety of annotators is needed to identify and quantify those effects. tasks we followed segment-level da scoring best practices established by wmt @xcite . the fluency and adequacy questions were taken directly from wmt16 @xcite . the believability question uses our definition of believability with an introductory phrase to assure the annotator that we understand that it is not possible to truly evaluate the meaning without the source: "even without having seen the source text, i believe the meaning of this translation is likely to match the meaning of the original." annotations were performed using the turkle 1 annotation platform. screenshots of the annotation interface are provided in appendix b. long documents were broken up into salient chunks and segments were annotated in their original order to provide discourse context, as in 1 https://github.com/hltcoe/turkle wmt19 "segment rating + document context" @xcite . for each chunk, annotators first scored fluency and believability based only on the mt output. they then scored the same segments for adequacy given both source and output. annotations for each segment, we calculate a z-score and a label for each feature. we calculate scores following @xcite . each annotator's raw scores are converted to z-scores based on their own mean and standard deviation, and the z-scores for each segment are averaged across annotators. segments with positive z-scores are labeled true and negative z-scores are labeled false. test data we chose a test set that is comparable across three typologically different languages with different amounts of training data. our test data comes from the multi-target ted talks task (mttt)-a collection of bitexts across 20 languages @xcite . the test set is fully sentence parallel with original talk transcripts as the english and human translations for the other languages. we use the non-english translations in mttt as "source" and machine translate into english. in the test set, there are 29 talks totalling 1,982 segments, however, we exclude one talk ("nellie mckay sings 'clonie"') that is too poetic for mt. the final set is 1,976 segments. ## quantitative analysis annotation statistics 63 translators participated in the annotation process. korean-to-english and arabic-to-english had the most annotators (26 and 27) and highest number of annotators per segment (10 for korean and at least 7 for arabic). there were three annotators whose annotations were deemed unreliable due to low correlation with the mean (< 0.5). only 10 annotators were available for farsi, resulting in fewer annotators per segment (median: 4) but more segments per annotator (median: 802), making the z-score process more reliable. after excluding the questionable annotators, we see strong correlation of individual annotator scores with the mean as shown in label distribution although the distribution of labels is specific to this set of output, it provides context for the other results. the first three rows of feature relationships as expected, fl-bl has the strongest correlation. this indicates that the two features are closely linked, but they are not identical. ## qualitative analysis based on informal examination of a random sample of segments, we find that those labeled as unbelievable often fail semantically, with strange phrases (e.g., "kidney steel", "iron code"), illogical clauses (e.g., "the only natural thing is that my son is the vending machine"), or unlikely argument structure (e.g., "to prove that it seems impossible", "...a state of treason for a raven, which explains that he's cute"). unbelievable translations may also be grammatical but unintelligible (e.g., "if you want a long time, i'm actually doing something about it."). by contrast, segments labeled as disfluent may include grammatical errors and/or awkward, non-idiomatic phrases such as "he set me a date" or "in a direct time". these observations support our intuition that believability is more influenced by semantic features than fluency is, but further analysis is needed. additional examples for each combination of fluency, believability, and adequacy are shown in ## conclusions and future work this work used traditional nlp annotation methods to measure users' perceptions of believability of mt output. these methods allow us to identify broad relationships between believability and traditional mt quality metrics, fluency and adequacy, showing that believability is strongly corre-lated with fluency and somewhat correlated with adequacy. preliminary qualitative analysis of examples where believability and fluency judgments disagreed suggests that semantic features can overwhelm grammatical features in believability judgments. a full qualitative analysis of the believabilityannotated examples would suggest features that may have influenced annotator's judgments and could indicate approaches that may be effective in automatically predicting believability. believability used alone could enable an adversarial mt system to deliberately mask errors and produce misleading output, but believability predictions combined with mt quality estimation @xcite could be used to flag potentially misleading output. because believability is a user-centric metric, gaining a complete understanding would require more user-centric methods. the annotator agreement in our results may indicate that believability is less subjective than one might expect, or it may simply indicate that our annotators were fairly homogeneous. a user study could not only tell us exactly what features were most salient in which contexts, but could indicate whether demographic features such as age or education affect perceptions of believability. figure 2: example question from bilingual annotation phase with adequacy question