Comprehension and Trust in Crises: Investigating the Impact of Machine Translation and Post-Editing (2020)

Abstract: We conducted a survey to understand the impact of machine translation and post-editing awareness on comprehension of and trust in messages disseminated to prepare the public for a weather-related crisis, i.e. flooding. The translation direction was English–Italian. Sixty-one participants—all native Italian speakers with different English proficiency levels—answered our survey. Each participant read and evaluated between three and six crisis messages using ratings and open-ended questions on comprehensibility and trust. The messages were in English and Italian. All the Italian messages had been machine translated and post-edited. Nevertheless, participants were told that only half had been post-edited, so that we could test the impact of post-editing awareness. We could not draw firm conclusions when comparing the scores for trust and comprehensibility assigned to the three types of messages—English, post-edits, and purported raw outputs. However, when scores were triangulated with open-ended answers, stronger patterns were observed, such as the impact of fluency of the translations on their comprehensibility and trustworthiness. We found correlations between comprehensibility and trustworthiness, and identified other factors influencing these aspects, such as the clarity and soundness of the messages. We conclude by outlining implications for crisis preparedness, limitations, and areas for future research.

## introduction societies are becoming increasingly multicultural and multilingual, mainly as a result of economic migration and displacement @xcite . in ireland, for example, there are more than 500 thousand non-irish nationals, the majority of whom come from a country where english is not the official language, e.g. poland, @xcite . non-native speakers of a language-and especially those with limited proficiency-need to overcome considerable communication challenges in the contexts of crises @xcite . taking again ireland as an example, flooding is the most common hazard that the country needs to manage @xcite . when substantial, flooding poses a threat to infrastructure, business, and also people's health (major emergency @xcite . in order to be safe and act upon the messages sent by emergency responders, linguistically diverse communities need to be able to comprehend and trust those messages @xcite . machine translation (mt) and post-editing (pe) can play a role in crisis communication but their application needs careful consideration. this paper describes the results of a survey whose goal was to address two important gaps in relation to the role of mt and pe as enablers of multilingual communication in crises. specifically, we set out to gather empirical evidence on the impact of mt and of pe awareness on comprehension of and trust in messages disseminated by emergency responders to prepare the public for a specific weather-related crisis: flooding. the translation direction under analysis was english to italian (see section 3 for our research questions). the choice of this translation direction was motivated by the substantial number of native speakers of italian living in englishspeaking countries where flooding is common, such as the united kingdom and ireland @xcite . it is worth underlining the lack of clear distinctions between the concepts of crisis, emergency, disaster, or hazard. for the purpose of this study, we adopted a broad definition of crisis, understood as a non-routine and disruptive event, that poses a threat, and that usually involves the phases of preparation, response, and recovery @xcite . the remainder of this paper is organized as follows: section 2 reviews and summarizes related work on mt, pe, comprehension, and trust, with a special focus on crisis contexts. section 3 presents our research questions and the methodology that we adopted in order to answer them. section 4 reports on the results of our survey, which are then discussed in section 5, along with implications, limitations, and avenues for future research. ## related work translation of crisis information into the first language of the target audience facilitates comprehension, as has been shown, for example, in the case of the 2014 ebola outbreak (o' @xcite . however, the importance of translation in crises is still either not acknowledged or discussed only superficially in policy documents and institutional checklists @xcite . this is surprising when considering that misunderstandings due to lack of translation have often resulted in increased vulnerability and loss of lives @xcite . in addition to comprehension, the language in which information is conveyed can influence trust in the message, particularly in crisis situations @xcite . previous research on trust, translation, and crises has mainly focused on how translation influences reasoning about trust among people affected by a crisis @xcite , with trust emerging as one of the challenges in the communication ef-forts of humanitarian organisations, along with low literacy levels and cultural sensitivity @xcite . in crisis situations, mt has been a component of some communications, as shown, for instance, during the haiti earthquake @xcite and, more recently, in refugee settings @xcite . mt is particularly helpful when large quantities of texts need quick translations into multiple languages @xcite . the utility of mt in crisis settings involving low-resource languages has also been empirically tested @xcite . the relationship between mt and trust has received some attention since machine-translated outputs are far from flawless and fully accurate, even after the quality improvements introduced by the neural paradigm @xcite , thus often requiring pe. research has revolved around approaches to identify machine-translated words, sentences or documents that pass a predetermined quality threshold and are therefore more trustworthy @xcite . the availability of these confidence, or trust, scores seems to be welcomed by translators @xcite , but the scores should be accompanied by an explanation of how they were obtained @xcite . attention has also been given to the level of trust that professional translators attribute to machinetranslated outputs and specific mt engines @xcite @xcite . furthermore, lack of trust in mt has emerged as one of the reasons for its nonadoption among language service providers (porro @xcite . previous works have also focused on students, with mixed resultsfrom a general lack of trust @xcite , to a tendency to almost uncritically trust the output @xcite . more relevant to our research, a limited number of studies have focused on end users of mt-who often read translations for gist understanding (specia and shah, 2018)-and on their reliance on mt to locate information on websites @xcite , as well as on their tendency to use mt to translate from languages or documents of which they already have some knowledge, which might indicate a lack of complete trust in the output @xcite . research has also focused on the broader areas of acceptability, usability, readability, and com-prehensibility of machine-translated texts among end users, and on how these aspects are influenced by different pe levels @xcite . however, most of the research so far has focused on technical documents. accordingly, there is a lack of empirical evidence on: (i) the potential benefits of mt (as opposed to lack of translation) for end users' comprehension of and trust in crisis communication; and (ii) the potential impact on comprehension and trust of being aware that crisis messages have been post-edited. we set out to fill these research gaps. ## discussion and conclusions with this survey, we set out to investigate the impact of mt and pe awareness, in the english to italian direction, on comprehension of and trust in messages disseminated to the public in the context of preparation for a specific weatherrelated crisis, i.e. flooding (section 3.1). overall, we found slight and non-significant differences in terms of scores between english, purported raw, and post-edited messages. however, some interesting trends emerged, namely: (i) some beneficial effect of mt on comprehension and trust among end users with low english proficiency; (ii) a tendency to comply more with messages in english, possibly as a result of their authoritative tone/style (section 4.2); and (iii) labelling of messages as post-edited resulting in some improvement in comprehension, but not in trust. the absence of a beneficial effect of pe awareness on trust might be due to: (i) the purported mt outputs having also been post-edited and, therefore, appearing equally trustworthy; (ii) the fact that the fluency of the declared post-edits could have been improved further (section 4.2). in line with these points, the fluency of the translations had some impact on how comprehensible and trustworthy the messages appeared to be to our participants. interestingly, after comparing the influence of different pe levels among end users, @xcite found that full pe led to positive judgements in terms of language use and style, but did not result in a significant improvement of the perceptions (including trustworthiness) of the sender of a product. regardless of how the messages were labelled, several aspects of crisis messages were recognized as particularly important by participants, especially the clarity, the soundness, the helpfulness, and the source of the messages. the importance of clarity for comprehension could be expected. on the other hand, results regarding trust are particularly interesting as they align with models of trust @xcite according to which the decision to trust is determined by, among others: the competence of the trustee (e.g. their ability to provide accurate and sensible information), corresponding to soundness in our study; their intentions (e.g. to help the public affected by a crisis), corresponding to our helpfulness theme; and their adherence to a set of accepted principles, e.g. as imposed by the profession on emergency managers, who were the source of our messages. furthermore, trust models discuss the trustor's propensity to trust @xcite , which, in our study, seemed to be mainly determined by previous experience of flooding. a final interesting finding from this study was the demonstration that greater comprehension is associated with greater trust. this finding provides empirical evidence of the role that clear crisis communications-through plain language and/or translation-can play in establishing a relationship of trust between emergency managers and the public, thus leading to higher compliance with instructions for crisis preparedness. a similar result, although related to advertisement disclaimers, is reported in @xcite . this study has several limitations, particularly the high level of english proficiency of most participants, and the limited number of messages that were evaluated. larger-scale experimental studies with different setups are warranted. additional research should focus on: the impact of different pe levels; the impact of labelling human translations as post-edits; different language pairs; and end users less familiar with mt (figure 2 ). it might also be interesting to observe end users' interactions with mt and to explain them using a trust and credibility lens (see e.g. @xcite . finally, future qualitative research could help determine the directionality of the relationship between comprehension and trust.