On The Real-world Performance of Machine Translation: Exploring Social Media Post-authors’ Perspectives (2023)

Abstract: Many social networking sites (SNS) offer machine translation of posts in an effort to increase understanding, engagement, and connectivity between users across language barriers. However, the translations of these posts are still not 100% accurate and can be a cause of misunderstandings that can harm post-authors’ professional or personal relationships. An exacerbating factor is on most SNS, authors cannot view the translation of their own posts, nor make corrections to inaccurate translations. This paper reports findings from a survey (N = 189) and an interview (N = 15) to explore users’ concerns regarding this automatic form of machine translation. Our findings show that users are concerned about potential inaccuracies in the meaning of the translations of their posts, and would thus appreciate being able to view and potentially correct such translations. Additionally, we found that when users write posts in their native language, they write them for specific audiences, so they do not always want them translated. This underscores the urgency of providing users with more control over the translation of their posts.

## introduction social networking sites (sns) allow users to connect with people from various language backgrounds, and a sizable proportion of sns users writes posts in a language that not everyone in their audience understands @xcite @xcite . to help people with diverse language backgrounds understand the posts of users who write in an unfamiliar language, sns offer machine translation (mt) to automatically translate users' posts from the original language into a language the reader understands @xcite . this may, in turn, increase users' engagement with an increasingly global audience @xcite . however, currently sns do not give users any control over the translation of their own posts. for example, facebook 1 allows the readers of posts with options to select "languages you'd like to have posts translated into", "languages you don't want to be offered translations for", and "languages you don't want automatically translated." in contrast, the only settings that are available for post-authors are "a feature that lets you post multiple language versions of a status" @xcite and "turn off translations" for each post. however, it still does not allow post-authors to view or edit the translated version of their own posts, nor does it allow them to choose the languages they would like their post(s) to be translated into. unlike facebook, instagram currently allows authors to read translations of their posts and stories but, like facebook, offers no control over these translations. other popular sns (e.g., twitter, linkedin) currently do not offer any controls over the mt feature, making it difficult for authors to judge the quality of the translations of their posts @xcite . in fact, the only way that users of these sns are exposed to mt-translated posts is by reading the translations of other users' posts. casacuberta et al. @xcite find that machine translations are often not accurate and can inadvertently distort the intended meaning of a post, which can lead to misunderstandings among sns users. as a result, sns users report feeling insecure when they think about how their posts might be translated and whether their translated posts accurately convey their intended meaning @xcite . beyond this, there is very little research done regarding how users feel about the translation of their posts, the use of automatic mt, or the fact that, in most cases, users have no control over the translation that gets posted. the current study bridges this gap by addressing the following research questions: ## related work in this section, we review research on mt methods, on tools that give authors control over their social media posts, and on authors' considerations when posting sensitive information to a growing and increasingly diverse audience. ## methodology we conducted a survey study (@xmath0, appreciation of, and desire for control over the mt of their posts. here we discuss the development of our survey questions and interview script and outline participant recruitment and data collection for both studies. the studies were approved by our irb, and we used an exploratory approach to answer our research questions. ## results this section presents the findings of our two studies. analysis was conducted to explore the effect of users' familiarity with mt (rq1), attitudes towards mt (rq2), and desire for control over the translation of their posts (rq3). both quantitative and qualitative data were collected from the survey; only qualitative data were collected from the interviews. we analyzed the quantitative data using correlation, regression, and t-tests, and we used thematic analysis for the qualitative data. ## discussion in this paper, we investigated authors' awareness of, and attitudes towards, the automatic translation of their sns posts. our findings show that people often write posts in languages other than english, and many of these people are familiar with and have experience with mt-likely because many of them interact with people from different language backgrounds, so they regularly see translations of posts from their friends which were originally written in different languages @xcite @xcite . users who are more (less) familiar with mt are more (less) likely to post on social networking sites (sec. 4.1), which was opposite to our expectations. interview participants who were aware of mt's capabilities explained that it made them more cautious about posting sensitive content that could be mistranslated and cause personal or professional harm. this heightened awareness lead to more selective posting, rather than decreased posting overall. in particular, such users are confident about mt translations in terms of actual meanings (which makes them tolerant towards the translation of posts about non-sensitive topics) but less confident in terms of connotative meaning @xcite (making them more cognizant of potential risks of the mt of posts containing figurative speech or slang). users want to avoid translating posts about sensitive topics-several participants wanted to share such information only to audiences who understand the language of the post. this may be because the information is specific to a particular culture and traditions that cannot be understood without context, or because another language may simply not have words for certain concepts @xcite . since mt currently is fully automated, users are not able to see, let alone control the translation of their own posts. as a result, most participants were concerned about the possibility of their posts being mistranslated and consequently misinterpreted by their audience, and many wished for features that give them more control over the translation of their posts. in contrast, a few of the interviewees mentioned that they feel more concerned about the translation of their posts-especially those writing in low-resourced languages. but even interviewees who wrote in high-resourced languages such as spanish and chinese mentioned that slang often gets mistranslated. this is because the quality of automated mt, both for low-and high-resourced languages, is still not 100% accurate. if the automated mt becomes more capable of accurately conveying the intended meaning of posts, users may have less desire to control their translations. that said, both hci theory and empirical findings show that users desire control over automated systems, even when the systems are so good that they rarely require intervention. having control over their posts' translation provides them a greater sense of security and ownership and helps individuals to present themselves more effectively online. thus, existing research corroborates the finding that users want to exercise control over machine translation, but also suggests that an intermediate level of control would be optimal to reduce cognitive load @xcite . an appropriate solution would be to allow users to view the mt of their posts and to correct potential translation errors. this will provide users a greater sense of control without requiring them to write the translation manually. this solution is not without limitations. for authors, the editing of posts would only be possible for output languages that they can read and write in. furthermore, authors' ability to edit the translations of their posts could result in ethical concerns, as this feature may be used to create language-specific misinformation. a solution to this problem is to present the reader with both the original machine translation and the user-edited version. alternative solutions include allowing users to disallow the automatic translation of posts on a case-by-case basis, or to filter the audience of a post based on its original language. this aligns with the fact that users' language choices are intentional, and depend heavily on the topic of the post and the target audience @xcite : many purposely write posts in a certain language as a means to target those who speak that language. overall, we see a substantial benefit in making sns users aware of how their posts are translated and shared with others, and in allowing them to remove the translation if the post was meant to be targeted to a specific audience or correcting the translation to preserve the original meaning of the post. together, these design solutions would ensure users that their information is conveyed accurately and only to their intended audiences. ## limitations and future work to recruit participants who are more likely to write posts in other languages, our recruitment was restricted to bi/multilingual individuals residing in the us. due to these criteria, most participants in both studies were spanish speakers and writers, which is the second most common language spoken in the us. since spanish is a rich-resourced language, findings about the quality or accuracy of mt may be different if we consider a more linguistically diverse participant sample. future work could recruit global bi/multi-lingual participants to understand broader perceptions of machine translation. our discussion section advocates for a number of control features that are currently not available in sns. our results suggest that these features would make users more comfortable with mt, but this suggestion would ideally be confirmed with a controlled experiment. our future research will be focused on designing the proposed controls. in particular, we plan to conduct an experiment on prototypes of three different translation features: one allowing the user to read but not edit the mt, one allowing the user to add a translation manually, and one allowing the user to read and edit the mt if there are any errors. each prototype will also have an option to include/not include the translation with each post. this experiment will measure which features most improves users' perceived control, perceived satisfaction, ease of use, and intention to use mt. ## conclusion in this paper we offer insights into users' awareness of and concerns regarding the translation of their social media posts. we consider improvements to the user experience of mt in terms of providing authors with more control over post translation. findings from our studies demonstrate that sns users find mt of posts with connotative meaning to be very poor, perhaps because the mt may fail to properly account for contextual information to accurately translate the intended meaning; users also express concern that mt may get the tone of the original post wrong. as a result, users were found to be concerned about how their posts are translated and subsequently interpreted by their audiences. this caused many users to be selective in which of their posts they would want to be translated: indeed, some posts were considered too sensitive to be translated, while at other times users intentionally used language as a means to limit the audience of their post. these findings stand in stark contrast with the current practice on most sns, where users' posts are translated indiscriminately and without users' explicit permission. thus, we call upon existing sns to give their users more control over the translation of their social media posts.