Measuring the Effects of Human and Machine Translation on Website Engagement (2022)

Abstract: With the internet growing increasingly multilingual, it is important to consider translating websites. However, professional translators are much more expensive than machines, and machine translation quality is continually increasing, so we must justify the cost of professional translation by measuring the effects of translation on website engagement, and how users interact with translations. This paper presents an in-the-wild study run on 2 websites fully translated into 15 and 11 languages respectively, where visitors with non-English preferred languages were randomized into being shown text translated by a professional translator, machine translated text, or untranslated English text. We find that both human and machine translations improve engagement, users rarely switch the page language manually, and that in-browser machine translation is often used when English is shown, particularly by users from countries with low English proficiency. We also release a dataset of interaction data collected during our studies, including 3,332,669 sessions from 190 countries across 2 websites.

## introduction the userbase of the internet is becoming increasingly linguistically diverse @xcite . as a result, publishers increasingly need to translate websites to make content available to global audiences. professional translators can be expensive, so localization decisions involve many tradeoffs. should we show translations generated by professional translators, or is machine translation quality sufficiently high that showing machine translations will not negatively impact user engagement? if a human translation is not available in the user's preferred language, is it preferable to show english or a machine translation? does the website need to be translated at all, given that browsers have built-in machine translation functionality? to make informed decisions, we need to know the effects of each form of translation on website engagement. this paper quantifies the effects of translation on engagement metrics and how users interact with translations, by running large-scale (over 3 million sessions), in-the-wild a/b tests on the homepages of two different open-source software projects. our contributions in this paper are a set of studies quantifying how users interact with website-provided translations and inbrowser machine translations, interface recommendations on how to incorporate translations into websites based on these findings, and a public dataset of interactions and code that can be used to reproduce our results and conduct follow-up research. ## related work previously published work has not evaluated the effects of machine translation, human translation, and non-translation on in-the-wild website user engagement in a randomized, a/b test fashion. while machine translation quality has historically been negatively perceived @xcite , some machine translation systems claim to have reached parity in translation quality with professional translators in certain settings @xcite @xcite . other work has questioned these claims of human parity @xcite @xcite . website translation has been found to benefit search engine optimization by attracting users who search in their native language @xcite . machine translation of product listings has been found to help increase purchases on ebay @xcite . this work seeks to measure the effects of human and machine translation on user engagement in the context of software-centric websites. ## research questions the studies in this paper aim to answer the following research questions: ## methodology we ran a/b tests on two sites, both of which are open-source software projects with a single-page design, shown in figure 1 . both sites had been translated from english to several languages foot_1 by professional translators two years before we began running this experiment. we will call these supported languages for each site. we refer to the preferred language in the user's browser settings as the preferred language. site 1 had 3,298,635 sessions total, of which 1,233,841 (37.4%) had a supported non-english preferred language. site 2 had 34,034 sessions total, of which 9,316 (27.4%) had a supported non-english preferred language. data was gathered from nov 25, 2020 to jan 14, 2022 (415 days). we obtained machine translated text from google translate in nov 2020. see the appendix for the demographics of visitors to the websites. ## study 1: effects of human and machine translation availability on engagement we study the effect of showing a human translation, machine translation, or no translation on the following engagement metrics: proceedings of the 15th biennial conference of the association for machine translation in the americas orlando, usa, september 12-16, 2022. volume 1: research track have the majority of textual content describing the software below the fold when viewed on a standard monitor. a visit duration in the top quartile (17 seconds or more) suggests the user is likely reading the page, as there are only images and no videos on the sites, so the non-textual portions of the websites can likely be skimmed in less than 17 seconds. proceedings of the 15th biennial conference of the association for machine translation in the americas orlando, usa, september 12-16, 2022. volume 1: research track to that language. another approach is to show untranslated pages by default, and ask the user to use a language selector or click a link in order to see the translation. we wished to see how often users would use a language selector, and the effects of requiring language selection on user engagement. the language selector interface used in this experiment is shown in figure 1 . figure 4 shows the fraction of users who used the language selector in each condition on sites 1 and 2. on both sites, there is a significant increase in usage of the language selector when english is shown by default (hte, mte) vs when a translation is shown by default (htt, mtt). on site 1, the language switcher is used in 1.95% of sessions in hte vs 0.21% of sessions in htt (χ 2 = 3095, @xmath0.0001). on site 1, the language switcher is used in 1.94% of sessions in mte vs 0.20% of sessions in mtt (χ 2 = 3123, @xmath1.0001). on site 2, the language switcher is used in 2.18% of sessions in hte vs 0.50% of sessions in htt (χ 2 = 15.81, @xmath2.0001). on site 2, the language switcher is used in 1.74% of sessions in mte vs 0.50% of sessions in mtt (χ 2 = 10.14, @xmath3.005). users hardly engage with language switchers on either site-even though it takes only a single button click to see the page in the user's preferred language, only 2% of users click it when english is shown by default. switching away from the preferred language is even less frequent. thus, automatically detecting the user's preferred language and showing translations accordingly is important-otherwise users will not see translations. showing english by default and requiring the user to use a language selector to view the page in their preferred language is detrimental to engagement. as we can see in figure 2 , the improvement in engagement metrics from translation we had previously observed for site 1 are lost in the hte and mte conditions. χ 2 tests are shown in there are no significant differences in language switcher use between hte vs mte (site 1: χ 2 = 0.021, @xmath4.886; site 2: χ 2 = 0.616, @xmath5.432) or between htt vs mtt (site 1: χ 2 = 0.342, @xmath6.559; site 2: χ 2 = 0, @xmath7. the fact that on both sites less than 0.6% in the mtt condition switch to english, and this is not significantly higher than the htt condition, suggests that the machine translation was sufficiently usable that users did not switch to english. ## study 3: in-browser machine translation use we were surprised by our results from study 1 that there was high engagement with untranslated english pages, even though the user's browser language preference was a non-english language that the site had been translated to, and our result from study 2 that users rarely switch the page to their preferred language. we suspected that users may be viewing pages through the machine translation functionality that is built into browsers, so we measured the use of in-browser machine translation. results are shown in figure 5 for sites 1 and 2. when a page is only available in english, 9.76% of users use in-browser machine translation on site 1, and this number is 5.23% on site 2. when a human translation is available in the user's preferred language, but the page is shown in english by default, 9.49% of users will use in-browser machine translation on site 1, and this number is 5.13% on site 2. this is higher than the 2% of users who used the language switcher from study 2, meaning that if users are required to click a button to see a translation, users will be more likely to end up seeing their browser's machine translation than clicking the button to see a human translation. we were curious whether the user's language and country has an influence on their use of in-browser machine translation. is in-browser machine translation used at roughly equal levels everywhere, or is it used much more in some countries than others? perhaps users whose preferred language is a language where the quality of machine translation from english is high will be more likely to use machine translation? perhaps users whose preferred language has high lexical overlap with english, or uses the same latin alphabet as english, will need to rely on machine translation less? perhaps users from countries with a high level of english proficiency will be less likely to use machine translation? we display the in-browser machine translation usage broken down by the target language from site 1 in figure 6 . site 2 results are in figure 8 of the appendix. interestingly, it does not appear to be the case that machine translation use is highest for languages where the machine translation quality is better. for example, observe that vietnamese (vi) has the highest proportion of users using the browser's built-in machine translation on site 1 (14.8%) and is the fourth highest on site 2 (11.3%), despite it being a low-resource language for which parallel data for training machine translation systems is scarce @xcite . we can also observe the contrast in machine translation use between spanish (es, 11.80% on site 1 figure 6 : use of in-browser machine translation on site 1 by language, for users whose preferred language is not english. error bars indicate 95% confidence intervals. and 6.41% on site 2) vs french (fr, 6.05% on site 1 and 2.90% on site 2)-despite both being high-resource romance languages with high machine translation quality from english, and high lexical overlap with english for technical vocabulary. the type of script the language is written in does not seem to be the main influence on machine translation use either-the top 4 languages with the most machine translation use in site 1 and site 2 include both those written in non-latin (japanese, thai, bulgarian) and latin scripts (vietnamese, spanish, romanian). perhaps we should consider countries rather than the properties of the languages themselves? we show the in-browser machine translation usage broken down by country from site 1 in figure 7 . site 2 results are in figure 8 of the appendix. since we suspected that english proficiency may influence machine translation usage, we group countries by their english proficiency ranking on the ef english proficiency index 2020 @xcite . we find that among users with non-english preferred languages, those from countries with lower english proficiency tend to use machine translation more. on site 1, 9.84% of sessions with non-english preferred languages from countries with "moderate" or lower english proficiency use in-browser machine translation, whereas only 6.10% of such sessions from counties with "high" or "very high" english proficiency used in-browser machine translation; this difference was statistically significant (χ 2 = 3797, @xmath8.0001). on site 2, 6.33% of sessions with non-english preferred languages from countries with "moderate" or lower english proficiency use in-browser machine translation, whereas only 4.05% of such sessions from counties with "high" or "very high" english proficiency used in-browser machine translation; this difference was statistically significant (χ 2 = 18.67, @xmath9.0001). ## dataset and code to help replicate this study and enable researchers to run further analyses, we publicly release the datasets for both websites and the notebooks to reproduce our results at https: //transabtest.github.io/ under the creative commons attribution license. the dataset represents the full 3,298,635 sessions from site 1, and the full 34,034 sessions from site 2, and the site content and translations. ## conclusion as the population of internet users grows increasingly linguistically diverse, it is increasingly important for websites to consider whether they should translate their websites, and how to present translations. machine translation quality has considerably increased over the years, and in-browser machine translation has likewise become integrated into major browsers, which leads us to ask how much benefit websites can expect from translating their website with human translators, as opposed to showing machine translations or leaving pages untranslated. through an a/b test we run on the homepages of two open-source software projects, we find that both human and machine translations improve engagement metrics that are indicative of reading the page, though download rates remain high regardless of whether a translation is shown or not. compared to machine translations, human translations increase two engagement metrics (scrolling and clicking on non-download links), but had no effect on others. if we require users to click a button to switch to their preferred language, they rarely do so, and the gains in engagement metrics we observed from translation are negated. a significant minority of users with non-english preferred languages use machine translation systems integrated into browsers, especially from countries with low english proficiency. this, along with our finding that only a small fraction of users switch to english if shown a machine translation to their preferred language, suggests that machine translations are often of acceptable quality to some users. while showing machine translations performed well on the software-centric sites we studied, our findings may not generalize to other types of content; perhaps human translations will have more visible benefits on more text-centric content such as news. the effects of translation on engagement will also change over time, as machine translation quality improves, the web's userbase becomes increasingly multilingual, and as foreign language reading abilities change. localization decision makers choosing between human and machine translations should also consider website demographics such as their visitors' countries and preferred languages, and the quality of machine translation for the languages they are considering supporting.