Agent and User-Generated Content and its Impact on Customer Support MT (2022)

Abstract: This paper illustrates a new evaluation framework developed at Unbabel for measuring the quality of source language text and its effect on both Machine Translation (MT) and Human Post-Edition (PE) performed by non-professional post-editors. We examine both agent and user-generated content from the Customer Support domain and propose that differentiating the two is crucial to obtaining high quality translation output. Furthermore, we present results of initial experimentation with a new evaluation typology based on the Multidimensional Quality Metrics (MQM) Framework Lommel et al., 2014), specifically tailored toward the evaluation of source language text. We show how the MQM Framework Lommel et al., 2014) can be adapted to assess errors of monolingual source texts and demonstrate how very specific source errors propagate to the MT and PE targets. Finally, we illustrate how MT systems are not robust enough to handle very specific source noise in the context of Customer Support data.

## introduction unbabel's language operations platform blends advanced artificial intelligence with humans in the loop, for fast, efficient, high-quality translations that get smarter over time. the company combines machine translation with human post-edition performed by non-professional posteditors to translate customer support content in a variety of formats including emails and chat messages. customer support is a highly unique domain given that it involves bilateral communication between customer support agents ('agents') and customers ('users'), each with their own nuanced discourse strategies and features. notwithstanding, primary literature such as @xcite , generally consider both sides of the interaction jointly, without regard for independent and differentiating factors. the fundamental differences can be characterized as follows: agents are call center employees who are usually nonnative speakers of english, which has generally evolved as the 'lingua franca' of customer support and the primary source language translated at unbabel. given that english is usually not the agent's first language, it is common to observe elements of language transfer (where the grammar rules of their native language are transferred to the english language) and other linguistic errors, more commonly the addition and omission of prepositions and articles, as also mentioned in @xcite and @xcite . according to @xcite , a person's experience and knowledge of their mother tongue will most definitely interfere with the learning of a second language, thus creating errors of different nature. the interaction established by agents is somewhat controlled because they are usually following a particular protocol for communication prescribed by their company. this might include response templates, branding and fixed terminology which discourages stylistic variance and can often result in a large amount of repetition both within and across interactions. agents also work quickly, aiming to provide consistently timely responses. this can often result in typographical and other linguistic errors. agents often operate in highly stressful circumstances; they might have to meet certain quotas which, for example, demand a brief turnaround time. this will ultimately influence their performance and the introduction of errors in the messages which, in our use case, are subsequently translated by mt and post-edited. user-generated content from customers, on the other hand, is highly variable and unstructured. common features include the use of abbreviations, emoticons and idiomatic expressions, all of which present unique challenges to mt. grammatical and typographical errors common to user-generated content resulting from keyboard or smartphone use are also present. most critically, content from customers in a customer service context is highly purposeful and sometimes emotionally volatile. customers often contact customer support to complain about a product or service and may exhibit high levels of impatience and frustration. linguistically, this is reflected in unique lexical choices, such as the use of profanities, and variable capitalization and punctuation. all of which can often result in degradation of translation quality where the interaction is translated into or out of the source language. additionally, the native language of the customer is not always predictable. as mentioned in @xcite , often times customers from non-english speaking countries will be interacting in a non-native tongue. this could be either because they live in a foreign country, or because they are engaging the services of a foreign company. finally, as mentioned in @xcite , being a native speaker of a language does not directly indicate a high proficiency of that language, another factor that can potentially indicate poor source text inputs. different types of customer support content also present another dimension of complexity: consider, for example, how a chat might differ from an email. the response time required in the former will often determine the fidelity and quality of the resulting interaction. as lind (2012) illustrates, time restrictions implied in chat language ultimately result in fragmented written content. because emails do not require real-time translation, at unbabel, translation is performed first by mt and subsequently post-edited. chat messages, however, require instantaneous translation and as such do not benefit from pe. this paper builds upon previous work by gonc @xcite and presents an evaluation framework for source text informed by the features of customer support interactions, which is currently being put into production at unbabel as a means of evaluating the quality of source language text. as well as the extent to which we are able to provide a methodology and a framework that can be used in the future to ensure accurate translation and improve the robustness of our mt models to source language noise. in this paper we seek to address the following questions: ## related work noisy source text input is a common issue in mt and the translation quality of user-generated content has received some limited attention in literature. @xcite for example, highlight the difficulties presented by source noise and introduce methods to improve mt system robustness to noisy source text from internet and social media, while @xcite propose to statistically model errors from grammatical-errorcorrection with state-of-the-art nlp systems. regarding human evaluation of noisy source input, the research on error typologies and evaluation frameworks has mainly focused on the annotation of translation errors. there is a scarcity of investigation on the annotation of source text and limited work on the impact of source noise on mt output. as a consequence, research and guidance on the annotation of source text is equally lacking. although there are no clear guidelines foot_0 , we acknowledge the recent developments on the core mqm framework @xcite to identify which categories of issue can be applied to both source and target errors. in addition, @xcite introduce the scate mt error taxonomy, which makes more of a clear distinction between the kinds of errors found in bilingual and monolingual text. notwithstanding the above, there is still very limited work on evaluation of source text independently of translation, particularly in the context of user-generated content and less so in the customer support use case. gonc ¸alves (2021) considers the similarities and differences of the mqm framework @xcite , the scate mt error taxonomy @xcite and proprietary error typologies developed internally at unbabel. the same work introduces an adaptation of the mqm framework @xcite , and proposes a prototype of mqm-compliant typology for the annotation of source text errors, specifically tailored to usergenerated content and customer support interactions. the proposed prototype included 4 parent categories and 28 terminal issues at 2 levels of granularity and was supported by specific annotation guidelines and decision trees to support annotation and the choice of the appropriate degree of severity of the selected errors. still, the resulting typology showed a low agreement among annotators, meaning it lacked some robustness and wasn't efficient in a production setting. ## methodology the primary goal of our revisions to the prototype typology presented in gonc ¸alves (2021) was to increase its robustness to customer support content and improve its effectiveness in a production setting. with this in mind, we made several adaptations which resulted in a new typology with 5 parent categories, 31 terminal issues and 2 levels of granularity. where appropriate, we merged terminal nodes that resulted in low agreement and added a parent category. the full revisions to the initial prototype are detailed in the following section: ## experimental setup the main application of source text annotation in a translation environment is to study and understand the propagation and the impact of source errors on the mt and pe steps. in order to do so, we conducted three experiments with real client data in order to study how a particular communication medium influences agents and their communication and how mt systems handle user-generated content for chat message translation. the first two experiments are client specific and, for privacy purposes, we refer to them hereafter as client a and client b respectively. the nature of the data from client a is formulaic and repetitive. for this reason, we expect high quality translation results. the content chosen for this experiment was email threads, translated with a combination of mt and pe. we conducted a three-step alignment in the annotated data, where we made a comparison between the source text, the mt output and the post-edited target text of email threads. the language pairs analyzed were english to german ('en-de'), english to french ('en-fr') and english to swedish ('en-sv'). on the other hand, client b's data, being realtime chat, was translated with mt only. we included this particular client setting in order to study how chat communication affects the quality of the final mt output without any final human revision. the language pairs analyzed were english to german ('en-de') and english to italian ('en-it'). finally, in the third experiment, we randomly selected a sample of source texts coming from five different customers to study how user-generated content, in the context of chat conversations, impacts the final quality of the mt output. the lan-guage pairs analyzed in this experiment are italian to english ('it-en') and brazilian portuguese to english ('pt-br-en'). ## results our goal was to evaluate how errors present in source text impact the quality of the mt output and how they propagate and may be overlooked in human pe. this section presents the results obtained in the experiments outlined in section 4. for simplification reasons, we refer to the experiment relating to client a as "client a experiment", and similarly the experiment related to client b, as "client b experiment" and, finally, the experiment run with user-generated content as the "usergenerated content experiment".