A Context-Aware Annotation Framework for Customer Support Live Chat Machine Translation (2023)

Abstract: To measure context-aware machine translation (MT) systems quality, existing solutions have recommended human annotators to consider the full context of a document. In our work, we revised a well known Machine Translation quality assessment framework, Multidimensional Quality Metrics (MQM), (Lommel et al., 2014) by introducing a set of nine annotation categories that allows to map MT errors to source document contextual phenomenon, for simplicity sake we named such phenomena as contextual triggers. Our analysis shows that the adapted categories set enhanced MQM’s potential for MT error identification, being able to cover up to 61% more errors, when compared to traditional non-context core MQM’s application. Subsequently, we analyzed the severity of these MT “contextual errors”, showing that the majority fall under the critical and major levels, further indicating the impact of such errors. Finally, we measured the ability of existing evaluation metrics in detecting the proposed MT “contextual errors”. The results have shown that current state-of-the-art metrics fall short in detecting MT errors that are caused by contextual triggers on the source document side. With the work developed, we hope to understand how impactful context is for enhancing quality within a MT workflow and draw attention to future integration of the proposed contextual annotation framework into current MQM’s core typology.

## introduction in past decades, the staggering growth in demand for shared knowledge has led to an increase in translation requests, exceeding human translators' work capacity. in order to accommodate to such request, many enterprises are now integrating mt systems to their workflow that allegedly provide human-like translations in record time. however, despite often claims of human-parity @xcite , there are plenty of work in the field @xcite ) that dispel such allegations, even showing that, under certain circumstances, state-of-the art conventional approaches under-perform and are unable to deal with language nuances, translating words instead of "meanings". aware of neural machine translation (nmt) limitations, in the last few years, new approaches have been devised to leverage document context for finer-grained mt outputs. despite sharing similar believes, we suspect that researchers have only now begun to scratch the surface on such complex subject matter, especially when it is not yet clear that context-aware mt systems are indeed able to account for context within a document @xcite . yet, there is scarce research into document-level mt quality assessment (qa) metrics for more reliable evaluations @xcite . taking into account the present scenario, we propose a framework that deals strictly with context issues instead of relying on more traditional qa metrics regarded as less suitable for document-level nmt assessment. to properly understand the weight of context within a document, we used the previously mqm annotated wmt-chat-task en-pt/br dataset foot_0 , from live chat customer support interactions, creating the perfect test environment for our research, that strives for more equitable and accurate qa mt metrics. ## state-of-the-art it is widely acknowledged that document context is critical for resolving a wide range of translation problems, nevertheless, the sentence-based translation approach remains the most salient characteristic of the prevailing mt paradigm @xcite . this method, in which documents are dismembered in self contained elements (independent sentences) for better translation management, fails in several accounts. first off, the mt system may translate words or phrases based solely on their individual usage, rather than considering their placement in the document as a whole, and second, it largely fails to maintain intersentential relationships within a document @xcite . such behavioral pattern ends up compromising essential textual parameters: cohesion and coherence, giving rise to a warped source text representation. realizing the limitations of sentence-level mt, in recent years, new proposals have surfaced, encouraging a paradigm shift. context aware mt models have started to be implemented and designed to leverage contextual information in a document @xcite @xcite , exposing the importance of context in improving mt quality @xcite , leading to new challenges: how to evaluate the quality of contextual mt models and how to identify if contextual mt models are actually using context? ## contextual annotation framework. recent context-aware mt models progress calls for developing new evaluation solutions that cover contextual errors. our framework allows to identify and classify contextual discourse structures linked to mt errors. this section will initially describe the most frequently addressed contextual categories in the literature, followed by our new set of contextual categories found to be relevant for the customer support live chat domain data. note that the framework was created with the possibility to be accustomed to other domains. ## results in this section we analysed the errors and characterized them according to the context aware typologies framework that we developed, providing an error and severity analysis, whilst, simultaneously, contrasting our annotation with the mqm's non-contextual annotation performed previously for the wmt 2002 chat shared task. ## automatic metrics of mt evaluation and contextual errors measuring the ability of the current state-of-the-art mt evaluation metrics in detecting the contextual errors is the first crucial step for developing new automated quality evaluation solutions for the mt systems using our proposed typology. hence, we measured the correlation of these metrics with the mqm annotations of the maia test-set. to have a reliable term of comparison, in addition to the contextual annotations, we also measured the correlation of the metrics on the original mqm annotations based on the existing framework. for the metrics, we used comet @xcite ) that is trained to predict the human translation quality judgments of the mt outputs. it evaluates the translations in isolation without considering their contexts at all. very recently, @xcite introduced an extension of this metric (i.e., doc-comet) that incorporates context when evaluating the mt outputs. @xcite show that doc-comet obtains a higher system-level pearson correlation with human judgments compared to its original sentence-level counterpart on ted talks and news domains for en-de, en-ru, and zh-en language pairs. since the system-level analysis does not provide detailed insights on the ability of the metrics in capturing the contextual errors, we focused our analysis on the sentence-level correlation of the metrics with human judgments on the maia dataset. given that our framework is tailored for the contextual errors only, for our analysis we concentrated on the samples that contain at least one contextual error in the output of the mt system. we also made sure that errors that do not have a contextual background have no reflection on the automatic metrics results. to this aim, and to not lose the context, we first obtained the scores of all the sentences of the test-set with each metric, and then used only the segments with contextual errors, 197 sentences, for the baseline model, and 123 sentences for the unbabel-ist model. as the results suggest, both comet and doc-comet have a lower correlation with the mqm scores of our annotation framework. this, however, is expected mainly because the comet models were trained on the data annotated with the existing mqm annotation framework. moreover, we clearly see that there is no reliable correlation between doccomet and the human judgments of both frameworks on the sentence level. this can be justified by the fact that the comet models were not trained on any document-level annotations, hence they cannot detect contextual errors accurately. these findings show that in order to measure the quality of the mt systems on the contextual errors, new datasets, metrics and tools need to be developed that not only cover the existing sentence-level errors, but also can cover the contextual errors that none of the current resources cover, and usually are categorized as severe errors (i.e., either critical or major). ## conclusion with our research, we have shown the significance of context for the mt. similarly, we exposed the inadequacy in conventional qa metrics for reliable qualitative assessments, since current qa models and frameworks show to be weak and deceptive as they have not been created to have in mind contextual mt errors. we have displayed first attempts in overcoming qa models and frameworks shortcomings in the form of contextual errors test-suites, but also those are scarce in terms of contextual typologies coverage and focus on common analysed domains. we instead propose an alternative contextual framework for document level mt qa, covering a relatively untapped domain in terms of contextual errors analysis. our framework shows significant gains of an average of 61% more contextual errors coverage than more conventional qa metrics, highlighting the fact that most of such contextual errors are deemed as critical and major, thus strengthening our beliefs that the field of qa for context aware mt is far from being effectively dealt with, on the one hand, and that contextual error severely compromise mt outputs, on the other hand.