Towards Personalised and Document-level Machine Translation of Dialogue (2021)

Abstract: State-of-the-art (SOTA) neural machine translation (NMT) systems translate texts at sentence level, ignoring context: intra-textual information, like the previous sentence, and extra-textual information, like the gender of the speaker. As a result, some sentences are translated incorrectly. Personalised NMT (PersNMT) and document-level NMT (DocNMT) incorporate this information into the translation process. Both fields are relatively new and previous work within them is limited. Moreover, there are no readily available robust evaluation metrics for them, which makes it difficult to develop better systems, as well as track global progress and compare different methods. This thesis proposal focuses on PersNMT and DocNMT for the domain of dialogue extracted from TV subtitles in five languages: English, Brazilian Portuguese, German, French and Polish. Three main challenges are addressed: (1) incorporating extra-textual information directly into NMT systems; (2) improving the machine translation of cohesion devices; (3) reliable evaluation for PersNMT and DocNMT.

## introduction neural machine translation (nmt) represents stateof-the-art (sota) results in many domains @xcite @xcite , with some authors claiming human parity @xcite . however, traditional methods process texts in short units like the utterance or sentence, isolating them from the entire dialogue or document, as well as ignoring extra-textual information (e.g. who is speaking, who they are talking to). this can result in a translation hypothesis' meaning or function being significantly different from the reference or make the text incohesive or illogical. for instance, the sentence in polish "nie poszłam." ("i didn't go." foot_0 ) incorporates gender information in the word poszłam (went fem ) -as opposed to poszedłem (went masc ) -while the english verb does not incorporate such information. when translating "i didn't go." into polish, the machine translation (mt) model must guess the gender of i, as this information is not rendered in the english sentence. @xcite show that when commercial mt engines need to "guess" the gender of a word, they do so by making implications based on its co-occurrence with other words in the training data. since training data is often biased @xcite , mt models will reproduce these biases, further propagating and reinforcing them. clearly, research on context-aware machine translation is needed. sentence-level nmt (sentnmt) is especially harmful in the domain of dialogue, where most utterances rely on previously spoken ones, both in content and in style. the way in which an interlocutor chooses to express themselves depends on what they perceive as the easiest for the other person to understand @xcite . dialogue is naturally cohesive @xcite , i.e. rid of redundancies, confusing redefinition of terms and unclear references. part of what makes a conversation fluent is the links between its elements, which sota nmt models fail to capture. for instance, the latter utterance in the following exchange: "they put something on the roof." "what?" translates to polish as "co takiego?" ("what something?"). the translation uses information unavailable in the utterance itself, i.e. the fact that what refers to the noun something. a sentence-level translation of what? would just be co?, which is more universal, but also more ambiguous. simply put, even when sentnmt pro-duces a feasible translation, its context agnosticism may prevent it from producing a far better one. there are growing appeals for developing nmt systems capable of incorporating additional information into hypothesis production: personalised nmt for extra-textual information (e.g. @xcite @xcite and document-level nmt for intratextual information (e.g. @xcite @xcite . evaluation methods predominant within both areas vary vastly from paper to paper, suggesting that for these applications a robust evaluation metric is not readily available. this view is further strengthened by the fact that @xcite , when assessing their mt for human parity, ignored document-level evaluation completely. @xcite later disputed this choice, showing that professional annotators still overwhelmingly prefer human translation at the level of the document, and therefore human parity has not yet been achieved. this case study shows how much a robust and widely accepted document-level metric is needed. currently, researchers working on persnmt and docnmt conduct evaluation primarily by reporting the bleu score for their systems. but they also commonly assert that the metric cannot reliably judge fine-grained translation improvements coming from context inclusion. as a way out, some of them report accuracy on specialised test suites (e.g. @xcite @xcite or manual evaluation. although both have limited potential for generalisation, their attention to detail makes them superior tactics of evaluation for applications such as persnmt and docnmt. in this work we utilise tv subtitles, a contextrich domain, in order to investigate whether mt of dialogue can be improved: directly, by enhancing document coherence and cohesion through incorporation of intra-and extra-textual information into translation, and indirectly, by designing suitable evaluation methods for persnmt and docnmt. dialogue extracted from tv content is an attractive domain for two reasons: (1) there is an abundance of parallel dialogue corpora extracted purely from subtitles, and (2) the data is rich in or could potentially be annotated for a range of meta information such as the gender of the speaker. in section 2, we discuss relevant contextual phenomena. we then present the research on persnmt and docnmt, and the applicability of mt evaluation metrics to both. in section 3 we delineate the research questions, the work conducted so far and our plans. section 4 concludes the paper. ## addressing research questions within this phd, we seek to answer three research questions (rqs): rq1 can machine translation of dialogue be personalised by supplying it with extra-textual information? rq2 is ellipsis problematic for mt, and can mt make use of marking of ellipsis and other cohesion devices to increase cohesion in translation of dialogue? rq3 how can automatic evaluation methods of mt be developed which confidently and reliably reward successful translations of contextual phenomena and, likewise, punish incorrect translations of the same phenomena? ## conclusions this work is the proposal of a phd addressing per-snmt and docnmt in the dialogue domain. we have presented evidence that sentence-level mt models make cohesion-and coherence-related errors and offered several approaches via which we aim to tackle this problem. we plan to conduct extensive experiments to analyse the problem of ellipsis translation and of the use of sentence-level evaluation metrics to evaluate contextual phenomena. the outcome of this work will also include publicly available test suites, a document-level translation model, a personalised translation model and a context-aware evaluation metric.