A Review of Discourse-level Machine Translation (2020)

Abstract: Machine translation (MT) models usually translate a text at sentence level by considering isolated sentences, which is based on a strict assumption that the sentences in a text are independent of one another. However, the fact is that the texts at discourse level have properties going beyond individual sentences. These properties reveal texts in the frequency and distribution of words, word senses, referential forms and syntactic structures. Dissregarding dependencies across sentences will harm translation quality especially in terms of coherence, cohesion, and consistency. To solve these problems, several approaches have previously been investigated for conventional statistical machine translation (SMT). With the fast growth of neural machine translation (NMT), discourse-level NMT has drawn increasing attention from researchers. In this work, we review major works on addressing discourse related problems for both SMT and NMT models with a survey of recent trends in the fields.

## introduction in the last several decades, the field of machine translation (mt) has experienced three main historical periods including rule-based mt (rmt) @xcite , statistical mt (smt) @xcite and neural mt (nmt) @xcite , unremittingly improving performances of mt systems. as an active research field in natural language processing (nlp), mt is regarded as a sequence-to-sequence prediction task, which aims to find the most probable target language text for a source language one that shares the most similar meaning. it is challenging to generate the high-quality translations, because mt models need to not only thoroughly understand the source text but also have good knowledge of the target text @xcite . natural languages, from bottom to top, are composed of several linguistic units including word, phrase, clause, sentence, paragraph, and discourse @xcite . the sentences in a discourse, like words in a sentence, are closely related to one another. however, mt systems usually translate a text in a sentence-by-sentence fashion, and neglect intersentence dependencies. loss of discourse information leads to a series of problems in understanding the semantic meaning of inputs and generating coherent and consistent translations of outputs for an mt system. along with the development of mt, a large amount of effort has been going into addressing discourse explicitly in translation models. the 1990s saw an intensification of research efforts aimed at endowing rmt-translated texts with the same document and discourse properties as their source texts @xcite . this included work on stylistics @xcite , discourse relations @xcite and referring forms @xcite and pronoun translation @xcite @xcite . discourse was widely investigated and demonstrated promising results in different aspects including language modelling @xcite , discourse connectives @xcite , lexical cohesion @xcite , anaphora resolution @xcite and topic adaption @xcite in smt development and researches. in recent years, nmt has made significant progress towards to constructing and utilizing a single large neural network to handle the entire translation task. the performance of nmt models has surpassed that of traditional smt in various language pairs @xcite . as discourse information have been shown useful for translation models, a growing number of work investigated a variety of neural architectures to model document structure @xcite @xcite , discourse structure @xcite and cache memory @xcite @xcite . similar to research line in smt, several work focusing on explicitly modelling different discourse phenomena especially anaphora @xcite . some researchers further studied the discourse-aware evaluation and analysis @xcite . the aim of this survey is to highlight the major works that have been undertaken in the space of discourse-level mt in smt and nmt. in the term of discourse-level mt, we mean works which utilise inter-sentential context information comprising discourse aspects of a text or surrounding sentences in the text. in addition to this, we also cover the evaluation strategies introduced to account for improvements in this domain and conclude by presenting avenues for future research. before moving on with the main agenda, we briefly describe the basics of statistical and neural mt models and their evaluation in the following section. ## discourse discourse contains seven fundamental properties including cohesion, coherence, intentionality, acceptability, informatively, situationality and intertextuality @xcite . cohesion and coherence are two most basic properties that establish "connectedness" in a text @xcite . cohesion is a surface property of the text that is realised by explicit clues. it occurs whenever "the interpretation of some element in the discourse is dependent on that of another" @xcite . cohesion is mainly realised by the way of pronominal reference including anaphora and coreference. anaphora is the use of an expression whose interpretation depends specifically upon antecedent expression. the anaphoric (referring) term is called an anaphor. sometimes anaphor may rely on the postcedent expression, and this phenomenon is called cataphora. taking sentence (a) in figure 1 for example, the pronoun it is an anaphor, which points to the left toward its antecedent audi. zero anaphora (pronoun-dropping) is a more complex case of anaphora. in some languages such as chinese and japanese, certain classes of words can be omitted to make the sentence compact yet comprehensible when the identity of the pronouns can be inferred from the context. coreference means two or more expressions (e.g. nouns) in a text refer to the same referent. as the referents point to persons or things in the real world, the coreference relation can exist independently of the context. taking sentence (b) in figure 1 for instance, the noun phrases hk chief executive and mr. tung chee-hwa point to the same person, although their surfaces are totally different. coherence is related to the connectedness of the "mental representation of the text rather than of the text itself" @xcite to make a text semantically meaningful. it is created referentially, when different parts of a text refer to the same entities, and relationally, by means of coherence relations such as "cause-consequence" between different discourse segments. researchers studied discourse structure mainly based on rhetorical structure theory (rst) @xcite and penn discourse tree bank (pdtb) annotation methodology @xcite . rst relations are applied recursively in a text, until all units in that text are constituents in a predefined relation. the result of such analysis is that rst structure are typically represented as trees, with one top level relation that encompasses other relations at lower levels. besides, the tree also contains a number of predefined relations such as "attribution" (cause-consequence), "elaboration" etc. the pdtb annotation methodology is proposed based on rst, but highlights the role of the connectives. according to whether containing a connective or not, discourse relations can be divided into two categories: explicit and implicit. ## discourse structure foster et al. @xcite try the first attempt to incorporate structural information into smt. they tagged each sentence with features such as kind of session, identity of the speaker, time period, and then used domain adaptation methods to balance between a lm trained from similar data and a background lm. marcu et al. @xcite found that there are significant differences in dis-course structure of between japanese and english. thus, they propose an "analysis-transfer-translate" pipeline: firstly, japanese text is parsed into rst tree; and then it is transferred into english style rst tree; finally process translation based on the rst tree. although all the training data are manually annotated (high cost), the method really improves the translation in term of coherence. besides, tu et al. @xcite propose a novel translation framework, which mainly includes three steps: 1) source rst tree acquisition: a source sentence is parsed into an rst tree; 2) rule extraction: translation rules are extracted from the source tree and the target string via bilingual word alignment; 3) rst-based translation: the source rst-tree is translated with translation rules. experiments show that their approach achieves improvements of about +2 bleu points than the baseline system on chinese-english. because of the superior ability to preserve sequence information over time, long short term memory (lstm) has obtained strong results on a variety of sequence modeling tasks. sequence models construct sentence representations as an order-sensitive function of the sequence of tokens. in contrast, tree-structured models compose each phrase and sentence representation from its constituent sub-phrases according to a given syntactic structure over the sentence. tai et al. @xcite introduce a tree-lstm, a generalization of lstms to tree-structured network topologies. the difference between tree-lstm and lstm is that the tree-lstm composes its state from an input vector and the hidden states of arbitrarily many child units. thus, the standard lstm can then be considered a special case of the tree-lstm where each internal node has exactly one child. they show its superiority for representing sentence meaning over a sequential lstm in two tasks: predicting the semantic relatedness of two sentences and sentiment classification. although their works show promising improvements, there are several underlying drawbacks: 1) some models are trained on small-scale or manually-created data sets, it is not reliable when adopting these approaches to large-scale mt task; and 2) the performance of discourse parser is still not reliable, thus incorporating the structure information into nmt will result in error propagation problems. ## document structure one direction is cache-based methods, which employ cache to retain bilingual phrase pairs from the best hypothesis of previously translated sentences and then use it as an additional feature in log-linear model of smt. tiedemann @xcite integrated cache-based language and translation models within a pbsmt decoder and used an exponential decay factor to carry over word preferences from one sentence to the next. when a source phrase is considered for translation, its cache translation score is computed using the phrase probabilities of matching phrases found in the cache and the decay factor. their examples illustrate better translation especially in repetition and consistency, however, the experimental score show modest improvements. gong et al. @xcite extended this work by using three caches: dynamic cache, static cache, and topic cache. they show a better improvement when all three caches are used in combination. other efforts are in document-level decoding. focusing on translation consistency, xiao et al. @xcite employed a forced-decoding method: identify ambiguous words in the output of baseline system, and then obtain a set of consistent translations based on frequencies and finally re-decode input using the filtered set of translation options. hardmeier et al. @xcite approach translation as an optimization task. he proposed a stochastic local search decoding method for pbsmt, which permits free document-wide dependencies in the models. their work on decoding try to reduce the searching space but it is difficult to incorporate new knowledge. recently, researchers began to explore neuralnetwork (nn)-based discourse-level approaches for sequence modeling. conversational models need to predict the next sentence by considering the historical utterances in a conversation. vinyals and le @xcite built an end-to-end conversational system using a sequence-to-sequence framework. in order to capture the lager-context information, they simply concatenate previous utterances together as the input. their preliminary results show that the method is able to converse well and extract knowledge from lager-context. li et al. @xcite argue that simply incorporating context information into context independent message will increase the workload of a generation system and has the risk of bringing in noise to the generation process. to better preserve the original search intent, sordoni et al. @xcite proposed a novel hierarchical recurrent encoder-decoder (hred) model to summarize these historical queries. besides, serban et al. @xcite adopt the framework to the task of dialogue response generation. they use hred to summarize a single representation from both the current and previous sentences. experiments demonstrated that availing of the historical representation helps to maintain the dialogue context. the continuous vector representation of a symbol encodes multiple dimensions of similarity, equivalent to encoding more than one meaning of a word. consequently, nmt needs to spend a substantial amount of its capacity in disambiguating source and target words based on the context defined by a source sentence. without additional information, standard nmt models are facing inconsistency and ambiguity problems. calixto and liu @xcite utilize global image features extracted using a pre-trained convolutional neural network and incorporate them in nmt. the work related to multi-source and multi-target nmt are investigated one-to-many or many-to-one languages translation tasks by integrating additional encoders or decoders into encoder-decoder framework, and their experiments show promising results. more recently, some researchers propose to use an additional set of an encoder and attention to model more information. for example, jean et al. @xcite use it to encode and select part of the previous source sentence for generating each target word. ## evaluation more recently, there are some new publications on discourse-level nmt evaluation. in order to evaluate discourse phenomena in nmt, bawden et al. @xcite conducted experiments from three aspects: 1) comparing multi-encoder models @xcite with different strategies; 2) investigating the impacts of source-and target-side history information on nmt; 3) presenting a novel evaluation through the use of two discourse test sets targeted at coreference and lexical coherence/cohesion. voita et al. @xcite introduced a contextaware model and demonstrated its usefulness for anaphora resolution as well as translation. besides, xiong @xcite proposed mod-eling method to use discourse context and reward to refine the translation quality from the perspective of coherence. some researchers proposed to extend the transformer model to take advantage of discourse-level context @xcite . following the work from tu et al. @xcite , @xcite and @xcite continue to exploit cache memory for improving the performance of discourselevel nmt. through human evaluation, läubli @xcite has found that discourse-level evaluation for mt can improve to discriminate the errors which are hard or impossible to spot at the sentence level. ## dropped pronoun issue as discussed in section 2.3, one issue of cohesion is pronominal anaphora. targeting cohesion phenomena, some researchers investigated approaches of incorporating anaphora information to improve the performance of mt. for instance, le nagard and koehn @xcite presented a method to aid english pronoun translation into french for smt by integrating an anaphora resolution system. in the thesis, we mainly focus on the more complicated phenomenon: dropped pronoun (dp), which can be regarded as a special case of pronominal anaphora. thus, in the following contents, we mainly review related work on dp.