Detecting context-dependent sentences in parallel corpora (2018)

Abstract: In this article, we provide several approaches to the automatic identification of parallel sentences that require sentence-external linguistic context to be correctly translated. Our long-term goal is to automatically construct a test set of context-dependent sentences in order to evaluate machine translation models designed to improve the translation of contextual, discursive phenomena. We provide a discussion and critique that show that current approaches do not allow us to achieve our goal, and suggest that for now evaluating individual phenomena is likely the best solution.

## introduction recent work in machine translation (mt) has focused on using information beyond the current sentence boundary to aid translation @xcite @xcite . the aim of these contextual mt systems is to remedy the flaw of traditional mt of translating sentences independently of each other, in particular to improve the translation of discourse phenomena. despite the progress made in incorporating linguistic context into @xcite , these gains are often not observable using automatic evaluation metrics, such as bleu @xcite , and manual analysis of translations is often anecdotal. whilst strategies such as producing contrastive sentence pairs to be reranked by mt models is a promising strategy for evaluation @xcite , producing the test sets is often time-consuming and unrepresentative of real data. morevoer, the distinction is often lacking between examples that need extra-sentential context to be translated and those that do not. a very useful addition to the test suites available would therefore be a test set of real, attested examples that require extra-sentential linguistic context to be correctly translated, as this would enable us to evaluate the progress made by contextual mt models specifically on the most difficult examples. since manually identifying real sentences is very time-consuming, our long-term goal is to automatically construct such a test set. in this paper, we aim to show that designing and implementing a method of automatically detecting these sentences in a parallel corpus remains problematic, as shown by a reflection on what such a method would entail and preliminary experiments using tools currently at our disposal to implement it. we begin by discussing the types of phenomena we wish to identify (section 2) and existing work on evaluating discourse phenomena (section 3). we then define the goals and principles such an identification method would adhere to (section 4). finally, in section 5, we critique two possible approaches to the problem, suggesting theoretical limitations of each approach. our hope is for this work to provide the basis for discussion on modelling contextual phenomena in a multilingual setting, in a view to automatically identifying context-dependent sentences in the long term. ## context-dependent phenomena in practice, many sentences can be correctly translated in isolation, without surrounding context, which explains why most mt systems today translate sentences independently of each other. however certain phenomena, mostly related to discourse, whose scope, by definition is defined at the discourse rather than the sentence level of discourse, cannot be systematically and correctly translated without extra context. examples include anaphoric pronoun translation @xcite @xcite , lexical disambiguation @xcite and cross-lingual discourse connective prediction @xcite . these phenomena have a common characteristic: they are cross-lingually ambiguous and can only be disambiguated with the help of linguistic context. this linguistic context can appear within the sentence containing the ambiguous element or elsewhere in the text, in which case we refer to it as extra-sentential linguistic context. cross-lingual ambiguity occurs because of mismatches in the language systems of the source and target languages, such that there are several translations possible out of context and only one correct in context. 1 the ambiguity can be morphological, syntactic, semantic and/or discursive. one example of the morphological level is the translation of anaphoric pronouns, which poses difficulties in mt due to structural differences in gender marking cross-lingually. for example, the french translation of english it is ambiguous between variants il (masc.) and elle (fem.), depending on the gender of the french noun with which the pronoun corefers. on the syntactic level, ambiguity can arise from an inherent ambiguity in the source language that is not preserved in the target language. for example, english 'green chestnuts and pears' is ambiguous between french 'des marrons verts et des poires' (only the chestnuts are green) and 'des marrons et des poires verts' (chestnuts and pears are green). cross-lingual semantic ambiguity is where semantic ambiguity in the source language is not preserved in the target language and a choice must be made between the different meanings. for example, the english word spade is ambiguous between french bêche 'gardening implement' and pique 'suit of cards' (cf. work on word sense disambiguation in mt by @xcite ). 2 finally, at the discursive level, elements such as discourse connectives are often language-specific and are often expressed differently cross-lingually (cf. work on implicitation of discourse connectives by @xcite ). although the examples given above are relatively well-studied phenomena, particularly in a monolingual setting (coreference resolution, word sense disambiguation, discourse relation labelling, etc.), this cannot be seen as an exhaustive list of context-dependent phenomena. ideally, it would be useful to study translation quality on all types of context-dependent sentences, not just those in a pre-defined list, especially as they are dependent on the particular language pair. ## evaluating discourse in mt evaluating discourse and other context-dependent phenomena in mt poses a problem for two main reasons. 3 firstly, most sentences do not require context to be translated. when they do, few words are affected by an incorrect translation relative to the total number of words in the dataset, despite the fact that these errors can be seriously detrimental to the understanding of the translation. 4 secondly, the correct translation of certain discourse phenomena, including anaphoric pronoun prediction, depends on previously made translation choices (ensuring translation coherence), ruling out metrics that rely on comparing surface forms of the predicted translation with a reference translation. as interest in contextual mt surges, the question of how to correctly evaluate the impact of the added context has not been far behind, with different solutions for evaluation, both manual and automatic, being proposed, aimed to overcome the problems described above. in terms of manual evaluation, the aim has been to construct a corpus containing only examples of interest in order to combat the sparsity problem cited above. isabelle et al. (2017) provides such a set of test examples designed to test different well-known problems faced by mt systems, including discourse phenomena. two solutions have been proposed for automatic evaluation of specific phenomenon. the first, adopted by shared task organisers for both the cross-lingual pronoun prediction task at wmt'16 and discomt'17 @xcite and the cross-lingual word sense disambiguation (wsd) task at @xcite , is to change the nature of the task, and to evaluate the models' ability to translate solely the word of interest, whilst the rest of the translation is imposed for all contestants. this alleviates the second problem of translation coherence. the second automatic method, which also involves avoiding comparison of different models' translations, is to evaluate the capacity of mt models to rerank different hypotheses. presenting models with contrastive pairs of examples and comparing the models on their ability to rank the correct hypothesis higher than an incorrect one is a way of indirectly evaluating them (cf. (sennrich, 2017) for grammatical errors @xcite for wsd and (bawden et al., 2018) for coreference and lexical coherence/cohesion. aside from @xcite , which imposes that the disambiguating context occur in the previous sentence, the other automatic evaluation methods do not control for the fact that the disambiguating context can appear within the current sentence or beyond the sentence boundary. 5 this means that many examples in the sets can be resolved using sentence-internal context and therefore do not directly evaluate the ability of contextual models to use context beyond the current sentence. this notably proved problematic for the evaluation of the 2016 pronoun task, of which the highest performing model did not use any extra-sentential context (achieved higher scores based on the inter-sentential examples alone). a useful complement to these test suites would therefore be a method of automatically constructing a test set of sentences that require linguistic context to be correctly translated. the advantages of such a method would be its automatic nature, given the difficulty of manually finding representative examples of context-dependent phenomena and the fact that it could potentially find more diverse phenomena than a human annotator is capable of finding. ## automatic context-dependent sentence detection our long-term goal is to propose and develop a method of identifying real corpus examples that are cross-lingually ambiguous and necessarily require extra-sentential context (as opposed to intrasentential context) to be correctly translated. 6 in theory, such a method would separate parallel sentences for which all information needed to produce the target sentence is found within the source sentence (non-context-dependent) from those for which part of the information can only be found in the surrounding sentences (context-dependent). ## comparison of methods an ideal method would be one relying on complete and comparable representations of the source sentence and of the target sentence both with and without linguistic context. intuitively, for contextdependent sentences to be correctly translated, the information present in the representation of the target sentence would be impossible to reconstruct from the representation of the source sentence, unless the information from the context is also included. we look at two different approaches for simulating this idealised scenario, working (i) at the sentence level and (ii) at the word level. ## conclusion we have described and motivated a theoretically interesting task of identifying sentences that are cross-lingually ambiguous and dependent on extra-sentential linguistic context. beyond translation, this could have a wider impact on nlp applications, including dialogue generation and understanding. through a reflection on the pre-requisites for such a detection method, and by exploring two different approaches to the problem, we have found that the task is very ambitious. the limitations identified have shown us that as long as complete and robust representations of all information within sentences are not yet achievable, the task of identifying context-dependent sentences using a method that is agnostic to the type of phenomenon is unlikely to be attainable. for now it appears that detecting contextual phenomena is better performed on a per-phenomenon basis.