Machine Translation of Speech-Like Texts: Strategies for the Inclusion of Context (2017)

Abstract: Whilst the focus of Machine Translation (MT) has for a long time been the translation of planned, written texts, more and more research is being dedicated to translating speech-like texts (informal or spontaneous discourse or dialogue). To achieve high quality and natural translation of speechlike texts, the integration of context is needed, whether it is extra-linguistic (speaker identity, the interaction between speaker and interlocutor) or linguistic (coreference and stylistic phenomena linked to the spontaneous and informal nature of the texts). However, the integration of contextual information in MT systems remains limited in most current systems. In this paper, we present and critique three experiments for the integration of context into a MT system, each focusing on a different type of context and exploiting a different method: adaptation to speaker gender, cross-lingual pronoun prediction and the generation of tag questions from French into English.

## introduction speech-like texts (social networking, speech transcriptions, subtitles and other informal written exchanges) 1 pose new challenges for natural language processing (nlp) and in particular machine translation (mt), when compared to the genres of parliamentary discourse and journalistic texts, which have been the main focus of text processing applications to date. as the need for high quality mt of this more informal genre increases, more and more mt research is being dedicated to exploring the difficulties speech-like texts present and how we might hope to overcome them @xcite . speech-like texts are fundamentally different from planned written texts for a number of different reasons. one of these is their highly contextualised nature. whilst it is true that all texts are related to a certain genre and context of production, the extra-linguistic context of a speech-like production is very often essential to understanding its linguistic content and therefore to producing accurate and coherent translations. unlike journalistic texts, which are addressed to a heterogenous audience, absent at the moment of the text's production, in discourse, the speaker and the interaction with the listener can greatly influence the style (in terms of formality and politeness), vocabulary choices and even the grammatical agreement (in languages that have grammatical gender agreement for example) of the texts. spontaneity of production allows speakers to continuously adapt to each other's reactions, making the aforementioned contextual aspects even more important. there is also the added possibility that speakers align to each other's linguistic choices, make reference to entities mentioned in the other speaker's speech turn and rely more on implicit common ground to root their speech, potentially resulting in more ambiguity for automatic processing. traditional mt methods suffer from a considerable flaw when it comes to dealing with context, this being that sentences are processed independently of each other, and translation choices are often made based on very local context, which is especially true of phrase-based statistical mt (smt) techniques. whilst attentional neural mt (nmt) approaches can partially alleviate this problem, the degree to which we can control which context is used in translation is limited. as for extra-linguistic information, it must first be identified and then integrated into the translation process, through pre-processing, during decoding or as a post-edition step. we illustrate the different strategies that can be used to integrate contextual information into the mt process by presenting a discussion orientated around three separate experiments for the language pair english-french, each illustrating the integration of a different type of contextual information. the scope and nature of context are very different in each case, presenting different challenges to mt. the experiments described illustrate the nature of the different problems faced, as well as provided a basis for the critique of the methods used, highlighting the fact that the problem is far from being solved. the first experiment is an illustration of the use of domain adaptation, a very simple technique used to take into account a very coarse-grained and static type of information, that of speaker gender, which applies on the sentence-level (section 2). the second experiment deals with a finer notion of context, necessitating a more sophisticated modelling of an utterance's linguistic context, in a task to predict the french translation of the english subject pronouns it and they, which, unlike their equivalent french pronouns, are unmarked for gender (section 3). in this case, a very specific contextual element (the antecedent) determines the translation of a very specific textual element (the pronoun). finally, in section 4, we present our third experiment, a post-editing task to improve the translation of a particular stylistic phenomenon common in spoken english, the english tag question, whose usage is largely determined by speaker attitude and the interaction between speaker and listener. whilst the phenomenon affected by context (the presence and form of the english tag question) is clearly identified, determining which context can be useful is much more complex. speaker identity (as defined by demographic factors such as age, gender and social background, as well as other aspects such as personality and mood) greatly influences the way in which speakers communicate, in terms of lexical choices, syntactic structure, politeness strategies, etc. this variation has long been studied in the sociolinguistics literature, in particular with respect to the impact of gender on communication style @xcite . whilst speaker adaptation has long been a part of speech recognition systems @xcite , only recently has research in nlp turned towards adapting systems to take into account this extra-linguistic information (cf. works by @xcite for sentiment analysis and @xcite for three separate nlp classification tasks). speaker identity is just as important in mt; @xcite show how mt performance fluctuates with respect to individual speaker identity, gender and register in five different language pairs for film and tv subtitles, and @xcite find that mt translated output does not conserve speaker personality and gender traits as well as human translations do. one of the simplest methods for integrating coarse-grained, sentence-level information such as aspects of speaker identity is to perform domain adaptation. domain in this context refers to a broader concept than the genre of text and can include any sentence-level aspect representing the class of sentence, such as sentence type, speaker gender and formality. domain adaptation is a well-known approach in mt and is used almost systematically. in its simplest form, it implies simply choosing an in-domain development set to tune the system (cf. @xcite for a study of the effectiveness of in-domain tuning for smt systems when dealing with a low-resource domain). but its implementation can be more complex, for example through similarity-based data selection techniques for both training and tuning data @xcite or by weighting different components of the mt system that have been trained on class-partitioned data @xcite . ## linguistically motivated cross-lingual pronoun prediction other forms of context necessarily require a more sophisticated representation of the linguistic properties of discourse. one of the most studied contextual aspects of discourse is coreference and the translation of pronouns, particularly between languages that do not have the same system of grammatical gender @xcite . take for example the language pair english-french. french common nouns and pronouns are marked for grammatical gender, whereas the english pronouns it and they are not. in order to correct translate the pronoun it in the english sentence "it was blue", it is necessary to know the grammatical gender of the french coreferential antecedent of the pronoun. for example, if it refers to a box (une boîte), the correct translation is the feminine pronoun elle, whereas if it refers to a toe (un orteil), the correction translation is the masculine pronoun il. there is also a third option (which can be used in certain contexts) to translate it using the gender-neutral demonstrative ce. the problem of ensuring accurate pronominal translation has received much interest in the discourse in mt community and a shared task has been organised for the past few years, dedicated to providing solutions to the problem (e.g. @xcite . the task is a cross-lingual pronoun prediction task, for which participating classification systems aim to correctly predict target pronoun forms in the target sentence, based on contextual information in the source and target sentences. we focus on one of the language directions proposed by the 2016 task: english into french, for which the aim was to predict the correct french translation of the english subject pronouns it and they, from the set of possible classes il, ils, elle, elles, on, ce, cela and other. an example of the data provided for the task is shown in figure 3 . target sentences were provided as lemma-tag pairs, rather than as surface forms, to avoid systems relying solely on the grammatical gender of the immediately surrounded words to determine gender, which is an unrealistic scenario in a real translation setting. 4 pronoun class lemma+tag english source sentence french target sentence word alignment ils il|pron they 're extremely costly . replace_0 être|ver très|adv coûteux|adj .|. 0-0 1-1 2-2 3-3 4-4 figure 1 : an example of pronoun prediction task data at wmt 2016. the first column represents the class label to be predicted and the second column the lemma and tag of the pronoun. the position of the pronoun to be predicted is marked by the placeholder replace_0 in this example. a variety of different strategies were used by task participants, who relied on different amounts of contextual, linguistic information. the winning system @xcite was a stacked recurrent neural network (rnn) system, which did not rely on any other contextual information than the information provided (as shown in figure 3 ) and did not look beyond sentence boundaries. a second neural network approach also scored highly in the task: dabre et al. (2016) used a simple rnn architecture with an attention mechanism and trained only using iwslt data (whereas the winning system was trained on all data provided). many of the other participants, including the current authors, chose to concentrate on the integration of more linguistic information into simpler (mainly linear) classification systems. many participants used part-of-speech (pos)-tagging, parsing and coreference chains. @xcite system, a linear classifier using multiple linguistic annotations, was ranked second out of nine systems, showing that simple linear classifiers can perform just as highly as more sophisticated architectures such as neural network systems if good quality linguistic information is used. ## stylistic choices: the case of tag questions the final focus of this paper is on an even more complicated notion of context, which is more difficult to characterise than the two previous types of contexts discussed in the two previous sections. unlike speaker identity or grammatical gender in coreference chains, which are deterministic and easily discernible, we focus here on one particular aspect of stylistic choice, the english tag question. common in spoken english (particularly british english), the tag question, which is best known in its canonical form, formed of an auxiliary verb and a pronoun of the form isn't it?, can we? or would you?, etc., is a much studied phenomenon in the field of linguistics @xcite . according to the longman grammar @xcite , approximately 20% of the questions found in the conversational part of the corpus they analysed were tag questions. question tags are peripheral interrogative elements, typically appended to a declarative sentence, with the effect of modifying the sentence, to express a range of different attitudes, including doubt, surprise, contempt, etc. or simply to facilitate the flow of conversation. there are two main types of tag question, grammatical tag questions, also known as canonical tag questions, which are grammatically bound to the main anchor clause, for example: however, there also exist lexical (or invariant) tag questions, which are not grammatical bound in the same way, and are in fact the most frequent form of tag question, also commonly appearing in languages other than english. for example: the function of tag questions is complex and multi-facetted. they have often been referred to as dialogue facilitators, as ways of "keeping the conversation going" and "inviting listeners to communicate" @xcite . however they have also been analysed as being modalisers to the main proposition of the utterance, modifying it to express the speaker's attitude to the proposition @xcite . this bias based on the speaker's belief in the truth of the initial proposition is also linked to the understanding and common knowledge between speaker and interlocutor. the choice of question tag can have an important impact on the meaning of an utterance, in particular concerning speaker certainty, politeness and conduciveness. for example, a "they're a bit strange, aren't they?" suggests that the speaker thinks them to be strange, but demands confirmation from the listener in an inviting way, whereas "they're a bit strange, are they?" suggests that the listener has stated they are a bit strange and the speaker is questioning this, potentially in a slightly aggressive manner. tag questions are notably difficult to translate automatically because there are no direct equivalents to the english system of tag questions in other languages, with respect to canonical (grammatical) tag questions of the form "isn't it?","won't you?". 6 the form of tag questions may be considered relatively easy to predict, once an utterance is known to be a tag question, because the grammatical structure of a tag question in most cases echoes the auxiliary verb of the main clause. however, there is also the option to use a lexical tag question (e.g. "right?", "see?"), which complicates this decision. the most difficult decision is to know when to produce a tag question when translating from a language that uses different strategies for the same communicative purpose. smt systems, which only use a very notion of local, immediate context, often struggle to provide a grammatical and coherent tag question. nmt systems theoretically far better, but the task of generating a tag question where one does not appear in the source sentence is still a considerable challenge. ## conclusion context, in its many different forms, is crucial when translating speech-like texts. we have shown, through three different experiments on three aspects of contextual information, how integrating context into the translation process can help translation performance, whether during decoding as with our first experiment on domain adaptation or as a post-edition step as with the last two experiments. however, these methods remain limited. domain adaptation by data partitioning restricts the amount of data used, pronoun predicting using external resources is, at present, limited by the lacunas in automatic coreference resolution, and we are far from being able to model the many different aspects of style, which, like tag questions, can greatly impact translation quality. the arrival of neural learning methods, which provide a promising architecture for taking into account richer linguistic information @xcite , pave the way for integrating rich, contextual information into mt systems, and importantly, could provide a unified approach for the integration of the different types of context.