Investigating gender adaptation for speech translation (2016)

Abstract: In this paper we investigate the impact of the integration of context into dialogue translation. We present a new contextual parallel corpus of television subtitles and show how taking into account speaker gender can significantly improve machine translation quality in terms of B LEU and M ETEOR scores. We perform a manual analysis, which suggests that these improvements are not necessary related to the morphological consequences of speaker gender, but to more general linguistic divergences.

## introduction journalistic and planned texts have long been the focus of attention in the statistical machine translation (smt) community largely due to the availability of large parallel corpora from parliamentary debates and the relative lack of data in other domains. over the last decade, more and more emphasis has been placed on speech translation (hardmeier, 2012), made possible thanks to the availability of speech-like parallel corpora such as the ted talks and parallel subtitles (lison & tiedemann, 2016). @xcite the translation of speech-like texts is a challenge for standard smt systems (trained on journalistic or parliamentary corpora) due to the very different nature of the texts; linguistically, spoken sentences are often ungrammatical, incomplete and contain a greater degree of lexical diversity, but they must also be situated within a dialogue context, in which prosody, common ground and speaker information can offer the only ways of providing a correct translation. one such example is that of french gender agreement, in utterances such as "i am tired", translated as "je suis fatiguée" for a female speaker and "je suis fatigué" for a male speaker, for which speaker gender is the only possible way of determining the correct adjectival suffix. @xcite in this work, we aim to evaluate the impact of integrating external contextual information on translation quality. we present two contributions: (i) the creation of a contextualised parallel corpus of spontaneous dialogues, taken from television subtitles adapted from the tvd dataset (roy et al., 2014) and (ii) exploratory experiments on adapting translation systems to one example of contextual information: the gender of the speaker. we provide preliminary results for the translation of english to french subtitles, using automatic evaluation metrics, as well as a manual evaluation of improvements. statistical approaches to machine translation heavily depend on training data used, and therefore adapting systems to specific corpora is a very natural move. the choice of data used to tune model parameters is also highly important, as shown by pecina et al. (2012) when tuning an out-of-domain model using in-domain data. gender-dependent modelling is a common technique used in speech recognition to adapt models to acoustic differences between men and women's speech (wahlster, 2000). however model adaptation is not restricted to speech recognition; kübler et al. (2010) use dialogue act tags to improve a pos tagger, and in machine translation, adaptive modelling has been used to create topic-dependent (foster & kuhn, 2007) and sentence-type-dependent (finch et al., 2009) models by training and tuning on class-partitioned data. our paper is organised as follows. we first describe our tv series corpus production process (section 2). in section 3 we present the baseline models, based on pre-existing corpora. in section 4 we suggest and evaluate a number of adaptations of the baseline models to gender-specific data. finally, section 5 provides a more detailed analysis of these changes through a contrastive manual evaluation between gender-specific and baseline models. ## the big bang theory reproducible corpus the tvd corpus: we focus on the translation from english to french of a small but contextually rich subtitle corpus: the first two seasons of the american television series the big bang theory. the dataset is generated using the tvd plugin (roy et al., 2014), developed to provide reproducible datasets, exploiting visual, auditory and textual data, directly extracted from dvds and freely available web sources. @xcite this is an important addition to primarily textual resources such as opensubtitles, which provide little extra contextual information. amongst the numerous contextual elements available (speech turns, audio signal and images), we choose, in this preliminary work, to focus on and integrate one aspect of this contextual information into translation -the gender of the speaker. extracting an english-french parallel subtitle corpus: to use the corpus for the purpose of translation, we first extracted an enriched parallel corpus from the big bang theory tvd plugin. the text is taken from the official ocr-extracted subtitles, 4 since, unlike the manual transcriptions, they exist in multiple languages and are official translations. following some correction of ocr errors using a small error lexicon and some manual rules, we align the french and english subtitles using their timestamps. since there is not always a one-to-one mapping between french and english subtitles, and a perfect temporal alignment is not always present, heuristics are used to concatenate subtitles where necessary in order to create parallel subtitle blocks. assigning gender to subtitles: gender is assigned to subtitles by manually mapping speaker names to their corresponding gender. speaker identities are provided by manual transcripts, foot_4 which are automatically aligned to the audio signal (bredin et al., 2014). we then transfer speaker identities to the subtitles, based on transcript and subtitle timestamps. we leave the task of using automatically predicted gender to future work; here we only consider reference genders to test our hypothesis. @xcite division into train, development and test sets: we divide this corpus into three datasets: bbttrain (the first 40 episodes), used to train translation and language models, bbt-dev (the next 6 episodes), used for tuning and bbt-test (the last 6 episodes), used for evaluation. we partition each set into two subsets, one for each gender. basic corpora statistics can be found in ## baseline systems: testing pre-existing corpora the first step in translating new data is to see how existing data fares for training a standard translation system. we provide baseline models using two pre-existing parallel corpora: europarl (koehn, 2005) and, more adapted to our domain, (though less commonly used) the film and television subtitle corpus opensubtitles (lison & tiedemann, 2016). @xcite as this corpus is very large, we used the modified moore-lewis (mml) filtering algorithm (axelrod et al., 2011) to keep only the 8% of opensubtitle sentences most similar to bbt-train. we hereafter refer to this set as opensubs-mml. a comparison of corpora is given in translation setup: we use the moses toolkit (koehn et al., 2007) for translation. alignments are computed over all three training sets. all language models are 4-gram models with kneser-ney smoothing. parameters are estimated on bbt-dev using kbmira to optimise the bleu score. for each corpus, a separate phrase table and language model are produced. multiple language and translation models are combined using the default moses model combination approach, and are assigned weights during the tuning step. results: we evaluate the different models using two metrics: bleu, and meteor. @xcite results are shown in ## gender-based adaptative modelling taking our best baseline system (opensubs-mml+europarl), we propose a series of adaptations to take speaker gender into account : (i) changing the tuning data, (ii) adding a gender-specific phrase table, (iii) adding a gender-specific language model, and (iv) adding both a gender-specific phrase table and language model. the additional gender-specific models are estimated using bbt-train sentences uttered by either male or female speakers. we distinguish three types of tuning data: 'all' corresponding to the entire bbt-dev, 'female' to sentences by female speakers and 'male' by male speakers. we test each of the models individually on the female and male bbt-test data. the results ( the improvement between the baseline and contextualised model was greater for female speakers, with an improvement of +1.09 bleu for the configuration (+lm f /female). when the gender-adapted models are used, (and the baseline for unknown genders), we see an improvement in scores for bbt-dev (24.61 bleu, 0.433 meteor) and bbt-test (25.11 bleu, 0.444 meteor). ## discussion and analysis of improvements we base our discussion on the differences between baseline predictions (∅/all) and those from the model that gave the greatest improvements for each gender: (lm m +tm m /male) and (lm f /female). given the language pair (english to french), one type of error that we could hope to have corrected is that of gender agreement, particularly for adjectives and past participles, in sentence such as "i am happy", translated as "je suis contente" for a female speaker but "je suis content" for a male speaker. however given the small size of our dataset, the cases of this phenomenon are few; we manually identify 11 cases in the female test set which, given the lexical choice, could have resulted in a correction of gender, only one of which actually resulted in a correction. @xcite so what are the improvements down to? as for many statistical systems, the improvements appear to be diverse and specific to the data used. to better understand the differences found between the baseline predictions and those from the gender-adapted models, we manually compared the quality of these two sets of translations and annotated their differences (see these observations suggest that the difference in bleu score might result from differences in sentence length; the bleu metric heavily penalises translation hypotheses that are shorter than the reference and, as shown in ## conclusion we have shown that it is possible to improve manual and automatic evaluation scores when testing our subtitle corpus on baseline and gender-adapted models. a manual evaluation indicates that these preliminary results do not yet enable us to link these improvements to gender-specific linguistic phenomena such as gender agreement. improvements appear to be due to other specificities of the datasets, such as average sentence length, and the higher gains for female speakers are almost certainly linked to class imbalance in the data. further investigations will be needed to fully understand our results. we also intend to extend the work to a larger and more balanced dataset, in order to see whether further improvements can be made. automatically identifying speaker gender will also enable us to forgo the need for manual transcriptions. we will also turn to other types of contextual information, such as the audio signal and speech turns, to improve translation using dialogue structure.