Multimodal Quality Estimation for Machine Translation (2020)

Abstract: We propose approaches to Quality Estimation (QE) for Machine Translation that explore both text and visual modalities for Multimodal QE. We compare various multimodality integration and fusion strategies. For both sentence-level and document-level predictions, we show that state-of-the-art neural and feature-based QE frameworks obtain better results when using the additional modality.

## introduction quality estimation (qe) for machine translation (mt) @xcite aims to predict the quality of a machine-translated text without using reference translations. it estimates a label (a category, such as 'good' or 'bad', or a numerical score) for a translation, given text in a source language and its machine translation in a target language @xcite . qe can operate at different linguistic levels, including sentence and document levels. sentence-level qe estimates the translation quality of a whole sentence, while document-level qe predicts the translation quality of an entire document, even though in practice in literature the documents have been limited to a small set of 3-5 sentences @xcite . existing work has only explored textual context. we posit that to judge (or estimate) the quality of a translated text, additional context is paramount. sentences or short documents taken out of context may lack information on the correct translation of certain (esp. ambiguous) constructions. inspired by recent work on multimodal machine learning @xcite , we propose to explore the visual modality in addition to the text modality for this task. source (en) danskin women's bermuda shorts mt (fr) bermuda danskin féminines court the main contributions of this paper are as follows: (i) we introduce the task of multimodal qe (mqe) for mt as an attempt to improve qe by using external sources of information, namely images; (ii) we propose several ways of incorporating visual information in neural-based and featurebased qe architectures; and (iii) we achieve the state-of-the-art performance for such architectures in document and sentence-level qe. ## multimodal qe we propose different ways to integrate visual features in our two monomodal qe approaches (sections 3.1 and 3.2). we compare each proposed model with its monomodal qe counterpart as baseline, both using the same hyperparameters. ## results we use the standard training, development and test datasets from the wmt'18 task 4 track. for feature-based systems, we follow the built-in crossvalidation in quest++, and train a single model with the hyperparameters found by cross-validation. for neural-based models, we use early-stopping with a patience of 10 to avoid over-fitting, and all reported figures are averaged over 5 runs corresponding to different seeds. we follow the evaluation method of the wmt qe tasks: pearson's r correlation as the main metric @xcite , mean-absolute error (mae) and root-mean-squared error (rmse) as secondary metrics. for statistical significance on pearson's r, we compute williams test @xcite as suggested by @xcite . for all neural-based models, we experiment with the all three integration strategies ('embed', 'annot' and 'last') and all three fusion strategies ('conc', 'mult' and 'mult2') presented in section 3.2. this leads to 6 multimodal models for each birnn and bert-birnn. in tables 2 and 4 , as well as in figures 2 and 3 , we report the top three performing models. we refer the reader to the appendix for the full set of results. ## conclusions we introduced multimodal quality estimation for machine translation, where an external modality -visual information -is incorporated to featurebased and neural-based qe approaches, on sentence and document levels. the use of visual features extracted from images has led to significant improvements in the results of state-of-the-art qe approaches, especially at sentence level. the version of deepquest for multimodal qe and scripts to convert document into sentencelevel data are available on https://github.com/ sheffieldnlp/deepquest .