C ross-modal L anguage G eneration using P ivot S tabilization for W eb-scale L anguage C overage (2020)

Abstract: Cross-modal language generation tasks such as image captioning are directly hurt in their ability to support non-English languages by the trend of data-hungry models combined with the lack of non-English annotations. We investigate potential solutions for combining existing language-generation annotations in English with translation capabilities in order to create solutions at web-scale in both domain and language coverage. We describe an approach called Pivot-Language Generation Stabilization (PLuGS), which leverages directly at training time both existing English annotations (gold data) as well as their machine-translated versions (silver data); at run-time, it generates first an English caption and then a corresponding target-language caption. We show that PLuGS models outperform other candidate solutions in evaluations performed over 5 different target languages, under a large-domain testset using images from the Open Images dataset. Furthermore, we find an interesting effect where the English captions generated by the PLuGS models are better than the captions generated by the original, monolingual English model.

## introduction data hungry state-of-the-art neural models for language generation have the undesired potential to widen the quality gap between english and non-english languages, given the scarcity of non-english labeled data. one notable exception is machine translation, which benefits from large amounts of bilingually or multilingually annotated data. but cross-modal language generation tasks, such as automatic image captioning, tend to be directly hurt by this trend: existing datasets such as flickr @xcite , mscoco @xcite , and conceptual captions @xcite have extensive labeled data for en- image tgt train generate translate ttg translate train generate plugs pivot language generation stabilization das logo ist auf dem computer zu sehen. (the logo can be seen on the computer.) bild mit dem titel live mit einem schritt (image titled live with a step) the iphone is seen in this undated image . <@xmath0. autoverkehr an einem regnerischen tag (car traffic on a rainy day) polizeiauto auf der straße (police car on the street) a car in the @xmath1. we call the latter the pivot-language generation stabilization (plugs) approach. examples of outputs produced by these three solutions are shown in fig. 1 . we perform extensive evaluations across five different languages (french, italian, german, spanish, hindi) to compare these three approaches. the results indicate that the bilingual plugs models consistently perform the best in terms of captioning accuracy. since there is very little support in the literature regarding the ability of standard evaluation metrics like bleu @xcite , rouge @xcite , meteor @xcite , cider @xcite , and spice @xcite to accurately measure captioning accuracy for non-english languages, our evaluations are done using fine-grained, side-by-side human evaluations using paid raters; we explain the evaluation protocol in detail in sec. 5. besides the evaluations on bilingual plugs models, we also train and evaluate a multilingual plugs model, in which all five non-english languages considered are supported through a single model capable of generating outputs in all 5 languages. the results indicate that similar languages are reinforcing each other in the common representation space, showing quantitative gains for the romance languages involved in our experiments. a related but perhaps less expected result is that the english captions generated by plugs models (what we call the stablizer outputs) are better, as measured using side-by-side human evaluations, than captions generated by the original, monolingual english model. there is a final additional advantage to having plugs models as a solution: in real-world applications of image captioning, quality estimation of the resulting captions is an important component that has recently received attention @xcite . again, labeled data for quality-estimation (qe) is only available for english 2 , and generating it separately for other languages of interest is expensive, time-consuming, and scales poorly. the tgt approach could directly apply a qe model at run-time on the en caption, but the subsequent translation step would need to be perfect in order not to ruin the predicted quality score. the ttg ap-proach cannot make use at run-time of an en qe model without translating the caption back to english and thus again requiring perfect translation in order not to ruin the predicted quality score. in contrast, the plugs approach appears to be best suited for leveraging an existing en qe model, due to the availability of the generated bilingual output that tends to maintain consistency between the generated en-& x-language outputs, with respect to accuracy; therefore, directly applying an english qe model appears to be the most appropriate scalable solution. ## related work there is a large body of work in automatic image captioning for english, starting with early work @xcite @xcite @xcite based on data offered by manually annotated datasets such as flickr30k @xcite and ms-coco @xcite , and more recently with work using transformer-based models @xcite @xcite based on the web-scale conceptual captions dataset @xcite . generating image captions in languages other than english has been explored in the context of the wmt 2017-2018 multimodal translation sub-task on multilingual caption generation @xcite . the goal of the task is to generate image captions in german and french, using a small training corpus with images and captions available in english, german and french (based on flickr30k). in the context of that work, we use the results reported in @xcite to quantitatively compare it against our approach. another relevant connection is with the work in @xcite , which explores several lstm-based encoder-decoder models that generate captions in different languages. the model most similar to our work is their dual attention model, which first generates an english caption, then an lstm with attention over the image and the generated english caption produces a german caption. their quantitative evaluations do not find any additional benefits for this approach. our work is related to this idea, but there are key technical differences. in the plugs approach, we train an end-to-end model based on a transformer @xcite decoder that exploits the generated english-prefix via the self-attention mechanism to learn to predict the non-english target caption, conditioned on the english tokens at multiple levels through the decoder stack. moreover, we approach this study as the search for a solution for web-scale multi-language image captioning: we employ the web-sized conceptual captions dataset for training, and consider the effects of using captions across multiple languages, as well as multi-language/single-model setups. ## model architecture we model the output caption using a sequencegeneration approach based on transformer networks @xcite . the output is the sequence of sub-tokens comprising the target caption. as shown in fig. 2 , the input sequence is obtained by concatenating the following features. ## datasets we perform our experiments using two different benchmarks. we use the multi30k @xcite dataset in order to compare the effect of the plugs model using a resource that has been widely used in the community. we focus on task 1 for french from @xcite , generating a translation in french based on an image and an english caption as input. the training set consists of images from the flickr30k train and validation splits, along with the corresponding french captions. the validation split consists of test2016 images and captions, and the test split consists of the test2017 images and captions. for the core results in this paper, we use the conceptual captions dataset @xcite as our english-annotated generation labels, in order to capture web-scale phenomena related to image captioning. in addition, we use google translate as the translation engine (both for the run-time translations needed for the tgt approach and the training-time translations needed for the ttg and plugs approaches), targeting french, italian, german, spanish, and hindi as target languages. we use the standard training and validation splits from conceptual captions for developing our models. we report the results using a set of 1,000 randomly samples images from the open images dataset @xcite . we refer to this test set as oid1k when reporting our results. ## evaluation in the experiments done using the multi30k dataset, we are reporting results using the me-teor @xcite ) metric, in line with previous work. for the experiments performed using the conceptual captions dataset, we have found that automated evaluation metrics for im-age captioning such as bleu @xcite , rouge @xcite , cider @xcite , and spice @xcite cannot accurately measure captioning accuracy for non-english languages. however, we are reporting cider numbers as a point of comparison, and contrast these numbers with human evaluation results. we describe the human evaluation framework we use next. ## conclusions we present a cross-modal language generation approach called plugs, which successfully combines the availability of an existing gold annotation (usually in english) with the availability of translation engines that automatically produce silver-data annotations. the result is a multilingual engine capable of generating high-quality outputs in the target languages, with no gold annotations needed for these languages. we show that, for image captioning, the plugs approach out-performs other alternatives, while also providing the ability to pack multiple languages in a single model for increased performance. surprisingly, by considering the generated outputs in the original language of the annotation (stabilizer outputs), we find that the quality of the stabilizers is higher compared to the outputs of a model trained on the original annotated data. overall, our results can be understood as a successful instance of transfer learning from a unimodal task (text-to-text translation) to a crossmodal task (image-to-text generation), which allows us to indirectly leverage the abundance of text-only parallel data annotations across many languages to improve the quality of an annotation-poor cross-modal setup.