Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination (2023)

Abstract: In this work, we investigate a more realistic unsupervised multimodal machine translation (UMMT) setup, inference-time image-free UMMT, where the model is trained with source-text image pairs, and tested with only source-text inputs. First, we represent the input images and texts with the visual and language scene graphs (SG), where such fine-grained vision-language features ensure a holistic understanding of the semantics. To enable pure-text input during inference, we devise a visual scene hallucination mechanism that dynamically generates pseudo visual SG from the given textual SG. Several SG-pivoting based learning objectives are introduced for unsupervised translation training. On the benchmark Multi30K data, our SG-based method outperforms the best-performing baseline by significant BLEU scores on the task and setup, helping yield translations with better completeness, relevance and fluency without relying on paired images. Further in-depth analyses reveal how our model advances in the task setting.

## introduction current neural machine translation (nmt) has achieved great triumph @xcite @xcite , however in the cost of creating large-scale parallel sentences, which obstructs the development of nmt for the minor languages. unsupervised nmt (umt) has thus been proposed to relieve the reliance of parallel corpora @xcite . the core idea of umt is to align the representation spaces between two languages with alternative pivot signals rather than parallel sentences, such as bilingual lexicons @xcite , multilingual language models (lm) @xcite and back-translation technique @xcite . recent trends have considered the incorporation of visual information, i.e., multimodal machine translation (mmt) @xcite . intuitively, visual modality can serve as language-agnostic signals, pivoting different languages by grounding the same textual semantics into the common visual space. therefore, solving umt with visual contents as pivot becomes a promising solution, a.k.a., unsupervised mmt (ummt) @xcite . ummt systems are trained with only the textimage pairs (<text-@xmath0, which can be easier to collect than the parallel source-target sentence pairs (<src-@xmath1. although exempting the parallel sentences for training, ummt still requires such text-image pairs as inputs for testing. yet such assumption might be unrealistic, because in most of the real-world scenarios such as online translation systems, paired images are not available during inference. especially for some scarce languages, @xmath2. in other words, practical ummt systems should not only avoid the parallel sentences during training, but also the text-image pairs during inference. as summarized in in this work, we present a novel ummt method that solves all aforementioned challenges. first of all, to better represent the visual (also the textual) inputs, we consider incorporating the visual scene graph (vsg) @xcite and language scene graph (lsg) @xcite . the scene graphs (sg) advance in intrinsically depicting the semantic structures of texts or images with rich details (cf. fig. 1 ), which offers a holistic viewpoint for more effective pivoting learning. then, we build the ummt framework as illustrated in fig. 2 . the input src text and paired image are first transformed into lsg and vsg, which are further fused into a mixed sg, and then translated into the tgt-side lsg. and the tgt sentence will be finally produced conditioned on the tgt lsg. several sgbased pivoting learning strategies are proposed for unsupervised training of ummt system. in addition, to support pure-text (image-free) input during inference, we devise a novel visual scene hallucination module, which dynamically generates a hallucinated vsg from the lsg compensatively. our system is evaluated on the standard mmt multi30k and nmt wmt data. extensive experimental results verify that the proposed method outperforms strong baselines on unsupervised multimodal translation by above 5 bleu score on average. we further reveal the efficacy of the visual scene hallucination mechanism in relieving the reliance on image inputs during inference. our sg-pivoting based ummt helps yield translations with higher completeness, relevance and fluency, and especially obtains improvements on the longer sentences. overall, we make the following contributions: ▶ 1) we are the first to study the inferencetime image-free unsupervised multimodal machine translation, solved with a novel visual scene hallucination mechanism. ▶ 2) we leverage the sgs to better represent the visual and language inputs. moreover, we design sg-based graph pivoting learning strategies for ummt training. ▶ 3) our model achieves huge boosts over strong baselines on benchmark data. code is available at https: //github.com/scofield7419/ummt-vsh . ## related work neural machine translation has achieved notable development in the era of deep learning @xcite @xcite . the constructions of powerful neural models and training paradigms as well as the collection of large-scale parallel corpora are the driving forces to nmt's success @xcite . the key of nmt is to learn a good mapping between two (or more) languages. in recent years, visual information has been intro-duced for stronger nmt (i.e., multimodal machine translation), by enhancing the alignments of language latent spaces with visual grounding @xcite . intuitively, people speaking different languages can actually refer to the same physical visual contents and conceptions. unsupervised machine translation aims to learn cross-lingual mapping without the use of largescale parallel corpora. the setting is practically meaningful to those minor languages with hard data accessibility. the basic idea is to leverage alternative pivoting contents to compensate the parallel signals based on the back-translation method @xcite , such as third-languages @xcite , bilingual lexicons @xcite or multilingual lm @xcite . the visual information can also serve as pivot signals for umt, i.e., unsupervised multimodal machine translation. comparing to the standard mmt that trains @xmath3, ummt takes as input only @xmath4. so far, few studies have explored the ummt setting, most of which try to enhance the back-translation with multimodal alignment mechanism @xcite @xcite . scene graph describes a scene of an image or text into a structure layout, by connecting discrete objects with attributes and with other objects via pairwise relations @xcite . as the sgs carry rich contextual and semantic information, they are widely integrated into downstream tasks for enhancements, e.g., image retrieval @xcite , image generation @xcite and image captioning @xcite . this work inherits wisdom, incorporating both the visual scene graph and language scene graph as pivots for ummt. all the ummt researches assume that @xmath5, yet we notice that this can be actually unrealistic. we thus propose a visual hallucination mechanism, achieving the inference-time image-free goal. there are relevant studies on supervised mmt that manage to avoid image inputs (with text only) during inference. the visual retrieval-base methods @xcite , which maintain an image lookup-table in advance, such that a text can retrieve the corresponding visual source from the lookup-table. li et al. (2022) directly build pseudo image representations from the input lsg step1: sketching skeleton skeleton vsg hallucinated vsg step2: completing vision object node attribute node relation node edge node augmentor relation augmentor sentence. differently, we consider generating the visual scene graph with richer and holistic visual structural information. ## learning with scene graph pivoting in this part, based on the sg pivot we introduce several learning strategies to accomplish the unsupervised training of machine translation. we mainly consider 1) cross-sg visual-language learning, and 2) sg-pivoted back-translation training. fig. 4 illustrates these learning strategies. ## conclusion we investigate an inference-time image-free setup in unsupervised multimodal machine translation. in specific, we integrate the visual and language scene graph to learn the fine-grained visionlanguage representations. moreover, we present a visual scene hallucination mechanism to generate pseudo visual features during inference. we then propose several sg-pivoting learning objectives for unsupervised translation training. experiments demonstrate the effectiveness of our sg-pivoting based ummt. further experimental analyses present a deep understanding of how our method advances the task and setup.