C ross2 S tr A : Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment (2023)

Abstract: Unpaired cross-lingual image captioning has long suffered from irrelevancy and disfluency issues, due to the inconsistencies of the semantic scene and syntax attributes during transfer. In this work, we propose to address the above problems by incorporating the scene graph (SG) structures and the syntactic constituency (SC) trees. Our captioner contains the semantic structure-guided image-to-pivot captioning and the syntactic structure-guided pivot-to-target translation, two of which are joined via pivot language. We then take the SG and SC structures as pivoting, performing cross-modal semantic structure alignment and cross-lingual syntactic structure alignment learning. We further introduce cross-lingual&cross-modal back-translation training to fully align the captioning and translation stages. Experiments on English-Chinese transfers show that our model shows great superiority in improving captioning relevancy and fluency.

## introduction generating texts to describe images (a.k.a., image captioning) has many real-world applications, such as virtual assistants and image indexing @xcite . current image captioning models have achieved impressive performance @xcite @xcite , yet are mainly limited to the english language due to the largescale paired image-caption datasets. subject to the scarcity of paired captioning data, the development of captioning in other languages is thus greatly hindered. while manually crafting sufficient paired data is prohibitively expensive, cross-lingual image captioning @xcite offers a promising solution, which aims to transfer a captioner trained at resource-rich language (e.g., english) to the resource-scarce language(s) without paired captioning data at target language(s). ## related work image captioning has been an emerging task in the past few years and received great research attention @xcite @xcite . later, the task of cross-lingual image captioning @xcite has been presented, to transfer the knowledge from resource-rich language to resource-poor language 1 , so as to spare the burden of manual data annotation for the minority languages. however, the task has been hindered and received limited attention due to two key issues: irrelevancy and disfluency of captions. there are two categories of cross-lingual captioning approaches: the translation-based @xcite and the pivoting-based @xcite methods. the former employs an off-the-shelf translator to translate the source (pivot) captions into the target language for targetside training or as the target-side captions. the latter reduces the noise introduction of the pipeline by jointly performing the image-to-pivot captioning step and pivot-to-target translation step, thus being the current sota paradigm. this work inherits the success of this line, and adopts the pivoting-based scheme as a backbone, but we further strengthen it by leveraging the semantic and syntactic structure information to better solve the two issues. scene graphs depict the intrinsic semantic scene structures of images or texts @xcite . in sgs, the key object and attribute nodes are connected to describe the semantic contexts, which have been shown useful as auxiliary features for wide ranges of downstream applications, e.g., image retrieval @xcite , image generation @xcite and image captioning @xcite . here we incorporate both the visual and language scene graphs to enhance the cross-modal alignment learning. note that @xcite also leverage the sg features for cross-lingual captioning, while ours differs from theirs in three aspects. first, they consider a fully unsupervised cross-lingual setup with no image-caption pairs at pivot language, while under such an unpaired assumption the visual and 1 without using target-side image-caption pairs, the task is also known as unpaired cross-lingual image captioning. language scene graphs are hard to align, and thus limits the utility of sgs. second, in this work we sufficiently align the two cross-modal sgs via unsupervised learning, such that the noises in sgs will be effectively screened. third, @xcite align the pivot and target languages with also the sg structure. we note that it could be ineffective to perform cross-lingual alignment based on textual sgs because the scene structures in different languages are essentially the same. in fact, two languages can be different the most in linguistic structures. almost all the erroneous sentences come with certain grammar or syntax errors @xcite . also syntax features have been extensively found to be effective in improving the language quality (e.g., fluency and grammatically-correctness) in cross-lingual scenario @xcite @xcite . for example, in machine translation, different languages show great correspondences in phrasal constituent structures @xcite . also, syntactic structure features have been integrated into a broad number of downstream applications @xcite @xcite . thus we consider making use of the syntax structures as cross-lingual supervision to enhance the captioning quality. ## syntactic semantic structure-guided cross-lingual captioning framework the original task is to learn a mapping f @xmath0. following @xcite ; @xcite , we decompose f @xmath1: 1) the image-to-pivot captioning f @xmath2, s p )}, and 2) the pivot-to-target translation f s @xmath3, s t )}. note that {(i, s p )} and {(s p , s t )} are two distinct datasets with possibly no intersection. in our setting, we also leverage the sg and sc structure features in two mappings. as shown in fig. 2 , the semantic structure-guided captioning phase (@xmath4,@xmath5, yielding the pivot caption s p . then, the syntactic structure-guided translating phase (@xmath6,@xmath7, finally producing the target caption s t . note that the input embeddings of the second step are shared with the output embeddings from the first step so as to avoid the isolation of the two parts. also we impose a residual connection from the sg feature representations to the sc feature representations to supervise the final target captioning with scene features. ## experimental results and analyses transfer between mscoco and aic-icc ## conclusion and future work in this paper we investigate the incorporation of semantic scene graphs and syntactic constituency structure information for cross-lingual image captioning. the framework includes two phrases, semantic structure-guided image-to-pivot captioning and syntactic structure-guided pivot-to-target translating. we take the sg and sc structures as pivots, performing cross-modal semantic structure alignment and cross-lingual syntactic structure alignment learning. a cross-lingual&cross-modal backtranslation training is further performed to align two phrases. on english↔chinese transfer experiments, our model shows great superiority in terms of captioning relevancy and fluency. bridging the gaps between the cross-modal and cross-lingual transfer with external semantic and syntactic structures has shown great potential. thus it is promising to extend the idea to other scenarios. also, exploiting the external structures potentially will introduce noises, and thus a dynamical structure induction is favorable.