Most recent image captioning works are conducted in English as the majority of image-caption datasets are in English . However, there are a large amount of non-native English speakers worldwide. Generating image captions in different languages is worth exploring . In this paper , we present a novel unsupervised  method to generate image captions  without using any caption corpus . Our method relies on 1 ) a cross-lingual auto-encoding , which learns the scene graph mapping function along with the scene graph encoders and sentence decoders on machine translation parallel corpora, and 2 ) an unsupervised feature mapping, which seeks to map the encoded scene graph features from image modality to sentence modality. By leveraging cross-lingual auto-encoding, cross-modal feature mapping, and adversarial learning, our method can learn an image captioner to generate captions in different languages. We verify the effectiveness of our proposed method on the Chinese image caption generation  . The comparisons against several baseline methods demonstrate the effectiveness of our approach.
Research in image captioning has mostly focused on English because of the availability of image-caption paired datasets in this language . However, building vision-language systems only for English deprives a large part of the world population of AI technologies' benefit. On the other hand, creating image-caption paired datasets for every target language is expensive . In this work , we present a novel unsupervised cross-lingual method to generate image captions in a target language without using any image-caption corpus in the source or target languages . Our method relies on (i ) a cross-lingual scene graph to sentence translation process , which learns to decode sentences in the target language from a cross-lingual encoding space of scene graphs using a sentence parallel (bitext) corpus, and (ii ) an unsupervised cross-modal feature mapping which seeks to map an encoded scene graph features from image modality to language modality. We verify the effectiveness of our proposed method on the Chinese image caption generation task . The comparisons against several existing methods demonstrate the effectiveness of our approach.
R
Most recent image captioning works are conducted in English as the majority
Research in image captioning has mostly focused on English because of the availability
R
datasets are in English
paired datasets in this language
R
there are a large amount of non-native English speakers worldwide. Generating image captions in different languages is worth exploring
building vision-language systems only for English deprives a large part of the world population of AI technologies' benefit. On the other hand, creating image-caption paired datasets for every target language is expensive
R
paper
work
A
cross-lingual
A
in a target language
R
caption corpus
image-caption corpus in the source or target languages
R
1
(i
R
auto-encoding
scene graph to sentence translation process
R
the scene graph mapping function along with the scene graph encoders and sentence decoders on machine translation parallel corpora, and 2
to decode sentences in the target language from a cross-lingual encoding space of scene graphs using a sentence parallel (bitext) corpus, and (ii
R
feature mapping,
cross-modal feature mapping
R
the
an
R
sentence modality. By leveraging cross-lingual auto-encoding, cross-modal feature mapping, and adversarial learning, our method can learn an image captioner to generate captions in different languages.
language modality.
A
task
R
baseline
existing