Language-agnostic Semantic Consistent Text-to-Image Generation (2022)

Abstract: Recent GAN-based text-to-image generation models have advanced that they can generate photo-realistic images matching semantically with descriptions. However, research on multi-lingual text-to-image generation has not been carried out yet much. There are two problems when constructing a multilingual text-to-image generation model: 1) language imbalance issue in text-to-image paired datasets and 2) generating images that have the same meaning but are semantically inconsistent with each other in texts expressed in different languages. To this end, we propose a Language-agnostic Semantic Consistent Generative Adversarial Network (LaSC-GAN) for text-to-image generation, which can generate semantically consistent images via language-agnostic text encoder and Siamese mechanism. Experiments on relatively low-resource language text-image datasets show that the model has comparable generation quality as images generated by high-resource language text, and generates semantically consistent images for texts with the same meaning even in different languages.

## introduction in this paper, we consider multilingual text-toimage generation. there are two problems with multilingual text-to-image generation. the first problem is the language imbalance issue in textto-image datasets. most text-to-image generation datasets are in english, so it is difficult to construct text-to-image generation models for other languages. furthermore, since existing multilingual datasets have a small amount of data, a discriminator overfitting may cause problems such as instability of learning in gan. the second is that generative models have difficulty extracting semantic commonality between languages.this can produce different images for captions with the same semantics but different languages. in @xcite , they treat the problem that captions with same meanings in english create semantically different images. we extend this awareness between languages. to solve those problems, we propose lasc-gan for text-to-image generation. lasc-gan consists of a language-agnostic text encoder and a hierarchical generator. language-agnostic text encoder generates text embeddings to be used in the hierarchical generator for the first problem mentioned above. and we exploit the siamese structure training to capture the semantic consistency between images generated in various languages. our main contributions are as follows: 1) by using a language-agnostic text encoder, images for low-resource language text can be generated only by learning the high-resource language. 2) texts with the same semantics in different languages can generate semantically consistent images using the siamese mechanism in hierarchical generator to extract semantic consistency between languages. we show the effect of each contribution in experiments using english ms-coco (coco-en), chinese ms-coco (coco-cn) and korean ms-coco (coco-ko) datasets. ## methods we propose a lasc-gan for text-to-image generation. our goal is to obtain as good visual quality of images created with low-resource language text as images generated with high-resource language text and to enable the model to reflect semantic consistency between languages in image generation. the lasc-gan consists of a languageagnostic text encoder and a hierarchical generator. the language-agnostic text encoder is used to obtain a text representation that will be fed as a condition to the generator. the hierarchical generator generates images for text conditions. training strategy of the model consists of two stages as shown in figure1. in stage 1, the model is trained followed by @xcite using only a high-resource language dataset. in stage 2, the model is trained with a siamese structure with two model branches using data from different language pairs (en-cn, en-ko). ## conclusion in this paper, we propose a lasc-gan for text-toimage generation. through language-agnostic text encoder, the model can generate images with lowresource language texts in zero-shot setting. furthermore, by siamese mechanism, the model can extract high-level consistent semantics between languages when generating images. the experiments on coco-en, ko, and cn show that our proposed method can generate photo-realistic images from the relatively low-resource language text and extract semantic commons between languages for image generation.