ARTIST : A Transformer-based C hinese Text-to-Image Synthesizer Digesting Linguistic and World Knowledge (2022)

Abstract: Text-to-Image Synthesis (TIS) is a popular task to convert natural language texts into realistic images. Recently, transformer-based TIS models (such as DALL-E) have been proposed using the encoder-decoder architectures. Yet, these billion-scale TIS models are difficult to tune and deploy in resource-constrained environments. In addition, there is a lack of language-specific TIS benchmarks for Chinese, together with high-performing models with moderate sizes. In this work, we present ARTIST, A tRansformer-based Chinese Text-to-Image SynThesizer for high-resolution image generation. In ARTIST, the rich linguistic and relational knowledge facts are injected into the model to ensure better model performance without the usage of ultra-large models. We further establish a large-scale Chinese TIS benchmark with the re-production results of state-of-the-art transformer-based TIS models.Results show ARTIST outperforms previous approaches.

## introduction text-to-image synthesis (tis) is a popular multimodality task that aims to convert natural language texts into realistic images @xcite . for accurate tis, various methods have been proposed based on generative adversarial networks (gans) @xcite . recently, with the wide popularity of the transformer model architecture @xcite , transformer-based tis has received more attention, which converts raw texts into "pseudo image tokens" by transformers and generates images based on models such as vqgan @xcite and @xcite . notable works include dall-e @xcite , dall-e 2 @xcite , cogview @xcite , cogview2 @xcite , ernie-vilg @xcite , m6 @xcite , ofa @xcite and a few others. despite the remarkable progress, we suggest that the mainstream transformer-based tis models may have a few drawbacks. i) most tis models have billion-scale parameters, making it challenging to fine-tune and deploy them in resourceconstrained environments (such as @xcite ; @xcite ). this highly limits the applications of these models in real-world applications. ii) the encoder-decoder architecture in the transformer @xcite does not explicitly model the semantics of key elements appearing in texts (i.e., entities or objects), and hence may lack the background knowledge for realistic image generation. iii) most existing works are benchmarked with english datasets, e.g., ms-coco @xcite . there is a lack of language-specific benchmarks for other languages (chinese in our work), together with high-performing tis models that are suitable for efficient domain-specific fine-tuning and online deployment. in addition, the multi-granularity of word segmentation of the chinese language @xcite makes the underlying transformer difficult to understand the true meanings of the input texts, which also causes ambiguity for knowledge injection to the models. we present artist, a transformer-based chinese text-to-image synthesizer with rich linguistic and world knowledge digested. it aims to gen-一架螺旋桨飞机在机场跑道上 ( a propeller plane is on the runway of the airport ) ## conclusion and future work in this paper, we present the artist framework for knowledge-enhanced chinese tis. the rich linguistic and relational knowledge facts are injected into the model for better chinese language understanding. for evaluation, we establish a largescale chinese tis benchmark and show that the proposed artist models outperform previous approaches. we will release our models and benchmark to the public and extend our work to other languages in the future.