Empowering Backbone Models for Visual Text Generation with Input Granularity Control and Glyph-Aware Training (2024)

Abstract: Diffusion-based text-to-image models have demonstrated impressive achievements in diversity and aesthetics but struggle to generate images with legible visual texts. Existing backbone models have limitations such as misspelling, failing to generate texts, and lack of support for Chinese texts, but their development shows promising potential. In this paper, we propose a series of methods, aiming to empower backbone models to generate visual texts in English and Chinese. We first conduct a preliminary study revealing that BPE tokenization and insufficient learning of cross-attention modules restrict the performance of the backbone models. Based on these observations, we make the following improvements: (1) We design a mixed granularity input strategy to provide more suitable text representations; (2) We propose to augment the conventional training objective with three glyph-aware training losses, which enhance the learning of cross-attention modules and encourage the model to focus on visual texts. Through experiments, we demonstrate that our methods can effectively empower backbone models to generate semantic relevant, aesthetically appealing, and accurate visual text images, while maintaining their fundamental image generation quality.

## introduction recently, diffusion-based models @xcite @xcite @xcite have revolutionized the field of text-to-image generation, particularly in terms of diversity and aesthetics. among various text-to-image tasks, visual text generation has attracted much attention due to the growing demand for generating images containing visual texts in the ai art community and commercial fields. despite their attractiveness, this task remains challenging, as most current diffusion models struggle to produce images with precise, readable visual texts. at present, dominant studies on this task can be roughly divided into two categories. some researchers focus on adding additional conditions to reduce the difficulty of generating images with visual texts, resulting in restricted diversity and visual texts not coherent with the background. @xcite @xcite @xcite . other researchers directly explore the performance of backbone models on visual text generation, which avoids the limitations of the previous type of methods but suffers from challenges such as misspelling, ignoring, and repeating words. to deal with these issues, early studies @xcite @xcite explore various text encoders to address misspelling issues. recent commercial models such as dall-e 3 @xcite and stable diffusion 3 @xcite demonstrate remarkable performance, further validating the potential of this research direction. however, they lack support for other languages, such as chinese. to explore potential avenues for improvements, we first conduct several preliminary experiments and observe that the visual text generation performance of backbone models are mainly constrained for two reasons: first, bpe tokenization requires the model to combine subwords to form complete visual words, increasing the difficulty of generating visual texts. second, the model is unable to effectively bind visual texts to the corresponding text tokens due to the insufficient learning of the cross-attention modules. based on these analyses, we propose a series of methods that significantly improve the visual text generation capability of backbone models, as shown in figure 1 . specifically, we first introduce a mixed granularity input strategy that provides more suitable text representations. then, we augment the conventional mse loss with three glyphaware losses: (1) attention alignment loss refines the cross-attention maps, thereby better binding visual texts to their corresponding text tokens; (2) local mse loss highlights the importance of visual text areas; (3) ocr recognition loss encourages the model to generate accurate visual texts. figure 2 demonstrates that our methods effectively enhance the backbone model's visual text generation ability while maintaining its fundamental capabilities. particularly, our methods can be transferred to the generation of chinese texts. ## preliminary study in this section, we first introduce the basic concepts of the diffusion based text-to-image backbone model, and then conduct experiments to identify potential avenues for improvements. ## conclusion in this paper, we conduct a preliminary study and find that bpe tokenization, as well as the model's insufficient learning of cross-attention modules, constrains the visual text generation performance of diffusion-based backbone models. based on these insights, we propose a series of methods, aiming to empower the backbone model with the ability to generate accurate and aesthetically appealing visual text images, while maintaining fundamental image generation quality. specifically, we introduce a mixed granularity input strategy to provide more suitable text representations. besides, we augment the conventional training objective with three glyph-aware training losses, which enhance the learning of the cross-attention modules and encourage the model to focus on visual texts. experiments demonstrate the effectiveness of our methods. typically, our methods can be transferred to chinese text generation. in the future, we intend to explore visual text generation for more languages, and generate texts in different styles @xcite . besides, we also plan to explore utilizing glyph enhanced diffusion models for image-to-image translation @xcite .