Character-Aware Models Improve Visual Text Rendering (2023)

Abstract: Current image generation models struggle to reliably produce well-formed visual text. In this paper, we investigate a key contributing factor: popular text-to-image models lack character-level input features, making it much harder to predict a word’s visual makeup as a series of glyphs. To quantify this effect, we conduct a series of experiments comparing character-aware vs. character-blind text encoders. In the text-only domain, we find that character-aware models provide large gains on a novel spelling task (WikiSpell). Applying our learnings to the visual domain, we train a suite of image generation models, and show that character-aware variants outperform their character-blind counterparts across a range of novel text rendering tasks (our DrawText benchmark). Our models set a much higher state-of-the-art on visual spelling, with 30+ point accuracy gains over competitors on rare words, despite training on far fewer examples.

## introduction over the last year, image generation models have made impressive quality gains @xcite @xcite . while many practical use cases are already within reach, rendering visual text in images remains a challenge. @xcite observe that dall•e-2 "struggles at producing coherent text," and the latest release of stable diffusion lists "cannot render legible text" as a known limitation. 1 in this paper, we seek to understand and improve the ability of image generation models to render high-quality visual text. to do so, we first investigate the spelling ability of text encoders in isolation. we find that despite their popularity, characterblind text encoders-which receive no direct signal as to the character-level makeup of their inputshave limited spelling ability. building on @xcite , we test the spelling ability of text encoders across scales, architectures, input representations, languages, and tuning methods. we document for the first time the miraculous ability of character-blind models to induce robust spelling knowledge (>99% accuracy) through web pretraining, but show that this does not generalize well beyond english, and is only achieved at scales over 100b parameters, making it infeasible for most applications. we find that character-aware text encoders, on the other hand, are able to achieve robust spelling ability at far smaller scales. applying these findings to image generation, we train a range of character-aware text-to-image models and demonstrate that they significantly outperform character-blind models on text rendering. for purely character-level models, this improved text rendering comes at a cost-decreasing image-text alignment for prompts that don't involve visual text. to alleviate this, we propose combining characterlevel and token-level input representations, and find that this delivers the best of both worlds. our main contributions are to: (1) measure the spelling ability of a range of text encoders, pulling apart the effects of scale, character-awareness, and multilinguality, using a new benchmark: wikispell. ## the spelling miracle language models can be categorized as to whether they have direct access to the characters making up their text input ("character-aware") or do not ("character-blind"). many early neural lan-character-blind: character-aware: guage models operated directly on characters, with no notion of multi-character "tokens" @xcite . later models moved to vocabulary-based tokenization, with some like elmo @xcite retaining characterawareness, and others like bert @xcite abandoning it in favor of more efficient pretraining. at present, most widely used language models are character-blind, relying on datadriven subword segmentation algorithms like byte pair encoding (bpe) @xcite to induce a vocabulary of subword pieces. while these methods back off to character-level representations for sufficiently rare sequences, they compress common character sequences into unbreakable units by design, as shown in figure 2 . recent work on "token-free" modeling has pointed to advantages of character-aware input representations. @xcite show that byt5a character-aware multilingual language model trained directly on utf-8 bytes-outperforms parameter-matched character-blind models on tasks related to spelling and pronunciation. while operating at the byte or character level comes at the cost of training and inference speed, additional work suggests that this can be overcome through downsampling @xcite . see @xcite for a recent overview of tokenization methods and character awareness. figure 2 : subword tokenization-used in most textto-image models-maps common character sequences onto ids that are looked up in an embedding table before being passed to the model, removing any signal about token-internal composition. this example uses the t5 sentencepiece tokenizer @xcite . in section §3 we find that, with sufficient scale, character-blind models can achieve near-perfect spelling accuracy. we dub this phenomenon the "spelling miracle", to emphasize the difficulty of inferring a token's spelling from its distribution alone. at the same time, we observe that character-blind text encoders of the sizes used in practice for image generation are lacking core spelling knowledge. with this in mind, it is unsurprising that to-day's image generation models struggle to translate input tokens into glyph sequences. these models' text encoders are all character-blind, with stable diffusion, dall•e, dall•e-2, imagen, parti and ediff-i all adopting bpe tokenizers @xcite @xcite @xcite . for image-text models, another key source of knowledge is supervised image-caption data. even if its text encoder is character-blind, could a model learn to spell by observing the makeup of words within images? while possible, we suspect this is an inefficient paradigm for learning, as each token would need to be learned separately, and would need to appear within an image-caption pair seen in training. in section §5 we find that, indeed, this "late-stage" learning of spelling is inferior to using a pretrained character-aware text encoder. ## measuring text encoder spelling ability since text-to-image generation models rely on text encoders to produce the representations for decoding, we first explore the ability of text encoders in isolation, using a text-only spelling evaluation task. ## the drawtext benchmark evaluating text-to-image models has been an ongoing topic of research, with the development of standard benchmarks from coco @xcite to drawbench @xcite , and metrics including fid @xcite , clip score @xcite , and human preferences @xcite . however, there has been a lack of work on text rendering and spelling evaluation. to that end, we present a new benchmark, drawtext, designed to measure the text rendering quality of text-to-image models. the benchmark consists of two parts, assessing distinct model capabilities: 1) drawtext spelling, which evaluates simple word rendering over a large set of english terms; and 2) drawtext creative, which evaluates end-to-end text rendering across diverse visual settings. ## image generation experiments in this section, we evaluate the spelling ability of text-to-image generative models with the proposed drawtext benchmark. state-of-the-art textto-image generative models consist of a text encoder plus a cascade of either diffusion models @xcite or autoregressive models @xcite that map the encoded text representations to realistic images. in section §3 we saw that character-aware text encoders greatly outperform character-blind models on spelling in a text-only setting; in this section, we investigate whether making the text encoder character-aware improves the text rendering ability of text-to-image models. ## conclusion in this paper, we set out to better understand what is needed for image generation models to reliably render well-formed visual text. using our novel wikispell and drawtext benchmarks, we were able to precisely quantify the effects of characterawareness and other design choices on spelling ability in both the text and visual domains. we found that character-aware text encoders confer large gains on spelling, and when used within an image generation model, these gains translate into improved visual text rendering. however, using exclusively character-level representations deteriorated overall text-image alignment-at least when evaluating our multilingual byt5 text encoder on english prompts with untuned guidance weight. to resolve this, we found that a hybrid model combining token-level and character-level signals offered the best of both worlds: dramatically improving visual text without significantly affecting alignment. while we saw substantial improvements on drawtext spelling accuracy (75% → 94% on common words and 47% → 83% on rare words), some failure modes remain unaddressed. even our strongest models were observed to occasionally drop, repeat, or merge letters within a word, or words within a phrase. our results strongly suggest that resolving these issues will require orthogonal improvements outside the text encoder, specifically changes to the image generation module. as a secondary finding, we demonstrated for the first time that, with sufficient scale, even models lacking a direct character-level view of their inputs can infer robust spelling information through knowledge gained via web pretraining-"the spelling miracle". while remarkable, this finding is less immediately practical, as it requires models over 100b parameters, and even these did not generalize well beyond english in our experiments.