C ontrol T ext: Unlocking Controllable Fonts in Multilingual Text Rendering without Font Annotations (2025)

Abstract: This work demonstrates that diffusion models can achieve font-controllable multilingual text rendering using just raw images without font label annotations. Visual text rendering remains a significant challenge. While recent methods condition diffusion on glyphs, it is impossible to retrieve exact font annotations from large-scale, real-world datasets, which prevents user-specified font control. To address this, we propose a data-driven solution that integrates the conditional diffusion model with a text segmentation model, utilizing segmentation masks to capture and represent fonts in pixel space in a self-supervised manner, thereby eliminating the need for any ground-truth labels and enabling users to customize text rendering with any multilingual font of their choice. The experiment provides a proof of concept of our algorithm in zero-shot text and font editing across diverse fonts and languages, providing valuable insights for the community and industry toward achieving generalized visual text rendering.

## introduction diffusion model is one of the dominant paradigms in image generation @xcite @xcite @xcite @xcite , because of its iterative denoising process that allows fine-grained image synthesis. while these models effectively capture data distributions of photorealistic or artistic images, they still fall short in generating high-fidelity text. rendering text in images is inherently more challenging as it requires precise knowledge of the geometric alignment among strokes, the arrangement of letters as words, the legibility across varying fonts, sizes, and styles, and the integration of text into visual backgrounds. at the same time, humans are more * co-first authors. this is preliminary work and code will be released at github.com/bowen-upenn/controltext. sensitive to minor errors in text, such as a missing character or an incorrectly shaped letter, compared to natural elements in a visual scene that allow for a much higher degree of variation. increasing attention has been paid to visual text rendering @xcite @xcite due to its high user demands. instead of relying solely on diffusion models to remember exactly how to render text, recent research is starting to embed the visual attributes of texts, such as glyphs @xcite @xcite , as input conditions to diffusion models. however, it is still difficult for users to specify the desired font in the open world, and there remain open challenges that burden the development of font-controllable text rendering: we also provide the community with a comprehensive dataset for font-aware glyph controls collected from diverse real-world images. we further propose a quantitative evaluation metric for handling fuzzy fonts in the open world. experimental results demonstrate that our method, controltext, as a text and font editing model, facilitates a humanin-the-loop process to generate multilingual text with user-controllable fonts in a zero-shot manner. despite these advancements, visual text rendering @xcite @xcite continues to pose significant challenges. several algorithms rely on text embeddings from user prompts or captions to control the diffusion process, such as textdiffuser @xcite , textdiffuser2 @xcite , and deepfloyd's if (deepfloyd-lab, 2023). @xcite utilizes intermediate features from ocr @xcite generation from glyphs the majority of algorithms rely on visual glyphs, pixel-level representations of texts, to guide the generation process. however, because most image training datasets lack ground-truth font annotations, most algorithms utilize a fixed standard font to render the texts on their glyph controls. for instance, glyphcontrol @xcite and glyphdraw @xcite render ocr-detected text using a fixed font, with the former adding a glyph-specific control-net @xcite . textmaster @xcite , diffute @xcite , and any-trans @xcite enforce font consistency within images, while @xcite introduces font variation through random sampling. layout generation is also addressed using language and vision models @xcite @xcite @xcite @xcite . in contrast, we focus on human-in-the-loop text editing without large language or multimodal models. our work builds upon the codebase of any-text @xcite , a glyph-based algorithm that trains a base controlnet @xcite model to render visual texts, with glyphs being generated in a fixed standard font due to unavailability of ground-truth font annotations, leaving the model to infer an appropriate font. font-controllable generation fewer recent works are more closely related to ours in enabling controllable fonts @xcite @xcite @xcite . however, none of these works provide a quantitative evaluation metric to assess the generated fonts in open-world settings. anytext2 @xcite ) is a concurrent work developed by the authors of anytext @xcite . we share a similar architecture @xcite , but we eliminate its use of lengthy language prompts in the inputs, separate models to support different languages, and the trainable ocr model to encode the font features. instead, we use ocr solely to filter out low-quality glyph controls. @xcite is also not yet open-sourced at the time of our submission. several works tackle font control through predefined font labels or additional supervision. glyph-byt5 @xcite requires languagespecific font labels and emphasizes language understanding, while we treat text rendering as purely visual. glyphdraw2 @xcite learns font features via cross-attention and a fine-tuned language model, but lacks quantitative font evaluation. joytype @xcite focuses on synthetic e-commerce images with 10 fixed fonts and vision-language models, assuming ocr is fontsensitive-unlike our font-agnostic assumption. fonts @xcite and customtext (paliwal et al., 2024a) rely on pre-specified font labels or user-defined font names. while @xcite highlights challenges with small fonts, we show that localized editing improves small-font quality. in contrast to above methods, we eliminate the need for font labels, special tokens, or predefined font names @xcite @xcite , enabling generalization to unseen fonts and languages. ## technical approach we envision this method being used as a modular plug-in for existing text-to-image generation frameworks. it works with images generated by any base models or actual photos. for instance, when incorrect text is generated, or the user wants to replace some text or modify its font, our algorithm can be specifically targeted to these localized regions without altering remaining parts in images. by leveraging a human-in-the-loop approach, the model aims to render controllable visual text within the user-specified region, perform background inpainting, and blend the modified region back into the original image, regardless of its original size. ## discussion this work presents a simple and scalable proofof-concept for multilingual visual text rendering with user-controllable fonts in the open world. we summarize our key findings as follows: font controls require no font label annotations a text segmentation model can capture nuanced font information in pixel space without requiring font label annotations in the dataset, enabling zeroshot generation on unseen languages and fonts (as evidenced by generated images with japanese in figure 1 ), as well as scalable training on web-scale image datasets as long as they contain text. ## limitations our model is based on controlnet @xcite with a clip text embedding model @xcite , although modified by any-text @xcite ) to incorporate glyph line information. however, the clip-text encoder has relatively limited language understanding capabilities compared to state-of-the-art foundation models. unlike text itself, this limitation affects the model's ability to accurately render complex artistic visual features or backgrounds, which users might specify in their input prompts, such as asking the text to appear like clouds or flames, that go beyond merely the font information. additionally, due to limited training resources, our experiments were conducted using a smaller diffusion model as a proof-of-concept compared to commercial ones. each epoch requires approximately 380 gpu hours on nvidia v100 gpus with 32 gb of memory, but we anticipate significantly improved efficiency on newer hardware and with a larger memory. this constraint may result in suboptimal inpainting of background regions within the text area, as well as instability in the quality of rendered text. this also made it difficult to try other base models for comparison. moreover, the limited training resources also made it difficult to conduct more ablation studies, which could have provided more nuanced insights into the architecture's effectiveness. the users also have limited controls of background pixels behind the text. some sacrifice in text quality is observed for non-latin languages on the anytext-benchmark in exchange for improved font controllability. the embedding layers of the glyph controls can also lead to reduced text quality, especially when the text in a font is very small, thin, or excessively long. in such cases, fine details of the font information in the glyphs may be lost. as with all other text-to-image algorithms that rely on diffusion models, our approach requires a certain number of denoising steps to generate a single image at inference. end-to-end transformerbased models @xcite ) may improve the time efficiency of the generation process. ## ethical impact this work is intended solely for academic research purposes. while our algorithm allows users to generate images with customized text, there is a potential risk of misuse for producing harmful or hateful content or misinformation. however, we do not identify any additional ethical concerns compared to existing research on visual text rendering. ## acknowledgment this work was supported by the national science foundation (nsf) grant ccf-2112665 (tilos).