A ny T rans: Translate A ny T ext in the Image with Large Scale Models (2024)

Abstract: This paper introduces AnyText, an all-encompassing framework for the task–In-Image Machine Translation (IIMT), which includes multilingual text translation and text fusion within images. Our framework leverages the strengths of large-scale models, such as Large Language Models (LLMs) and text-guided diffusion models, to incorporate contextual cues from both textual and visual elements during translation. The few-shot learning capability of LLMs allows for the translation of fragmented texts by considering the overall context. Meanwhile, diffusion models’ advanced inpainting and editing abilities make it possible to fuse translated text seamlessly into the original image while preserving its style and realism. Our framework can be constructed entirely using open-source models and requires no training, making it highly accessible and easily expandable. To encourage advancement in the IIMT task, we have meticulously compiled a test dataset called MTIT6, which consists of multilingual text image translation data from six language pairs.

## introduction recently, notable progress in natural language processing (nlp) and computer vision (cv) has been realized. a convergent area emerging from these disciplines is in-image machine translation (iimt), which focuses on transforming images with text in one language into equivalent images displaying that text translated into another language. the integration of these diverse capabilities carries immense significance in both scholarly research and practical application domains, including enhancing cross-cultural interactions, bolstering educational methodologies, and playing a pivotal part in the international business landscape. recently a few works attempted to explore the iimt task. works like @xcite @xcite have embraced an end-toend approach to tackle this task. however, these methodologies are constrained in their application as they solely cater to images containing straightforward textual content and are further restricted to a narrow range of languages. the model @xcite , along with popular products like google image translation * , microsoft image translation † , and apple ios image translation ‡ , embraces this cascaded paradigm. the cascaded model stands out in its ability to effectively manage a wide range of image translations, even those with complex backgrounds, thanks to its inherent scalability and the integration of advanced models. however, as illustrated in figure 1 (a), microsoft image translation, for instance, utilizes traditional machine translation to translate text recognized by ocr models. it then employs a simple rule to insert the translated text back into the original image. unfortunately, this approach often overlooks the contextual relationship between textual elements within images. this oversight can result in inaccurate translations and visual inconsistencies, thereby compromising the authenticity of the newly generated image. to address the identified shortcomings, our framework illustrated in figure 1 (b) significantly diverges from conventional text translation tasks in images. by leveraging the advanced contextual comprehension capabilities of llms, our approach achieves superior translation accuracy. alternatively, the integration of a vision language model (vlm) may allow a dual consideration of both visual and textual contexts within the source images, further enhancing translation quality. our methodology unfolds in three consecutive steps. initially, we utilize the latest pp-ocr @xcite to accurately locate the text within the image and decipher its content. this step is crucial for determining the exact area for text editing and translating the text content precisely. secondly, once the text is identified, we employ a few-shot prompt learning strategy that enables (visual) language models to maintain the format during contextual translation. this approach ensures that the translation is both contextually appropriate and linguistically accurate. finally, we apply a modified anytext @xcite to render the translated text back into the original image. in this phase, the translated text is fused into its original location, identified during the initial step. we propose resizing the anticipated text box by considering the length of the detected box, the original source text, and the translated target text. this modification maximizes the preservation of the original image's style and produces a clean, new image. as shown in figure 1 (b), our method does achieve superior translation quality and visual effects while preserving the image's legibility and aesthetic appeal. the new text seamlessly blends with the original visual context, maintaining both coherence and style. our main contributions are as follows: ## methodology in this section, we will detail each component of our anytrans. following the module order shown in figure 2 , we begin by introducing the detection and recognition of text in the image. following this, we introduce how to leverage (vision) llms for translation. lastly, we describe the text editing process informed by the translation outcomes. ## discussions as the first paper to introduce (vision) llms and diffusion model into the iimt task, significant opportunities exist for further improvement. below, we enumerate several potential directions for future advancements: ## conclusion we introduce a novel framework named any-trans designed for in-image machine translation (iimt). distinguished from existing closed-source products, our anytrans can be built upon opensource models and is training-free. uniquely, we integrate (vision) llms and diffusion models into iimt task for the first time, achieving both accurate translations and authentic translated images. furthermore, we have curated a multilingual text image translation dataset mtit6 to promote development in this field. we meticulously selected a sample of 50 images for each of the six languages, summing up to a total of 300 images. to objectively and accurately assess the authenticity of translated images along with the maintenance of font styles, we utilize both human evaluation and gpt-4o evaluation. for human evaluation, we enlisted the help of three annotators. for each image assessed, the annotators were provided with the original image alongside the translation outputs from google, microsoft, apple image translations, and our any-trans. they then scored each translation based on predetermined criteria, with the final score for each image being the average of the three annotators'. for the evaluation involving gpt-4o, to minimize biases associated with the order in which translations are presented, the evaluation is conducted on a one-to-one basis: compare the source image with the translated image from one of the four different methods. the detailed evaluation criteria are outlined as follows: in actual evaluation, these two aspects can be considered comprehensively based on the overall effect of the translated image on the score.