Paper: AnyText: Multilingual Visual Text Generation and Editing (ICLR-2024)

Reviewer comment (Experiments):
using models trained own dataset to compare with previous baselines is not so fair

Author response:
Dear reviewer,

We attempted to compare our model with the method and data used in another work, specifically TextDiffuser. However, we encountered some challenges that made it not as "easy to go" as expected. Following the instructions provided in the [Official Code](https://github.com/microsoft/unilm/blob/master/textdiffuser), we found that it only provides rendered images with all text lines. Since AnyText requires the position and corresponding text for each line to make the Auxiliary Latent and Text Embedding modules work, perform inference became impossible. To overcome this issue, we attempted an alternative approach by using our OCR model to generate annotations, followed by generating images and comparing them using their metrics. After spending lots of time configuring the OCR model they used ([MaskTextSpotterV3](https://github.com/MhLiao/MaskTextSpotterV3)), and made guesses for missing details (presumably modified in the MaskTextSpotterV3 code that was not provided), we roughly replicated their evaluation metrics. Unfortunately, we encountered another issue that our OCR results are not consistent with the ground truth they used in their precision and recall calculations, which prevent us from obtaining reliable and fair evaluation results. Regarding GlyphControl, we were even unable to reproduce their own evaluation results. From the [Official Code](https://github.com/AIGText/GlyphControl-release/tree/main), they only shared 200+ prompt templates used for evaluation, without providing the texts they randomly selected and the positions of them, and the evaluation code was not provided, either. Consequently, we are unable to use evaluation set and metric from others to assess our model. 

In the other question, it is worth considering that other methods might have seen some of our evaluation sets. However, even that if a generated model has seen an image during training, fully restoring the pixels accurately is still challenging. Furthermore, the inputs provided to the model during inference are not the actual images themselves but rather captions and OCR annotations, which could differ from those used by other models. So, we believe that compared to traditional methods such as detection and recognition, the impact of this issue is relatively minor. 

Regarding whether the presence of Chinese data improves the performance of English text, our answer is: very likely. Chinese characters encompass thousands of distinct shapes, and if a model can accurately generate these characters, it becomes comparatively much easier to generate the additional 26 concise letterforms of English. This stems from our method's comprehensive approach to designing methods and processing datasets to support multilingual capabilities, which may not be present in other methods. However, even when considering the same training data, if we remove $l_p$ and $l_m$ in the Auxiliary Latent module, disable the Text Embedding module, and exclude the Text Perceptual loss, our method bears resemblance to GlyphControl. In our ablation study (see Table 2 in Supplementary Materials for latest), we have demonstrated performance improvements achieved by each of these self-designed algorithms.

We hope this reply has addressed your concerns, and we are looking forward to further discussion with you. Thank you!

Reviewer comment (Experiments):
As revealed in the ablation study, the most significant performance improvement of this method occurs upon the introduction of text embedding, which is attributed to the performance of PP-OCRv3. If one were not to use the OCR module's embedding and instead employ a general image encoder like the CLIP visual encoder, it is questionable whether the same level of performance improvement would have been achieved.

Author response:
Coincidentally, before utilizing the OCR model, our initial attempt was to employ the CLIP visual encoder. However, we found that the results were unsatisfactory. We suspect that this is because images with only rendered text fall under the Out-of-Distribution data category for the pre-trained CLIP vision model, hindering its text stroke understanding and encoding capabilities. Subsequently, we experimented with a trainable module composed of stacked convolutions and an average pooling layer. However, without strong supervision specifically tailored to this task, this module also struggled to perform well. In this study, we have reorganized our experiments, and the relevant results can be found in Table 2 of Supplementary Material. From Exp. 3 and Exp. 4, we noticed that using the vit or conv module for token replacement in the text embedding resulted in even lower performance compared to the original CLIP text encoder in Exp .2. Comparing with Exp. 5, we can conclude that utilizing a pre-trained OCR model effectively encodes stroke-level information, significantly enhancing text generation accuracy. You can also find qualitative comparisons in Fig. 4 and Fig. 5 in Supplementary Material.

Reviewer comment (Evaluation):
Many modules are added to the vanilla text-to-image diffusion model, but the paper fails to mention the computational overhead that arises as a result.

Author response:
As mentioned in the paper, our framework is based on ControlNet. Despite the addition of some modules, it did not significantly increase the computational overhead. Please refer to the parameter details below:
Params | ControlNet | AnyText
:---:|:---:|:---:
UNet|859M|859M
VAE|83.7M|83.7M
CLIP Text Encoder|123M|123M
ControlNet|361M|-
TextControlNet|-|360M
Glyph Block|-|0.35M
Position Block|-|0.04M
Fuse Layer|-|0.93M
OCR Model|-|2.6M 
Linear Layer|-|2.0M
Total|1426.7M|1431.6M

We compared the computational overhead of both models using a batch size of 4 on a single Tesla V100. The results are as follows:
Model | Inference Time
:---:|:---:
ControlNet|3476ms/image
AnyText|3512ms/image

In summary, compared to ControlNet, AnyText has an increase of 0.34% in parameter size and 1.04% in inference time.

Reviewer comment (Experiments):
Although the paper highlights multilingual capabilities and provides qualitative results for Korean and Japanese in the figures, it is disappointing that these two languages are excluded from the quantitative results, falling under the "others" category.

Author response:
Generating quantitative results for these languages requires the careful selection of reliable OCR models and the meticulous preparation of benchmark datasets. However, to be honest, none of us are familiar with languages other than Chinese and English. Furthermore, even if we were to calculate quantitative results, there are currently no comparable methods available to the public. Therefore, we have only provided qualitative results for now. Nevertheless, we plan to gradually improve this aspect in the future.

Reviewer comment (Experiments):
The lack of qualitative results corresponding to each ablation experiment is also a drawback.

Author response:
We have added qualitative results in Fig. 4 and Fig. 5 in Supplemental Material.

Reviewer comment (Novelty):
The architecture seems somewhat reliant on pre-established technologies such as Latent/Stable Diffusion and ControlNet, which slightly shadows its novelty.

Author response:
Stable Diffusion and ControlNet have been gaining significant popularity in the open-source community recently, and various interesting models have emerged. Our intention was to provide a plug-and-play module that allows more people to work on text generation and editing based on their own models. However, technically speaking, pre-established technologies are not strict dependencies for AnyText. In the future, we may explore other methods such as GANs.

Reviewer comment (Reproducibility):
Encumbered by a complex array of components and a multi-stage training regimen, the model’s re-implementation emerges as a challenging task, compounded further by numerous critical hyperparameters requiring manual assignment.

Author response:
Text generation and editing can be seen as a multi-task model. It is logical to first focus on "generating" text well before proceeding to "editing" it, which is why we implemented a staged training approach. The current solution does involve several modules and parameters, which is why we built a smaller-scale dataset (mentioned in Sec. 5.3) and conducted extensive validation work. In the future, we will open-source our code and dataset to facilitate the reproducibility of AnyText and consider refining the approach to make it more streamlined.

Reviewer comment (Writing):
Certain aspects, such as token replacement, require a more elaborate discourse for clearer comprehension, primarily concerning the identification of corresponding tokens and their subsequent utility in text-image generation.

Author response:
The process of token replacement is as follows: First, we obtain the CLIP token index of the placeholder string. For example, the index of '$\*$' is 265. Then, for the input $y'$ (which is processed from the original caption $y$ with all text lines replaced by '$*$'), after tokenization, we obtain a sequence of token indices. At this point, we keep track of the positions where the index is 265. After obtaining all token embeddings through a lookup table, we replace the embeddings at the marked positions, with the embeddings encoded by the OCR model. The replaced tokens then go through the self-attention mechanism of the CLIP text encoder, which combines stroke information with semantic information.

Reviewer comment (Evaluation):
There exists a potential ambiguity concerning the intermediate generative results (x'_0), where the possible presence of noise or blur could compromise the precision of perceptual loss computation.

Author response:
Yes, the $x'\_{0}$ may appear blurry, and the blurriness is correlated with the time step t. This is why we introduced the weight adjustment function $\varphi(t) = \bar{\alpha_t}$. We provide a more detailed explanation in Fig. 6 and Fig. 7 in Supplementary Material. According to the original DDPM paper, $\alpha_t=1-\beta_t$, where $\beta_t$ represents the forward process variances that increase linearly from 0.0001 to 0.02. $\bar{\alpha_t}$ is the cumprod of $\alpha_t$, with the range from 0 to 1, this aligns perfectly with our requirements for the weight adjustment. When t is large, the predicted image $x'_{0}$ may be more blurry, but the corresponding weight is also small, and the text perceptual loss has little effect.

Reviewer comment (Reproducibility):
A clearer depiction of computational resource demands (GPU Hours), beyond the ambiguity of epochs, would enhance the paper’s practicability and replicability.

Author response:
As mentioned in Sec. 5.1, our model was trained for 10 epochs, with the training speed being consistent for the first 8 epochs at approximately 29h/epoch. For the last 2 epochs, it increased to ~40h/epoch due to the enabling of the text perceptual loss. Overall, the total training time for the model amounted to 312 hours on 8XA100 GPUs.

Reviewer comment (Writing):
A more explicit elucidation on the operational synergy between Z_a from the Auxiliary Latent Module and Z_0 from VAE, as depicted in Fig. 2, alongside their application within Text-ControlNet, would augment the manuscript's clarity.

Author response:
In the standard stable diffusion process, training the model on $z_t$ alone cannot generate text well, since it does not explicitly inform the model of how (*glyphs*) and where (*position*) to generate them. Therefore, we designed the Auxiliary Latent Module to combine the information into $z_a$. As you mentioned, we hoped to use $z_a$ as a "catalyst" for text generation by ensuring that its shape aligns with $z_t$ and adding them together. However, we did not directly input this combined latent into the UNet because the parameters of UNet are trained on billions of images, giving it a large knowledge capacity. We did not want to influence its parameters using our relatively smaller dataset of millions of text-containing images. Hence, we freeze these parameters and fed $z_a+z_t$ into an additional trainable TextControlNet, essentially patching the original model to enable text generation.

Reviewer comment (Reproducibility):
The method and dataset collection miss a lot of details. For example, is the linear projection layer trained or fixed for ocr encoder in the text embedding module?

Author response:
Yes, the linear transformation $\xi$ is randomly initialized and trained during the training process.

Reviewer comment (Reproducibility):
Which feature is extracted from PP-OCR to serve as text embedding?

Author response:
We use the feature map before the last fully connected layer of the PP-OCR model. The feature map mentioned here and the $\hat{m}\_{p},\hat{m'}\_{p}$ referred to in the text perceptual loss(Sec. 3.4) are the same, the difference is that the feature is flattened and linear projected later, to match the token dimension.

Reviewer comment (Reproducibility):
Is the linear projection layer for the OCR encoder trained?

Author response:
Yes, the linear transformation $\xi$ is randomly initialized and trained during the training process.

Reviewer comment (Reproducibility):
What is the configuration for the fuse layer? A single convolution layer or a stacked convotion?

Author response:
A single layer, as mentioned in Sec. 3.2 : “...we utilize **a convolutional fusion layer** $f$ to merge $l_g$, $l_p$, and $l_m$...”. The kernel_size=3, stride=1, in_channels=324(256+64+4, channels for $l_g$, $l_p$, and $l_m$) and out_channels=320.

Reviewer comment (Presentation):
What is the input to text controlnet? Based on Figure 2, it seems to be the concatenation of $z_a$ and $z_t$.

Author response:
There is a "⊕" in Fig. 2, which means the operation is addition rather than concatenation. To provide an accurate description, we will modify the text in Sec. 3.1 as: "...to control the generation of text, we ~~combine~~ **add** $z_a$ with $z_t$ and fed them into a ..."

Reviewer comment (Experiments):
Why is the image resolution for glyph image 1024x1024 instead of 512x512? This does not align with the final image resolution.

Author response:
We have observed that when rendering small text onto glyph images at a resolution of 512x512, the text becomes nearly illegible, whereas at 1024x1024 there is no issue (see Supplemental Material Fig. 2, the first and fourth images in the second column). Additionally, we employed 4 downsampling layers for the Glyph block, while the Position block only employed 3. This incurs additional parameter overhead (see our replay to reviewer 5GTp's Q2), but the results demonstrate that it does not affect the alignment with the final image resolution.

Reviewer comment (Experiments):
How is the tight position mask $l_p$ annotated? As far as I know, the PP-OCR model does not support arbitrary shape text detection.

Author response:
Our dataset does not rely solely on OCR models but also includes human annotations. Just as mentioned in Sec. 4, "Except for the OCR datasets, **where the annotated information is used directly**, all other images are processed using the PP-OCRv3...". The image used in Fig. 2 is sourced from an OCR dataset called Art.

Reviewer comment (Reproducibility):
The glyph image $l_g$ contains rotated text; how is the bounding box annotated from the position mask?

Author response:
As mentioned in Sec. 3.2: "...we simplify the process by rendering characters **based on the enclosing rectangle** of the text position...". Specifically, use cv2.minAreaRect( ) on the polygon of the position mask.

Reviewer comment (Experiments):
The captions are generated using BLIP-2 model instead of the original captions. Could authors provide some statistics like the length of the captions and show some example captions for images in AnyWord-3M? How does this difference affect the model performance?

Author response:
The average caption length by BLIP-2 is **~8.26 words** per image, excluding the additional description of the text to be rendered. You can find caption examples in Fig. 2 and Fig. 3 in Supplemental Material. We utilize BLIP-2 not due to performance considerations, but because images as in OCR datasets or Noah-Wukong do not have existing English descriptions, and BLIP-2 is one of the most common approaches adopted by many studies. We believe that even perfect captions would have negligible influence on text-generation tasks. Therefore, we did not focus on evaluating it.

Reviewer comment (Presentation):
Could authors provide more information about the input for the visual examples? For example, the layouts for Figures 1, 6, 8, 12, the captions for Figures 5, 10, 11.

Author response:
To provide more information, we have regenerated several images, as shown in Fig. 1 in Supplementary Material. We just use Gradio's brush as input and after a simple process convert it into Glyph and Positon. The layouts and captions for Figures 5, 10, and 11 are available as they are sourced from our benchmark. See Fig. 2 and Fig.3 in Supplemental Material.

Reviewer comment (Evaluation):
The Sen. ACC and NED metric is measured using the same OCR model as the encoder in model training, which might be unsuitable. Could authors evaluate the OCR performance using another OCR model?

Author response:
Thanks for your suggestions. We have chosen a recently popular OCR model in the ModelScope community, DuGuangOCR[1], to replace PPOCR_v3 as our evaluation model, which supports recognition for both Chinese and English. The updated quantitative comparison can be seen in Table 1 in Supplementary Material. We observed that in all methods, there were minimal changes in the Sen.Acc and NED metrics, except for AnyText, which showed a significant decrease. This indicates that PPOCR_v3 indeed introduced bias to the evaluation. However, the updated results still demonstrate that AnyText has significant advantages over competing methods.

Reviewer comment (Experiments):
Table 3 shows the improvement brought by visual text embedding in Chinese text generation. I wonder if this also improves the English word generation.

Author response:
Yes, the text embedding works for English. In fact, as we use the feature before the fully connected layer, it is not particularly sensitive to language and instead focuses more on stroke. The Chinese and English evaluation results for Exp. 2 and Exp. 3 in Table 3 are as follows:
Exp. № | Text Embedding |Sen. Acc↑(CH) | NED↑(CH) | Sen. Acc↑(EN) | NED↑(EN)
:---:|:---:|:---:|:---:|:---:|:---:
2|w/o|0.2024|0.4649|0.3159|0.6298
3|w/|0.4595|0.7072|0.4524|0.7459

There is also a qualitative comparison in Fig. 4 in Supplementary Material(Exp. 2 vs. Exp. 5).

Reviewer comment (Experiments):
In the experiment, the text diffuser model is not trained on the same dataset as GlyphControl and AnyText Model. Is it possible that authors fine-tune them on the same data and compare final performance?

Author response:
The training dataset used for TextDiffuser is MARIO-10M, out of which 9.2M images were filtered from LAION-400M. Similarly, the English data in AnyWord-3M was also filtered from LAION-400M. During the data processing, we indeed found that there were nearly 10 million images containing text in LAION-400M. However, in order to maintain a balance in terms of scale with the Chinese data in AnyWord-3M, we adjusted our filtering rules and retained only 1.3M images. More details about this can be found in Appendix A.2. To some extent, we are using a subset of the training data as for TextDiffuser.

The GlyphControl dataset, LAION-Glyph-10M, was filtered from LAION-2B-en, and it is likely to have a significant overlap with the text-containing images in LAION-400M. Additionally, in our ablation experiments (refer to Table 2 in Supplementary Material for the latest results), the setup of Exp. 2 is very similar to GlyphControl. It involves disabling editing, text embedding, and perceptual loss while retaining only the glyph image that renders tests based on their positions, along with the corresponding text position image. Under the exact same training dataset, our method (Exp. 8) demonstrates a significant advantage.

In addition, we randomly sampled some text lines to assess the accuracy of our OCR annotations. The error rate for the English portion was 2.4%, while for the Chinese portion was significantly higher at 11.8%. This difference may be attributed to the inherent complexity of Chinese character recognition, as well as the performance of PPOCR_v3.

In summary, during model training, we only utilized 1.3 million out of 10 million English data and incorporated over 1.6 million non-English data with noise. While our English metrics have a clear advantage over competing methods, this advantage is certainly not derived from the benefits of the data.