Paper: AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ (ICLR-2024)

Reviewer comment (Experiments):
As shown in Table 1, DaTikZ consists of multiple data sources. If DaTikZ consists of multiple data sources, the authors should evaluate which data sources contribute to the accuracy of the generation on the test data and to what extent. This would not only justify the use of each data source, but may also suggest what further data should be collected in the future. The contribution of data augmentation should also be evaluated.

Author response:
> As shown in Table 1, DaTikZ consists of multiple data sources. If DaTikZ
> consists of multiple data sources, the authors should evaluate which data
> sources contribute to the accuracy of the generation on the test data and to
> what extent. This would not only justify the use of each data source, but may
> also suggest what further data should be collected in the future. 

We agree with the reviewer that this should be investigated. We are currently running these experiments and will report back once we have the results.

> The contribution of data augmentation should also be evaluated.

We are currently in the process of evaluating this. Please refer to our general response for further details.

Reviewer comment (Novelty):
Since this paper is also about a novel dataset for scientific vector graphics and their benchmarking, the technical contribution of the baseline method is modest.

Author response:
> Since this paper is also about a novel dataset for scientific vector graphics
> and their benchmarking, the technical contribution of the baseline method is
> modest.

We agree with the reviewer that the creation of our dataset plays a central role in our work, however, we believe our methodology carries innovation as well. First, we are, to our best knowledge, the first to combine LLMs with vision encoders to extract visual concepts of *textual data*, which improved CLiMA over LLaMA, despite having the same inputs. This result is by no means trivial. In addition, the multimodality of our data (text/code/images) allows for automated evaluation that is new in this field and lead to new insights. For instance, we identified CLIPScore's vulnerability to typographic attacks and the relative robustness of CLIPScore<sub>img</sub>. Notably, we're perhaps the first to report and use CLIPScore<sub>img</sub> in this work. We have made that more clear in our latest revision.

Reviewer comment (Presentation):
Figure 2 shows two radar charts. Although the main purpose of the radar charts is to make relative comparisons within the same chart, it would be possible to make some comparisons among the method in the left and right charts if the ranges of each score were adjusted.

Author response:
> Figure 2 shows two radar charts. Although the main purpose of the radar
> charts is to make relative comparisons within the same chart, it would be
> possible to make some comparisons among the method in the left and right
> charts if the ranges of each score were adjusted.

We designed the left radar chart to evaluate our own primary models (architecture & model size), while the right one evaluates other, more heterogeneous models (GPT-4, Claude 2, CLiMA<sub>img</sub>). This leads to more pronounced score differences in the right chart, necessitating different scales. We agree with the reviewer, however, that comparisons across charts would be insightful, which is why we plot the best model from the left chart also in the right one as a reference, and provide exact scores for all models in Appendix C.

Reviewer comment (Evaluation):
When the authors conduct a human annotation campaign, they could also evaluate how well the automatic evaluation metrics used in section 5.1 correlate with subjective evaluation; are there any plans to do so?

Author response:
> When the authors conduct a human annotation campaign, they could also
> evaluate how well the automatic evaluation metrics used in section 5.1
> correlate with subjective evaluation; are there any plans to do so?

Acknowledging the author's feedback, we've updated our paper accordingly and report Spearman correlations for (Caption Similarity-CLIPScore) and (Reference Similarity-CLIPScore<sub>img</sub>). Segment-level correlations are 0.23 and 0.25 (which is similar to what we observe for e.g. machine translation metrics [[1](https://aclanthology.org/2021.tacl-1.87/),[2](https://aclanthology.org/2022.eamt-1.9/)]) and system-level correlations are 0.5 and 0.8, respectively. The higher correlations for image-image over caption-image similarity strengthen our hypothesis regarding CLIPScore's biases in favor of images of texts.

Reviewer comment (Evaluation):
Can you please provide a couple of examples of different models' behavior to typographic attacks?

Author response:
An example of a typographic attack can be seen in Figure 5 (low-right subfigure). We have made that more clear. In our latest revision we have added additional examples. Again, please refer to the general response.

Reviewer comment (Experiments):
Have others considered comparing with ChatGPT? Maybe fine-tuned on a very small subset of the dataset.

Author response:
We have considered ChatGPT in the initial stages of this project, but due to other authors reporting that ChatGPT's performance on generating TikZ [[1](https://arxiv.org/abs/2305.18583), [2](https://arxiv.org/abs/2303.12712)] is vastly below GPT-4, we chose to not include it in our experiments, with GPT-4 effectively serving as an upper bound.

Reviewer comment (Experiments):
The main weakness of the paper is the limited ablation study: The authors compare and contrast the two versions of the model, LLaMa and CLiMA (LLaMa + CLIP adapter) but don't highlight other important choices made, for example, the effect of the artificial samples. 50% of the proposed dataset is the artificial samples, but the evaluation is done on real samples. The effect of augmentation with artificial samples is not measured.

Author response:
While the examples are not artificial but augmented, we agree with the reviewer that we should measure this effect. Please refer to our general response for further details.

Reviewer comment (Experiments):
The effect of data augmentation for CLiMa. As specified in section 5, during training CLIP is given either the input caption or the reference image (50-50%) and during inference it is given the input caption. The effect of this data augmentation is not reported.

Author response:
Mixing text-only and text-image modalities is common when training models which accept both unimodal and multimodal input [[1](https://arxiv.org/abs/1908.06066),[2](https://arxiv.org/abs/2202.03052)]. However, we do agree with the reviewer that quantifying the effect of this particular augmentation would be insightful. We will add this to our final version.  

In response, we conducted an ablation experiment and have included our positive findings in Appendix C.3. This data augmentation indeed positively influenced the model's performance, as we initially hypothesized.

Reviewer comment (Experiments):
The performance of vanilla LLaMa or prompt-tuned LLaMa is not reported, making the comparison between fine-tuned versions and prompted GPT-4 / Claude-2 not exactly fair.

Author response:
In our preliminary experiments, vanilla LLaMA and its derivatives had negligible TikZ generating capabilities, which is why we did not include them in our comparison. We have cleared this up in our latest revision. We do not think our comparison is necessarily unfair for GPT-4, however. For example, GPT-4 is [rumored](https://news.ycombinator.com/item?id=36675934) to have been trained on joint data (rendered LaTeX/text). In this case, the training would be very similar to our setting and even put our models at a disadvantage, given the likely difference in model size. Either way, since not much is known on how these models operate or how they were trained, we think it is difficult to assess if a comparison is fair or not. What we do show is that our models are a good choice for generating TikZ, outperforming these proprietary models under varying conditions. We have added this to our paper as well.

Reviewer comment (Experiments):
How did the authors arrive at the specific prompt that was used for GPT-4 / Claude-2 and how do they know if that is the one that indeed yields a reasonably high performance from the models?

Author response:
We designed our prompts similarly to the ones reported in related work, especially [this one](https://arxiv.org/abs/2305.18583), which arrived at their prompt through prompt engineering. After also experimenting with our own prompts and other prompts stated in [other works](https://arxiv.org/abs/2303.12712), we found the former to work the best in our case. We have made that more clear in our latest version.

Reviewer comment (Experiments):
Given the iterative re-generation, the models could have also made use of the TikZ compiler error / stack trace, what was the rationale for not using this information?

Author response:
Due to GPU constraints, our models were limited to a 1200-token context size. Most examples fit within this size but most were also quite close to the upper limit. Consequently, we cannot fit an additional stack trace and a corrected version within this context. Hence, we offer our iterative resampling technique as a resource-efficient alternative. We have added this information to our latest version.

Reviewer comment (Experiments):
Seems like author didn't show any conversation example where users can edit or optimize the code while in the chat.

Author response:
While we agree that this would be an interesting direction for future work, we want to remark that our models are not general-purpose, instruction-tuned models, i.e., it is not possible to chat with them. We emphasize this more in our latest revision. This deliberate design choice is akin to text-to-image models like Stable Diffusion to which we draw parallels in this work.

Reviewer comment (Novelty):
The error handling strategy seems quite standard and wildly used in programming languages.

Author response:
Since most related works are about high-resource programming languages, they can leverage the much richer available tooling to prevent errors. Consequently, we are not aware of any other works which employ our error mitigating technique designed for low-resource settings. Therefore, we do not think it is as a standard technique and we believe it is a meaningful contribution.