Paper: Modelling complex vector drawings with stroke-clouds (ICLR-2024)

Reviewer comment (Experiments):
How well do you expect the system to generalise to other objects?

Author response:
**(R3-6) Generating a sketch of a particular noun class**

Enabling more classes is possible in a naive setting by sourcing a similar dataset where we can readily obtain vectorised training sketches, and training on millions of these sketches. This was however not possible within the rebuttal period, especially recognising our limited computation resources.  Another option could be conditioning our SRM module on feature spaces of large pretrained models and focusing on the transfer learning problem, this is however out of the scope of our work.

**Summary**

Given the answers 3-6, we expect our model to generalize well to new classes and styles, with stroke design being an area of study moving forward.

Reviewer comment (Evaluation):
How well do you expect the system to generalise to other points of view?

Author response:
**(R3-5) Results view angles and classes**

The short answer is the results simply reflect the training data. We train only on the dataset of faces. But of course, our method is agnostic to the exact nature of data, as long as they can be categorized as *complex sketches*.

Additional results for a different dataset (QuickDraw) are shown in the newly added Figure 17 in the Appendix.

**Summary**

Given the answers 3-6, we expect our model to generalize well to new classes and styles, with stroke design being an area of study moving forward.

Reviewer comment (Evaluation):
How well do you expect the system to generalise to the true complexity of real strokes?

Author response:
**(R3-3) The use of quadratic curves to represent strokes (strokes complexity)**

First, we would like to stress that, theoretically, SRM has no restriction on the exact stroke model to use. The reason why we used quadratic Bezier curves is that the method used to create vector sketches produces fairly simple strokes, and quadratic Bezier curves are more than enough to model strokes in the training set. 

To show that our model supports Bezier curves of a high degree (can represent more complex strokes), we train on the sketches from the QuickDraw dataset with Bezier curves with 15 control points.
We added Figure 17 to the Appendix which shows that our model can flawlessly support more complex curves.

Reviewer comment (Experiments):
Why did you conduct no experiment?

Author response:
**(R3-7) Numerical evaluation** 

Thanks. We have added a comparison of the FID scores of the generated drawings under different sampling conditions in Section 4.2 Table 1.

Reviewer comment (Novelty):
Why have you not cited any paper prior to 2015, when there is plenty of relevant work?

Author response:
**(R3-2) Additional references**

We focus our discussion on recent papers as the focus of our paper is on deep generative approaches for vector graphics. We are happy to provide a more extensive discussion of other related topics if reviewers think that this will benefit the paper's exposition.

Reviewer comment (Evaluation):
I think the main weakness of this paper is the poor visual quality, which is far from artistic application. And the stroke design of the Bezier curve is too simple. Could the Bezier curve contain more parameters such as width and transparency?

Author response:
**(R2-1)** **The visual quality of the sketches**
We focus on *complex vector sketch* generation as a new genre where sketches can have a large number of constituent strokes. We make no assumption about the exact nature of sketches in our theory as long as it fits into the said criteria. However, one needs to take into account the fact that there are no large-scale vector datasets of complex sketches available. The limitation on the visual quality of sketches primarily comes from the training data and the quality of the outputs of the available vectorization methods.

**(R2-2) Bezier curves complexity**
Firstly, we would like to mention that, even though we used quadratic Bezier curves as a design choice, we never made any hard assumptions about the exact stroke representation in our theoretical setup. To further clarify any doubts, we have now added Figure 17 to the appendix, generating sketches from a model trained on the QuickDraw dataset. In this case, we used Bezier curves with 15 control points (able to represent significantly complex strokes). It can be seen that our model copes well with more complex strokes when the appropriate training data is available.

**(R2-3) Additional stroke parameters**
Thanks, great suggestion. While we leave extensive evaluation of adding additional properties to strokes to future work, as a proof of concept, we experimented with adding line thickness to a set of control point positions, describing each stroke. A newly added Figure 16 in the supplemental shows that our model manages to leverage line thickness as well and represents eyelashes with thicker lines, showcasing flexibility in our set transformer approach.

Reviewer comment (Experiments):
Lack of comparison. I do recognize that there may be no methods similar to your technique route. But you should compare with at least one method related to the topic of sketch generation, even though there may be raster-based such as GANs.

Author response:
**(R2-3) Quantitate results**

Thank you for the suggestion. We have added a comparison of the FID scores of the generated drawings under different sampling conditions in section 4.2 Table 1.

Reviewer comment (Experiments):
There seems to be a flaw in the experiment of section 4.4, figure 8. I can see the sketches corresponding to the interpolated results in Figure 8 (the middle ones) are obviously not in the correct domain. This may be because your interpolation function does not fit this distribution (and I didn't see your interpolation function), which leads to the interpolated latent codes are not on their true distribution（you can analogy the manifold of Swiss Roll).

Author response:
**(R2-4) Interpolation results in Figure 8**
We agree that the interpolated results in Figure 8 (now Figure 10.) are of lower quality. We however argue that this might not be a limitation of our SRM module but is a result of the lack of training data for the latent generative model. This however should not weaken our main contribution which is complex vector sketch generation. We will now move this Figure to the Appendix and provide a careful discussion around it.

Reviewer comment (Experiments):
The authors clearly point out that the number of strokes is an issue, but simply brush it away by saying we use 1000 strokes. This is a glaring flaw. What if the sketch is simpler? What happens to the other strokes? Are they duplicated?

Author response:
**Strokes sampling during inference**

SRM (Set Reconstruction Module) is trained on sketches with a variable number of strokes, thanks to the arbitrary sum (i.e. $\sum_{\mathbf{s}\in \mathcal{S}}$) present in the true likelihood in Eq(3). Our SRM is conditioned on a latent vector, and at inference time, we can indeed generate any number of strokes. However, this is a parameter that has to be set manually. In Section 4.2 we discuss what happens when we vary the number of strokes during generation, and Figure 4 shows the results with varying numbers of strokes. For the results in the paper (unless specified otherwise), we generated 1000 strokes per sketch as we found it to produce good results. Since we assume that strokes are IID, we need to sample more strokes than the average cardinality of sketches in the training set (in Anime-Vec10k, there are 305 strokes per sketch on average). We discuss in detail in Section 4.2 oversampling and under-sampling.

Reviewer comment (Experiments):
The discussion in Appendix E is simply not enough to provide a solution to the number of strokes issue.

Author response:
**Strokes sampling during inference**

SRM (Set Reconstruction Module) is trained on sketches with a variable number of strokes, thanks to the arbitrary sum (i.e. $\sum_{\mathbf{s}\in \mathcal{S}}$) present in the true likelihood in Eq(3). Our SRM is conditioned on a latent vector, and at inference time, we can indeed generate any number of strokes. However, this is a parameter that has to be set manually. In Section 4.2 we discuss what happens when we vary the number of strokes during generation, and Figure 4 shows the results with varying numbers of strokes. For the results in the paper (unless specified otherwise), we generated 1000 strokes per sketch as we found it to produce good results. Since we assume that strokes are IID, we need to sample more strokes than the average cardinality of sketches in the training set (in Anime-Vec10k, there are 305 strokes per sketch on average). We discuss in detail in Section 4.2 oversampling and under-sampling.

Reviewer comment (Theory):
In the diffusion model equation (3), the authors decompose the sketch into iid strokes, which is a **VERY** strong assumption. There is no explanation for how, during sampling, one cannot get a degenerate solution of repetitively denoising into the same stroke, nor is there any discussion of how the iid sampling affects the proposed pipeline.

Author response:
**Assumption of IID strokes:** 

We do not claim that strokes are independent — the strokes are indeed correlated in their true sense. However, we use the fact that the strokes are conditionally independent given a set representation (i.e. a compact latent code), powered by De-Finetti’s Theorem. The implication of this is that conditioned on a latent set representation, our SRM model will not result in a degenerate solution.

Reviewer comment (Evaluation):
The paper appears to take in a **variable** number of input strokes and generate a **fixed** (1000) number of output strokes. This contradicts the paper's apparent motivation and suggestion that the model can generate a variable number of strokes, implying that such capability might not actually be possible.

Author response:
**Strokes sampling during inference**

SRM (Set Reconstruction Module) is trained on sketches with a variable number of strokes, thanks to the arbitrary sum (i.e. $\sum_{\mathbf{s}\in \mathcal{S}}$) present in the true likelihood in Eq(3). Our SRM is conditioned on a latent vector, and at inference time, we can indeed generate any number of strokes. However, this is a parameter that has to be set manually. In Section 4.2 we discuss what happens when we vary the number of strokes during generation, and Figure 4 shows the results with varying numbers of strokes. For the results in the paper (unless specified otherwise), we generated 1000 strokes per sketch as we found it to produce good results. Since we assume that strokes are IID, we need to sample more strokes than the average cardinality of sketches in the training set (in Anime-Vec10k, there are 305 strokes per sketch on average). We discuss in detail in Section 4.2 oversampling and under-sampling.

Reviewer comment (Evaluation):
While the paper contains a decent number of qualitative results, the dataset itself is small (10k samples). The authors should check for overfitting by visualizing the closest training set example to each generated sample, both in RGB space and perceptual space (e.g., using VGG features), following sketch retrieval or similar literature.

Author response:
**Additional qualitative results** 

Thanks for the suggestions! We have added additional qualitative results in Figure 7 and Figure 12 as well as examples of the training data in Figure 13.

Reviewer comment (Presentation):
To better analyze dataset and generation quality, the authors should include a large 10x10 grid of samples from both the training set and generated outputs somewhere in the paper.

Author response:
**Additional qualitative results** 

Thanks for the suggestions! We have added additional qualitative results in Figure 7 and Figure 12 as well as examples of the training data in Figure 13.

Reviewer comment (Novelty):
Missing citations — the paper omits citing relevant works:  
1. Paschalidou et al., "ATISS: Autoregressive Transformers for Indoor Scene Synthesis," NeurIPS 2021.  
2. Para et al., "COFS: COntrollable Furniture Layout Synthesis," SIGGRAPH 2023.

Author response:
**Additional references**

Thanks for the additional references!
We added a discussion of additional works to a revised introduction.

Reviewer comment (Evaluation):
Figure 5 Question — Why does the DDPM sampler seem to have more strokes than the DDIM sampler? Or are the strokes just placed closer together in the DDIM sample?

Author response:
**Figure 5: Why does the DDPM sampler seem to have more strokes than the DDIM sampler? or are the strokes just placed closer together in the DDIM sample?**

All samples generated in Figure 5 have the same number of generated strokes. Since DDPM sampling is stochastic, there is more diversity in stroke positioning. In DDIM sampling (which is deterministic), strokes are positioned so closely that it is hard to see that there are multiple strokes on top of each other. In Figure 15 in the Appendix, we reduce stroke width to better show the alignment between strokes for the DDIM sampling method.

Reviewer comment (Reproducibility):
Question — Do the authors plan to release the dataset?

Author response:
**Do you plan to release the dataset?**

Yes, most definitely — this is a dataset that (we think) is much needed to progress the sketch community in terms of studying complex sketches. We will also release the code as well.