Paper: RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches (ICLR-2024)

Reviewer comment (Experiments):
The model, like its predecessor, the Robot Transformer (RT), remains an end-to-end agent. Although it showcases impressive performance, task comprehension is tied to a specific robot, hindering the system's ability to undertake out-of-domain tasks or generalize across various robots without necessitating retraining.

Author response:
While we implement and evaluate our approach on a particular robot platform, we would like to emphasize that there is nothing in principle preventing our approach from applying to other robot embodiments and manipulation tasks more broadly. The primary contribution of this work is a framework for training a sketch-conditioned IL agent, without specific assumptions on the robot platform or even IL algorithm used. We are currently working on extending this platform to a Franka Panda robot by collecting demonstrations for a rearrangement task, and training the image-to-sketch GAN on a few paired image-sketch examples. 

Given recent efforts around crowdsourcing robot datasets [1], our image-to-sketch network could also certainly be trained across different robot embodiment images and paired sketches. This general-purpose image-to-sketch translation network could then be used to post-process many manipulation datasets in a hindsight fashion, enabling sketch-IL for free on different robots without any additional demo collection.

Reviewer comment (Novelty):
In terms of communication, a sketch represents more than just an enhanced image; it abstracts visual information and fosters the emergence of graphical conventions to boost efficiency (Qiu et al., 2022; Chen et al., 2023). However, the sketches in this work, even at the lowest specificity, seem to be merely processed images rather than abstractions, making this work more like an augmented version of RT-1 to sketch images.

Author response:
We would like to clarify that although we train on GAN-generated sketches (amidst other augmentations), **we evaluate RT-Sketch on real, hand-drawn sketches at test-time across all experiments.** We also come up with a principled way to evaluate whether RT-Sketch can handle sketches with varied levels of input specificity, ranging from free-hand drawings to line sketches to sketches with shading, all of which are drawn by hand at test-time rather than GAN-generated.

Reviewer comment (Evaluation):
Overall the reviewer is not convinced that the examples and arguments given motivate the superiority of using a sketch as a goal over using natural language.

Author response:
We apologize if the current framing makes it seem as though we view sketches as being strictly favorable to language in all cases. While we point out situations in which language alone might be ambiguous and sketches have a potential for allowing more spatial-aware task specification, this is not to say that language should be replaced by sketches altogether. We have made changes to the introduction and conclusion to better contextualize this work with this in mind.  

Regarding what scenarios language alone may be inconvenient, and where a sketch can serve as a meaningful alternative or complementary modality, we have since added [a section on our website](https://rt-sketch-anon.github.io/additional.html#why_sketches) with motivating examples.

Our focus on sketches is largely an exploration around pushing the boundaries of goal specification more broadly beyond traditional modalities like language, and the design choices we can make to enable this at all. In the future, we are certainly excited about multimodal goal specification, and as a step towards this, we have since implemented a sketch-and-language-conditioned model which performs on par with, and better in some cases, than either modality alone. Specifically, in the reorientation task which our sketch-alone policy initially struggled with, we observe that sketches alleviate spatial ambiguity introduced by language, while language can help specify that reorientation (over just repositioning) is needed. Please refer to the [Additional Experiments section](https://rt-sketch-anon.github.io/additional.html#multimodal) on the website for more details. This result further suggests that sketches can powerfully complement other modalities.

Reviewer comment (Theory):
Regarding the granularity argument that the authors use in the introduction including the examples of "put utensils, ..., on the table" and "put the fork 2cm to the right...". Doesn't this demonstrate the flexibility that language has as a goal? Even if a human had to communicate the placement of utensils on a table, this would still seem easier than drawing a corresponding representation for an entire table.

Author response:
We agree that these examples do illustrate the flexibility language offers, but we retain that having so much flexibility can also become highly inconvenient as the complexity of a task scales or the degree of spatial precision matters. For instance, in the given example of setting a table with multiple plates, utensils, napkins, decorations, etc., it would seem very inconvenient for a user to have to provide an instruction for the placement of every single object, along with such fine-grained corrections. On the other hand, a very long-winded description that captures all the placements of the objects in one shot would be difficult for any language-conditioned agent to interpret, let alone for someone to describe. Even an imperfect sketch of the objects and their relative placements could be quick to provide, and provide much more spatial context than a sentence for the policy to be able to roughly place objects. 

We have also shown in our experiments with visual distractors that sketches are capable of attending only to the objects we care about manipulating. Here, a human need not draw the entire table in one shot, but can use partial sketches to indicate subgoals in a long-horizon sequence. We fully agree that viewing these modalities in isolation is not necessarily the answer; a combination of a rough sketch, and verbal corrections, for instance, can go a long way in this scenario and others described in our [motivating examples on the website.](https://rt-sketch-anon.github.io/additional.html#why_sketches)

Reviewer comment (Evaluation):
It is not obvious how big of a difference the errors are when looking at the RMSE in pixels. Could the errors when finding the centroid and the manual keypoint annotations be an issue in measuring this? Can a visualization be created to show this?

Author response:
We agree that the numbers alone are difficult to contextualize without a visualization, and have since updated our Appendix and website [here](https://rt-sketch-anon.github.io/additional.html#eval_metrics) to illustrate the discrepancy in achieved vs. target object placements in different images, with which we measure RMSE. As in our response above, we think that using manual keypoint annotation actually allows us to more accurately quantify policy imprecision without confounding with object detection errors.

Reviewer comment (Presentation):
Typically bolded numbers in a column/row represent the method with best performance. However, in the column for failure occurrences, RT-Goal-Image is bolded despite having the highest failure occurrence.

Author response:
Thanks for your attention to detail and apologies for the confusion. We have since changed the bolding/shading in order to obey standard conventions and updated the caption.

Reviewer comment (Presentation):
The meaning of the shading of the cells of Table 1 adds a lot of confusion and should have been defined in the caption to improve readability. This confusion comes because, in one portion of the table, darker gray colors represent lower centroid distance and in another portion it represents the frequency of failures. This frequency of failure metric is also not well defined, how is this different than failure occurrence?

Author response:
Thanks for your attention to detail and apologies for the confusion. We have since changed the bolding/shading in order to obey standard conventions and updated the caption.

Reviewer comment (Experiments):
Given the potential (vast) cost associated with collecting human sketches for various scenarios, perhaps scalability could be a concern when applying this approach to broader scenarios, rather than solely relying on the benchmark presented in this study.

Author response:
Scalability is an important consideration, and we kindly ask that you refer to our shared response above for a more thorough response. To highlight a few things, we train our image-to-sketch GAN on 500 examples, which is a mere 0.6% of the overall demonstration trajectory dataset size. This amount of examples is also on par with datasets in vision literature; namely the original PhotoSketching GAN [1] trains on 5,000 sketches, and ControlNet [2] is trained with 1K examples at the minimum. While 500 may still seem like a lot, in reality it took a few hours of data collection over 3 days, and we believe the benefits of training this GAN are worth it if the end result is a much more convenient means of goal specification which  takes seconds to provide. 

Also, while goal-image datasets are very much tied to the environment they were collected in, we could certainly train our image-to-sketch network on the same amount of images spanning different datasets. The original PhotoSketching GAN which we fine-tuned was actually trained on internet-scale data with varied categories of people, animals, etc., so training this network with multiple robot environments is certainly within scope. This would be one straightforward way to scale sketch-based IL to multiple robot setups without the need to collect any additional demonstration or sketch data.

Reviewer comment (Experiments):
The authors did not conduct experiments to determine how various image-to-sketch generation methods can affect the final results of RT-Sketch, which could potentially create a bottleneck in the entire pipeline.

Author response:
As you point out, the choice of what technique we use for image-to-sketch generation is critical to the overall success of the pipeline. We did in fact experiment with various approaches for image-to-sketch generation before converging on the proposed method. We have since updated our website with [additional results for alternate image-to-sketch approaches](https://rt-sketch-anon.github.io/additional.html#alt_methods) we tried, and their pitfalls. A critical challenge is that we want generated sketches to be minimal in nature (i.e. background objects are ignored, rough outlines are acceptable). Within the vision and graphics communities, the most recent work on image-to-sketch generation (CLIPasso, CLIPAscene) is concerned with producing aesthetically pleasing results, which is not a priority in our setting. We find that a GAN-based approach gives us control over the kinds of output sketches we would like, and allows us to make use of existing large-scale, crowdsourced data of paired images and sketches while just fine-tuning on our own data. 

We also acknowledge that sketch-to-image generation is another possibility to consider. We did consider state-of-the-art models like ControlNet and other Diffusion-type variants for image synthesis, but the challenge here is still overcoming hallucinations in generated images and jointly enabling style transfer (generated images must match the style of robot images if we want to condition policies on them). That said, generative models for image synthesis are ever-improving, and the RT-Sketch policy architecture is agnostic to the exact choice of visual input. In the future, we are excited about revisiting these techniques for sketch-based IL, but we hope that this provides more context into our design choices for the time being.

Reviewer comment (Novelty):
In recent years, there has been a significant body of work at the intersection of sketches for visual understanding. I would suggest the author add one separate subsection discussing a few major works on 'Sketch for Visual Understanding' in the related work parts. The authors could use this for their reference: https://github.com/MarkMoHR/Awesome-Sketch-Based-Applications/blob/master/README.md. Some representative works include: a) https://arxiv.org/abs/2303.15149, CVPR'23, b) https://arxiv.org/pdf/2302.05543.pdf (ControlNet uses sketch for image generation), ICCV'23, c) https://arxiv.org/pdf/2303.11502.pdf, CVPR'23, d) https://arxiv.org/abs/2203.14843, CVPR'22, e) https://arxiv.org/abs/2204.11964, CVPR'23.

Author response:
Thanks for putting these alternate sketch-image translation approaches on our radar. In the [updated draft](https://rt-sketch-anon.github.io/assets/rt_sketch_revised.pdf), we have since referenced them throughout Related Works and greatly appreciate the references.

Reviewer comment (Experiments):
Free-hand sketches and edge maps are different, and many existing works on sketches have claimed that models trained from edge maps do not generalize well to free-hand sketches. Some relevant works are https://openaccess.thecvf.com/content/CVPR2023/papers/Koley_Picture_That_Sketch_Photorealistic_Image_Generation_From_Abstract_Sketches_CVPR_2023_paper.pdf and https://github.com/mtli/PhotoSketch. Some discussions around that could be helpful.

Author response:
This is a great point. We do fully acknowledge that there is a potentially large domain gap between edge-detected / GAN-generated images and manually drawn sketches, but in the absence of manual sketches at scale, we make certain design choices to make use of modalities available to us. Specifically, we train RT-Sketch on a mix of GAN-generated sketches (60%) to be able to handle hand-drawn sketches at test-time, but we include edge-detected images (20%) and goal images (20%) to encourage the policy to afford variations in style, shading, and level of detail in sketches.

Empirically, we do find that RT-Sketch performs well on hand-drawn sketches at test-time, as validated by H1-4, despite being trained on these non hand-drawn representations. This finding suggests that our image-to-sketch GAN produces reasonably realistic sketches in the first place, such that real sketches do not throw the policy off at test-time.

We are also encouraged, somewhat surprisingly, that the policy can handle colored sketches and completely free-hand drawings, despite not being trained on such representations. Since training on GAN-generated line sketches alone would be a huge distribution shift from these kinds of sketches, this suggests that our affine/color augmentations and the inclusion of other  modalities (edges/goal images) enables this emergent robustness. Prior work in sketch-based object detection has found that similar kinds of augmentations can help bridge the domain gap [1].

Finally, we note that we use a Transformer backbone [2], and Transformers have been shown to have a high model capacity; we posit that this might enable the policy to interpret these different modalities well.

Reviewer comment (Presentation):
Some self-contained caption could be helpful for Fig. 2.

Author response:
Thank you! We agree and have updated the caption.

Reviewer comment (Presentation):
Figure 3 could benefit from a more comprehensive caption to guide the reader through the observations, making it easier to understand.

Author response:
We have since updated the caption for Figure 3, with changes highlighted in blue, but please let us know if anything is still unclear.

Reviewer comment (Presentation):
Is it possible to add more visual examples where a sketch is found to be better than a text-only counterpart?

Author response:
Absolutely, we agree that the motivation for sketches could be made more clear. We have since addressed this in the shared response as well as in this [section on the website](https://rt-sketch-anon.github.io/additional.html#why_sketches) where we illustrate scenarios in which sketches could either complement or address limitations of language alone.