Paper: Learning with Language-Guided State Abstractions (ICLR-2024)

Reviewer comment (Novelty):
The primary weakness of the paper lies in the absence of a lack of substantial technical contribution. While the proposed strategy mitigates the issue of spurious correlations, it appears more as a choice of system design to generate heuristics rather than a fundamental method to identify useful state features. Notably, the importance of state features not only depends on task semantics, but also the low-level geometric constraints imposed by the environment and robot embodiment. For example, some objects not mentioned in the task specification may be important for robot collision avoidance. Consequently, the learning of important state features and motion policies are coupled and should ideally be learned together, such as in VIOLA [2].

Author response:
> it appears more as a choice of system design to generate heuristics rather than a fundamental method

We respectfully disagree with the reviewer that, because we are demonstrating a capability of applying language priors to imitation learning, our contribution is not “a fundamental method”. 

First, as we know from the recent boom of LM+robotics papers, the premise of applying strong pretrained language priors to problems of decision-making is of high value and immensely impactful to the field. We simply take an alternative approach from those of previous methods: rather than leveraging LMs for *action abstraction* i.e. action stitching as in SayCan, we use them for *state abstraction*, i.e. identifying relevant features. LGA therefore demonstrates the utility of strong LM priors in a novel facet of decision-making. 

Second, we agree that VIOLA is an important related work and will update our paper to discuss it in more detail. However, it is not evident to us that learning important state features should *ideally* be learned with the motion policy: indeed, this forces the agent to re-learn motor policies for novel tasks (e.g. a new linguistic specification), whereas our approach allows the re-use of motor policies across tasks (if the new linguistic utterance has been found to be the same as previously learned task with an alternative specification). However, we believe that learning how to blend these different state representations (both abstract and raw) is an important direction for future work, and will emphasize in the discussion.

Finally, we evaluate LGA with BC here since it is the clearest, simplest method for isolating the impact of good abstractions, but it is entirely feasible to utilize the abstraction generation process of LGA in tandem with a method like VIOLA for jointly learning well-attended policies (e.g. by also conditioning the transformer-based policy in VIOLA on LGA-generated abstractions). We are happy to clarify this scoping of LGA in the new revision.

Reviewer comment (Experiments):
Another drawback is that the imitation learning setup is overly simplified. The use of VIMA [1] benchmark, which employs high-level primitive actions like “pick” and “place” with continuous goal poses, significantly simplifies the training of a behavior cloning policy. Given that the major objective is to assess the proposed state abstraction in imitation learning, it would be advisable for the authors to consider a more rigorous robot manipulation benchmark with continuous actions, such as RoboSuite [3].

Author response:
> VIMA is too simple

We appreciate the reviewer for pointing out the limitations of VIMA as an evaluation domain, and for the suggestion of RoboSuite. Although evaluating on harder benchmarks is a great suggestion for future work, we felt it prudent to instead spend time on demonstrating the practical feasibility of LGA on a real robot. As we wrote about in Sec. 7, we demonstrated the full LGA pipeline on a Spot robot, including using raw robot observations segmented via SAM, captioned via Dedic, and then imitation learned on real user tele-op’d trajectories. Consistently (~9/10 test tasks), the state abstractions produced via LGA were able to produce policies successful at completing the tasks. The few failure cases observed were due to the captioning model failing (e.g. SAM reliably detected different shapes and sizes of water bottles but Dedic was unable to always produce the text description “water bottle”). 

Thus, the LGA pipeline can be reliably deployed in real-world scenarios, and we expect that it can be straightforwardly generalized to even more complex ones, especially as individual components of the pipeline improve. While we can always evaluate on harder/alternative simulated benchmarks, we hope the reviewer appreciates the effort spent on ensuring the full pipeline is able to be practically implemented with real hardware!

Reviewer comment (Novelty):
State abstraction is a fundamental problem in decision making. I think it would be helpful for the readers to understand the context better if the authors refer and discuss the related literature (such as [4,5]).

Author response:
We thank the reviewer for bringing missing references to our attention, for they indeed help frame the historical context of state abstraction in decision-making! We will add them to the related work.

Reviewer comment (Reproducibility):
Regarding the observation input to behavior cloning in LGA, do you use the binary mask directly (as visualized in Figure 1) or the masked image? If you use the former one, I wonder how the latter one works.

Author response:
> do you use the binary mask directly (as visualized in Figure 1) or the masked image?

We used the binary mask directly, although the reviewer suggests a reasonable alternative. In our approach, we explicitly highlighted the flexibility of LGA to operate on both the binary mask as well as one conditioned on the raw observation, although note we did not include masked image as an alternative. We suspect that for tasks where texture/color is not meaningful, e.g. once the system has detected it’s an image, there’s no need to preserve the color “orange” in the observation”, although either would likely work. We’re happy to clarify this additional option in the revision.