Paper: Every Mistake Counts: Spatial and Temporal Beliefs for Mistake Detection in Assembly Tasks (ICLR-2024)

Reviewer comment (Presentation):
The paper does not discuss any limitations.

Author response:
Thanks for this feedback! We acknowledge that, in its current form,  our current approach doesn't generalize well to new object parts and is heavily influenced by perception quality. We'll incorporate these points into our revision.

Reviewer comment (Writing):
The paper lacks a better exposition (text quality, mathematical notation, figures), which makes it difficult to fully assess the contribution.

Author response:
We appreciate the reviewer's feedback and will clarify and refine our contributions in our paper and will enhance its clarity.

Reviewer comment (Experiments):
Missing ablation studies.

Author response:
Unfortunately, our framework does not allow for various ablations. In Table 2,  we've explored various perception module levels to show the gap between GT and computer vision-based perception modules.

Reviewer comment (Presentation):
The in-text description of Figure 2 refers to 'edges', but no edges are shown in Figure 2 (left). Does this mean the graph has no edges?

Author response:
Thanks for bringing this to our attention!  We’ll revise the corresponding text to refer to Figure 2 (right), when discussing edges,

Reviewer comment (Writing):
Section 3.2: The text states that 'completion occurs when the episodic context M fully traverses the graph'. Does this refer to the rule graph?

Author response:
Yes, your understanding is correct! It does refer to the rule graph.

Reviewer comment (Writing):
What is a 'focal component pair'?

Author response:
The term is to denote the part pairs of interest that the temporal rule is designed for as described in Sec3.3 (the second paragraph).

Reviewer comment (Writing):
The mathematical notation appears convoluted and more difficult to follow than it seems necessary. After reading Section 3.2 and 3.3, it was not clear how Spatial and Temporal beliefs are defined.

Author response:
Sec 3.2 explains that spatial belief is defined as the allowed connection between parts, which is also considered as the edge of the rule graph. Sec 3.3 essentially talks about two types of temporal beliefs and how they can be used to infer ordering mistakes.

Reviewer comment (Writing):
Section 4.3: What are 'fine-grained' mistakes?

Author response:
The coarse mistakes only care about mistakes, correct and corrections without accounting for their root causes. Following this, the fine-grained mistakes are introduced in Sec 4.2 (the last paragraph on page 7), which include generic ordering mistakes (B1), accumulated mistakes (B2), unnecessary detachment (B3), and Misorientation mistakes (B4). In  addition, we provided more information on the mistake types in our supplementary.

Reviewer comment (Experiments):
To assess the performance of the Inferences vs the BeliefBuilder, I would appreciate ablation studies.

Author response:
The BeliefBuild essentially builds the rule graph which is later used by the Inferencer during testing only. We therefore cannot compare the Inferencer vs the Beliefbuilder.

Reviewer comment (Experiments):
Trivial task definition: Examining only the errors related to the ordering in assembling tasks using the ground truth annotated object names and action verbs seems trivial, unless the issues related to orientation or fastening and the uncertainties in the object detection, shape, size or pose estimation are taken into consideration.

Author response:
**A**: We focus on ordering mistakes in this paper, which comprise 2/3 of overall mistakes in Assembly101 and require temporal reasoning. We do agree with the reviewer that object pose prediction and action recognition are worth investigating as challenging tasks in the computer vision domain. We refrain from shifting our attention to such aspects in this work for the following considerations:

- Object pose prediction [A] primarily focuses on large objects, requiring object meshes for seen classes for some models; Assembly101 lacks these meshes. Additionally, studying orientation issues is a frame-based investigation problem, which is also orthogonal to our temporal reasoning for the ordering mistake detection.
- Action recognition on the well-studied egocentric dataset Epic-Kitchens reaches only 56% accuracy [B]. Research [C] on adverbs like "tight" modifying the verb "fasten" has been limited to small-scale studies, achieving 71.9% accuracy for just 5 adverbs.

Reviewer comment (Novelty):
Limitations in the approach: The proposed approach is based on pure logic, set theory and graph representation of the spatial and the temporal relations. The entire pipeline relies on hard-coded logic and does not employ any learning mechanisms, leading to re-initialisation and re-population of the spatial and temporal knowledge, along with the preconditions, each time this method needs to assess a new object assembly. On top of that further improvements in this deterministic pipeline needs to be manually accounted for, hence inclusion of this paper is of little importance to the learning community.

Author response:
**A**: We acknowledge that our approach is based on logic which build the spatial and temproal knowledge, however, we respectfully disagree with the reviewer that the inclusion of our work in ICLR is of less importance. Specifically, our spatial and temproal beliefs can be of great interest to the audience working with robotics, planning.

Reviewer comment (Experiments):
Limited evaluation: The Assembly 101 dataset has only 328 distinct action sequences to construct 101 different toys and out of this only 1 randomly selected sequence is used for testing for each toy and remaining are used to create the spatial & temporal knowledge and the preconditions. The complexity of sequence in assembling a toy is not an apt illustration of the difficulty in assembling an object with numerous parts, involving a large complex sequence, say an electronic gadget, automobile etc. In addition, this dataset is too small in size to effectively represent all the possible scenarios of ordering mistakes that occur in real-life large scale assembly, for example a factory. The criteria of the logic module appears to be tailor made for the toy assembly dataset. Therefore, its performance in a different assembly scenario, with more parts and complex sequences needs to be sufficiently evaluated to establish the generalizability.

Author response:
**A**: We politely disagree that our approach is tailored for the toy assembly. 
- Our spatial and temporal beliefs are generic, not specific to toy parts; as we tested them on 101 different toys. Through synthetic experiments in our paper, we demonstrate the adaptability of our approach for EGT assembly in industry settings, serving as a proof of concept for its potential to generalize and improve assembly tasks across various domains.
- Assembly101 is the sole dataset with these annotations. And it encompasses intricate assembly scenarios, averaging 14 steps, making it a valuable resource for studying mistake detection and learning. Even adults assembling children's toys make numerous mistakes, highlighting the complexity of the task and the need for improved learning strategies.

Reviewer comment (Theory):
Limitations in the logic: In Tab. 1, the proposed method shows a significant improvement over Sohn et al. (2020), but that is likely because of the inclusion of an additional case (intransitive temporal error) in the logic module. Yet, the proposed approach is unable to achieve 100% accuracy, precision or recall, clearly showing gaps in their logic module to determine all the possible cases of preconditions. Similarly in Tab. 2,  the proposed approach performs better than the existing methods, but I am unable to contemplate a reason for its failure to achieve perfect accuracy and F1, except the possibility of missing out on certain scenarios in its logic criterion, despite being a hard coded algorithm.

Author response:
**A**: The reason that our approach is not 100% accurate is due to the training data.  Our approach's accuracy is limited by training data constraints. When we only have access to a subset of error-free action sequences, it will hinder our algorithm's ability to handle mistakes during testing. We performed our algorithm over 4 splits to account for this.

Reviewer comment (Writing):
Clumsy Notations: Too many redundant symbols and equations appear in the paper whose purpose could have been simply stated in words.

Author response:
**A**: Thanks for the comment. 
- We do state everything in words, either above or below the equations. We found that the equations provide a precise and concise way to present complex relationships, making the information more unambiguous. They enable precise communication of ideas which can be challenging to convey in words alone.   
- Thanks for pointing this out and we’ll highlight the differences and update in the figures.

Reviewer comment (Presentation):
Clarity in Fig 3: Edges for both transitive and intransitive graphs have the same color. Especially in the intransitive graph, the “final mistake” edge could have been highlighted differently.

Author response:
**A**: Thanks for the comment. 
- We do state everything in words, either above or below the equations. We found that the equations provide a precise and concise way to present complex relationships, making the information more unambiguous. They enable precise communication of ideas which can be challenging to convey in words alone.   
- Thanks for pointing this out and we’ll highlight the differences and update in the figures.

Reviewer comment (Experiments):
Is it really difficult to integrate state of the art perception with this pipeline? If so, why? It would be really helpful if you could provide the end-to-end results of this method and state-of-the-art (SOTA) with perception. A real-time video demonstration showing the ordering mistake detection in the assembly task, using only the RGB frame, would be very intriguing.

Author response:
**A**: In this paper, we decouple the perception and reasoning modules. We can utilize the output of the perception module, simply by replacing the GT labels with the output of the perception module which are expected to be the per-frame/segment action labels,  as seen in Table 2.

**A**: We integrate our mistake detection module with the perception module's outputs in Table 2. Notably, when AR Acc is 86.4%, errors in the perception module result in a decrease in mistake recall to 42.3%, highlighting the module's impact. 

**A**: We will provide these for both ground truth labels and perception module outputs.

Reviewer comment (Experiments):
If perception is included in this pipeline and you are able to estimate the object pose, how difficult would it be to extend this method to detect orientation errors?

Author response:
**A**: Orientation is independent of temporal reasoning and can be addressed as a frame-wise issue through object pose prediction. We expect that with an accurate object pose prediction method, detecting orientation mistakes is feasible. 
To avoid confusion, we are considering renaming our paper to clarify that we focus on detecting ordering mistakes.

Reviewer comment (Writing):
Writing needs some improvements. For example, the task needs to be explained better. Especially for readers that are unfamiliar with the assembly ordering mistake task, it is not clear whether the input is a sequence of images or labels representing the actions. I assume it is the latter, since the authors provide a separate Visual Integration experiment towards the end of the paper. Moreover, if the part-to-part annotations and the synthetic dataset are treated as contributions, then more information should be included in the main paper, rather than the supplementary material. Please add a couple examples of the synthetic sequences with mistakes.

Author response:
- Yes, our approach generally takes as input labels and such labels can be obtained from multiple sources, for example, the video annotation, parsed from narration in the video, or the output from a perception module. In this work, we evaluate our approach with both annotation and perception outputs in Table 2. 
-  Thanks for highlighting the synthetic data issues. We'll add the examples to the main paper for revision.

Reviewer comment (Theory):
In my understanding, the BeliefBuilder is creating an explicit rule graph that only applies to a specific object and its set of components. Aren't explicit knowledge grounded beliefs very limited in terms of generalization? Wouldn't it be more useful to learn a soft graph with latent representations that could potentially be applied to new objects?

Author response:
We haven't explicitly tested for generalization, but our model is expected to perform reasonably for shared part pairs like interior-cabin and sound module-roof on different toys. However, for novel object parts across toys, prior knowledge is necessary, e.g., informing the model about the similarity of attaching a dumper vs a container which are usually positioned in the rear part of the toy. One option to get such information could be through leveraging LLMs, so that we can generalize to new objects like unknown containers by recognizing structural similarities aligned with linguistic semantics. We thank the reviewer and will incorporate these discussions into our paper.

Reviewer comment (Evaluation):
What is the performance of the Inferencer on Seen vs Unseen objects?

Author response:
As we mentioned above, we focus on seen objects in this paper, but with structural prior knowledge, our rule graphs can be extended to unseen objects.

Reviewer comment (Writing):
The paper is hard to read and follow. The system is described in a complicated way, though the underlying mechanism is straightforward.

Author response:
Thank you for your comments. Even though the underlying mechanism appears straightforward at first glance, unraveling the less apparent challenges within ordering mistake detection requires a detailed and comprehensive description. We will improve the readability and ease of comprehension to ensure our paper is accessible to a broader audience.

Reviewer comment (Experiments):
The method avoids perception and reasoning, but instead resorts to human annotations for the parts and the steps, which is an unrealistic simplification of the problem.

Author response:
We respectfully disagree.  Our work simply separates perception and reasoning into two separate components, and two are developed independently.  
* Reasoning: the separation of reasoning from the perception is standard in robotics research [A];  Mistake detection is essential for many applications, and waiting for perfect perception to be solved is not a practical approach for advancing research.  

* Perception: This is not our main focus so we rely on existing model (TSM), which we train for our problem setting.  We verify that using perception-based outptus integrates seamlessly in our approach as shown Table 2.  While there are some performance drops due to imperfect perception, we don’t think this takes away from the underlying contributions of our proposed work. 


[A] Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement. In ICRA, 2023