Paper: AED: Adaptable Error Detection for Few-shot Imitation Policy (ICLR-2024)

Reviewer comment (Experiments):
Baselines are not recent and strong enough for meaningful comparisons. All the baselines do not have access to the policy network while the proposed method utilizes it. There are a couple of recent works in error detection in robot manipulation, including MoMaRT [1], ThriftyDAgger [2], and some model-based approaches [3] worth comparing.

Author response:
**How does the method compare with recent papers, such as MoMaRT, ThrifyDAgger, etc.? (W1, Q1)**

Thank you for recommending these references.

As explained in Q3, in our setting, we don’t have access to human assistance (intervention and newly labeled data) and environmental exploration before inference, which are the natural limits of the FSI literature. Therefore, the works that need human-in-the-loop (reference [2]) or the world model (reference [3]) cannot be compared with our work.

The reference [1] is already included in our references. However, as we pointed out in the Introduction, these reconstruction-based methods (one-class classification) suffer when dealing with our problem. 

They monitor the policies based on the *reconstruction errors of observations*. However, they will raise errors for **every frame in the novel environments** since they did not see them before. We have tried to add [1] as one of our baselines, but we cannot find appropriate thresholds since it yields large errors no matter if the frame is normal or an error in the novel environments.

Moreover, we actually build our baselines **with various attributes**, including policy dependent, OCC (trained with only normal rollouts), temporal information, and demonstration-conditioned. Please find them in Appendix Table 3. Besides, there are also powerful models in our baselines. For example, DCTED (2021) is a transformer model, which should be more complicated than cVAE used in [1].

However, we are grateful for receiving the references related to our work. We think summarizing and introducing these interactive error detection works is valuable, even though our settings are different. **We will include them in our final paper**.

Please let us know if there is any other paper the reviewer thinks is related to us or fits our setting and should be compared.

Thank you again for suggesting the references.

Reviewer comment (Experiments):
Since both the policy and error detector are conditioned on demonstrations, it’s worth seeing the performance of the error detector under different demonstration qualities. When deploying a human-robot interactive system, we should be able to separate out human error and robot error.

Author response:
**How does the performance change with different demonstration qualities? (W2, Q2)**

It’s really a good question.

Indeed, the demonstration qualities might affect AED methods that leverage the demonstration features. We are now experimenting with this; please wait for our follow-up updates.

Nevertheless, for now, we can still analyze the possible results from the method characteristics:

1. Those methods without accessing the demonstrations should not have significant performance differences, e.g., LSTMED.
2. TaskEmbED should also obtain similar results if the task is still completed in the demonstration since it only uses the first and last frames of demonstrations.
3. The remaining methods might be affected by the demonstration quality.

Among them, SVDDED and DCTED should have a similar influence across different policies.

However, our PrObe’s performance difference might depend on policies since it accesses the demonstration features from the policy.

We will update the actual results once the experiment is completed. Thank you again for this interesting question.

---

Dear Reviewer upjt,

We would like to share that the experiment on the demonstration qualities is completed. However, we found that observing the change patterns is a little hard since the *policy performance will also change* when adjusting the demonstration qualities.

Specifically, the *policy performance usually drops* when the demonstration quality decreases (note that we are in the few-shot imitation problem). As a result, *the AED methods should get a higher AUPRC score* if they are not affected by the quality changes of demonstrations since it’s easier to find an error.

---

Next, we present how the experiment is conducted. We test it on the *Press Button* task. We collect sub-optimal demonstrations that failed to press the button at the first trial but pressed the button successfully on the second try. We provide these sub-optimal demonstrations and let the policies and AED methods perform the task. 

Since our setting is few-shot, we have tested two cases, including
1. all five demonstrations are replaced with sub-optimal ones 
2. only two out of five are sub-optimal demonstrations. 

The results are summarized below.

---

- NaiveDC policy

| Demonstration setting    |    Policy SR    | SVDDED | TaskEmbED | LSTMED |  DCTED |  PrObe |
|--------------------------|:---------------:|:------:|:---------:|:------:|:------:|:------:|
| All optimal              | 51.94% ± 18.79% | 0.6956 |   0.7280  | 0.7051 | 0.8405 | 0.8851 |
| 3 optimal & 2 sub-optimal | 50.83% ± 18.50% | 0.6861 |   0.7407  | 0.7168 | 0.8832 | 0.9059 |
| All sub-optimal | 51.67% ± 17.24% | 0.6414 |   0.7276  | 0.7044 | 0.8446 | 0.8930 |

- DCT policy

| Demonstration setting    |    Policy SR    | SVDDED | TaskEmbED | LSTMED |  DCTED |  PrObe |
|--------------------------|:---------------:|:------:|:---------:|:------:|:------:|:------:|
| All optimal              | 80.56% ± 11.65% | 0.3789 |   0.3597  | 0.4468 | 0.5734 | 0.7474 |
| 3 optimal & 2 sub-optimal | 78.61% ± 14.02% | 0.3908 |   0.3811  | 0.4606 | 0.6019 | 0.7530 |
| All sub-optimal | 77.22% ± 14.16% | 0.3899 |   0.4051  | 0.4718 | 0.5341 | 0.7588 |

- SCAN policy

| Demonstration setting    |    Policy SR    | SVDDED | TaskEmbED | LSTMED |  DCTED |  PrObe |
|--------------------------|:---------------:|:------:|:---------:|:------:|:------:|:------:|
| All optimal              | 75.56% ± 12.68% | 0.4653 |   0.4657  | 0.4487 | 0.6757 | 0.7505 |
| 3 optimal & 2 sub-optimal | 71.67% ± 11.18% | 0.4985 |   0.5165  | 0.4979 | 0.7467 | 0.7804 |
| All sub-optimal | 69.72% ± 12.30% | 0.4886 |   0.5262  | 0.5096 | 0.7140 | 0.7752 |

---

From the results, we can find that **not all AED methods** can achieve a better result when policy performance decreases because **they are also affected** by the dropping of demonstration quality. Notably, our proposed **PrObe is the only method** that achieves better results for all cases - *kindly compare the first and second rows and the first and third rows for each policy*. 

Regarding our previous forecasted result, it can be observed in *the case of NaiveDC policy (all optimal vs. all sub-optimal)* because there is no significant difference in the policy performances. 

As expected, the LSTMED and the TaskEmbED obtained similar AUPRC scores. SVDDED baseline suffered in this situation since it averages every demonstration frame (including those where errors occurred). DCTED and our PrObe get a slightly better result. 

In conclusion, the AED methods without accessing the demonstrations or intermediate process of demonstrations will produce a similar AUPRC score. Besides, the AED methods that have filtered useful information from demonstrations (e.g., DCTED and PrObe) can achieve a better result.

---

Thank you again for this interesting question.

Besides, feel free to let us know if there is any other experiment the reviewer would like to check.

Reviewer comment (Evaluation):
In the visualizations, the failure is detected after the fact (5 frames later), which is undesirable if the damage is already happened. This is expected as the design doesn’t include a world model to predict possible failures. I hope the authors can discuss this in more detail and point out the intended application of the proposed error detector.

Author response:
**What’s the use case of the proposed system if it only detects failure after the fact? (W3, Q3)**

Thank you for this question.

The fundamental assumption of our work is that  we are monitoring the few-shot imitation (FSI) policies, and they usually have the following settings:

- **Training and testing environments are disjoint**. (base env. vs. novel env.)
- **No exploration** before inference in novel environments.
- **The expert only provides demonstrations** in the novel environment but may not be around the robot when it executes.

These settings make our AED methods unable to get the information of human assistance (e.g., intervention, labeling new data) or world dynamic model as used in the references the reviewer listed. A practical scenario of our AED study is provided at the end of our response.

Moreover, we are dealing with robot manipulation tasks, where the robot interacts closely with objects. Therefore, in our definition, an error (failure) occurs (right) after a failed interaction. From this moment, the current observation is inconsistent with demonstrations.

In addition, to our knowledge, most existing studies related to failure detection on robot manipulation also detect the error after the fact, e.g., the reference [1] the reviewer listed. 

We are not saying that our setting is better than the one in [3], but the settings, targeted policies, and scenarios between ours and [3] are different. Thus, in our AED task, the goal is to detect the error as early as possible (after the fact). And our PrObe detects errors at the most accurate timing.

Regarding the issue of delaying 5 frames, the camera collects the observations with fps=20. It means that our PrObe only causes a delay less than 0.25 seconds, which should be acceptable in practice. From our demo video, our PrObe can detect the error before the mug falls on the table. We believe the AED problem and proposed PrObe suit many real-world situations.


**A practical scenario of our AED problem:**

An intelligent robot (containing policy and AED method) is trained in the factory with daily life tasks. Then, it is sold to the customer's house to perform those tasks. The customer will provide few demonstrations for the robot but may not stay around the robot for the whole time. The robot itself should be able to recognize if it performs the task correctly or not. Also, it should terminate the policy and wait for expert's help or apply the recovery policy (in our appendix) when it detects a behavior mismatch. Note that in this case, the expert is the customer, who can provide demonstrations but may not be able to label corrected actions for the robot.

Reviewer comment (Presentation):
As discussed in related work, adaptable error detection (AED) is more challenging than few-shot anomaly detection (FSAD), and previous error detection methods are infeasible for AED. After reading the introduction, I understand that the main goal of AED is to monitor FSI policies and report their behavior errors. PrObe is proposed to address the challenges of AED. However, sections 4 and 5 are relatively not well connected to each other. If Figure 2 describes the whole framework of this paper, which part does PrObe contribute to? In AED training or AED inference? What is the relationship between equations (1) (2) and (3)?

Author response:
**Which parts of the framework in Figure 2 do our PrObe contribute to? (W1, Q2)**

Thanks for the question. 

Our PrObe is an AED method (i.e., adaptable error detector, denoted as $\phi$ in Figure 2), which involves the phases of AED training and AED inference.

The framework of our AED problem contains three phases:
- **Policy training phase:** The few-shot imitation policy $\pi$  first trains on the base environments. 
- **AED training phase:** The AED methods also train on the base environments. In this phase, some methods, e.g., our PrObe, might leverage the knowledge from the trained policy.
- **AED inference phase:** The trained policy $\pi$ performs the FSI tasks in novel environments. AED methods aim to monitor the policy and report timely once the policy behavior error occurs, i.e., the policy rollout is inconsistent with demonstrations provided by the expert in the novel environments. 

Thus, all AED methods involve the last two phases of Figure 2; however, our PrObe can tackle the challenging AED problem with superior performance. We will make this more clear in the revised version. 

Besides, please note that the FSI tasks are semantically similar in the base and novel environments (e.g., both are pick&place tasks). However, the interactable objects and backgrounds are disjoint between them. 

**A practical scenario of our AED problem:**

An intelligent robot (containing policy and AED method) is trained in the factory with daily life tasks. Then, it is sold to the customer's house to perform those tasks. The customer will provide few demonstrations for the robot but may not stay around the robot for the whole time. The robot itself should be able to recognize if it performs the task correctly or not. Also, it should terminate the policy and wait for expert's help or apply the recovery policy (in our appendix) when it detects a behavior mismatch.

---

**What's the relationship between Eq. 1, 2 and 3? (W1, Q2)**

Thanks for the question. 

Eq. 1 and 2 are the formal representations of how the AED method predicts the error status and how they will be evaluated.

- Eq. 1: The AED method $\phi$ takes rollout history (observations) $h$ and few demonstrations $\mathcal{D}^{n}$ as inputs and then predicts the error status $\hat{y}$.
- Eq. 2: The AED method is evaluated by its prediction $\hat{y}$ and ground-truth $y$ across all novel environments.

Eq. 3 is a new objective we proposed to train the PrObe, which is a temporal-aware variant of triplet loss. The complete objectives of PrObe can be found in Appendix Section B.

Reviewer comment (Writing):
It is very hard to read the paper. The prose could use significant improvements. There are also too many abbreviations in the paper making it hard to follow the content. What does "No impulse signals in behavior errors" mean? The problem and its challenges itself are unclear.

Author response:
**The problem and its challenges itself are unclear. What does "No impulse signals in behavior errors" mean? (W1)**

We propose a new problem named adaptable error detection (AED), which aims to detect the behavior inconsistency between the few-shot imitation policy rollouts and demonstrations.

The proposed AED contains three challenges, which makes it more challenging than an existing video few-shot anomaly detection problem. The second challenge, “no impulse signals in behavior errors,” means no significant difference between the consecutive frames when the error happens. We also provide two cases with detailed visual explanations in Figure 1. 

It is worth mentioning that all other reviewers agree **our proposed AED problem is crucial** for robot learning when considering the safety perspective.

---

**Presentation Issues (W1, W5)**

We are perplexed about these issues. 

For W1, there are only three new abbreviations in our paper, i.e., AED for adaptable error detection, PrObe for Pattern Observer, and vFSAD for video few-shot anomaly detection.

All remaining abbreviations are well-known or the method names from existing literature. For example, OCC for one-class classification, OOD for out-of-distribution, or DC for demonstration-conditioned. Also, we usually write the full name of the abbreviation at the beginning of each section if it does not frequently appear.

For W5, we do not find any inappropriateness when emphasizing the importance of our work since we propose a new problem in this paper. Most importantly, the two cases the reviewer mentioned are not even our proposed problem. Instead, they are the two problems related to us. We state the fact that they are relatively new and have drawn the researcher's attention recently.

At last, we want to highlight that we got >= 3 presentation scores from all other reviewers. We would appreciate it if the reviewer could point out which parts of the paper are unclear so that we can further improve it.

Reviewer comment (Evaluation):
The paper makes many claims without supporting evidence, e.g. "We emphasize that addressing AED is as critical as enhancing FSI policies’ performance".

Author response:
**The papers make many claims without supporting evidence, e.g. "We emphasize that addressing AED is as critical as enhancing FSI policies’ performance". (W2)**

It is the motivation of our work, which we spent **more than 13 lines** (first two paragraphs of Introduction) to describe the situation that the few-shot imitation policies may provoke damage when applying them to real-world scenarios. Thus, we should also consider promoting their ability for safety executions rather than the performance of their own tasks (e.g., manipulation tasks). 

We believe this motivation is clearly presented, which all other reviewers recognize.

We would appreciate it if the reviewer could point out other unsupported claims or motivations we made in the paper. We would love to explain or improve it.

Reviewer comment (Novelty):
As far as I understand, the term "vFSAD" refers to OOD detection in time-series data which includes a large quantity of work that has not been compared with or even mentioned. Example: Kaur et al. "CODiT: Conformal Out-of-Distribution Detection in Time-Series Data."

Author response:
**Works related to OOD detection in time-series data should be compared. (W3)**

Thank you for pointing out this question.

The fundamental assumption of our work is that we are monitoring the few-shot imitation (FSI) policies, and they usually have the following settings:

- **Training and testing environments are disjoint**. (base env. vs. novel env.)
- **No exploration before inference** in novel environments.
- **The expert only provides demonstrations** in the novel environment but may not be around the robot when it executes.

Thus, the traditional methods on OOD detection in time-series data are unavailable. In our case, the error detector works in an unseen environment. And we only have few demonstrations in the environment which makes it hard to compute those p-values in the reference [1] or other information in the frequency space. 

Moreover, the OOD samples defined in our work is the inconsistency between the policy rollout and demonstrations, which have two time-series. The OOD samples can be only detected by comparing these two video streams rather than simply computed on any one of them.

The vFSAD problem is the most related problem we can find. But our problem raises more challenges than the vFSAD problem, as we stated in the Introduction.

Please let us know if there is any other paper the reviewer thinks is related to us or fits our setting and should be compared.

Thank you again for suggesting the reference.

Reviewer comment (Experiments):
The stated aim of the experiments is confusing -- do you aim to check if detection is useful or if detection works?

Author response:
**The stated aim of the experiments is confusing -- do you aim to check if detection is useful or if detection works? (W6)**

In this work, we formulate **a new problem**. We also propose **a tailored method**, PrObe, to tackle the problem, which leverages the auxiliary knowledge of policy features. 

Therefore, our experiments aim to verify whether learning in the feature space of policy benefits to tackle the problem, i.e., **our motivation and strategy are useful and supported**. Thus, we examine tremendous evaluations across different tasks and policies. We also provide detailed ablation studies and visualization results.

Meanwhile, we would like to express **the practicality of the proposed problem**, and thus, we conduct the error correction experiment in Appendix, which incorporates our problem with the error correction strategies.

We believe that *the aims of our experiment are clear, and the evaluations of our experiment are comprehensive*.