Paper: Put on your detective hat: What’s wrong in this video? (ICLR-2024)

Reviewer comment (Novelty):
The additional benefit of this new dataset is unclear. Looking at Table 1, the added benefit compared to existing work "Assembly 101" seems to be limited to adding "depth" channel. However, it is unclear whether the "depth" channel is really useful in the context of error recognition. The authors need to provide more evidence on why this new dataset is significantly more useful than "Assembly 101".

Author response:
> The additional benefit of this new dataset is unclear.

Firstly, Error recognition as a task is quite challenging, and we conjecture that it requires the development of techniques that understand the context, meaning, and cause of various errors. 

Our dataset is different from Assembly 101 in the following ways:

- **Domain:** We stress the fact that Assembly 101 explores the domain of Assembly and Disassembly tasks, and we focus on Cooking activities. 
   - In assembly, the majority of the errors performed can be corrected, but cooking as a domain includes activities which do not have corrective actions. 
   - Unlike assembly, where shapes and colours of the objects remain fixed during the whole procedure. Cooking involves continuous changes in the shape and colour of ingredients.
- **Environment:** Assembly 101 is a dataset that is collected in a lab environment, and we collect our dataset in real kitchen environments.
- **Modalities:** As we primarily employ HoloLens2 in capturing the data, we leverage the wealth of information it enables to capture. For example, our synchronized dataset enables the complete reconstruction of a 3D map of the environment while the activity is being performed.

In conclusion, we emphasize that the category of errors we explore is different from the category of errors explored by activities in the assembly domain.

Reviewer comment (Evaluation):
The AUC score between Table 2 (Error Recognition) & Table 3 (Early Error Prediction) looks surprisingly similar. Shouldn't the task of "Early Error Prediction" be much harder than the typical "Error Recognition" task?

Author response:
> The AUC score between Table 2 (Error Recognition)

Notice that although the AUC scores are roughly the same, the F1 scores are always lower because recall is low.

Reviewer comment (Presentation):
In Figure 4, Table 2-3, can you highlight the best performing model?

Author response:
> In Figure 4, Table 2-3, can you highlight the best-performing model?

Sure, we have updated the manuscript following your suggestion

Reviewer comment (Presentation):
It's better to present the distribution of different error categories in the main paper, e.g., Figure 16-17?

Author response:
> It's better to present the distribution of different error categories in the main paper, e.g., Figure 16-17.

Sure, we have updated the manuscript following your suggestion

Reviewer comment (Experiments):
Compared to Assembly 101 (error detection), the paper seems like an inferior / less complicated dataset. Claims like higher ratio of error to normal videos needs to be validated.

Author response:
Firstly, Error recognition as a task is quite challenging, and we conjecture that it requires the development of techniques that understand the context, meaning, and cause of various errors. Secondly, we respectfully deny the statement that this paper is an inferior dataset compared to Assembly 101. We justify the statement with the following reasons

- **Domain:** We stress the fact that Assembly 101 explores the domain of Assembly and Disassembly tasks, and we focus on Cooking activities.
  - In assembly, the majority of the errors performed can be corrected, but cooking as a domain includes activities which do not have corrective actions. 
  - Unlike assembly, where shapes and colours of the objects remain fixed during the whole procedure. Cooking involves continuous changes in the shape and colour of ingredients.
- **Environment:** Assembly 101 is a dataset which is collected in a lab environment, and we collect our dataset in real-kitchen environments.
- **Modalities:** As we primarily employ HoloLens2 in capturing the data, we leverage the wealth of information it enables to capture. For example, our synchronized dataset enables the complete reconstruction of a 3D map of the environment while the activity is being performed. 

In conclusion, we emphasize that the category of errors we explore is different from the category of errors explored by activities in the assembly domain.

Reviewer comment (Experiments):
Compared to datasets, the dataset prides itself on adding different modalities especially depth channel (RGB-D). The paper fails to validate the necessities of such modality. One crucial difference between assembly dataset is use of depth values. What role does it play in training baseline models? Does it boost the model’s performance if these weren’t present. In current deep learning area, depth channels should reasonably be producible via the help of existing models.

Author response:
We believe that the task of error recognition is a challenging task and requires complete awareness of the 3D environment. So, we employed Hololens2 to capture the data. 

You are right that the current monocular depth estimation techniques are indeed quite powerful and can equal the capacity of a low-quality depth sensor. But, we leverage the wealth of information provided by Hololens2, thus enabling the creation of a less noisy 3D map of the environment.  

In the current version of the manuscript, we do not have results on incorporating depth data into the proposed tasks. We shall add the results incorporating data captured from all the modalities in future versions of this manuscript.

Reviewer comment (Evaluation):
I’m not convinced that the binary classification is a justifiable baseline metric. While I agree that the TAL task is really important here and a good problem to solve, I’m not sure how coarse grained binary classification can assess models’ understanding of fine-grained error like technique error.

Author response:
We do agree that binary classification is a preliminary baseline and conjecture that any technique developed for the task of error recognition should interpret the action, understand the context and infer the cause. 

As this manuscript introduces the dataset, we followed the convention of providing a supervised baseline using transfer learning techniques for the specific tasks.

Reviewer comment (Experiments):
Timing Error (Duration of an activity) and Temperature based error — do these really need ML based solutions? In sensitive tasks, simple sensor readings can indicate error. I’m not sure testing computer vision models on such tasks is justifiable. These require more heuristics-based methods, working with if-else statements.

Author response:
It is an interesting question. In our experiments (see Table 5) on the proposed tasks and categorization of common errors that could occur while cooking, we believe that timing, temperature and measurement are the most challenging errors to detect.  

Whenever there's a conditional transition based on a change of state involving a heating component, sensor-based measurements fall short. For example, \textit{Saute the tofu on a pan for 3 minutes or until it turns brown}. Here, having a perception component that is capable of judging the state change to brown is necessary.

Reviewer comment (Writing):
Procedure Learning is very vaguely defined, mostly left unexplained and seems like an afterthought. I recommend authors devote passage to methods “M1 (Dwibedi et al., 2019)” and “M2 (Bansal, Siddhant et al., 2022)”. In Table 5, value of lambda? is not mentioned.

Author response:
We implemented your suggestion and added an extra paragraph explaining this section. Please look at the updated section on Procedure Learning.

Reviewer comment (Evaluation):
The authors are dealing with a degree of subjectivity in terms of severity of errors. It would have been greatly beneficial if the errors could be finely measured. For example, if the person uses a tablespoon instead of teaspoon, is it still an error? Some errors are more grave than others — is there a weighted score? Is there a way to measure level of deviation for each type of error or time stamp of occurrence of error? Is one recipe more difficult than another recipe?

Author response:
In the current version of the manuscript, we flag any deviation from the procedure as an error. As stated in the above question, error recognition completely depends on the context; your example of using a tablespoon instead of a teaspoon may not be a significant error in the procedure when the step is to **stir**. However, the step involves **adding an ingredient like salt**, which completely changes the taste of the recipe, and this is indeed very significant. 

We do understand that not all errors have the same impact on the procedure. It is an interesting idea to automatically infer the impact of an error on the current activity.

Reviewer comment (Evaluation):
Please validate claims like higher ratio of error to normal videos.

Author response:
Firstly, Error recognition as a task is quite challenging, and we conjecture that it requires the development of techniques that understand the context, meaning, and cause of various errors. Secondly, we respectfully deny the statement that this paper is an inferior dataset compared to Assembly 101. We justify the statement with the following reasons

- **Domain:** We stress the fact that Assembly 101 explores the domain of Assembly and Disassembly tasks, and we focus on Cooking activities.
  - In assembly, the majority of the errors performed can be corrected, but cooking as a domain includes activities which do not have corrective actions. 
  - Unlike assembly, where shapes and colours of the objects remain fixed during the whole procedure. Cooking involves continuous changes in the shape and colour of ingredients.
- **Environment:** Assembly 101 is a dataset which is collected in a lab environment, and we collect our dataset in real-kitchen environments.
- **Modalities:** As we primarily employ HoloLens2 in capturing the data, we leverage the wealth of information it enables to capture. For example, our synchronized dataset enables the complete reconstruction of a 3D map of the environment while the activity is being performed. 

In conclusion, we emphasize that the category of errors we explore is different from the category of errors explored by activities in the assembly domain.

Reviewer comment (Experiments):
Please provide the utility of depth channels in error detection tasks or provide baseline performances.

Author response:
We believe that the task of error recognition is a challenging task and requires complete awareness of the 3D environment. So, we employed Hololens2 to capture the data. 

You are right that the current monocular depth estimation techniques are indeed quite powerful and can equal the capacity of a low-quality depth sensor. But, we leverage the wealth of information provided by Hololens2, thus enabling the creation of a less noisy 3D map of the environment.  

In the current version of the manuscript, we do not have results on incorporating depth data into the proposed tasks. We shall add the results incorporating data captured from all the modalities in future versions of this manuscript.

Reviewer comment (Presentation):
I feel like binary classification section can be minimized, and instead procedural activity section needs to be emphasized and described in much more detail.

Author response:
We do agree that binary classification is a preliminary baseline and conjecture that any technique developed for the task of error recognition should interpret the action, understand the context and infer the cause. 

As this manuscript introduces the dataset, we followed the convention of providing a supervised baseline using transfer learning techniques for the specific tasks.

Reviewer comment (Evaluation):
Some form of comparisons of errors needs to be there, or a severity level of errors.

Author response:
In the current version of the manuscript, we flag any deviation from the procedure as an error. As stated in the above question, error recognition completely depends on the context; your example of using a tablespoon instead of a teaspoon may not be a significant error in the procedure when the step is to **stir**. However, the step involves **adding an ingredient like salt**, which completely changes the taste of the recipe, and this is indeed very significant. 

We do understand that not all errors have the same impact on the procedure. It is an interesting idea to automatically infer the impact of an error on the current activity.

Reviewer comment (Experiments):
In terms of statistics, still not convinced if the proposed dataset is better than Assembly 101. None of the baseline models have been shown to have any adverse impact of these proposed added difficulties: “Shape and color changes ingredients”, “real-kitchen environments”. Higher ratio of error to normal videos (not validated). Reviewer I weakness-1 pointed out the same issue.

Author response:
Firstly, Error recognition as a task is quite challenging, and we conjecture that it requires the development of techniques that understand the context, meaning, and cause of various errors. Secondly, we respectfully deny the statement that this paper is an inferior dataset compared to Assembly 101. We justify the statement with the following reasons

- **Domain:** We stress the fact that Assembly 101 explores the domain of Assembly and Disassembly tasks, and we focus on Cooking activities.
  - In assembly, the majority of the errors performed can be corrected, but cooking as a domain includes activities which do not have corrective actions. 
  - Unlike assembly, where shapes and colours of the objects remain fixed during the whole procedure. Cooking involves continuous changes in the shape and colour of ingredients.
- **Environment:** Assembly 101 is a dataset which is collected in a lab environment, and we collect our dataset in real-kitchen environments.
- **Modalities:** As we primarily employ HoloLens2 in capturing the data, we leverage the wealth of information it enables to capture. For example, our synchronized dataset enables the complete reconstruction of a 3D map of the environment while the activity is being performed. 

In conclusion, we emphasize that the category of errors we explore is different from the category of errors explored by activities in the assembly domain.

Reviewer comment (Evaluation):
Binary classification as an error detection baseline metric is not justified. (Similar issue pointed out by Reviewer I weakness 2). I fully agree that the baselines do not provide many insights.

Author response:
We do agree that binary classification is a preliminary baseline and conjecture that any technique developed for the task of error recognition should interpret the action, understand the context and infer the cause. 

As this manuscript introduces the dataset, we followed the convention of providing a supervised baseline using transfer learning techniques for the specific tasks.

Reviewer comment (Evaluation):
I’m not entirely convinced metrics-based (Time / Temperature) errors are the right metrics to measure error detection capabilities via Computer Vision or Machine Learning algorithms. (Similar issue pointed out by Reviewer I weakness 2)

Author response:
It is an interesting question. In our experiments (see Table 5) on the proposed tasks and categorization of common errors that could occur while cooking, we believe that timing, temperature and measurement are the most challenging errors to detect.  

Whenever there's a conditional transition based on a change of state involving a heating component, sensor-based measurements fall short. For example, \textit{Saute the tofu on a pan for 3 minutes or until it turns brown}. Here, having a perception component that is capable of judging the state change to brown is necessary.

Reviewer comment (Novelty):
The proposed dataset has nothing standing out from previous ones. By looking at Table 1, the proposed dataset are bucked in small or medium compared with the existing ones in terms of number of videos or number of hours, and even number of tasks. The only difference may be that the proposed dataset provides more modalities. However, experiments shown in the paper only used RGB so far. Another claim made by the paper is that it has more error instances (compared by error:normal ratio) than Assembly101, which is true. However, the ways of capturing error videos have some problems: in three scenario of capturing error videos (sec 3.1.2), the first twos were scripted, the last one is instructed. Although it helps providing more error videos, however, those error videos will be intentional (the mistakes won't look realistic). In practice, the unintentional mistakes or errors are more relevant and often happened in reality.

Author response:
Firstly, Error recognition as a task is quite challenging, and we conjecture that it requires the development of techniques that understand the context, meaning, and cause of various errors. Secondly, we respectfully question the statement the proposed dataset has nothing standing out from previous ones. We justify the statement with the following reasons

- **Domain:** We stress the fact that Assembly 101 explores the domain of Assembly and Disassembly tasks, and we focus on Cooking activities. 
  - In assembly, the majority of the errors performed can be corrected, but cooking as a domain includes activities which do not have corrective actions. 
  - Unlike assembly, where shapes and colours of the objects remain fixed during the whole procedure. Cooking involves transformations in the shape or colour of ingredients.
- **Environment:** Assembly 101 is a dataset that is collected in a lab environment, and we collect our dataset in real kitchen environments.
- **Modalities:**  As we primarily employ HoloLens2 in capturing the data, we leverage the wealth of information it enables to capture. For example, our synchronized dataset enables the complete reconstruction of a 3D map of the environment while the activity is being performed.

In conclusion, we emphasize that the category of errors we explore is different from the category of errors explored by activities in the assembly domain. 

---

> However, the ways of capturing error videos have some problems

We respectfully deny this statement.
- **Intentional videos based on scripts**
  - It has been a common practice to use scripted videos for activity understanding. Popular dataset [1] is constructed based on pre-prepared scripts.
  - Since the submission of this manuscript, two works have been published in a similar direction, and both explored assembly tasks [2-3] and both used pre-prepared scripts to capture data.
- **Unintentional videos**
  - Although participants followed scripts to perform activities. Due to the complex nature of cooking tasks and the participants' lack of experience in cooking, we do capture unintentional errors.
  - We have updated the following sections of the manuscript to explicitly state this information Table 1, Figure 3 and Section 3.1.2
- **Visual Appearance**
  - It is important to note that the visual outcome of the majority of the errors remains the same i,e independent of errors generated intentionally by following a script or unintentionally.

Reviewer comment (Experiments):
The baselines provide not much insights. For Error Recognition (Table 2 & 3), the observation is Omnivore is the best backbone / embedding for error / normal video classification. The early prediction problem is still formulated as classification with partly-observed data. The same finding is confirmed (Omnivore works best for this, this brings no surprise as both are technically the same problem, the later one is a bit harder). For Multi-Step Localization, the problem is formulated as supervised TAL and the same set of features are used and an ActionFormer head was used for localization. The same finding is that Omnivore works best. For procedure learning, two baselines (Dwibedi et al. 2019 and Siddhant et al. 2022) were used and provided similar performance. No real insights were observed in these experiments. Since this paper is about the new dataset which is claimed to focus on error recognition, however there are not much new insights about the significance of bringing more error videos to procedural video dataset: neither in the way data is captured nor in significant baselines or experiments to demonstrate why it matters.

Author response:
Error recognition and robust multi-step localization with errors are harder tasks which require the development of techniques that semantically understand the environment, perceptually identify the changes in the state of the activity and infer the cause. We do understand that the provided baselines are preliminary, but they provide a starting point for the development of task-specific methods. 

The purpose of this manuscript is to explore and categorize common errors that could occur in the cooking domain and follow the convention of evaluating the dataset using task-specific (error-recognition and multi-step localization) transfer learning techniques.

Reviewer comment (Writing):
Writing is not clear in some parts. In 4.2, it is not clear whether the supervised TAL is trained task-agnostically or task-specifically, meaning TAL is trained for all or per tasks (24 recipes).

Author response:
As stated in the paper, we extracted features from 4 pre-trained models, used three criteria to construct training, validation and test splits and trained 12 models. In all these scenarios, we train models task-agnostically.

Reviewer comment (Writing):
Section 4.2: what does "refer to 16" mean?

Author response:
It is a typo. Thanks for catching it. 
It should be \textit{refers to Figure 16}, where figure 16 is from the supplementary material.