Paper: Beyond the Benchmark: Detecting Diverse Anomalies in Videos (ICLR-2024)

Reviewer comment (Novelty):
The proposed method is incremental w.r.t. AI-VAD.

Author response:
Our contribution on top of AI-VAD:

Our paper's core contribution lies in addressing the underexplored realm of diverse multi-frame anomalies, a facet largely unaddressed in prior research. While drawing inspiration from AI-VAD, our work crucially emphasizes the comparison between prevalent benchmark datasets, primarily featuring simple anomalies, and our novel datasets, introducing intricate, action-based anomalies. This contrast aims to encourage a broader perspective on anomalies, reflecting real-world scenarios. The significance of this comparison is evident in Table 3, which demonstrates a major improvement, of 19.9% and 9.7%, when using our method on the proposed datasets, while exposing limitations in the existing approaches.

Reviewer comment (Evaluation):
The comparison in Table 1 does not reflect the difficulty / diversity advantages suggested by the authors. I do not see the benefits of the proposed datasets w.r.t. recent benchmarks such as UCF-Crime, XD-Violence or UBnormal.

Author response:
UCF-Crime, XD-Violence or UBnormal:

Though indeed used for VAD, UCF-Crime, XD-Violence and UBnormal are built for supervised VAD learning which is less compatible with the popular use case for Ped2, Avenue and ShanghaiTech Campus which is unsupervised VAD.

While UCF-Crime, XD-Violence, and UBnormal datasets have been employed in VAD contexts, they are primarily tailored for supervised VAD learning. This framework contrasts with the prevalent use case for Ped2, Avenue, and ShanghaiTech Campus datasets, which predominantly function in unsupervised VAD scenarios.

Considering the aforementioned time-complexity issue, adapting UCF-Crime and XD-Violence datasets for unsupervised VAD poses a challenge, due to the severe increase in their size compared to current benchmark datasets. UCF-Crime has a total of 128 hours of data and XD-Violence has around 217 hours, both orders of magnitude above ShanghaiTech which has 3.5 hours of data, Avenue with around 20 minutes and Ped2 with less than 5 minutes. Our smaller-sized HMDB-AD (around one hour of data) and HMDB-Violence (less than 2.5 hours) datasets fulfill a unique role in advancing VAD research, specifically addressing complex anomalies and keeping realistic processing run times.

Following your interest on this topic we also added a short explanation to section 2.1 for future readers.

Reviewer comment (Evaluation):
There is no time evaluation reported for the presented method. Anomaly detection methods are expected to run in real-time, but it is not clear if MFAD can do this. To me, the method is a bit heavy.

Author response:
Time complexity & Real-Time constraints:

Our method's multi-stage process and diverse feature extraction are aligned with the state-of-the-art Video Anomaly Detection (VAD) methods [1]. The reliance on established models ensures that our method is comparable in run time to various prior approaches [2, 3].

We acknowledge that real-time application remains a challenge in current VAD research. As part of our future endeavors, we're exploring strategies to enhance computational efficiency. Our approach's modular structure enables parallel computation of individual features, offering a pathway to expedite processing. Furthermore, it provides our system an option for easily integrating future improved components that can speed up our method even more.

Reviewer comment (Writing):
There are some language corrections to be made, e.g.:  
- "It’s crucial to" => "It is crucial to" (language abbreviations should be avoided in formal language).

Author response:
Language corrections:

We revised our paper with the language corrections you rightfully pointed out.

Reviewer comment (Experiments):
The paper lacks a more detailed description of the datasets HMDB-AD and HMDB-Violence. It would be beneficial to provide more information on the distribution of normal and abnormal activities, and any specific challenges or characteristics of the datasets.

Author response:
Dataset Description:

In Section 3 of our paper, we provide a comprehensive overview of our datasets—HMDB-AD and HMDB-Violence. These datasets are derived from the HMDB51 action recognition dataset and consist of short videos carefully trimmed around specific actions. HMDB-AD encompasses two normal behaviors, walking and running, widely used as fundamental normal human activities in VAD studies. The abnormal classes in HMDB-AD include climbing and performing a cartwheel, distinct activities characterized by varied orientation and movement direction. On the other hand, HMDB-Violence comprises nine normal non-violent activities and seven abnormal violent actions. In both datasets, we also include some randomly picked normal videos and add them to the test set.

Appendix A showcases examples from both datasets to offer a clearer understanding.

Reviewer comment (Presentation):
The paper lacks a more detailed and visual explanation of the proposed method.

Author response:
Methodology:

Our methodology draws upon the principles established in AI-VAD. In their research, AI-VAD showcased the additional benefits of incorporating pose estimation, object velocity, and deep image encoding. To clarify our feature extraction pipeline, we offer a visualization in Figure 1 and an explanation in Section 4.1.


Feature Synthesis:

Following feature extraction, we compute anomaly scores for each feature independently and aggregate the results. This approach allows us to address the variations in dimensionality and scale across the four distinct features. We take note of this process in Section 4.2, supported by a visualization in Figure 2.

Reviewer comment (Evaluation):
The sub-optimal performance of the method proposed on the STC and Avenue datasets and the reasons for this should be analysed.

Author response:
Performance on ShanghaiTech and Avenue Datasets:

MFAD sub-optimal performance on the ShanghaiTech and Avenue datasets is in line with our shift in approach to anomaly detection. These datasets primarily showcase a narrow definition of anomalies, whereas our methodology aims for a broader spectrum that also includes complex, multi-frame anomalies, as demonstrated in our proposed datasets. Consequently, MFAD faces similar challenges to previous methods when evaluated against these specific datasets and achieves comparable results. Particularly, the object-oriented aspect of MFAD struggles when confronted with scenarios involving closely clustered pedestrians. In contrast, as illustrated in Figure 3, our method outperforms previous methodologies when handling the limited subset of complex anomaly videos in the benchmark datasets.

Following your note we addressed this shortly in Section 5.3.

Reviewer comment (Experiments):
Utilizing specific video foundation models may reduce adaptability to different datasets or domains.

Author response:
Foundation models are designed to be versatile and applicable across diverse domains [4]. Specifically, VideoMAEv2  [5] demonstrated their adaptability across various domains (Video Action Classification, Spatial Action Detection and Temporal Action Detection), and tested on multiple datasets.

Reviewer comment (Experiments):
Computationally intensive and a requirement for separate training/testing data make real-time application challenging.

Author response:
We acknowledge that real-time remains a challenge in current VAD research. As part of our future endeavors, we're exploring strategies to enhance computational efficiency. Our approach's modular structure enables parallel computation of individual features, offering a pathway to expedite processing. Furthermore, it provides our system an option for easily integrating future improved components that can speed up our method even more.

Reviewer comment (Evaluation):
How does MFAD handle the challenge of real-time video anomaly detection given its computational complexity? As I can see the author has provided the note for reproducibility, some insights would be helpful to understand the scope.

Author response:
We acknowledge that real-time remains a challenge in current VAD research. As part of our future endeavors, we're exploring strategies to enhance computational efficiency. Our approach's modular structure enables parallel computation of individual features, offering a pathway to expedite processing. Furthermore, it provides our system an option for easily integrating future improved components that can speed up our method even more.