Paper: Bayesian Knowledge Distillation for Online Action Detection (ICLR-2024)

Reviewer comment (Novelty):
This paper seems to simply combine existing knowledge distillation and uncertainty techniques. Knowledge distillation is already proposed in PKD (Zhao et al. (2020)), and uncertainty technique have been used in Uncertainty-OAD (Guo et al. (2022)) for online action detection.

Author response:
Th knowledge distillation in PKD (Zhao et al. (2020)) aims at distilling the knowledge of offline model to an online model. Our knowledge distillation aims at distilling the distribution and mutual information of an online teacher model to an online student model and improving the efficiency. The overall mechanism is different. In addition, our student model can quantify the predictive uncertainty in a single forward pass, which is practical for real applications.

Reviewer comment (Experiments):
More details about the teacher model should be included. The results of the teacher model are also missing.

Author response:
The teacher model is trained in a deterministic way. The Laplace approximation is performed as a post-processing. Code will be made publicly available. For the teacher model, we include the performance below:

| Dataset | ActivityNet (mAP %) | Kinetics (mcAP%) |
|-----------|------|--------|
|THUMOS'14|71.6|73.9|
|TVSeries|88.9|90.6|

The teacher model has bettern performance than the student model and SOTAs. But it is more computationally expensive, which does not satisfies the objective of our overall goal

Reviewer comment (Presentation):
More visual analysis should be included instead of all numerical analysis.

Author response:
We will update the paper with visual analysis. Demos will also be available on our project page

Reviewer comment (Novelty):
Lack of novelty. Although it may be the first work which applies the distillation architecture on online action detection, there is little innovation on components design and theoretical analysis. Teacher-student architecture, Bayesian neural network, and attention network are all very common tools.

Author response:
The novelties of this paper mainly lie in: (1) we are the first work leveraging knowledge distillation to improve the inference efficiency of online action detection, which is a big concern for real applications. (2) The student model are trained to identify important features automatically based on the mutual information. (3) We build an evidential student network, which can quantify the predictive uncertainty by a single forward pass.

Reviewer comment (Theory):
Misleading/Inappropriate use of evidential deep learning (EDL). EDL is based on the Subjective Logic theory, which is implemented by its unique optimization objective and is accompanied by its own uncertainty calculation method. In the work, the authors construct a simple Dirichlet distribution and then claim they adopt EDL, which is not true.

Author response:
The concept of evidential deep learning is first introduced by [1] to quantify uncertainty for classification problems. By applying a Dirichlet distribution to class probabilities, they treated predictions of a neural network as subjective opinions and learned the function to collect the evidence leading to these opinions from data. The uncertainty estimation is guided by the theory of evidence. Subsequent research has aimed to enhance uncertainty quantification in the framework of [1], particularly by including out-of-distribution data or density models. These developments are comprehensively reviewed in the survey paper [2], which also inspired our use of the term “evidential deep learning”. For a more detailed discussion on this topic, please refer to [2]. We will further clarify evidential deep learning for quantifying classification uncertainty.  \
[1] Sensoy, Murat, Lance Kaplan, and Melih Kandemir. "Evidential deep learning to quantify classification uncertainty." Advances in neural information processing systems 31 (2018).\
[2] Ulmer, Dennis, Christian Hardmeier, and Jes Frellsen. "Prior and posterior networks: A survey on evidential deep learning methods for uncertainty estimation." Transactions on Machine Learning Research (2023).  

Question of using CE loss:  
Our work focuses on evidential deep learning that is designed specifically for classification uncertainty, as detailed in Section 3 of [2]. We use the Dirichlet distribution to efficiently capture uncertainty, allowing it to be determined in a single forward pass of the neural network. This is because Dirichlet distribution is conjugate to the target categorical distribution of y. Since traditional EDL's uncertainty quantification might not effectively distinguish between aleatoric and epistemic uncertainties, we address this through knowledge distillation to minimize the distance between the Dirichlet distribution and the corresponding distribution of the teacher model. By distilling both epistemic and aleatoric information from the Bayesian teacher model, we can employ Eq. 10 as outlined in Section 3.3 of [2] to separate these two types of uncertainty.

Reviewer comment (Experiments):
Unfair (or at least incomplete) comparison of computation efficiency and model complexity. Authors claim that the proposed method BKD achieves competitive performance with less model complexity and computational cost, and provide comparison results on Table 2. However, the other methods did not adopt a teacher-student distillation manner as BKD, and the Bayesian neural network which is used as the teacher model by BKD is quite computationally heavy. It is a very natural result for BKD to achieve fast inference at the cost of much larger computation in the training phase via distillation, and the comparison of training speed is not provided.

Author response:
We agree the training of Bayesian neural network is expensive. Our proposed BKD performs Laplace approximation (LA) after training one deterministic teacher model. So the main computation gain lies in the LA process. Following the comparison in E2E-LOAD, we made a comparison of training cost with other methods below, the mAP is reported on THUMOS’14 with Kinetics-pretrained features :  
| Method | mAP (%) | GPU Mem | Time (min/epoch) | # of parameters |  
|----------|------|-------------------------------|--------|-----|  
| LSTR | 59.2 | 8x31.4 |1.5 | 105.9|  
| E2E-LOAD | 72.4 | 8x16.9 |9.6 | 53.5|  
| BKD (ours) | 71.3 | 2x24 |13.2 | 18.7|  

In addition, the inference speed is the first priority of online action detection so the inference FPS is the most important factor.  
For the unfair comparison concern, the knowledge distillation is one of our contributions. If other methods also leverage knowledge distillation, the performance will drop as the student model usually performs worth than the teacher model.

Reviewer comment (Theory):
The motivation behind incorporating uncertainty prediction into the design is unclear. While the reviewer acknowledges the need to consider uncertainty due to the inherent unpredictability of the future, the authors have not highlighted the limitations of not modeling uncertainty. For example, a deterministic prediction of the current action may not be necessary, and predictions of potential opportunities for different actions may suffice.

Author response:
Weakness 1b: The student model learns from the teacher model. We aim to transfer as much as the knowledge to the student model, i.e. we aim to distill the distribution of the teacher model to student model. In addition, online action detection is used for many safety-critical appplications such as autonomous drving. Therefore, the uncertainty quantification is important for decision making which another motivation of modeling the uncertainty.

Reviewer comment (Experiments):
The second claim is that the proposed framework can perform feature selection using mutual information and output Bayesian predictive uncertainties. The reviewer was expecting experimental evidence to support this claim. However, the reviewer did not find sufficient evidence regarding the effectiveness of the architectural design. To substantiate this claim, the reviewer requests experiments that can validate its effectiveness.

Author response:
To validate that the proposed framework can leverage mutual information to select important features, we re-implemented the framework without the attention module. The other parts of the framework remain the same. The experiment results are shown as follows:

| Method | THUMOS'14-ANet | THUMOS'14-Kinetics | TVSeries-ANet | TVSeries-Kinetics |
|----------|------|-------------------------------|--------|-----|
| BKD (no attention) | 68.2 | 70.1 | 88.0 | 88.9 |
| BKD | 69.6 | 71.3 | 88.4 | 89.9|

The results demonstrate the effectiveness of our proposed attention modeling with mutual information.  

For the effectiveness of predictive uncertainty quantification, the results in Table 7 of the paper show that the uncertainties can be used for detecting abnormal actions accurately.

Reviewer comment (Experiments):
As mentioned in point 1, the authors primarily focus on demonstrating that the proposed framework performs on par with existing methods on benchmarks for online action detection. However, the reviewer believes that there is a lack of insights regarding the ablation of teacher-student architectures. To establish the value of the proposed evidential deep learning approach, the reviewer requests experiments that can validate this claim.

Author response:
For the insights of teacher-student architectures. We implemented different knowledge distillation methods with a same teacher model for comparison. We did grid searches to select the optimal hyper-parameters for fair comparisons. The experiment results are shown as follows:
| Method | THUMOS'14-ANet | THUMOS'14-Kinetics | TVSeries-ANet | TVSeries-Kinetics |
|----------|------|-------------------------------|--------|-----|
| PKD [1] | - | 64.5 | - | 86.4 |
| Feature-based [2] | 65.2 | 68.1 | 87.0 | 88.4 |
| Output-based [3] | 66.1 | 68.3 | 87.4 | 88.5 |
| BKD | 69.6 | 71.3 | 88.4 | 89.9|

The results shown our BKD obtains better performance than other knowledge distillation methods, which demonstrates the effectiveness of Bayesian distillation.  

[1] Zhao, Peisen, et al. "Privileged knowledge distillation for online action detection." arXiv preprint arXiv:2011.09158 (2020).  
[2] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).  
[3] Romero, Adriana, et al. "Fitnets: Hints for thin deep nets." arXiv preprint arXiv:1412.6550 (2014).