Paper: Don't Judge by the Look: Towards Motion Coherent Video Representation (ICLR-2024)

Reviewer comment (Novelty):
The main weakness of the paper is the lack of solid motivation.

Author response:
**Weakness 1: Lack of motivation.**  

Thanks for the comment. To better clarify our motivation, we divide our motivation into two folds:  

(1) Why do we study the problem of Hue Jittering?  

As shown in Fig. 1, Hue Jittering is often neglected in current supervised video recognition methods because of its detrimental effect in object recognition. We rethink this problem in the context of video recognition and find the behavior of hue variance, which leads to appearance changes, is beneficial as static appearances are less important in videos. This justifies why MCA can lead to improvement on different datasets, architectures, and other data augmentation methods, as this problem generally exists in video recognition and is often neglected in current methods.  

(2) Why do we propose MCA instead of directly using Hue Jittering?  

The current implementation of Hue Jittering is highly time-consuming and its efficacy is limited by the distribution shift problem. To solve these two issues, we propose SwapMix to simulate the effect of hue variance while being significantly faster in practice, and then we present Variation Alignment to resolve the distribution shift and implicitly encourage the model to focus more on the motion information.

The first point explains why our method is very general and the second point justifies the effectiveness of MCA.

Reviewer comment (Novelty):
The paper presents an insufficient survey. It claims in section 2 that “there has been a lack of studies for data augmentation methods in video recognition.” However, there are many related works such as:  
* Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foreground, ICCV2023  
* Learning representational invariances for data-efficient action recognition, CVIU 2023  
* Frequency Selective Augmentation for Video Representation Learning, AAAI 2023  
* Enabling Detailed Action Recognition Evaluation Through Video Dataset Augmentation, NeurIPS 2022  
* Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognition, ECCV 2022  
* ObjectMix: Data Augmentation by Copy-Pasting Objects in Videos for Action Recognition, MMAsia 2022  
* Exploring Temporally Dynamic Data Augmentation for Video Recognition, arXiv 2022  
* VideoMix: Rethinking Data Augmentation for Video Classification, arXiv 2020

Author response:
**Weakness 2: Insufficient survey and comparisons.**  

We thank the Reviewer for pointing out these works and we will add discussion with all of them in the final version. As for empirical comparisons, we stress that it is hard to fairly compare our method with most of them for the following reasons:

<1> *'Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foreground'*: This paper mainly targets the problem of out-of-distribution (OOD) generalization and it results in inferior performance on independent and identically distributed (IID) test data (shown in their Tab. 2/3/4), which means it leads to performance drop in the supervised learning setting.  

<2> *'Learning representational invariances for data-efficient action recognition'*: This work proposes ActorCutMix to swap the video background and needs a human detection model to generate human/background mask for each frame which relies on extra knowledge and is very time-consuming for implementation. Moreover, this paper mainly focuses on the semi-supervised learning setting which is different from us.  

<3> *'Frequency Selective Augmentation for Video Representation Learning'*: To our best knowledge, there is no publicly available code for the proposed method and this work mainly focuses on the self-supervised learning setting. We have tried to re-implement their method based on the pseudo-code, but errors appeared as the output is complex numbers.

<4> *'Enabling Detailed Action Recognition Evaluation Through Video Dataset Augmentation'*: Instead of as a data augmentation method, the aim of this paper is to deliver an analysis toolkit for evaluation.

<5> *'Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognition'*: To our best knowledge, there is no publicly available code for the proposed method and it needs to train a selector using reinforcement learning.

<6> *'ObjectMix: Data Augmentation by Copy-Pasting Objects in Videos for Action Recognition'*: To our best knowledge, there is no publicly available code for this method. 

<7> *'Exploring Temporally Dynamic Data Augmentation for Video Recognition'*: To our best knowledge, there is no publicly available code for this method.  

<8> *'VideoMix: Rethinking Data Augmentation for Video Classification'*: We implement the VideoMix on Something-Something V1 dataset over TSM[1] to compare with MCA and test different choices of hyperparameter as they do.

| Method | $\alpha$ | Top-1 Acc. |
| :-----: | :-----: | :-----: |
| TSM | - | 45.63% |
| TSM+VideoMix | 0.4 | 45.62% |
| TSM+VideoMix | 2 | 45.25% |
| TSM+VideoMix | 8 | 44.48% |
| TSM+MCA | - | 47.57% |

One can observe that VideoMix exhibits worse performance compared to the baseline under different hyperparameters, while MCA clearly improves the generalization ability of the baseline method which proves the efficacy of our method. Thanks again for listing these works and we will add discussion in our final version.

Reviewer comment (Novelty):
Many prior works have tried to tackle background bias or scene bias (similar to “static appearance” mentioned in page 2), which exists in action recognition datasets. The paper should discuss how the proposed method can contribute to solving this issue.

Author response:
**Weakness 3: How MCA contributes to the background bias issue.**  

Thanks for the great question. Unlike prior works which divide each frame into foreground and background, we simply treat every image as a whole and change its appearance by SwapMix. That means we will also change the appearance of the foreground, which is usually human, as the variance in their looks does not affect the action conveyed in videos (shown in Fig. 1). By doing so, we can implicitly enforce the model to focus more on the motion information, rather than static appearances. This indicates that MCA is compatible with those works as our motivation is quite different from theirs. To validate it, we implement the prior work BE[2], which is robust to background bias, as a data augmentation operation and adapt it in supervised learning where we train it on Something-Something V1 dataset over TSM for comparisons:
| Method | Top-1 Acc. |
| :-----: | :-----: |
| TSM | 45.63% |
| TSM(BE) | 46.45%(+0.82) |
| TSM(MCA) | 47.57%(+1.94) |
| TSM(BE+MCA) | 48.05%(+2.42) |

As shown in this table, MCA clearly outperforms BE and is compatible with BE which leads to significant improvement of the baseline method. It implies that MCA is compatible with those works and can contribute to further addressing the background bias issue.

Reviewer comment (Experiments):
There is weak motivation for the choice of backbone model. The experiments used TSM, a 3D CNN with shift modules, but it is only one among many action recognition models. While other models (slowfast and uniformer) are used, they are not applied for all datasets. The paper should justify why TSM is the best for demonstrating the proposed method’s effect or otherwise compare with more recent methods mentioned in the related work.

Author response:
**Weakness 5: Weak motivation for TSM.**

Thanks for this question. The reasons for choosing TSM can be summarized into two folds:  

(1) The implementation of TSM does not contain other modern data augmentations so it will be easier for us to analyze the efficacy of MCA. In contrast, Uniformer[3] is a competing method that involves multiple strong data augmentations, e.g., Mixup[4]/CutMix[5]/RandAugment[6], and the efficacy of MCA upon these methods will be different from the improvement brought by its own. Nevertheless, we highlight that MCA is compatible with those methods based on our analysis in Fig. 7.

(2) TSM is a 2D network which is based on ResNet50[7] and is efficient in implementation. Considering the resources we have, we choose TSM to save the training time so that we can conduct more experiments to validate the effectiveness and generalization ability of MCA.

To further address the Reviewer's concern, we implement Uniformer, which is a very competitive transformer method, on other datasets for empirical validation:
| Method  | Mini-Kinetics Acc. | HMDB51 Acc. |  UCF101 Acc. |
| :-----: | ------------ | :-----: | :-----: |
| Uniformer | 76.12% | 45.75% | 82.45% |
| Uniformer+MCA | 77.18% | 50.20% | 84.27% |

It can be noticed that MCA consistently boosts the performance of Uniformer across different datasets which validates its effectiveness and generalization ability.

Reviewer comment (Novelty):
Channel swap is not a novel augmentation technique and has already been implemented in libraries. Examples include:  
https://kornia.readthedocs.io/en/latest/augmentation.module.html#kornia.augmentation.RandomChannelShuffle  
https://albumentations.ai/docs/api_reference/full_reference/#albumentations.augmentations.transforms.ChannelShuffle

Author response:
**Weakness 6: Channel Swap has been defined.**  

Thanks for pointing it out. First, we want to clarify that Channel Swap is only part of the design of SwapMix. Further, we emphasize that our motivation is different from theirs as Channel Swap is proposed to simulate the effect of hue variance in video recognition and there is no prior work have done so based on our best knowledge.

Reviewer comment (Writing):
Definitions for key terms are incomplete. Affinity, Diversity, and high Expected Calibration Error (ECE) are mentioned in the paper, but only Affinity is defined. ECE is only shown in experiments (Affinity may be inferred from Figure 5 indirectly).

Author response:
**Weakness 7: Definition of Diversity and ECE.**  

Thanks for bringing it out and we will add the definitions of Diversity and Expected Calibration Error (ECE) in our final version.

Reviewer comment (Experiments):
How does this approach compare to the one that uses only optical flow or other motion representation and not appearance?

Author response:
Thanks for the good question. Similar to other data augmentation methods, MCA is designed for RGB modality and it is hard to fairly compare the results from different modalities as they exhibit very different performance. To validate the efficacy of MCA, we revise the self-supervised learning method BE[1], which is robust to background bias and prioritizes motion information, to data augmentation operation and adapt it in supervised learning where we train it on Something-Something V1 dataset over TSM[2] for fair comparisons:
| Method | Top-1 Acc. |
| :-----: | :-----: |
| TSM | 45.63% |
| TSM(BE) | 46.45%(+0.82) |
| TSM(MCA) | 47.57%(+1.94) |
| TSM(BE+MCA) | 48.05%(+2.42) |

As shown in this table, MCA clearly outperforms BE and is compatible with BE which leads to significant improvement of baseline method. This further demonstrates the effectiveness and generalization capability of MCA and we will add the results in the final version.

Reviewer comment (Experiments):
Does the MCA approach work better with other more modern and robust core networks than TCM?

Author response:
Thanks for asking this question. We validate MCA over different architectures in Tab. 2 where it is implemented on the popular 3D network: SlowFast[3] and very competing Transformer network: Uniformer[4] and MCA consistently improves the accuracy of these methods. Further, we stress that Uniformer is a competitive modern network that is composed of multiple data augmentations and exhibits very strong performance. From our analysis in Fig. 6/7, MCA is compatible with those augmentation operations and can further boost its performance across different frame numbers.
| Method | Frame | Top-1 Acc. | GFLOPs |
| :-----: | :-----: | :-----: | :-----: |
| Uniformer-S | 32 | 58.8% | 329 |
| Uniformer-S+MCA | 20 | 59.3%(+0.5) | 169 |

Notably, Uniformer-S (20 Frame) with MCA obtains the accuracy of 59.3% on Something-Something V1 dataset, outperforming the official results Uniformer-S (32 Frame) with higher accuracy and markedly less computational costs.