Paper: A unique M-pattern for micro-expression spotting in long videos (ICLR-2024)

Reviewer comment (Theory):
No more detailed theoretical analyses of why the M-pattern is effective in mitigating the influence of other global movements.

Author response:
In long videos, changes in head posture, image background, brightness, and other factors become more and more apparent over time. Therefore, an obvious drawback of mainstream spotting algorithms based on fixed reference frame is that frames distant from the reference frame are subject to interference from these changes, resulting in the inability to accurately extract subtle micro-expression features. 

The M-pattern is effective in mitigating the influence of global movements for two main reasons: (1) The M-pattern is derived from the proposed skip-$k$-frame (unfixed reference frame strategy) feature extraction, i.e., computing the feature between frame $i$ and frame $i + k$. The two frames involved in optical flow computation are only $k$ frames apart, minimizing changes in  head posture, brightness and etc. with much shorter time span. This allows us to focus on changes predominantly originating from facial muscle movements within RoIs. (2) The M-pattern is derived from the unique movement pattern of micro-expressions. It contains information about the occurrence phase, restoration phase, intensity and duration of micro-expressions. This information can clearly distinguish between micro-expressions and global movements, since the properties of global movements (global, prolong, large-magnitude) are almost opposite to those of micro-expressions (localized, transient, small-magnitude).

We apologize for neglecting to analyze in depth why the M-pattern is effective in mitigating the influence of other global movements and will refine the relevant content in our future versions.

Reviewer comment (Theory):
The method in the article appears to establish a set of conditions and rules, and I am skeptical about whether this is a universally applicable approach in this field.

Author response:
YES. It is a universally applicable approach in the field of MES to establish a set of conditions and rules, primarily aimed at reducing the number of false positives. Different methods will design different rules according to their own characteristics. Our method is designed strictly based on the characteristics of micro-expressions, which has achieved favorable results.

Reviewer comment (Experiments):
Is it possible to demonstrate through experimentation that this method exhibits generalizability on other datasets?

Author response:
YES, it is indeed possible to demonstrate through experimentation that our method exhibits generalizability on other datasets. MES in long videos has only gained attention in recent years, and there are relatively few available datasets. Following the guidelines in Facial Micro-expression Grand Challenge (MEGC), almost all researchers use CAS(ME)$^2$ or SAMM-LV or both to validate their methods. Two new spontaneous micro-expression datasets MMEW and CAS(ME)$^3$ were proposed in 2021 and 2022, respectively. But currently, few works utilize these two datasets; and they are more commonly employed in micro-expression recognition task. We will conduct experiments on these two datasets to validate the generalizability of our method in future works. 

In addition, in our experiments, we added CASME II to validate our method and yielded 0.8571 F1-score. Although CASME II contains fewer global movements, the experimental results are sufficient to illustrate the superiority of our method in describing, quantifying, and spotting micro-expressions.

Reviewer comment (Experiments):
While the proposed system has many parameters, it is not clear how to choose the parameter value.

Author response:
**[A1]** We apologize for not clearly explaining how to select the parameter value. It is a universally applicable approach in the field of micro-expression spotting (MES) to establish a set of rules and parameters to reduce the number of false positive intervals, mainly resulting from global movements. Our method establishes rules and parameters completely based on the characteristics of micro-expression. Below we explain in detail how the parameters are selected.

- **$M_{min}$** and **$M_{max}$**. 

**[Role]** In facial alignment stage,  we use the number of mismatched pixels between the current frame and the reference frame to align the face. Consider two extreme cases: (1) if the alignment is performed on every frame, then successive frames will be very jittery; (2) if all frames are aligned to the same reference frame, then it will not be possible to align them when a significant change in head pose or facial appearance occurs. Therefore, $M_{min}$ is set to reduce the jitter between images and $M_{max}$ is a threshold that implies a significant change in head pose or facial appearance. 

**[How to choose]** We chose them randomly within a certain range, because through multiple experiments, we found that the alignment performance and the spotting results are not sensitive to these two parameters due to the superiority of our proposed method in eliminating global movements. $M_{min}$ and $M_{max}$ are set to 0.1% and 8% of the cutting box area, respectively. Specifically, (1) $M_{min}$ is chosen from [0.05%, 0.1%]. Smaller $M_{min}$ results in continuous jitter between frames, and larger values results in abrupt changes between frames. (2) $M_{max}$ is chosen from [5%, 10%]. While the definition of “significant change” is ambiguous, we can determine that a “significant change” must correspond to a relatively large mismatch value in terms of pixelmatch. 

- **$IoU_{merge}$**. 

**[Role]** Expression is the result of coordinated movements across multiple RoIs (Regions of Interest). For example, a smiling expression may involve movements of the mouth and eyebrow regions. We performed MES on 12 facial RoIs. Each RoI generates several predicted micro-expression intervals. Intervals with $IoU$ (Intersection over Union) larger than $IoU_{merge}$ will be merged since they belong to the same expression.

**[How to choose]** We utilize a grid search with range [0.2, 0.9] and a stride of 0.05 to determine its value. 

- **$T_{ddeg}$** and **$T_{max}$**. 

**[Role]** $T_{ddeg}$ and $T_{max}$ are proposed to distinguish between micro-expressions and interference movements. Interference movements are the primary cause of low spotting accuracy, mainly including global movements (head rotation, image blurring...) and macro-expression movements. From the perspective of RoI, the magnitude of global movements and macro-expressions on certain RoIs is close to the magnitude of micro-expressions, making it challenging for existing methods to differentiate between micro-expressions and global movements. However, from the perspective of expression, The overall magnitude of interference movements is significantly larger than the overall magnitude of of micro-expressions. Therefore, we propose $T_{ddeg}$ and $T_{max}$ to spot micro-expression at the expression-level.

**[How to choose]** We first conducted a coarse grid search with stride 10, followed by a finer search with a smaller stride.

Reviewer comment (Presentation):
In general I think the structure of the paper could be improved. For example, putting dataset description in appendix makes the experiment section harder to follow.

Author response:
**[A2]** Thank you for bringing this issue to our attention. We all agree with your constructive comment that the structure of this paper needs improvement and apologize for any confusion caused by the inappropriate structure. We placed the dataset description in appendix due to limitations in the length of the paper. We will surely refine the structure of the paper in our future versions.

Reviewer comment (Experiments):
For different type of micro-expression spotting, does the system need different set of parameters?

Author response:
**[A3&4]** We apologize for not explaining the micro-expression spotting (MES) task clearly in paper. As a matter of fact, we had not discuss the type of micro-expression in paper. Similar to temporal action localization, we discuss locating all the micro-expression intervals in a long video, i.e., determine the onset frame (the start of expression) and offset frame (end of expression) of the micro-expression. We don't focus on **WHAT** category micro-expression belongs to; we focus on **WHEN** micro-expression occurs. According to the guidelines presented in Facial Micro-expression Grand Challenge (MEGC) [1], we spot two types of expressions in long videos, macro-expression and micro-expression. The expressions we refer to in our daily life are macro-expression, which include prolonged, large-magnitude, global facial muscle movements such as laughing and crying. The expression that unconsciously leaks out when we attempt to conceal our emotions is referred to as micro-expression. Opposite to macro-expression, micro-expression is characterized by transient, small-magnitude, localized. For this reason, micro-expression and macro-expression are spotted separately. 
The proposed M-pattern is used to characterize the movement of micro-expressions, includes information about the occurrence phase, restoration phase, intensity and duration of micro-expressions. And it is also applicable to describing macro-expressions. The experimental results demonstrate the superiority of our method for both micro-expression spotting and macro-expression spotting.

For micro-expression spotting and macro-expression spotting, the system needs different set of parameters, which is mentioned in Appendix A.4. Micro-expression and macro-expression have completely opposite properties and no consensus has yet been reached on the definition of these two types of expressions. It is difficult to distinguish between macro and micro-expression without designing effective parameters.

Reviewer comment (Experiments):
Does the dataset contain a validation part for parameter selection, and a separate test part for metric reporting?

Author response:
**[A5]** As mentioned above, MES aims to locate intervals in long videos where micro-expressions occur. Unlike recognition task that focus on the category of micro-expressions, spotting task concentrates on when micro-expressions occur and predicts the onset (the start of expression) and offset (the end of expression) frames of micro-expression. However, the presence of a large amount of global movements in long videos severely interferes with the spotting of micro-expressions. Therefore, our work focuses on eliminating global movements as well as proposing a unique micro-expression movement pattern for better spotting. We follow the performance metric proposed in Facial Micro-expression Grand Challenge (MEGC) [1]. For a spotted interval $W_{spotted}$, the prediction is considered as a true positive (TP) when there is a ground truth interval $W_{groundTruth}$ fitting the following condition:

$$
\frac{W_{spotted} \bigcap W_{groundTruth}}{W_{spotted} \bigcup W_{groundTruth}} \geq 0.5.
$$

Otherwise, it is false positive(FP). Specifically, if there are $m$ ground-truth intervals in dataset and $a$ ground-truth intervals are spotted; and if there are $n$ predicted intervals with $b$ TPs, then the recall, precision and F1-score are computed as follows:

$$
Recall=\frac{a}{m}, Precision=\frac{b}{n},
F1-score=\frac{2\times Recall \times Precision}{Recall+Precision}.
$$

Traditional methods test spotting performance directly on the entire dataset and do not divide the dataset into train, validation and test sets. Currently, tradition methods are able to effectively distinguish micro-expressions from global movements by utilizing prior knowledge in the field of micro-expressions, and the spotting results are comparable to, or even better than, most of the deep-learning methods, which have relatively poor spotting results due to the excessive number of false positive intervals.

Reviewer comment (Experiments):
It was unclear to the reviewer if the authors updated values of k across datasets (based on the average ME duration calculated from the chosen sample set).

Author response:
We apologize for any confusion caused by our unclear presentation. The value of $k$ is updated across datasets (Appendix A.3). The $k$ value in skip-$k$-frame is calculated as ($N$+1)/2, where $N$ is the average duration of all micro-expressions in the dateset. For datasets CAS(ME)$^2$, SAMM-LV, CASME II, k is set to 7, 37, 34, respectively. The details are shown as follows.

|   |   $~~$CAS(ME)$^2$|  SAMM-LV  | CASME II |
|:-:|:---------:|:---------:|:--------:|
|   | $~~$ME \| MaE | ME \| MaE |    ME    |
| $N$|14 \| 36 | 74 \| 259 |    67    |
| $k$ | $~~$7 \| 18 | 37 \| 130 |    34    |

For each dataset, k is fixed, and all videos in the dataset use the same k value, regardless of the duration. Therefore, $k$ is not updated per sample set, so it does not affect the spotting results in the case of very long videos.

Reviewer comment (Experiments):
If indeed k was updated as per the sample set, how would this impact its update in the case of very long videos (which were omitted by authors of this work)?

Author response:
We apologize for any confusion caused by our unclear presentation. The value of $k$ is updated across datasets (Appendix A.3). The $k$ value in skip-$k$-frame is calculated as ($N$+1)/2, where $N$ is the average duration of all micro-expressions in the dateset. For datasets CAS(ME)$^2$, SAMM-LV, CASME II, k is set to 7, 37, 34, respectively. The details are shown as follows.

|   |   $~~$CAS(ME)$^2$|  SAMM-LV  | CASME II |
|:-:|:---------:|:---------:|:--------:|
|   | $~~$ME \| MaE | ME \| MaE |    ME    |
| $N$|14 \| 36 | 74 \| 259 |    67    |
| $k$ | $~~$7 \| 18 | 37 \| 130 |    34    |

For each dataset, k is fixed, and all videos in the dataset use the same k value, regardless of the duration. Therefore, $k$ is not updated per sample set, so it does not affect the spotting results in the case of very long videos.

Reviewer comment (Experiments):
In section 3.1 Face Alignment, how do you determine the values for M_min and M_max?

Author response:
$M_{min}$ is set to reduce the jitter between images and $M_{max}$ is a threshold that implies a significant change in head pose or facial appearance. $M_{min}$ and $M_{max}$ are set to 0.1% and 8% of the cutting box area respectively, which we have mentioned in Appendix A.4. But we did not discuss why we chose 0.1% and 8%. We apologize for any confusion caused to our readers and will refine the relevant content in our future versions. As a matter of fact, we randomly chose these two parameters within a certain range because through multiple experiments, we found that the alignment performance and the spotting results are not sensitive to these two parameters. Specifically, (1) $M_{min}$ is chosen from [0.05%, 0.1%]. Smaller $M_{min}$ results in continuous jitter between frames, and larger $M_{min}$ values results in abrupt changes between frames. (2) $M_{max}$ is chosen from [5%, 10%]. While the definition of “significant change” is ambiguous, we can determine that a “significant change” must correspond to a relatively large mismatch value in terms of pixelmatch. Due to the proposed algorithm's robustness and its superiority in eliminating global movements, changes in these two parameters will not lead to significant variations in spotting results.

Reviewer comment (Presentation):
Some of the graphs are very hard to read in a printed paper as the font size is so small, for example Figures 9 and 10.

Author response:
Thanks for bringing this issue to our attention. We will recreate these images to ensure clarity for reading on printed paper.

Reviewer comment (Presentation):
The order of Section 3.4 should be before Section 3.2, to match the order of the steps inside the pipeline.

Author response:
**[A1]** The MES process proceeds as follows: (1) facial alignment, (2) feature extraction, (3) spotting. Following the pipeline, the order of Section 3 should be 3.1, 3.4, 3.2, 3.3. The spotting rules are performed on the feature curves obtained in the feature extraction phase. However, we placed feature extraction in Section 3.4 and introduced M-pattern at first. We adopted this writing order for several reasons: (1) the M-pattern was derived from the computation of skip-$k$-frame, and the strategy of unfixed reference frame (skip-$k$-frame) was used throughout the paper; (2) we inferred M-pattern first (Section 3.2), followed closely by the introduction of M-pattern-based spotting rules (Section 3.3). However, the spotting rules cannot address global movements that exhibit single-wave with similar magnitudes to ME on the feature curve. Therefore, we must conduct a global analysis in the early feature extraction stage to eliminate such global movements (Section 3.4).

We have considered the possibility that such an arrangement might cause distress to the reader when writing the paper. And we will reorganize the order and revise the relevant content to make it consistent with the pipeline in future revisions.

Reviewer comment (Presentation):
The spotting rules are presented in a really verbose manner, making Section 3.3 denser than it should be. The authors could instead present the details in a graphical way (probably a flowchart) and dedicate the main text to describing the decision pipeline in more plain words.

Author response:
**[A2]** We thank the reviewer for the constructive suggestions. The spotting rules proposed in this paper are quite different and novel compared to existing methods. In step 2, we provided a detailed explanation on how to utilize the M-pattern to detect ME of varying durations. We also elaborated on spotting of ME without a restoration phase, thereby enhancing the overall comprehensiveness and robustness of the spotting capability. In step 3, we introduced a novel measurement muscle deformation degree $ddeg$ to describe muscle intensity and explain in detail how to reduce FPs at the expression-level, rather than at the RoI-level as in existing methods. As a result, Section 3.3 has an extended length. Thank you for the constructive suggestions and we will illustrate the spotting rules in the most intuitive way using graph and refine the content of Section 3.3.

Reviewer comment (Writing):
There are some typos in the paper:  
- Page 3: "Alignment is the process of adjusting the center coordinate until m between the current matching box and the reference matching box is minimized."  
- "to prevent subsequent frames from being aligned with misaligned frame"  
- "even without facial alignmen."  
- "Thus, we propose a spotting rules"

Author response:
**[A4]** We apologize for typos! We have corrected the typos carefully and refined relevant contents. Part of the corrected sentence is displayed as follows:

(1)Based on the limitations above, we adopt **an** unfixed RF strategy for both facial alignment and feature extraction.

(2)Yan et al. (2014) **utilize** Local Weighted Mean(LWM) transformation to...

(3)Common feature descriptors include MDMO (Liu et al., 2016), LCN (Yap et al., 2022), LBP-TOP(Tran et al., 2019), HOOF (Verburg & Menkovski, 2019)**, etc.**

(4)Third, based on M-pattern and ME characteristics, we **provide novel ME spotting rules**.

(5)But we must ensure that the updated RF is well-aligned to prevent subsequent frames **from aligning with a misaligned frame**.

(6)... satisfy the assumptions of OF (minor displacement and constant apparent brightness) even without facial **alignment**.

(7)Thus, we **propose spotting rules** based on M-pattern and ME characteristics.

Reviewer comment (Presentation):
Is there any particular reason to present the feature extraction procedure before describing the M-pattern? (Possibly the order of the steps in the method is misunderstood and feature extraction goes last.)

Author response:
**[A1]** The MES process proceeds as follows: (1) facial alignment, (2) feature extraction, (3) spotting. Following the pipeline, the order of Section 3 should be 3.1, 3.4, 3.2, 3.3. The spotting rules are performed on the feature curves obtained in the feature extraction phase. However, we placed feature extraction in Section 3.4 and introduced M-pattern at first. We adopted this writing order for several reasons: (1) the M-pattern was derived from the computation of skip-$k$-frame, and the strategy of unfixed reference frame (skip-$k$-frame) was used throughout the paper; (2) we inferred M-pattern first (Section 3.2), followed closely by the introduction of M-pattern-based spotting rules (Section 3.3). However, the spotting rules cannot address global movements that exhibit single-wave with similar magnitudes to ME on the feature curve. Therefore, we must conduct a global analysis in the early feature extraction stage to eliminate such global movements (Section 3.4).

We have considered the possibility that such an arrangement might cause distress to the reader when writing the paper. And we will reorganize the order and revise the relevant content to make it consistent with the pipeline in future revisions.

Reviewer comment (Experiments):
Do you get a unique M-curve for the whole sequence or is there one M-curve per ROI?

Author response:
**[A5]** In the task of MES in long videos, each video contains several or zero micro-expressions, macro-expressions, along with a considerable amount of global and habitual movements. Therefore, for a video, we get several or zero M-curves (predicted intervals) per RoI, and intervals with IoU (Intersection over Union) larger than $T_{merge}$ will be merged since they belong to the same expression.