Paper: Modulated Phase Diffusor: Content-Oriented Feature Synthesis for Detecting Unknown Objects (ICLR-2024)

Reviewer comment (Theory):
The idea of replacing Gaussian noise in the diffusion process with Gaussian average operation seems to be an experimental result, lacking theoretical explanation and formula derivation.

Author response:
Q1: Theoretical Analysis of Gaussian Average Operation

A: Thanks for your helpful comment. Here, we give some theoretical analysis of Gaussian Average Operation.

We first introduce the background on diffusion models. For each training data $\mathbf x_0 \sim q_0(\mathbf x)$, a forward process of the diffusion models is defined from the following Markov chain:

$\mathbf x_i = \sqrt{1-\beta_i}\mathbf x_{i-1} + \sqrt{\beta_i}\mathbf z_{i}, \ \ i=1,...,N$, (1)

where $z_i\sim \mathcal N(0,\mathbf I)$, and $\{\beta_i\}_{i=1}^N$ is a pre-defined noise schedule. Next, we generalize this process in a rorated coordinate system. We first define an orthogonal matrix ${\rm U}$, and subsequently some vector rotated by the matrix as $\bar{\rm x} := {\rm U}^T {\rm x}$.

Then, a generalized forward diffusion process is defined as the following Markon chain:

$q(\bar{\rm x}{\tiny{i}} |\bar{\rm x}{\tiny{i-1}}) = \mathcal{N}(\bar{\rm x}{\tiny{i}}; ({\rm I} - {\rm B}{\tiny{i}})^{\frac{1}{2}}\bar{\rm x}{\tiny{i-1}}, {\rm B}{\tiny{i}}{\rm I})$, (2)

where ${\rm B}_i$ is a diagonal matrix that defines the noise schedule of the process. It is worth noting that Eq. (2) is a generalized version of the standard diffusion, as standard diffusion is retrieved when we set ${\rm U} = {\rm I}$, and ${\rm B}_i = \beta_i {\rm I}$.

Next, we analyze the proposed Gaussian Average Diffusion:

While the choice of the rotation matrix ${\rm U}$ and the noise schedule ${\rm B}_i$ are flexible, we propose to replace adding noise with effective Gaussian Average. Since Gaussian Average is a linear operation, it can be approximated as a matrix multiplication using a circular symmetric matrix $\mathbf W$. At this time, the process of Gaussian Average is defined as follows:

$q({\rm x}{\tiny{i}}|{\rm x}{\tiny{i-1}}) = \mathcal{N}({\rm x}{\tiny{i}}; \sqrt{1-\beta{\tiny{i}}}{\rm W}{\tiny{i}}{\rm x}{\tiny{i-1}}, {\rm C}{\tiny{i}})$,   (3)

where we set ${\rm C}_i = {\rm I} - (1-\beta_i){\rm W}_i^2$ to ensure the process preserves unit variance. Meanwhile, Eq. (3) can also be written as follows:

$\mathbf x_i = \mathbf x_{i-1} -\mathbf H(\mathbf x_{i-1}, i-1) +  {\rm C}_i^{\frac{1}{2}}{\rm z}_i$, (4)

where $\mathbf H(\mathbf x_{i}, i) = \mathbf x_{i} - \sqrt{1-\beta_{i+1}}{\rm W}_{i+1}{\rm x}_i$ is an unnormalized Gaussian high-pass filter. Thus, the forward process gradually destroys high frequencies. To match the definition of the generalized diffusion, we propose to factor the symmetric matrix ${\rm W}$ by eigenvalue-decomposition ${\rm W} = \tilde{\rm U}{\rm D}\tilde{\rm U}^T$ and subsequently ${\rm W}_i = \tilde{\rm U}{\rm D}^{f(i)}\tilde{\rm U}^T$, where $f(i)$ determines a blur schedule.

Meanwhile, when ${\rm B}_i = {\rm I} - (1-\beta_i) {\rm D}^{2f(i)}$ and ${\rm U} = \tilde{\rm U}$, Eq. (3) is equivalent to Eq. (2).

Besides, in Sec. A.6 of Appendix, we also give some analysis about our method. These analyses all demonstrate that the diffusion with Gaussian Average could be used to synthesize specific features effectively, which improves the discrimination ability.

Reviewer comment (Theory):
Why choose to generate phase instead of the original image or amplitude? What are the phase advantages?

Author response:
Q2: The reason of generating phase

A: For unsupervised OOD-OD, since there is no OOD data available, we could not ensure the generated images contain expected OOD contents. Meanwhile, compared with generating features, generating images may require more computational costs. And the generated images are taken as the input for training, increasing the training costs.

In the third paragraph of the Introduction Section, we have interpreted the reason of generating phase. In general, the extracted features of input images could be recognized as containing style and content information. And the content of the OOD features should be different from that of the ID features. Besides, from the frequency perspective, recent researches have shown that the amplitude and phase could be separately regarded as the style and content of the input. To this end, we pay more attention to exploiting the phase information to perform content-oriented feature synthesis, which is instrumental in addressing the key challenge of unsupervised OOD-OD.

Extensive experiments and visualization analysis also demonstrate that leveraging the phase components is indeed helpful for generating expected virtual features, which is instrumental in improving the discrimination ability.

Reviewer comment (Experiments):
Why choose a 5x5 kernel for the Gaussian average? Has the author tried other types or sizes of kernels?

Author response:
Q3: On the other sizes of Gaussian kernel

A: Thanks for your helpful comment. In this paper, we empirically select $5 \times 5$ kernel.

In the Appendix (i.e., Sec A.4), we have made an ablation analysis of the kernel size. Taking PASCAL VOC as ID data for training and MS-COCO as OOD data for evaluation, when the kernel size is separately set to $3 \times 3$, $5 \times 5$, and $7 \times 7$, the corresponding performance is 41.76%, 41.28%, and 42.84%.

Reviewer comment (Theory):
If Gaussian noise is replaced by the Gaussian average, is the diffusion model still valid? Can you provide the formula derivation?

Author response:
Q1: Theoretical Analysis of Gaussian Average Operation

A: Thanks for your helpful comment. Here, we give some theoretical analysis of Gaussian Average Operation.

We first introduce the background on diffusion models. For each training data $\mathbf x_0 \sim q_0(\mathbf x)$, a forward process of the diffusion models is defined from the following Markov chain:

$\mathbf x_i = \sqrt{1-\beta_i}\mathbf x_{i-1} + \sqrt{\beta_i}\mathbf z_{i}, \ \ i=1,...,N$, (1)

where $z_i\sim \mathcal N(0,\mathbf I)$, and $\{\beta_i\}_{i=1}^N$ is a pre-defined noise schedule. Next, we generalize this process in a rorated coordinate system. We first define an orthogonal matrix ${\rm U}$, and subsequently some vector rotated by the matrix as $\bar{\rm x} := {\rm U}^T {\rm x}$.

Then, a generalized forward diffusion process is defined as the following Markon chain:

$q(\bar{\rm x}{\tiny{i}} |\bar{\rm x}{\tiny{i-1}}) = \mathcal{N}(\bar{\rm x}{\tiny{i}}; ({\rm I} - {\rm B}{\tiny{i}})^{\frac{1}{2}}\bar{\rm x}{\tiny{i-1}}, {\rm B}{\tiny{i}}{\rm I})$, (2)

where ${\rm B}_i$ is a diagonal matrix that defines the noise schedule of the process. It is worth noting that Eq. (2) is a generalized version of the standard diffusion, as standard diffusion is retrieved when we set ${\rm U} = {\rm I}$, and ${\rm B}_i = \beta_i {\rm I}$.

Next, we analyze the proposed Gaussian Average Diffusion:

While the choice of the rotation matrix ${\rm U}$ and the noise schedule ${\rm B}_i$ are flexible, we propose to replace adding noise with effective Gaussian Average. Since Gaussian Average is a linear operation, it can be approximated as a matrix multiplication using a circular symmetric matrix $\mathbf W$. At this time, the process of Gaussian Average is defined as follows:

$q({\rm x}{\tiny{i}}|{\rm x}{\tiny{i-1}}) = \mathcal{N}({\rm x}{\tiny{i}}; \sqrt{1-\beta{\tiny{i}}}{\rm W}{\tiny{i}}{\rm x}{\tiny{i-1}}, {\rm C}{\tiny{i}})$,   (3)

where we set ${\rm C}_i = {\rm I} - (1-\beta_i){\rm W}_i^2$ to ensure the process preserves unit variance. Meanwhile, Eq. (3) can also be written as follows:

$\mathbf x_i = \mathbf x_{i-1} -\mathbf H(\mathbf x_{i-1}, i-1) +  {\rm C}_i^{\frac{1}{2}}{\rm z}_i$, (4)

where $\mathbf H(\mathbf x_{i}, i) = \mathbf x_{i} - \sqrt{1-\beta_{i+1}}{\rm W}_{i+1}{\rm x}_i$ is an unnormalized Gaussian high-pass filter. Thus, the forward process gradually destroys high frequencies. To match the definition of the generalized diffusion, we propose to factor the symmetric matrix ${\rm W}$ by eigenvalue-decomposition ${\rm W} = \tilde{\rm U}{\rm D}\tilde{\rm U}^T$ and subsequently ${\rm W}_i = \tilde{\rm U}{\rm D}^{f(i)}\tilde{\rm U}^T$, where $f(i)$ determines a blur schedule.

Meanwhile, when ${\rm B}_i = {\rm I} - (1-\beta_i) {\rm D}^{2f(i)}$ and ${\rm U} = \tilde{\rm U}$, Eq. (3) is equivalent to Eq. (2).

Besides, in Sec. A.6 of Appendix, we also give some analysis about our method. These analyses all demonstrate that the diffusion with Gaussian Average could be used to synthesize specific features effectively, which improves the discrimination ability.

Reviewer comment (Presentation):
The description of the OOD phase in Figure 2 is confusing, the meaning of approximately equal and not equal symbols is unclear.

Author response:
Q4: On the approximately equal and not equal symbols in Figure 2

A: Thanks for your valuable comment. We have revised Figure 2 and removed the approximately equal and not equal symbols. And OOD Phase in Figure 2 is replaced by OOD Feat. Phase.

Reviewer comment (Evaluation):
What are the shortcomings of this method? Does its training and reasoning time have any advantages compared to previous methods?

Author response:
Q5: On the shortcomings of this method

A: During training, since our method introduces the diffusion process, the training time is slightly longer than the baseline method [1], which could be considered a shortcoming. However, compared with GAN-based methods [2], the training and reasoning time is better than them.

[1] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don’t know by virtual outlier synthesis. ICLR, 2022.

[2] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. ICLR, 2018.

Reviewer comment (Writing):
This paper writting has a large space for improvement and not easy to follow. Although the authors presented the main motivations of this paper, there are still many places unclear. The phase information of ID features is used to generate OOD features. Since phase represents more the content, the amplititude information may be more important for different styles (OOD featuers).

Author response:
Q2: The phase information of ID features is used to generate OOD features

A: Sorry for confusing you. Since the phase information has been shown to be related to the content, to synthesize expected OOD features owning different content from that of ID features, we explore generating OOD features by enlarging the phase gap between the ID and generated OOD features. Experimental results demonstrate that the phase information of ID features could be effectively used to generate proper OOD features.

Reviewer comment (Writing):
The motivation on the augmented features (ID or OOD?) is not estabilished.

Author response:
Q1: On the motivation of the augmented features

A: In the Introduction Section, we have indicated that due to the lack of OOD data for training, the challenge of unsupervised OOD-OD mainly lies in how to leverage the known ID data to enhance the ability of distinguishing OOD objects while reducing the impact on the performance of detecting ID objects.

To address this challenge, we leverage the phase components to synthesize expected virtual OOD features, which alleviates the impact of lacking OOD data for training. Meanwhile, we further generate augmented ID features that are beneficial for retaining the detection performance of ID objects. Extensive experimental results and visualization analysis demonstrate that our method is indeed helpful for addressing the critical challenge of unsupervised OOD-OD.

Reviewer comment (Writing):
What is the difference between OOD-OD and open-set OD? The authors seem list them as different. But as I see, they are the same problem.

Author response:
Q2: The difference between OOD-OD and open-set OD

A: The difference between OOD-OD and open-set OD mainly lies in their settings. Compared with OOD-OD, open-set OD requires indicating the number of unknown categories. However, their goal is consistent, i.e., detecting unknown objects. Thus, we simultaneously verify our method on OOD-OD and open-set OD. The performance improvement demonstrates the effectiveness of our method.

Reviewer comment (Presentation):
Lacking the visualization results of phase based OOD data synthesis.

Author response:
Q3: On the visualization results of phase

A: In Fig. 4 and 5, we have shown the phase of the synthesized features. The phase of OOD features is significantly different from that of original features and augmented features, demonstrating that our MPD method could effectively synthesize expected features that differ from the content of the original features and alleviate the impact of lacking OOD data.

Reviewer comment (Experiments):
How about the conventional unknown object detection based on a simple threshold, such as entropy based.

Author response:
Q4: The performance of the conventional unknown object detection

A: The performance of the conventional methods is very poor. In Table 1, MSP is simple baseline method. Based on PASCAL VOC dataset, we can observe that FPR95 value is 70.99% and 73.13%, which is very poor.

Reviewer comment (Theory):
Minimizing the KL between ID and OOD is strange in Eq. 7.

Author response:
Q5: On minimizing the KL in Eq. 7

A: In Eq. 7, we have explained the role of the KL divergence loss, i.e., constraining the prediction consistency between ID and augmented features. This loss term is beneficial for ameliorating the ability of the object classifier for discriminating ID objects.

Reviewer comment (Experiments):
In Eq. 9, there are many losses, which makes the training not easy.

Author response:
Q6: On the training process

A: In Eq. 9, to ensure the effectiveness of our method, we introduce more loss terms. In the experiments, we observe that using more loss terms does not increase the training difficulty. In fig8, we show the training loss curves of VOS (baseline) and our method. The decreasing speed of the loss value is around consistent with the baseline method. Meanwhile, we can see that the loss curve of our method is significantly lower than that of the baseline, which further demonstrates the effectiveness of our method.

Reviewer comment (Novelty):
Actually, studying the processing of features from a phase perspective is not the first of its kind in this paper. In other words, the method [1] has proven that phase-related features are content-oriented in the DG field, which is very similar to OOD. This paper directly uses such a conclusion of DG in OOD and cannot be regarded as a complete innovation.

Author response:
A: We agree with you that our method is inspired by recent observations, i.e., the phase components usually involve content-related information. In general, the goal of DG task is to transfer the model to new domains with different styles. To this end, most DG methods attempt to keep the phase components unchanged and leverage the amplitude components involving style-related information to perform style-level augmentation, which is instrumental in improving the generalization ability. 

However, the phase components are rarely explored. Besides, different from DG tasks, the task of unsupervised OOD is to discriminate known from unknown objects without reliance on any auxiliary data. The challenge mainly lies in how to leverage the given ID data to improve the discrimination ability. One feasible solution is to synthesize virtual OOD features. Meanwhile, we consider that the OOD features should involve the content that differs from the known ID features. To this end, we explore leverage the phase components to synthesize expected virtual OOD features to alleviate the impact of lacking OOD data.

Thus, the novelty of our method mainly is that we first propose a dedicated phase-driven diffusor that leverages the phase components to separately synthesize OOD features and augmented ID features, which improves the ability of discriminating OOD objects without degrading the detection performance of ID objects.

Reviewer comment (Experiments):
As we all know, Faster R-CNN is a classic but outdated detector. It gives a weak baseline of detection comparing to recent new ones. The actual value of OOD-OD is to achieve robust and generalizable object detection in real applications. Thus, using advanced basic detectors such as YOLOv5 [2], YOLOv8 [3], TOOD [4] and DETRs is more meaningful. And it will be important to see if the proposed MPD works or not on these superior detectors. It may not be practical to do more experiments. The authors could give similar explanations and discussions. For example, is MPD universal to these superior detectors?

Author response:
A: Thanks for your helpful advice. To address the challenge of unsupervised OOD-OD, i.e., leveraging the known ID data to enhance the ability of distinguishing OOD objects while reducing the impact on the performance of detecting ID objects, our MPD method aims to leverage the phase components to synthesize virtual OOD features and augmented ID features. Therefore, our method could be applied to both Faster R-CNN and other advanced detectors.

Here, we make an experiment, i.e., plugging our method into YOLOv5. We separately take PASCAL VOC and MS-COCO as the ID dataset and OOD dataset. We observe that plugging our method reduces FPR95 performance by around 7.38%. This further shows that synthesizing virtual features for unsupervised OOD-OD is an effective solution. Meanwhile, our MPD method could be applied to other detectors and improve their performance of discriminating OOD objects.