Paper: Learning with Mixture of Prototypes for Out-of-Distribution Detection (ICLR-2024)

Reviewer comment (Experiments):
**Hyperparameter Settings:** While it's commendable that ablation studies were performed for the introduced hyperparameters like $K$ (number of prototypes) and $\alpha$ (momentum), how do we choose these hyperparameters for different models or datasets? Is there a potential performance dip with varying settings?

Author response:
**Q1. Discussions on hyperparameters for different models or datasets.**

Fig. 5\(c\) and 8(b) show the performance using different hyperparameters $K$ and $\alpha$, which are produced on the benchmark using CIFAR-100 as ID data and using others (such as SVHN, Place365, LSUN, etc) as OOD data. The curves in Fig. 5\(c\) and 8(b) are the performance of the same model trained on the same ID data (e.g., CIFAR-100) on different unseen OOD datasets. They show that the proposed method can perform well and stably given various proper hyperparameters. 

Considering the training is only applied on ID data (where OOD data can only be seen in testing), the setting of hyperparameters is mainly for reflecting the ID data characteristics from obtaining high-quality embedding space (with good OOD detection behaviour in testing). In experiments, we observed the method is robust to the hyperparameter setting on different training ID datasets (such as CIFAR-100, CIFAR-10, ImageNet-100). Although tuning the hyperparameters in a proper range may slightly increase or decrease the performance, we use the **same default** hyperparameters to produce the state-of-the-art results in Tables 1-8, demonstrating the robustness of hyperparameter settings on different datasets.

Reviewer comment (Evaluation):
**Intra-class Heterogeneity:** Fig. 5(c) shows varying prototype effects across datasets. Is this due to different levels of intra-class variability? Do authors have any insights or experiments on why this happens?

Author response:
**Q3. Varying behaviours across datasets in Fig. 5\(c\) (i.e., Fig. 4\(c\) in the revised version).**

As discussed in Q1, the curves are for different OOD datasets used in inference, where the model is trained on CIFAR-100 as the ID dataset. It shows that the proposed method can perform well consistently with different settings of the prototypes. Although there may exist an ideal setting for a given training dataset, considering that the main role of prototypes is to regularize/shape the embedding space, a properly set (i.e., not too small or too large) parameter is enough for producing good results. 

The behaviours of OOD detection on different testing OOD datasets are slightly different, **mainly due to the difference in data characteristics and the gaps between ID and OOD data**, which is not strongly associated to the intra-class variability in the OOD datasets seen only in testing. For example, while using the place recognition dataset Place365 as OOD data, an OOD input image from Place365 (with a place as the label) can easily contain similar semantic information as the images in CIFAR-100, making OOD detection on it difficult and leading to more unstable results.

Reviewer comment (Novelty):
**Technical Contribution:** The paper's core idea, i.e., prototypical learning with a mixture of prototypes, seems like a straightforward and easy extension of CIDER. I acknowledge the improvements shown in experiments and welcome further discussion on this.

Author response:
**Q4. Highlight the difference to CIDER.**

Thank you for the comments. We highlight the significance of our method and the difference to CIDER. 

Among distance-based OOD methods, the main objective is to learn the embedding space (using only ID data) that can discriminate the ID and unseen OOD sample. Different regularizations and supervisions are investigated in various methods based on similar and fair basic frameworks, such as CSI, SSD, CIDER, and our method PALM.

Despite the outstanding performance of CIDER, we identified the limitations of the straightforward (single-prototype) modelling in CIDER. **1)** By analyzing the underlying insights, we first propose to use mixture modelling for better regularizing/shaping the embedding space and thus formulate the mixture vMF models different from CIDER. **2)** Achieving multi-prototype modelling (i.e., moving from a single prototype to multiple prototypes) is not trivial, since, in single prototype modelling, all samples in a class have been assigned to a shared prototype (which is also the simplified assumption). To make multi-prototype modelling achievable, we propose a series of techniques, including prototype updating, sample-prototype assignment considering diversity, and assignment pruning. **3)** Based on that, the novel MLE and contrastive losses based on prototypes use more flexible modelling to shape the embedding space. **4)** Based on the carefully designed modelling and algorithm, the proposed method achieves state-of-the-art performances. We further conducted extensive analyses to analyze the behaviour of the methods. Overall, the relationship between the proposed method and CIDER does not influence the significance of our method.

Reviewer comment (Presentation):
Figure 5 is presented before Figure 4.

Author response:
**Q5. Minor points for figure presentation.**

Thank you for your suggestion. We have revised the presentation of Fig. 4-7 accordingly. 1) The order of Fig. 4 and 5 is changed. 2) The font size in Fig. 4 and 6 is adjusted to be larger.

Reviewer comment (Presentation):
Figures 4 & 6 are too small to recognize when printed. Consider resizing for clarity.

Author response:
**Q5. Minor points for figure presentation.**

Thank you for your suggestion. We have revised the presentation of Fig. 4-7 accordingly. 1) The order of Fig. 4 and 5 is changed. 2) The font size in Fig. 4 and 6 is adjusted to be larger.

Reviewer comment (Theory):
PALM, like its predecessor CIDER, heavily relies on the hyperspherical representation of the learned embeddings to formulate and shape the embedding space. In CIDER, this representation was crucial in achieving strong ID-OOD separability and ID classification. However, since PALM is a mixture of Gaussian, that assumption that the embeddings need to be normalized to unit-norm or need to lie in a hyper spherical space may not necessarily be needed. I wonder if this assumption could be lifted so as to make PALM more general.

Author response:
> **Q1. Mixture modeling and normalizing to hyperspherical embedding space**

In our method, we use the mixture of von Mises-Fisher (vMF) distributions (instead of the mixture of Gaussian distributions). The vMF distribution models the points on the hypersphere. Based on this model, we normalize the embeddings to the unit norm. The vMF distribution is commonly used to formulate and explain the embedding space from contrastive learning (Wang & Isola, 2020) ([1] in the following). Based on the contrastive representation learning on the hypersphere (and proper regularizations), the learned embeddings can be discriminative and beneficial for OOD detection, as commonly used in many methods (e.g., SimCLR, SupCpon, CIDER, and PALM) including our PALM. Different from CIDER using single vMF distribution for a class, we use the mixture modelling of vMF to capture the more complex and realistic data. 

[1] Wang, T. and Isola, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning (pp. 9929-9939). PMLR.

Reviewer comment (Reproducibility):
It is unclear from the manuscript how the mixture proportions $\omega_{i, k}$ are learned. As the authors may know, GMMs are very sensitive to the initialization of the mixture proportions. A suboptimal initialization may lead to the model getting stuck in a local minimum.

Author response:
> **Q3. Estimation of the soft assignment weights $w_{i,k}$.**

To avoid learning a collapsed model, we encourage samples of the same class in a batch to be assigned to diverse prototypes. We optimize $w_{i,k}$'s in $\mathbf{W}^c$ (for each class $c$) by solving the transportation polytope problem in Eq. (8) and (9) (in the revised paper) to maximize the similarity between samples and the assigned prototypes with the regularization. The solution is obtained via Eq. (6). More details of the updating problem are in Sec. 3.3 and Appendix A.2. 
Considering that, although some samples are the top-K neighbours of a prototype, some of them may still be far away like outliers, we further prune the assignment weights for more robust and stable updating, achieving better performance as indicated in Fig. 5(a). 

Given the initialized prototypes, the assignment weights (between prototypes and each sample in the batches) in the iterations are calculated relying on the distances and the diversity regularization mentioned above, which avoids the potential trivial and collapsed solution. The prototypes and the subsequent weights are quickly updated to proper positions in the embedding space. Analyses in Appendix C (Tables 7 and 12) show the robustness of initializations (of the prototypes).

Reviewer comment (Theory):
It has sounded quite unintuitive to me why the likelihood of a sample would be expressed as a mixture of prototypes given the strong ID separability flavor the authors would want to endow to their ID detection method. This concern is somewhat assuaged by the assignment pruning, but I think more justification is needed as to why one would want to enforce separability and yet mix the prototypes.

Author response:
> **Q4. Intuitions of mixture modelling and assignment pruning.**

The basic idea is to encourage intra-class compactness and inter-class separability for better ID-OOD separability, as other distance-based OOD detection methods. The basic idea is rooted in representation learning, where better ID representation/separability can lead to better handling of unseen/OOD data (Wang & Isola, 2020; Khosla et al., 2020). 

By using a single prototype to model a class, enforcing all samples (in the class) close to the same prototype in the same way may learn compact intra-class for the majority of samples. However, there may exist outlier samples and diverse sub-classes, which cannot be regularized by a single shared prototype, leading to the mixed embeddings of ID sample and OOD samples, as visualized in Fig. 3(a)(b)\(c\). 
To address the issue, we introduce a mixture of multiple prototypes for each class and expect any sample can be closely modelled by the associate prototype. Note that we do **not** enforce separability between the intra-class prototypes. Instead, the prototype contrastive loss in Eq. (4) encourages the compactness of the inta-class prototypes and separability of the inter-class prototypes, where the supervision signal is delivered to samples via the prototypes. Multiple prototypes help to capture the diverse characteristics of the samples even in a class, leading to better embedding space (See Fig. 3(d)). 

As discussed above, during updating prototypes, some samples in selected top-K neighbour samples of a prototype may be too far away from the prototype (but close to another prototype). To approach the mutual neighbour relationship between the assigned samples and prototypes (with limited computations), we use assignment pruning to remove the far-away samples from the top-K neighbour for more reliable prototype updating.

Reviewer comment (Experiments):
By having K dedicated prototypes for each class, PALM significantly increases the memory footprint unlike its predecessor CIDER and other distance-based OOD detection methods. This needs to be addressed in the manuscript.

Author response:
> **Q5. Discussions on memory cost.**

During the training phase, the only additional memory induced by our proposed method is the storing of multiple prototypes as vectors and the estimated assignment weights temporarily held during training. Compared to CIDER which uses one prototype for each class, the requirements for additional parameters are minimal. We only induce a 1.2% increase in parameters. Specifically, the parameter size for standard ResNet-34 is 21.60M, for CIDER is 21.65M, and for PALM is 21.91M. The paper has been revised accordingly.

Reviewer comment (Experiments):
I have some concerns about scalability. The introduction of multiple prototypes and their dynamic updating could lead to scalability issues, especially when handling very large datasets or a vast number of classes.

Author response:
> **Q1. Discussion on scalability with multiple prototypes and computational cost.**

Although our method involves multiple prototypes per class, it introduces minimal additional overhead on both **computation** and **memory**, not influencing the scalability. 
- In terms of memory, the additional parameters introduced by the multiple-prototype design are only the prototypes as vectors, and the assignment weights for each batch are temporally calculated during training. Under the default setting with 6 prototypes for each of 100 classes (i.e., 600 prototypes in total), it introduces 1.2% of additional parameters compared to the single prototype method. Specifically, based on ResNet-34 with 21.60M, CIDER introduces one prototype per class, increasing the parameters to 21.65M, while our method is 21.91M. The additional computation caused by ours is restricted. 
- The additional computations of our method are mainly caused by calculating the soft assignment weight using the efficient Sinkhorn-Knopp approximation, which takes only 32 milliseconds each time. As discussed in Appendix C.1, the computation time is less than or similar to the previous efficient and powerful methods. With an optimized implementation, our method with multiple prototypes can be faster than CIDER. The computation times are based on the GeForce RTX 3090 GPU. 

In the paper, we validated the proposed method with the ImageNet-100 (as ID data used in training), which is a relatively large dataset for OOD, where our method works well and performs better than other methods.

Reviewer comment (Evaluation):
In terms of computational cost, PALM might demand additional computational resources.

Author response:
> **Q1. Discussion on scalability with multiple prototypes and computational cost.**

Although our method involves multiple prototypes per class, it introduces minimal additional overhead on both **computation** and **memory**, not influencing the scalability. 
- In terms of memory, the additional parameters introduced by the multiple-prototype design are only the prototypes as vectors, and the assignment weights for each batch are temporally calculated during training. Under the default setting with 6 prototypes for each of 100 classes (i.e., 600 prototypes in total), it introduces 1.2% of additional parameters compared to the single prototype method. Specifically, based on ResNet-34 with 21.60M, CIDER introduces one prototype per class, increasing the parameters to 21.65M, while our method is 21.91M. The additional computation caused by ours is restricted. 
- The additional computations of our method are mainly caused by calculating the soft assignment weight using the efficient Sinkhorn-Knopp approximation, which takes only 32 milliseconds each time. As discussed in Appendix C.1, the computation time is less than or similar to the previous efficient and powerful methods. With an optimized implementation, our method with multiple prototypes can be faster than CIDER. The computation times are based on the GeForce RTX 3090 GPU. 

In the paper, we validated the proposed method with the ImageNet-100 (as ID data used in training), which is a relatively large dataset for OOD, where our method works well and performs better than other methods.

Reviewer comment (Evaluation):
The experimental results from tables 1, 2, and 3 all indicate that PALM does not achieve optimal performance on the texture dataset as it does in other tasks. I am quite curious as to why this phenomenon occurs.

Author response:
> **Q3. Discussion on performance on Texture dataset in Table 1, 2, and 3 of paper.**

Our method can achieve state-of-the-art results on most of the metrics, especially among the methods with similar frameworks (e.g., distance-based methods), such as CIDER, KNN, and SSD. In Table 1, NPOS achieves the best FPR on Textures as the OOD data, since it directly generates OOD samples to boost training. The Textures dataset, characterized by its diverse textural patterns, may be more amenable to artificial OOD sample generation, especially when compared to the synthesis of natural images, which are more challenging to synthesise. Similarly, in Table 2, our method can perform very well via a simple extension from the labelled setting, while other methods include some specific designs for the unlabelled setting (without semantic labels in the ID training data). In Table 3, we validate the effectiveness of our method with different distance measurements, while the Mahalanobis distance-based version is the best.

Reviewer comment (Experiments):
There have been some OOD detection benchmark datasets, such as Openood: Benchmarking generalized out-of-distribution detection. Advances in Neural Information Processing Systems 2022, 35, 32598-32611. Most of the datasets used in experiments are based on standard benchmark datasets. How are these datasets, such as CIFAR, used for OOD in this work?

Author response:
> **Q1. Experiments on OpenOOD benchmark. ``Most of the datasets used in experiments are based on standard benchmark datasets. How are these datasets, such as CIFAR are used for OOD in this work?''**


Thank you for the suggestion. We evaluate the methods with the OpenOOD benchmark and report the results in the following Table 1. In OpenOOD, while using CIFAR-10 and CIFAR-100 as in-distribution data, other datasets, such as TinyImageNet and MNIST, are used as "Near-OOD" or "Far-OOD". As shown in Table 1, our method is better than or competitive with other methods on most metrics, which validates the effectiveness. 

In the paper, the CIFAR datasets are used as in-distribution (ID) data, and other datasets are used as OOD data, which follows the standard experimental protocol widely used in previous works, e.g., CIDER, SSD, Energy, and KNN, as another benchmark. The OOD datasets used in this benchmark are different from OpenOOD. In both the standard and near-OOD experiments in Table 1 and Appendix C - Table 5, our method can achieve the best performance average. 

We have added the results on OpenOOD in the paper and updated the discussions. 

**Table 1**: Results on OpenOOD benchmarks. 

|   Methods   | CIFAR-10 (ID) |           | CIFAR-100 (ID) |           |
|:-----------:|:---------:|:---------:|:---------:|:---------:|
|             |  Near-OOD |  Far-OOD  |  Near-OOD |  Far-OOD  |
| MSP         |   88.03   |   90.73   |   80.27   |   77.76   |
| ODIN        |   82.87   |   87.96   |   79.90   |   79.28   |
| Vim         |   88.68   |   93.48   |   74.98   |   81.70   |
| Energy      |   87.58   |   91.21   |   80.91   |   79.77   |
| OpenGAN     |   53.71   |   54.61   |   65.98   |   67.88   |
| VOS         |   87.70   |   90.83   | **80.93** |   81.32   |
| CSI         |   89.51   |   92.00   |   71.45   |   66.31   |
| kNN         |   90.64   |   92.96   |   80.18   |   82.40   |
| NPOS        |   89.78   |   94.07   |   78.35   |   82.29   |
| CIDER       |   90.71   |   94.71   |   73.10   |   80.49   |
| PALM (Ours) | **92.96** | **98.09** |   76.89   | **92.97** |

Reviewer comment (Experiments):
One of the reasons that the proposed model outperforms the compared baselines is it better estimates the class or sample distribution due to the multi-centroid assumption. How about the comparison with generative based models, such as GAN based model? Reference: Out-of-domain detection based on generative adversarial network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 714-718).

Author response:
> **Q2. Comparison with generative-based (GAN-based) methods.**

Thank you for the suggestion. We discuss the generative-based (GAN-based) methods [1,2] for OOD detection in the revised paper. Considering the GAN-based method in [1] was originally designed for NLP tasks, we mainly conduct the comparison using the related method OpenGAN [2] on the OpenOOD benchmark, as shown in Table 1. The comparison further provides a broader perspective on the analysis. The results and discussions have been added to the revised version. 


[1] Ryu, S., Koo, S., Yu, H. and Lee, G.G., 2018. Out-of-domain detection based on generative adversarial network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 714-718).

[2] Kong, S. and Ramanan, D., 2021. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 813-822).

Reviewer comment (Reproducibility):
Some technical details are not clearly described. For example, how to have diag(u) and diag(v) in Eqn. (6). Appendix C.1 is not about the detail of this.

Author response:
> **Q3. Details of $\text{diag}(\mathbf{u})$ and $\text{diag}(\mathbf{v})$ on Eqn.(6).**

As introduced in Sec. 3.3 and Appendix A.2, $\mathbf{u}$ and $\mathbf{v}$ are two non-negative vectors for obtaining the solution of $\mathbf{W}^c$ constrained by the transportation polytope $\mathcal{W}$ via Eq.(6). As shown in Lemma 2 in (Cuturi, 2013) in the paper, the solution of $\mathbf{W}^c$ is unique and can be obtained via the form of Eq.(6). $\mathbf{u}$ and $\mathbf{v}$ are two non-negative vectors for obtaining the solution, which can be computed with Shinkhorn's fixed point iteration $(\mathbf{u}, \mathbf{v}) \leftarrow (1/(K\mathbf{H}\mathbf{v}),  1/(B\mathbf{H}\mathbf{u}) )$. $\text{diag}(\mathbf{u})$ and $\text{diag}(\mathbf{v})$ are the diagonal matrix of the two vectors $\mathbf{u}$ and $\mathbf{v}$. The detailed proof and algorithms are referred to (Cuturi, 2013). We revised the paper and appendix A.2 with more details accordingly.