Paper: SODA: Stream Out-of-Distribution Adaptation (ICLR-2024)

Reviewer comment (Novelty):
The proposed algorithm requires strong supervision feedback on whether each instance is OOD data in the online learning procedure, which is difficult to satisfy in real-world applications. In this problem setting (whether an instance is OOD data is available), the learning task is then over-simplified. Such a strong supervision assumption makes the proposed method straightforward, which is a simple extension of online learning algorithms with a specific loss function, that is, a combination of ID loss and OOD loss in this draft.

Author response:
> The reviewer's concern regarding the supervised feedback requirement of SODA is understandable, and it is a large part of why we designed the no-feedback unsupervised extension to SODA. However, we would like to direct the reviewer to consider several other factors regarding the supervised online learning setting. Although obtaining feedback might appear challenging at first glance, there are many real-world scenarios where relevant feedback is provided during online interaction. For instance, in cases where the model aims to predict whether an image posted online receives impressions, clicks, or content violation reports (OOD), the environment will eventually provide labels based on future data, indicating whether the image indeed garners impressions/clicks or triggers content violation (OOD) reports. Moreover, there may also be cases where implementor-provided feedback is given during the online interaction. Finally, even in the worst-case scenario where no feedback is possible, the unsupervised extension (USODA) can still be leveraged, which demonstrates very strong performance on the same comprehensive set of evaluations as SODA.

Reviewer comment (Novelty):
The theoretical results in this work are well established in the online learning literature, which limits the technical contribution of this research. These results are relatively straightforward to obtain when a specific convex loss function is specified. In this draft, it is the combination of the ID loss and the OOD loss.

Author response:
> Our primary objective with this work is to address a significant gap in modern Out-of-Distribution (OOD) detection, where OOD detection is often confined to the static supervised setting. Therefore, the goal of the theoretical analysis is to showcase how to leverage the classic online learning analysis for the specific use case of online OOD detection. Furthermore, in online OOD, our model is learning to be uncertainty-aware, and built to function as a detector for semantically shifted OOD samples. In other words, standard online learning analysis differs from our work in that, in standard online learning, no OOD detection occurs throughout the online interaction, and the model is not learning to be more uncertainty-aware. Thus, we believe our theoretical contribution is valuable and complementary to existing online learning literature.

Reviewer comment (Reproducibility):
Regarding the implementation of the algorithm, determining the hyperparameter $\lambda$ for OOD detection before and during the online learning process seems to be challenging. Furthermore, the authors do not elaborate on how $\lambda$ is determined in the experiments or how it affects the algorithm's performance. There is also no published code.

Author response:
> We understand the reviewer's confusion, as comprehending where the lambda threshold lies in our evaluations can be perplexing. Throughout our work, we’ve consistently adhered to established norms in OOD detection. In our online simulation, we internally utilize the widely adopted OOD score, when the true positive rate of a validation ID set is 95%, as the lambda threshold. However, it is crucial to note that this threshold is neither utilized nor required when evaluating SODA/USODA or any prior OOD detection works. In OOD detection literature, the selection of the lambda threshold is generally arbitrary and left to the implementer's discretion. For a more comprehensive look at this, we encourage the reviewer to refer to Appendix C.4. Additionally, further information on standard thresholding practices can also be found in [1, 2]. Should any lingering confusion persist, we would also be happy to provide further clarification.

Reviewer comment (Experiments):
How is $\lambda$ in the algorithm determined in the experiments and how does $\lambda$ affect the performance of the algorithm?

Author response:
> We understand the reviewer's confusion, as comprehending where the lambda threshold lies in our evaluations can be perplexing. Throughout our work, we’ve consistently adhered to established norms in OOD detection. In our online simulation, we internally utilize the widely adopted OOD score, when the true positive rate of a validation ID set is 95%, as the lambda threshold. However, it is crucial to note that this threshold is neither utilized nor required when evaluating SODA/USODA or any prior OOD detection works. In OOD detection literature, the selection of the lambda threshold is generally arbitrary and left to the implementer's discretion. For a more comprehensive look at this, we encourage the reviewer to refer to Appendix C.4. Additionally, further information on standard thresholding practices can also be found in [1, 2]. Should any lingering confusion persist, we would also be happy to provide further clarification.

Reviewer comment (Novelty):
Originality and Significance: My main concern about the paper is the originality of the proposed method. It seems to me that Algorithm 2 (and Algorithm 3 in the Appendix) is a direct application of the classical Online Gradient Descent (OGD) algorithm [1, Chapter 3.1] with the cross-entropy loss. The regret analysis for OGD is somewhat standard. Although the authors have provided some related work discussion, I am unconvinced by the claim "...our framework differs in that we deal with non-stationary OOD data in the context of OOD detection." For the classical online convex optimization framework, the loss function can be arbitrarily different at each round (under certain boundedness assumptions); the framework can already be used for non-stationary data.

Author response:
**Originality and differences with Online Learning**

> We would like to offer several new perspectives for the reviewer to consider as they engage with our work. In contrast to traditional OOD detection methods that operate in the static supervised setting without leveraging any online observations, and traditional online learning methods that do not adequately address semantically shifted OOD, our work is designed to bridge this gap and provide insights into effective OOD detection in an online setting [2].
> We recognize that leveraging online learning to address OOD detection might seem straightforward. However, we want to emphasize that (1) this problem has never been previously addressed and (2) this problem offers a necessary contribution for liberating existing OOD detection away from the confines of its static supervised setting. Additionally, the Online OOD regime is disjoint from both OOD detection and online learning, presenting unique and previously unaddressed challenges such as:
> - How do we define an online mixture environment embedded with semantically shifted out-of-distribution?
> - What does "non-stationary" mean in the context of semantically shifted OOD and OOD detection?
> - What is the most appropriate way to incrementally adjust OOD detectors while preserving model classification in the online setting?
> - How do the unsupervised and supervised regimes differ, and is it even feasible to build an algorithm with both unsupervised ID and OOD observations?
> - Lastly, how do we appropriately evaluate Online OOD methods, and what metrics/datasets should we employ?

**Response on Non-stationarity**

> The reviewer raises an excellent point, regarding the fact that traditional online convex optimization (OCO) already addresses non-stationary environments. However, we would like to note that the non-stationary environments defined in traditional OCO significantly differ from the non-stationary setting we address in SODA. Crucially, our online OOD detection framework specifically targets cases where a non-stationary, semantically shifted out-of-distribution is presented alongside an implementation of an OOD detector. More specifically, the goal of traditional OCO is to correctly classify/generalize observations as the non-stationary in-distribution environment undergoes covariate distributional shifts. In contrast, our online OOD detector aims to detect observations, that should not be classified into any in-distribution classes, as the non-stationary out-of-distribution undergoes semantic shifts. In summary, our primary objective in online OOD is to address the uncertainty awareness of non-stationary, semantically shifted OOD observations. While traditional OCO focuses on the classification of non-stationary or covariate-shifted in-distribution observations.

Reviewer comment (Theory):
About the Feedback: In the main paper, the algorithm is developed based on the assumption that the labels of the OOD data are available after the learner has made the prediction. This seems like a strong assumption to me since the main challenge of OOD detection lies in the lack of supervision of OOD data. When the labeled OOD data are available, I am confused about what the difference is between the proposed framework and supervised online learning. Although the authors show in the Appendix that one can also use unsupervised loss functions to perform OOD detection, it is unclear how the proposed unsupervised loss is (theoretically) guaranteed to output a well-performing OOD detector, which makes the method less appealing.

Author response:
**USODA and Supervised Feedback**

> The reviewer raises an excellent point regarding the strong OOD feedback assumption in SODA. Although obtaining feedback may appear challenging at first glance, there are many real-world scenarios where feedback is available during online interaction. For instance, in cases where the model aims to predict whether an image posted online receives impressions, clicks, or content violation reports (OOD), the environment will eventually provide labels based on future data, indicating whether the image indeed garners impressions/clicks or triggers content violation (OOD) reports. Similarly, any human-in-the-loop environment would also have readily available feedback. Even in the worst-case scenario where no feedback is possible, we provided an unsupervised extension to SODA (USODA). However, given that the unsupervised method lacks clear guarantees on its OOD estimates, it is challenging for us to provide strong theoretical results without making stronger assumptions on either the robustness of the model or the underlying online environment. Nevertheless, USODA still demonstrates exceptional performance on the same comprehensive set of evaluations as SODA (Appendix D).  Additionally, the goal with the L^H_t loss function is to function alongside L^{ood}_t to min-max entropy, thereby making the model confidence more separable between ID and OOD samples. Although it may be initially confusing how minimizing entropy can correlate with minimizing the loss, there are actually many prior test-time adaptation works that already leverage and provide great insights on this question [1].

Reviewer comment (Theory):
About the Theoretical Analysis: I think the statement of Theorem 3.1 is somewhat informal, as the conditions of the loss functions and parameter settings are not provided. It seems to me that the analysis in the Appendix crucially relies on the convexity of the loss function, and thus, the linear model is applied. However, in the main paper, the algorithm is equipped with a neural network. I think it would be better if the authors could clearly mention the discrepancy between the theory and algorithm implementation.

Author response:
**Linear Assumption in the Theorem**

> The reviewer raises a valid point regarding the assumption of linear classifiers in Theorem 3.1. The theorem has its empirical relevance. Specifically, our guarantee ensures that all linear probing methods, a common approach used throughout online and continual learning, will achieve optimality. Furthermore, it is worth mentioning that linear probing of the final layer is largely employed in our unsupervised extension of SODA. Thus, our findings should still hold practical value in our USODA algorithm and various online applications.

Reviewer comment (Novelty):
Could you provide more discussion on the differences between the proposed methods (Algorithms 2 and 3) and the online learning algorithm (e.g., OGD)?

Author response:
**Originality and differences with Online Learning**

> We would like to offer several new perspectives for the reviewer to consider as they engage with our work. In contrast to traditional OOD detection methods that operate in the static supervised setting without leveraging any online observations, and traditional online learning methods that do not adequately address semantically shifted OOD, our work is designed to bridge this gap and provide insights into effective OOD detection in an online setting [2].
> We recognize that leveraging online learning to address OOD detection might seem straightforward. However, we want to emphasize that (1) this problem has never been previously addressed and (2) this problem offers a necessary contribution for liberating existing OOD detection away from the confines of its static supervised setting. Additionally, the Online OOD regime is disjoint from both OOD detection and online learning, presenting unique and previously unaddressed challenges such as:
> - How do we define an online mixture environment embedded with semantically shifted out-of-distribution?
> - What does "non-stationary" mean in the context of semantically shifted OOD and OOD detection?
> - What is the most appropriate way to incrementally adjust OOD detectors while preserving model classification in the online setting?
> - How do the unsupervised and supervised regimes differ, and is it even feasible to build an algorithm with both unsupervised ID and OOD observations?
> - Lastly, how do we appropriately evaluate Online OOD methods, and what metrics/datasets should we employ?

Reviewer comment (Theory):
Could we have certain guarantees for the unsupervised OOD detection loss $\mathcal{L}_t^H$? It seems to me one can also apply similar arguments as Theorem 2 to obtain a regret bound in terms of $\mathcal{L}_t^H$, but it is unclear why minimizing $\mathcal{L}_t^H$ can lead to a well-performed detector that can eventually minimize the true loss function $\mathcal{L}_t$.

Author response:
**USODA and Supervised Feedback**

> The reviewer raises an excellent point regarding the strong OOD feedback assumption in SODA. Although obtaining feedback may appear challenging at first glance, there are many real-world scenarios where feedback is available during online interaction. For instance, in cases where the model aims to predict whether an image posted online receives impressions, clicks, or content violation reports (OOD), the environment will eventually provide labels based on future data, indicating whether the image indeed garners impressions/clicks or triggers content violation (OOD) reports. Similarly, any human-in-the-loop environment would also have readily available feedback. Even in the worst-case scenario where no feedback is possible, we provided an unsupervised extension to SODA (USODA). However, given that the unsupervised method lacks clear guarantees on its OOD estimates, it is challenging for us to provide strong theoretical results without making stronger assumptions on either the robustness of the model or the underlying online environment. Nevertheless, USODA still demonstrates exceptional performance on the same comprehensive set of evaluations as SODA (Appendix D).  Additionally, the goal with the L^H_t loss function is to function alongside L^{ood}_t to min-max entropy, thereby making the model confidence more separable between ID and OOD samples. Although it may be initially confusing how minimizing entropy can correlate with minimizing the loss, there are actually many prior test-time adaptation works that already leverage and provide great insights on this question [1].

Reviewer comment (Reproducibility):
As shown by the equation above (63), it seems to me that the setting of the learning rate $\eta$ requires the knowledge of $T$, $\mathbb{K}^{id}$ and $\mathbb{K}^{ood}$, which are unknown in practice. It would be better if the authors could provide more discussions on how to select the parameters for the algorithm.

Author response:
**Selecting for Hyperparameters**

> While the selection of hyperparameters may seem challenging at first, in practice, choosing the learning rate is actually quite straightforward. In OOD detection, even without online OOD learning, we are guaranteed to have some degree of OOD detection capabilities. This is especially true if one opts to use the stronger OOD detectors as showcased in Section 4.3. Therefore, in practice, it boils down to choosing a learning rate that is not excessively large relative to the model architecture, as the online OOD detection method will still function even with a suboptimal small learning rate. For example, we've found that as long as the learning rate stays lower than 1e-3, the algorithm will generally perform optimally.

Reviewer comment (Novelty):
The solution seems less technical to me. This needs further discussion during rebuttal.

Author response:
The reviewer raises a valid concern and we would like to rephrase some sections of our work that may not have been fully expressed in the initial reading. We want to clarify that our primary objective with SODA is to address a significant gap in modern OOD detection literature, specifically the need for a concrete formalization on how to enable OOD detection in the online setting. Given the limited amount of relevant literature in this area, the base SODA method was designed to serve as a simple yet rigorous realization of our online OOD framework, with the capability of being adaptable to other existing OOD scoring functions (Section 4.3). In short, we presented SODA to provide a necessary initial formalization for the problem introduced in online OOD. Additionally, while the base SODA method offers an algorithmically simple solution to the online OOD problem, we encourage the reviewer to read the unsupervised extension to SODA (USODA) presented in Appendix D. The unsupervised extension places SODA in a more distinct and challenging unsupervised online framework. Sadly, due to space constraints, much of our unsupervised algorithm has been moved to the Appendix. Nevertheless, we hope that the USODA algorithm will better clarify the uniqueness of our problem setting, and showcase how our formalizations can guide new solutions for tackling the online OOD problems.

Reviewer comment (Novelty):
The current version seems to embed an OOD detector into a naive online learning framework.

Author response:
The reviewer raises a valid concern and we would like to rephrase some sections of our work that may not have been fully expressed in the initial reading. We want to clarify that our primary objective with SODA is to address a significant gap in modern OOD detection literature, specifically the need for a concrete formalization on how to enable OOD detection in the online setting. Given the limited amount of relevant literature in this area, the base SODA method was designed to serve as a simple yet rigorous realization of our online OOD framework, with the capability of being adaptable to other existing OOD scoring functions (Section 4.3). In short, we presented SODA to provide a necessary initial formalization for the problem introduced in online OOD. Additionally, while the base SODA method offers an algorithmically simple solution to the online OOD problem, we encourage the reviewer to read the unsupervised extension to SODA (USODA) presented in Appendix D. The unsupervised extension places SODA in a more distinct and challenging unsupervised online framework. Sadly, due to space constraints, much of our unsupervised algorithm has been moved to the Appendix. Nevertheless, we hope that the USODA algorithm will better clarify the uniqueness of our problem setting, and showcase how our formalizations can guide new solutions for tackling the online OOD problems.

Reviewer comment (Novelty):
This paper claims its novelty in online OOD detection. However, [1] is also designed for online OOD detection. Please compare the difference, and specify on what aspect this work is better than [1].  
[1] Wu, Xinheng, et al. "Meta OOD Learning For Continuously Adaptive OOD Detection." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Author response:
We thank the reviewer for bringing to our attention a recent work on continuously adaptive OOD detection (CAOOD) [1]. After further reading, we would like to highlight several points to help clarify the differences between CAOOD and our work. 
- A primary goal of our work is to provide a necessary initial framework to better define the problem of online OOD. However, the framework setting as proposed in CAOOD, differs significantly from online OOD detection. In particular, the distributional shifts observed in the CAOOD are limited to small gradual distributional variations, which puts implicit constraints on what observations are in the online environment [1].
- We also want to clarify that the initial pre-training setting for MOL is significantly different from our work. In particular, MOL requires a model to be pre-trained through a specific training regime, meaning that any existing models would need to be retrained in order to operate with MOL [1]. In other words, MOL is not pre-training agnostic while both SODA and its unsupervised extension (USODA) are agnostic to how the initial model is pre-trained. Furthermore, MOL also requires virtual OOD synthesis to generate OOD samples during initial pre-training, which further alters the pre-training algorithm and adds additional computational complexity.
- Additionally, SODA has the benefit of being incredibly versatile and able to function across various OOD scoring functions (ODIN & energy), while MOL is mostly limited to just the energy scoring function. Our paper also incorporates theoretical analyses on regret that are not present in MOL, helping further distinguish the contributions of our work.
- We would also like to emphasize that the experiments in MOL differ from the experiments presented in our work. For example, many of the evaluations are limited to small gradual distributional shifts over time on low-resolution CIFAR datasets [1].
- Lastly, we want to clarify that the objective of our work is to provide a framework for future research to address the challenges of online OOD. While there may be similarities in the underlying topic, SODA and MOL consider different frameworks, settings, and methodologies for addressing this issue. Therefore, we believe there should be minimal concern regarding overlap, with each work contributing unique insights to the broader topic of OOD detection.

Reviewer comment (Experiments):
Could you specify which items in (2) and (3) are designed specially for an online non-stationary environment?

Author response:
The items in Sections 2 and 3 that specifically pertain to the non-stationary setting include the environment setup in Section 2.1 and the environment declaration in Section 3. In particular, both the distribution P^{out}_t and \pi_t enable a non-stationary environment, as the distribution and its mixture ratio can change throughout the online interaction.

Reviewer comment (Experiments):
In the current SODA, OOD detector and classifier are updated at every new t. However, this update is not always good. Please explain why it is necessary to update OOD detector and classifier for EVERY new t, rather than retain the previous one for some t.

Author response:
The reviewer raises a valid question regarding the optimality of mandating an update at every timestep t. Indeed, in cases where feedback is unavailable or noisy, updating at every timestep t may be suboptimal. However, we would like to direct the reviewer to our unsupervised extension (USODA) presented in Appendix D, where this requirement of updating at every timestep is no longer strictly true. But for the supervised SODA algorithm, given that the supervised setting provides ground truth feedback, adding an update to occur every timestep t should not hurt the model's overall performance.

Reviewer comment (Evaluation):
Will SODA have a high computational cost? Please also specify the computational cost for update process.

Author response:
Additionally, the update process incurs minimal computational cost as it involves just a gradient step for both the supervised SODA and unsupervised USODA methods.

Reviewer comment (Evaluation):
Definition of performance measures such as FPR and AUROC.

Author response:
We understand the reviewer's confusion, and would like to provide some additional background to help clarify the reviewer’s questions regarding evaluation metrics and datasets. Throughout our work, all evaluation metrics, and datasets found in Table 1, closely follow the standard used in OOD detection literature. More specifically, FPR refers to the false positive rate of declaring OOD examples as ID when the true positive rate of ID is set at 95%, and AUROC refers to the area under the receiver operating characteristic curve. A more detailed explanation of these metrics, and the specific OOD datasets employed, can be found in Appendices C.4 and C.1. Figure 1(b) also illustrates an experiment involving an OOD stream defined with respect to time. Works [1, 2] may also be helpful for better contextualizing the metrics and datasets used in OOD detection. Given that our goal with SODA is to provide a means to move OOD detection beyond its traditional offline supervised setting, we felt it is important to align closely with the standards used in OOD detection. This alignment can also help to facilitate a clear and easy path for incorporating the online environment into future OOD detection works.

Reviewer comment (Experiments):
The rationale behind choosing the datasets in Table 1. Why is SCHN considered OOD compared to CIFAR? Is there a way to quantify how far the evaluated distribution is from the training one?

Author response:
We understand the reviewer's confusion, and would like to provide some additional background to help clarify the reviewer’s questions regarding evaluation metrics and datasets. Throughout our work, all evaluation metrics, and datasets found in Table 1, closely follow the standard used in OOD detection literature. More specifically, FPR refers to the false positive rate of declaring OOD examples as ID when the true positive rate of ID is set at 95%, and AUROC refers to the area under the receiver operating characteristic curve. A more detailed explanation of these metrics, and the specific OOD datasets employed, can be found in Appendices C.4 and C.1. Figure 1(b) also illustrates an experiment involving an OOD stream defined with respect to time. Works [1, 2] may also be helpful for better contextualizing the metrics and datasets used in OOD detection. Given that our goal with SODA is to provide a means to move OOD detection beyond its traditional offline supervised setting, we felt it is important to align closely with the standards used in OOD detection. This alignment can also help to facilitate a clear and easy path for incorporating the online environment into future OOD detection works.

Reviewer comment (Evaluation):
Discussion of limitation: The proposed SODA, while being effective in many cases, can fail in other scenarios. For instance, consider a stream with very small visual variations with respect to time (e.g. a surveillance camera). SODA will then be presented with the same batch (with small variations) repeatedly for some time. Would this make the online updates overfit the network parameters and fail to generalize on new novel batches? A discussion on the limitations of SODA should be included.

Author response:
The reviewer raises an excellent question regarding the potential limitation of SODA under small distributional shifts over time. In the traditional OOD detection setting, we define OOD as any observation that undergoes a significant semantic shift from the in-distribution. This means that the small distributional shifts, such as the covariate shifts observed in surveillance footage, may not be considered out-of-distribution in the OOD detection setting. In other words, our goal in OOD detection is to detect semantically shifted observations rather than engaging with covariate distributional shifts. Therefore, the reviewer's question about generalizing to online covariate OOD shifts, would primarily fall within the domain adaptation regime, which is beyond the scope of our current work. Nonetheless, there are many existing works that deal with this problem of life-long domain adaptation [3, 4] and these techniques present existing solutions that can be integrated with our methods to mitigate any anticipated domain adaptation issues.