Original: Is Out-of-Distribution Detection Learnable?
Revised: Is Out-of-Distribution Detection Learnable?

Original: Abstract
Revised: Abstract

Original: Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution.
Revised: Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution.

Original: To ease the above assumption, researchers have studied a more realistic setting: out-of-distribution (OOD) detection, where test data may come from classes that are unknown during training ( i.e. , OOD data).
Revised: To ease the above assumption, researchers have studied a more realistic setting: out-of-distribution (OOD) detection, where test data may come from classes that are unknown during training ( i.e. , OOD data).

Original: Due to the unavailability and diversity of OOD data, good generalization ability is crucial for effective OOD detection algorithms.
Revised: Due to the unavailability and diversity of OOD data, good generalization ability is crucial for effective OOD detection algorithms.

Original: To study the generalization of
Revised: To study the generalization of

Original: OOD detection, in this paper, we investigate the probably approximately correct (PAC) learning theory of OOD detection, which is proposed by researchers as an open problem .
Revised: OOD detection, in this paper, we investigate the probably approximately correct (PAC) learning theory of OOD detection, which is proposed by researchers as an open problem .

Original: First, we find a necessary condition for the learnability of OOD detection.
Revised: First, we find a necessary condition for the learnability of OOD detection.

Original: Then, using this condition, we prove several impossibility theorems for the learnability of OOD detection under some scenarios.
Revised: Then, using this condition, we prove several impossibility theorems for the learnability of OOD detection under some scenarios.

Original: Although the impossibility theorems are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios.
Revised: Although the impossibility theorems are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios.

Original: Based on this observation, we next give several necessary and sufficient conditions to characterize the learnability of OOD detection in some practical scenarios.
Revised: Based on this observation, we next give several necessary and sufficient conditions to characterize the learnability of OOD detection in some practical scenarios.

Original: Lastly, we also offer theoretical supports for several representative OOD detection works based on our OOD theory.
Revised: Lastly, we also offer theoretical supports for several representative OOD detection works based on our OOD theory.

Original:  Introduction
Revised:  Introduction

Original: The success of supervised learning is established on an implicit assumption that training and test data share a same distribution, i e. , in-distribution (ID) [1, 2, 3, 4].
Revised: The success of supervised learning is established on an implicit assumption that training and test data share a same distribution, i e. , in-distribution (ID) [1, 2, 3, 4].

Original: However, test data distribution in many real-world scenarios may violate the assumption and, instead, contain out-of-distribution (OOD) data whose labels have not been seen during the training process [5, 6].
Revised: However, test data distribution in many real-world scenarios may violate the assumption and, instead, contain out-of-distribution (OOD) data whose labels have not been seen during the training process [5, 6].

Original: To mitigate the risk of OOD data, researchers have considered a more practical learning scenario:
Revised: To mitigate the risk of OOD data, researchers have considered a more practical learning scenario:

Original: OOD detection which determines whether an input is ID/OOD, while classifying the ID data into respective classes.
Revised: OOD detection which determines whether an input is ID/OOD, while classifying the ID data into respective classes.

Original: OOD detection has shown great potential to ensure the reliable deployment of machine learning models in the real world.
Revised: OOD detection has shown great potential to ensure the reliable deployment of machine learning models in the real world.

Original: A rich line of algorithms have been developed to empirically address the OOD detection problem [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].
Revised: A rich line of algorithms have been developed to empirically address the OOD detection problem [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].

Original: However, very few works study theory of OOD detection, which hinders the rigorous path forward for the field.
Revised: However, very few works study theory of OOD detection, which hinders the rigorous path forward for the field.

Original: This paper aims to bridge the gap.
Revised: This paper aims to bridge the gap.

Original: In this paper, we provide a theoretical framework to understand the learnability of the OOD detection problem.
Revised: In this paper, we provide a theoretical framework to understand the learnability of the OOD detection problem.

Original: We investigate the probably approximately correct (PAC) learning theory of OOD detection, which is posed as an open problem to date.
Revised: We investigate the probably approximately correct (PAC) learning theory of OOD detection, which is posed as an open problem to date.

Original: Unlike the classical PAC learning theory in a supervised setting, our problem setting is fundamentally challenging due to the absence of OOD data in training.
Revised: Unlike the classical PAC learning theory in a supervised setting, our problem setting is fundamentally challenging due to the absence of OOD data in training.

Original: In many real-world scenarios, OOD data can be diverse and priori-unknown.
Revised: In many real-world scenarios, OOD data can be diverse and priori-unknown.

Original: Given this, we study whether there exists an algorithm that can be used to detect various OOD data instead of merely some specified OOD data.
Revised: Given this, we study whether there exists an algorithm that can be used to detect various OOD data instead of merely some specified OOD data.

Original: Such is the significance of studying the learning theory for OOD detection [4].
Revised: Such is the significance of studying the learning theory for OOD detection [4].

Original: This motivates our question: is OOD detection PAC learnable?
Revised: This motivates our question: is OOD detection PAC learnable?

Original: i.e., is there the PAC learning theory to guarantee the generalization ability of OOD detection?
Revised: i.e., is there the PAC learning theory to guarantee the generalization ability of OOD detection?

Original: To investigate the learning theory, we mainly focus on two basic spaces: domain space and hypothesis space.
Revised: To investigate the learning theory, we mainly focus on two basic spaces: domain space and hypothesis space.

Original: The domain space is a space consisting of some distributions, and the hypothesis space is a space consisting of some classifiers.
Revised: The domain space is a space consisting of some distributions, and the hypothesis space is a space consisting of some classifiers.

Original: Existing agnostic PAC theories in supervised learning [21, 22] are distribution-free, i.e. , the domain space consists of all domains.
Revised: Existing agnostic PAC theories in supervised learning [21, 22] are distribution-free, i.e. , the domain space consists of all domains.

Original: Yet, in Theorem 4, we shows that the learning theory of OOD detection is not distribution-free.
Revised: Yet, in Theorem 4, we shows that the learning theory of OOD detection is not distribution-free.

Original: In fact, we discover that OOD detection is learnable only if the domain space and the hypothesis space satisfy some special conditions, e.g.,
Revised: In fact, we discover that OOD detection is learnable only if the domain space and the hypothesis space satisfy some special conditions, e.g.,

Original: Conditions 1 and 3.
Revised: Conditions 1 and 3.

Original: Notably, there are many conditions and theorems in existing learning theories and many OOD detection algorithms in the literature.
Revised: Notably, there are many conditions and theorems in existing learning theories and many OOD detection algorithms in the literature.

Original: Thus, it is very difficult to analyze the relation between these theories and algorithms, and explore useful conditions to ensure the learnability of
Revised: Thus, it is very difficult to analyze the relation between these theories and algorithms, and explore useful conditions to ensure the learnability of

Original: OOD detection, especially when we have to explore them from the scratch .
Revised: OOD detection, especially when we have to explore them from the scratch .

Original: Thus, the main aim of our paper is to study these essential conditions.
Revised: Thus, the main aim of our paper is to study these essential conditions.

Original: From these essential conditions, we can know when OOD detection can be successful in practical scenarios.
Revised: From these essential conditions, we can know when OOD detection can be successful in practical scenarios.

Original: We restate our question and goal in following:
Revised: We restate our question and goal in following:

Original: Given hypothesis spaces and several representative domain spaces, what are the conditions to ensure the learnability of OOD detection?
Revised: Given hypothesis spaces and several representative domain spaces, what are the conditions to ensure the learnability of OOD detection?

Original: If possible, we hope that these conditions are necessary and sufficient in some scenarios.
Revised: If possible, we hope that these conditions are necessary and sufficient in some scenarios.

Original: Main Results.
Revised: Main Results.

Original: We investigate the learnability of OOD detection starting from the largest space—the total space, and give a necessary condition (Condition 1) for the learnability.
Revised: We investigate the learnability of OOD detection starting from the largest space—the total space, and give a necessary condition (Condition 1) for the learnability.

Original: However, we find that the overlap between ID and OOD data may result in that the necessary condition does not hold.
Revised: However, we find that the overlap between ID and OOD data may result in that the necessary condition does not hold.

Original: Therefore, we give an impossibility theorem to demonstrate that OOD detection fails in the total space (Theorem 4).
Revised: Therefore, we give an impossibility theorem to demonstrate that OOD detection fails in the total space (Theorem 4).

Original: Next, we study OOD detection in the separate space, where there are no overlaps between the ID and OOD data.
Revised: Next, we study OOD detection in the separate space, where there are no overlaps between the ID and OOD data.

Original: Unfortunately, there still exists impossibility theorem (Theorem 5), which demonstrates that OOD detection is not learnable in the separate space under some conditions.
Revised: Unfortunately, there still exists impossibility theorem (Theorem 5), which demonstrates that OOD detection is not learnable in the separate space under some conditions.

Original: Although the impossibility theorems obtained in the separate space are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios.
Revised: Although the impossibility theorems obtained in the separate space are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios.

Original: Based on this observation, we give several necessary and sufficient conditions to characterize the learnability of
Revised: Based on this observation, we give several necessary and sufficient conditions to characterize the learnability of

Original: OOD detection in the separate space (Theorems 6 and 10).
Revised: OOD detection in the separate space (Theorems 6 and 10).

Original: Especially, when our model is based on fully-connected neural network (FCNN), OOD detection is learnable in the separate space if and only if the feature space is finite.
Revised: Especially, when our model is based on fully-connected neural network (FCNN), OOD detection is learnable in the separate space if and only if the feature space is finite.

Original: Furthermore, we investigate the learnability of OOD detection in other more practical domain spaces, e.g. , the finite-ID-distribution space (Theorem 8) and the densitybased space (Theorem 9).
Revised: Furthermore, we investigate the learnability of OOD detection in other more practical domain spaces, e.g. , the finite-ID-distribution space (Theorem 8) and the densitybased space (Theorem 9).

Original: By studying the finite-ID-distribution space, we discover a compatibility condition (Condition 3) that is a necessary and sufficient condition for this space.
Revised: By studying the finite-ID-distribution space, we discover a compatibility condition (Condition 3) that is a necessary and sufficient condition for this space.

Original: Next, we further investigate the compatibility condition in the density-based space, and find that such condition is also the necessary and sufficient condition in some practical scenarios (Theorem 11).
Revised: Next, we further investigate the compatibility condition in the density-based space, and find that such condition is also the necessary and sufficient condition in some practical scenarios (Theorem 11).

Original: Implications and Impacts of Theory.
Revised: Implications and Impacts of Theory.

Original: Our study is not of purely theoretical interest; it has also practical impacts.
Revised: Our study is not of purely theoretical interest; it has also practical impacts.

Original: First, when we design OOD detection algorithms, we normally only have finite ID datasets, corresponding to the finite-ID-distribution space.
Revised: First, when we design OOD detection algorithms, we normally only have finite ID datasets, corresponding to the finite-ID-distribution space.

Original: In this case, Theorem 8 gives the necessary and sufficient condition to the success of OOD detection.
Revised: In this case, Theorem 8 gives the necessary and sufficient condition to the success of OOD detection.

Original: Second, our theory provides theoretical support (Theorems 10 and 11) for several representative OOD detection works [7, 8, 23].
Revised: Second, our theory provides theoretical support (Theorems 10 and 11) for several representative OOD detection works [7, 8, 23].

Original: Third, our theory shows that OOD detection is learnable in image-based scenarios when ID images have clearly different semantic labels and styles ( far-OOD ) from OOD images.
Revised: Third, our theory shows that OOD detection is learnable in image-based scenarios when ID images have clearly different semantic labels and styles ( far-OOD ) from OOD images.

Original: Fourth, we should not expect a universally working algorithm.
Revised: Fourth, we should not expect a universally working algorithm.

Original: It is necessary to design different algorithms in different scenarios.
Revised: It is necessary to design different algorithms in different scenarios.

Original:  Learning Setups
Revised:  Learning Setups

Original: We start by introducing the necessary concepts and notations for our theoretical framework.
Revised: We start by introducing the necessary concepts and notations for our theoretical framework.

Original: Given a feature space X ⊂ R d and a label space Y :
Revised: Given a feature space X ⊂ R d and a label space Y :

Original: = { 1 , ..., K } , we have an ID joint distribution D X
Revised: = { 1 , ..., K } , we have an ID joint distribution D X

Original: I Y I over X × Y , where X I ∈
Revised: I Y I over X × Y , where X I ∈

Original: X and Y I ∈ Y are random variables.
Revised: X and Y I ∈ Y are random variables.

Original: We also have an OOD joint distribution D X O Y O , where X O is a random variable from X , but Y O is a random variable whose outputs do not belong to Y .
Revised: We also have an OOD joint distribution D X O Y O , where X O is a random variable from X , but Y O is a random variable whose outputs do not belong to Y .

Original: During testing, we will meet a mixture of ID and OOD joint distributions:
Revised: During testing, we will meet a mixture of ID and OOD joint distributions:

Original: D XY :
Revised: D XY :

Original: = (1 − π out ) D X
Revised: = (1 − π out ) D X

Original: I Y
Revised: I Y

Original: I + π out D X
Revised: I + π out D X

Original: O Y O ,
Revised: O Y O ,

Original: and can only observe the marginal distribution D X :=
Revised: and can only observe the marginal distribution D X :=

Original: (1 − π out ) D X
Revised: (1 − π out ) D X

Original: I + π out D X
Revised: I + π out D X

Original: O , where the constant π out ∈ [0 , 1) is an unknown class-prior probability.
Revised: O , where the constant π out ∈ [0 , 1) is an unknown class-prior probability.

Original: Problem 1 (OOD Detection [4]) .
Revised: Problem 1 (OOD Detection [4]) .

Original: Given an ID joint distribution D X I Y I and a training data S :
Revised: Given an ID joint distribution D X I Y I and a training data S :

Original: = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn independent and identically distributed from D X
Revised: = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn independent and identically distributed from D X

Original: I Y
Revised: I Y

Original: I , the aim of OOD detection is to train a classifier f by using the training data S such that, for any test data x drawn from the mixed marginal distribution D X : 1) if x is an observation from D X I , f can classify x into correct ID classes; and 2) if x is an observation from D X O , f can detect x as OOD data.
Revised: I , the aim of OOD detection is to train a classifier f by using the training data S such that, for any test data x drawn from the mixed marginal distribution D X : 1) if x is an observation from D X I , f can classify x into correct ID classes; and 2) if x is an observation from D X O , f can detect x as OOD data.

Original: According to the survey [4], when K > 1 , OOD detection is also known as the open-set recognition or open-set learning [24, 25]; and when K = 1 , OOD detection reduces to one-class novelty detection and semantic anomaly detection [26, 27, 28].
Revised: According to the survey [4], when K > 1 , OOD detection is also known as the open-set recognition or open-set learning [24, 25]; and when K = 1 , OOD detection reduces to one-class novelty detection and semantic anomaly detection [26, 27, 28].

Original: OOD Label and Domain Space.
Revised: OOD Label and Domain Space.

Original: Based on Problem 1, we know it is not necessary to classify OOD data into the correct OOD classes.
Revised: Based on Problem 1, we know it is not necessary to classify OOD data into the correct OOD classes.

Original: Without loss of generality, let all OOD data be allocated to one big OOD class, i.e. , Y O = K + 1 [24, 29].
Revised: Without loss of generality, let all OOD data be allocated to one big OOD class, i.e. , Y O = K + 1 [24, 29].

Original: To investigate the PAC learnability of OOD detection, we define a domain space D XY , which is a set consisting of some joint distributions D XY mixed by some ID joint distributions and some OOD joint distributions.
Revised: To investigate the PAC learnability of OOD detection, we define a domain space D XY , which is a set consisting of some joint distributions D XY mixed by some ID joint distributions and some OOD joint distributions.

Original: In this paper, the joint distribution
Revised: In this paper, the joint distribution

Original: D XY mixed by ID joint distribution D X
Revised: D XY mixed by ID joint distribution D X

Original: I Y I and OOD joint distribution D X
Revised: I Y I and OOD joint distribution D X

Original: O Y O is called domain .
Revised: O Y O is called domain .

Original: Hypothesis Spaces and Scoring Function Spaces.
Revised: Hypothesis Spaces and Scoring Function Spaces.

Original: A hypothesis space H is a subset of function space, i.e. , H ⊂ { h : X → Y ∪ { K + 1 }} .
Revised: A hypothesis space H is a subset of function space, i.e. , H ⊂ { h : X → Y ∪ { K + 1 }} .

Original: We set H in ⊂ { h : X → Y} to the ID hypothesis space.
Revised: We set H in ⊂ { h : X → Y} to the ID hypothesis space.

Original: We also define H b ⊂ { h : X → { 1 , 2 }} as the hypothesis space for binary classification, where 1 represents the ID data, and 2 represents the OOD data.
Revised: We also define H b ⊂ { h : X → { 1 , 2 }} as the hypothesis space for binary classification, where 1 represents the ID data, and 2 represents the OOD data.

Original: The function h is called the hypothesis function.
Revised: The function h is called the hypothesis function.

Original: A scoring function space is a subset of function space, i.e. , F l ⊂ { f : X → R l } , where l is the output’s dimension of the vector-valued function f .
Revised: A scoring function space is a subset of function space, i.e. , F l ⊂ { f : X → R l } , where l is the output’s dimension of the vector-valued function f .

Original: The function f is called the scoring function.
Revised: The function f is called the scoring function.

Original: Loss and Risks.
Revised: Loss and Risks.

Original: Let Y all =
Revised: Let Y all =

Original: Y ∪ { K + 1 } .
Revised: Y ∪ { K + 1 } .

Original: Given a loss function ℓ 2 : Y all × Y all → R ≥ 0 satisfying that ℓ ( y 1 , y 2 )
Revised: Given a loss function ℓ 2 : Y all × Y all → R ≥ 0 satisfying that ℓ ( y 1 , y 2 )

Original: =
Revised: =

Original: if and only if y 1 = y 2 , and any h ∈ H , then the risk with respect to D XY is
Revised: if and only if y 1 = y 2 , and any h ∈ H , then the risk with respect to D XY is

Original: [Equation]
Revised: [Equation]

Original: The α -risk R αD ( h ) :
Revised: The α -risk R αD ( h ) :

Original: = (1 − α ) R in D ( h ) + αR out D ( h ) , ∀ α ∈ [0 , 1] , where the risks R in D ( h ) , R out D ( h ) are
Revised: = (1 − α ) R in D ( h ) + αR out D ( h ) , ∀ α ∈ [0 , 1] , where the risks R in D ( h ) , R out D ( h ) are

Original: [Equation]
Revised: [Equation]

Original: Learnability.
Revised: Learnability.

Original: We aim to select a hypothesis function h ∈ H with approximately minimal risk, based on finite data.
Revised: We aim to select a hypothesis function h ∈ H with approximately minimal risk, based on finite data.

Original: Generally, we expect the approximation to get better, with the increase in sample size.
Revised: Generally, we expect the approximation to get better, with the increase in sample size.

Original: Algorithms achieving this are said to be consistent.
Revised: Algorithms achieving this are said to be consistent.

Original: Formally, we introduce the following definition:
Revised: Formally, we introduce the following definition:

Original: Definition 1 (Learnability of OOD Detection) .
Revised: Definition 1 (Learnability of OOD Detection) .

Original: Given a domain space D XY and a hypothesis space
Revised: Given a domain space D XY and a hypothesis space

Original: H ⊂ { h : X → Y all } , we say OOD detection is learnable in D XY for H , if there exists an algorithm A 3 : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,
Revised: H ⊂ { h : X → Y all } , we say OOD detection is learnable in D XY for H , if there exists an algorithm A 3 : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,

Original: [Equation]
Revised: [Equation]

Original: An algorithm A for which this holds is said to be consistent with respect to D XY .
Revised: An algorithm A for which this holds is said to be consistent with respect to D XY .

Original: Definition 1 is a natural extension of agnostic PAC learnability of supervised learning [30].
Revised: Definition 1 is a natural extension of agnostic PAC learnability of supervised learning [30].

Original: If for any
Revised: If for any

Original: D XY ∈ D XY , π out = 0 , then Definition 2 is the agnostic PAC learnability of supervised learning.
Revised: D XY ∈ D XY , π out = 0 , then Definition 2 is the agnostic PAC learnability of supervised learning.

Original: Although the expression of Definition 1 is different from the normal definition of agnostic PAC learning in [21], one can easily prove that they are equivalent when ℓ is bounded, see Appendix D.3.
Revised: Although the expression of Definition 1 is different from the normal definition of agnostic PAC learning in [21], one can easily prove that they are equivalent when ℓ is bounded, see Appendix D.3.

Original: Since OOD data are unavailable, it is impossible to obtain information about the class-prior probability π out .
Revised: Since OOD data are unavailable, it is impossible to obtain information about the class-prior probability π out .

Original: Furthermore, in the real world, it is possible that π out can be any value in [0 , 1) .
Revised: Furthermore, in the real world, it is possible that π out can be any value in [0 , 1) .

Original: Therefore, the imbalance issue between ID and OOD distributions, and the priori-unknown issue ( i.e. , π out is unknown) are the core challenges.
Revised: Therefore, the imbalance issue between ID and OOD distributions, and the priori-unknown issue ( i.e. , π out is unknown) are the core challenges.

Original: To ease these challenges, researchers use AUROC, AUPR and
Revised: To ease these challenges, researchers use AUROC, AUPR and

Original: FPR95 to estimate the performance of OOD detection [18, 31, 32, 33, 34, 35].
Revised: FPR95 to estimate the performance of OOD detection [18, 31, 32, 33, 34, 35].

Original: It seems that there is a gap between Definition 1 and existing works.
Revised: It seems that there is a gap between Definition 1 and existing works.

Original: To eliminate this gap, we revise Eq.
Revised: To eliminate this gap, we revise Eq.

Original: (2) as follows:
Revised: (2) as follows:

Original: [Equation]
Revised: [Equation]

Original: If an algorithm A satisfies Eq.
Revised: If an algorithm A satisfies Eq.

Original: (3), then the imbalance issue and the prior-unknown issue disappear.
Revised: (3), then the imbalance issue and the prior-unknown issue disappear.

Original: That is, A can simultaneously classify the ID data and detect the OOD data well.
Revised: That is, A can simultaneously classify the ID data and detect the OOD data well.

Original: Based on the above discussion, we define the strong learnability of OOD detection as follows:
Revised: Based on the above discussion, we define the strong learnability of OOD detection as follows:

Original: Definition 2 (Strong Learnability of OOD Detection) .
Revised: Definition 2 (Strong Learnability of OOD Detection) .

Original: Given a domain space D XY and a hypothesis space H ⊂ { h : X → Y all } , we say OOD detection is strongly learnable in D XY for H , if there exists an algorithm A : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,
Revised: Given a domain space D XY and a hypothesis space H ⊂ { h : X → Y all } , we say OOD detection is strongly learnable in D XY for H , if there exists an algorithm A : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,

Original: [Equation]
Revised: [Equation]

Original: In Theorem 1, we have shown that the strong learnability of OOD detection is equivalent to the learnability of OOD detection, if the domain space D XY is a prior-unknown space (see Definition 3).
Revised: In Theorem 1, we have shown that the strong learnability of OOD detection is equivalent to the learnability of OOD detection, if the domain space D XY is a prior-unknown space (see Definition 3).

Original: In this paper, we mainly discuss the learnability in the prior-unknown space.
Revised: In this paper, we mainly discuss the learnability in the prior-unknown space.

Original: Therefore, when we mention that OOD detection is learnable, we also mean that OOD detection is strongly learnable .
Revised: Therefore, when we mention that OOD detection is learnable, we also mean that OOD detection is strongly learnable .

Original: Goal of Theory.
Revised: Goal of Theory.

Original: Note that the agnostic PAC learnability of supervised learning is distribution-free, i.e. , the domain space D XY consists of all domains.
Revised: Note that the agnostic PAC learnability of supervised learning is distribution-free, i.e. , the domain space D XY consists of all domains.

Original: However, due to the absence of OOD data during the training process [8, 14, 24], it is obvious that the learnability of OOD detection is not distribution-free ( i.e. , Theorem 4).
Revised: However, due to the absence of OOD data during the training process [8, 14, 24], it is obvious that the learnability of OOD detection is not distribution-free ( i.e. , Theorem 4).

Original: In fact, we discover that the learnability of OOD detection is deeply correlated with the relationship between the domain space D XY and the hypothesis space H .
Revised: In fact, we discover that the learnability of OOD detection is deeply correlated with the relationship between the domain space D XY and the hypothesis space H .

Original: That is, OOD detection is learnable only when the domain space D XY and the hypothesis space H satisfy some special conditions, e.g., Condition 1 and Condition 3.
Revised: That is, OOD detection is learnable only when the domain space D XY and the hypothesis space H satisfy some special conditions, e.g., Condition 1 and Condition 3.

Original: We present our goal as follows:
Revised: We present our goal as follows:

Original: Goal:
Revised: Goal:

Original: given a hypothesis space H
Revised: given a hypothesis space H

Original: and several representative domain spaces D
Revised: and several representative domain spaces D

Original: XY
Revised: XY

Original: , what are the conditions to ensure the learnability of OOD detection?
Revised: , what are the conditions to ensure the learnability of OOD detection?

Original: Furthermore, if possible, we hope that these conditions are necessary and sufficient in some scenarios.
Revised: Furthermore, if possible, we hope that these conditions are necessary and sufficient in some scenarios.

Original: Therefore, compared to the agnostic PAC learnability of supervised learning, our theory doesn’t focus on the distribution-free case, but focuses on discovering essential conditions to guarantee the learnability of OOD detection in several representative and practical domain spaces D XY .
Revised: Therefore, compared to the agnostic PAC learnability of supervised learning, our theory doesn’t focus on the distribution-free case, but focuses on discovering essential conditions to guarantee the learnability of OOD detection in several representative and practical domain spaces D XY .

Original: By these essential conditions, we can know when OOD detection can be successful in real applications.
Revised: By these essential conditions, we can know when OOD detection can be successful in real applications.

Original:  Learning in Priori-unknown Spaces
Revised:  Learning in Priori-unknown Spaces

Original: We first investigate a special space, called prior-unknown space.
Revised: We first investigate a special space, called prior-unknown space.

Original: In such space, Definition 1 and
Revised: In such space, Definition 1 and

Original: Definition 2 are equivalent.
Revised: Definition 2 are equivalent.

Original: Furthermore, we also prove that if OOD detection is strongly learnable in a space D XY , then one can discover a larger domain space, which is prior-unknown, to ensure the learnability of OOD detection.
Revised: Furthermore, we also prove that if OOD detection is strongly learnable in a space D XY , then one can discover a larger domain space, which is prior-unknown, to ensure the learnability of OOD detection.

Original: These results imply that it is enough to consider our theory in the prior-unknown spaces.
Revised: These results imply that it is enough to consider our theory in the prior-unknown spaces.

Original: The prior-unknown space is introduced as follows:
Revised: The prior-unknown space is introduced as follows:

Original: Definition 3.
Revised: Definition 3.

Original: Given a domain space D XY , we say D XY is a priori-unknown space, if for any domain
Revised: Given a domain space D XY , we say D XY is a priori-unknown space, if for any domain

Original: D XY ∈ D XY and any α ∈ [0 , 1)
Revised: D XY ∈ D XY and any α ∈ [0 , 1)

Original: , we have D αXY :
Revised: , we have D αXY :

Original: = (1 − α )
Revised: = (1 − α )

Original: D
Revised: D

Original: X
Revised: X

Original: I Y
Revised: I Y

Original: I + αD X O Y O ∈ D XY .
Revised: I + αD X O Y O ∈ D XY .

Original: Theorem 1.
Revised: Theorem 1.

Original: Given domain spaces D XY and D ′ XY
Revised: Given domain spaces D XY and D ′ XY

Original: = { D αXY : ∀ D XY ∈ D XY , ∀ α ∈ [0 , 1) } , then
Revised: = { D αXY : ∀ D XY ∈ D XY , ∀ α ∈ [0 , 1) } , then

Original: 1) D ′ XY is a priori-unknown space and D XY ⊂ D ′ XY ; 2) if D XY is a priori-unknown space, then Definition 1 and Definition 2 are equivalent ; 3) OOD detection is strongly learnable in D XY if and only if OOD detection is learnable in D ′ XY .
Revised: 1) D ′ XY is a priori-unknown space and D XY ⊂ D ′ XY ; 2) if D XY is a priori-unknown space, then Definition 1 and Definition 2 are equivalent ; 3) OOD detection is strongly learnable in D XY if and only if OOD detection is learnable in D ′ XY .

Original: The second result of Theorem 1 bridges the learnability and strong learnability, which implies that if an algorithm A is consistent with respect to a prior-unknown space, then this algorithm A can address the imbalance issue between ID and OOD distributions, and the priori-unknown issue well.
Revised: The second result of Theorem 1 bridges the learnability and strong learnability, which implies that if an algorithm A is consistent with respect to a prior-unknown space, then this algorithm A can address the imbalance issue between ID and OOD distributions, and the priori-unknown issue well.

Original: Based on Theorem 1, we focus on our theory in the prior-unknown spaces.
Revised: Based on Theorem 1, we focus on our theory in the prior-unknown spaces.

Original: Furthermore, to demystify the learnability of OOD detection, we introduce five representative priori-unknown spaces:
Revised: Furthermore, to demystify the learnability of OOD detection, we introduce five representative priori-unknown spaces:

Original: • Single-distribution space D D XY XY .
Revised: • Single-distribution space D D XY XY .

Original: For a domain D XY , D D XY XY
Revised: For a domain D XY , D D XY XY

Original: [Equation]
Revised: [Equation]

Original: • Total space D all XY , which consists of all domains.
Revised: • Total space D all XY , which consists of all domains.

Original: • Separate space D sXY , which consists of all domains that satisfy the separate condition, that is for any D XY ∈ D s XY , supp D X O ∩ supp D
Revised: • Separate space D sXY , which consists of all domains that satisfy the separate condition, that is for any D XY ∈ D s XY , supp D X O ∩ supp D

Original: X
Revised: X

Original: I = ∅ , where supp means the support set.
Revised: I = ∅ , where supp means the support set.

Original: • Finite-ID-distribution space D FXY , which is a prior-unknown space satisfying that the number of distinct ID joint distributions D X
Revised: • Finite-ID-distribution space D FXY , which is a prior-unknown space satisfying that the number of distinct ID joint distributions D X

Original: I Y I in D FXY is finite, i.e. , |{ D X
Revised: I Y I in D FXY is finite, i.e. , |{ D X

Original: I Y
Revised: I Y

Original: I : ∀ D XY ∈ D FXY }| < + ∞ .
Revised: I : ∀ D XY ∈ D FXY }| < + ∞ .

Original: • Density-based space D µ,bXY , which is a prior-unknown space consisting of some domains satisfying that: for any D XY , there exists a density function f with 1 /b ≤ f ≤ b in supp µ and 0 .
Revised: • Density-based space D µ,bXY , which is a prior-unknown space consisting of some domains satisfying that: for any D XY , there exists a density function f with 1 /b ≤ f ≤ b in supp µ and 0 .

Original: 5 ∗ D X
Revised: 5 ∗ D X

Original: I +
Revised: I +

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: f d µ , where µ is a measure defined over X .
Revised: f d µ , where µ is a measure defined over X .

Original: Note that if µ is discrete, then D X is a discrete distribution; and if µ is the Lebesgue measure, then D X is a continuous distribution.
Revised: Note that if µ is discrete, then D X is a discrete distribution; and if µ is the Lebesgue measure, then D X is a continuous distribution.

Original: The above representative spaces widely exist in real applications.
Revised: The above representative spaces widely exist in real applications.

Original: For example, 1) if the images from different semantic labels with different styles are clearly different, then those images can form a distribution belonging to a separate space D sXY ; and 2) when designing an algorithm, we only have finite ID datasets, e.g. , CIFAR-10, MNIST, SVHN, and ImageNet, to build a model.
Revised: For example, 1) if the images from different semantic labels with different styles are clearly different, then those images can form a distribution belonging to a separate space D sXY ; and 2) when designing an algorithm, we only have finite ID datasets, e.g. , CIFAR-10, MNIST, SVHN, and ImageNet, to build a model.

Original: Then, finite-ID-distribution space D FXY can handle this real scenario.
Revised: Then, finite-ID-distribution space D FXY can handle this real scenario.

Original: Note that the single-distribution space is a special case of the finite-ID-distribution space.
Revised: Note that the single-distribution space is a special case of the finite-ID-distribution space.

Original: In this paper, we mainly discuss these five spaces.
Revised: In this paper, we mainly discuss these five spaces.

Original:  Impossibility Theorems for OOD Detection
Revised:  Impossibility Theorems for OOD Detection

Original: In this section, we first give a necessary condition for the learnability of OOD detection.
Revised: In this section, we first give a necessary condition for the learnability of OOD detection.

Original: Then, we show this necessary condition does not hold in the total space D all XY and the separate space D sXY .
Revised: Then, we show this necessary condition does not hold in the total space D all XY and the separate space D sXY .

Original: Necessary Condition.
Revised: Necessary Condition.

Original: We find a necessary condition for the learnability of OOD detection, i.e. , Condition 1, motivated by the experiments in Figure 1.
Revised: We find a necessary condition for the learnability of OOD detection, i.e. , Condition 1, motivated by the experiments in Figure 1.

Original: Details of Figure 1 can be found in Appendix C.2.
Revised: Details of Figure 1 can be found in Appendix C.2.

Original: Condition 1 (Linear Condition) .
Revised: Condition 1 (Linear Condition) .

Original: For any D XY ∈ D XY and any α ∈ [0 , 1) ,
Revised: For any D XY ∈ D XY and any α ∈ [0 , 1) ,

Original: [Equation]
Revised: [Equation]

Original: To reveal the importance of Condition 1, Theorem 2 shows that Condition 1 is a necessary and sufficient condition for the learnability of OOD detection if the D XY is the single-distribution space.
Revised: To reveal the importance of Condition 1, Theorem 2 shows that Condition 1 is a necessary and sufficient condition for the learnability of OOD detection if the D XY is the single-distribution space.

Original: Theorem 2.
Revised: Theorem 2.

Original: Given a hypothesis space H and a domain D XY , OOD detection is learnable in the single-distribution space D D XY XY for H if and only if linear condition (i.e., Condition 1) holds.
Revised: Given a hypothesis space H and a domain D XY , OOD detection is learnable in the single-distribution space D D XY XY for H if and only if linear condition (i.e., Condition 1) holds.

Original: Theorem 2 implies that Condition 1 is important for the learnability of OOD detection.
Revised: Theorem 2 implies that Condition 1 is important for the learnability of OOD detection.

Original: Due to the simplicity of single-distribution space, Theorem 2 implies that Condition 1 is the necessary condition for the learnability of OOD detection in the prior-unknown space, see Lemma 1 in Appendix F.
Revised: Due to the simplicity of single-distribution space, Theorem 2 implies that Condition 1 is the necessary condition for the learnability of OOD detection in the prior-unknown space, see Lemma 1 in Appendix F.

Original: Impossibility Theorems.
Revised: Impossibility Theorems.

Original: Here, we first study whether Condition 1 holds in the total space D all XY .
Revised: Here, we first study whether Condition 1 holds in the total space D all XY .

Original: If
Revised: If

Original: Condition 1 does not hold, then OOD detection is not learnable.
Revised: Condition 1 does not hold, then OOD detection is not learnable.

Original: Theorem 3 shows that Condition 1 is not always satisfied, especially, when there is an overlap between the ID and OOD distributions:
Revised: Theorem 3 shows that Condition 1 is not always satisfied, especially, when there is an overlap between the ID and OOD distributions:

Original: Definition 4 (Overlap Between ID and OOD) .
Revised: Definition 4 (Overlap Between ID and OOD) .

Original: We say a domain D XY has overlap between ID and
Revised: We say a domain D XY has overlap between ID and

Original: OOD distributions, if there is a σ -finite measure ˜ µ such that D X is absolutely continuous with respect to ˜ µ , and ˜ µ ( A overlap ) > 0 , where A overlap = { x ∈ X : f I ( x ) > 0 and f O ( x ) > 0 } .
Revised: OOD distributions, if there is a σ -finite measure ˜ µ such that D X is absolutely continuous with respect to ˜ µ , and ˜ µ ( A overlap ) > 0 , where A overlap = { x ∈ X : f I ( x ) > 0 and f O ( x ) > 0 } .

Original: Here f I and f O are the representers of D X I and D X O in Radon–Nikodym Theorem [36],
Revised: Here f I and f O are the representers of D X I and D X O in Radon–Nikodym Theorem [36],

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: Given a hypothesis space H and a prior-unknown space D XY , if there is D XY ∈ D XY , which has overlap between ID and OOD, and inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 , then
Revised: Given a hypothesis space H and a prior-unknown space D XY , if there is D XY ∈ D XY , which has overlap between ID and OOD, and inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 , then

Original: Condition 1 does not hold.
Revised: Condition 1 does not hold.

Original: Therefore, OOD detection is not learnable in D XY for H .
Revised: Therefore, OOD detection is not learnable in D XY for H .

Original: Theorem 3 clearly shows that under proper conditions, Condition 1 does not hold, if there exists a domain whose ID and OOD distributions have overlap.
Revised: Theorem 3 clearly shows that under proper conditions, Condition 1 does not hold, if there exists a domain whose ID and OOD distributions have overlap.

Original: By Theorem 3, we can obtain that the OOD detection is not learnable in the total space D all XY for any non-trivial hypothesis space H .
Revised: By Theorem 3, we can obtain that the OOD detection is not learnable in the total space D all XY for any non-trivial hypothesis space H .

Original: Theorem 4 (Impossibility Theorem for Total Space) .
Revised: Theorem 4 (Impossibility Theorem for Total Space) .

Original: OOD detection is not learnable in the total space D all XY for H , if | ϕ ◦ H| > 1 , where ϕ maps ID labels to 1 and maps OOD labels to 2 .
Revised: OOD detection is not learnable in the total space D all XY for H , if | ϕ ◦ H| > 1 , where ϕ maps ID labels to 1 and maps OOD labels to 2 .

Original: Since the overlaps between ID and OOD distributions may cause that Condition 1 does not hold, we then consider studying the learnability of OOD detection in the separate space D sXY , where there are no overlaps between the ID and OOD distributions.
Revised: Since the overlaps between ID and OOD distributions may cause that Condition 1 does not hold, we then consider studying the learnability of OOD detection in the separate space D sXY , where there are no overlaps between the ID and OOD distributions.

Original: However, Theorem 5 shows that even if we consider the separate space, the OOD detection is still not learnable in some scenarios.
Revised: However, Theorem 5 shows that even if we consider the separate space, the OOD detection is still not learnable in some scenarios.

Original: Before introducing the impossibility theorem for separate space, i.e. , Theorem 5, we need a mild assumption:
Revised: Before introducing the impossibility theorem for separate space, i.e. , Theorem 5, we need a mild assumption:

Original: Assumption 1 (Separate Space for OOD) .
Revised: Assumption 1 (Separate Space for OOD) .

Original: A hypothesis space H is separate for OOD data, if for each data point x ∈ X , there exists at least one hypothesis function h x ∈ H such that h x ( x ) =
Revised: A hypothesis space H is separate for OOD data, if for each data point x ∈ X , there exists at least one hypothesis function h x ∈ H such that h x ( x ) =

Original: K + 1 .
Revised: K + 1 .

Original: Assumption 1 means that every data point x has the possibility to be detected as OOD data.
Revised: Assumption 1 means that every data point x has the possibility to be detected as OOD data.

Original: Assumption 1 is mild and can be satisfied by many hypothesis spaces, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel space.
Revised: Assumption 1 is mild and can be satisfied by many hypothesis spaces, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel space.

Original: Next, we use Vapnik–Chervonenkis (VC) dimension [22] to measure the size of hypothesis space, and study the learnability of OOD detection in D sXY based on the VC dimension.
Revised: Next, we use Vapnik–Chervonenkis (VC) dimension [22] to measure the size of hypothesis space, and study the learnability of OOD detection in D sXY based on the VC dimension.

Original: Theorem 5 (Impossibility Theorem for Separate Space) .
Revised: Theorem 5 (Impossibility Theorem for Separate Space) .

Original: If Assumption 1 holds, VCdim( ϕ ◦ H ) <
Revised: If Assumption 1 holds, VCdim( ϕ ◦ H ) <

Original: + ∞ and sup h ∈H |{ x ∈ X : h ( x )
Revised: + ∞ and sup h ∈H |{ x ∈ X : h ( x )

Original: ∈ Y}| =
Revised: ∈ Y}| =

Original: + ∞ , then OOD detection is not learnable in separate space D sXY for H , where ϕ maps ID labels to 1 and maps OOD labels to 2 .
Revised: + ∞ , then OOD detection is not learnable in separate space D sXY for H , where ϕ maps ID labels to 1 and maps OOD labels to 2 .

Original: The finite VC dimension normally implies the learnability of supervised learning.
Revised: The finite VC dimension normally implies the learnability of supervised learning.

Original: However, in our results, the finite VC dimension cannot guarantee the learnability of OOD detection in the separate space, which reveals the difficulty of the OOD detection.
Revised: However, in our results, the finite VC dimension cannot guarantee the learnability of OOD detection in the separate space, which reveals the difficulty of the OOD detection.

Original: Although the above impossibility theorems are frustrating, there is still room to discuss the conditions in Theorem 5, and to find out the proper conditions for ensuring the learnability of OOD detection in the separate space (see Sections 5 and 6).
Revised: Although the above impossibility theorems are frustrating, there is still room to discuss the conditions in Theorem 5, and to find out the proper conditions for ensuring the learnability of OOD detection in the separate space (see Sections 5 and 6).

Original:  When OOD Detection Can Be Successful
Revised:  When OOD Detection Can Be Successful

Original: Here, we discuss when the OOD detection can be learnable in the separate space D sXY , finite-IDdistribution space D FXY and density-based space D µ,bXY .
Revised: Here, we discuss when the OOD detection can be learnable in the separate space D sXY , finite-IDdistribution space D FXY and density-based space D µ,bXY .

Original: We first study the separate space D sXY .
Revised: We first study the separate space D sXY .

Original: OOD Detection in the Separate Space.
Revised: OOD Detection in the Separate Space.

Original: Theorem 5 has indicated that VCdim( ϕ ◦ H ) =
Revised: Theorem 5 has indicated that VCdim( ϕ ◦ H ) =

Original: + ∞ or sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ is necessary to ensure the learnability of OOD detection in D s
Revised: + ∞ or sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ is necessary to ensure the learnability of OOD detection in D s

Original: XY if Assumption 1 holds.
Revised: XY if Assumption 1 holds.

Original: However, generally, hypothesis spaces generated by feed-forward neural networks with proper activation functions have finite VC dimension [37, 38].
Revised: However, generally, hypothesis spaces generated by feed-forward neural networks with proper activation functions have finite VC dimension [37, 38].

Original: Therefore, we study the learnability of OOD detection in the case that |X| < + ∞ , which implies that sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ .
Revised: Therefore, we study the learnability of OOD detection in the case that |X| < + ∞ , which implies that sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ .

Original: Additionally, Theorem 10 also implies that |X| <
Revised: Additionally, Theorem 10 also implies that |X| <

Original: + ∞ is the necessary and sufficient condition for the learnability of OOD detection in separate space, when the hypothesis space is generated by FCNN.
Revised: + ∞ is the necessary and sufficient condition for the learnability of OOD detection in separate space, when the hypothesis space is generated by FCNN.

Original: Hence, |X| < + ∞ may be necessary in the space D sXY .
Revised: Hence, |X| < + ∞ may be necessary in the space D sXY .

Original: For simplicity, we first discuss the case that K
Revised: For simplicity, we first discuss the case that K

Original: = 1 , i.e. , the one-class novelty detection.
Revised: = 1 , i.e. , the one-class novelty detection.

Original: We show the necessary and sufficient condition for the learnability of OOD detection in D sXY , when |X| < + ∞ .
Revised: We show the necessary and sufficient condition for the learnability of OOD detection in D sXY , when |X| < + ∞ .

Original: Theorem 6.
Revised: Theorem 6.

Original: Let K = 1 and |X| < + ∞ .
Revised: Let K = 1 and |X| < + ∞ .

Original: Suppose that Assumption 1 holds and the constant function h in := 1 ∈ H .
Revised: Suppose that Assumption 1 holds and the constant function h in := 1 ∈ H .

Original: Then OOD detection is learnable in D sXY for H if and only if H all − { h out } ⊂ H , where H all is the hypothesis space consisting of all hypothesis functions, and h out is a constant function that h out := 2 , here 1 represents ID data and 2 represents OOD data.
Revised: Then OOD detection is learnable in D sXY for H if and only if H all − { h out } ⊂ H , where H all is the hypothesis space consisting of all hypothesis functions, and h out is a constant function that h out := 2 , here 1 represents ID data and 2 represents OOD data.

Original: The condition h in ∈ H presented in Theorem 6 is mild.
Revised: The condition h in ∈ H presented in Theorem 6 is mild.

Original: Many practical hypothesis spaces satisfy this condition, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel-based hypothesis space.
Revised: Many practical hypothesis spaces satisfy this condition, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel-based hypothesis space.

Original: Theorem 6 implies that if K = 1 and OOD detection is learnable in D sXY for H , then the hypothesis space H should contain almost all hypothesis functions, implying that if the OOD detection can be learnable in the distribution-agnostic case, then a large-capacity model is necessary.
Revised: Theorem 6 implies that if K = 1 and OOD detection is learnable in D sXY for H , then the hypothesis space H should contain almost all hypothesis functions, implying that if the OOD detection can be learnable in the distribution-agnostic case, then a large-capacity model is necessary.

Original: Next, we extend Theorem 6 to a general case, i.e. , K > 1 .
Revised: Next, we extend Theorem 6 to a general case, i.e. , K > 1 .

Original: When K > 1 , we will first use a binary classifier h b to classify the ID and OOD data.
Revised: When K > 1 , we will first use a binary classifier h b to classify the ID and OOD data.

Original: Then, for the ID data identified by h b , an ID hypothesis function h in will be used to classify them into corresponding ID classes.
Revised: Then, for the ID data identified by h b , an ID hypothesis function h in will be used to classify them into corresponding ID classes.

Original: We state this strategy as follows: given a hypothesis space H in for ID distribution and a binary classification hypothesis space
Revised: We state this strategy as follows: given a hypothesis space H in for ID distribution and a binary classification hypothesis space

Original: H b introduced in Section 2, we use H in and H b to construct an OOD detection’s hypothesis space H , which consists of all hypothesis functions h satisfying the following condition: there exist h in ∈ H in and h b ∈ H b such that for any x ∈ X ,
Revised: H b introduced in Section 2, we use H in and H b to construct an OOD detection’s hypothesis space H , which consists of all hypothesis functions h satisfying the following condition: there exist h in ∈ H in and h b ∈ H b such that for any x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: We use H in • H b to represent a hypothesis space consisting of all h defined in Eq.
Revised: We use H in • H b to represent a hypothesis space consisting of all h defined in Eq.

Original: In addition, we also need an additional condition for the loss function ℓ .
Revised: In addition, we also need an additional condition for the loss function ℓ .

Original: This condition is shown as follows:
Revised: This condition is shown as follows:

Original: Condition 2.
Revised: Condition 2.

Original: ℓ ( y 2 , y 1 ) ≤ ℓ ( K + 1 , y 1 ) , for any in-distribution labels y 1 and y 2 ∈ Y .
Revised: ℓ ( y 2 , y 1 ) ≤ ℓ ( K + 1 , y 1 ) , for any in-distribution labels y 1 and y 2 ∈ Y .

Original: Theorem 7.
Revised: Theorem 7.

Original: Let |X| < + ∞ and H
Revised: Let |X| < + ∞ and H

Original: = H in • H b .
Revised: = H in • H b .

Original: If H all − { h out } ⊂ H b and Condition 2 holds, then OOD detection is learnable in D sXY for H , where H all and h out are defined in Theorem 6.
Revised: If H all − { h out } ⊂ H b and Condition 2 holds, then OOD detection is learnable in D sXY for H , where H all and h out are defined in Theorem 6.

Original: OOD Detection in the Finite-ID-Distribution Space.
Revised: OOD Detection in the Finite-ID-Distribution Space.

Original: Since researchers can only collect finite ID datasets as the training data in the process of algorithm design, it is worthy to study the learnability of
Revised: Since researchers can only collect finite ID datasets as the training data in the process of algorithm design, it is worthy to study the learnability of

Original: OOD detection in the finite-ID-distribution space D FXY .
Revised: OOD detection in the finite-ID-distribution space D FXY .

Original: We first show two necessary concepts below.
Revised: We first show two necessary concepts below.

Original: Definition 5 (ID Consistency) .
Revised: Definition 5 (ID Consistency) .

Original: Given a domain space D XY , we say any two domains D XY ∈ D XY and D ′ XY ∈ D XY are ID consistency, if D X
Revised: Given a domain space D XY , we say any two domains D XY ∈ D XY and D ′ XY ∈ D XY are ID consistency, if D X

Original: I Y I
Revised: I Y I

Original: = D ′
Revised: = D ′

Original: X
Revised: X

Original: I Y
Revised: I Y

Original: I .
Revised: I .

Original: We use the notation ∼ to represent the ID consistency, i.e., D XY ∼ D ′ XY if and only if D XY and D ′ XY are ID consistency.
Revised: We use the notation ∼ to represent the ID consistency, i.e., D XY ∼ D ′ XY if and only if D XY and D ′ XY are ID consistency.

Original: It is easy to check that the ID consistency ∼ is an equivalence relation.
Revised: It is easy to check that the ID consistency ∼ is an equivalence relation.

Original: Therefore, we define the set [ D XY ] := { D ′ XY ∈ D XY : D XY ∼ D ′ XY } as the equivalence class with respect to space D XY .
Revised: Therefore, we define the set [ D XY ] := { D ′ XY ∈ D XY : D XY ∼ D ′ XY } as the equivalence class with respect to space D XY .

Original: Condition 3 (Compatibility) .
Revised: Condition 3 (Compatibility) .

Original: For any equivalence class [ D ′ XY ] with respect to D XY and any ϵ > 0 , there exists a hypothesis function h ϵ ∈ H such that for any domain D XY ∈
Revised: For any equivalence class [ D ′ XY ] with respect to D XY and any ϵ > 0 , there exists a hypothesis function h ϵ ∈ H such that for any domain D XY ∈

Original: [ D ′ XY ] ,
Revised: [ D ′ XY ] ,

Original: [Equation]
Revised: [Equation]

Original: In Appendix F, Lemma 2 has implied that Condition 3 is a general version of Condition 1.
Revised: In Appendix F, Lemma 2 has implied that Condition 3 is a general version of Condition 1.

Original: Next,
Revised: Next,

Original: Theorem 8 indicates that Condition 3 is the necessary and sufficient condition in the space D FXY .
Revised: Theorem 8 indicates that Condition 3 is the necessary and sufficient condition in the space D FXY .

Original: Theorem 8.
Revised: Theorem 8.

Original: Suppose that X is a bounded set.
Revised: Suppose that X is a bounded set.

Original: OOD detection is learnable in the finite-IDdistribution space D FXY for H if and only if the compatibility condition (i.e., Condition 3) holds.
Revised: OOD detection is learnable in the finite-IDdistribution space D FXY for H if and only if the compatibility condition (i.e., Condition 3) holds.

Original: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .
Revised: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .

Original: Theorem 8 shows that, in the process of algorithm design, OOD detection cannot be successful without the compatibility condition.
Revised: Theorem 8 shows that, in the process of algorithm design, OOD detection cannot be successful without the compatibility condition.

Original: Theorem 8 also implies that Condition 3 is essential for the learnability of OOD detection.
Revised: Theorem 8 also implies that Condition 3 is essential for the learnability of OOD detection.

Original: This motivates us to study whether OOD detection can be successful in more general spaces ( e.g. , the density-based space), when the compatibility condition holds.
Revised: This motivates us to study whether OOD detection can be successful in more general spaces ( e.g. , the density-based space), when the compatibility condition holds.

Original: OOD Detection in the Density-based Space.
Revised: OOD Detection in the Density-based Space.

Original: To ensure that Condition 3 holds, we consider a basic assumption in learning theory— Realizability Assumption (see Appendix D.2), i.e. , for any
Revised: To ensure that Condition 3 holds, we consider a basic assumption in learning theory— Realizability Assumption (see Appendix D.2), i.e. , for any

Original: D XY ∈ D XY , there exists h ∗ ∈ H such that R D ( h ∗ )
Revised: D XY ∈ D XY , there exists h ∗ ∈ H such that R D ( h ∗ )

Original: = 0 .
Revised: = 0 .

Original: We discover that in the density-based space D µ,bXY , Realizability Assumption can conclude the compatibility condition ( i.e. , Condition 3).
Revised: We discover that in the density-based space D µ,bXY , Realizability Assumption can conclude the compatibility condition ( i.e. , Condition 3).

Original: Based on this observation, we can prove the following theorem:
Revised: Based on this observation, we can prove the following theorem:

Original: Theorem 9.
Revised: Theorem 9.

Original: Given a density-based space D µ,bXY , if µ ( X ) < + ∞ , the Realizability Assumption holds, then when H has finite Natarajan dimension [21], OOD detection is learnable in D µ,bXY for
Revised: Given a density-based space D µ,bXY , if µ ( X ) < + ∞ , the Realizability Assumption holds, then when H has finite Natarajan dimension [21], OOD detection is learnable in D µ,bXY for

Original: H .
Revised: H .

Original: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .
Revised: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .

Original: To further investigate the importance and necessary of Realizability Assumption, Theorem 11 has indicated that in some practical scenarios, Realizability Assumption is the necessary and sufficient condition for the learnability of OOD detection in the density-based space.
Revised: To further investigate the importance and necessary of Realizability Assumption, Theorem 11 has indicated that in some practical scenarios, Realizability Assumption is the necessary and sufficient condition for the learnability of OOD detection in the density-based space.

Original: Therefore, Realizability
Revised: Therefore, Realizability

Original: Assumption may be indispensable for the learnability of OOD detection in some practical scenarios.
Revised: Assumption may be indispensable for the learnability of OOD detection in some practical scenarios.

Original:  Connecting Theory to Practice
Revised:  Connecting Theory to Practice

Original: In Section 5, we have shown the successful scenarios where OOD detection problem can be addressed in theory.
Revised: In Section 5, we have shown the successful scenarios where OOD detection problem can be addressed in theory.

Original: In this section, we will discuss how the proposed theory is applied to two representative hypothesis spaces—neural-network-based hypothesis spaces and score-based hypothesis spaces.
Revised: In this section, we will discuss how the proposed theory is applied to two representative hypothesis spaces—neural-network-based hypothesis spaces and score-based hypothesis spaces.

Original: Fully-connected Neural Networks.
Revised: Fully-connected Neural Networks.

Original: Given a sequence q
Revised: Given a sequence q

Original: =
Revised: =

Original: ( l 1 , l 2 , ..., l g ) , where l i and g are positive integers and g > 2 , we use g to represent the depth of neural network and use l i to represent the width of the i -th layer.
Revised: ( l 1 , l 2 , ..., l g ) , where l i and g are positive integers and g > 2 , we use g to represent the depth of neural network and use l i to represent the width of the i -th layer.

Original: After the activation function σ is selected 4 , we can obtain the architecture of FCNN according to the sequence q .
Revised: After the activation function σ is selected 4 , we can obtain the architecture of FCNN according to the sequence q .

Original: Let f w , b be the function generated by FCNN with weights w and bias b .
Revised: Let f w , b be the function generated by FCNN with weights w and bias b .

Original: An FCNN-based scoring function space is defined as: F σ q := { f w , b : ∀ weights w , ∀ bias b } .
Revised: An FCNN-based scoring function space is defined as: F σ q := { f w , b : ∀ weights w , ∀ bias b } .

Original: In addition, for simplicity, given any two sequences q = ( l 1 , ..., l g ) and q ′ =
Revised: In addition, for simplicity, given any two sequences q = ( l 1 , ..., l g ) and q ′ =

Original: ( l ′ 1 , ..., l ′ g ′ ) , we use the notation q ≲ q ′ to represent the following equations and inequalities:
Revised: ( l ′ 1 , ..., l ′ g ′ ) , we use the notation q ≲ q ′ to represent the following equations and inequalities:

Original: 1) g ≤ g ′ , l 1 = l ′ 1 , l g = l ′ g ′ ; 2) l i ≤ l ′
Revised: 1) g ≤ g ′ , l 1 = l ′ 1 , l g = l ′ g ′ ; 2) l i ≤ l ′

Original: i , ∀ i = 1 , ..., g − 1; and 3) l g − 1 ≤ l ′
Revised: i , ∀ i = 1 , ..., g − 1; and 3) l g − 1 ≤ l ′

Original: i , ∀
Revised: i , ∀

Original: i = g, ..., g ′ − 1 .
Revised: i = g, ..., g ′ − 1 .

Original: In Appendix L, Lemma 10 shows q ≲ q ′ ⇒ F σ q ⊂ F σ q ′ .
Revised: In Appendix L, Lemma 10 shows q ≲ q ′ ⇒ F σ q ⊂ F σ q ′ .

Original: We use ≲ to compare the sizes of FCNNs.
Revised: We use ≲ to compare the sizes of FCNNs.

Original: FCNN-based Hypothesis Space.
Revised: FCNN-based Hypothesis Space.

Original: Let l g
Revised: Let l g

Original: =
Revised: =

Original: K
Revised: K

Original: + 1 .
Revised: + 1 .

Original: The FCNN-based scoring function space F σ q can induce an FCNN-based hypothesis space.
Revised: The FCNN-based scoring function space F σ q can induce an FCNN-based hypothesis space.

Original: For any f w , b ∈ F σ q , the induced hypothesis function is:
Revised: For any f w , b ∈ F σ q , the induced hypothesis function is:

Original: [Equation]
Revised: [Equation]

Original: k w , b , where f k w , b is the k -th coordinate of f w , b .
Revised: k w , b , where f k w , b is the k -th coordinate of f w , b .

Original: Then, the FCNN-based hypothesis space is defined as H σ q := { h w , b : ∀ weights w , ∀ bias b } .
Revised: Then, the FCNN-based hypothesis space is defined as H σ q := { h w , b : ∀ weights w , ∀ bias b } .

Original: Score-based Hypothesis Space.
Revised: Score-based Hypothesis Space.

Original: Many OOD detection algorithms detect OOD data by using a score-based strategy.
Revised: Many OOD detection algorithms detect OOD data by using a score-based strategy.

Original: That is, given a threshold λ , a scoring function space F l ⊂ { f : X → R l } and a scoring function E : F l → R , then x is regarded as ID data if and only if E ( f ( x )) ≥ λ .
Revised: That is, given a threshold λ , a scoring function space F l ⊂ { f : X → R l } and a scoring function E : F l → R , then x is regarded as ID data if and only if E ( f ( x )) ≥ λ .

Original: We introduce several representative scoring functions E as follows: for any f = [ f 1 , ..., f l ] ⊤ ∈ F l ,
Revised: We introduce several representative scoring functions E as follows: for any f = [ f 1 , ..., f l ] ⊤ ∈ F l ,

Original: • softmax-based function [7] and temperature-scaled function [
Revised: • softmax-based function [7] and temperature-scaled function [

Original: : λ ∈ (
Revised: : λ ∈ (

Original: 1 l and T
Revised: 1 l and T

Original: >
Revised: >

Original: [Equation]
Revised: [Equation]

Original: • energy-based function [23]: λ ∈ (0 , + ∞ ) and T > 0 ,
Revised: • energy-based function [23]: λ ∈ (0 , + ∞ ) and T > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Using E , λ and f ∈ F σ q , we have a classifier:
Revised: Using E , λ and f ∈ F σ q , we have a classifier:

Original: h λ f ,E ( x ) = 1 , if E ( f ( x )) ≥ λ ; otherwise, h λ f ,E ( x ) = 2 , where 1 represents the ID data and 2 represents the OOD data.
Revised: h λ f ,E ( x ) = 1 , if E ( f ( x )) ≥ λ ; otherwise, h λ f ,E ( x ) = 2 , where 1 represents the ID data and 2 represents the OOD data.

Original: Hence, a binary classification hypothesis space H b , which consists of all h λ f ,E , is generated.
Revised: Hence, a binary classification hypothesis space H b , which consists of all h λ f ,E , is generated.

Original: We define H σ,λ q ,E :=
Revised: We define H σ,λ q ,E :=

Original: { h λ f ,E : ∀ f ∈ F σ q } .
Revised: { h λ f ,E : ∀ f ∈ F σ q } .

Original: Learnability of OOD Detection in Different Hypothesis Spaces.
Revised: Learnability of OOD Detection in Different Hypothesis Spaces.

Original: Next, we present applications of our theory regarding the above two practical and important hypothesis spaces H σ q and H σ,λ q ,E .
Revised: Next, we present applications of our theory regarding the above two practical and important hypothesis spaces H σ q and H σ,λ q ,E .

Original: Theorem 10.
Revised: Theorem 10.

Original: Suppose that Condition 2 holds and the hypothesis space H is FCNN-based or scorebased
Revised: Suppose that Condition 2 holds and the hypothesis space H is FCNN-based or scorebased

Original: , i.e., H = H σ q or H =
Revised: , i.e., H = H σ q or H =

Original: H in • H b , where H in is an ID hypothesis space, H b
Revised: H in • H b , where H in is an ID hypothesis space, H b

Original: = H σ,λ q ,E and
Revised: = H σ,λ q ,E and

Original: H = H in • H b is introduced below Eq.
Revised: H = H in • H b is introduced below Eq.

Original: (4) , here E is introduced in Eqs.
Revised: (4) , here E is introduced in Eqs.

Original: (5) or (6) .
Revised: (5) or (6) .

Original: Then
Revised: Then

Original: There is a sequence q =
Revised: There is a sequence q =

Original: ( l 1 , ..., l g )
Revised: ( l 1 , ..., l g )

Original: such that OOD detection is learnable in the separate space D sXY for H if and only if |X| < + ∞ .
Revised: such that OOD detection is learnable in the separate space D sXY for H if and only if |X| < + ∞ .

Original: Furthermore, if |X| < + ∞ , then there exists a sequence q =
Revised: Furthermore, if |X| < + ∞ , then there exists a sequence q =

Original: ( l 1 , ..., l g )
Revised: ( l 1 , ..., l g )

Original: such that for any sequence q ′ satisfying that q ≲ q ′ , OOD detection is learnable in D sXY for H .
Revised: such that for any sequence q ′ satisfying that q ≲ q ′ , OOD detection is learnable in D sXY for H .

Original: Theorem 10 states that 1) when the hypothesis space is FCNN-based or score-based, the finite feature space is the necessary and sufficient condition for the learnability of OOD detection in the separate space; and 2) a larger architecture of FCNN has a greater probability to achieve the learnability of
Revised: Theorem 10 states that 1) when the hypothesis space is FCNN-based or score-based, the finite feature space is the necessary and sufficient condition for the learnability of OOD detection in the separate space; and 2) a larger architecture of FCNN has a greater probability to achieve the learnability of

Original: OOD detection in the separate space.
Revised: OOD detection in the separate space.

Original: Note that when we select Eqs.
Revised: Note that when we select Eqs.

Original: (5) or (6) as the scoring function
Revised: (5) or (6) as the scoring function

Original: E , Theorem 10 also shows that the selected scoring functions E can guarantee the learnability of
Revised: E , Theorem 10 also shows that the selected scoring functions E can guarantee the learnability of

Original: OOD detection, which is a theoretical support for the representative works [8, 23, 7].
Revised: OOD detection, which is a theoretical support for the representative works [8, 23, 7].

Original: Furthermore,
Revised: Furthermore,

Original: Theorem 11 also offers theoretical supports for these works in the density-based space, when K = 1 .
Revised: Theorem 11 also offers theoretical supports for these works in the density-based space, when K = 1 .

Original: Theorem 11.
Revised: Theorem 11.

Original: Suppose that each domain D XY in D µ,bXY is attainable, i.e., arg min
Revised: Suppose that each domain D XY in D µ,bXY is attainable, i.e., arg min

Original: h ∈H R D ( h ) ̸
Revised: h ∈H R D ( h ) ̸

Original: = ∅ (the finite discrete domains satisfy this).
Revised: = ∅ (the finite discrete domains satisfy this).

Original: Let K = 1 and the hypothesis space H be score-based (
Revised: Let K = 1 and the hypothesis space H be score-based (

Original: H = H σ,λ q ,E , where E is in Eqs.
Revised: H = H σ,λ q ,E , where E is in Eqs.

Original: (5) or (6) ) or FCNN-based ( H = H σ q ) .
Revised: (5) or (6) ) or FCNN-based ( H = H σ q ) .

Original: If µ ( X ) < + ∞ , then the following four conditions are equivalent :
Revised: If µ ( X ) < + ∞ , then the following four conditions are equivalent :

Original: [Equation]
Revised: [Equation]

Original: Theorem 11 still holds if the function space F σ q is generated by Convolutional Neural Network.
Revised: Theorem 11 still holds if the function space F σ q is generated by Convolutional Neural Network.

Original: Overlap and Benefits of Multi-class Case.
Revised: Overlap and Benefits of Multi-class Case.

Original: We investigate when the hypothesis space is FCNN-based or score-based, what will happen if there exists an overlap between the ID and OOD distributions?
Revised: We investigate when the hypothesis space is FCNN-based or score-based, what will happen if there exists an overlap between the ID and OOD distributions?

Original: Theorem 12.
Revised: Theorem 12.

Original: Let K = 1 and the hypothesis space H be score-based ( H = H σ,λ q ,E , where E is in
Revised: Let K = 1 and the hypothesis space H be score-based ( H = H σ,λ q ,E , where E is in

Original: Eqs.
Revised: Eqs.

Original: (5) or (6) ) or FCNN-based ( H = H σ q ) .
Revised: (5) or (6) ) or FCNN-based ( H = H σ q ) .

Original: Given a prior-unknown space D XY , if there exists a domain D XY ∈ D XY , which has an overlap between ID and OOD distributions (see Definition 4), then OOD detection is not learnable in the domain space D XY for H .
Revised: Given a prior-unknown space D XY , if there exists a domain D XY ∈ D XY , which has an overlap between ID and OOD distributions (see Definition 4), then OOD detection is not learnable in the domain space D XY for H .

Original: When K = 1 and the hypothesis space is FCNN-based or score-based, Theorem 12 shows that overlap between ID and OOD distributions is the sufficient condition for the unlearnability of OOD detection.
Revised: When K = 1 and the hypothesis space is FCNN-based or score-based, Theorem 12 shows that overlap between ID and OOD distributions is the sufficient condition for the unlearnability of OOD detection.

Original: Theorem 12 takes roots in the conditions inf h ∈H R in D ( h )
Revised: Theorem 12 takes roots in the conditions inf h ∈H R in D ( h )

Original: = 

Revised: =

Original: and inf h ∈H R out D ( h ) = 0 .
Revised: and inf h ∈H R out D ( h ) = 0 .

Original: However, when K > 1 , we can ensure inf h ∈H R in D ( h ) > 

Revised: However, when K > 1 , we can ensure inf h ∈H R in D ( h ) >

Original: if ID distribution D X
Revised: if ID distribution D X

Original: I Y
Revised: I Y

Original: I has overlap between ID classes.
Revised: I has overlap between ID classes.

Original: By this observation, we conjecture that when K > 1 , OOD detection is learnable in some special cases where overlap exists, even if the hypothesis space is FCNN-based or score-based.
Revised: By this observation, we conjecture that when K > 1 , OOD detection is learnable in some special cases where overlap exists, even if the hypothesis space is FCNN-based or score-based.

Original:  Discussion
Revised:  Discussion

Original: Understanding Far-OOD Detection.
Revised: Understanding Far-OOD Detection.

Original: Many existing works [7, 39] study the far-OOD detection issue.
Revised: Many existing works [7, 39] study the far-OOD detection issue.

Original: Existing benchmarks include 1) MNIST [40] as ID dataset, and Texture [41], CIFAR- 10 [42] or Place 365 [43] as OOD datasets; and 2)
Revised: Existing benchmarks include 1) MNIST [40] as ID dataset, and Texture [41], CIFAR- 10 [42] or Place 365 [43] as OOD datasets; and 2)

Original: CIFAR-
Revised: CIFAR-

Original: 10 [42] as ID dataset, and MNIST [40], or FashionMNIST [43] as OOD datasets.
Revised: 10 [42] as ID dataset, and MNIST [40], or FashionMNIST [43] as OOD datasets.

Original: In far-OOD case, we find that the ID and OOD datasets have different semantic labels and different styles.
Revised: In far-OOD case, we find that the ID and OOD datasets have different semantic labels and different styles.

Original: From the theoretical view, we can define far-OOD detection tasks as follows: for τ > 0 , a domain space D XY is τ -far-OOD, if for any domain D XY ∈ D XY , dist(supp D X O , supp D X I ) > τ.
Revised: From the theoretical view, we can define far-OOD detection tasks as follows: for τ > 0 , a domain space D XY is τ -far-OOD, if for any domain D XY ∈ D XY , dist(supp D X O , supp D X I ) > τ.

Original: Theorems 7, 8 and 10 imply that under appropriate hypothesis space, τ -far-OOD detection is learnable.
Revised: Theorems 7, 8 and 10 imply that under appropriate hypothesis space, τ -far-OOD detection is learnable.

Original: In Theorem 7, the condition |X| <
Revised: In Theorem 7, the condition |X| <

Original: + ∞ is necessary for the separate space.
Revised: + ∞ is necessary for the separate space.

Original: However, one can prove that in the far-OOD case, when H in is agnostic PAC learnable for ID distribution, the results in
Revised: However, one can prove that in the far-OOD case, when H in is agnostic PAC learnable for ID distribution, the results in

Original: Theorem 7 still holds, if the condition |X| <
Revised: Theorem 7 still holds, if the condition |X| <

Original: + ∞ is replaced by a weaker condition that X is compact.
Revised: + ∞ is replaced by a weaker condition that X is compact.

Original: In addition, it is notable that when H in is agnostic PAC learnable for ID distribution and X is compact, the KNN-based OOD detection algorithm [44] is consistent in the τ -far-OOD case.
Revised: In addition, it is notable that when H in is agnostic PAC learnable for ID distribution and X is compact, the KNN-based OOD detection algorithm [44] is consistent in the τ -far-OOD case.

Original: Understanding Near-OOD Detection.
Revised: Understanding Near-OOD Detection.

Original: When the ID and OOD datasets have similar semantics or styles, OOD detection tasks become more challenging.
Revised: When the ID and OOD datasets have similar semantics or styles, OOD detection tasks become more challenging.

Original: [45, 46] consider this issue and name it near-OOD detection.
Revised: [45, 46] consider this issue and name it near-OOD detection.

Original: Existing benchmarks include 1) MNIST [40] as ID dataset, and Fashion-MNIST [43] or Not-MNIST [47] as OOD datasets; and 2) CIFAR-
Revised: Existing benchmarks include 1) MNIST [40] as ID dataset, and Fashion-MNIST [43] or Not-MNIST [47] as OOD datasets; and 2) CIFAR-

Original: 10 [42] as ID dataset, and CIFAR- 100 [48] as OOD dataset.
Revised: 10 [42] as ID dataset, and CIFAR- 100 [48] as OOD dataset.

Original: From the theoretical view, some near-OOD tasks may imply the overlap condition, i.e. Definition 4.
Revised: From the theoretical view, some near-OOD tasks may imply the overlap condition, i.e. Definition 4.

Original: Therefore, Theorems 3 and 12 imply that near-OOD detection may be not learnable.
Revised: Therefore, Theorems 3 and 12 imply that near-OOD detection may be not learnable.

Original: Developing a theory to understand the feasibility of near-OOD detection is still an open question .
Revised: Developing a theory to understand the feasibility of near-OOD detection is still an open question .

Original: Understanding One-class Novelty Detection.
Revised: Understanding One-class Novelty Detection.

Original: In one-class novelty detection and semantic anomaly detection ( i.e. K = 1 ), Theorem 6 has revealed that it is necessary to use a large-capacity model to ensure the good generalization in the separate space.
Revised: In one-class novelty detection and semantic anomaly detection ( i.e. K = 1 ), Theorem 6 has revealed that it is necessary to use a large-capacity model to ensure the good generalization in the separate space.

Original: Theorem 3 and Theorem 12 suggest that we should try to avoid the overlap between ID and OOD distributions in the one-class case.
Revised: Theorem 3 and Theorem 12 suggest that we should try to avoid the overlap between ID and OOD distributions in the one-class case.

Original: If the overlap cannot be avoided, we suggest considering the multi-class OOD detection instead of the one-class case.
Revised: If the overlap cannot be avoided, we suggest considering the multi-class OOD detection instead of the one-class case.

Original: Additionally, in the density-based space, Theorem 11 has shown that it is necessary to select a suitable hypothesis space satisfying the Realizability Assumption to ensure the learnability of OOD detection in the density-based space.
Revised: Additionally, in the density-based space, Theorem 11 has shown that it is necessary to select a suitable hypothesis space satisfying the Realizability Assumption to ensure the learnability of OOD detection in the density-based space.

Original: Generally, a large-capacity model can be helpful to guarantee that the Realizability Assumption holds.
Revised: Generally, a large-capacity model can be helpful to guarantee that the Realizability Assumption holds.

Original:  Related Work
Revised:  Related Work

Original: We briefly review the related theoretical works below.
Revised: We briefly review the related theoretical works below.

Original: See Appendix A for detailed related works.
Revised: See Appendix A for detailed related works.

Original: OOD Detection Theory.
Revised: OOD Detection Theory.

Original: [49] understands the OOD detection via goodness-of-fit tests and typical set hypothesis, and argues that minimal density estimation errors can lead to OOD detection failures without assuming an overlap between ID and OOD distributions.
Revised: [49] understands the OOD detection via goodness-of-fit tests and typical set hypothesis, and argues that minimal density estimation errors can lead to OOD detection failures without assuming an overlap between ID and OOD distributions.

Original: Beyond [49], [50] paves a new avenue to designing provable OOD detection algorithms.
Revised: Beyond [49], [50] paves a new avenue to designing provable OOD detection algorithms.

Original: Compared to [50, 49], our theory focuses on the PAC learnable theory of OOD detection and identifies several necessary and sufficient conditions for the learnability of OOD detection, opening a door to study OOD detection in theory.
Revised: Compared to [50, 49], our theory focuses on the PAC learnable theory of OOD detection and identifies several necessary and sufficient conditions for the learnability of OOD detection, opening a door to study OOD detection in theory.

Original: Open-set Learning Theory.
Revised: Open-set Learning Theory.

Original: [51] and [29, 52] propose the agnostic PAC learning bounds for open-set detection and open-set domain adaptation, respectively.
Revised: [51] and [29, 52] propose the agnostic PAC learning bounds for open-set detection and open-set domain adaptation, respectively.

Original: Unfortunately, [29, 51, 52] all require that the test data are indispensable during the training process.
Revised: Unfortunately, [29, 51, 52] all require that the test data are indispensable during the training process.

Original: To investigate open-set learning (OSL) without accessing the test data during training, [24] proposes and investigates the almost agnostic
Revised: To investigate open-set learning (OSL) without accessing the test data during training, [24] proposes and investigates the almost agnostic

Original: PAC learnability for OSL.
Revised: PAC learnability for OSL.

Original: However, the assumptions used in [24] are very strong and unpractical.
Revised: However, the assumptions used in [24] are very strong and unpractical.

Original: Learning Theory for Classification with Reject Option.
Revised: Learning Theory for Classification with Reject Option.

Original: Many works [53, 54] also investigate the classification with reject option (CwRO) problem, which is similar to OOD detection in some cases.
Revised: Many works [53, 54] also investigate the classification with reject option (CwRO) problem, which is similar to OOD detection in some cases.

Original: [55, 56, 57, 58, 59] study the learning theory and propose the PAC learning bounds for CwRO.
Revised: [55, 56, 57, 58, 59] study the learning theory and propose the PAC learning bounds for CwRO.

Original: However, compared to our work regarding OOD detection, existing CwRO theories mainly focus on how the ID risk R in D ( i.e. , the risk that ID data is wrongly classified) is influenced by special rejection rules.
Revised: However, compared to our work regarding OOD detection, existing CwRO theories mainly focus on how the ID risk R in D ( i.e. , the risk that ID data is wrongly classified) is influenced by special rejection rules.

Original: Our theory not only focuses on the ID risk, but also pays attention to the OOD risk.
Revised: Our theory not only focuses on the ID risk, but also pays attention to the OOD risk.

Original: Robust Statistics.
Revised: Robust Statistics.

Original: In the field of robust statistics [60], researchers aim to propose estimators and testers that can mitigate the negative effects of outliers (similar to OOD data).
Revised: In the field of robust statistics [60], researchers aim to propose estimators and testers that can mitigate the negative effects of outliers (similar to OOD data).

Original: The proposed estimators are supposed to be independent of the potentially high dimensionality of the data [61, 62, 63].
Revised: The proposed estimators are supposed to be independent of the potentially high dimensionality of the data [61, 62, 63].

Original: Existing works [64, 65, 66] in the field have identified and resolved the statistical limits of outlier robust statistics by constructing estimators and proving impossibility results.
Revised: Existing works [64, 65, 66] in the field have identified and resolved the statistical limits of outlier robust statistics by constructing estimators and proving impossibility results.

Original: In the future, it is a promising and interesting research direction to study the robustness of OOD detection based on robust statistics.
Revised: In the future, it is a promising and interesting research direction to study the robustness of OOD detection based on robust statistics.

Original: PQ Learning Theory.
Revised: PQ Learning Theory.

Original: Under some conditions, PQ learning theory [67, 68] can be regarded as the
Revised: Under some conditions, PQ learning theory [67, 68] can be regarded as the

Original: PAC theory for OOD detection in the semi-supervised or transductive learning cases, i.e. , test data are required during training.
Revised: PAC theory for OOD detection in the semi-supervised or transductive learning cases, i.e. , test data are required during training.

Original: Besides, [67, 68] aim to give the PAC estimation under Realizability
Revised: Besides, [67, 68] aim to give the PAC estimation under Realizability

Original: Assumption [21].
Revised: Assumption [21].

Original: Our theory does not only study the PAC estimation in the realization cases, but also studies the other cases, which are more difficult than PAC theory under Realizability Assumption.
Revised: Our theory does not only study the PAC estimation in the realization cases, but also studies the other cases, which are more difficult than PAC theory under Realizability Assumption.

Original:  Conclusions and Future Works
Revised:  Conclusions and Future Works

Original: Detecting OOD data has shown its significance in improving the reliability of machine learning.
Revised: Detecting OOD data has shown its significance in improving the reliability of machine learning.

Original: However, very few works discuss OOD detection in theory, which hinders real-world applications of OOD detection algorithms.
Revised: However, very few works discuss OOD detection in theory, which hinders real-world applications of OOD detection algorithms.

Original: In this paper, we are the first to provide the PAC theory for OOD detection.
Revised: In this paper, we are the first to provide the PAC theory for OOD detection.

Original: Our results imply that we cannot expect a universally consistent algorithm to handle all scenarios in OOD detection.
Revised: Our results imply that we cannot expect a universally consistent algorithm to handle all scenarios in OOD detection.

Original: Yet, it is still possible to make OOD detection learnable in certain scenarios.
Revised: Yet, it is still possible to make OOD detection learnable in certain scenarios.

Original: For example, when we design OOD detection algorithms, we normally only have finite ID datasets.
Revised: For example, when we design OOD detection algorithms, we normally only have finite ID datasets.

Original: In this real scenario, Theorem 8 provides a necessary and sufficient condition for the success of OOD detection.
Revised: In this real scenario, Theorem 8 provides a necessary and sufficient condition for the success of OOD detection.

Original: Our theory reveals many necessary and sufficient conditions for the learnability of
Revised: Our theory reveals many necessary and sufficient conditions for the learnability of

Original: OOD detection, hence opening a door to studying the learnability of OOD detection.
Revised: OOD detection, hence opening a door to studying the learnability of OOD detection.

Original: In the future, we will focus on studying the robustness of OOD detection based on robust statistics [64, 69].
Revised: In the future, we will focus on studying the robustness of OOD detection based on robust statistics [64, 69].

Original: Acknowledgment
Revised: Acknowledgment

Original: JL and ZF were supported by the Australian Research Council (ARC) under FL190100149.
Revised: JL and ZF were supported by the Australian Research Council (ARC) under FL190100149.

Original: YXL was supported by the Wisconsin Alumni Research Foundation (WARF).
Revised: YL is supported by the AFOSR Young Investigator Program Award.

Original: BH was supported by the 
Revised: BH was supported by the RGC Early

Original: RGC Early Career Scheme No. 22200720 and NSFC Young Scientists Fund No. 62006202.
Revised: Career Scheme No. 22200720 and NSFC Young Scientists Fund No. 62006202.

Original: ZF would also like to thank Prof. Peter Bartlett and Dr. Tongliang Liu for productive discussions.
Revised: ZF would also like to thank Prof. Peter Bartlett and Dr. Tongliang Liu for productive discussions.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Appendix B (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] See Appendix B (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Appendix B (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] See Appendix B (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e
Revised: [N/A] (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A] 4.
Revised: [N/A] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]