Original: Is Out-of-distribution Detection Learnable?
Revised: Is Out-of-distribution Detection Learnable?

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Supervised learning aims to train a classifier under the assumption that training andtest data are from the same distribution.
Revised: Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution.

Original: To ease the above assumption, researchershave studied a more realistic setting: out-of-distribution (OOD) detection, wheretest data may come from classes that are unknown during training ( i.e. , OOD data).
Revised: To ease the above assumption, researchers have studied a more realistic setting: out-of-distribution (OOD) detection, where test data may come from classes that are unknown during training ( i.e. , OOD data).

Original: Due to the unavailability and diversity of OOD data, good generalization abilityis crucial for effective OOD detection algorithms.
Revised: Due to the unavailability and diversity of OOD data, good generalization ability is crucial for effective OOD detection algorithms.

Original: To study the generalization of
Revised: To study the generalization of

Original: OOD detection, in this paper, we investigate the probably approximately correct(PAC) learning theory of OOD detection, which is proposed by researchers as anopen problem .
Revised: OOD detection, in this paper, we investigate the probably approximately correct (PAC) learning theory of OOD detection, which is proposed by researchers as an open problem .

Original: First, we find a necessary condition for the learnability of OODdetection.
Revised: First, we find a necessary condition for the learnability of OOD detection.

Original: Then, using this condition, we prove several impossibility theorems forthe learnability of OOD detection under some scenarios.
Revised: Then, using this condition, we prove several impossibility theorems for the learnability of OOD detection under some scenarios.

Original: Although the impossibil- ity theorems are frustrating, we find that some conditions of these impossibilitytheorems may not hold in some practical scenarios.
Revised: Although the impossibility theorems are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios.

Original: Based on this observation, we next give several necessary and sufficient conditions to characterize the learnabilityof OOD detection in some practical scenarios.
Revised: Based on this observation, we next give several necessary and sufficient conditions to characterize the learnability of OOD detection in some practical scenarios.

Original: Lastly, we also offer theoreticalsupports for several representative OOD detection works based on our OOD theory.
Revised: Lastly, we also offer theoretical supports for several representative OOD detection works based on our OOD theory.

Original:  Introduction
Revised:  Introduction

Original: The success of supervised learning is established on an implicit assumption that training and test datashare a same distribution, i e. , in-distribution (ID) [1, 2, 3, 4].
Revised: The success of supervised learning is established on an implicit assumption that training and test data share a same distribution, i.e. , in-distribution (ID) [1, 2, 3, 4].

Original: However, test data distribution in manyreal-world scenarios may violate the assumption and, instead, contain out-of-distribution (OOD) datawhose labels have not been seen during the training process [5, 6].
Revised: However, test data distribution in many real-world scenarios may violate the assumption and, instead, contain out-of-distribution (OOD) data whose labels have not been seen during the training process [5, 6].

Original: To mitigate the risk of OOD data,researchers have considered a more practical learning scenario:
Revised: To mitigate the risk of OOD data, researchers have considered a more practical learning scenario:

Original: OOD detection which determineswhether an input is ID/OOD, while classifying the ID data into respective classes.
Revised: OOD detection which determines whether an input is ID/OOD, while classifying the ID data into respective classes.

Original: OOD detection hasshown great potential to ensure the reliable deployment of machine learning models in the real world.
Revised: OOD detection has shown great potential to ensure the reliable deployment of machine learning models in the real world.

Original: A rich line of algorithms have been developed to empirically address the OOD detection problem[6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].
Revised: A rich line of algorithms have been developed to empirically address the OOD detection problem [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].

Original: However, very few works study theory of OODdetection, which hinders the rigorous path forward for the field.
Revised: However, very few works study theory of OOD detection, which hinders the rigorous path forward for the field.

Original: This paper aims to bridge the gap.
Revised: This paper aims to bridge the gap.

Original: In this paper, we provide a theoretical framework to understand the learnability of the OOD detectionproblem.
Revised: In this paper, we provide a theoretical framework to understand the learnability of the OOD detection problem.

Original: We investigate the probably approximately correct (PAC) learning theory of OOD detection,which is posed as an open problem to date.
Revised: We investigate the probably approximately correct (PAC) learning theory of OOD detection, which is posed as an open problem to date.

Original: Unlike the classical PAC learning theory in a supervisedsetting, our problem setting is fundamentally challenging due to the absence of OOD data in training.
Revised: Unlike the classical PAC learning theory in a supervised setting, our problem setting is fundamentally challenging due to the absence of OOD data in training.

Original: In many real-world scenarios, OOD data can be diverse and priori-unknown.
Revised: In many real-world scenarios, OOD data can be diverse and priori-unknown.

Original: Given this, we studywhether there exists an algorithm that can be used to detect various OOD data instead of merely somespecified OOD data.
Revised: Given this, we study whether there exists an algorithm that can be used to detect various OOD data instead of merely some specified OOD data.

Original: Such is the significance of studying the learning theory for OOD detection [4].
Revised: Such is the significance of studying the learning theory for OOD detection [4].

Original: This motivates our question: is OOD detectionagnostic PAC learnable?
Revised: This motivates our question: is OOD detection PAC learnable?

Original: i.e., is there theagnostic PAC learning theory to guarantee the generalization ability of OOD detection?
Revised: i.e., is there the PAC learning theory to guarantee the generalization ability of OOD detection?

Original: To investigate the learning theory, we mainly focus on two basic spaces: domain space and hypothesisspace.
Revised: To investigate the learning theory, we mainly focus on two basic spaces: domain space and hypothesis space.

Original: The domain space is a space consisting of some distributions, and the hypothesis space is a space consisting of some classifiers.
Revised: The domain space is a space consisting of some distributions, and the hypothesis space is a space consisting of some classifiers.

Original: Existing agnostic PAC theories in supervised learning [21, 22]are distribution-free, i.e. , the domain space consists of all domains.
Revised: Existing agnostic PAC theories in supervised learning [21, 22] are distribution-free, i.e. , the domain space consists of all domains.

Original: Yet, in Theorem 4, we shows thatthe learning theory of OOD detection is not distribution-free.
Revised: Yet, in Theorem 4, we shows that the learning theory of OOD detection is not distribution-free.

Original: In fact, we discover that OOD detectionis learnable only if the domain space and the hypothesis space satisfy some special conditions, e.g.,
Revised: In fact, we discover that OOD detection is learnable only if the domain space and the hypothesis space satisfy some special conditions, e.g.,

Original: Conditions 1 and 3.
Revised: Conditions 1 and 3.

Original: Notably, there are many conditions and theorems in existing learning theoriesand many OOD detection algorithms in the literature.
Revised: Notably, there are many conditions and theorems in existing learning theories and many OOD detection algorithms in the literature.

Original: Thus, it is very difficult to analyze the relationbetween these theories and algorithms, and explore useful conditions to ensure the learnability of
Revised: Thus, it is very difficult to analyze the relation between these theories and algorithms, and explore useful conditions to ensure the learnability of

Original: OOD detection, especially when we have to explore them from the scratch .
Revised: OOD detection, especially when we have to explore them from the scratch .

Original: Thus, the main aim of ourpaper is to study these essential conditions.
Revised: Thus, the main aim of our paper is to study these essential conditions.

Original: From these essential conditions, we can know when OODdetection can be successful in practical scenarios.
Revised: From these essential conditions, we can know when OOD detection can be successful in practical scenarios.

Original: We restate our question and goal in following:
Revised: We restate our question and goal in following:

Original: Given hypothesis spaces and several representative domain spaces, what are the conditions to ensure the learnability of OOD detection?
Revised: Given hypothesis spaces and several representative domain spaces, what are the conditions to ensure the learnability of OOD detection?

Original: If possible, we hope that these conditions are necessary and sufficient in some scenarios.
Revised: If possible, we hope that these conditions are necessary and sufficient in some scenarios.

Original: Main Results.
Revised: Main Results.

Original: We investigate the learnability of OOD detection starting from the largest space—thetotal space, and give a necessary condition (Condition 1) for the learnability.
Revised: We investigate the learnability of OOD detection starting from the largest space—the total space, and give a necessary condition (Condition 1) for the learnability.

Original: However, we find thatthe overlap between ID and OOD data may result in that the necessary condition does not hold.
Revised: However, we find that the overlap between ID and OOD data may result in that the necessary condition does not hold.

Original: Therefore, we give an impossibility theorem to demonstrate that OOD detection fails in the totalspace (Theorem 4).
Revised: Therefore, we give an impossibility theorem to demonstrate that OOD detection fails in the total space (Theorem 4).

Original: Next, we study OOD detection in the separate space, where there are no overlapsbetween the ID and OOD data.
Revised: Next, we study OOD detection in the separate space, where there are no overlaps between the ID and OOD data.

Original: Unfortunately, there still exists impossibility theorem (Theorem 5),which demonstrates that OOD detection is not learnable in the separate space under some conditions.
Revised: Unfortunately, there still exists impossibility theorem (Theorem 5), which demonstrates that OOD detection is not learnable in the separate space under some conditions.

Original: Although the impossibility theorems obtained in the separate space are frustrating, we find that someconditions of these impossibility theorems may not hold in some practical scenarios.
Revised: Although the impossibility theorems obtained in the separate space are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios.

Original: Based on thisobservation, we give several necessary and sufficient conditions to characterize the learnability of
Revised: Based on this observation, we give several necessary and sufficient conditions to characterize the learnability of

Original: OOD detection in the separate space (Theorems 6 and 10).
Revised: OOD detection in the separate space (Theorems 6 and 10).

Original: Especially, when our model is based onfully-connected neural network (FCNN), OOD detection is learnable in the separate space if andonly if the feature space is finite.
Revised: Especially, when our model is based on fully-connected neural network (FCNN), OOD detection is learnable in the separate space if and only if the feature space is finite.

Original: Furthermore, we investigate the learnability of OOD detection inother more practical domain spaces, e.g. , the finite-ID-distribution space (Theorem 8) and the densitybased space (Theorem 9).
Revised: Furthermore, we investigate the learnability of OOD detection in other more practical domain spaces, e.g. , the finite-ID-distribution space (Theorem 8) and the densitybased space (Theorem 9).

Original: By studying the finite-ID-distribution space, we discover a compatibilitycondition (Condition 3) that is a necessary and sufficient condition for this space.
Revised: By studying the finite-ID-distribution space, we discover a compatibility condition (Condition 3) that is a necessary and sufficient condition for this space.

Original: Next, we furtherinvestigate the compatibility condition in the density-based space, and find that such condition is alsothe necessary and sufficient condition in some practical scenarios (Theorem 11).
Revised: Next, we further investigate the compatibility condition in the density-based space, and find that such condition is also the necessary and sufficient condition in some practical scenarios (Theorem 11).

Original: Implications and Impacts of Theory.
Revised: Implications and Impacts of Theory.

Original: Our study is not of purely theoretical interest; it has alsopractical impacts.
Revised: Our study is not of purely theoretical interest; it has also practical impacts.

Original: First, when we design OOD detection algorithms, we normally only have finite ID datasets, corresponding to the finite-ID-distribution space.
Revised: First, when we design OOD detection algorithms, we normally only have finite ID datasets, corresponding to the finite-ID-distribution space.

Original: In this case, Theorem 8 provides thenecessary and sufficient condition to the success of OOD detection.
Revised: In this case, Theorem 8 gives the necessary and sufficient condition to the success of OOD detection.

Original: Second, our theory also providestheoretical support (Theorems 10 and 11) for several representative OOD detection works [7, 8, 23].
Revised: Second, our theory provides theoretical support (Theorems 10 and 11) for several representative OOD detection works [7, 8, 23].

Original: Third, our theory shows that OOD detection can be addressed in image-based distributions as long as ID images have clearly different semantic meanings  from OOD images.
Revised: Third, our theory shows that OOD detection can be learnable in image-based distributions as long as ID images have clearly different semantic meanings and styles from OOD images.

Original: Fourth, we should not expecta universally working algorithm.
Revised: Fourth, we should not expect a universally working algorithm.

Original: It is necessary to design different algorithms in different scenarios.
Revised: It is necessary to design different algorithms in different scenarios.

Original:  Learning Setups
Revised:  Learning Setups

Original: We start by introducing the necessary concepts and notations for our theoretical framework.
Revised: We start by introducing the necessary concepts and notations for our theoretical framework.

Original: Givena feature space X ⊂ R d and a label space Y :
Revised: Given a feature space X ⊂ R d and a label space Y :

Original: = { 1 , ..., K } , we have an ID joint distribution D X
Revised: = { 1 , ..., K } , we have an ID joint distribution D X

Original: I Y Iover X × Y , where X I ∈
Revised: I Y I over X × Y , where X I ∈

Original: X and Y I ∈ Y are random variables.
Revised: X and Y I ∈ Y are random variables.

Original: We also have an OOD jointdistribution D X O Y O , where X O is a random variable from X , but Y O is a random variable whoseoutputs do not belong to Y .
Revised: We also have an OOD joint distribution D X O Y O , where X O is a random variable from X , but Y O is a random variable whose outputs do not belong to Y .

Original: During testing, we will meet a mixture of ID and OOD joint distributions:
Revised: During testing, we will meet a mixture of ID and OOD joint distributions:

Original: D XY :
Revised: D XY :

Original: = (1 − π out ) D X
Revised: = (1 − π out ) D X

Original: I Y
Revised: I Y

Original: I + π out D X
Revised: I + π out D X

Original: O Y O ,
Revised: O Y O ,

Original: and can only observe the marginal distribution D X :=(1 − π out ) D X
Revised: and can only observe the marginal distribution D X := (1 − π out ) D X

Original: I + π out D X
Revised: I + π out D X

Original: O , where the constant π out ∈ [0 , 1) is an unknown class-prior probability.
Revised: O , where the constant π out ∈ [0 , 1) is an unknown class-prior probability.

Original: Problem 1 (OOD Detection [4]) .
Revised: Problem 1 (OOD Detection [4]) .

Original: Given an ID joint distribution D X I Y I and a training data S :
Revised: Given an ID joint distribution D X I Y I and a training data S :

Original: ={ ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn independent and identically distributed from D X
Revised: = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn independent and identically distributed from D X

Original: I Y
Revised: I Y

Original: I , the aim of OODdetection is to train a classifier f by using the training data S such that, for any test data x drawnfrom the mixed marginal distribution D X : 1) if x is an observation from D X I , f can classify x intocorrect ID classes; and 2) if x is an observation from D X O , f can detect x as OOD data.
Revised: I , the aim of OOD detection is to train a classifier f by using the training data S such that, for any test data x drawn from the mixed marginal distribution D X : 1) if x is an observation from D X I , f can classify x into correct ID classes; and 2) if x is an observation from D X O , f can detect x as OOD data.

Original: According to the survey [4], when K > 1 , OOD detection is also known as the open-set recognitionor open-set learning [24, 25]; and when K = 1 , OOD detection reduces to one-class novelty detectionand semantic anomaly detection [26, 27, 28].
Revised: According to the survey [4], when K > 1 , OOD detection is also known as the open-set recognition or open-set learning [24, 25]; and when K = 1 , OOD detection reduces to one-class novelty detection and semantic anomaly detection [26, 27, 28].

Original: OOD Label and Domain Space.
Revised: OOD Label and Domain Space.

Original: Based on Problem 1, we know it is not necessary to classify OODdata into the correct OOD classes.
Revised: Based on Problem 1, we know it is not necessary to classify OOD data into the correct OOD classes.

Original: Without loss of generality, let all OOD data be allocated to one big OOD class, i.e. , Y O = K [-+
-]
Revised: Without loss of generality, let all OOD data be allocated to one big OOD class, i.e. , Y O = K [-+
-]+ 1 [24, 29].

Original: To investigate theagnostic PAC learnability of OOD detection,we define a domain space D XY , which is a set consisting of some joint distributions D XY mixed bysome ID joint distributions and some OOD joint distributions.
Revised: To investigate the PAC learnability of OOD detection, we define a domain space D XY , which is a set consisting of some joint distributions D XY mixed by some ID joint distributions and some OOD joint distributions.

Original: In this paper, the joint distribution
Revised: In this paper, the joint distribution

Original: D XY mixed by ID joint distribution D X
Revised: D XY mixed by ID joint distribution D X

Original: I Y I and OOD joint distribution D X
Revised: I Y I and OOD joint distribution D X

Original: O Y O is called domain .
Revised: O Y O is called domain .

Original: Hypothesis Spaces and Scoring Function Spaces.
Revised: Hypothesis Spaces and Scoring Function Spaces.

Original: A hypothesis space H is a subset of functionspace, i.e. , H ⊂ { h : X → Y ∪ { K + 1 }} .
Revised: A hypothesis space H is a subset of function space, i.e. , H ⊂ { h : X → Y ∪ { K + 1 }} .

Original: We set H in ⊂ { h : X → Y} to the ID hypothesis space.
Revised: We set H in ⊂ { h : X → Y} to the ID hypothesis space.

Original: We also define H b ⊂ { h : X → { 1 , 2 }} as the hypothesis space for binary classification, where1 represents the ID data, and 2 represents the OOD data.
Revised: We also define H b ⊂ { h : X → { 1 , 2 }} as the hypothesis space for binary classification, where 1 represents the ID data, and 2 represents the OOD data.

Original: The function h is called the hypothesisfunction.
Revised: The function h is called the hypothesis function.

Original: A scoring function space is a subset of function space, i.e. , F l ⊂ { f : X → R l } , where l isthe output’s dimension of the vector-valued function f .
Revised: A scoring function space is a subset of function space, i.e. , F l ⊂ { f : X → R l } , where l is the output’s dimension of the vector-valued function f .

Original: The function f is called the scoring function.
Revised: The function f is called the scoring function.

Original: Loss and Risks.
Revised: Loss and Risks.

Original: Let Y all =
Revised: Let Y all =

Original: Y ∪ { K + 1 } .
Revised: Y ∪ { K + 1 } .

Original: Given a loss function ℓ  : Y all × Y all → R ≥ 0 satisfyingthat ℓ ( y 1 , y 2 ) =[-
-]
Revised: Given a loss function ℓ 2 : Y all × Y all → R ≥ 0 satisfying that ℓ ( y 1 , y 2 ) =[-
-]

Original: if and only if y 1 = y 2 , and any h ∈ H , then the risk with respect to D XY is
Revised: if and only if y 1 = y 2 , and any h ∈ H , then the risk with respect to D XY is

Original: [Equation]
Revised: [Equation]

Original: The α -risk R αD ( h ) :
Revised: The α -risk R αD ( h ) :

Original: = (1 − α ) R in D ( h ) + αR out D ( h ) , ∀ α ∈ [0 , 1] , where the risks R in D ( h ) , R out D ( h ) areR in D ( h ) : =
Revised: = (1 − α ) R in D ( h ) + αR out D ( h ) , ∀ α ∈ [0 , 1] , where the risks R in D ( h ) , R out D ( h ) are

Original: E ( x ,y ) ∼ D X I Y I ℓ ( h ( x ) , y ) , R out D ( h ) := E x ∼ D X O ℓ ( h ( x ) , K + 1) .
Revised: [Equation]

Original: Learnability.
Revised: Learnability.

Original: We aim to select a hypothesis function h ∈ H with approximately minimal risk, basedon finite data.
Revised: We aim to select a hypothesis function h ∈ H with approximately minimal risk, based on finite data.

Original: Generally, we expect the approximation to get better, with the increase in sample size.
Revised: Generally, we expect the approximation to get better, with the increase in sample size.

Original: Algorithms achieving this are said to be consistent.
Revised: Algorithms achieving this are said to be consistent.

Original: Formally, we introduce the following definition:
Revised: Formally, we introduce the following definition:

Original: Definition 1 (Learnability of OOD Detection) .
Revised: Definition 1 (Learnability of OOD Detection) .

Original: Given a domain space D XY and a hypothesis space
Revised: Given a domain space D XY and a hypothesis space

Original: H ⊂ { h : X → Y all } , we say OOD detection is learnable in D XY for H , if there exists an algorithm A  : ∪ + ∞ n =1 ( X ×Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 ,as n → + ∞ , and for any domain D XY ∈ D XY ,
Revised: H ⊂ { h : X → Y all } , we say OOD detection is learnable in D XY for H , if there exists an algorithm A 3 : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,

Original: [Equation]
Revised: [Equation]

Original: An algorithm A for which this holds is said to be consistent with respect to D XY .
Revised: An algorithm A for which this holds is said to be consistent with respect to D XY .

Original: Definition 1 is a natural extension of agnostic PAC learnability of supervised learning [30].
Revised: Definition 1 is a natural extension of agnostic PAC learnability of supervised learning [30].

Original: If for any
Revised: If for any

Original: D XY ∈ D XY , π out = 0 , then Definition 2 is the agnostic PAC learnability of supervised learning.
Revised: D XY ∈ D XY , π out = 0 , then Definition 2 is the agnostic PAC learnability of supervised learning.

Original: Although themathematical expression of Definition 1 is different from the normal definition ofagnostic PAC learning in [21], one can easily prove that they are equivalent, see Appendix D.3.
Revised: Although the expression of Definition 1 is different from the normal definition of agnostic PAC learning in [21], one can easily prove that they are equivalent when ℓ is bounded, see Appendix D.3.

Original: Since OOD data are unavailable, it is impossible to obtain information about the class-prior probabilityπ out .
Revised: Since OOD data are unavailable, it is impossible to obtain information about the class-prior probability π out .

Original: Furthermore, in the real world, it is possible that π out can be any value in [0 , 1) .
Revised: Furthermore, in the real world, it is possible that π out can be any value in [0 , 1) .

Original: Therefore,the imbalance issue between ID and OOD distributions, and the priori-unknown issue ( i.e. , π out isunknown) are the core challenges.
Revised: Therefore, the imbalance issue between ID and OOD distributions, and the priori-unknown issue ( i.e. , π out is unknown) are the core challenges.

Original: To ease these challenges, researchers use AUROC, AUPR and
Revised: To ease these challenges, researchers use AUROC, AUPR and

Original: FPR95 to estimate the performance of OOD detection [18, 31, 32, 33, 34].
Revised: FPR95 to estimate the performance of OOD detection [18, 31, 32, 33, 34, 35].

Original: It seems that there is agap between Definition 1 and existing works.
Revised: It seems that there is a gap between Definition 1 and existing works.

Original: To eliminate this gap, we revise Eq.
Revised: To eliminate this gap, we revise Eq.

Original: (2) as follows:
Revised: (2) as follows:

Original: [Equation]
Revised: [Equation]

Original: If an algorithm A satisfies Eq.
Revised: If an algorithm A satisfies Eq.

Original: (3), then the imbalance issue and the prior-unknown issue disappear.
Revised: (3), then the imbalance issue and the prior-unknown issue disappear.

Original: That is, A can simultaneously classify the ID data and detect the OOD data well.
Revised: That is, A can simultaneously classify the ID data and detect the OOD data well.

Original: Based on the abovediscussion, we define the strong learnability of OOD detection as follows:
Revised: Based on the above discussion, we define the strong learnability of OOD detection as follows:

Original: Definition 2 (Strong Learnability of OOD Detection) .
Revised: Definition 2 (Strong Learnability of OOD Detection) .

Original: Given a domain space D XY and a hypothesisspace H ⊂ { h : X → Y all } , we say OOD detection is strongly learnable in D XY for H , if thereexists an algorithm A : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) ,such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,
Revised: Given a domain space D XY and a hypothesis space H ⊂ { h : X → Y all } , we say OOD detection is strongly learnable in D XY for H , if there exists an algorithm A : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,

Original: [Equation]
Revised: [Equation]

Original: In Theorem 1, we have shown that the strong learnability of OOD detection is equivalent to thelearnability of OOD detection, if the domain space D XY is a prior-unknown space (see Definition 3).
Revised: In Theorem 1, we have shown that the strong learnability of OOD detection is equivalent to the learnability of OOD detection, if the domain space D XY is a prior-unknown space (see Definition 3).

Original: In this paper, we mainly discuss the learnability in the prior-unknown space.
Revised: In this paper, we mainly discuss the learnability in the prior-unknown space.

Original: Therefore, when wemention that OOD detection is learnable, we also mean that OOD detection is strongly learnable .
Revised: Therefore, when we mention that OOD detection is learnable, we also mean that OOD detection is strongly learnable .

Original: Goal of Theory.
Revised: Goal of Theory.

Original: Note that the agnostic PAC learnability of supervised learning is distribution-free,i.e. , the domain space D XY consists of all domains.
Revised: Note that the agnostic PAC learnability of supervised learning is distribution-free, i.e. , the domain space D XY consists of all domains.

Original: However, due to the absence of OOD dataduring the training process [8, 14, 24], it is obvious that the learnability of OOD detection is notdistribution-free ( i.e. , Theorem 4).
Revised: However, due to the absence of OOD data during the training process [8, 14, 24], it is obvious that the learnability of OOD detection is not distribution-free ( i.e. , Theorem 4).

Original: In fact, we discover that the learnability of OOD detection isdeeply correlated with the relationship between the domain space D XY and the hypothesis space H .
Revised: In fact, we discover that the learnability of OOD detection is deeply correlated with the relationship between the domain space D XY and the hypothesis space H .

Original: That is, OOD detection is learnable only when the domain space D XY and the hypothesis space Hsatisfy some special conditions, e.g., Condition 1 and Condition 3.
Revised: That is, OOD detection is learnable only when the domain space D XY and the hypothesis space H satisfy some special conditions, e.g., Condition 1 and Condition 3.

Original: We present our goal as follows: 
Revised: We present our goal as follows: Goal:

Original: Goal: given a hypothesis space H and several representative domain spaces D XY , what are the conditions to ensure the learnability of OOD detection?
Revised: given a hypothesis space H and several representative domain spaces D XY , what are the conditions to ensure the learnability of OOD detection?

Original: Furthermore, if possible, we hope that these conditions are necessary and sufficient in some scenarios.
Revised: Furthermore, if possible, we hope that these conditions are necessary and sufficient in some scenarios.

Original: Therefore, compared to the agnostic PAC learnability of supervised learning, our theory doesn’tfocus on the distribution-free case, but focuses on discovering essential conditions to guarantee thelearnability of OOD detection in several representative and practical domain spaces D XY .
Revised: Therefore, compared to the agnostic PAC learnability of supervised learning, our theory doesn’t focus on the distribution-free case, but focuses on discovering essential conditions to guarantee the learnability of OOD detection in several representative and practical domain spaces D XY .

Original: By theseessential conditions, we can know when OOD detection can be successful in real applications.
Revised: By these essential conditions, we can know when OOD detection can be successful in real applications.

Original: The guidance for real applications based on our theory and all proofs can be found in Appendices .
Revised: The guidance for real applications based on our theory and all proofs can be found in Appendices .

Original:  Learning in Priori-unknown Spaces
Revised:  Learning in Priori-unknown Spaces

Original: We first investigate a special space, called prior-unknown space.
Revised: We first investigate a special space, called prior-unknown space.

Original: In such space, Definition 1 and
Revised: In such space, Definition 1 and

Original: Definition 2 are equivalent.
Revised: Definition 2 are equivalent.

Original: Furthermore, we also prove that if OOD detection is strongly learnablein a space D XY , then one can discover a larger domain space, which is prior-unknown, to ensurethe learnability of OOD detection.
Revised: Furthermore, we also prove that if OOD detection is strongly learnable in a space D XY , then one can discover a larger domain space, which is prior-unknown, to ensure the learnability of OOD detection.

Original: These results imply that it is enough to consider our theory in theprior-unknown spaces.
Revised: These results imply that it is enough to consider our theory in the prior-unknown spaces.

Original: The prior-unknown space is introduced as follows:
Revised: The prior-unknown space is introduced as follows:

Original: Definition 3.
Revised: Definition 3.

Original: Given a domain space D XY , we say D XY is a priori-unknown space, if for any domain
Revised: Given a domain space D XY , we say D XY is a priori-unknown space, if for any domain

Original: D XY ∈ D XY and any α ∈ [0 , 1)
Revised: D XY ∈ D XY and any α ∈ [0 , 1)

Original: , we have D αXY :
Revised: , we have D αXY :

Original: = (1 − α )
Revised: = (1 − α )

Original: D
Revised: D

Original: X
Revised: X

Original: I Y
Revised: I Y

Original: I + αD X O Y O ∈ D XY .
Revised: I + αD X O Y O ∈ D XY .

Original: Theorem 1.
Revised: Theorem 1.

Original: Given domain spaces D XY and D ′ XY
Revised: Given domain spaces D XY and D ′ XY

Original: = { D αXY : ∀ D XY ∈ D XY , ∀ α ∈ [0 , 1) } , then1) D XY is
Revised: = { D αXY : ∀ D XY ∈ D XY , ∀ α ∈ [0 , 1) } , then

Original: a priori-unknown space and D

Original: XY

Original: ⊂ D

Original: XY

Original:  2) if D XY is a priori-unknown space, then Definition  and Definition 2 are equivalent ; 3) OOD detection is strongly learnable in D XY if and only if OOD detection is learnable in D ′ XY .
Revised: 1) D ′ XY is a priori-unknown space and D XY ⊂ D ′ XY ; 2) if D XY is a priori-unknown space, then Definition 1 and Definition 2 are equivalent ; 3) OOD detection is strongly learnable in D XY if and only if OOD detection is learnable in D ′ XY .

Original: The second result of Theorem 1 bridges the learnability and strong learnability, which implies thatif an algorithm A is consistent with respect to a prior-unknown space, then this algorithm A canaddress the imbalance issue between ID and OOD distributions, and the priori-unknown issue well.
Revised: The second result of Theorem 1 bridges the learnability and strong learnability, which implies that if an algorithm A is consistent with respect to a prior-unknown space, then this algorithm A can address the imbalance issue between ID and OOD distributions, and the priori-unknown issue well.

Original: Based on Theorem 1, we focus on our theory in the prior-unknown spaces.
Revised: Based on Theorem 1, we focus on our theory in the prior-unknown spaces.

Original: Furthermore, to demystifythe learnability of OOD detection, we introduce five representative priori-unknown spaces:
Revised: Furthermore, to demystify the learnability of OOD detection, we introduce five representative priori-unknown spaces:

Original: • Single-distribution space D D XY XY .
Revised: • Single-distribution space D D XY XY .

Original: For a domain D XY , D D XY XY := { D αXY : ∀ α ∈ [0 , 1) } .
Revised: For a domain D XY , D D [Equation]

Original: • Total space D all XY , which consists of all domains.
Revised: • Total space D all XY , which consists of all domains.

Original: • Separate space D sXY , which consists of all domains that satisfy the separate condition, that is forany D XY ∈ D sXY , supp D X O
Revised: • Separate space D s XY , which consists of all domains that satisfy the separate condition, that is for any D XY ∈ D sXY , supp D X O

Original: ∩ supp D
Revised: ∩ supp D

Original: X
Revised: X

Original: I = ∅ , where supp means the support set.
Revised: I = ∅ , where supp means the support set.

Original: • Finite-ID-distribution space D FXY , which is a prior-unknown space satisfying that the number of distinct ID joint distributions D X
Revised: • Finite-ID-distribution space D FXY , which is a prior-unknown space satisfying that the number of distinct ID joint distributions D X

Original: I Y I in D FXY is finite, i.e. , 
Revised: I Y I in D FXY is finite, i.e. , |{ D X I Y

Revised: I : ∀ D XY ∈ D FXY }| < + ∞ .

Revised: • Density-based space D µ,bXY , which is a prior-unknown space consisting of some domains satisfying

Original: [Equation]
Revised: [Figure]

Original: • Density-based space D µ,bXY , which is a prior-unknown space consisting of some domains satisfyingthat: for any D XY , there exists a density function f with 1 /b ≤ f ≤ b in supp µ and 0 .
Revised: that: for any D XY , there exists a density function f with 1 /b ≤ f ≤ b in supp µ and 0 .

Original: 5 ∗ D X
Revised: 5 ∗ D X

Original: I +0 .
Revised: I + 0 .

Original: 5 ∗ D X O = (cid:82) f d µ , where µ is a measure defined over X .
Revised: 5 ∗ D X O = (cid:82) f d µ , where µ is a measure defined over X .

Original: Note that if µ is discrete, then D X is adiscrete distribution; and if µ is the Lebesgue measure, then D X is a continuous distribution.
Revised: Note that if µ is discrete, then D X is a discrete distribution; and if µ is the Lebesgue measure, then D X is a continuous distribution.

Original: The above representative spaces widely exist in real applications.
Revised: The above representative spaces widely exist in real applications.

Original: For example, 1) if the images fromdifferent semantic labels  are clearly different ( e.g. , cats and airplanes), then those images can forma distribution belonging to a separate space D sXY ; and 2) when designing an algorithm, we onlyhave finite ID datasets, e.g. , CIFAR-10, MNIST, SVHN, and ImageNet, to build a model.Then, [Figure]
Revised: For example, 1) if the images from different semantic labels with different styles are clearly different, then those images can form a distribution belonging to a separate space D sXY ; and 2) when designing an algorithm, we only have finite ID datasets, e.g. , CIFAR-10, MNIST, SVHN, and ImageNet, to build a model.

Original:  finite-ID-distribution space D F XY can handle this real scenario.
Revised: Then, finite-ID-distribution space D FXY can handle this real scenario.

Original: Note that the single-distribution spaceis a special case of the finite-ID-distribution space.
Revised: Note that the single-distribution space is a special case of the finite-ID-distribution space.

Original: In this paper, we mainly discuss these five spaces.
Revised: In this paper, we mainly discuss these five spaces.

Original:  Impossibility Theorems for OOD Detection
Revised:  Impossibility Theorems for OOD Detection

Original: In this section, we first give a necessary condition for the learnability of OOD detection.
Revised: In this section, we first give a necessary condition for the learnability of OOD detection.

Original: Then, weshow this necessary condition does not hold in the total space D all XY and the separate space D sXY .
Revised: Then, we show this necessary condition does not hold in the total space D all XY and the separate space D sXY .

Original: Necessary Condition.
Revised: Necessary Condition.

Original: We find a necessary condition for the learnability of OOD detection, i.e. , Condition 1, motivated by the experiments in Figure 1.
Revised: We find a necessary condition for the learnability of OOD detection, i.e. , Condition 1, motivated by the experiments in Figure 1.

Original: Details of Figure 1 can be found in Appendix C.3.
Revised: Details of Figure 1 can be found in Appendix C.3.

Original: Condition 1 (Linear Condition) .
Revised: Condition 1 (Linear Condition) .

Original: For any D XY ∈ D XY and any α ∈ [0 , 1) ,
Revised: For any D XY ∈ D XY and any α ∈ [0 , 1) ,

Original: [Equation]
Revised: [Equation]

Original: To reveal the importance of Condition 1, Theorem 2 shows that Condition 1 is a necessary andsufficient condition for the learnability of OOD detection if the D XY is the single-distribution space.
Revised: To reveal the importance of Condition 1, Theorem 2 shows that Condition 1 is a necessary and sufficient condition for the learnability of OOD detection if the D XY is the single-distribution space.

Original: Theorem 2.
Revised: Theorem 2.

Original: Given a hypothesis space H and a domain D XY , OOD detection is learnable in the single-distribution space D D XY XY for H if and only if linear condition (i.e., Condition 1) holds.
Revised: Given a hypothesis space H and a domain D XY , OOD detection is learnable in the single-distribution space D D XY XY for H if and only if linear condition (i.e., Condition 1) holds.

Original: Theorem 2 implies that Condition 1 is important for the learnability of OOD detection.
Revised: Theorem 2 implies that Condition 1 is important for the learnability of OOD detection.

Original: Due to thesimplicity of single-distribution space, Theorem 2 implies that Condition 1 is the necessary conditionfor the learnability of OOD detection in the prior-unknown space, see Lemma 1 in Appendix F.
Revised: Due to the simplicity of single-distribution space, Theorem 2 implies that Condition 1 is the necessary condition for the learnability of OOD detection in the prior-unknown space, see Lemma 1 in Appendix F.

Original: Impossibility Theorems.
Revised: Impossibility Theorems.

Original: Here, we first study whether Condition 1 holds in the total space D all XY .
Revised: Here, we first study whether Condition 1 holds in the total space D all XY .

Original: If
Revised: If

Original: Condition 1 does not hold, then OOD detection is not learnable.
Revised: Condition 1 does not hold, then OOD detection is not learnable.

Original: Theorem 3 shows that Condition 1 isnot always satisfied, especially, when there is an overlap between the ID and OOD distributions:
Revised: Theorem 3 shows that Condition 1 is not always satisfied, especially, when there is an overlap between the ID and OOD distributions:

Original: Definition 4 (Overlap Between ID and OOD) .
Revised: Definition 4 (Overlap Between ID and OOD) .

Original: We say a domain D XY has overlap between ID and
Revised: We say a domain D XY has overlap between ID and

Original: OOD distributions, if there is a σ -finite measure ˜ µ such that D X is absolutely continuous with respectto ˜ µ , and ˜ µ ( A overlap ) > 0 , where A overlap = { x ∈ X : f I ( x ) > 0 and f O ( x ) > 0 } .
Revised: OOD distributions, if there is a σ -finite measure ˜ µ such that D X is absolutely continuous with respect to ˜ µ , and ˜ µ ( A overlap ) > 0 , where A overlap = { x ∈ X : f I ( x ) > 0 and f O ( x ) > 0 } .

Original: Here f I andf O are the representers of D X I and D X O in Radon–Nikodym Theorem [35],
Revised: Here f I and f O are the representers of D X I and D X O in Radon–Nikodym Theorem [36],

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: Given a hypothesis space H and a prior-unknown space D XY , if there is D XY ∈ D XY ,which has overlap between ID and OOD, and inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 , then
Revised: Given a hypothesis space H and a prior-unknown space D XY , if there is D XY ∈ D XY , which has overlap between ID and OOD, and inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 , then

Original: Condition 1 does not hold.
Revised: Condition 1 does not hold.

Original: Therefore, OOD detection is not learnable in D XY for H .
Revised: Therefore, OOD detection is not learnable in D XY for H .

Original: Theorem 3 clearly shows that under proper conditions, Condition 1 does not hold, if there exists adomain whose ID and OOD distributions have overlap.
Revised: Theorem 3 clearly shows that under proper conditions, Condition 1 does not hold, if there exists a domain whose ID and OOD distributions have overlap.

Original: By Theorem 3, we can obtain that the OODdetection is not learnable in the total space D all XY for any non-trivial hypothesis space H .
Revised: By Theorem 3, we can obtain that the OOD detection is not learnable in the total space D all XY for any non-trivial hypothesis space H .

Original: Theorem 4 (Impossibility Theorem for Total Space) .
Revised: Theorem 4 (Impossibility Theorem for Total Space) .

Original: OOD detection is not learnable in the total space D all XY for H , if | ϕ ◦ H| > 1 , where ϕ maps ID labels to 1 and maps OOD labels to 2 .
Revised: OOD detection is not learnable in the total space D all XY for H , if | ϕ ◦ H| > 1 , where ϕ maps ID labels to 1 and maps OOD labels to 2 .

Original: Since the overlaps between ID and OOD distributions may cause that Condition 1 does not hold, wethen consider studying the learnability of OOD detection in the separate space D sXY , where thereare no overlaps between the ID and OOD distributions.
Revised: Since the overlaps between ID and OOD distributions may cause that Condition 1 does not hold, we then consider studying the learnability of OOD detection in the separate space D sXY , where there are no overlaps between the ID and OOD distributions.

Original: However, Theorem 5 shows that even ifwe consider the separate space, the OOD detection is still not learnable in some scenarios.
Revised: However, Theorem 5 shows that even if we consider the separate space, the OOD detection is still not learnable in some scenarios.

Original: Beforeintroducing the impossibility theorem for separate space, i.e. , Theorem 5, we need a mild assumption:213 Assumption 1 (Separate Space for OOD) .
Revised: Before introducing the impossibility theorem for separate space, i.e. , Theorem 5, we need a mild assumption: Assumption 1 (Separate Space for OOD) .

Original: A hypothesis space H is separate for OOD data, if for eachdata point x ∈ X , there exists at least one hypothesis function h x ∈ H such that h x ( x ) =
Revised: A hypothesis space H is separate for OOD data, if for each data point x ∈ X , there exists at least one hypothesis function h x ∈ H such that h x ( x ) =

Original: K + 1 .
Revised: K + 1 .

Original: Assumption 1 means that every data point x has the possibility to be detected as OOD data.
Revised: Assumption 1 means that every data point x has the possibility to be detected as OOD data.

Original: Assumption 1 is mild and can be satisfied by many hypothesis spaces, e.g. , the FCNN-based hypothesis space(Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) anduniversal kernel space.
Revised: Assumption 1 is mild and can be satisfied by many hypothesis spaces, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel space.

Original: Next, we use Vapnik–Chervonenkis (VC) dimension [22] to measure the sizeof hypothesis space, and study the learnability of OOD detection in D sXY based on the VC dimension.
Revised: Next, we use Vapnik–Chervonenkis (VC) dimension [22] to measure the size of hypothesis space, and study the learnability of OOD detection in D sXY based on the VC dimension.

Original: Theorem 5 (Impossibility Theorem for Separate Space) .
Revised: Theorem 5 (Impossibility Theorem for Separate Space) .

Original: If Assumption 1 holds, VCdim( ϕ ◦ H ) <
Revised: If Assumption 1 holds, VCdim( ϕ ◦ H ) <

Original: + ∞ and sup h ∈H |{ x ∈ X : h ( x )
Revised: + ∞ and sup h ∈H |{ x ∈ X : h ( x )

Original: ∈ Y}| =
Revised: ∈ Y}| =

Original: + ∞ , then OOD detection is not learnable in separate space D s XY for H , where ϕ maps ID labels to 1 and maps OOD labels to 2 .
Revised: + ∞ , then OOD detection is not learnable in separate space D sXY for H , where ϕ maps ID labels to 1 and maps OOD labels to 2 .

Original: The finite VC dimension normally implies the learnability of supervised learning.
Revised: The finite VC dimension normally implies the learnability of supervised learning.

Original: However, in ourresults, the finite VC dimension cannot guarantee the learnability of OOD detection in the separatespace, which reveals the difficulty of the OOD detection.
Revised: However, in our results, the finite VC dimension cannot guarantee the learnability of OOD detection in the separate space, which reveals the difficulty of the OOD detection.

Original: Although the above impossibility theoremsare frustrating, there is still room to discuss the conditions in Theorem 5, and to find out the properconditions for ensuring the learnability of OOD detection in the separate space (see Sections 5 and 6).
Revised: Although the above impossibility theorems are frustrating, there is still room to discuss the conditions in Theorem 5, and to find out the proper conditions for ensuring the learnability of OOD detection in the separate space (see Sections 5 and 6).

Original:  When OOD Detection Can Be Successful
Revised:  When OOD Detection Can Be Successful

Original: Here, we discuss when the OOD detection can be successful in the separate space D sXY , finite-IDdistribution space D FXY and density-based space D µ,bXY .
Revised: Here, we discuss when the OOD detection can be successful in the separate space D sXY , finite-IDdistribution space D FXY and density-based space D µ,bXY .

Original: We first study the separate space D sXY .
Revised: We first study the separate space D sXY .

Original: OOD Detection in the Separate Space.
Revised: OOD Detection in the Separate Space.

Original: Theorem 5 has indicated that VCdim( ϕ ◦ H ) =
Revised: Theorem 5 has indicated that VCdim( ϕ ◦ H ) =

Original: + ∞ orsup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ is necessary to ensure the learnability of OOD detection in232 D sXY if Assumption 1 holds.
Revised: + ∞ or sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ is necessary to ensure the learnability of OOD detection in D s XY if Assumption 1 holds.

Original: However, generally, hypothesis spaces generated by feed-forward neuralnetworks with proper activation functions have finite VC dimension [36, 37].
Revised: However, generally, hypothesis spaces generated by feed-forward neural networks with proper activation functions have finite VC dimension [37, 38].

Original: Therefore, we studythe learnability of OOD detection in the case that |X| < + ∞ , which implies that sup h ∈H |{ x ∈ X :h ( x ) ∈ Y}| < + ∞ .
Revised: Therefore, we study the learnability of OOD detection in the case that |X| < + ∞ , which implies that sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ .

Original: Additionally, Theorem 10 also implies that |X| <
Revised: Additionally, Theorem 10 also implies that |X| <

Original: + ∞ is the necessary andsufficient condition for the learnability of OOD detection in separate space, when the hypothesisspace is generated by FCNN.
Revised: + ∞ is the necessary and sufficient condition for the learnability of OOD detection in separate space, when the hypothesis space is generated by FCNN.

Original: Hence, |X| < + ∞ may be necessary in the space D sXY .
Revised: Hence, |X| < + ∞ may be necessary in the space D sXY .

Original: For simplicity, we first discuss the case that K
Revised: For simplicity, we first discuss the case that K

Original: = 1 , i.e. , the one-class novelty detection.
Revised: = 1 , i.e. , the one-class novelty detection.

Original: We show thenecessary and sufficient condition for the learnability of OOD detection in D sXY , when |X| < + ∞ .
Revised: We show the necessary and sufficient condition for the learnability of OOD detection in D sXY , when |X| < + ∞ .

Original: Theorem 6.
Revised: Theorem 6.

Original: Let K = 1 and |X| < + ∞ .
Revised: Let K = 1 and |X| < + ∞ .

Original: Suppose that Assumption 1 holds and the constant function h in := 1 ∈ H .
Revised: Suppose that Assumption 1 holds and the constant function h in := 1 ∈ H .

Original: Then OOD detection is learnable in D sXY for H if and only if H all − { h out } ⊂ H , where H all is the hypothesis space consisting of all hypothesis functions, and h out is a constant function that h out := 2 , here 1 represents ID data and 2 represents OOD data.
Revised: Then OOD detection is learnable in D sXY for H if and only if H all − { h out } ⊂ H , where H all is the hypothesis space consisting of all hypothesis functions, and h out is a constant function that h out := 2 , here 1 represents ID data and 2 represents OOD data.

Original: The condition h in ∈ H presented in Theorem 6 is mild.
Revised: The condition h in ∈ H presented in Theorem 6 is mild.

Original: Many practical hypothesis spaces satisfy thiscondition, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel-based hypothesis space.
Revised: Many practical hypothesis spaces satisfy this condition, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel-based hypothesis space.

Original: Theorem6 implies that if K = 1 and OOD detection is learnable in D sXY for H , then the hypothesis space Hshould contain almost all hypothesis functions, implying that if the OOD detection can be learnablein the distribution-agnostic case, then a large-capacity model is necessary.
Revised: Theorem 6 implies that if K = 1 and OOD detection is learnable in D sXY for H , then the hypothesis space H should contain almost all hypothesis functions, implying that if the OOD detection can be learnable in the distribution-agnostic case, then a large-capacity model is necessary.

Original: Next, we extend Theorem 6 to a general case, i.e. , K > 1 .
Revised: Next, we extend Theorem 6 to a general case, i.e. , K > 1 .

Original: When K > 1 , we will first use a binaryclassifier h b to classify the ID and OOD data.
Revised: When K > 1 , we will first use a binary classifier h b to classify the ID and OOD data.

Original: Then, for the ID data identified by h b , an ID hypothesisfunction h in will be used to classify them into corresponding ID classes.
Revised: Then, for the ID data identified by h b , an ID hypothesis function h in will be used to classify them into corresponding ID classes.

Original: We state this strategy asfollows: given a hypothesis space H in for ID distribution and a binary classification hypothesis space
Revised: We state this strategy as follows: given a hypothesis space H in for ID distribution and a binary classification hypothesis space

Original: H b introduced in Section 2, we use H in and H b to construct an OOD detection’s hypothesis space H ,which consists of all hypothesis functions h satisfying the following condition: there exist h in ∈ H in and h b ∈ H b such that for any x ∈ X,h ( x ) = i, if h in ( x ) = i and h b ( x ) = 1; otherwise ,h ( x ) = K + 1 .
Revised: H b introduced in Section 2, we use H in and H b to construct an OOD detection’s hypothesis space H , which consists of all hypothesis functions h satisfying the following condition: there exist h in ∈ H in and h b ∈ H b such that for any x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: We use H in • H b to represent a hypothesis space consisting of all h defined in Eq.
Revised: We use H in • H b to represent a hypothesis space consisting of all h defined in Eq.

Original: In addition,we also need an additional condition for the loss function ℓ .
Revised: In addition, we also need an additional condition for the loss function ℓ .

Original: This condition is shown as follows:
Revised: This condition is shown as follows:

Original: Condition 2.
Revised: Condition 2.

Original: ℓ ( y 2 , y 1 ) ≤ ℓ ( K + 1 , y 1 ) , for any in-distribution labels y 1 and y 2 ∈ Y .
Revised: ℓ ( y 2 , y 1 ) ≤ ℓ ( K + 1 , y 1 ) , for any in-distribution labels y 1 and y 2 ∈ Y .

Original: Theorem 7.
Revised: Theorem 7.

Original: Let |X| < + ∞ and H
Revised: Let |X| < + ∞ and H

Original: = H in • H b .
Revised: = H in • H b .

Original: If H all − { h out } ⊂ H b and Condition 2 holds,then OOD detection is learnable in D sXY for H , where H all and h out are defined in Theorem 6.
Revised: If H all − { h out } ⊂ H b and Condition 2 holds, then OOD detection is learnable in D sXY for H , where H all and h out are defined in Theorem 6.

Original: OOD Detection in the Finite-ID-Distribution Space.
Revised: OOD Detection in the Finite-ID-Distribution Space.

Original: Since researchers can only collect finite IDdatasets as the training data in the process of algorithm design, it is worthy to study the learnability of
Revised: Since researchers can only collect finite ID datasets as the training data in the process of algorithm design, it is worthy to study the learnability of

Original: OOD detection in the finite-ID-distribution space D FXY .
Revised: OOD detection in the finite-ID-distribution space D FXY .

Original: We first show two necessary concepts below.
Revised: We first show two necessary concepts below.

Original: Definition 5 (ID Consistency) .
Revised: Definition 5 (ID Consistency) .

Original: Given a domain space D XY , we say any two domains D XY ∈ D XYand D ′ XY ∈ D XY are ID consistency, if D X
Revised: Given a domain space D XY , we say any two domains D XY ∈ D XY and D ′ XY ∈ D XY are ID consistency, if D X

Original: I Y I
Revised: I Y I

Original: = D ′
Revised: = D ′

Original: X
Revised: X

Original: I Y
Revised: I Y

Original: I .
Revised: I .

Original: We use the notation ∼ to represent the IDconsistency, i.e., D XY ∼ D ′ XY if and only if D XY and D ′ XY are ID consistency.
Revised: We use the notation ∼ to represent the ID consistency, i.e., D XY ∼ D ′ XY if and only if D XY and D ′ XY are ID consistency.

Original: It is easy to check that the ID consistency ∼ is an equivalence relation.
Revised: It is easy to check that the ID consistency ∼ is an equivalence relation.

Original: Therefore, we define the set[ D XY ] := { D ′ XY ∈ D XY : D XY ∼ D ′ XY } as the equivalence class with respect to space D XY .
Revised: Therefore, we define the set [ D XY ] := { D ′ XY ∈ D XY : D XY ∼ D ′ XY } as the equivalence class with respect to space D XY .

Original: Condition 3 (Compatibility) .
Revised: Condition 3 (Compatibility) .

Original: For any equivalence class [ D ′ XY ] with respect to D XY and any ϵ > 0 ,there exists a hypothesis function h ϵ ∈ H such that for any domain D XY ∈
Revised: For any equivalence class [ D ′ XY ] with respect to D XY and any ϵ > 0 , there exists a hypothesis function h ϵ ∈ H such that for any domain D XY ∈

Original: [ D ′ XY ] ,
Revised: [ D ′ XY ] ,

Original: [Equation]
Revised: [Equation]

Original: In Appendix F, Lemma 2 has implied that Condition 3 is a general version of Condition 1.
Revised: In Appendix F, Lemma 2 has implied that Condition 3 is a general version of Condition 1.

Original: Next,
Revised: Next,

Original: Theorem 8 indicates that Condition 3 is the necessary and sufficient condition in the space D FXY .
Revised: Theorem 8 indicates that Condition 3 is the necessary and sufficient condition in the space D FXY .

Original: Theorem 8.
Revised: Theorem 8.

Original: Suppose that X is a bounded set.
Revised: Suppose that X is a bounded set.

Original: OOD detection is learnable in the finite-IDdistribution space D FXY for H if and only if the compatibility condition (i.e., Condition 3) holds.
Revised: OOD detection is learnable in the finite-IDdistribution space D FXY for H if and only if the compatibility condition (i.e., Condition 3) holds.

Original: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .
Revised: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .

Original: Theorem 8 shows that, in the process of algorithm design, OOD detection cannot be successfulwithout the compatibility condition.
Revised: Theorem 8 shows that, in the process of algorithm design, OOD detection cannot be successful without the compatibility condition.

Original: Theorem 8 also implies that Condition 3 is essential for thelearnability of OOD detection.
Revised: Theorem 8 also implies that Condition 3 is essential for the learnability of OOD detection.

Original: This motivates us to study whether OOD detection can be successfulin more general spaces ( e.g. , the density-based space), when the compatibility condition holds.
Revised: This motivates us to study whether OOD detection can be successful in more general spaces ( e.g. , the density-based space), when the compatibility condition holds.

Original: OOD Detection in the Density-based Space.
Revised: OOD Detection in the Density-based Space.

Original: To ensure that Condition 3 holds, we consider a basicassumption in learning theory— Realizability Assumption , i.e. , for any D XY ∈ D XY , there existsh ∗ ∈ H such that R D ( h ∗ )
Revised: To ensure that Condition 3 holds, we consider a basic assumption in learning theory— Realizability Assumption , i.e. , for any D XY ∈ D XY , there exists h ∗ ∈ H such that R D ( h ∗ )

Original: = 0 .
Revised: = 0 .

Original: We discover that in the density-based space D µ,bXY , Realizability
Revised: We discover that in the density-based space D µ,bXY , Realizability

Original: Assumption can conclude the compatibility condition ( i.e. , Condition 3).
Revised: Assumption can conclude the compatibility condition ( i.e. , Condition 3).

Original: Based on this observation,we can prove the following theorem:
Revised: Based on this observation, we can prove the following theorem:

Original: Theorem 9.
Revised: Theorem 9.

Original: Given a density-based space D µ,b XY , if µ ( X ) < + ∞ , the Realizability Assumption holds, then when H has finite Natarajan dimension [21], OOD detection is learnable in D µ,bXY for
Revised: Given a density-based space D µ,bXY , if µ ( X ) < + ∞ , the Realizability Assumption holds, then when H has finite Natarajan dimension [21], OOD detection is learnable in D µ,bXY for

Original: H .
Revised: H .

Original: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .
Revised: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .

Original: To further investigate the importance and necessary of Realizability Assumption, Theorem 11 hasindicated that in some practical scenarios, Realizability Assumption is the necessary and sufficientcondition for the learnability of OOD detection in the density-based space.
Revised: To further investigate the importance and necessary of Realizability Assumption, Theorem 11 has indicated that in some practical scenarios, Realizability Assumption is the necessary and sufficient condition for the learnability of OOD detection in the density-based space.

Original: Therefore, Realizability
Revised: Therefore, Realizability

Original: Assumption may be indispensable for the learnability of OOD detection in some practical scenarios.
Revised: Assumption may be indispensable for the learnability of OOD detection in some practical scenarios.

Original:  Connecting Theory to Practice
Revised:  Connecting Theory to Practice

Original: In Section 5, we have shown the successful scenarios where OOD detection problem can be addressedin theory.
Revised: In Section 5, we have shown the successful scenarios where OOD detection problem can be addressed in theory.

Original: In this section, we will discuss how the proposed theory is applied to two representativehypothesis spaces—neural-network-based hypothesis spaces and score-based hypothesis spaces.
Revised: In this section, we will discuss how the proposed theory is applied to two representative hypothesis spaces—neural-network-based hypothesis spaces and score-based hypothesis spaces.

Original: Fully-connected Neural Networks.
Revised: Fully-connected Neural Networks.

Original: Given a sequence q
Revised: Given a sequence q

Original: =
Revised: =

Original: ( l 1 , l 2 , ..., l g ) , where l i and g are positiveintegers and g > 2 , we use g to represent the depth of neural network and use l i to represent the width of the i -th layer.
Revised: ( l 1 , l 2 , ..., l g ) , where l i and g are positive integers and g > 2 , we use g to represent the depth of neural network and use l i to represent the width of the i -th layer.

Original: After the activation function σ is selected 1 , we can obtain the architecture of FCNNaccording to the sequence q .
Revised: After the activation function σ is selected 4 , we can obtain the architecture of FCNN according to the sequence q .

Original: Let f w , b be the function generated by FCNN with weights w and biasb .
Revised: Let f w , b be the function generated by FCNN with weights w and bias b .

Original: An FCNN-based scoring function space is defined as: F σ q := { f w , b : ∀ weights w , ∀ bias b } .
Revised: An FCNN-based scoring function space is defined as: F σ q := { f w , b : ∀ weights w , ∀ bias b } .

Original: Inaddition, for simplicity, given any two sequences q = ( l 1 , ..., l g ) and q ′ =
Revised: In addition, for simplicity, given any two sequences q = ( l 1 , ..., l g ) and q ′ =

Original: ( l ′ 1 , ..., l ′ g ′ ) , we use thenotation q ≲ q ′ to represent the following equations and inequalities:
Revised: ( l ′ 1 , ..., l ′ g ′ ) , we use the notation q ≲ q ′ to represent the following equations and inequalities:

Original: [Equation]
Revised: 1) g ≤ g ′ , l 1 = l ′ 1 , l g = l ′ g ′ ; 2) l i ≤ l ′ i , ∀ i = 1 , ..., g − 1; and 3) l g − 1 ≤ l ′ i , ∀ i = g, ..., g ′ − 1 .

Original: In Appendix L, Lemma 10 shows q ≲ q ′ ⇒ F σ q ⊂ F σ q ′ .
Revised: In Appendix L, Lemma 10 shows q ≲ q ′ ⇒ F σ q ⊂ F σ q ′ .

Original: We use ≲ to compare the sizes of FCNNs.
Revised: We use ≲ to compare the sizes of FCNNs.

Original: FCNN-based Hypothesis Space.
Revised: FCNN-based Hypothesis Space.

Original: Let l g
Revised: Let l g

Original: =
Revised: =

Original: K
Revised: K

Original: + 1 .
Revised: + 1 .

Original: The FCNN-based scoring function space F σ q caninduce an FCNN-based hypothesis space.
Revised: The FCNN-based scoring function space F σ q can induce an FCNN-based hypothesis space.

Original: For any f w , b ∈ F σ q , the induced hypothesis function is:[Equation] b
Revised: For any f w , b ∈ F σ q , the induced hypothesis function is:

Original: [Equation]
Revised: [Equation]

Original: k w , b , where f k w , b is the k -th coordinate of f w , b .
Revised: k w , b , where f k w , b is the k -th coordinate of f w , b .

Original: Then, the FCNN-based hypothesis space is defined as H σ q := { h w , b : ∀ weights w , ∀ bias b } .
Revised: Then, the FCNN-based hypothesis space is defined as H σ q := { h w , b : ∀ weights w , ∀ bias b } .

Original: Score-based Hypothesis Space.
Revised: Score-based Hypothesis Space.

Original: Many OOD detection algorithms detect OOD data by using ascore-based strategy.
Revised: Many OOD detection algorithms detect OOD data by using a score-based strategy.

Original: That is, given a threshold λ , a scoring function space F l ⊂ { f : X → R l }and a scoring function E : F l → R , then x is regarded as ID data if and only if E ( f ( x )) ≥ λ .
Revised: That is, given a threshold λ , a scoring function space F l ⊂ { f : X → R l } and a scoring function E : F l → R , then x is regarded as ID data if and only if E ( f ( x )) ≥ λ .

Original: Weintroduce several representative scoring functions E as follows: for any f = [ f 1 , ..., f l ] ⊤ [Equation] F [Equation]
Revised: We introduce several representative scoring functions E as follows: for any f = [ f 1 , ..., f l ] ⊤ ∈ F l ,

Original: • softmax-based function [7] and temperature-scaled function [8]: λ ∈ ( 1 l, 1) and T >0 ,
Revised: • softmax-based function [7] and temperature-scaled function [ : λ ∈ ( 1 l and T >

Original: [Equation]
Revised: [Equation]

Original: • energy-based function [23]: λ ∈ (0 , + ∞ ) and T > 0 ,
Revised: • energy-based function [23]: λ ∈ (0 , + ∞ ) and T > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Using E , λ and f ∈ F σ q , we have a classifier:
Revised: Using E , λ and f ∈ F σ q , we have a classifier:

Original: h λ f ,E ( x ) = 1 , if E ( f ( x )) ≥ λ ; otherwise, h λ f ,E ( x ) = 2 ,where 1 represents the ID data and 2 represents the OOD data.
Revised: h λ f ,E ( x ) = 1 , if E ( f ( x )) ≥ λ ; otherwise, h λ f ,E ( x ) = 2 , where 1 represents the ID data and 2 represents the OOD data.

Original: Hence, a binary classificationhypothesis space H b , which consists of all h λ f ,E , is generated.
Revised: Hence, a binary classification hypothesis space H b , which consists of all h λ f ,E , is generated.

Original: We define H σ,λ q ,E :=
Revised: We define H σ,λ q ,E :=

Original: { h λ f ,E : ∀ f ∈ F σ q } .
Revised: { h λ f ,E : ∀ f ∈ F σ q } .

Original: Learnability of OOD Detection in Different Hypothesis Spaces.
Revised: Learnability of OOD Detection in Different Hypothesis Spaces.

Original: Next, we present applications ofour theory regarding the above two practical and important hypothesis spaces H σ q and H σ,λ q ,E .
Revised: Next, we present applications of our theory regarding the above two practical and important hypothesis spaces H σ q and H σ,λ q ,E .

Original: Theorem 10.
Revised: Theorem 10.

Original: Suppose that Condition 2 holds and the hypothesis space H is FCNN-based or scorebased
Revised: Suppose that Condition 2 holds and the hypothesis space H is FCNN-based or scorebased

Original: , i.e., H = H σ q or H =
Revised: , i.e., H = H σ q or H =

Original: H in • H b , where H in is an ID hypothesis space, H b
Revised: H in • H b , where H in is an ID hypothesis space, H b

Original: = H σ,λ q ,E and
Revised: = H σ,λ q ,E and

Original: H = H in • H b is introduced below Eq.
Revised: H = H in • H b is introduced below Eq.

Original: (4) , here E is introduced in Eqs.
Revised: (4) , here E is introduced in Eqs.

Original: (5) or (6) .
Revised: (5) or (6) .

Original: Then
Revised: Then

Original: There is a sequence q =
Revised: There is a sequence q =

Original: ( l 1 , ..., l g )
Revised: ( l 1 , ..., l g )

Original: such that OOD detection is learnable in the separate space D sXY for H if and only if |X| < + ∞ .
Revised: such that OOD detection is learnable in the separate space D sXY for H if and only if |X| < + ∞ .

Original: Furthermore, if |X| < + ∞ , then there exists a sequence q =
Revised: Furthermore, if |X| < + ∞ , then there exists a sequence q =

Original: ( l 1 , ..., l g ) such that for any sequenceq ′ satisfying that q ≲ q ′ , OOD detection is learnable in D s XY for H .
Revised: ( l 1 , ..., l g ) such that for any sequence q ′ satisfying that q ≲ q ′ , OOD detection is learnable in D sXY for H .

Original: Theorem 10 states that 1) when the hypothesis space is FCNN-based or score-based, the finite featurespace is the necessary and sufficient condition for the learnability of OOD detection in the separatespace; and 2) a larger architecture of FCNN has a greater probability to achieve the learnability of
Revised: Theorem 10 states that 1) when the hypothesis space is FCNN-based or score-based, the finite feature space is the necessary and sufficient condition for the learnability of OOD detection in the separate space; and 2) a larger architecture of FCNN has a greater probability to achieve the learnability of

Original: OOD detection in the separate space.
Revised: OOD detection in the separate space.

Original: Note that when we select Eqs.
Revised: Note that when we select Eqs.

Original: (5) or (6) as the scoring function
Revised: (5) or (6) as the scoring function

Original: E , Theorem 10 also shows that the selected scoring functions E can guarantee the learnability of
Revised: E , Theorem 10 also shows that the selected scoring functions E can guarantee the learnability of

Original: OOD detection, which is a theoretical support for the representative works [8, 23, 7].
Revised: OOD detection, which is a theoretical support for the representative works [8, 23, 7].

Original: Furthermore,
Revised: Furthermore,

Original: Theorem 11 also offers theoretical supports for these works in the density-based space, when K = 1 .
Revised: Theorem 11 also offers theoretical supports for these works in the density-based space, when K = 1 .

Original: Theorem 11.
Revised: Theorem 11.

Original: Suppose that each domain D XY in D µ,bXY is attainable, i.e., arg min
Revised: Suppose that each domain D XY in D µ,bXY is attainable, i.e., arg min

Original: h ∈H R D ( h ) ̸
Revised: h ∈H R D ( h ) ̸

Original: = ∅(the finite discrete domains satisfy this).
Revised: = ∅ (the finite discrete domains satisfy this).

Original: Let K = 1 and the hypothesis space H be score-based(
Revised: Let K = 1 and the hypothesis space H be score-based (

Original: H = H σ,λ q ,E , where E is in Eqs.
Revised: H = H σ,λ q ,E , where E is in Eqs.

Original: (5) or (6) ) or FCNN-based ( H = H σ q ) .
Revised: (5) or (6) ) or FCNN-based ( H = H σ q ) .

Original: If µ ( X ) < + ∞ , then thefollowing four conditions are equivalent :
Revised: If µ ( X ) < + ∞ , then the following four conditions are equivalent :

Original: [-Learnability in D µ,bXY for H ⇐⇒ Condition 1 ⇐⇒ Realizability Assumption ⇐⇒ Condition 
-]
Revised: [-Learnability in D µ,bXY for H ⇐⇒ Condition 1 ⇐⇒ Realizability Assumption ⇐⇒ Condition 
-][Equation]

Original: Theorem 11 still holds if the function space F σ q is generated by Convolutional Neural Network.
Revised: Theorem 11 still holds if the function space F σ q is generated by Convolutional Neural Network.

Original: Overlap and Benefits of Multi-class Case.
Revised: Overlap and Benefits of Multi-class Case.

Original: We investigate when the hypothesis space is FCNN-basedor score-based, what will happen if there exists an overlap between the ID and OOD distributions?
Revised: We investigate when the hypothesis space is FCNN-based or score-based, what will happen if there exists an overlap between the ID and OOD distributions?

Original: Theorem 12.
Revised: Theorem 12.

Original: Let K = 1 and the hypothesis space H be score-based ( H = H σ,λ q ,E , where E is in
Revised: Let K = 1 and the hypothesis space H be score-based ( H = H σ,λ q ,E , where E is in

Original: Eqs.
Revised: Eqs.

Original: (5) or (6) ) or FCNN-based ( H = H σ q ) .
Revised: (5) or (6) ) or FCNN-based ( H = H σ q ) .

Original: Given a prior-unknown space D XY , if there exists adomain D XY ∈ D XY , which has an overlap between ID and OOD distributions (see Definition 4),then OOD detection is not learnable in the domain space D XY for H .
Revised: Given a prior-unknown space D XY , if there exists a domain D XY ∈ D XY , which has an overlap between ID and OOD distributions (see Definition 4), then OOD detection is not learnable in the domain space D XY for H .

Original: When K = 1 and the hypothesis space is FCNN-based or score-based, Theorem 12 shows that overlapbetween ID and OOD distributions is the sufficient condition for the unlearnability of OOD detection.
Revised: When K = 1 and the hypothesis space is FCNN-based or score-based, Theorem 12 shows that overlap between ID and OOD distributions is the sufficient condition for the unlearnability of OOD detection.

Original: Theorem 12 takes roots in the conditions inf h ∈H R in D ( h )
Revised: Theorem 12 takes roots in the conditions inf h ∈H R in D ( h )

Original: = 

Revised: =

Original: and inf h ∈H R out D ( h ) = 0 .
Revised: and inf h ∈H R out D ( h ) = 0 .

Original: However,when K > 1 , we can ensure inf h ∈H R in D ( h ) > 

Revised: However, when K > 1 , we can ensure inf h ∈H R in D ( h ) >

Original: if ID distribution D X
Revised: if ID distribution D X

Original: I Y
Revised: I Y

Original: I has overlap between IDclasses.
Revised: I has overlap between ID classes.

Original: By this observation, we conjecture that when K > 1 , OOD detection is learnable in somespecial cases where overlap exists, even if the hypothesis space is FCNN-based or score-based.
Revised: By this observation, we conjecture that when K > 1 , OOD detection is learnable in some special cases where overlap exists, even if the hypothesis space is FCNN-based or score-based.

Original:  Related Work
Revised:  Related Work

Original: We briefly review the related theoretical works below.
Revised: We briefly review the related theoretical works below.

Original: See Appendix A for detailed related works.
Revised: See Appendix A for detailed related works.

Original: OOD Detection Theory.
Revised: OOD Detection Theory.

Original: [38] understands the OOD detection via goodness-of-fit tests and typicalset hypothesis, and argues that minimal density estimation errors can lead to OOD detection failures,when there exists an overlap between ID and OOD distributions.
Revised: [39] understands the OOD detection via goodness-of-fit tests and typical set hypothesis, and argues that minimal density estimation errors can lead to OOD detection failures, when there exists an overlap between ID and OOD distributions.

Original: Beyond [38], [39] paves a newavenue to designing provable OOD detection algorithms.
Revised: Beyond [39], [40] paves a new avenue to designing provable OOD detection algorithms.

Original: Compared to [39, 38], our theory focuseson theagnostic PAC learnable theory of OOD detection and identifies several necessary and sufficientconditions for the learnability of OOD detection, opening a door to study OOD detection in theory.
Revised: Compared to [40, 39], our theory focuses on the PAC learnable theory of OOD detection and identifies several necessary and sufficient conditions for the learnability of OOD detection, opening a door to study OOD detection in theory.

Original: Open-set Learning Theory.
Revised: Open-set Learning Theory.

Original: [40] and [29, 41] propose the agnostic PAC learning bounds for open-setdetection and open-set domain adaptation, respectively.
Revised: [41] and [29, 42] propose the agnostic PAC learning bounds for open-set detection and open-set domain adaptation, respectively.

Original: Unfortunately, [29, 40, 41] all require thatthe test data are indispensable during the training process.
Revised: Unfortunately, [29, 41, 42] all require that the test data are indispensable during the training process.

Original: To investigate open-set learning (OSL)without accessing the test data during training, [24] proposes and investigates the almost agnostic
Revised: To investigate open-set learning (OSL) without accessing the test data during training, [24] proposes and investigates the almost agnostic

Original: PAC learnability for OSL.
Revised: PAC learnability for OSL.

Original: However, the assumptions used in [24] are very strong and unpractical.
Revised: However, the assumptions used in [24] are very strong and unpractical.

Original: Learning Theory for Classification with Reject Option.
Revised: Learning Theory for Classification with Reject Option.

Original: Many works [42, 43] also investigate theclassification with reject option (CwRO) problem, which is similar to OOD detection in some cases.[44, 45, 46, 47, 48] study the learning theory and propose theagnostic PAC learning bounds for CwRO.
Revised: Many works [43, 44] also investigate the classification with reject option (CwRO) problem, which is similar to OOD detection in some cases. [45, 46, 47, 48, 49] study the learning theory and propose the PAC learning bounds for CwRO.

Original: However, compared to our work regarding OOD detection, existing CwRO theories mainlyfocus on how the ID risk R in D ( i.e. , the risk that ID data is wrongly classified) is influenced by specialrejection rules.
Revised: However, compared to our work regarding OOD detection, existing CwRO theories mainly focus on how the ID risk R in D ( i.e. , the risk that ID data is wrongly classified) is influenced by special rejection rules.

Original: Our theory not only focuses on the ID risk, but also pays attention to the OOD risk.
Revised: Our theory not only focuses on the ID risk, but also pays attention to the OOD risk.

Original: PQ Learning Theory.
Revised: PQ Learning Theory.

Original: Under some conditions, PQ learning theory [49, 50] can be regarded as the
Revised: Under some conditions, PQ learning theory [50, 51] can be regarded as the

Original: PAC theory for OOD detection in the semi-supervised or transductive learning cases, i.e. , test dataare required during training.
Revised: PAC theory for OOD detection in the semi-supervised or transductive learning cases, i.e. , test data are required during training.

Original: Besides, [49, 50] aim to give the PAC estimation under Realizability
Revised: Besides, [50, 51] aim to give the PAC estimation under Realizability

Original: Assumption [21].
Revised: Assumption [21].

Original: Our theory does not only study the PAC estimation in the realization cases, but alsostudies the agnostic cases, which are more difficult than PAC theory under Realizability Assumption.
Revised: Our theory does not only study the PAC estimation in the realization cases, but also studies the other cases, which are more difficult than PAC theory under Realizability Assumption.

Original:  Conclusions
Revised:  Conclusions

Original: Detecting OOD data has shown its significance in improving the reliability of machine learning.
Revised: Detecting OOD data has shown its significance in improving the reliability of machine learning.

Original: However, very few works discuss OOD detection in theory, which hinders real-world applicationsof OOD detection algorithms.
Revised: However, very few works discuss OOD detection in theory, which hinders real-world applications of OOD detection algorithms.

Original: In this paper, we are the first to provide theagnostic PAC theory for OOD detection.
Revised: In this paper, we are the first to provide the PAC theory for OOD detection.

Original: Our results imply that we cannot expect a universally consistent algorithm to handleall scenarios in OOD detection.
Revised: Our results imply that we cannot expect a universally consistent algorithm to handle all scenarios in OOD detection.

Original: Yet, it is still possible to make OOD detection learnable in certainscenarios.
Revised: Yet, it is still possible to make OOD detection learnable in certain scenarios.

Original: For example, when the ID and OOD images have clearly different semantic meanings,

Original: Theorems 10 and 11 show that the image-based OOD detection is learnable for some practicalhypothesis spaces.

Original: In addition, when we design OOD detection algorithms, we normally only havefinite ID datasets.
Revised: For example, when we design OOD detection algorithms, we normally only have finite ID datasets.

Original: In this real scenario, Theorem 8 provides a necessary and sufficient condition forthe success of OOD detection.
Revised: In this real scenario, Theorem 8 provides a necessary and sufficient condition for the success of OOD detection.

Original: Our theory reveals many necessary and sufficient condition for thelearnability of OOD detection, hence opening a door to studying the learnability of OOD detection. 
Revised: Our theory reveals many necessary and sufficient condition for the learnability of OOD detection, hence opening a door to studying the learnability of OOD detection. Acknowledgment

Revised: JL and ZF were supported by the Australian Research Council (ARC) under FL190100149.

Revised: YXL was supported by the Wisconsin Alumni Research Foundation (WARF).

Revised: BH was supported by the

Revised: RGC Early Career Scheme No. 22200720 and NSFC Young Scientists Fund

Revised: No. 62006202.

Revised: The authors would also like to thank NeurIPS reviewers for the helpful suggestions and feedback.

Revised: ZF would also like to thank Dr. Tongliang Liu for productive discussions.