Original: Is Out-of-Distribution Detection Learnable?
Revised: Is Out-of-Distribution Detection Learnable?

Original: Abstract
Revised: Abstract

Original: Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution.
Revised: Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution.

Original: To ease the above assumption, researchers have studied a more realistic setting: out-of-distribution (OOD) detection, where test data may come from classes that are unknown during training ( i.e. , OOD data).
Revised: To ease the above assumption, researchers have studied a more realistic setting: out-of-distribution (OOD) detection, where test data may come from classes that are unknown during training ( i.e. , OOD data).

Original: Due to the unavailability and diversity of OOD data, good generalization ability is crucial for effective OOD detection algorithms.
Revised: Due to the unavailability and diversity of OOD data, good generalization ability is crucial for effective OOD detection algorithms.

Original: To study the generalization of
Revised: To study the generalization of

Original: OOD detection, in this paper, we investigate the probably approximately correct (PAC) learning theory of OOD detection, which is proposed by researchers as an open problem .
Revised: OOD detection, in this paper, we investigate the probably approximately correct (PAC) learning theory of OOD detection, which is proposed by researchers as an open problem .

Original: First, we find a necessary condition for the learnability of OOD detection.
Revised: First, we find a necessary condition for the learnability of OOD detection.

Original: Then, using this condition, we prove several impossibility theorems for the learnability of OOD detection under some scenarios.
Revised: Then, using this condition, we prove several impossibility theorems for the learnability of OOD detection under some scenarios.

Original: Although the impossibility theorems are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios.
Revised: Although the impossibility theorems are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios.

Original: Based on this observation, we next give several necessary and sufficient conditions to characterize the learnability of OOD detection in some practical scenarios.
Revised: Based on this observation, we next give several necessary and sufficient conditions to characterize the learnability of OOD detection in some practical scenarios.

Original: Lastly, we also offer theoretical supports for several representative OOD detection works based on our OOD theory.
Revised: Lastly, we also offer theoretical supports for several representative OOD detection works based on our OOD theory.

Original:  Introduction
Revised:  Introduction

Original: The success of supervised learning is established on an implicit assumption that training and test data share a same distribution, i e. , in-distribution (ID) [1, 2, 3, 4].
Revised: The success of supervised learning is established on an implicit assumption that training and test data share a same distribution, i e. , in-distribution (ID) [1, 2, 3, 4].

Original: However, test data distribution in many real-world scenarios may violate the assumption and, instead, contain out-of-distribution (OOD) data whose labels have not been seen during the training process [5, 6].
Revised: However, test data distribution in many real-world scenarios may violate the assumption and, instead, contain out-of-distribution (OOD) data whose labels have not been seen during the training process [5, 6].

Original: To mitigate the risk of OOD data, researchers have considered a more practical learning scenario:
Revised: To mitigate the risk of OOD data, researchers have considered a more practical learning scenario:

Original: OOD detection which determines whether an input is ID/OOD, while classifying the ID data into respective classes.
Revised: OOD detection which determines whether an input is ID/OOD, while classifying the ID data into respective classes.

Original: OOD detection has shown great potential to ensure the reliable deployment of machine learning models in the real world.
Revised: OOD detection has shown great potential to ensure the reliable deployment of machine learning models in the real world.

Original: A rich line of algorithms have been developed to empirically address the OOD detection problem [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].
Revised: A rich line of algorithms have been developed to empirically address the OOD detection problem [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].

Original: However, very few works study theory of OOD detection, which hinders the rigorous path forward for the field.
Revised: However, very few works study theory of OOD detection, which hinders the rigorous path forward for the field.

Original: This paper aims to bridge the gap.
Revised: This paper aims to bridge the gap.

Original: In this paper, we provide a theoretical framework to understand the learnability of the OOD detection problem.
Revised: In this paper, we provide a theoretical framework to understand the learnability of the OOD detection problem.

Original: We investigate the probably approximately correct (PAC) learning theory of OOD detection, which is posed as an open problem to date.
Revised: We investigate the probably approximately correct (PAC) learning theory of OOD detection, which is posed as an open problem to date.

Original: Unlike the classical PAC learning theory in a supervised setting, our problem setting is fundamentally challenging due to the absence of OOD data in training.
Revised: Unlike the classical PAC learning theory in a supervised setting, our problem setting is fundamentally challenging due to the absence of OOD data in training.

Original: In many real-world scenarios, OOD data can be diverse and priori-unknown.
Revised: In many real-world scenarios, OOD data can be diverse and priori-unknown.

Original: Given this, we study whether there exists an algorithm that can be used to detect various OOD data instead of merely some specified OOD data.
Revised: Given this, we study whether there exists an algorithm that can be used to detect various OOD data instead of merely some specified OOD data.

Original: Such is the significance of studying the learning theory for OOD detection [4].
Revised: Such is the significance of studying the learning theory for OOD detection [4].

Original: This motivates our question: is OOD detection PAC learnable?
Revised: This motivates our question: is OOD detection PAC learnable?

Original: i.e., is there the PAC learning theory to guarantee the generalization ability of OOD detection?
Revised: i.e., is there the PAC learning theory to guarantee the generalization ability of OOD detection?

Original: To investigate the learning theory, we mainly focus on two basic spaces: domain space and hypothesis space.
Revised: To investigate the learning theory, we mainly focus on two basic spaces: domain space and hypothesis space.

Original: The domain space is a space consisting of some distributions, and the hypothesis space is a space consisting of some classifiers.
Revised: The domain space is a space consisting of some distributions, and the hypothesis space is a space consisting of some classifiers.

Original: Existing agnostic PAC theories in supervised learning [21, 22] are distribution-free, i.e. , the domain space consists of all domains.
Revised: Existing agnostic PAC theories in supervised learning [21, 22] are distribution-free, i.e. , the domain space consists of all domains.

Original: Yet, in Theorem 4, we shows that the learning theory of OOD detection is not distribution-free.
Revised: Yet, in Theorem 4, we shows that the learning theory of OOD detection is not distribution-free.

Original: In fact, we discover that OOD detection is learnable only if the domain space and the hypothesis space satisfy some special conditions, e.g.,
Revised: In fact, we discover that OOD detection is learnable only if the domain space and the hypothesis space satisfy some special conditions, e.g.,

Original: Conditions 1 and 3.
Revised: Conditions 1 and 3.

Original: Notably, there are many conditions and theorems in existing learning theories and many OOD detection algorithms in the literature.
Revised: Notably, there are many conditions and theorems in existing learning theories and many OOD detection algorithms in the literature.

Original: Thus, it is very difficult to analyze the relation between these theories and algorithms, and explore useful conditions to ensure the learnability of
Revised: Thus, it is very difficult to analyze the relation between these theories and algorithms, and explore useful conditions to ensure the learnability of

Original: OOD detection, especially when we have to explore them from the scratch .
Revised: OOD detection, especially when we have to explore them from the scratch .

Original: Thus, the main aim of our paper is to study these essential conditions.
Revised: Thus, the main aim of our paper is to study these essential conditions.

Original: From these essential conditions, we can know when OOD detection can be successful in practical scenarios.
Revised: From these essential conditions, we can know when OOD detection can be successful in practical scenarios.

Original: We restate our question and goal in following:
Revised: We restate our question and goal in following:

Original: Given hypothesis spaces and several representative domain spaces, what are the conditions to ensure the learnability of OOD detection?
Revised: Given hypothesis spaces and several representative domain spaces, what are the conditions to ensure the learnability of OOD detection?

Original: If possible, we hope that these conditions are necessary and sufficient in some scenarios.
Revised: If possible, we hope that these conditions are necessary and sufficient in some scenarios.

Original: Main Results.
Revised: Main Results.

Original: We investigate the learnability of OOD detection starting from the largest space—the total space, and give a necessary condition (Condition 1) for the learnability.
Revised: We investigate the learnability of OOD detection starting from the largest space—the total space, and give a necessary condition (Condition 1) for the learnability.

Original: However, we find that the overlap between ID and OOD data may result in that the necessary condition does not hold.
Revised: However, we find that the overlap between ID and OOD data may result in that the necessary condition does not hold.

Original: Therefore, we give an impossibility theorem to demonstrate that OOD detection fails in the total space (Theorem 4).
Revised: Therefore, we give an impossibility theorem to demonstrate that OOD detection fails in the total space (Theorem 4).

Original: Next, we study OOD detection in the separate space, where there are no overlaps between the ID and OOD data.
Revised: Next, we study OOD detection in the separate space, where there are no overlaps between the ID and OOD data.

Original: Unfortunately, there still exists impossibility theorem (Theorem 5), which demonstrates that OOD detection is not learnable in the separate space under some conditions.
Revised: Unfortunately, there still exists impossibility theorem (Theorem 5), which demonstrates that OOD detection is not learnable in the separate space under some conditions.

Original: Although the impossibility theorems obtained in the separate space are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios.
Revised: Although the impossibility theorems obtained in the separate space are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios.

Original: Based on this observation, we give several necessary and sufficient conditions to characterize the learnability of
Revised: Based on this observation, we give several necessary and sufficient conditions to characterize the learnability of

Original: OOD detection in the separate space (Theorems 6 and 10).
Revised: OOD detection in the separate space (Theorems 6 and 10).

Original: Especially, when our model is based on fully-connected neural network (FCNN), OOD detection is learnable in the separate space if and only if the feature space is finite.
Revised: Especially, when our model is based on fully-connected neural network (FCNN), OOD detection is learnable in the separate space if and only if the feature space is finite.

Original: Furthermore, we investigate the learnability of OOD detection in other more practical domain spaces, e.g. , the finite-ID-distribution space (Theorem 8) and the densitybased space (Theorem 9).
Revised: Furthermore, we investigate the learnability of OOD detection in other more practical domain spaces, e.g. , the finite-ID-distribution space (Theorem 8) and the densitybased space (Theorem 9).

Original: By studying the finite-ID-distribution space, we discover a compatibility condition (Condition 3) that is a necessary and sufficient condition for this space.
Revised: By studying the finite-ID-distribution space, we discover a compatibility condition (Condition 3) that is a necessary and sufficient condition for this space.

Original: Next, we further investigate the compatibility condition in the density-based space, and find that such condition is also the necessary and sufficient condition in some practical scenarios (Theorem 11).
Revised: Next, we further investigate the compatibility condition in the density-based space, and find that such condition is also the necessary and sufficient condition in some practical scenarios (Theorem 11).

Original: Implications and Impacts of Theory.
Revised: Implications and Impacts of Theory.

Original: Our study is not of purely theoretical interest; it has also practical impacts.
Revised: Our study is not of purely theoretical interest; it has also practical impacts.

Original: First, when we design OOD detection algorithms, we normally only have finite ID datasets, corresponding to the finite-ID-distribution space.
Revised: First, when we design OOD detection algorithms, we normally only have finite ID datasets, corresponding to the finite-ID-distribution space.

Original: In this case, Theorem 8 gives the necessary and sufficient condition to the success of OOD detection.
Revised: In this case, Theorem 8 gives the necessary and sufficient condition to the success of OOD detection.

Original: Second, our theory provides theoretical support (Theorems 10 and 11) for several representative OOD detection works [7, 8, 23].
Revised: Second, our theory provides theoretical support (Theorems 10 and 11) for several representative OOD detection works [7, 8, 23].

Original: Third, our theory shows that OOD detection is learnable in image-based scenarios when ID images have clearly different semantic labels and styles ( far-OOD ) from OOD images.
Revised: Third, our theory shows that OOD detection is learnable in image-based scenarios when ID images have clearly different semantic labels and styles ( far-OOD ) from OOD images.

Original: Fourth, we should not expect a universally working algorithm.
Revised: Fourth, we should not expect a universally working algorithm.

Original: It is necessary to design different algorithms in different scenarios.
Revised: It is necessary to design different algorithms in different scenarios.

Original:  Learning Setups
Revised:  Learning Setups

Original: We start by introducing the necessary concepts and notations for our theoretical framework.
Revised: We start by introducing the necessary concepts and notations for our theoretical framework.

Original: Given a feature space X ⊂ R d and a label space Y :
Revised: Given a feature space X ⊂ R d and a label space Y :

Original: = { 1 , ..., K } , we have an ID joint distribution D X
Revised: = { 1 , ..., K } , we have an ID joint distribution D X

Original: I Y I over X × Y , where X I ∈
Revised: I Y I over X × Y , where X I ∈

Original: X and Y I ∈ Y are random variables.
Revised: X and Y I ∈ Y are random variables.

Original: We also have an OOD joint distribution D X O Y O , where X O is a random variable from X , but Y O is a random variable whose outputs do not belong to Y .
Revised: We also have an OOD joint distribution D X O Y O , where X O is a random variable from X , but Y O is a random variable whose outputs do not belong to Y .

Original: During testing, we will meet a mixture of ID and OOD joint distributions:
Revised: During testing, we will meet a mixture of ID and OOD joint distributions:

Original: D XY :
Revised: D XY :

Original: = (1 − π out ) D X
Revised: = (1 − π out ) D X

Original: I Y
Revised: I Y

Original: I + π out D X
Revised: I + π out D X

Original: O Y O ,
Revised: O Y O ,

Original: and can only observe the marginal distribution D X :=
Revised: and can only observe the marginal distribution D X :=

Original: (1 − π out ) D X
Revised: (1 − π out ) D X

Original: I + π out D X
Revised: I + π out D X

Original: O , where the constant π out ∈ [0 , 1) is an unknown class-prior probability.
Revised: O , where the constant π out ∈ [0 , 1) is an unknown class-prior probability.

Original: Problem 1 (OOD Detection [4]) .
Revised: Problem 1 (OOD Detection [4]) .

Original: Given an ID joint distribution D X I Y I and a training data S :
Revised: Given an ID joint distribution D X I Y I and a training data S :

Original: = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn independent and identically distributed from D X
Revised: = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn independent and identically distributed from D X

Original: I Y
Revised: I Y

Original: I , the aim of OOD detection is to train a classifier f by using the training data S such that, for any test data x drawn from the mixed marginal distribution D X : 1) if x is an observation from D X I , f can classify x into correct ID classes; and 2) if x is an observation from D X O , f can detect x as OOD data.
Revised: I , the aim of OOD detection is to train a classifier f by using the training data S such that, for any test data x drawn from the mixed marginal distribution D X : 1) if x is an observation from D X I , f can classify x into correct ID classes; and 2) if x is an observation from D X O , f can detect x as OOD data.

Original: According to the survey [4], when K > 1 , OOD detection is also known as the open-set recognition or open-set learning [24, 25]; and when K = 1 , OOD detection reduces to one-class novelty detection and semantic anomaly detection [26, 27, 28].
Revised: According to the survey [4], when K > 1 , OOD detection is also known as the open-set recognition or open-set learning [24, 25]; and when K = 1 , OOD detection reduces to one-class novelty detection and semantic anomaly detection [26, 27, 28].

Original: OOD Label and Domain Space.
Revised: OOD Label and Domain Space.

Original: Based on Problem 1, we know it is not necessary to classify OOD data into the correct OOD classes.
Revised: Based on Problem 1, we know it is not necessary to classify OOD data into the correct OOD classes.

Original: Without loss of generality, let all OOD data be allocated to one big OOD class, i.e. , Y O = K + 1 [24, 29].
Revised: Without loss of generality, let all OOD data be allocated to one big OOD class, i.e. , Y O = K + 1 [24, 29].

Original: To investigate the PAC learnability of OOD detection, we define a domain space D XY , which is a set consisting of some joint distributions D XY mixed by some ID joint distributions and some OOD joint distributions.
Revised: To investigate the PAC learnability of OOD detection, we define a domain space D XY , which is a set consisting of some joint distributions D XY mixed by some ID joint distributions and some OOD joint distributions.

Original: In this paper, the joint distribution
Revised: In this paper, the joint distribution

Original: D XY mixed by ID joint distribution D X
Revised: D XY mixed by ID joint distribution D X

Original: I Y I and OOD joint distribution D X
Revised: I Y I and OOD joint distribution D X

Original: O Y O is called domain .
Revised: O Y O is called domain .

Original: Hypothesis Spaces and Scoring Function Spaces.
Revised: Hypothesis Spaces and Scoring Function Spaces.

Original: A hypothesis space H is a subset of function space, i.e. , H ⊂ { h : X → Y ∪ { K + 1 }} .
Revised: A hypothesis space H is a subset of function space, i.e. , H ⊂ { h : X → Y ∪ { K + 1 }} .

Original: We set H in ⊂ { h : X → Y} to the ID hypothesis space.
Revised: We set H in ⊂ { h : X → Y} to the ID hypothesis space.

Original: We also define H b ⊂ { h : X → { 1 , 2 }} as the hypothesis space for binary classification, where 1 represents the ID data, and 2 represents the OOD data.
Revised: We also define H b ⊂ { h : X → { 1 , 2 }} as the hypothesis space for binary classification, where 1 represents the ID data, and 2 represents the OOD data.

Original: The function h is called the hypothesis function.
Revised: The function h is called the hypothesis function.

Original: A scoring function space is a subset of function space, i.e. , F l ⊂ { f : X → R l } , where l is the output’s dimension of the vector-valued function f .
Revised: A scoring function space is a subset of function space, i.e. , F l ⊂ { f : X → R l } , where l is the output’s dimension of the vector-valued function f .

Original: The function f is called the scoring function.
Revised: The function f is called the scoring function.

Original: Loss and Risks.
Revised: Loss and Risks.

Original: Let Y all =
Revised: Let Y all =

Original: Y ∪ { K + 1 } .
Revised: Y ∪ { K + 1 } .

Original: Given a loss function ℓ 2 : Y all × Y all → R ≥ 0 satisfying that ℓ ( y 1 , y 2 )
Revised: Given a loss function ℓ 2 : Y all × Y all → R ≥ 0 satisfying that ℓ ( y 1 , y 2 )

Original: =
Revised: =

Original: if and only if y 1 = y 2 , and any h ∈ H , then the risk with respect to D XY is
Revised: if and only if y 1 = y 2 , and any h ∈ H , then the risk with respect to D XY is

Original: [Equation]
Revised: [Equation]

Original: The α -risk R αD ( h ) :
Revised: The α -risk R αD ( h ) :

Original: = (1 − α ) R in D ( h ) + αR out D ( h ) , ∀ α ∈ [0 , 1] , where the risks R in D ( h ) , R out D ( h ) are
Revised: = (1 − α ) R in D ( h ) + αR out D ( h ) , ∀ α ∈ [0 , 1] , where the risks R in D ( h ) , R out D ( h ) are

Original: [Equation]
Revised: [Equation]

Original: Learnability.
Revised: Learnability.

Original: We aim to select a hypothesis function h ∈ H with approximately minimal risk, based on finite data.
Revised: We aim to select a hypothesis function h ∈ H with approximately minimal risk, based on finite data.

Original: Generally, we expect the approximation to get better, with the increase in sample size.
Revised: Generally, we expect the approximation to get better, with the increase in sample size.

Original: Algorithms achieving this are said to be consistent.
Revised: Algorithms achieving this are said to be consistent.

Original: Formally, we introduce the following definition:
Revised: Formally, we introduce the following definition:

Original: Definition 1 (Learnability of OOD Detection) .
Revised: Definition 1 (Learnability of OOD Detection) .

Original: Given a domain space D XY and a hypothesis space
Revised: Given a domain space D XY and a hypothesis space

Original: H ⊂ { h : X → Y all } , we say OOD detection is learnable in D XY for H , if there exists an algorithm A 3 : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,
Revised: H ⊂ { h : X → Y all } , we say OOD detection is learnable in D XY for H , if there exists an algorithm A 3 : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,

Original: [Equation]
Revised: [Equation]

Original: An algorithm A for which this holds is said to be consistent with respect to D XY .
Revised: An algorithm A for which this holds is said to be consistent with respect to D XY .

Original: Definition 1 is a natural extension of agnostic PAC learnability of supervised learning [30].
Revised: Definition 1 is a natural extension of agnostic PAC learnability of supervised learning [30].

Original: If for any
Revised: If for any

Original: D XY ∈ D XY , π out = 0 , then Definition 2 is the agnostic PAC learnability of supervised learning.
Revised: D XY ∈ D XY , π out = 0 , then Definition 2 is the agnostic PAC learnability of supervised learning.

Original: Although the expression of Definition 1 is different from the normal definition of agnostic PAC learning in [21], one can easily prove that they are equivalent when ℓ is bounded, see Appendix D.3.
Revised: Although the expression of Definition 1 is different from the normal definition of agnostic PAC learning in [21], one can easily prove that they are equivalent when ℓ is bounded, see Appendix D.3.

Original: Since OOD data are unavailable, it is impossible to obtain information about the class-prior probability π out .
Revised: Since OOD data are unavailable, it is impossible to obtain information about the class-prior probability π out .

Original: Furthermore, in the real world, it is possible that π out can be any value in [0 , 1) .
Revised: Furthermore, in the real world, it is possible that π out can be any value in [0 , 1) .

Original: Therefore, the imbalance issue between ID and OOD distributions, and the priori-unknown issue ( i.e. , π out is unknown) are the core challenges.
Revised: Therefore, the imbalance issue between ID and OOD distributions, and the priori-unknown issue ( i.e. , π out is unknown) are the core challenges.

Original: To ease these challenges, researchers use AUROC, AUPR and
Revised: To ease these challenges, researchers use AUROC, AUPR and

Original: FPR95 to estimate the performance of OOD detection [18, 31, 32, 33, 34, 35].
Revised: FPR95 to estimate the performance of OOD detection [18, 31, 32, 33, 34, 35].

Original: It seems that there is a gap between Definition 1 and existing works.
Revised: It seems that there is a gap between Definition 1 and existing works.

Original: To eliminate this gap, we revise Eq.
Revised: To eliminate this gap, we revise Eq.

Original: (2) as follows:
Revised: (2) as follows:

Original: [Equation]
Revised: [Equation]

Original: If an algorithm A satisfies Eq.
Revised: If an algorithm A satisfies Eq.

Original: (3), then the imbalance issue and the prior-unknown issue disappear.
Revised: (3), then the imbalance issue and the prior-unknown issue disappear.

Original: That is, A can simultaneously classify the ID data and detect the OOD data well.
Revised: That is, A can simultaneously classify the ID data and detect the OOD data well.

Original: Based on the above discussion, we define the strong learnability of OOD detection as follows:
Revised: Based on the above discussion, we define the strong learnability of OOD detection as follows:

Original: Definition 2 (Strong Learnability of OOD Detection) .
Revised: Definition 2 (Strong Learnability of OOD Detection) .

Original: Given a domain space D XY and a hypothesis space H ⊂ { h : X → Y all } , we say OOD detection is strongly learnable in D XY for H , if there exists an algorithm A : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,
Revised: Given a domain space D XY and a hypothesis space H ⊂ { h : X → Y all } , we say OOD detection is strongly learnable in D XY for H , if there exists an algorithm A : ∪ + ∞ n =1 ( X × Y ) n → H and a monotonically decreasing sequence ϵ cons ( n ) , such that ϵ cons ( n ) → 0 , as n → + ∞ , and for any domain D XY ∈ D XY ,

Original: [Equation]
Revised: [Equation]

Original: In Theorem 1, we have shown that the strong learnability of OOD detection is equivalent to the learnability of OOD detection, if the domain space D XY is a prior-unknown space (see Definition 3).
Revised: In Theorem 1, we have shown that the strong learnability of OOD detection is equivalent to the learnability of OOD detection, if the domain space D XY is a prior-unknown space (see Definition 3).

Original: In this paper, we mainly discuss the learnability in the prior-unknown space.
Revised: In this paper, we mainly discuss the learnability in the prior-unknown space.

Original: Therefore, when we mention that OOD detection is learnable, we also mean that OOD detection is strongly learnable .
Revised: Therefore, when we mention that OOD detection is learnable, we also mean that OOD detection is strongly learnable .

Original: Goal of Theory.
Revised: Goal of Theory.

Original: Note that the agnostic PAC learnability of supervised learning is distribution-free, i.e. , the domain space D XY consists of all domains.
Revised: Note that the agnostic PAC learnability of supervised learning is distribution-free, i.e. , the domain space D XY consists of all domains.

Original: However, due to the absence of OOD data during the training process [8, 14, 24], it is obvious that the learnability of OOD detection is not distribution-free ( i.e. , Theorem 4).
Revised: However, due to the absence of OOD data during the training process [8, 14, 24], it is obvious that the learnability of OOD detection is not distribution-free ( i.e. , Theorem 4).

Original: In fact, we discover that the learnability of OOD detection is deeply correlated with the relationship between the domain space D XY and the hypothesis space H .
Revised: In fact, we discover that the learnability of OOD detection is deeply correlated with the relationship between the domain space D XY and the hypothesis space H .

Original: That is, OOD detection is learnable only when the domain space D XY and the hypothesis space H satisfy some special conditions, e.g., Condition 1 and Condition 3.
Revised: That is, OOD detection is learnable only when the domain space D XY and the hypothesis space H satisfy some special conditions, e.g., Condition 1 and Condition 3.

Original: We present our goal as follows:
Revised: We present our goal as follows:

Original: Goal:
Revised: Goal:

Original: given a hypothesis space H
Revised: given a hypothesis space H

Original: and several representative domain spaces D
Revised: and several representative domain spaces D

Original: XY
Revised: XY

Original: , what are the conditions to ensure the learnability of OOD detection?
Revised: , what are the conditions to ensure the learnability of OOD detection?

Original: Furthermore, if possible, we hope that these conditions are necessary and sufficient in some scenarios.
Revised: Furthermore, if possible, we hope that these conditions are necessary and sufficient in some scenarios.

Original: Therefore, compared to the agnostic PAC learnability of supervised learning, our theory doesn’t focus on the distribution-free case, but focuses on discovering essential conditions to guarantee the learnability of OOD detection in several representative and practical domain spaces D XY .
Revised: Therefore, compared to the agnostic PAC learnability of supervised learning, our theory doesn’t focus on the distribution-free case, but focuses on discovering essential conditions to guarantee the learnability of OOD detection in several representative and practical domain spaces D XY .

Original: By these essential conditions, we can know when OOD detection can be successful in real applications.
Revised: By these essential conditions, we can know when OOD detection can be successful in real applications.

Original:  Learning in Priori-unknown Spaces
Revised:  Learning in Priori-unknown Spaces

Original: We first investigate a special space, called prior-unknown space.
Revised: We first investigate a special space, called prior-unknown space.

Original: In such space, Definition 1 and
Revised: In such space, Definition 1 and

Original: Definition 2 are equivalent.
Revised: Definition 2 are equivalent.

Original: Furthermore, we also prove that if OOD detection is strongly learnable in a space D XY , then one can discover a larger domain space, which is prior-unknown, to ensure the learnability of OOD detection.
Revised: Furthermore, we also prove that if OOD detection is strongly learnable in a space D XY , then one can discover a larger domain space, which is prior-unknown, to ensure the learnability of OOD detection.

Original: These results imply that it is enough to consider our theory in the prior-unknown spaces.
Revised: These results imply that it is enough to consider our theory in the prior-unknown spaces.

Original: The prior-unknown space is introduced as follows:
Revised: The prior-unknown space is introduced as follows:

Original: Definition 3.
Revised: Definition 3.

Original: Given a domain space D XY , we say D XY is a priori-unknown space, if for any domain
Revised: Given a domain space D XY , we say D XY is a priori-unknown space, if for any domain

Original: D XY ∈ D XY and any α ∈ [0 , 1)
Revised: D XY ∈ D XY and any α ∈ [0 , 1)

Original: , we have D αXY :
Revised: , we have D αXY :

Original: = (1 − α )
Revised: = (1 − α )

Original: D
Revised: D

Original: X
Revised: X

Original: I Y
Revised: I Y

Original: I + αD X O Y O ∈ D XY .
Revised: I + αD X O Y O ∈ D XY .

Original: Theorem 1.
Revised: Theorem 1.

Original: Given domain spaces D XY and D ′ XY
Revised: Given domain spaces D XY and D ′ XY

Original: = { D αXY : ∀ D XY ∈ D XY , ∀ α ∈ [0 , 1) } , then
Revised: = { D αXY : ∀ D XY ∈ D XY , ∀ α ∈ [0 , 1) } , then

Original: 1) D ′ XY is a priori-unknown space and D XY ⊂ D ′ XY ; 2) if D XY is a priori-unknown space, then Definition 1 and Definition 2 are equivalent ; 3) OOD detection is strongly learnable in D XY if and only if OOD detection is learnable in D ′ XY .
Revised: 1) D ′ XY is a priori-unknown space and D XY ⊂ D ′ XY ; 2) if D XY is a priori-unknown space, then Definition 1 and Definition 2 are equivalent ; 3) OOD detection is strongly learnable in D XY if and only if OOD detection is learnable in D ′ XY .

Original: The second result of Theorem 1 bridges the learnability and strong learnability, which implies that if an algorithm A is consistent with respect to a prior-unknown space, then this algorithm A can address the imbalance issue between ID and OOD distributions, and the priori-unknown issue well.
Revised: The second result of Theorem 1 bridges the learnability and strong learnability, which implies that if an algorithm A is consistent with respect to a prior-unknown space, then this algorithm A can address the imbalance issue between ID and OOD distributions, and the priori-unknown issue well.

Original: Based on Theorem 1, we focus on our theory in the prior-unknown spaces.
Revised: Based on Theorem 1, we focus on our theory in the prior-unknown spaces.

Original: Furthermore, to demystify the learnability of OOD detection, we introduce five representative priori-unknown spaces:
Revised: Furthermore, to demystify the learnability of OOD detection, we introduce five representative priori-unknown spaces:

Original: • Single-distribution space D D XY XY .
Revised: • Single-distribution space D D XY XY .

Original: For a domain D XY , D D XY XY
Revised: For a domain D XY , D D XY XY

Original: [Equation]
Revised: [Equation]

Original: • Total space D all XY , which consists of all domains.
Revised: • Total space D all XY , which consists of all domains.

Original: • Separate space D sXY , which consists of all domains that satisfy the separate condition, that is for any D XY ∈ D s XY , supp D X O ∩ supp D
Revised: • Separate space D sXY , which consists of all domains that satisfy the separate condition, that is for any D XY ∈ D s XY , supp D X O ∩ supp D

Original: X
Revised: X

Original: I = ∅ , where supp means the support set.
Revised: I = ∅ , where supp means the support set.

Original: • Finite-ID-distribution space D FXY , which is a prior-unknown space satisfying that the number of distinct ID joint distributions D X
Revised: • Finite-ID-distribution space D FXY , which is a prior-unknown space satisfying that the number of distinct ID joint distributions D X

Original: I Y I in D FXY is finite, i.e. , |{ D X
Revised: I Y I in D FXY is finite, i.e. , |{ D X

Original: I Y
Revised: I Y

Original: I : ∀ D XY ∈ D FXY }| < + ∞ .
Revised: I : ∀ D XY ∈ D FXY }| < + ∞ .

Original: • Density-based space D µ,bXY , which is a prior-unknown space consisting of some domains satisfying that: for any D XY , there exists a density function f with 1 /b ≤ f ≤ b in supp µ and 0 .
Revised: • Density-based space D µ,bXY , which is a prior-unknown space consisting of some domains satisfying that: for any D XY , there exists a density function f with 1 /b ≤ f ≤ b in supp µ and 0 .

Original: 5 ∗ D X
Revised: 5 ∗ D X

Original: I +
Revised: I +

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: f d µ , where µ is a measure defined over X .
Revised: f d µ , where µ is a measure defined over X .

Original: Note that if µ is discrete, then D X is a discrete distribution; and if µ is the Lebesgue measure, then D X is a continuous distribution.
Revised: Note that if µ is discrete, then D X is a discrete distribution; and if µ is the Lebesgue measure, then D X is a continuous distribution.

Original: The above representative spaces widely exist in real applications.
Revised: The above representative spaces widely exist in real applications.

Original: For example, 1) if the images from different semantic labels with different styles are clearly different, then those images can form a distribution belonging to a separate space D sXY ; and 2) when designing an algorithm, we only have finite ID datasets, e.g. , CIFAR-10, MNIST, SVHN, and ImageNet, to build a model.
Revised: For example, 1) if the images from different semantic labels with different styles are clearly different, then those images can form a distribution belonging to a separate space D sXY ; and 2) when designing an algorithm, we only have finite ID datasets, e.g. , CIFAR-10, MNIST, SVHN, and ImageNet, to build a model.

Original: Then, finite-ID-distribution space D FXY can handle this real scenario.
Revised: Then, finite-ID-distribution space D FXY can handle this real scenario.

Original: Note that the single-distribution space is a special case of the finite-ID-distribution space.
Revised: Note that the single-distribution space is a special case of the finite-ID-distribution space.

Original: In this paper, we mainly discuss these five spaces.
Revised: In this paper, we mainly discuss these five spaces.

Original:  Impossibility Theorems for OOD Detection
Revised:  Impossibility Theorems for OOD Detection

Original: In this section, we first give a necessary condition for the learnability of OOD detection.
Revised: In this section, we first give a necessary condition for the learnability of OOD detection.

Original: Then, we show this necessary condition does not hold in the total space D all XY and the separate space D sXY .
Revised: Then, we show this necessary condition does not hold in the total space D all XY and the separate space D sXY .

Original: Necessary Condition.
Revised: Necessary Condition.

Original: We find a necessary condition for the learnability of OOD detection, i.e. , Condition 1, motivated by the experiments in Figure 1.
Revised: We find a necessary condition for the learnability of OOD detection, i.e. , Condition 1, motivated by the experiments in Figure 1.

Original: Details of Figure 1 can be found in Appendix C.2.
Revised: Details of Figure 1 can be found in Appendix C.2.

Original: Condition 1 (Linear Condition) .
Revised: Condition 1 (Linear Condition) .

Original: For any D XY ∈ D XY and any α ∈ [0 , 1) ,
Revised: For any D XY ∈ D XY and any α ∈ [0 , 1) ,

Original: [Equation]
Revised: [Equation]

Original: To reveal the importance of Condition 1, Theorem 2 shows that Condition 1 is a necessary and sufficient condition for the learnability of OOD detection if the D XY is the single-distribution space.
Revised: To reveal the importance of Condition 1, Theorem 2 shows that Condition 1 is a necessary and sufficient condition for the learnability of OOD detection if the D XY is the single-distribution space.

Original: Theorem 2.
Revised: Theorem 2.

Original: Given a hypothesis space H and a domain D XY , OOD detection is learnable in the single-distribution space D D XY XY for H if and only if linear condition (i.e., Condition 1) holds.
Revised: Given a hypothesis space H and a domain D XY , OOD detection is learnable in the single-distribution space D D XY XY for H if and only if linear condition (i.e., Condition 1) holds.

Original: Theorem 2 implies that Condition 1 is important for the learnability of OOD detection.
Revised: Theorem 2 implies that Condition 1 is important for the learnability of OOD detection.

Original: Due to the simplicity of single-distribution space, Theorem 2 implies that Condition 1 is the necessary condition for the learnability of OOD detection in the prior-unknown space, see Lemma 1 in Appendix F.
Revised: Due to the simplicity of single-distribution space, Theorem 2 implies that Condition 1 is the necessary condition for the learnability of OOD detection in the prior-unknown space, see Lemma 1 in Appendix F.

Original: Impossibility Theorems.
Revised: Impossibility Theorems.

Original: Here, we first study whether Condition 1 holds in the total space D all XY .
Revised: Here, we first study whether Condition 1 holds in the total space D all XY .

Original: If
Revised: If

Original: Condition 1 does not hold, then OOD detection is not learnable.
Revised: Condition 1 does not hold, then OOD detection is not learnable.

Original: Theorem 3 shows that Condition 1 is not always satisfied, especially, when there is an overlap between the ID and OOD distributions:
Revised: Theorem 3 shows that Condition 1 is not always satisfied, especially, when there is an overlap between the ID and OOD distributions:

Original: Definition 4 (Overlap Between ID and OOD) .
Revised: Definition 4 (Overlap Between ID and OOD) .

Original: We say a domain D XY has overlap between ID and
Revised: We say a domain D XY has overlap between ID and

Original: OOD distributions, if there is a σ -finite measure ˜ µ such that D X is absolutely continuous with respect to ˜ µ , and ˜ µ ( A overlap ) > 0 , where A overlap = { x ∈ X : f I ( x ) > 0 and f O ( x ) > 0 } .
Revised: OOD distributions, if there is a σ -finite measure ˜ µ such that D X is absolutely continuous with respect to ˜ µ , and ˜ µ ( A overlap ) > 0 , where A overlap = { x ∈ X : f I ( x ) > 0 and f O ( x ) > 0 } .

Original: Here f I and f O are the representers of D X I and D X O in Radon–Nikodym Theorem [36],
Revised: Here f I and f O are the representers of D X I and D X O in Radon–Nikodym Theorem [36],

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: Given a hypothesis space H and a prior-unknown space D XY , if there is D XY ∈ D XY , which has overlap between ID and OOD, and inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 , then
Revised: Given a hypothesis space H and a prior-unknown space D XY , if there is D XY ∈ D XY , which has overlap between ID and OOD, and inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 , then

Original: Condition 1 does not hold.
Revised: Condition 1 does not hold.

Original: Therefore, OOD detection is not learnable in D XY for H .
Revised: Therefore, OOD detection is not learnable in D XY for H .

Original: Theorem 3 clearly shows that under proper conditions, Condition 1 does not hold, if there exists a domain whose ID and OOD distributions have overlap.
Revised: Theorem 3 clearly shows that under proper conditions, Condition 1 does not hold, if there exists a domain whose ID and OOD distributions have overlap.

Original: By Theorem 3, we can obtain that the OOD detection is not learnable in the total space D all XY for any non-trivial hypothesis space H .
Revised: By Theorem 3, we can obtain that the OOD detection is not learnable in the total space D all XY for any non-trivial hypothesis space H .

Original: Theorem 4 (Impossibility Theorem for Total Space) .
Revised: Theorem 4 (Impossibility Theorem for Total Space) .

Original: OOD detection is not learnable in the total space D all XY for H , if | ϕ ◦ H| > 1 , where ϕ maps ID labels to 1 and maps OOD labels to 2 .
Revised: OOD detection is not learnable in the total space D all XY for H , if | ϕ ◦ H| > 1 , where ϕ maps ID labels to 1 and maps OOD labels to 2 .

Original: Since the overlaps between ID and OOD distributions may cause that Condition 1 does not hold, we then consider studying the learnability of OOD detection in the separate space D sXY , where there are no overlaps between the ID and OOD distributions.
Revised: Since the overlaps between ID and OOD distributions may cause that Condition 1 does not hold, we then consider studying the learnability of OOD detection in the separate space D sXY , where there are no overlaps between the ID and OOD distributions.

Original: However, Theorem 5 shows that even if we consider the separate space, the OOD detection is still not learnable in some scenarios.
Revised: However, Theorem 5 shows that even if we consider the separate space, the OOD detection is still not learnable in some scenarios.

Original: Before introducing the impossibility theorem for separate space, i.e. , Theorem 5, we need a mild assumption:
Revised: Before introducing the impossibility theorem for separate space, i.e. , Theorem 5, we need a mild assumption:

Original: Assumption 1 (Separate Space for OOD) .
Revised: Assumption 1 (Separate Space for OOD) .

Original: A hypothesis space H is separate for OOD data, if for each data point x ∈ X , there exists at least one hypothesis function h x ∈ H such that h x ( x ) =
Revised: A hypothesis space H is separate for OOD data, if for each data point x ∈ X , there exists at least one hypothesis function h x ∈ H such that h x ( x ) =

Original: K + 1 .
Revised: K + 1 .

Original: Assumption 1 means that every data point x has the possibility to be detected as OOD data.
Revised: Assumption 1 means that every data point x has the possibility to be detected as OOD data.

Original: Assumption 1 is mild and can be satisfied by many hypothesis spaces, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel space.
Revised: Assumption 1 is mild and can be satisfied by many hypothesis spaces, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel space.

Original: Next, we use Vapnik–Chervonenkis (VC) dimension [22] to measure the size of hypothesis space, and study the learnability of OOD detection in D sXY based on the VC dimension.
Revised: Next, we use Vapnik–Chervonenkis (VC) dimension [22] to measure the size of hypothesis space, and study the learnability of OOD detection in D sXY based on the VC dimension.

Original: Theorem 5 (Impossibility Theorem for Separate Space) .
Revised: Theorem 5 (Impossibility Theorem for Separate Space) .

Original: If Assumption 1 holds, VCdim( ϕ ◦ H ) <
Revised: If Assumption 1 holds, VCdim( ϕ ◦ H ) <

Original: + ∞ and sup h ∈H |{ x ∈ X : h ( x )
Revised: + ∞ and sup h ∈H |{ x ∈ X : h ( x )

Original: ∈ Y}| =
Revised: ∈ Y}| =

Original: + ∞ , then OOD detection is not learnable in separate space D sXY for H , where ϕ maps ID labels to 1 and maps OOD labels to 2 .
Revised: + ∞ , then OOD detection is not learnable in separate space D sXY for H , where ϕ maps ID labels to 1 and maps OOD labels to 2 .

Original: The finite VC dimension normally implies the learnability of supervised learning.
Revised: The finite VC dimension normally implies the learnability of supervised learning.

Original: However, in our results, the finite VC dimension cannot guarantee the learnability of OOD detection in the separate space, which reveals the difficulty of the OOD detection.
Revised: However, in our results, the finite VC dimension cannot guarantee the learnability of OOD detection in the separate space, which reveals the difficulty of the OOD detection.

Original: Although the above impossibility theorems are frustrating, there is still room to discuss the conditions in Theorem 5, and to find out the proper conditions for ensuring the learnability of OOD detection in the separate space (see Sections 5 and 6).
Revised: Although the above impossibility theorems are frustrating, there is still room to discuss the conditions in Theorem 5, and to find out the proper conditions for ensuring the learnability of OOD detection in the separate space (see Sections 5 and 6).

Original:  When OOD Detection Can Be Successful
Revised:  When OOD Detection Can Be Successful

Original: Here, we discuss when the OOD detection can be learnable in the separate space D sXY , finite-IDdistribution space D FXY and density-based space D µ,bXY .
Revised: Here, we discuss when the OOD detection can be learnable in the separate space D sXY , finite-IDdistribution space D FXY and density-based space D µ,bXY .

Original: We first study the separate space D sXY .
Revised: We first study the separate space D sXY .

Original: OOD Detection in the Separate Space.
Revised: OOD Detection in the Separate Space.

Original: Theorem 5 has indicated that VCdim( ϕ ◦ H ) =
Revised: Theorem 5 has indicated that VCdim( ϕ ◦ H ) =

Original: + ∞ or sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ is necessary to ensure the learnability of OOD detection in D s
Revised: + ∞ or sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ is necessary to ensure the learnability of OOD detection in D s

Original: XY if Assumption 1 holds.
Revised: XY if Assumption 1 holds.

Original: However, generally, hypothesis spaces generated by feed-forward neural networks with proper activation functions have finite VC dimension [37, 38].
Revised: However, generally, hypothesis spaces generated by feed-forward neural networks with proper activation functions have finite VC dimension [37, 38].

Original: Therefore, we study the learnability of OOD detection in the case that |X| < + ∞ , which implies that sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ .
Revised: Therefore, we study the learnability of OOD detection in the case that |X| < + ∞ , which implies that sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| < + ∞ .

Original: Additionally, Theorem 10 also implies that |X| <
Revised: Additionally, Theorem 10 also implies that |X| <

Original: + ∞ is the necessary and sufficient condition for the learnability of OOD detection in separate space, when the hypothesis space is generated by FCNN.
Revised: + ∞ is the necessary and sufficient condition for the learnability of OOD detection in separate space, when the hypothesis space is generated by FCNN.

Original: Hence, |X| < + ∞ may be necessary in the space D sXY .
Revised: Hence, |X| < + ∞ may be necessary in the space D sXY .

Original: For simplicity, we first discuss the case that K
Revised: For simplicity, we first discuss the case that K

Original: = 1 , i.e. , the one-class novelty detection.
Revised: = 1 , i.e. , the one-class novelty detection.

Original: We show the necessary and sufficient condition for the learnability of OOD detection in D sXY , when |X| < + ∞ .
Revised: We show the necessary and sufficient condition for the learnability of OOD detection in D sXY , when |X| < + ∞ .

Original: Theorem 6.
Revised: Theorem 6.

Original: Let K = 1 and |X| < + ∞ .
Revised: Let K = 1 and |X| < + ∞ .

Original: Suppose that Assumption 1 holds and the constant function h in := 1 ∈ H .
Revised: Suppose that Assumption 1 holds and the constant function h in := 1 ∈ H .

Original: Then OOD detection is learnable in D sXY for H if and only if H all − { h out } ⊂ H , where H all is the hypothesis space consisting of all hypothesis functions, and h out is a constant function that h out := 2 , here 1 represents ID data and 2 represents OOD data.
Revised: Then OOD detection is learnable in D sXY for H if and only if H all − { h out } ⊂ H , where H all is the hypothesis space consisting of all hypothesis functions, and h out is a constant function that h out := 2 , here 1 represents ID data and 2 represents OOD data.

Original: The condition h in ∈ H presented in Theorem 6 is mild.
Revised: The condition h in ∈ H presented in Theorem 6 is mild.

Original: Many practical hypothesis spaces satisfy this condition, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel-based hypothesis space.
Revised: Many practical hypothesis spaces satisfy this condition, e.g. , the FCNN-based hypothesis space (Proposition 1 in Appendix K), score-based hypothesis space (Proposition 2 in Appendix K) and universal kernel-based hypothesis space.

Original: Theorem 6 implies that if K = 1 and OOD detection is learnable in D sXY for H , then the hypothesis space H should contain almost all hypothesis functions, implying that if the OOD detection can be learnable in the distribution-agnostic case, then a large-capacity model is necessary.
Revised: Theorem 6 implies that if K = 1 and OOD detection is learnable in D sXY for H , then the hypothesis space H should contain almost all hypothesis functions, implying that if the OOD detection can be learnable in the distribution-agnostic case, then a large-capacity model is necessary.

Original: Next, we extend Theorem 6 to a general case, i.e. , K > 1 .
Revised: Next, we extend Theorem 6 to a general case, i.e. , K > 1 .

Original: When K > 1 , we will first use a binary classifier h b to classify the ID and OOD data.
Revised: When K > 1 , we will first use a binary classifier h b to classify the ID and OOD data.

Original: Then, for the ID data identified by h b , an ID hypothesis function h in will be used to classify them into corresponding ID classes.
Revised: Then, for the ID data identified by h b , an ID hypothesis function h in will be used to classify them into corresponding ID classes.

Original: We state this strategy as follows: given a hypothesis space H in for ID distribution and a binary classification hypothesis space
Revised: We state this strategy as follows: given a hypothesis space H in for ID distribution and a binary classification hypothesis space

Original: H b introduced in Section 2, we use H in and H b to construct an OOD detection’s hypothesis space H , which consists of all hypothesis functions h satisfying the following condition: there exist h in ∈ H in and h b ∈ H b such that for any x ∈ X ,
Revised: H b introduced in Section 2, we use H in and H b to construct an OOD detection’s hypothesis space H , which consists of all hypothesis functions h satisfying the following condition: there exist h in ∈ H in and h b ∈ H b such that for any x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: We use H in • H b to represent a hypothesis space consisting of all h defined in Eq.
Revised: We use H in • H b to represent a hypothesis space consisting of all h defined in Eq.

Original: In addition, we also need an additional condition for the loss function ℓ .
Revised: In addition, we also need an additional condition for the loss function ℓ .

Original: This condition is shown as follows:
Revised: This condition is shown as follows:

Original: Condition 2.
Revised: Condition 2.

Original: ℓ ( y 2 , y 1 ) ≤ ℓ ( K + 1 , y 1 ) , for any in-distribution labels y 1 and y 2 ∈ Y .
Revised: ℓ ( y 2 , y 1 ) ≤ ℓ ( K + 1 , y 1 ) , for any in-distribution labels y 1 and y 2 ∈ Y .

Original: Theorem 7.
Revised: Theorem 7.

Original: Let |X| < + ∞ and H
Revised: Let |X| < + ∞ and H

Original: = H in • H b .
Revised: = H in • H b .

Original: If H all − { h out } ⊂ H b and Condition 2 holds, then OOD detection is learnable in D sXY for H , where H all and h out are defined in Theorem 6.
Revised: If H all − { h out } ⊂ H b and Condition 2 holds, then OOD detection is learnable in D sXY for H , where H all and h out are defined in Theorem 6.

Original: OOD Detection in the Finite-ID-Distribution Space.
Revised: OOD Detection in the Finite-ID-Distribution Space.

Original: Since researchers can only collect finite ID datasets as the training data in the process of algorithm design, it is worthy to study the learnability of
Revised: Since researchers can only collect finite ID datasets as the training data in the process of algorithm design, it is worthy to study the learnability of

Original: OOD detection in the finite-ID-distribution space D FXY .
Revised: OOD detection in the finite-ID-distribution space D FXY .

Original: We first show two necessary concepts below.
Revised: We first show two necessary concepts below.

Original: Definition 5 (ID Consistency) .
Revised: Definition 5 (ID Consistency) .

Original: Given a domain space D XY , we say any two domains D XY ∈ D XY and D ′ XY ∈ D XY are ID consistency, if D X
Revised: Given a domain space D XY , we say any two domains D XY ∈ D XY and D ′ XY ∈ D XY are ID consistency, if D X

Original: I Y I
Revised: I Y I

Original: = D ′
Revised: = D ′

Original: X
Revised: X

Original: I Y
Revised: I Y

Original: I .
Revised: I .

Original: We use the notation ∼ to represent the ID consistency, i.e., D XY ∼ D ′ XY if and only if D XY and D ′ XY are ID consistency.
Revised: We use the notation ∼ to represent the ID consistency, i.e., D XY ∼ D ′ XY if and only if D XY and D ′ XY are ID consistency.

Original: It is easy to check that the ID consistency ∼ is an equivalence relation.
Revised: It is easy to check that the ID consistency ∼ is an equivalence relation.

Original: Therefore, we define the set [ D XY ] := { D ′ XY ∈ D XY : D XY ∼ D ′ XY } as the equivalence class with respect to space D XY .
Revised: Therefore, we define the set [ D XY ] := { D ′ XY ∈ D XY : D XY ∼ D ′ XY } as the equivalence class with respect to space D XY .

Original: Condition 3 (Compatibility) .
Revised: Condition 3 (Compatibility) .

Original: For any equivalence class [ D ′ XY ] with respect to D XY and any ϵ > 0 , there exists a hypothesis function h ϵ ∈ H such that for any domain D XY ∈
Revised: For any equivalence class [ D ′ XY ] with respect to D XY and any ϵ > 0 , there exists a hypothesis function h ϵ ∈ H such that for any domain D XY ∈

Original: [ D ′ XY ] ,
Revised: [ D ′ XY ] ,

Original: [Equation]
Revised: [Equation]

Original: In Appendix F, Lemma 2 has implied that Condition 3 is a general version of Condition 1.
Revised: In Appendix F, Lemma 2 has implied that Condition 3 is a general version of Condition 1.

Original: Next,
Revised: Next,

Original: Theorem 8 indicates that Condition 3 is the necessary and sufficient condition in the space D FXY .
Revised: Theorem 8 indicates that Condition 3 is the necessary and sufficient condition in the space D FXY .

Original: Theorem 8.
Revised: Theorem 8.

Original: Suppose that X is a bounded set.
Revised: Suppose that X is a bounded set.

Original: OOD detection is learnable in the finite-IDdistribution space D FXY for H if and only if the compatibility condition (i.e., Condition 3) holds.
Revised: OOD detection is learnable in the finite-IDdistribution space D FXY for H if and only if the compatibility condition (i.e., Condition 3) holds.

Original: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .
Revised: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .

Original: Theorem 8 shows that, in the process of algorithm design, OOD detection cannot be successful without the compatibility condition.
Revised: Theorem 8 shows that, in the process of algorithm design, OOD detection cannot be successful without the compatibility condition.

Original: Theorem 8 also implies that Condition 3 is essential for the learnability of OOD detection.
Revised: Theorem 8 also implies that Condition 3 is essential for the learnability of OOD detection.

Original: This motivates us to study whether OOD detection can be successful in more general spaces ( e.g. , the density-based space), when the compatibility condition holds.
Revised: This motivates us to study whether OOD detection can be successful in more general spaces ( e.g. , the density-based space), when the compatibility condition holds.

Original: OOD Detection in the Density-based Space.
Revised: OOD Detection in the Density-based Space.

Original: To ensure that Condition 3 holds, we consider a basic assumption in learning theory— Realizability Assumption (see Appendix D.2), i.e. , for any
Revised: To ensure that Condition 3 holds, we consider a basic assumption in learning theory— Realizability Assumption (see Appendix D.2), i.e. , for any

Original: D XY ∈ D XY , there exists h ∗ ∈ H such that R D ( h ∗ )
Revised: D XY ∈ D XY , there exists h ∗ ∈ H such that R D ( h ∗ )

Original: = 0 .
Revised: = 0 .

Original: We discover that in the density-based space D µ,bXY , Realizability Assumption can conclude the compatibility condition ( i.e. , Condition 3).
Revised: We discover that in the density-based space D µ,bXY , Realizability Assumption can conclude the compatibility condition ( i.e. , Condition 3).

Original: Based on this observation, we can prove the following theorem:
Revised: Based on this observation, we can prove the following theorem:

Original: Theorem 9.
Revised: Theorem 9.

Original: Given a density-based space D µ,bXY , if µ ( X ) < + ∞ , the Realizability Assumption holds, then when H has finite Natarajan dimension [21], OOD detection is learnable in D µ,bXY for
Revised: Given a density-based space D µ,bXY , if µ ( X ) < + ∞ , the Realizability Assumption holds, then when H has finite Natarajan dimension [21], OOD detection is learnable in D µ,bXY for

Original: H .
Revised: H .

Original: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .
Revised: Furthermore, the learning rate ϵ cons ( n ) can attain O (1 / √ n 1 − θ ) , for any θ ∈ (0 , 1) .

Original: To further investigate the importance and necessary of Realizability Assumption, Theorem 11 has indicated that in some practical scenarios, Realizability Assumption is the necessary and sufficient condition for the learnability of OOD detection in the density-based space.
Revised: To further investigate the importance and necessary of Realizability Assumption, Theorem 11 has indicated that in some practical scenarios, Realizability Assumption is the necessary and sufficient condition for the learnability of OOD detection in the density-based space.

Original: Therefore, Realizability
Revised: Therefore, Realizability

Original: Assumption may be indispensable for the learnability of OOD detection in some practical scenarios.
Revised: Assumption may be indispensable for the learnability of OOD detection in some practical scenarios.

Original:  Connecting Theory to Practice
Revised:  Connecting Theory to Practice

Original: In Section 5, we have shown the successful scenarios where OOD detection problem can be addressed in theory.
Revised: In Section 5, we have shown the successful scenarios where OOD detection problem can be addressed in theory.

Original: In this section, we will discuss how the proposed theory is applied to two representative hypothesis spaces—neural-network-based hypothesis spaces and score-based hypothesis spaces.
Revised: In this section, we will discuss how the proposed theory is applied to two representative hypothesis spaces—neural-network-based hypothesis spaces and score-based hypothesis spaces.

Original: Fully-connected Neural Networks.
Revised: Fully-connected Neural Networks.

Original: Given a sequence q
Revised: Given a sequence q

Original: =
Revised: =

Original: ( l 1 , l 2 , ..., l g ) , where l i and g are positive integers and g > 2 , we use g to represent the depth of neural network and use l i to represent the width of the i -th layer.
Revised: ( l 1 , l 2 , ..., l g ) , where l i and g are positive integers and g > 2 , we use g to represent the depth of neural network and use l i to represent the width of the i -th layer.

Original: After the activation function σ is selected 4 , we can obtain the architecture of FCNN according to the sequence q .
Revised: After the activation function σ is selected 4 , we can obtain the architecture of FCNN according to the sequence q .

Original: Let f w , b be the function generated by FCNN with weights w and bias b .
Revised: Let f w , b be the function generated by FCNN with weights w and bias b .

Original: An FCNN-based scoring function space is defined as: F σ q := { f w , b : ∀ weights w , ∀ bias b } .
Revised: An FCNN-based scoring function space is defined as: F σ q := { f w , b : ∀ weights w , ∀ bias b } .

Original: In addition, for simplicity, given any two sequences q = ( l 1 , ..., l g ) and q ′ =
Revised: In addition, for simplicity, given any two sequences q = ( l 1 , ..., l g ) and q ′ =

Original: ( l ′ 1 , ..., l ′ g ′ ) , we use the notation q ≲ q ′ to represent the following equations and inequalities:
Revised: ( l ′ 1 , ..., l ′ g ′ ) , we use the notation q ≲ q ′ to represent the following equations and inequalities:

Original: 1) g ≤ g ′ , l 1 = l ′ 1 , l g = l ′ g ′ ; 2) l i ≤ l ′
Revised: 1) g ≤ g ′ , l 1 = l ′ 1 , l g = l ′ g ′ ; 2) l i ≤ l ′

Original: i , ∀ i = 1 , ..., g − 1; and 3) l g − 1 ≤ l ′
Revised: i , ∀ i = 1 , ..., g − 1; and 3) l g − 1 ≤ l ′

Original: i , ∀
Revised: i , ∀

Original: i = g, ..., g ′ − 1 .
Revised: i = g, ..., g ′ − 1 .

Original: In Appendix L, Lemma 10 shows q ≲ q ′ ⇒ F σ q ⊂ F σ q ′ .
Revised: In Appendix L, Lemma 10 shows q ≲ q ′ ⇒ F σ q ⊂ F σ q ′ .

Original: We use ≲ to compare the sizes of FCNNs.
Revised: We use ≲ to compare the sizes of FCNNs.

Original: FCNN-based Hypothesis Space.
Revised: FCNN-based Hypothesis Space.

Original: Let l g
Revised: Let l g

Original: =
Revised: =

Original: K
Revised: K

Original: + 1 .
Revised: + 1 .

Original: The FCNN-based scoring function space F σ q can induce an FCNN-based hypothesis space.
Revised: The FCNN-based scoring function space F σ q can induce an FCNN-based hypothesis space.

Original: For any f w , b ∈ F σ q , the induced hypothesis function is:
Revised: For any f w , b ∈ F σ q , the induced hypothesis function is:

Original: [Equation]
Revised: [Equation]

Original: k w , b , where f k w , b is the k -th coordinate of f w , b .
Revised: k w , b , where f k w , b is the k -th coordinate of f w , b .

Original: Then, the FCNN-based hypothesis space is defined as H σ q := { h w , b : ∀ weights w , ∀ bias b } .
Revised: Then, the FCNN-based hypothesis space is defined as H σ q := { h w , b : ∀ weights w , ∀ bias b } .

Original: Score-based Hypothesis Space.
Revised: Score-based Hypothesis Space.

Original: Many OOD detection algorithms detect OOD data by using a score-based strategy.
Revised: Many OOD detection algorithms detect OOD data by using a score-based strategy.

Original: That is, given a threshold λ , a scoring function space F l ⊂ { f : X → R l } and a scoring function E : F l → R , then x is regarded as ID data if and only if E ( f ( x )) ≥ λ .
Revised: That is, given a threshold λ , a scoring function space F l ⊂ { f : X → R l } and a scoring function E : F l → R , then x is regarded as ID data if and only if E ( f ( x )) ≥ λ .

Original: We introduce several representative scoring functions E as follows: for any f = [ f 1 , ..., f l ] ⊤ ∈ F l ,
Revised: We introduce several representative scoring functions E as follows: for any f = [ f 1 , ..., f l ] ⊤ ∈ F l ,

Original: • softmax-based function [7] and temperature-scaled function [
Revised: • softmax-based function [7] and temperature-scaled function [

Original: : λ ∈ (
Revised: : λ ∈ (

Original: 1 l and T
Revised: 1 l and T

Original: >
Revised: >

Original: [Equation]
Revised: [Equation]

Original: • energy-based function [23]: λ ∈ (0 , + ∞ ) and T > 0 ,
Revised: • energy-based function [23]: λ ∈ (0 , + ∞ ) and T > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Using E , λ and f ∈ F σ q , we have a classifier:
Revised: Using E , λ and f ∈ F σ q , we have a classifier:

Original: h λ f ,E ( x ) = 1 , if E ( f ( x )) ≥ λ ; otherwise, h λ f ,E ( x ) = 2 , where 1 represents the ID data and 2 represents the OOD data.
Revised: h λ f ,E ( x ) = 1 , if E ( f ( x )) ≥ λ ; otherwise, h λ f ,E ( x ) = 2 , where 1 represents the ID data and 2 represents the OOD data.

Original: Hence, a binary classification hypothesis space H b , which consists of all h λ f ,E , is generated.
Revised: Hence, a binary classification hypothesis space H b , which consists of all h λ f ,E , is generated.

Original: We define H σ,λ q ,E :=
Revised: We define H σ,λ q ,E :=

Original: { h λ f ,E : ∀ f ∈ F σ q } .
Revised: { h λ f ,E : ∀ f ∈ F σ q } .

Original: Learnability of OOD Detection in Different Hypothesis Spaces.
Revised: Learnability of OOD Detection in Different Hypothesis Spaces.

Original: Next, we present applications of our theory regarding the above two practical and important hypothesis spaces H σ q and H σ,λ q ,E .
Revised: Next, we present applications of our theory regarding the above two practical and important hypothesis spaces H σ q and H σ,λ q ,E .

Original: Theorem 10.
Revised: Theorem 10.

Original: Suppose that Condition 2 holds and the hypothesis space H is FCNN-based or scorebased
Revised: Suppose that Condition 2 holds and the hypothesis space H is FCNN-based or scorebased

Original: , i.e., H = H σ q or H =
Revised: , i.e., H = H σ q or H =

Original: H in • H b , where H in is an ID hypothesis space, H b
Revised: H in • H b , where H in is an ID hypothesis space, H b

Original: = H σ,λ q ,E and
Revised: = H σ,λ q ,E and

Original: H = H in • H b is introduced below Eq.
Revised: H = H in • H b is introduced below Eq.

Original: (4) , here E is introduced in Eqs.
Revised: (4) , here E is introduced in Eqs.

Original: (5) or (6) .
Revised: (5) or (6) .

Original: Then
Revised: Then

Original: There is a sequence q =
Revised: There is a sequence q =

Original: ( l 1 , ..., l g )
Revised: ( l 1 , ..., l g )

Original: such that OOD detection is learnable in the separate space D sXY for H if and only if |X| < + ∞ .
Revised: such that OOD detection is learnable in the separate space D sXY for H if and only if |X| < + ∞ .

Original: Furthermore, if |X| < + ∞ , then there exists a sequence q =
Revised: Furthermore, if |X| < + ∞ , then there exists a sequence q =

Original: ( l 1 , ..., l g )
Revised: ( l 1 , ..., l g )

Original: such that for any sequence q ′ satisfying that q ≲ q ′ , OOD detection is learnable in D sXY for H .
Revised: such that for any sequence q ′ satisfying that q ≲ q ′ , OOD detection is learnable in D sXY for H .

Original: Theorem 10 states that 1) when the hypothesis space is FCNN-based or score-based, the finite feature space is the necessary and sufficient condition for the learnability of OOD detection in the separate space; and 2) a larger architecture of FCNN has a greater probability to achieve the learnability of
Revised: Theorem 10 states that 1) when the hypothesis space is FCNN-based or score-based, the finite feature space is the necessary and sufficient condition for the learnability of OOD detection in the separate space; and 2) a larger architecture of FCNN has a greater probability to achieve the learnability of

Original: OOD detection in the separate space.
Revised: OOD detection in the separate space.

Original: Note that when we select Eqs.
Revised: Note that when we select Eqs.

Original: (5) or (6) as the scoring function
Revised: (5) or (6) as the scoring function

Original: E , Theorem 10 also shows that the selected scoring functions E can guarantee the learnability of
Revised: E , Theorem 10 also shows that the selected scoring functions E can guarantee the learnability of

Original: OOD detection, which is a theoretical support for the representative works [8, 23, 7].
Revised: OOD detection, which is a theoretical support for the representative works [8, 23, 7].

Original: Furthermore,
Revised: Furthermore,

Original: Theorem 11 also offers theoretical supports for these works in the density-based space, when K = 1 .
Revised: Theorem 11 also offers theoretical supports for these works in the density-based space, when K = 1 .

Original: Theorem 11.
Revised: Theorem 11.

Original: Suppose that each domain D XY in D µ,bXY is attainable, i.e., arg min
Revised: Suppose that each domain D XY in D µ,bXY is attainable, i.e., arg min

Original: h ∈H R D ( h ) ̸
Revised: h ∈H R D ( h ) ̸

Original: = ∅ (the finite discrete domains satisfy this).
Revised: = ∅ (the finite discrete domains satisfy this).

Original: Let K = 1 and the hypothesis space H be score-based (
Revised: Let K = 1 and the hypothesis space H be score-based (

Original: H = H σ,λ q ,E , where E is in Eqs.
Revised: H = H σ,λ q ,E , where E is in Eqs.

Original: (5) or (6) ) or FCNN-based ( H = H σ q ) .
Revised: (5) or (6) ) or FCNN-based ( H = H σ q ) .

Original: If µ ( X ) < + ∞ , then the following four conditions are equivalent :
Revised: If µ ( X ) < + ∞ , then the following four conditions are equivalent :

Original: [Equation]
Revised: [Equation]

Original: Theorem 11 still holds if the function space F σ q is generated by Convolutional Neural Network.
Revised: Theorem 11 still holds if the function space F σ q is generated by Convolutional Neural Network.

Original: Overlap and Benefits of Multi-class Case.
Revised: Overlap and Benefits of Multi-class Case.

Original: We investigate when the hypothesis space is FCNN-based or score-based, what will happen if there exists an overlap between the ID and OOD distributions?
Revised: We investigate when the hypothesis space is FCNN-based or score-based, what will happen if there exists an overlap between the ID and OOD distributions?

Original: Theorem 12.
Revised: Theorem 12.

Original: Let K = 1 and the hypothesis space H be score-based ( H = H σ,λ q ,E , where E is in
Revised: Let K = 1 and the hypothesis space H be score-based ( H = H σ,λ q ,E , where E is in

Original: Eqs.
Revised: Eqs.

Original: (5) or (6) ) or FCNN-based ( H = H σ q ) .
Revised: (5) or (6) ) or FCNN-based ( H = H σ q ) .

Original: Given a prior-unknown space D XY , if there exists a domain D XY ∈ D XY , which has an overlap between ID and OOD distributions (see Definition 4), then OOD detection is not learnable in the domain space D XY for H .
Revised: Given a prior-unknown space D XY , if there exists a domain D XY ∈ D XY , which has an overlap between ID and OOD distributions (see Definition 4), then OOD detection is not learnable in the domain space D XY for H .

Original: When K = 1 and the hypothesis space is FCNN-based or score-based, Theorem 12 shows that overlap between ID and OOD distributions is the sufficient condition for the unlearnability of OOD detection.
Revised: When K = 1 and the hypothesis space is FCNN-based or score-based, Theorem 12 shows that overlap between ID and OOD distributions is the sufficient condition for the unlearnability of OOD detection.

Original: Theorem 12 takes roots in the conditions inf h ∈H R in D ( h )
Revised: Theorem 12 takes roots in the conditions inf h ∈H R in D ( h )

Original: = 

Revised: =

Original: and inf h ∈H R out D ( h ) = 0 .
Revised: and inf h ∈H R out D ( h ) = 0 .

Original: However, when K > 1 , we can ensure inf h ∈H R in D ( h ) > 

Revised: However, when K > 1 , we can ensure inf h ∈H R in D ( h ) >

Original: if ID distribution D X
Revised: if ID distribution D X

Original: I Y
Revised: I Y

Original: I has overlap between ID classes.
Revised: I has overlap between ID classes.

Original: By this observation, we conjecture that when K > 1 , OOD detection is learnable in some special cases where overlap exists, even if the hypothesis space is FCNN-based or score-based.
Revised: By this observation, we conjecture that when K > 1 , OOD detection is learnable in some special cases where overlap exists, even if the hypothesis space is FCNN-based or score-based.

Original:  Discussion
Revised:  Discussion

Original: Understanding Far-OOD Detection.
Revised: Understanding Far-OOD Detection.

Original: Many existing works [7, 39] study the far-OOD detection issue.
Revised: Many existing works [7, 39] study the far-OOD detection issue.

Original: Existing benchmarks include 1) MNIST [40] as ID dataset, and Texture [41], CIFAR- 10 [42] or Place 365 [43] as OOD datasets; and 2)
Revised: Existing benchmarks include 1) MNIST [40] as ID dataset, and Texture [41], CIFAR- 10 [42] or Place 365 [43] as OOD datasets; and 2)

Original: CIFAR-
Revised: CIFAR-

Original: 10 [42] as ID dataset, and MNIST [40], or FashionMNIST [43] as OOD datasets.
Revised: 10 [42] as ID dataset, and MNIST [40], or FashionMNIST [43] as OOD datasets.

Original: In far-OOD case, we find that the ID and OOD datasets have different semantic labels and different styles.
Revised: In far-OOD case, we find that the ID and OOD datasets have different semantic labels and different styles.

Original: From the theoretical view, we can define far-OOD detection tasks as follows: for τ > 0 , a domain space D XY is τ -far-OOD, if for any domain D XY ∈ D XY , dist(supp D X O , supp D X I ) > τ.
Revised: From the theoretical view, we can define far-OOD detection tasks as follows: for τ > 0 , a domain space D XY is τ -far-OOD, if for any domain D XY ∈ D XY , dist(supp D X O , supp D X I ) > τ.

Original: Theorems 7, 8 and 10 imply that under appropriate hypothesis space, τ -far-OOD detection is learnable.
Revised: Theorems 7, 8 and 10 imply that under appropriate hypothesis space, τ -far-OOD detection is learnable.

Original: In Theorem 7, the condition |X| <
Revised: In Theorem 7, the condition |X| <

Original: + ∞ is necessary for the separate space.
Revised: + ∞ is necessary for the separate space.

Original: However, one can prove that in the far-OOD case, when H in is agnostic PAC learnable for ID distribution, the results in
Revised: However, one can prove that in the far-OOD case, when H in is agnostic PAC learnable for ID distribution, the results in

Original: Theorem 7 still holds, if the condition |X| <
Revised: Theorem 7 still holds, if the condition |X| <

Original: + ∞ is replaced by a weaker condition that X is compact.
Revised: + ∞ is replaced by a weaker condition that X is compact.

Original: In addition, it is notable that when H in is agnostic PAC learnable for ID distribution and X is compact, the KNN-based OOD detection algorithm [44] is consistent in the τ -far-OOD case.
Revised: In addition, it is notable that when H in is agnostic PAC learnable for ID distribution and X is compact, the KNN-based OOD detection algorithm [44] is consistent in the τ -far-OOD case.

Original: Understanding Near-OOD Detection.
Revised: Understanding Near-OOD Detection.

Original: When the ID and OOD datasets have similar semantics or styles, OOD detection tasks become more challenging.
Revised: When the ID and OOD datasets have similar semantics or styles, OOD detection tasks become more challenging.

Original: [45, 46] consider this issue and names it near-OOD detection.
Revised: [45, 46] consider this issue and names it near-OOD detection.

Original: Existing benchmarks include 1) MNIST [40] as ID dataset, and Fashion-MNIST [43] or Not-MNIST [47] as OOD datasets; and 2) CIFAR-
Revised: Existing benchmarks include 1) MNIST [40] as ID dataset, and Fashion-MNIST [43] or Not-MNIST [47] as OOD datasets; and 2) CIFAR-

Original: 10 [42] as ID dataset, and CIFAR- 100 [48] as OOD dataset.
Revised: 10 [42] as ID dataset, and CIFAR- 100 [48] as OOD dataset.

Original: From the theoretical view, some near-OOD tasks may imply the overlap condition, i.e. Definition 4.
Revised: From the theoretical view, some near-OOD tasks may imply the overlap condition, i.e. Definition 4.

Original: Therefore, Theorems 3 and 12 imply that near-OOD detection may be not learnable.
Revised: Therefore, Theorems 3 and 12 imply that near-OOD detection may be not learnable.

Original: Developing a theory to understand the feasibility of near-OOD detection is still an open question .
Revised: Developing a theory to understand the feasibility of near-OOD detection is still an open question .

Original: Understanding One-class Novelty Detection.
Revised: Understanding One-class Novelty Detection.

Original: In one-class novelty detection and semantic anomaly detection ( i.e. K = 1 ), Theorem 6 has revealed that it is necessary to use a large-capacity model to ensure the good generalization in the separate space.
Revised: In one-class novelty detection and semantic anomaly detection ( i.e. K = 1 ), Theorem 6 has revealed that it is necessary to use a large-capacity model to ensure the good generalization in the separate space.

Original: Theorem 3 and Theorem 12 suggest that we should try to avoid the overlap between ID and OOD distributions in the one-class case.
Revised: Theorem 3 and Theorem 12 suggest that we should try to avoid the overlap between ID and OOD distributions in the one-class case.

Original: If the overlap cannot be avoided, we suggest considering the multi-class OOD detection instead of the one-class case.
Revised: If the overlap cannot be avoided, we suggest considering the multi-class OOD detection instead of the one-class case.

Original: Additionally, in the density-based space, Theorem 11 has shown that it is necessary to select a suitable hypothesis space satisfying the Realizability Assumption to ensure the learnability of OOD detection in the density-based space.
Revised: Additionally, in the density-based space, Theorem 11 has shown that it is necessary to select a suitable hypothesis space satisfying the Realizability Assumption to ensure the learnability of OOD detection in the density-based space.

Original: Generally, a large-capacity model can be helpful to guarantee that the Realizability Assumption holds.
Revised: Generally, a large-capacity model can be helpful to guarantee that the Realizability Assumption holds.

Original: FPR and AUROC Metrics.
Revised: FPR and AUROC Metrics.

Original: FPR ( i.e. , False Positive Rate) and AUROC ( i.e. , Area under the Receiver
Revised: FPR ( i.e. , False Positive Rate) and AUROC ( i.e. , Area under the Receiver

Original: Operating Characteristic Curve) metrics are commonly used metrics in OOD detection [4].
Revised: Operating Characteristic Curve) metrics are commonly used metrics in OOD detection [4].

Original: TPR ( i.e. , True Positive Rate) is a corresponding metric with respect to FPR.
Revised: TPR ( i.e. , True Positive Rate) is a corresponding metric with respect to FPR.

Original: Generally, we hope that an
Revised: Generally, we hope that an

Original: OOD detection algorithm can approximate the optimal TPR and FPR performances.
Revised: OOD detection algorithm can approximate the optimal TPR and FPR performances.

Original: With minor adjustment, our theory can also be used to understand OOD detection under the TPR and FPR metrics.
Revised: With minor adjustment, our theory can also be used to understand OOD detection under the TPR and FPR metrics.

Original: However, due to the complexity of AUROC, we still cannot give an OOD detection theory under
Revised: However, due to the complexity of AUROC, we still cannot give an OOD detection theory under

Original: AUROC metric.
Revised: AUROC metric.

Original: Understanding OOD detection under AUROC metric is still an open question .
Revised: Understanding OOD detection under AUROC metric is still an open question .

Original:  Related Work
Revised:  Related Work

Original: We briefly review the related theoretical works below.
Revised: We briefly review the related theoretical works below.

Original: See Appendix A for detailed related works.
Revised: See Appendix A for detailed related works.

Original: OOD Detection Theory.
Revised: OOD Detection Theory.

Original: [49] understands the OOD detection via goodness-of-fit tests and typical set hypothesis, and argues that minimal density estimation errors can lead to OOD detection failures, when there exists an overlap between ID and OOD distributions.
Revised: [49] understands the OOD detection via goodness-of-fit tests and typical set hypothesis, and argues that minimal density estimation errors can lead to OOD detection failures, when there exists an overlap between ID and OOD distributions.

Original: Beyond [49], [50] paves a new avenue to designing provable OOD detection algorithms.
Revised: Beyond [49], [50] paves a new avenue to designing provable OOD detection algorithms.

Original: Compared to [50, 49], our theory focuses on the PAC learnable theory of OOD detection and identifies several necessary and sufficient conditions for the learnability of OOD detection, opening a door to study OOD detection in theory.
Revised: Compared to [50, 49], our theory focuses on the PAC learnable theory of OOD detection and identifies several necessary and sufficient conditions for the learnability of OOD detection, opening a door to study OOD detection in theory.

Original: Open-set Learning Theory.
Revised: Open-set Learning Theory.

Original: [51] and [29, 52] propose the agnostic PAC learning bounds for open-set detection and open-set domain adaptation, respectively.
Revised: [51] and [29, 52] propose the agnostic PAC learning bounds for open-set detection and open-set domain adaptation, respectively.

Original: Unfortunately, [29, 51, 52] all require that the test data are indispensable during the training process.
Revised: Unfortunately, [29, 51, 52] all require that the test data are indispensable during the training process.

Original: To investigate open-set learning (OSL) without accessing the test data during training, [24] proposes and investigates the almost agnostic
Revised: To investigate open-set learning (OSL) without accessing the test data during training, [24] proposes and investigates the almost agnostic

Original: PAC learnability for OSL.
Revised: PAC learnability for OSL.

Original: However, the assumptions used in [24] are very strong and unpractical.
Revised: However, the assumptions used in [24] are very strong and unpractical.

Original: Learning Theory for Classification with Reject Option.
Revised: Learning Theory for Classification with Reject Option.

Original: Many works [53, 54] also investigate the classification with reject option (CwRO) problem, which is similar to OOD detection in some cases.
Revised: Many works [53, 54] also investigate the classification with reject option (CwRO) problem, which is similar to OOD detection in some cases.

Original: [55, 56, 57, 58, 59] study the learning theory and propose the PAC learning bounds for CwRO.
Revised: [55, 56, 57, 58, 59] study the learning theory and propose the PAC learning bounds for CwRO.

Original: However, compared to our work regarding OOD detection, existing CwRO theories mainly focus on how the ID risk R in D ( i.e. , the risk that ID data is wrongly classified) is influenced by special rejection rules.
Revised: However, compared to our work regarding OOD detection, existing CwRO theories mainly focus on how the ID risk R in D ( i.e. , the risk that ID data is wrongly classified) is influenced by special rejection rules.

Original: Our theory not only focuses on the ID risk, but also pays attention to the OOD risk.
Revised: Our theory not only focuses on the ID risk, but also pays attention to the OOD risk.

Original: PQ Learning Theory.
Revised: PQ Learning Theory.

Original: Under some conditions, PQ learning theory [60, 61] can be regarded as the
Revised: Under some conditions, PQ learning theory [60, 61] can be regarded as the

Original: PAC theory for OOD detection in the semi-supervised or transductive learning cases, i.e. , test data are required during training.
Revised: PAC theory for OOD detection in the semi-supervised or transductive learning cases, i.e. , test data are required during training.

Original: Besides, [60, 61] aim to give the PAC estimation under Realizability
Revised: Besides, [60, 61] aim to give the PAC estimation under Realizability

Original: Assumption [21].
Revised: Assumption [21].

Original: Our theory does not only study the PAC estimation in the realization cases, but also studies the other cases, which are more difficult than PAC theory under Realizability Assumption.
Revised: Our theory does not only study the PAC estimation in the realization cases, but also studies the other cases, which are more difficult than PAC theory under Realizability Assumption.

Original:  Conclusions and Future Works
Revised:  Conclusions and Future Works

Original: Detecting OOD data has shown its significance in improving the reliability of machine learning.
Revised: Detecting OOD data has shown its significance in improving the reliability of machine learning.

Original: However, very few works discuss OOD detection in theory, which hinders real-world applications of OOD detection algorithms.
Revised: However, very few works discuss OOD detection in theory, which hinders real-world applications of OOD detection algorithms.

Original: In this paper, we are the first to provide the PAC theory for OOD detection.
Revised: In this paper, we are the first to provide the PAC theory for OOD detection.

Original: Our results imply that we cannot expect a universally consistent algorithm to handle all scenarios in OOD detection.
Revised: Our results imply that we cannot expect a universally consistent algorithm to handle all scenarios in OOD detection.

Original: Yet, it is still possible to make OOD detection learnable in certain scenarios.
Revised: Yet, it is still possible to make OOD detection learnable in certain scenarios.

Original: For example, when we design OOD detection algorithms, we normally only have finite ID datasets.
Revised: For example, when we design OOD detection algorithms, we normally only have finite ID datasets.

Original: In this real scenario, Theorem 8 provides a necessary and sufficient condition for the success of OOD detection.
Revised: In this real scenario, Theorem 8 provides a necessary and sufficient condition for the success of OOD detection.

Original: Our theory reveals many necessary and sufficient conditions for the learnability of
Revised: Our theory reveals many necessary and sufficient conditions for the learnability of

Original: OOD detection, hence opening a door to studying the learnability of OOD detection.
Revised: OOD detection, hence opening a door to studying the learnability of OOD detection.

Original: In future, we will focus on studying the robustness of OOD detection based on robust statistics [62, 63].
Revised: In future, we will focus on studying the robustness of OOD detection based on robust statistics [62, 63].

Original: Acknowledgment
Revised: Acknowledgment

Original: JL and ZF were supported by the Australian Research Council (ARC) under FL190100149.
Revised: JL and ZF were supported by the Australian Research Council (ARC) under FL190100149.

Original: YXL was supported by the Wisconsin Alumni Research Foundation (WARF).
Revised: YXL was supported by the Wisconsin Alumni Research Foundation (WARF).

Original: BH was supported by the
Revised: BH was supported by the

Original: RGC Early Career Scheme No. 22200720. The authors thank the reviewers and the meta-reviewer for their helpful comments.
Revised: RGC Early Career Scheme No. 22200720 and NSFC Young Scientists Fund No. 62006202.

Original: ZF would also like to thank Dr. Tongliang Liu for productive discussions.
Revised: ZF would also like to thank Dr

Original: B Limitations and Potential Negative Societal Impacts
Revised: B Limitations and Potential Negative Societal Impacts

Original: C Discussions and Details about Experiments in Figure
Revised: C Discussions and Details about Experiments in Figure

Original: C.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.2 Details of Experiments in Figure 1 . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.2 Details of Experiments in Figure 1 . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D Notations
Revised: D Notations

Original: D.1 Main Notations and Their Descriptions . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1 Main Notations and Their Descriptions . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.2 Realizability Assumption . .
Revised: D.2 Realizability Assumption . .

Original: D.3 Learnability and PAC learnability . .
Revised: D.3 Learnability and PAC learnability . .

Original: D.4 Explanations for Some Notations in Section 2 . . . . . . . . . . . . . . . . . . . .
Revised: D.4 Explanations for Some Notations in Section 2 . . . . . . . . . . . . . . . . . . . .

Original: E Proof of Theorem
Revised: E Proof of Theorem

Original: F Proof of Theorem
Revised: F Proof of Theorem

Original: G Proofs of Theorem 3 and Theorem
Revised: G Proofs of Theorem 3 and Theorem

Original: G.1 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.1 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.2 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H Proof of Theorem
Revised: H Proof of Theorem

Original: I
Revised: I

Original: Proofs of Theorem 6 and Theorem
Revised: Proofs of Theorem 6 and Theorem

Original: I.
Revised: I.

Original: Proof of Theorem 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Proof of Theorem 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: J Proofs of Theorems 8 and
Revised: J Proofs of Theorems 8 and

Original: J
Revised: J

Original: Proof
Revised: Proof

Original: of Theorem 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: of Theorem 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: J.
Revised: J.

Original: Proof
Revised: Proof

Original: of Theorem 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: of Theorem 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: K Proof of Proposition 1 and Proof of Proposition
Revised: K Proof of Proposition 1 and Proof of Proposition

Original: L Proof of Theorem
Revised: L Proof of Theorem

Original: M Proofs of Theorem 11 and Theorem
Revised: M Proofs of Theorem 11 and Theorem

Original: M.1 Proof of Theorem 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: M.1 Proof of Theorem 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: M.2 Proof of Theorem 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: M.2 Proof of Theorem 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A Detailed Related Work
Revised: A Detailed Related Work

Original: OOD Detection Algorithms.
Revised: OOD Detection Algorithms.

Original: We will briefly review many representative OOD detection algorithms in three categories.
Revised: We will briefly review many representative OOD detection algorithms in three categories.

Original: 1) Classification-based methods use an ID classifier to detect OOD data [7] 5 .
Revised: 1) Classification-based methods use an ID classifier to detect OOD data [7] 5 .

Original: Representative works consider using the maximum softmax score [7], temperature-scaled score [14] and energy-based score [23, 65] to identify OOD data.
Revised: Representative works consider using the maximum softmax score [7], temperature-scaled score [14] and energy-based score [23, 65] to identify OOD data.

Original: 2) Density-based methods aim to estimate an ID distribution and identify the low-density area as OOD data [10].
Revised: 2) Density-based methods aim to estimate an ID distribution and identify the low-density area as OOD data [10].

Original: The recent development of generative models provides promising ways to make them successful in OOD detection [11, 12, 14, 66, 67].
Revised: The recent development of generative models provides promising ways to make them successful in OOD detection [11, 12, 14, 66, 67].

Original: Distance-based methods are based on the assumption that OOD data should be relatively far away from the centroids of ID classes [9], including Mahalanobis distance [9, 45], cosine similarity [68], and kernel similarity [69].
Revised: Distance-based methods are based on the assumption that OOD data should be relatively far away from the centroids of ID classes [9], including Mahalanobis distance [9, 45], cosine similarity [68], and kernel similarity [69].

Original: Early works consider using the maximum softmax score to express the ID-ness [7].
Revised: Early works consider using the maximum softmax score to express the ID-ness [7].

Original: Then, temperature scaling functions are used to amplify the separation between the ID and OOD data [14].
Revised: Then, temperature scaling functions are used to amplify the separation between the ID and OOD data [14].

Original: Recently, researchers propose hyperparameter-free energy scores to improve the OOD uncertainty estimation [23, 65].
Revised: Recently, researchers propose hyperparameter-free energy scores to improve the OOD uncertainty estimation [23, 65].

Original: Additionally, researchers also consider using the information contained in gradients to help improve the performance of OOD detection [18].
Revised: Additionally, researchers also consider using the information contained in gradients to help improve the performance of OOD detection [18].

Original: Except for the above algorithms, researchers also study the situation, where auxiliary OOD data can be obtained during the training process [13, 64].
Revised: Except for the above algorithms, researchers also study the situation, where auxiliary OOD data can be obtained during the training process [13, 64].

Original: These methods are called outlier exposure, and have much better performance than the above methods due to the appearance of OOD data.
Revised: These methods are called outlier exposure, and have much better performance than the above methods due to the appearance of OOD data.

Original: However, the exposure of OOD data is a strong assumption [4].
Revised: However, the exposure of OOD data is a strong assumption [4].

Original: Thus, researchers also consider generating OOD data to help the separation of OOD and ID data [70].
Revised: Thus, researchers also consider generating OOD data to help the separation of OOD and ID data [70].

Original: In this paper, we do not make an assumption that OOD data are available during training, since this assumption may not hold in real world.
Revised: In this paper, we do not make an assumption that OOD data are available during training, since this assumption may not hold in real world.

Original: OOD Detection Theory.
Revised: OOD Detection Theory.

Original: [49] understands the OOD detection via goodness-of-fit tests and typical set hypothesis, and discovers that relevant OOD distributions can lie in the high likelihood regions of a data distribution.
Revised: [49] understands the OOD detection via goodness-of-fit tests and typical set hypothesis, and discovers that relevant OOD distributions can lie in the high likelihood regions of a data distribution.

Original: [49] argues that minimal density estimation errors can lead to OOD detection failures, when there is an overlap between ID and OOD distributions.
Revised: [49] argues that minimal density estimation errors can lead to OOD detection failures, when there is an overlap between ID and OOD distributions.

Original: Compared to [49], our theory focuses on the PAC learnable theory of OOD detection.
Revised: Compared to [49], our theory focuses on the PAC learnable theory of OOD detection.

Original: If detectors are generated by FCNN, our theory (Theorem 12) shows that the overlap is the sufficient condition to the failure of learnability of OOD detection, which is complementary to [49].
Revised: If detectors are generated by FCNN, our theory (Theorem 12) shows that the overlap is the sufficient condition to the failure of learnability of OOD detection, which is complementary to [49].

Original: In addition, we identify several necessary and sufficient conditions for the learnability of OOD detection, which opens a door to studying OOD detection in theory.
Revised: In addition, we identify several necessary and sufficient conditions for the learnability of OOD detection, which opens a door to studying OOD detection in theory.

Original: Beyond [49], [50] paves a new avenue to designing provable OOD detection algorithms.
Revised: Beyond [49], [50] paves a new avenue to designing provable OOD detection algorithms.

Original: Compared to [50], our paper aims to characterize the learnability of OOD detection to answer the question: is OOD detection PAC learnable?
Revised: Compared to [50], our paper aims to characterize the learnability of OOD detection to answer the question: is OOD detection PAC learnable?

Original: Open-set Learning Theory.
Revised: Open-set Learning Theory.

Original: [51] is the first to propose the agnostic PAC guarantees for open-set detection.
Revised: [51] is the first to propose the agnostic PAC guarantees for open-set detection.

Original: Unfortunately, the test data must be used during the training process.
Revised: Unfortunately, the test data must be used during the training process.

Original: [29] considers the open-set domain adaptation (OSDA) [52] and proposes the first learning bound for OSDA.
Revised: [29] considers the open-set domain adaptation (OSDA) [52] and proposes the first learning bound for OSDA.

Original: [29] mainly depends on the positive-unlabeled learning techniques [71, 72, 73].
Revised: [29] mainly depends on the positive-unlabeled learning techniques [71, 72, 73].

Original: However, similar to [51], the test data must be available during training.
Revised: However, similar to [51], the test data must be available during training.

Original: To study open-set learning (OSL) without accessing the test data during training, [24] proposes and studies the almost PAC learnability for OSL, which is motivated by transfer learning [74, 75].
Revised: To study open-set learning (OSL) without accessing the test data during training, [24] proposes and studies the almost PAC learnability for OSL, which is motivated by transfer learning [74, 75].

Original: In our paper, we study the PAC learnability for OOD detection, which is an open problem proposed by [24].
Revised: In our paper, we study the PAC learnability for OOD detection, which is an open problem proposed by [24].

Original: Learning Theory for Classification with Reject Option.
Revised: Learning Theory for Classification with Reject Option.

Original: Many works [53, 54] also investigate the classification with reject option (CwRO) problem, which is similar to OOD detection in some cases.
Revised: Many works [53, 54] also investigate the classification with reject option (CwRO) problem, which is similar to OOD detection in some cases.

Original: [55, 56, 57, 58, 59] study the learning theory and propose the agnostic PAC learning bounds for
Revised: [55, 56, 57, 58, 59] study the learning theory and propose the agnostic PAC learning bounds for

Original: CwRO.
Revised: CwRO.

Original: However, compared to our work regarding OOD detection, existing CwRO theories mainly focus on how the ID risk ( i.e. , the risk that ID data is wrongly classified) is influenced by special rejection rules.
Revised: However, compared to our work regarding OOD detection, existing CwRO theories mainly focus on how the ID risk ( i.e. , the risk that ID data is wrongly classified) is influenced by special rejection rules.

Original: Our theory not only focuses on the ID risk, but also pays attention to the OOD risk.
Revised: Our theory not only focuses on the ID risk, but also pays attention to the OOD risk.

Original: PQ Learning Theory.
Revised: PQ Learning Theory.

Original: Under some conditions, PQ learning theory [60, 61] can be regarded as the
Revised: Under some conditions, PQ learning theory [60, 61] can be regarded as the

Original: PAC theory for OOD detection in the semi-supervised or transductive learning cases, i.e. , test data are required during the training process.
Revised: PAC theory for OOD detection in the semi-supervised or transductive learning cases, i.e. , test data are required during the training process.

Original: Additionally, PQ learning theory in [60, 61] aims to give the PAC estimation under Realizability Assumption [21].
Revised: Additionally, PQ learning theory in [60, 61] aims to give the PAC estimation under Realizability Assumption [21].

Original: Our theory focuses on the PAC theory in different cases, which is more difficult and more practical than PAC theory under Realizability
Revised: Our theory focuses on the PAC theory in different cases, which is more difficult and more practical than PAC theory under Realizability

Original: Assumption.
Revised: Assumption.

Original: B Limitations and Potential Negative Societal Impacts
Revised: B Limitations and Potential Negative Societal Impacts

Original: Limitations.
Revised: Limitations.

Original: The main limitation of our work lies in that we do not answer the most general question:
Revised: The main limitation of our work lies in that we do not answer the most general question:

Original: Given any hypothesis space H and space D XY , what is the necessary and sufficient condition to ensure the PAC learnability of OOD detection?
Revised: Given any hypothesis space H and space D XY , what is the necessary and sufficient condition to ensure the PAC learnability of OOD detection?

Original: However, this question is still difficult to be addressed, due to limited mathematical skills.
Revised: However, this question is still difficult to be addressed, due to limited mathematical skills.

Original: Yet, based on our observations and the main results in our paper, we believe the following result may hold:
Revised: Yet, based on our observations and the main results in our paper, we believe the following result may hold:

Original: Conjecture : If H is agnostic learnable for supervised learning, then OOD detection is learnable in D
Revised: Conjecture : If H is agnostic learnable for supervised learning, then OOD detection is learnable in D

Original: XY if and only if compatibility condition (i.e., Condition 3) holds.
Revised: XY if and only if compatibility condition (i.e., Condition 3) holds.

Original: We leave this question as a future work.
Revised: We leave this question as a future work.

Original: Potential Negative Societal Impacts.
Revised: Potential Negative Societal Impacts.

Original: Since our paper is a theoretical paper and the OOD detection problem is significant to ensure the safety of deploying existing machine learning algorithms, there are no potential negative societal impacts in our paper.
Revised: Since our paper is a theoretical paper and the OOD detection problem is significant to ensure the safety of deploying existing machine learning algorithms, there are no potential negative societal impacts in our paper.

Original: C Discussions and Details about Experiments in Figure
Revised: C Discussions and Details about Experiments in Figure

Original: In this section, we summarize our main results, then give the details of the experiments in Figure 1.
Revised: In this section, we summarize our main results, then give the details of the experiments in Figure 1.

Original: C.1 Summary
Revised: C.1 Summary

Original: We summarize our main results as follows:
Revised: We summarize our main results as follows:

Original: • A necessary condition ( i.e. , Condition 1) for the learnability of OOD detection is proposed.
Revised: • A necessary condition ( i.e. , Condition 1) for the learnability of OOD detection is proposed.

Original: Theorem
Revised: Theorem

Original: shows that Condition 1 is the necessary and sufficient condition for the learnability of OOD detection, when the domain space is the single-distribution space D D XY XY .
Revised: shows that Condition 1 is the necessary and sufficient condition for the learnability of OOD detection, when the domain space is the single-distribution space D D XY XY .

Original: This implies the Condition 1 is the necessary condition for the learnability of OOD detection.
Revised: This implies the Condition 1 is the necessary condition for the learnability of OOD detection.

Original: • Theorem 3 has shown that the overlap between ID and OOD data can lead the failures of OOD detection under some mild assumptions.
Revised: • Theorem 3 has shown that the overlap between ID and OOD data can lead the failures of OOD detection under some mild assumptions.

Original: Furthermore, Theorem 12 shows that when K = 1 , the overlap is the sufficient condition for the failures of OOD detection, when the hypothesis space is FCNN-based or score-based.
Revised: Furthermore, Theorem 12 shows that when K = 1 , the overlap is the sufficient condition for the failures of OOD detection, when the hypothesis space is FCNN-based or score-based.

Original: • Theorem 4 provides an impossibility theorem for the total space D all XY .
Revised: • Theorem 4 provides an impossibility theorem for the total space D all XY .

Original: OOD detection is not learnable in D all XY for any non-trivial hypothesis space.
Revised: OOD detection is not learnable in D all XY for any non-trivial hypothesis space.

Original: • Theorem 5 gives impossibility theorems for the separate space D sXY .
Revised: • Theorem 5 gives impossibility theorems for the separate space D sXY .

Original: To ensure the impossibility theorems hold, mild assumptions are required.
Revised: To ensure the impossibility theorems hold, mild assumptions are required.

Original: Theorem 5 also implies that OOD detection may be learnable in the separate space D sXY , if the feature space is finite, i.e. , |X| < + ∞ .
Revised: Theorem 5 also implies that OOD detection may be learnable in the separate space D sXY , if the feature space is finite, i.e. , |X| < + ∞ .

Original: Additionally, Theorem 10 implies that the finite feature space may be the necessary condition to ensure the learnability of OOD detection in the separate space.
Revised: Additionally, Theorem 10 implies that the finite feature space may be the necessary condition to ensure the learnability of OOD detection in the separate space.

Original: • When |X| < + ∞ and K = 1 , Theorem 6 provides the necessary and sufficient condition for the learnability of OOD detection in the separate space D sXY .
Revised: • When |X| < + ∞ and K = 1 , Theorem 6 provides the necessary and sufficient condition for the learnability of OOD detection in the separate space D sXY .

Original: Theorem 6 implies that if the OOD detection can be learnable in the distribution-agnostic case, then a large-capacity model is necessary.
Revised: Theorem 6 implies that if the OOD detection can be learnable in the distribution-agnostic case, then a large-capacity model is necessary.

Original: Based on Theorem 6, Theorem 7 studies the learnability in the K > 1 case.
Revised: Based on Theorem 6, Theorem 7 studies the learnability in the K > 1 case.

Original: • The compatibility condition ( i.e. , Condition 3) for the learnability of OOD detection is proposed.
Revised: • The compatibility condition ( i.e. , Condition 3) for the learnability of OOD detection is proposed.

Original: Theorem 8 shows that Condition 3 is the necessary and sufficient condition for the learnability of OOD detection in the finite-ID-distribution space D FXY .
Revised: Theorem 8 shows that Condition 3 is the necessary and sufficient condition for the learnability of OOD detection in the finite-ID-distribution space D FXY .

Original: This also implies Condition 3 is the necessary condition for any prior-unknown space.
Revised: This also implies Condition 3 is the necessary condition for any prior-unknown space.

Original: Note that we can only collect finite ID datasets to build models.
Revised: Note that we can only collect finite ID datasets to build models.

Original: Hence, Theorem 8 can handle the most practical scenarios.
Revised: Hence, Theorem 8 can handle the most practical scenarios.

Original: • To further understand the importance of the compatibility condition (Condition 3).
Revised: • To further understand the importance of the compatibility condition (Condition 3).

Original: Theoremconsiders the density-based space D µ,bXY .
Revised: Theoremconsiders the density-based space D µ,bXY .

Original: We discover that Realizability Assumption implies the compatibility condition in the density-based space.
Revised: We discover that Realizability Assumption implies the compatibility condition in the density-based space.

Original: Based on this observation, we prove that OOD detection is learnable in D µ,bXY under Realizability Assumption.
Revised: Based on this observation, we prove that OOD detection is learnable in D µ,bXY under Realizability Assumption.

Original: • Theorem 10 gives practical applications of our theory.
Revised: • Theorem 10 gives practical applications of our theory.

Original: In this theorem, we discover that the finite feature space is a necessary and sufficient condition for the learnability of OOD detection in the separate space D sXY , when the hypothesis space is FCNN-based or score-based.
Revised: In this theorem, we discover that the finite feature space is a necessary and sufficient condition for the learnability of OOD detection in the separate space D sXY , when the hypothesis space is FCNN-based or score-based.

Original: • Theorem 11 has shown that when K = 1 and the hypothesis space is FCNN-based or score-based, Realizability Assumption, Condition 3, Condition 1 and the learnability of OOD detection in the density-based space D µ,bXY are all equivalent .
Revised: • Theorem 11 has shown that when K = 1 and the hypothesis space is FCNN-based or score-based, Realizability Assumption, Condition 3, Condition 1 and the learnability of OOD detection in the density-based space D µ,bXY are all equivalent .

Original: Meaning of Our Theory.
Revised: Meaning of Our Theory.

Original: In classical statistical learning theory, the generalization theory guarantees that a well-trained classifier can be generalized well on the test set as long as the training and test sets are from the same distribution [21, 22].
Revised: In classical statistical learning theory, the generalization theory guarantees that a well-trained classifier can be generalized well on the test set as long as the training and test sets are from the same distribution [21, 22].

Original: However, since the OOD data are unseen during the training process, it is very difficult to determine whether the generalization theory holds for OOD detection.
Revised: However, since the OOD data are unseen during the training process, it is very difficult to determine whether the generalization theory holds for OOD detection.

Original: Normally, OOD data are unseen and can be various.
Revised: Normally, OOD data are unseen and can be various.

Original: We hope that there exists an algorithm that can be used for the various OOD data instead of some certain OOD data, which is the reason why the generalization theory for OOD detection needs to be developed.
Revised: We hope that there exists an algorithm that can be used for the various OOD data instead of some certain OOD data, which is the reason why the generalization theory for OOD detection needs to be developed.

Original: In this paper, we investigate the generalization theory regarding OOD detection and point out when the OOD detection can be successful.
Revised: In this paper, we investigate the generalization theory regarding OOD detection and point out when the OOD detection can be successful.

Original: Our theory is based on the PAC learning theory.
Revised: Our theory is based on the PAC learning theory.

Original: The impossibility theorems and the given necessary and sufficient conditions outlined provide important perspectives from which to think about OOD detection.
Revised: The impossibility theorems and the given necessary and sufficient conditions outlined provide important perspectives from which to think about OOD detection.

Original: C.2 Details of Experiments in Figure
Revised: C.2 Details of Experiments in Figure

Original: In this subsection we present details of the experiments in Figure 1, including data generation, configuration and OOD detection procedure.
Revised: In this subsection we present details of the experiments in Figure 1, including data generation, configuration and OOD detection procedure.

Original: Data Generation.
Revised: Data Generation.

Original: ID and OOD data are drawn from the following uniform (U) distributions (note that we use U( I ) to present the uniform distribution in region I )
Revised: ID and OOD data are drawn from the following uniform (U) distributions (note that we use U( I ) to present the uniform distribution in region I )

Original: • The marginal distribution of ID distribution for class c : for any c ∈ { 1 , ..., 10 } ,
Revised: • The marginal distribution of ID distribution for class c : for any c ∈ { 1 , ..., 10 } ,

Original: [Equation]
Revised: [Equation]

Original: here d i = 5 + gap II ∗ ( i − 1) + 4( i − 2) and gap II is a positive constant.
Revised: here d i = 5 + gap II ∗ ( i − 1) + 4( i − 2) and gap II is a positive constant.

Original: • The class-prior probability for class c : for any c
Revised: • The class-prior probability for class c : for any c

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The
Revised: The

Original: marginal distribution of OOD distribution:
Revised: marginal distribution of OOD distribution:

Original: [Equation]
Revised: [Equation]

Original: Figure 2 shows the OOD and ID distributions, when gap II = 20 and gap IO = − 2 .
Revised: Figure 2 shows the OOD and ID distributions, when gap II = 20 and gap IO = − 2 .

Original: In Figure 1, we draw n data from ID distribution ( n = 15 , 000 , 20 , 000 , 25 , 000 ) and 25 , 000 data from the OOD distribution.
Revised: In Figure 1, we draw n data from ID distribution ( n = 15 , 000 , 20 , 000 , 25 , 000 ) and 25 , 000 data from the OOD distribution.

Original: Configuration.
Revised: Configuration.

Original: The architecture of ID classifier is a four-layer FCNN.
Revised: The architecture of ID classifier is a four-layer FCNN.

Original: The number of neurons in hidden layers is set to 100 , and the number of neurons of output layer is set to 10 .
Revised: The number of neurons in hidden layers is set to 100 , and the number of neurons of output layer is set to 10 .

Original: These neurons use sigmoid activations.
Revised: These neurons use sigmoid activations.

Original: We use the Adam optimizer [76] to optimize the network’s parameters (with the ℓ 2 loss).
Revised: We use the Adam optimizer [76] to optimize the network’s parameters (with the ℓ 2 loss).

Original: The learning rate is set to 0 .
Revised: The learning rate is set to 0 .

Original: 001 , and the max number of training iterations is set to 10 , 000 .
Revised: 001 , and the max number of training iterations is set to 10 , 000 .

Original: Within each iteration, we use full batch to update the network’s parameters.
Revised: Within each iteration, we use full batch to update the network’s parameters.

Original: gap II is set to 20 in our experiments.
Revised: gap II is set to 20 in our experiments.

Original: In Figure 1b, gap IO = − 2 (the overlap exists, see Figure 2), and in Figure 1c, gap IO = 100 (no overlap).
Revised: In Figure 1b, gap IO = − 2 (the overlap exists, see Figure 2), and in Figure 1c, gap IO = 100 (no overlap).

Original: OOD Detection Procedure.
Revised: OOD Detection Procedure.

Original: We first train an ID classifier with n data drawn from the ID distribution.
Revised: We first train an ID classifier with n data drawn from the ID distribution.

Original: Then, according to [23], we apply the free-energy score to identify the OOD data and calculate the α -risk (with the 0 - 1 loss).
Revised: Then, according to [23], we apply the free-energy score to identify the OOD data and calculate the α -risk (with the 0 - 1 loss).

Original: We repeat the above detection procedure 20 times and report the average α -risk in Figure 1.
Revised: We repeat the above detection procedure 20 times and report the average α -risk in Figure 1.

Original: Note that, following [23], we choose the threshold used by the free-energy method so that 95% of ID data are correctly identified as the ID classes by the OOD detector.
Revised: Note that, following [23], we choose the threshold used by the free-energy method so that 95% of ID data are correctly identified as the ID classes by the OOD detector.

Original: [Figure]
Revised: [Figure]

Original: D Notations D.1 Main Notations and Their Descriptions
Revised: D Notations D.1 Main Notations and Their Descriptions

Original: In this section, we summarize important notations in Table 1.
Revised: In this section, we summarize important notations in Table 1.

Original: Given f =
Revised: Given f =

Original: [ f 1 , ..., f l ] ⊤ , for any x ∈ X ,
Revised: [ f 1 , ..., f l ] ⊤ , for any x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where f k is the k -th coordinate of f and f i is the i -th coordinate of f .
Revised: where f k is the k -th coordinate of f and f i is the i -th coordinate of f .

Original: The above definition about arg max aims to overcome some special cases.
Revised: The above definition about arg max aims to overcome some special cases.

Original: For example, there exist k 1 , k 2 ( k 1 < k 2 ) such that f k 1 ( x ) =
Revised: For example, there exist k 1 , k 2 ( k 1 < k 2 ) such that f k 1 ( x ) =

Original: f k 2 ( x ) and f k 1 ( x ) >
Revised: f k 2 ( x ) and f k 1 ( x ) >

Original: f i ( x ) , f k 2 ( x ) >
Revised: f i ( x ) , f k 2 ( x ) >

Original: f i ( x ) , ∀ i ∈ { 1 , ..., l }−{ k 1 , k 2 } .
Revised: f i ( x ) , ∀ i ∈ { 1 , ..., l }−{ k 1 , k 2 } .

Original: Then, according to the above definition, k 2 = arg max k ∈{ 1 ,...,l } f k ( x ) .
Revised: Then, according to the above definition, k 2 = arg max k ∈{ 1 ,...,l } f k ( x ) .

Original: D.2 Realizability Assumption
Revised: D.2 Realizability Assumption

Original: Assumption 2 (Realizability Assumption) .
Revised: Assumption 2 (Realizability Assumption) .

Original: A domain space D XY and hypothesis space H satisfy the Realizability Assumption, if for each domain D XY ∈ D XY , there exists at least one hypothesis function h ∗ ∈ H such that R D ( h ∗ )
Revised: A domain space D XY and hypothesis space H satisfy the Realizability Assumption, if for each domain D XY ∈ D XY , there exists at least one hypothesis function h ∗ ∈ H such that R D ( h ∗ )

Original: = 0 .
Revised: = 0 .

Original: D.3 Learnability and PAC learnability
Revised: D.3 Learnability and PAC learnability

Original: Here we give a proof to show that Learnability given in Definition 1 and PAC learnability are equivalent.
Revised: Here we give a proof to show that Learnability given in Definition 1 and PAC learnability are equivalent.

Original: First , we prove that Learnability concludes the PAC learnability.
Revised: First , we prove that Learnability concludes the PAC learnability.

Original: According to Definition 1,
Revised: According to Definition 1,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: sequence ( l 1 , ..., l g ) to represent the architecture of FCNN
Revised: sequence ( l 1 , ..., l g ) to represent the architecture of FCNN

Original: [Table]
Revised: [Table]

Original: activation function.
Revised: activation function.

Original: In this paper, we use ReLU function
Revised: In this paper, we use ReLU function

Original: [Table]
Revised: [Table]

Original: function
Revised: function

Original: [Table]
Revised: [Table]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: Note that R D ( A ( S )) − inf h ∈H R D ( h ) ≥ 0 .
Revised: Note that R D ( A ( S )) − inf h ∈H R D ( h ) ≥ 0 .

Original: Therefore, by Markov’s inequality, we have
Revised: Therefore, by Markov’s inequality, we have

Original: [Equation]
Revised: [Equation]

Original: Because ϵ cons ( n ) is monotonically decreasing, we can find a smallest m such that ϵ cons ( m ) ≥ ϵδ and ϵ cons ( m − 1) <
Revised: Because ϵ cons ( n ) is monotonically decreasing, we can find a smallest m such that ϵ cons ( m ) ≥ ϵδ and ϵ cons ( m − 1) <

Original: ϵδ , for δ ∈ (0 , 1) .
Revised: ϵδ , for δ ∈ (0 , 1) .

Original: We define that m
Revised: We define that m

Original: ( ϵ, δ )
Revised: ( ϵ, δ )

Original: = m .
Revised: = m .

Original: Therefore, for any ϵ > 0 and δ ∈ (0 , 1) , there exists a function m ( ϵ, δ ) such that when n > m ( ϵ, δ ) , with the probability at least 1 − δ , we have
Revised: Therefore, for any ϵ > 0 and δ ∈ (0 , 1) , there exists a function m ( ϵ, δ ) such that when n > m ( ϵ, δ ) , with the probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: which is the definition of PAC learnability.
Revised: which is the definition of PAC learnability.

Original: Second , we prove that the PAC learnability concludes Learnability.
Revised: Second , we prove that the PAC learnability concludes Learnability.

Original: PAC-learnability: for any ϵ > 0 and 0 < δ < 1 , there exists a function m ( ϵ, δ ) > 0 such that when the sample size n > m ( ϵ, δ ) , we have that with the probability at least 1 − δ > 0 ,
Revised: PAC-learnability: for any ϵ > 0 and 0 < δ < 1 , there exists a function m ( ϵ, δ ) > 0 such that when the sample size n > m ( ϵ, δ ) , we have that with the probability at least 1 − δ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Note that the loss ℓ defined in Section 2 has upper bound (because Y ∪ { K + 1 } is a finite set).
Revised: Note that the loss ℓ defined in Section 2 has upper bound (because Y ∪ { K + 1 } is a finite set).

Original: We assume the upper bound of ℓ is M .
Revised: We assume the upper bound of ℓ is M .

Original: Hence, according to the definition of PAC-learnability, when the sample size n > m ( ϵ, δ ) , we have that
Revised: Hence, according to the definition of PAC-learnability, when the sample size n > m ( ϵ, δ ) , we have that

Original: [Equation]
Revised: [Equation]

Original: If we set δ = ϵ , then when the sample size n > m ( ϵ, ϵ ) , we have that
Revised: If we set δ = ϵ , then when the sample size n > m ( ϵ, ϵ ) , we have that

Original: [Equation]
Revised: [Equation]

Original: this implies that
Revised: this implies that

Original: [Equation]
Revised: [Equation]

Original: which implies the Learnability in Definition 1.
Revised: which implies the Learnability in Definition 1.

Original: We have completed this proof.
Revised: We have completed this proof.

Original: D.4 Explanations for Some Notations in Section
Revised: D.4 Explanations for Some Notations in Section

Original: First, we explain the concept that S ∼ D nX
Revised: First, we explain the concept that S ∼ D nX

Original: I Y I in Eq.
Revised: I Y I in Eq.

Original: S = { ( x 1 , y 1 ) , ..., ( x n , y n ) } is training data drawn independent and identically distributed from
Revised: S = { ( x 1 , y 1 ) , ..., ( x n , y n ) } is training data drawn independent and identically distributed from

Original: D X
Revised: D X

Original: I Y
Revised: I Y

Original: I .
Revised: I .

Original: D nX
Revised: D nX

Original: I
Revised: I

Original: Y
Revised: Y

Original: I denotes the probability over n -tuples induced by applying D X
Revised: I denotes the probability over n -tuples induced by applying D X

Original: I Y I to pick each element of the tuple independently of the other members of the tuple.
Revised: I Y I to pick each element of the tuple independently of the other members of the tuple.

Original: Because these samples are i.i.d.
Revised: Because these samples are i.i.d.

Original: drawn n times, researchers often use ” S ∼ D nX
Revised: drawn n times, researchers often use ” S ∼ D nX

Original: I Y I ” to represent a sample set S (of size n ) whose each element is drawn i.i.d.
Revised: I Y I ” to represent a sample set S (of size n ) whose each element is drawn i.i.d.

Original: from D X I Y I .
Revised: from D X I Y I .

Original: Second, we explain the concept ” + ” in (1 − π out )
Revised: Second, we explain the concept ” + ” in (1 − π out )

Original: D X
Revised: D X

Original: I + π out D X O .
Revised: I + π out D X O .

Original: For convenience, let P =
Revised: For convenience, let P =

Original: (1 − π out ) D X I and Q = π out
Revised: (1 − π out ) D X I and Q = π out

Original: D X
Revised: D X

Original: O .
Revised: O .

Original: It is clear that P and Q are measures.
Revised: It is clear that P and Q are measures.

Original: Then P + Q is also a measure, which is defined as follows: for any measurable set
Revised: Then P + Q is also a measure, which is defined as follows: for any measurable set

Original: A ⊂ X , we have
Revised: A ⊂ X , we have

Original: [Equation]
Revised: [Equation]

Original: For example, when P and Q are discrete measures, then P + Q is also discrete measure: for any x ∈ X ,
Revised: For example, when P and Q are discrete measures, then P + Q is also discrete measure: for any x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: When P and Q are continuous measures with density functions f and g , then P + Q is also continuous measure with density function f + g : for any measurable A ⊂ X ,
Revised: When P and Q are continuous measures with density functions f and g , then P + Q is also continuous measure with density function f + g : for any measurable A ⊂ X ,

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: Third, we explain the concept E ( x ,y ) ∼ D XY ℓ ( h ( x ) , y ) .
Revised: Third, we explain the concept E ( x ,y ) ∼ D XY ℓ ( h ( x ) , y ) .

Original: The concept E ( x ,y ) ∼ D XY ℓ ( h ( x ) , y ) can be computed as follows:
Revised: The concept E ( x ,y ) ∼ D XY ℓ ( h ( x ) , y ) can be computed as follows:

Original: [Equation]
Revised: [Equation]

Original: For example, when D XY is a finite discrete distribution: let Z = { ( x 1 , y 1 ) , ..., ( x m , y m ) } be the support set of D XY , and assume that a i is the probability for ( x i , y i ) ,
Revised: For example, when D XY is a finite discrete distribution: let Z = { ( x 1 , y 1 ) , ..., ( x m , y m ) } be the support set of D XY , and assume that a i is the probability for ( x i , y i ) ,

Original: i.e. , a i = D XY ( x i , y i ) .
Revised: i.e. , a i = D XY ( x i , y i ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: When D X is a continuous distribution with density f , and D Y | X
Revised: When D X is a continuous distribution with density f , and D Y | X

Original: ( Y = k
Revised: ( Y = k

Original: | X = x ) ( k -th classconditional distribution for x ) is a k ( x ) , then
Revised: | X = x ) ( k -th classconditional distribution for x ) is a k ( x ) , then

Original: [Equation]
Revised: [Equation]

Original: where D Y | X ( Y = k
Revised: where D Y | X ( Y = k

Original: | X = x ) is the k -th class-conditional distribution.
Revised: | X = x ) is the k -th class-conditional distribution.

Original: E Proof of Theorem
Revised: E Proof of Theorem

Original: Theorem 1.
Revised: Theorem 1.

Original: Given domain spaces D XY and D ′ XY
Revised: Given domain spaces D XY and D ′ XY

Original: = { D αXY : ∀ D XY ∈ D XY , ∀ α ∈ [0 , 1) } , then
Revised: = { D αXY : ∀ D XY ∈ D XY , ∀ α ∈ [0 , 1) } , then

Original: 1) D ′ XY is a priori-unknown space and D XY ⊂ D ′ XY ; 2) if D XY is a priori-unknown space, then Definition 1 and Definition 2 are equivalent ; 3) OOD detection is strongly learnable in D XY if and only if OOD detection is learnable in D ′ XY .
Revised: 1) D ′ XY is a priori-unknown space and D XY ⊂ D ′ XY ; 2) if D XY is a priori-unknown space, then Definition 1 and Definition 2 are equivalent ; 3) OOD detection is strongly learnable in D XY if and only if OOD detection is learnable in D ′ XY .

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Proof of the First Result.
Revised: Proof of the First Result.

Original: To prove that D ′ XY is a priori-unknown space, we need to show that for any D α ′ XY ∈ D
Revised: To prove that D ′ XY is a priori-unknown space, we need to show that for any D α ′ XY ∈ D

Original: ′ XY , then
Revised: ′ XY , then

Original: D αXY ∈ D ′ XY for any α ∈ [0 , 1) .
Revised: D αXY ∈ D ′ XY for any α ∈ [0 , 1) .

Original: According to the definition of D ′ XY , for any D α ′ XY ∈ D ′ XY , we can find a domain D XY ∈ D XY , which can be written as D XY =
Revised: According to the definition of D ′ XY , for any D α ′ XY ∈ D ′ XY , we can find a domain D XY ∈ D XY , which can be written as D XY =

Original: (1 − π out ) D X
Revised: (1 − π out ) D X

Original: I Y
Revised: I Y

Original: I + π out D X
Revised: I + π out D X

Original: O Y O (here π out ∈ [0 , 1) )
Revised: O Y O (here π out ∈ [0 , 1) )

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: Note that D αXY =
Revised: Note that D αXY =

Original: (1 − α )
Revised: (1 − α )

Original: D X
Revised: D X

Original: I Y
Revised: I Y

Original: I + αD X
Revised: I + αD X

Original: O Y O .
Revised: O Y O .

Original: Therefore, based on the definition of D ′ XY , for any α ∈ [0 , 1) , D αXY ∈ D ′ XY , which implies that D ′
Revised: Therefore, based on the definition of D ′ XY , for any α ∈ [0 , 1) , D αXY ∈ D ′ XY , which implies that D ′

Original: XY is a prior-known space.
Revised: XY is a prior-known space.

Original: Additionally, for any D XY ∈ D XY , we can rewrite D XY as D π out XY , thus D XY
Revised: Additionally, for any D XY ∈ D XY , we can rewrite D XY as D π out XY , thus D XY

Original: = D π out XY ∈ D
Revised: = D π out XY ∈ D

Original: ′ XY , which implies that D XY ⊂ D ′ XY .
Revised: ′ XY , which implies that D XY ⊂ D ′ XY .

Original: Proof of the Second Result.
Revised: Proof of the Second Result.

Original: First, we prove that Definition 1 concludes Definition 2, if D XY is a prior-unknown space:
Revised: First, we prove that Definition 1 concludes Definition 2, if D XY is a prior-unknown space:

Original: The domain space D XY is a priori-unknown space, and OOD detection is learnable in D XY for H .
Revised: The domain space D XY is a priori-unknown space, and OOD detection is learnable in D XY for H .

Original: ⇓
Revised: ⇓

Original: OOD detection is strongly learnable in D XY for H : there exist an algorithm A : ∪ + ∞ n =1 ( X ×Y ) n →
Revised: OOD detection is strongly learnable in D XY for H : there exist an algorithm A : ∪ + ∞ n =1 ( X ×Y ) n →

Original: H , and a monotonically decreasing sequence ϵ ( n ) , such that ϵ ( n ) → 0 , as n → + ∞
Revised: H , and a monotonically decreasing sequence ϵ ( n ) , such that ϵ ( n ) → 0 , as n → + ∞

Original: [Equation]
Revised: [Equation]

Original: In the priori-unknown space, for any D XY ∈ D XY , we have that for any α ∈ [0 , 1) ,
Revised: In the priori-unknown space, for any D XY ∈ D XY , we have that for any α ∈ [0 , 1) ,

Original: [Equation]
Revised: [Equation]

Original: Then, according to the definition of learnability of OOD detection, we have an algorithm A and a monotonically decreasing sequence ϵ cons ( n ) → 0 , as n → + ∞ , such that for any α ∈ [0 , 1) ,
Revised: Then, according to the definition of learnability of OOD detection, we have an algorithm A and a monotonically decreasing sequence ϵ cons ( n ) → 0 , as n → + ∞ , such that for any α ∈ [0 , 1) ,

Original: [Equation]
Revised: [Equation]

Original: n ) , ( by the property of priori-unknown space )
Revised: n ) , ( by the property of priori-unknown space )

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Since R D α ( A ( S )) =
Revised: Since R D α ( A ( S )) =

Original: R αD ( A ( S )) and R D α ( h ) =
Revised: R αD ( A ( S )) and R D α ( h ) =

Original: R αD ( h ) , we have that
Revised: R αD ( h ) , we have that

Original: [Equation]
Revised: [Equation]

Original: Next, we consider the case that α = 1 .
Revised: Next, we consider the case that α = 1 .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Then, we assume that h ϵ ∈ H satisfies that
Revised: Then, we assume that h ϵ ∈ H satisfies that

Original: [Equation]
Revised: [Equation]

Original: It is obvious that
Revised: It is obvious that

Original: [Equation]
Revised: [Equation]

Original: Let α → 1 .
Revised: Let α → 1 .

Original: Then, for any ϵ > 0 ,
Revised: Then, for any ϵ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (10) with Eq.
Revised: (10) with Eq.

Original: (11), we have
Revised: (11), we have

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Hence, Lebesgue’s Dominated Convergence Theorem [36] implies that
Revised: Hence, Lebesgue’s Dominated Convergence Theorem [36] implies that

Original: [Equation]
Revised: [Equation]

Original: Using Eq.
Revised: Using Eq.

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (13), Eq.
Revised: (13), Eq.

Original: (14) with Eq.
Revised: (14) with Eq.

Original: [Equation]
Revised: [Equation]

Original: we obtain that
Revised: we obtain that

Original: [Equation]
Revised: [Equation]

Original: Since R out D ( A ( S ))
Revised: Since R out D ( A ( S ))

Original: =
Revised: =

Original: R 1 D
Revised: R 1 D

Original: ( A ( S ))
Revised: ( A ( S ))

Original: and R out D ( h )
Revised: and R out D ( h )

Original: =
Revised: =

Original: R 1 D ( h ) , we obtain that
Revised: R 1 D ( h ) , we obtain that

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (9) and Eq.
Revised: (9) and Eq.

Original: (16), we have proven that: if the domain space D XY is a priori-unknown space, then OOD detection is learnable in D XY for H .
Revised: (16), we have proven that: if the domain space D XY is a priori-unknown space, then OOD detection is learnable in D XY for H .

Original: ⇓
Revised: ⇓

Original: OOD detection is strongly learnable in D XY for H : there exist an algorithm A : ∪ + ∞ n =1 ( X ×Y ) n → H , and a monotonically decreasing sequence ϵ ( n ) , such that ϵ ( n ) → 0 , as n → + ∞ ,
Revised: OOD detection is strongly learnable in D XY for H : there exist an algorithm A : ∪ + ∞ n =1 ( X ×Y ) n → H , and a monotonically decreasing sequence ϵ ( n ) , such that ϵ ( n ) → 0 , as n → + ∞ ,

Original: [Equation]
Revised: [Equation]

Original: Second , we prove that Definition 2 concludes Definition 1:
Revised: Second , we prove that Definition 2 concludes Definition 1:

Original: OOD detection is strongly learnable in D XY for H : there exist an algorithm A : ∪ + ∞ n =1 ( X ×Y ) n → H , and a monotonically decreasing sequence ϵ ( n ) , such that ϵ ( n ) → 0 , as n → + ∞ ,
Revised: OOD detection is strongly learnable in D XY for H : there exist an algorithm A : ∪ + ∞ n =1 ( X ×Y ) n → H , and a monotonically decreasing sequence ϵ ( n ) , such that ϵ ( n ) → 0 , as n → + ∞ ,

Original: [Equation]
Revised: [Equation]

Original: ∀ D XY ∈
Revised: ∀ D XY ∈

Original: [Equation]
Revised: [Equation]

Original: XY .
Revised: XY .

Original: ⇓
Revised: ⇓

Original: OOD detection is learnable in D XY for H .
Revised: OOD detection is learnable in D XY for H .

Original: If we set α = π out , then E S ∼ D nX
Revised: If we set α = π out , then E S ∼ D nX

Original: I Y I R αD ( A ( S ))
Revised: I Y I R αD ( A ( S ))

Original: ≤ inf h ∈H R αD ( h ) + ϵ ( n ) implies that
Revised: ≤ inf h ∈H R αD ( h ) + ϵ ( n ) implies that

Original: [Equation]
Revised: [Equation]

Original: which means that OOD detection is learnable in D XY for H .
Revised: which means that OOD detection is learnable in D XY for H .

Original: We have completed this proof.
Revised: We have completed this proof.

Original: Proof of the Third Result.
Revised: Proof of the Third Result.

Original: The third result is a simple conclusion of the second result.
Revised: The third result is a simple conclusion of the second result.

Original: Hence, we omit it.
Revised: Hence, we omit it.

Original: F Proof of Theorem
Revised: F Proof of Theorem

Original: Before introducing the proof of Theorem 2, we extend Condition 1 to a general version (Condition 4).
Revised: Before introducing the proof of Theorem 2, we extend Condition 1 to a general version (Condition 4).

Original: Then, Lemma 1 proves that Conditions 1 and 4 are the necessary conditions for the learnability of
Revised: Then, Lemma 1 proves that Conditions 1 and 4 are the necessary conditions for the learnability of

Original: OOD detection.
Revised: OOD detection.

Original: First, we provide the details of Condition 4.
Revised: First, we provide the details of Condition 4.

Original: Let ∆ o l = { ( λ 1 , ..., λ l ) : (cid:80) lj
Revised: Let ∆ o l = { ( λ 1 , ..., λ l ) : (cid:80) lj

Original: =1 λ j < 1 and λ j ≥ 0 , ∀ j = 1 , ..., l } , where l is a positive integer.
Revised: =1 λ j < 1 and λ j ≥ 0 , ∀ j = 1 , ..., l } , where l is a positive integer.

Original: Next, we introduce an important definition as follows:
Revised: Next, we introduce an important definition as follows:

Original: Definition 6 (OOD Convex Decomposition and Convex Domain) .
Revised: Definition 6 (OOD Convex Decomposition and Convex Domain) .

Original: Given any domain D XY ∈ D XY , we say joint distributions Q 1 , ..., Q l , which are defined over X × { K + 1 } , are the OOD convex decomposition for D XY , if
Revised: Given any domain D XY ∈ D XY , we say joint distributions Q 1 , ..., Q l , which are defined over X × { K + 1 } , are the OOD convex decomposition for D XY , if

Original: [Equation]
Revised: [Equation]

Original: for some ( λ 1 , ..., λ l ) ∈ ∆ o l .
Revised: for some ( λ 1 , ..., λ l ) ∈ ∆ o l .

Original: We also say domain D XY ∈ D XY is an OOD convex domain corresponding to OOD convex decomposition Q 1 , ..., Q l , if for any ( α 1 ,
Revised: We also say domain D XY ∈ D XY is an OOD convex domain corresponding to OOD convex decomposition Q 1 , ..., Q l , if for any ( α 1 ,

Original: [Equation]
Revised: [Equation]

Original: .., α l ) ∈ ∆
Revised: .., α l ) ∈ ∆

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We extend the linear condition (Condition 1) to a multi-linear scenario.
Revised: We extend the linear condition (Condition 1) to a multi-linear scenario.

Original: Condition 4 (Multi-linear Condition) .
Revised: Condition 4 (Multi-linear Condition) .

Original: For each OOD convex domain D XY ∈ D XY corresponding to OOD convex decomposition Q 1 , ..., Q l , the following function
Revised: For each OOD convex domain D XY ∈ D XY corresponding to OOD convex decomposition Q 1 , ..., Q l , the following function

Original: [Equation]
Revised: [Equation]

Original: satisfies that
Revised: satisfies that

Original: [Equation]
Revised: [Equation]

Original: where 0 is the 1 × l vector, whose elements are 0 , and α j is the 1 × l vector, whose j -th element isand other elements are 0 .
Revised: where 0 is the 1 × l vector, whose elements are 0 , and α j is the 1 × l vector, whose j -th element isand other elements are 0 .

Original: When l = 1 and the domain space D XY is a priori-unknown space, Condition 4 degenerates into
Revised: When l = 1 and the domain space D XY is a priori-unknown space, Condition 4 degenerates into

Original: Condition 1.
Revised: Condition 1.

Original: Lemma 1 shows that Condition 4 is necessary for the learnability of OOD detection.
Revised: Lemma 1 shows that Condition 4 is necessary for the learnability of OOD detection.

Original: Lemma 1.
Revised: Lemma 1.

Original: Given a priori-unknown space D XY and a hypothesis space H , if OOD detection is learnable in D XY for H , then Conditions 1 and 4 hold.
Revised: Given a priori-unknown space D XY and a hypothesis space H , if OOD detection is learnable in D XY for H , then Conditions 1 and 4 hold.

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Since Condition 1 is a special case of Condition 4, we only need to prove that Condition 4 holds.
Revised: Since Condition 1 is a special case of Condition 4, we only need to prove that Condition 4 holds.

Original: For any OOD convex domain D XY ∈ D XY corresponding to OOD convex decomposition Q 1 , ..., Q l , and any ( α 1 , ..., α l ) ∈ ∆ o l , we set
Revised: For any OOD convex domain D XY ∈ D XY corresponding to OOD convex decomposition Q 1 , ..., Q l , and any ( α 1 , ..., α l ) ∈ ∆ o l , we set

Original: [Equation]
Revised: [Equation]

Original: Then, we define
Revised: Then, we define

Original: [Equation]
Revised: [Equation]

Original: ) Q α , which belongs to D XY .
Revised: ) Q α , which belongs to D XY .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Since OOD detection is learnable in D XY for H , there exist an algorithm A : ∪ + ∞ n =1 ( X × Y ) n → H , and a monotonically decreasing sequence ϵ ( n ) , such that ϵ ( n ) → 0 , as n → + ∞ , and
Revised: Since OOD detection is learnable in D XY for H , there exist an algorithm A : ∪ + ∞ n =1 ( X × Y ) n → H , and a monotonically decreasing sequence ϵ ( n ) , such that ϵ ( n ) → 0 , as n → + ∞ , and

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have that for any ( α 1 , ..., α l ) ∈
Revised: Therefore, we have that for any ( α 1 , ..., α l ) ∈

Original: ∆ o l ,
Revised: ∆ o l ,

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Note that Eq.
Revised: Note that Eq.

Original: (17) implies that
Revised: (17) implies that

Original: [Equation]
Revised: [Equation]

Original: Step 1.
Revised: Step 1.

Original: Since α j / ∈ ∆ o l , we need to prove that
Revised: Since α j / ∈ ∆ o l , we need to prove that

Original: [Equation]
Revised: [Equation]

Original: where α j is the 1 × l vector, whose j -th element is 1 and other elements are 0 .
Revised: where α j is the 1 × l vector, whose j -th element is 1 and other elements are 0 .

Original: Let ˜ D XY
Revised: Let ˜ D XY

Original: = 0 .
Revised: = 0 .

Original: 5 ∗ D X
Revised: 5 ∗ D X

Original: I Y I + 0 .
Revised: I Y I + 0 .

Original: 5 ∗ Q j .
Revised: 5 ∗ Q j .

Original: The second result of Theorem 1 implies that
Revised: The second result of Theorem 1 implies that

Original: [Equation]
Revised: [Equation]

Original: Since R out˜ D ( A ( S ))
Revised: Since R out˜ D ( A ( S ))

Original: =
Revised: =

Original: R Q j ( A ( S )) and R out˜
Revised: R Q j ( A ( S )) and R out˜

Original: D ( h ) =
Revised: D ( h ) =

Original: R Q j ( h ) ,
Revised: R Q j ( h ) ,

Original: [Equation]
Revised: [Equation]

Original: Note that inf
Revised: Note that inf

Original: h ∈H R Q j ( h ) ≤ E S ∼ D nX
Revised: h ∈H R Q j ( h ) ≤ E S ∼ D nX

Original: I Y I
Revised: I Y I

Original: R Q
Revised: R Q

Original: j
Revised: j

Original: ( A ( S )) .
Revised: ( A ( S )) .

Original: We have
Revised: We have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Eq.
Revised: Eq.

Original: (20) implies that
Revised: (20) implies that

Original: [Equation]
Revised: [Equation]

Original: We note that inf h ∈H R Q j ( h ) =
Revised: We note that inf h ∈H R Q j ( h ) =

Original: f D,Q ( α j ) .
Revised: f D,Q ( α j ) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: α j ) , i.e. , lim n → + ∞ g n ( α j ) =
Revised: α j ) , i.e. , lim n → + ∞ g n ( α j ) =

Original: f ( α j ) .
Revised: f ( α j ) .

Original: Step 2.
Revised: Step 2.

Original: It is easy to check that for any ( α 1 , ..., α l ) ∈
Revised: It is easy to check that for any ( α 1 , ..., α l ) ∈

Original: ∆ o l ,
Revised: ∆ o l ,

Original: [Equation]
Revised: [Equation]

Original: According to Eq.
Revised: According to Eq.

Original: (18) and Eq.
Revised: (18) and Eq.

Original: (22), we have
Revised: (22), we have

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (24) with Eq.
Revised: (24) with Eq.

Original: (23), we complete the proof.
Revised: (23), we complete the proof.

Original: Lemma 2.
Revised: Lemma 2.

Original: [Equation]
Revised: [Equation]

Original: if and only if for any ϵ > 0 ,
Revised: if and only if for any ϵ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: For the sake of convenience, we set f D ( α )
Revised: For the sake of convenience, we set f D ( α )

Original: =
Revised: =

Original: inf h ∈H R αD ( h ) , for any α ∈ [0 , 1] .
Revised: inf h ∈H R αD ( h ) , for any α ∈ [0 , 1] .

Original: First , we prove that f D ( α
Revised: First , we prove that f D ( α

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ∈ [0 , 1) implies
Revised: ∈ [0 , 1) implies

Original: [Equation]
Revised: [Equation]

Original: For any ϵ > 0 and 0 ≤ α < 1 , we can find h αϵ ∈ H satisfying that
Revised: For any ϵ > 0 and 0 ≤ α < 1 , we can find h αϵ ∈ H satisfying that

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Note that f D ( α ) =
Revised: Note that f D ( α ) =

Original: (1 − α )
Revised: (1 − α )

Original: f D (0) + αf D (1) , ∀ α ∈ [0 , 1) , i.e.
Revised: f D (0) + αf D (1) , ∀ α ∈ [0 , 1) , i.e.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using Eqs.
Revised: Using Eqs.

Original: (25) and (26), we have that for any 0 ≤ α < 1 , ϵ ≥
Revised: (25) and (26), we have that for any 0 ≤ α < 1 , ϵ ≥

Original: [Equation]
Revised: [Equation]

Original: Since R out D ( h αϵ ) − inf h ∈H R out D ( h ) ≥ 0 and R in D ( h αϵ ) − inf h ∈H R in D ( h ) ≥ 0 , Eq.
Revised: Since R out D ( h αϵ ) − inf h ∈H R out D ( h ) ≥ 0 and R in D ( h αϵ ) − inf h ∈H R in D ( h ) ≥ 0 , Eq.

Original: (27) implies that: for any 0 < α < 1 ,
Revised: (27) implies that: for any 0 < α < 1 ,

Original: R in D ( h αϵ ) ≤ inf h ∈H R in D ( h ) + ϵ/ (1 − α ) ,
Revised: R in D ( h αϵ ) ≤ inf h ∈H R in D ( h ) + ϵ/ (1 − α ) ,

Original: R out D ( h αϵ ) ≤ inf h ∈H R out D ( h ) + ϵ/α.
Revised: R out D ( h αϵ ) ≤ inf h ∈H R out D ( h ) + ϵ/α.

Original: Therefore, h αϵ ∈ { h ′ ∈ H : R in D ( h ′ ) ≤ inf h ∈H R in D ( h )+ ϵ/ (1 − α ) }∩{ h ′ ∈ H : R out D ( h ′ ) ≤ inf h ∈H R out D ( h )+ ϵ/α } .
Revised: Therefore, h αϵ ∈ { h ′ ∈ H : R in D ( h ′ ) ≤ inf h ∈H R in D ( h )+ ϵ/ (1 − α ) }∩{ h ′ ∈ H : R out D ( h ′ ) ≤ inf h ∈H R out D ( h )+ ϵ/α } .

Original: If we set α = 0 .
Revised: If we set α = 0 .

Original: 5 , we obtain that for any ϵ > 0 ,
Revised: 5 , we obtain that for any ϵ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Second , we prove that for any ϵ > 0 , if
Revised: Second , we prove that for any ϵ > 0 , if

Original: [Equation]
Revised: [Equation]

Original: then f D ( α ) =
Revised: then f D ( α ) =

Original: (1 − α )
Revised: (1 − α )

Original: f D (0) + αf D (1) , for any α ∈ [0 , 1) .
Revised: f D (0) + αf D (1) , for any α ∈ [0 , 1) .

Original: Let h ϵ ∈ { h ′ ∈ H :
Revised: Let h ϵ ∈ { h ′ ∈ H :

Original: R in D ( h ′ ) ≤ inf h
Revised: R in D ( h ′ ) ≤ inf h

Original: [Equation]
Revised: [Equation]

Original: R in D ( h
Revised: R in D ( h

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: ∈ H
Revised: ∈ H

Original: [Equation]
Revised: [Equation]

Original: R out
Revised: R out

Original: [Equation]
Revised: [Equation]

Original: ( h
Revised: ( h

Original: [Equation]
Revised: [Equation]

Original: ≤ inf h ∈H R out D ( h )+2 ϵ } .
Revised: ≤ inf h ∈H R out D ( h )+2 ϵ } .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: which implies that | f D ( α ) − (1 − α )
Revised: which implies that | f D ( α ) − (1 − α )

Original: f D (0) − αf D (1) | ≤ 2 ϵ .
Revised: f D (0) − αf D (1) | ≤ 2 ϵ .

Original: As ϵ → 0 , | f D ( α ) − (1 − α )
Revised: As ϵ → 0 , | f D ( α ) − (1 − α )

Original: f D (0) − αf D (1) | ≤ 0 .
Revised: f D (0) − αf D (1) | ≤ 0 .

Original: We have completed the proof.
Revised: We have completed the proof.

Original: Theorem 2.
Revised: Theorem 2.

Original: Given a hypothesis space H and a domain D XY , OOD detection is learnable in the single-distribution space D D XY XY for H if and only if linear condition (i.e., Condition 1) holds.
Revised: Given a hypothesis space H and a domain D XY , OOD detection is learnable in the single-distribution space D D XY XY for H if and only if linear condition (i.e., Condition 1) holds.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Based on Lemma 1, we obtain that Condition 1 is the necessary condition for the learnability of OOD detection in the single-distribution space D D XY XY .
Revised: Based on Lemma 1, we obtain that Condition 1 is the necessary condition for the learnability of OOD detection in the single-distribution space D D XY XY .

Original: Next, it suffices to prove that
Revised: Next, it suffices to prove that

Original: Condition 1 is the sufficient condition for the learnability of OOD detection in the single-distribution space D D XY XY .
Revised: Condition 1 is the sufficient condition for the learnability of OOD detection in the single-distribution space D D XY XY .

Original: We use Lemma 2 to prove the sufficient condition.
Revised: We use Lemma 2 to prove the sufficient condition.

Original: Let F be the infinite sequence set that consists of all infinite sequences, whose coordinates are hypothesis functions, i.e. ,
Revised: Let F be the infinite sequence set that consists of all infinite sequences, whose coordinates are hypothesis functions, i.e. ,

Original: F = { h
Revised: F = { h

Original: =
Revised: =

Original: ( h 1 , ..., h n , ... ) :
Revised: ( h 1 , ..., h n , ... ) :

Original: ∀ h n ∈ H , n = 1 , ...., + ∞} .
Revised: ∀ h n ∈ H , n = 1 , ...., + ∞} .

Original: For each h ∈ F , there is a corresponding algorithm
Revised: For each h ∈ F , there is a corresponding algorithm

Original: A h 6 : A h ( S ) =
Revised: A h 6 : A h ( S ) =

Original: h n , if | S | = n .
Revised: h n , if | S | = n .

Original: F generates an algorithm class A
Revised: F generates an algorithm class A

Original: = { A h : ∀ h ∈ F } .
Revised: = { A h : ∀ h ∈ F } .

Original: We select a consistent algorithm from the algorithm class A .
Revised: We select a consistent algorithm from the algorithm class A .

Original: We construct a special infinite sequence ˜ h
Revised: We construct a special infinite sequence ˜ h

Original: = (˜ h 1 , ..., ˜ h n , ... ) ∈ F .
Revised: = (˜ h 1 , ..., ˜ h n , ... ) ∈ F .

Original: For each positive integer n , we select ˜ h n from { h ′ ∈ H : R in D ( h ′ ) ≤ inf h ∈H R in D ( h ) + 2 /n } ∩ { h ′ ∈
Revised: For each positive integer n , we select ˜ h n from { h ′ ∈ H : R in D ( h ′ ) ≤ inf h ∈H R in D ( h ) + 2 /n } ∩ { h ′ ∈

Original: H : R out D ( h ′ ) ≤ inf h ∈H R out D ( h )
Revised: H : R out D ( h ′ ) ≤ inf h ∈H R out D ( h )

Original: + 2 /n } (the existence of ˜ h n is based on Lemma 2).
Revised: + 2 /n } (the existence of ˜ h n is based on Lemma 2).

Original: It is easy to check that
Revised: It is easy to check that

Original: [Equation]
Revised: [Equation]

Original: Since (1 − α ) inf h ∈H R in D ( h )+ α inf h ∈H R out D ( h ) ≤ inf h ∈H R αD ( h ) , we obtain that for any α ∈ [0 , 1] ,
Revised: Since (1 − α ) inf h ∈H R in D ( h )+ α inf h ∈H R out D ( h ) ≤ inf h ∈H R αD ( h ) , we obtain that for any α ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: We have completed this proof.
Revised: We have completed this proof.

Original: G Proofs of Theorem 3 and TheoremG.1 Proof of Theorem
Revised: G Proofs of Theorem 3 and TheoremG.1 Proof of Theorem

Original: Theorem 3.
Revised: Theorem 3.

Original: Given a hypothesis space H and a prior-unknown space D XY , if there is D XY ∈ D XY , which has overlap between ID and OOD, and inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 , then
Revised: Given a hypothesis space H and a prior-unknown space D XY , if there is D XY ∈ D XY , which has overlap between ID and OOD, and inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 , then

Original: Condition 1 does not hold.
Revised: Condition 1 does not hold.

Original: Therefore, OOD detection is not learnable in D XY for H .
Revised: Therefore, OOD detection is not learnable in D XY for H .

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: We first explain how we get f I and f O in Definition 4.
Revised: We first explain how we get f I and f O in Definition 4.

Original: Since D X is absolutely continuous respect to µ ( D X ≪ µ ), then D X I ≪ µ and D X O ≪ µ .
Revised: Since D X is absolutely continuous respect to µ ( D X ≪ µ ), then D X I ≪ µ and D X O ≪ µ .

Original: By Radon-Nikodym Theorem [36], we know there exist two non-negative functions defined over X : f I and f O such that for any µ -measurable set
Revised: By Radon-Nikodym Theorem [36], we know there exist two non-negative functions defined over X : f I and f O such that for any µ -measurable set

Original: A ⊂ X ,
Revised: A ⊂ X ,

Original: [Equation]
Revised: [Equation]

Original: Second , we prove that for any α ∈ (0 , 1) , inf h ∈H R αD ( h ) > 0 .
Revised: Second , we prove that for any α ∈ (0 , 1) , inf h ∈H R αD ( h ) > 0 .

Original: We define A m = { x ∈ X : f I ( x ) ≥ 1 m and f O ( x ) ≥ 1 m } .
Revised: We define A m = { x ∈ X : f I ( x ) ≥ 1 m and f O ( x ) ≥ 1 m } .

Original: It is clear that
Revised: It is clear that

Original: [Equation]
Revised: [Equation]

Original: >
Revised: >

Original: [Equation]
Revised: [Equation]

Original: and f O ( x ) > 0 } =
Revised: and f O ( x ) > 0 } =

Original: A overlap , and
Revised: A overlap , and

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: which implies that there exists m 0 such that
Revised: which implies that there exists m 0 such that

Original: [Equation]
Revised: [Equation]

Original: For any α ∈ (0 , 1) , we define c α = min y 1 ∈Y all (cid:0) (1 − α ) min y 2 ∈Y ℓ ( y 1 , y 2 ) + αℓ ( y 1 , K + 1) (cid:1) .
Revised: For any α ∈ (0 , 1) , we define c α = min y 1 ∈Y all (cid:0) (1 − α ) min y 2 ∈Y ℓ ( y 1 , y 2 ) + αℓ ( y 1 , K + 1) (cid:1) .

Original: It is clear that c α > 0 for α ∈ (0 , 1) .
Revised: It is clear that c α > 0 for α ∈ (0 , 1) .

Original: Then, for any h ∈ H ,
Revised: Then, for any h ∈ H ,

Original: [Equation]
Revised: [Equation]

Original: ×Y
Revised: ×Y

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Third , Condition 1 indicates that inf h ∈H R αD ( h ) =
Revised: Third , Condition 1 indicates that inf h ∈H R αD ( h ) =

Original: [Equation]
Revised: [Equation]

Original: 1 − α ) inf h ∈H R in D ( h ) + α inf h ∈H R in D ( h ) =(here we have used conditions inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 ), which contradicts with inf h ∈H R αD ( h ) > 0 ( α ∈ (0 , 1) ).
Revised: 1 − α ) inf h ∈H R in D ( h ) + α inf h ∈H R in D ( h ) =(here we have used conditions inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 ), which contradicts with inf h ∈H R αD ( h ) > 0 ( α ∈ (0 , 1) ).

Original: Therefore, Condition 1 does not hold.
Revised: Therefore, Condition 1 does not hold.

Original: Using Lemma 1, we obtain that OOD detection in D XY is not learnable for H .
Revised: Using Lemma 1, we obtain that OOD detection in D XY is not learnable for H .

Original: G.2 Proof of Theorem
Revised: G.2 Proof of Theorem

Original: Theorem 4 (Impossibility Theorem for Total Space) .
Revised: Theorem 4 (Impossibility Theorem for Total Space) .

Original: OOD detection is not learnable in the total space D all XY for H , if | ϕ ◦ H| > 1 , where ϕ maps ID labels to 1 and maps OOD labels to 2 .
Revised: OOD detection is not learnable in the total space D all XY for H , if | ϕ ◦ H| > 1 , where ϕ maps ID labels to 1 and maps OOD labels to 2 .

Original: Proof of Theorem 4.
Revised: Proof of Theorem 4.

Original: We need to prove that OOD detection is not learnable in the total space D all XY for H , if H is non-trivial, i.e. , { x ∈ X : ∃ h 1 , h 2 ∈ H , s.t.
Revised: We need to prove that OOD detection is not learnable in the total space D all XY for H , if H is non-trivial, i.e. , { x ∈ X : ∃ h 1 , h 2 ∈ H , s.t.

Original: h 1 ( x ) ∈ Y , h 2 ( x ) =
Revised: h 1 ( x ) ∈ Y , h 2 ( x ) =

Original: K + 1 } ̸ = ∅ .
Revised: K + 1 } ̸ = ∅ .

Original: The main idea is to construct a domain D XY satisfying that:
Revised: The main idea is to construct a domain D XY satisfying that:

Original: 1) the ID and OOD distributions have overlap (Definition 4); and
Revised: 1) the ID and OOD distributions have overlap (Definition 4); and

Original: ) R
Revised: ) R

Original: in
Revised: in

Original: D ( h 1 ) = 0 , R out D ( h 2 ) =
Revised: D ( h 1 ) = 0 , R out D ( h 2 ) =

Original: According to the condition that H is non-trivial, we know that there exist h 1 , h 2 ∈ H such that
Revised: According to the condition that H is non-trivial, we know that there exist h 1 , h 2 ∈ H such that

Original: h 1 ( x 1 ) ∈ Y , h 2 ( x 1 ) =
Revised: h 1 ( x 1 ) ∈ Y , h 2 ( x 1 ) =

Original: K +1 , for some x 1 ∈ X .
Revised: K +1 , for some x 1 ∈ X .

Original: We set D XY
Revised: We set D XY

Original: = 0 .
Revised: = 0 .

Original: 5 ∗ δ ( x 1 ,h 1 ( x 1 ))
Revised: 5 ∗ δ ( x 1 ,h 1 ( x 1 ))

Original: +0 .
Revised: +0 .

Original: 5 ∗ δ ( x 1 ,h 2 ( x 1 )) , where δ is the Dirac measure.
Revised: 5 ∗ δ ( x 1 ,h 2 ( x 1 )) , where δ is the Dirac measure.

Original: It is easy to check that R in D ( h 1 ) = 0 , R out D ( h 2 ) = 0 , which implies that inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 .
Revised: It is easy to check that R in D ( h 1 ) = 0 , R out D ( h 2 ) = 0 , which implies that inf h ∈H R in D ( h ) = 0 and inf h ∈H R out D ( h ) = 0 .

Original: In addition, the ID distribution δ ( x 1 ,h 1 ( x 1 )) and OOD distribution δ ( x 1 ,h 2 ( x 1 )) have overlap x 1 .
Revised: In addition, the ID distribution δ ( x 1 ,h 1 ( x 1 )) and OOD distribution δ ( x 1 ,h 2 ( x 1 )) have overlap x 1 .

Original: By using Theorem 3, we have completed this proof.
Revised: By using Theorem 3, we have completed this proof.

Original: H Proof of Theorem
Revised: H Proof of Theorem

Original: Before proving Theorem 5, we need three important lemmas.
Revised: Before proving Theorem 5, we need three important lemmas.

Original: Lemma 3.
Revised: Lemma 3.

Original: Suppose that D XY is a domain with OOD convex decomposition Q 1 , ..., Q l (convex decomposition is given by Definition 6 in Appendix F), and D XY is a finite discrete distribution, then (the definition of f D,Q is given in Condition 4)
Revised: Suppose that D XY is a domain with OOD convex decomposition Q 1 , ..., Q l (convex decomposition is given by Definition 6 in Appendix F), and D XY is a finite discrete distribution, then (the definition of f D,Q is given in Condition 4)

Original: [Equation]
Revised: [Equation]

Original: if and only if
Revised: if and only if

Original: [Equation]
Revised: [Equation]

Original: where 0 is the 1 × l vector, whose elements are
Revised: where 0 is the 1 × l vector, whose elements are

Original: [Equation]
Revised: [Equation]

Original: and α j is the 1 × l vector, whose j -th element isand other elements are 0 , and
Revised: and α j is the 1 × l vector, whose j -th element isand other elements are 0 , and

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 3.
Revised: Proof of Lemma 3.

Original: To better understand this proof
Revised: To better understand this proof

Original: [Equation]
Revised: [Equation]

Original: we recall the definition of f D,Q ( α 1 , ..., α l ) :
Revised: we recall the definition of f D,Q ( α 1 , ..., α l ) :

Original: [Equation]
Revised: [Equation]

Original: First , we prove that if
Revised: First , we prove that if

Original: [Equation]
Revised: [Equation]

Original: then,
Revised: then,

Original: [Equation]
Revised: [Equation]

Original: Let D XY
Revised: Let D XY

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: I Y
Revised: I Y

Original: I + (cid:80) lj
Revised: I + (cid:80) lj

Original: =

Revised: =

Original: λ j Q j , for some ( λ 1 , ..., λ l ) ∈ ∆ o l .
Revised: λ j Q j , for some ( λ 1 , ..., λ l ) ∈ ∆ o l .

Original: Since D XY has finite support set, we have
Revised: Since D XY has finite support set, we have

Original: [Equation]
Revised: [Equation]

Original: We can find that h 0 ∈ arg min h ∈H (cid:16) (1 − (cid:80) lj
Revised: We can find that h 0 ∈ arg min h ∈H (cid:16) (1 − (cid:80) lj

Original: =1 λ j ) R in D ( h ) + (cid:80) lj
Revised: =1 λ j ) R in D ( h ) + (cid:80) lj

Original: =

Revised: =

Original: λ j R Q j ( h ) (cid:17) .
Revised: λ j R Q j ( h ) (cid:17) .

Original: Hence, (1 −
Revised: Hence, (1 −

Original: [Equation]
Revised: [Equation]

Original: Note that the condition f D,Q ( α
Revised: Note that the condition f D,Q ( α

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 − (cid:80)
Revised: 1 − (cid:80)

Original: [Equation]
Revised: [Equation]

Original: Q (
Revised: Q (

Original: [Equation]
Revised: [Equation]

Original: ) implies (1 −
Revised: ) implies (1 −

Original: [Equation]
Revised: [Equation]

Original: Therefore, Eq.
Revised: Therefore, Eq.

Original: (28) and Eq.
Revised: (28) and Eq.

Original: (29) imply that
Revised: (29) imply that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since R in D ( h 0 ) ≥ inf h ∈H R in D ( h ) and R Q j ( h 0 ) ≥ inf h ∈H R in Q j ( h ) , for j = 1 , ..., l , then using Eq.
Revised: Since R in D ( h 0 ) ≥ inf h ∈H R in D ( h ) and R Q j ( h 0 ) ≥ inf h ∈H R in Q j ( h ) , for j = 1 , ..., l , then using Eq.

Original: (30), we have that
Revised: (30), we have that

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Additionally, using
Revised: Additionally, using

Original: [Equation]
Revised: [Equation]

Original: we obtain that for any h ′ ∈ (cid:84) lj =1 arg min h ∈H R Q j ( h ) (cid:84) arg min h ∈H R in D ( h ) , inf h ∈H R D ( h ) =
Revised: we obtain that for any h ′ ∈ (cid:84) lj =1 arg min h ∈H R Q j ( h ) (cid:84) arg min h ∈H R in D ( h ) , inf h ∈H R D ( h ) =

Original: inf h ∈H (cid:16) (1 − l (cid:88)
Revised: inf h ∈H (cid:16) (1 − l (cid:88)

Original: j = λ j ) R in D ( h ) + l (cid:88)
Revised: j = λ j ) R in D ( h ) + l (cid:88)

Original: j = λ j R Q j ( h ) (cid:17) =(1 − l (cid:88)
Revised: j = λ j R Q j ( h ) (cid:17) =(1 − l (cid:88)

Original: j = λ j ) inf h ∈H R in D ( h ) + l (cid:88)
Revised: j = λ j ) inf h ∈H R in D ( h ) + l (cid:88)

Original: j = λ j inf h ∈H R Q j ( h ) =(1 − l (cid:88)
Revised: j = λ j inf h ∈H R Q j ( h ) =(1 − l (cid:88)

Original: j = λ j ) R in D ( h ′ )
Revised: j = λ j ) R in D ( h ′ )

Original: + l (cid:88)
Revised: + l (cid:88)

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (31) with Eq.
Revised: (31) with Eq.

Original: (32), we obtain that
Revised: (32), we obtain that

Original: [Equation]
Revised: [Equation]

Original: Second , we prove that if
Revised: Second , we prove that if

Original: [Equation]
Revised: [Equation]

Original: then,
Revised: then,

Original: [Equation]
Revised: [Equation]

Original: We set h
Revised: We set h

Original: [Equation]
Revised: [Equation]

Original: then, for any ( α 1 , ..., α l ) ∈ ∆ o l , (1 − l
Revised: then, for any ( α 1 , ..., α l ) ∈ ∆ o l , (1 − l

Original: [Equation]
Revised: [Equation]

Original: ( h ) .
Revised: ( h ) .

Original: Therefore, for any ( α 1 , ..., α l ) ∈ ∆ o l ,
Revised: Therefore, for any ( α 1 , ..., α l ) ∈ ∆ o l ,

Original: [Equation]
Revised: [Equation]

Original: 1 − l (cid:88)
Revised: 1 − l (cid:88)

Original: [Equation]
Revised: [Equation]

Original: which implies that: for any ( α 1 , ..., α l ) ∈ ∆ o l ,
Revised: which implies that: for any ( α 1 , ..., α l ) ∈ ∆ o l ,

Original: [Equation]
Revised: [Equation]

Original: We have completed this proof.
Revised: We have completed this proof.

Original: Lemma 4.
Revised: Lemma 4.

Original: Suppose that Assumption 1 holds.
Revised: Suppose that Assumption 1 holds.

Original: If there is a finite discrete domain D XY ∈ D sXY such that inf h ∈H R out D ( h ) > 0 , then OOD detection is not learnable in D sXY for H .
Revised: If there is a finite discrete domain D XY ∈ D sXY such that inf h ∈H R out D ( h ) > 0 , then OOD detection is not learnable in D sXY for H .

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: Suppose that supp D X O = { x out1 , ..., x out l } , then it is clear that D XY has OOD convex decomposition δ x out1 , ..., δ x out l , where δ x is the dirac measure whose support set is { x } .
Revised: Suppose that supp D X O = { x out1 , ..., x out l } , then it is clear that D XY has OOD convex decomposition δ x out1 , ..., δ x out l , where δ x is the dirac measure whose support set is { x } .

Original: Since H is the separate space for OOD ( i.e. , Assumption 1 holds), then ∀ j = 1 , ..., l , inf h ∈H R δ x out j ( h ) = 0 , where
Revised: Since H is the separate space for OOD ( i.e. , Assumption 1 holds), then ∀ j = 1 , ..., l , inf h ∈H R δ x out j ( h ) = 0 , where

Original: [Equation]
Revised: [Equation]

Original: This implies that: if (cid:84) lj =1 arg min h ∈H R δ x out j ( h ) ̸ = ∅ , then for ∀ h ′ ∈ (cid:84) lj
Revised: This implies that: if (cid:84) lj =1 arg min h ∈H R δ x out j ( h ) ̸ = ∅ , then for ∀ h ′ ∈ (cid:84) lj

Original: =1 arg min h ∈H R δ x out j ( h ) , h ′ ( x out i )
Revised: =1 arg min h ∈H R δ x out j ( h ) , h ′ ( x out i )

Original: =
Revised: =

Original: K + 1 , ∀ i = 1 , ..., l.
Revised: K + 1 , ∀ i = 1 , ..., l.

Original: Therefore, if (cid:84) lj =1 arg min h ∈H R
Revised: Therefore, if (cid:84) lj =1 arg min h ∈H R

Original: δ x out j ( h ) (cid:84) arg min h ∈H R in D ( h ) ̸
Revised: δ x out j ( h ) (cid:84) arg min h ∈H R in D ( h ) ̸

Original: = ∅ , then for any h ∗ ∈ (cid:84) l j =1 arg min h ∈H R δ x out j ( h ) (cid:84) arg min h ∈H R in D ( h ) , we have that h ∗ ( x out i ) =
Revised: = ∅ , then for any h ∗ ∈ (cid:84) l j =1 arg min h ∈H R δ x out j ( h ) (cid:84) arg min h ∈H R in D ( h ) , we have that h ∗ ( x out i ) =

Original: K + 1 , ∀ i = 1 , ..., l.
Revised: K + 1 , ∀ i = 1 , ..., l.

Original: Proof by Contradiction : assume OOD detection is learnable in D sXY for H , then Lemmas 1 andimply that
Revised: Proof by Contradiction : assume OOD detection is learnable in D sXY for H , then Lemmas 1 andimply that

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any h ∗ ∈ arg min h ∈H R D ( h ) , we have that
Revised: Therefore, for any h ∗ ∈ arg min h ∈H R D ( h ) , we have that

Original: [Equation]
Revised: [Equation]

Original: which implies that for any h ∗ ∈ arg min h ∈H R D ( h ) , we have R out D ( h ∗ )
Revised: which implies that for any h ∗ ∈ arg min h ∈H R D ( h ) , we have R out D ( h ∗ )

Original: = 0 , which implies that inf h ∈H R out D ( h ) = 0 .
Revised: = 0 , which implies that inf h ∈H R out D ( h ) = 0 .

Original: It is clear that inf h ∈H R out D ( h ) = 0 is inconsistent with the condition inf h ∈H R out D ( h ) > 0 .
Revised: It is clear that inf h ∈H R out D ( h ) = 0 is inconsistent with the condition inf h ∈H R out D ( h ) > 0 .

Original: Therefore, OOD detection is not learnable in D sXY for H .
Revised: Therefore, OOD detection is not learnable in D sXY for H .

Original: Lemma 5.
Revised: Lemma 5.

Original: If Assumption 1 holds, VCdim( ϕ ◦ H )
Revised: If Assumption 1 holds, VCdim( ϕ ◦ H )

Original: =
Revised: =

Original: v < + ∞ and sup h ∈H |{ x ∈ X : h ( x ) ∈
Revised: v < + ∞ and sup h ∈H |{ x ∈ X : h ( x ) ∈

Original: Y}| > m such that v < m , then OOD detection is not learnable in D sXY for H , where ϕ maps ID’s labels to 1 and maps OOD’s labels to 2 .
Revised: Y}| > m such that v < m , then OOD detection is not learnable in D sXY for H , where ϕ maps ID’s labels to 1 and maps OOD’s labels to 2 .

Original: Proof of Lemma 5.
Revised: Proof of Lemma 5.

Original: Due to sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| > m , we can obtain a set
Revised: Due to sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| > m , we can obtain a set

Original: C = { x 1 , ..., x m , x m +1 } , which satisfies that there exists
Revised: C = { x 1 , ..., x m , x m +1 } , which satisfies that there exists

Original: ˜ h ∈ H such that ˜ h ( x i ) ∈ Y for any i
Revised: ˜ h ∈ H such that ˜ h ( x i ) ∈ Y for any i

Original: = 1 , ..., m, m + 1 .
Revised: = 1 , ..., m, m + 1 .

Original: Let H ϕC = { ( ϕ ◦ h ( x 1 ) , ..., ϕ ◦ h ( x m ) , ϕ ◦ h
Revised: Let H ϕC = { ( ϕ ◦ h ( x 1 ) , ..., ϕ ◦ h ( x m ) , ϕ ◦ h

Original: ( x m +1 ) : h ∈ H} .
Revised: ( x m +1 ) : h ∈ H} .

Original: It is clear that
Revised: It is clear that

Original: [Equation]
Revised: [Equation]

Original: , where (1 , 1 , ..., 1) means all elements are 1 .
Revised: , where (1 , 1 , ..., 1) means all elements are 1 .

Original: Let H ϕ m +

Revised: Let H ϕ m +

Original: = { ( ϕ ◦ h ( x 1 ) , ..., ϕ ◦ h ( x m ) , ϕ ◦ h ( x m +1 ) : h is any hypothesis function from X to Y all } .
Revised: = { ( ϕ ◦ h ( x 1 ) , ..., ϕ ◦ h ( x m ) , ϕ ◦ h ( x m +1 ) : h is any hypothesis function from X to Y all } .

Original: Clearly, H ϕC ⊂ H
Revised: Clearly, H ϕC ⊂ H

Original: ϕm +1 and |H
Revised: ϕm +1 and |H

Original: ϕm +1 | = 2 m +1 .
Revised: ϕm +1 | = 2 m +1 .

Original: Sauer-Shelah-Perles Lemma (Lemma 6.10 in [21]) implies that
Revised: Sauer-Shelah-Perles Lemma (Lemma 6.10 in [21]) implies that

Original: [Equation]
Revised: [Equation]

Original: Since (cid:80) vi =0 (cid:0) m +1 i (cid:1) < 2 m +1 − 1 (because v < m ), we obtain that |H ϕC | ≤ 2 m +1 − 2 .
Revised: Since (cid:80) vi =0 (cid:0) m +1 i (cid:1) < 2 m +1 − 1 (because v < m ), we obtain that |H ϕC | ≤ 2 m +1 − 2 .

Original: Therefore,
Revised: Therefore,

Original: H ϕC ∪{ (2 , 2 ..., 2) } is a proper subset of H
Revised: H ϕC ∪{ (2 , 2 ..., 2) } is a proper subset of H

Original: ϕm +1 , where (2 , 2 , ..., 2) means that all elements are 2 .
Revised: ϕm +1 , where (2 , 2 , ..., 2) means that all elements are 2 .

Original: Note that (1 , 1 ..., 1) (all elements are 1) also belongs to H ϕC .
Revised: Note that (1 , 1 ..., 1) (all elements are 1) also belongs to H ϕC .

Original: Hence, H ϕC ∪ { (2 , 2 ..., 2) } ∪ { (1 , 1 ..., 1) } is a proper subset of H
Revised: Hence, H ϕC ∪ { (2 , 2 ..., 2) } ∪ { (1 , 1 ..., 1) } is a proper subset of H

Original: ϕm +1 , which implies that we can obtain a hypothesis function
Revised: ϕm +1 , which implies that we can obtain a hypothesis function

Original: h ′ satisfying that:
Revised: h ′ satisfying that:

Original: [Equation]
Revised: [Equation]

Original: ϕC
Revised: ϕC

Original: [Equation]
Revised: [Equation]

Original: There
Revised: There

Original: exist x j , x
Revised: exist x j , x

Original: p
Revised: p

Original: ∈ C such that ϕ ◦ h ′
Revised: ∈ C such that ϕ ◦ h ′

Original: ( x j )
Revised: ( x j )

Original: = 2 and ϕ ◦
Revised: = 2 and ϕ ◦

Original: h ′ ( x p ) =
Revised: h ′ ( x p ) =

Original: Let C I = C ∩ { x ∈ X : ϕ ◦ h ′ ( x ) = 1 } and C O = C ∩ { x ∈ X : ϕ ◦
Revised: Let C I = C ∩ { x ∈ X : ϕ ◦ h ′ ( x ) = 1 } and C O = C ∩ { x ∈ X : ϕ ◦

Original: h ′ ( x ) = 2 } ;
Revised: h ′ ( x ) = 2 } ;

Original: Then, we construct a special domain D XY :
Revised: Then, we construct a special domain D XY :

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: = 0 .
Revised: = 0 .

Original: 5 ∗ D X I ∗ D Y
Revised: 5 ∗ D X I ∗ D Y

Original: I |
Revised: I |

Original: X I
Revised: X I

Original: + 0 .
Revised: + 0 .

Original: 5 ∗ D X
Revised: 5 ∗ D X

Original: O ∗ D
Revised: O ∗ D

Original: Y O
Revised: Y O

Original: | X O , where
Revised: | X O , where

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: C I δ x and D Y I
Revised: C I δ x and D Y I

Original: |
Revised: |

Original: X I ( y | x )
Revised: X I ( y | x )

Original: = 1 , if ˜ h ( x ) =
Revised: = 1 , if ˜ h ( x ) =

Original: y and x ∈ C
Revised: y and x ∈ C

Original: I ; and
Revised: I ; and

Original: D X O =
Revised: D X O =

Original: | C O | (cid:88) x ∈ C
Revised: | C O | (cid:88) x ∈ C

Original: O δ x and D Y O
Revised: O δ x and D Y O

Original: | X O ( K + 1 | x ) = 1 , if x ∈ C O .
Revised: | X O ( K + 1 | x ) = 1 , if x ∈ C O .

Original: Since D XY is a finite discrete distribution and ( ϕ ◦ h ′ ( x 1 ) , ..., ϕ ◦ h ′ ( x m ) , ϕ ◦ h ′ (
Revised: Since D XY is a finite discrete distribution and ( ϕ ◦ h ′ ( x 1 ) , ..., ϕ ◦ h ′ ( x m ) , ϕ ◦ h ′ (

Original: x m +1 ))
Revised: x m +1 ))

Original: / ∈ H ϕC , it is clear that arg min h ∈H R D ( h ) ̸ = ∅ and inf h ∈H R D ( h ) > 0 .
Revised: / ∈ H ϕC , it is clear that arg min h ∈H R D ( h ) ̸ = ∅ and inf h ∈H R D ( h ) > 0 .

Original: Additionally, R in D (˜ h ) = 0 .
Revised: Additionally, R in D (˜ h ) = 0 .

Original: Therefore, inf h ∈H R in D ( h ) = 0 .
Revised: Therefore, inf h ∈H R in D ( h ) = 0 .

Original: Proof by Contradiction : suppose that OOD detection is learnable in D sXY for H , then Lemmaimplies that
Revised: Proof by Contradiction : suppose that OOD detection is learnable in D sXY for H , then Lemmaimplies that

Original: [Equation]
Revised: [Equation]

Original: Therefore, if OOD detection is learnable in D sXY for H , then inf h ∈H R out D ( h ) > 0 .
Revised: Therefore, if OOD detection is learnable in D sXY for H , then inf h ∈H R out D ( h ) > 0 .

Original: Until now, we have constructed a domain D XY (defined over X × Y all ) with finite support and satisfying that inf h ∈H R out D ( h ) > 0 .
Revised: Until now, we have constructed a domain D XY (defined over X × Y all ) with finite support and satisfying that inf h ∈H R out D ( h ) > 0 .

Original: Note that H is the separate space for OOD data (Assumption 1 holds).
Revised: Note that H is the separate space for OOD data (Assumption 1 holds).

Original: Using Lemma 4, we know that OOD detection is not learnable in D sXY for H , which is inconsistent with our assumption that OOD detection is learnable in D sXY for H .
Revised: Using Lemma 4, we know that OOD detection is not learnable in D sXY for H , which is inconsistent with our assumption that OOD detection is learnable in D sXY for H .

Original: Therefore, OOD detection is not learnable in D sXY for H .
Revised: Therefore, OOD detection is not learnable in D sXY for H .

Original: We have completed the proof.
Revised: We have completed the proof.

Original: Theorem 5 (Impossibility Theorem for Separate Space) .
Revised: Theorem 5 (Impossibility Theorem for Separate Space) .

Original: If Assumption 1 holds, VCdim( ϕ ◦ H ) <
Revised: If Assumption 1 holds, VCdim( ϕ ◦ H ) <

Original: + ∞ and sup h ∈H |{ x ∈ X : h ( x )
Revised: + ∞ and sup h ∈H |{ x ∈ X : h ( x )

Original: ∈ Y}| =
Revised: ∈ Y}| =

Original: + ∞ , then OOD detection is not learnable in separate space D sXY for H , where ϕ maps ID labels to 1 and maps OOD labels to 2 .
Revised: + ∞ , then OOD detection is not learnable in separate space D sXY for H , where ϕ maps ID labels to 1 and maps OOD labels to 2 .

Original: Proof of Theorem 5.
Revised: Proof of Theorem 5.

Original: Let VCdim( ϕ ◦ H )
Revised: Let VCdim( ϕ ◦ H )

Original: = v .
Revised: = v .

Original: Since sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| =
Revised: Since sup h ∈H |{ x ∈ X : h ( x ) ∈ Y}| =

Original: + ∞ , it is clear that sup h ∈H |{ x ∈ X : h ( x ) ∈
Revised: + ∞ , it is clear that sup h ∈H |{ x ∈ X : h ( x ) ∈

Original: Y}| > v .
Revised: Y}| > v .

Original: Using Lemma 5, we complete this proof.
Revised: Using Lemma 5, we complete this proof.

Original: I Proofs of Theorem 6 and TheoremI.1 Proof of Theorem
Revised: I Proofs of Theorem 6 and TheoremI.1 Proof of Theorem

Original: Firstly, we need two lemmas, which are motivated by Lemma 19.2 and Lemma 19.3 in [21].
Revised: Firstly, we need two lemmas, which are motivated by Lemma 19.2 and Lemma 19.3 in [21].

Original: Lemma 6.
Revised: Lemma 6.

Original: Let C 1 ,..., C r be a cover of space X , i.e., (cid:80) ri =1 C i
Revised: Let C 1 ,..., C r be a cover of space X , i.e., (cid:80) ri =1 C i

Original: =
Revised: =

Original: X .
Revised: X .

Original: Let S X
Revised: Let S X

Original: = { x 1 , ..., x n } be a sequence of n data drawn from D X I , i.i.d.
Revised: = { x 1 , ..., x n } be a sequence of n data drawn from D X I , i.i.d.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 6.
Revised: Proof of Lemma 6.

Original: [Equation]
Revised: [Equation]

Original: where 1 is the characteristic function.
Revised: where 1 is the characteristic function.

Original: For each i ,
Revised: For each i ,

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: here we have used inequality: max i ∈{ 1 ,...,r } a i e − na i ≤ 1 / ( ne ) .
Revised: here we have used inequality: max i ∈{ 1 ,...,r } a i e − na i ≤ 1 / ( ne ) .

Original: The proof has been completed.
Revised: The proof has been completed.

Original: Lemma 7.
Revised: Lemma 7.

Original: Let K
Revised: Let K

Original: = 1 .
Revised: = 1 .

Original: When X ⊂ R d is a bounded set, there exists a monotonically decreasing sequence ϵ cons ( m ) satisfying that ϵ cons ( m ) → 0 , as m → 0 , such that
Revised: When X ⊂ R d is a bounded set, there exists a monotonically decreasing sequence ϵ cons ( m ) satisfying that ϵ cons ( m ) → 0 , as m → 0 , such that

Original: [Equation]
Revised: [Equation]

Original: where dist is the Euclidean distance, π 1 ( x , S )
Revised: where dist is the Euclidean distance, π 1 ( x , S )

Original: = arg
Revised: = arg

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: ( x , ˜ x ) , here S X is the feature part of S , i.e., S X = { x 1 , ..., x n } , if S = { ( x 1 , y 1 ) , ..., ( x n , y n ) } .
Revised: ( x , ˜ x ) , here S X is the feature part of S , i.e., S X = { x 1 , ..., x n } , if S = { ( x 1 , y 1 ) , ..., ( x n , y n ) } .

Original: Proof of Lemma 7.
Revised: Proof of Lemma 7.

Original: Since X is bounded, without loss of generality, we set X ⊂ [0 , 1) d .
Revised: Since X is bounded, without loss of generality, we set X ⊂ [0 , 1) d .

Original: Fix ϵ
Revised: Fix ϵ

Original: = 1 /T , for some integer T .
Revised: = 1 /T , for some integer T .

Original: Let r = T d and C 1 , C 2 , ..., C r be a cover of X : for every ( a 1 , ...,
Revised: Let r = T d and C 1 , C 2 , ..., C r be a cover of X : for every ( a 1 , ...,

Original: a T ) ∈
Revised: a T ) ∈

Original: [ T ] d := [1 , ..., T ] d , there exists a C
Revised: [ T ] d := [1 , ..., T ] d , there exists a C

Original: i = { x = ( x 1 , ..., x d ) : ∀ j ∈ { 1 , ..., d } , x j ∈ [( a j − 1) /T, a j /T ) } .
Revised: i = { x = ( x 1 , ..., x d ) : ∀ j ∈ { 1 , ..., d } , x j ∈ [( a j − 1) /T, a j /T ) } .

Original: If x , x ′ belong to some C i , then dist( x , x ′ ) ≤ √ dϵ ; otherwise, dist( x , x ′ ) ≤ √ d .
Revised: If x , x ′ belong to some C i , then dist( x , x ′ ) ≤ √ dϵ ; otherwise, dist( x , x ′ ) ≤ √ d .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Note that C 1 , ..., C r are disjoint.
Revised: Note that C 1 , ..., C r are disjoint.

Original: Therefore, (cid:80) i : C i ∩ S X ̸
Revised: Therefore, (cid:80) i : C i ∩ S X ̸

Original: = ∅ D X
Revised: = ∅ D X

Original: I ( C i ) ≤ D X I ( (cid:80) i : C
Revised: I ( C i ) ≤ D X I ( (cid:80) i : C

Original: i ∩ S X ̸ = ∅ C i ) ≤ 1 .
Revised: i ∩ S X ̸ = ∅ C i ) ≤ 1 .

Original: Using Lemma 6, we obtain
Revised: Using Lemma 6, we obtain

Original: E x ∼ D X
Revised: E x ∼ D X

Original: I ,S ∼ D nX
Revised: I ,S ∼ D nX

Original: I Y I dist( x , π 1 ( x , S ))
Revised: I Y I dist( x , π 1 ( x , S ))

Original: ≤ √ dϵ + r √ d ne = √ dϵ + √ d neϵ d .
Revised: ≤ √ dϵ + r √ d ne = √ dϵ + √ d neϵ d .

Original: If we set ϵ = 2 n − 1 / ( d +1) , then
Revised: If we set ϵ = 2 n − 1 / ( d +1) , then

Original: E x ∼ D X
Revised: E x ∼ D X

Original: I ,S ∼ D nX
Revised: I ,S ∼ D nX

Original: I Y I dist( x , π 1 ( x , S ))
Revised: I Y I dist( x , π 1 ( x , S ))

Original: ≤ 2 √ d n 1 / ( d +1)
Revised: ≤ 2 √ d n 1 / ( d +1)

Original: + √ d 2 d en 1 / ( d +1) .
Revised: + √ d 2 d en 1 / ( d +1) .

Original: If we set ϵ cons ( n ) = 2 √ d n 1 / ( d +1) + √ d 2 d en 1 / ( d +1) , we complete this proof.
Revised: If we set ϵ cons ( n ) = 2 √ d n 1 / ( d +1) + √ d 2 d en 1 / ( d +1) , we complete this proof.

Original: Theorem 6.
Revised: Theorem 6.

Original: Let K = 1 and |X| < + ∞ .
Revised: Let K = 1 and |X| < + ∞ .

Original: Suppose that Assumption 1 holds and the constant function h in := 1 ∈ H .
Revised: Suppose that Assumption 1 holds and the constant function h in := 1 ∈ H .

Original: Then OOD detection is learnable in D sXY for H if and only if H all − { h out } ⊂ H , where H all is the hypothesis space consisting of all hypothesis functions, and h out is a constant function that h out := 2 , here 1 represents ID data and 2 represents OOD data.
Revised: Then OOD detection is learnable in D sXY for H if and only if H all − { h out } ⊂ H , where H all is the hypothesis space consisting of all hypothesis functions, and h out is a constant function that h out := 2 , here 1 represents ID data and 2 represents OOD data.

Original: Proof of Theorem 6.
Revised: Proof of Theorem 6.

Original: First , we prove that if the hypothesis space H is a separate space for OOD ( i.e. ,
Revised: First , we prove that if the hypothesis space H is a separate space for OOD ( i.e. ,

Original: Assumption 1 holds), the constant function h in := 1 ∈ H , then that OOD detection is learnable in D sXY for H implies H all − { h out } ⊂ H .
Revised: Assumption 1 holds), the constant function h in := 1 ∈ H , then that OOD detection is learnable in D sXY for H implies H all − { h out } ⊂ H .

Original: Proof by Contradiction : suppose that there exists h ′ ∈ H all such
Revised: Proof by Contradiction : suppose that there exists h ′ ∈ H all such

Original: that h ′ ̸
Revised: that h ′ ̸

Original: = h out and h ′ / ∈ H .
Revised: = h out and h ′ / ∈ H .

Original: Let X = { x 1 , ..., x m } , C I = { x ∈ X : h ′ ( x ) ∈ Y} and C
Revised: Let X = { x 1 , ..., x m } , C I = { x ∈ X : h ′ ( x ) ∈ Y} and C

Original: O
Revised: O

Original: =
Revised: =

Original: { x ∈ X : h ′ ( x )
Revised: { x ∈ X : h ′ ( x )

Original: =
Revised: =

Original: K + 1 } .
Revised: K + 1 } .

Original: Because h ′ ̸ = h out , we know that C I ̸
Revised: Because h ′ ̸ = h out , we know that C I ̸

Original: = ∅ .
Revised: = ∅ .

Original: We construct a special domain D XY ∈ D sXY : if C O = ∅ , then D XY
Revised: We construct a special domain D XY ∈ D sXY : if C O = ∅ , then D XY

Original: =
Revised: =

Original: D X
Revised: D X

Original: I ∗ D Y I
Revised: I ∗ D Y I

Original: |
Revised: |

Original: X I ; otherwise,
Revised: X I ; otherwise,

Original: D XY
Revised: D XY

Original: = 0 .
Revised: = 0 .

Original: 5 ∗ D X I ∗ D Y
Revised: 5 ∗ D X I ∗ D Y

Original: I |
Revised: I |

Original: X I
Revised: X I

Original: + 0 .
Revised: + 0 .

Original: 5 ∗ D X
Revised: 5 ∗ D X

Original: O ∗ D
Revised: O ∗ D

Original: Y O
Revised: Y O

Original: | X O , where
Revised: | X O , where

Original: [Equation]
Revised: [Equation]

Original: | (cid:88) x
Revised: | (cid:88) x

Original: [Equation]
Revised: [Equation]

Original: C I δ x and D Y I
Revised: C I δ x and D Y I

Original: |
Revised: |

Original: X I ( y | x )
Revised: X I ( y | x )

Original: = 1 , if h ′ ( x ) =
Revised: = 1 , if h ′ ( x ) =

Original: y and x ∈ C I , and
Revised: y and x ∈ C I , and

Original: D X O =
Revised: D X O =

Original: | C O | (cid:88) x ∈ C
Revised: | C O | (cid:88) x ∈ C

Original: O δ x and D Y O
Revised: O δ x and D Y O

Original: | X O ( K + 1 | x ) = 1 , if x ∈ C O .
Revised: | X O ( K + 1 | x ) = 1 , if x ∈ C O .

Original: Since h ′ / ∈ H and |X| < + ∞ , then arg min h ∈H R D ( h ) ̸ = ∅ , and inf h ∈H R D ( h ) > 0 .
Revised: Since h ′ / ∈ H and |X| < + ∞ , then arg min h ∈H R D ( h ) ̸ = ∅ , and inf h ∈H R D ( h ) > 0 .

Original: Additionally,
Revised: Additionally,

Original: R in D ( h in )
Revised: R in D ( h in )

Original: = 

Revised: =

Original: (here h in = 1 ), hence, inf h ∈H R in D ( h ) = 0 .
Revised: (here h in = 1 ), hence, inf h ∈H R in D ( h ) = 0 .

Original: Since OOD detection is learnable in D sXY for H , Lemma 1 implies that
Revised: Since OOD detection is learnable in D sXY for H , Lemma 1 implies that

Original: [Equation]
Revised: [Equation]

Original: where π out = D Y ( Y = K + 1) = 1 or 0 .
Revised: where π out = D Y ( Y = K + 1) = 1 or 0 .

Original: Since inf h ∈H R in D ( h ) = 0 and inf h ∈H R D ( h ) > 0 , we obtain that inf h ∈H R out D ( h ) > 0 .
Revised: Since inf h ∈H R in D ( h ) = 0 and inf h ∈H R D ( h ) > 0 , we obtain that inf h ∈H R out D ( h ) > 0 .

Original: Until now, we have constructed a special domain D XY ∈ D sXY satisfying that inf h ∈H R out D ( h ) > 0 .
Revised: Until now, we have constructed a special domain D XY ∈ D sXY satisfying that inf h ∈H R out D ( h ) > 0 .

Original: Using Lemma 4, we know that OOD detection in D sXY is not learnable for H , which is inconsistent with the condition that OOD detection is learnable in D sXY for H .
Revised: Using Lemma 4, we know that OOD detection in D sXY is not learnable for H , which is inconsistent with the condition that OOD detection is learnable in D sXY for H .

Original: Therefore, the assumption (there exists h ′ ∈ H all such that h ′ ̸
Revised: Therefore, the assumption (there exists h ′ ∈ H all such that h ′ ̸

Original: = h out and h / ∈ H ) doesn’t hold, which implies that H all −{ h out } ⊂ H .
Revised: = h out and h / ∈ H ) doesn’t hold, which implies that H all −{ h out } ⊂ H .

Original: Second , we prove that if H all − { h out } ⊂ H , then OOD detection is learnable in D sXY for H .
Revised: Second , we prove that if H all − { h out } ⊂ H , then OOD detection is learnable in D sXY for H .

Original: To prove this result, we need to design a special algorithm.
Revised: To prove this result, we need to design a special algorithm.

Original: Let d 0 = min x , x ′ ∈X and x ̸ = x ′ dist( x , x ′ ) , where dist is the Euclidean distance.
Revised: Let d 0 = min x , x ′ ∈X and x ̸ = x ′ dist( x , x ′ ) , where dist is the Euclidean distance.

Original: It is clear that d 0 > 0 .
Revised: It is clear that d 0 > 0 .

Original: Let
Revised: Let

Original: A ( S )( x ) = (cid:26) 1 , if dist( x , π 1 ( x , S )) < 0 .
Revised: A ( S )( x ) = (cid:26) 1 , if dist( x , π 1 ( x , S )) < 0 .

Original: 5 ∗ d 0 ; 2 , if dist( x , π 1 ( x , S )) ≥ 0 .
Revised: 5 ∗ d 0 ; 2 , if dist( x , π 1 ( x , S )) ≥ 0 .

Original: 5 ∗ d 0 , where π 1 ( x , S ) =
Revised: 5 ∗ d 0 , where π 1 ( x , S ) =

Original: arg min ˜ x ∈ S X dist( x , ˜ x ) , here S X is the feature part of S , i.e. , S X
Revised: arg min ˜ x ∈ S X dist( x , ˜ x ) , here S X is the feature part of S , i.e. , S X

Original: = { x 1 , ..., x n } , if S = { ( x 1 , y 1 ) , ..., ( x n , y n ) } .
Revised: = { x 1 , ..., x n } , if S = { ( x 1 , y 1 ) , ..., ( x n , y n ) } .

Original: For any x ∈ supp D X I , it is easy to check that for almost all S ∼ D nX I Y
Revised: For any x ∈ supp D X I , it is easy to check that for almost all S ∼ D nX I Y

Original: I ,
Revised: I ,

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: hence,
Revised: hence,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using Lemma 7, for any x ∈ supp D
Revised: Using Lemma 7, for any x ∈ supp D

Original: X I , we have
Revised: X I , we have

Original: [Equation]
Revised: [Equation]

Original: where ϵ cons ( n ) → 0 , as n → 0 and ϵ cons ( n ) is a monotonically decreasing sequence.
Revised: where ϵ cons ( n ) → 0 , as n → 0 and ϵ cons ( n ) is a monotonically decreasing sequence.

Original: Hence, we have that
Revised: Hence, we have that

Original: [Equation]
Revised: [Equation]

Original: where D X I × D nX
Revised: where D X I × D nX

Original: I Y
Revised: I Y

Original: I is the product measure of D X I and D nX I Y
Revised: I is the product measure of D X I and D nX I Y

Original: I [36].
Revised: I [36].

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: where B = max { ℓ (1 , 2) , ℓ (2 , 1) } .
Revised: where B = max { ℓ (1 , 2) , ℓ (2 , 1) } .

Original: Using Eq.
Revised: Using Eq.

Original: (33) and Eq.
Revised: (33) and Eq.

Original: (34), we have proved that
Revised: (34), we have proved that

Original: [Equation]
Revised: [Equation]

Original: It is easy to check that A ( S ) ∈ H all − { h out } .
Revised: It is easy to check that A ( S ) ∈ H all − { h out } .

Original: Therefore, we have constructed a consistent algorithm
Revised: Therefore, we have constructed a consistent algorithm

Original: A for H .
Revised: A for H .

Original: We have completed this proof.
Revised: We have completed this proof.

Original: I.2 Proof of Theorem
Revised: I.2 Proof of Theorem

Original: Theorem 7.
Revised: Theorem 7.

Original: Let |X| < + ∞ and H
Revised: Let |X| < + ∞ and H

Original: = H in • H b .
Revised: = H in • H b .

Original: If H all − { h out } ⊂ H b and Condition 2 holds, then OOD detection is learnable in D sXY for H , where H all and h out are defined in Theorem 6.
Revised: If H all − { h out } ⊂ H b and Condition 2 holds, then OOD detection is learnable in D sXY for H , where H all and h out are defined in Theorem 6.

Original: Proof of Theorem 7.
Revised: Proof of Theorem 7.

Original: Since |X| < + ∞ , we know that |H| < + ∞ , which implies that H in is agnostic PAC learnable for supervised learning in classification.
Revised: Since |X| < + ∞ , we know that |H| < + ∞ , which implies that H in is agnostic PAC learnable for supervised learning in classification.

Original: Therefore, there exist an algorithm
Revised: Therefore, there exist an algorithm

Original: A in : ∪ + ∞ n =1 ( X × Y ) n → H in and a monotonically decreasing sequence ϵ ( n ) , such that ϵ ( n ) → 0 , as n → + ∞ , and for any D XY ∈ D sXY ,
Revised: A in : ∪ + ∞ n =1 ( X × Y ) n → H in and a monotonically decreasing sequence ϵ ( n ) , such that ϵ ( n ) → 0 , as n → + ∞ , and for any D XY ∈ D sXY ,

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: A in ( S ))
Revised: A in ( S ))

Original: [Equation]
Revised: [Equation]

Original: Since |X| < + ∞ and H b almost contains all binary classifiers, then using Theorem 6 and Theorem 1, we obtain that there exist an algorithm A b : ∪ + ∞ n =1 ( X × { 1 , 2 } ) n → H b and a monotonically decreasing sequence ϵ ′ ( n ) , such that ϵ ′ ( n ) → 0 , as n → + ∞ , and for any D XY ∈ D sXY ,
Revised: Since |X| < + ∞ and H b almost contains all binary classifiers, then using Theorem 6 and Theorem 1, we obtain that there exist an algorithm A b : ∪ + ∞ n =1 ( X × { 1 , 2 } ) n → H b and a monotonically decreasing sequence ϵ ′ ( n ) , such that ϵ ′ ( n ) → 0 , as n → + ∞ , and for any D XY ∈ D sXY ,

Original: [Equation]
Revised: [Equation]

Original: where ϕ maps ID’s labels to 1 and OOD’s label to 2 ,
Revised: where ϕ maps ID’s labels to 1 and OOD’s label to 2 ,

Original: [Equation]
Revised: [Equation]

Original: ) and
Revised: ) and

Original: [Equation]
Revised: [Equation]

Original: 39) here ϕ ( S ) =
Revised: 39) here ϕ ( S ) =

Original: { ( x 1 , ϕ
Revised: { ( x 1 , ϕ

Original: ( y 1 )) , ..., ( x n , ϕ ( y n )) } , if S = { ( x
Revised: ( y 1 )) , ..., ( x n , ϕ ( y n )) } , if S = { ( x

Original: [Equation]
Revised: [Equation]

Original: , y
Revised: , y