Original: T HE D EVIL IS IN THE W RONGLY - CLASSIFIED S AMPLES : T OWARDS U NIFIED O PEN - SET R ECOGNITION
Revised: T HE D EVIL IS IN THE W RONGLY - CLASSIFIED S AMPLES : T OWARDS U NIFIED O PEN - SET R ECOGNITION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Open-set Recognition (OSR) aims to identify test samples whose classes are not seen during the training process.
Revised: Open-set Recognition (OSR) aims to identify test samples whose classes are not seen during the training process.

Original: Recently, Unified Open-set Recognition (UOSR) has been proposed to reject not only unknown samples but also known but wrongly classified samples, which tends to be more practical in real-world applications.
Revised: Recently, Unified Open-set Recognition (UOSR) has been proposed to reject not only unknown samples but also known but wrongly classified samples, which tends to be more practical in real-world applications.

Original: In this paper, we deeply analyze the UOSR task under different training and evaluation settings to shed light on this promising research direction.
Revised: In this paper, we deeply analyze the UOSR task under different training and evaluation settings to shed light on this promising research direction.

Original: For this purpose, we first evaluate the UOSR performance of several OSR methods and show a significant finding that theuncertainty distribution of almost all these methods is actually closer to the expectation of UOSR than OSR.
Revised: For this purpose, we first evaluate the UOSR performance of several OSR methods and show a significant finding that the UOSR performance consistently surpasses the OSR performance by a large margin for the same method.

Original: We show that the reason lies in the known but wrongly classified samples, as their uncertainty distribution is extremely close to unknown samples rather than known and correctly classified samples.
Revised: We show that the reason lies in the known but wrongly classified samples, as their uncertainty distribution is extremely close to unknown samples rather than known and correctly classified samples.

Original: Second, we analyze how the two training settings of OSR ( i.e. , pre-training and outlier exposure) influence the UOSR.
Revised: Second, we analyze how the two training settings of OSR ( i.e. , pre-training and outlier exposure) influence the UOSR.

Original: We find although they are both beneficial for distinguishing known and correctly classified samples from unknown samples, pre-training is also helpful for identifying known but wrongly classified samples while outlier exposure is not.
Revised: We find although they are both beneficial for distinguishing known and correctly classified samples from unknown samples, pre-training is also helpful for identifying known but wrongly classified samples while outlier exposure is not.

Original: In addition to different training settings , we also formulate a new evaluation setting for UOSR which is called few-shot UOSR, where only one or five samples per unknown class are available during evaluation to help identify unknown samples.
Revised: In addition to different training settings , we also formulate a new evaluation setting for UOSR which is called few-shot UOSR, where only one or five samples per unknown class are available during evaluation to help identify unknown samples.

Original: We propose FS-KNNS for the few-shot UOSR to achieve state-of-the-art performance under all settings.
Revised: We propose FS-KNNS for the few-shot UOSR to achieve state-of-the-art performance under all settings.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks have achieved tremendous success in the closed-set classification (Deng et al., 2009), where the test samples share the same In-Distribution (InD) class set with training samples.
Revised: Neural networks have achieved tremendous success in the closed-set classification (Deng et al., 2009), where the test samples share the same In-Distribution (InD) class set with training samples.

Original: Open-Set Recognition (OSR) (Scheirer et al., 2013) is proposed to tackle the challenge that some samples whose classes are not seen during training, which are Out-of-Distribution (OoD) data, may occur in the real world applications and should be rejected.
Revised: Open-Set Recognition (OSR) (Scheirer et al., 2013) is proposed to tackle the challenge that some samples whose classes are not seen during training, which are Out-of-Distribution (OoD) data, may occur in the real world applications and should be rejected.

Original: However, some researchers have argued that the model should not only reject OoD samples but also InD samples that are Wrongly classified (InW), as the model gives the wrong answers for both of them.
Revised: However, some researchers have argued that the model should not only reject OoD samples but also InD samples that are Wrongly classified (InW), as the model gives the wrong answers for both of them.

Original: So Unified Open-set Recognition (UOSR) is proposed to only accept InD samples that are correctly classified (InC) and reject OoD and InW samples (Kim et al., 2021) simultaneously.
Revised: So Unified Open-set Recognition (UOSR) is proposed to only accept InD samples that are correctly classified (InC) and reject OoD and InW samples (Kim et al., 2021) simultaneously.

Original: The difference between the UOSR and OSR lies in the InW samples, where OSR is supposed to accept them while UOSR has the opposite purpose.
Revised: The difference between the UOSR and OSR lies in the InW samples, where OSR is supposed to accept them while UOSR has the opposite purpose.

Original: Actually, UOSR is more useful in most real-world applications, but it receives little attention from the research community as it has been proposed very recently and lacks comprehensive systematic research.
Revised: Actually, UOSR is more useful in most real-world applications, but it receives little attention from the research community as it has been proposed very recently and lacks comprehensive systematic research.

Original: Therefore, we deeply analyze the UOSR problem in this work to fill this gap.
Revised: Therefore, we deeply analyze the UOSR problem in this work to fill this gap.

Original: We first apply existing OSR methods for UOSR in Sec. 3, and then analyze UOSR under different training settings and evaluation settings in Sec. 4 and Sec. 5 respectively.
Revised: We first apply existing OSR methods for UOSR in Sec. 3, and then analyze UOSR under different training settings and evaluation settings in Sec. 4 and Sec. 5 respectively.

Original: In Sec. 3, several existing
Revised: In Sec. 3, several existing

Original: OSR methods are applied for UOSR, and we demonstrate a significant finding that theuncertainty distribution of almost all existing methods designed for OSR is actually closer to the expectation of UOSR  than  OSR .
Revised: OSR methods are applied for UOSR, and we find that the UOSR performance is consistently and significantly better than the OSR performance for the same method, as shown in Fig.

Original: In other words, simply changing the uncertainty ground truth of InW samples from 0 (OSR) to 1 (UOSR) can bring a very large performance gain, as shown in Fig.

Original: 1 (a).
Revised: 1 (a).

Original: We show that this phenomenon holds for different network architectures, datasets, and domains (image and video recognition).
Revised: We show that this phenomenon holds for different network architectures, datasets, and domains (image and video recognition).

Original: We find the devil is in the InW samples that have similar uncertainty distribution with OoD samples rather than InC samples.
Revised: We find the devil is in the InW samples that have similar uncertainty distribution with OoD samples rather than InC samples .

Revised: Therefore, the false positive predictions in OSR tend to be InW samples, which is extremely important but dismissed by all existing OSR works.

Original: TSTPTS w/ OE TP w/ OE
Revised: TSTPTS w/ OE TP w/ OE

Original: TP w/ FS
Revised: TP w/ FS

Original: (Ours) (a) (b) (Sec. 3)
Revised: (Ours) (a) (b) (Sec. 3)

Original: (Sec. 4) (Sec. 5)
Revised: (Sec. 4) (Sec. 5)

Original: In Sec. 4, we introduce two training settings into UOSR, including pre-training (Hendrycks et al., 2019a) and outlier exposure (Hendrycks et al., 2019b; Yu & Aizawa, 2019; Thulasidasan et al., 2021), as they are both helpers of the OSR that introduce extra information beyond the training set.
Revised: In Sec. 4, we introduce two training settings into UOSR, including pre-training (Hendrycks et al., 2019a) and outlier exposure (Hendrycks et al., 2019b; Yu & Aizawa, 2019; Thulasidasan et al., 2021), as they are both helpers of the OSR that introduce extra information beyond the training set.

Original: Pre-training is to use the weights that are trained on a large-scale dataset for better down-task performance, and outlier exposure is to introduce some background data without labels into training to help the model classify InD and OoD samples.
Revised: Pre-training is to use the weights that are trained on a large-scale dataset for better down-task performance, and outlier exposure is to introduce some background data without labels into training to help the model classify InD and OoD samples.

Original: We find both of them have better performance for InC/OoD discrimination, which explains why they are beneficial for OSR.
Revised: We find both of them have better performance for InC/OoD discrimination, which explains why they are beneficial for OSR.

Original: However, pre-training is also helpful for InC/InW discrimination, while outlier exposure has a comparable or even worse performance to distinguish InC and InW samples.
Revised: However, pre-training is also helpful for InC/InW discrimination, while outlier exposure has a comparable or even worse performance to distinguish InC and InW samples.

Original: The performance of UOSR can be regarded as the comprehensive results of InC/OoD and InC/InW discrimination so that both techniques can boost the performance of UOSR.
Revised: The performance of UOSR can be regarded as the comprehensive results of InC/OoD and InC/InW discrimination so that both techniques can boost the performance of UOSR.

Original: We build up a comprehensive UOSR benchmark that involves both pretraining and outlier exposure settings, as shown in Fig.
Revised: We build up a comprehensive UOSR benchmark that involves both pretraining and outlier exposure settings, as shown in Fig.

Original: 1 (b).
Revised: 1 (b).

Original: In addition to the two aforementioned training settings , we introduce a new evaluation setting into
Revised: In addition to the two aforementioned training settings , we introduce a new evaluation setting into

Original: UOSR in Sec.
Revised: UOSR in Sec.

Original: We formulate the few-shot UOSR, similar to SSD (Sehwag et al., 2022) that proposes few-shot OSR, where 1 or 5 samples per OoD class are introduced for reference to better identify OoD samples.
Revised: We formulate the few-shot UOSR, similar to SSD (Sehwag et al., 2022) that proposes few-shot OSR, where 1 or 5 samples per OoD class are introduced for reference to better identify OoD samples.

Original: We first develop a KNN-based baseline (Sun et al., 2022) FS-KNN for the fewshot UOSR.
Revised: We first develop a KNN-based baseline (Sun et al., 2022) FS-KNN for the fewshot UOSR.

Original: Although InC/OoD discrimination is improved due to the introduced OoD reference samples, the InC/InW discrimination is severely harmed compared to SoftMax baseline (Hendrycks & Gimpel, 2017).
Revised: Although InC/OoD discrimination is improved due to the introduced OoD reference samples, the InC/InW discrimination is severely harmed compared to SoftMax baseline (Hendrycks & Gimpel, 2017).

Original: To alleviate this problem, we propose FS-KNNS that dynamically fuses the
Revised: To alleviate this problem, we propose FS-KNNS that dynamically fuses the

Original: FS-KNN with SoftMax uncertainty scores to keep high InC/InW and InC/OoD performance simultaneously.
Revised: FS-KNN with SoftMax uncertainty scores to keep high InC/InW and InC/OoD performance simultaneously.

Original: Our FS-KNNS achieves state-of-the-art performance under all settings in the UOSR benchmark, as shown in Fig.
Revised: Our FS-KNNS achieves state-of-the-art performance under all settings in the UOSR benchmark, as shown in Fig.

Original: 1 (b), even without outlier exposure during training.
Revised: 1 (b), even without outlier exposure during training.

Original: Note that InC/OoD performances are comparable between FS-KNNS and FS-KNN, but their distinct InC/InW performances makes FS-KNN better at OSR and FS-KNNS better at UOSR, which illustrates the difference between few-shot OSR and UOSR and the importance of InW samples during evaluation.
Revised: Note that InC/OoD performances are comparable between FS-KNNS and FS-KNN, but their distinct InC/InW performances makes FS-KNN better at OSR and FS-KNNS better at UOSR, which illustrates the difference between few-shot OSR and UOSR and the importance of InW samples during evaluation.

Original:  T OWARDS U NIFIED O PEN - SET R ECOGNITION
Revised:  T OWARDS U NIFIED O PEN - SET R ECOGNITION

Original: In this section, we first formalize the UOSR problem and then discuss the relation between UOSR and other uncertainty-related tasks, as the core of UOSR is also how to estimate better uncertainty scores to reject InW and OoD samples.
Revised: In this section, we first formalize the UOSR problem and then discuss the relation between UOSR and other uncertainty-related tasks, as the core of UOSR is also how to estimate better uncertainty scores to reject InW and OoD samples.

Original: Unified Open-set Recognition.
Revised: Unified Open-set Recognition.

Original: Suppose the training dataset is D train
Revised: Suppose the training dataset is D train

Original: = { ( x i , y i ) } Ni =1 ⊂ X × C , where X refers to the input space, e.g. , images or videos, and C refers to the InD sets.
Revised: = { ( x i , y i ) } Ni =1 ⊂ X × C , where X refers to the input space, e.g. , images or videos, and C refers to the InD sets.

Original: In closed-set recognition, all test samples come from the InD sets, i.e. , D closedtest =
Revised: In closed-set recognition, all test samples come from the InD sets, i.e. , D closedtest =

Original: { ( x i , y i ) } Mi =1 ⊂ X × C .
Revised: { ( x i , y i ) } Mi =1 ⊂ X × C .

Original: In OSR and UOSR, the test samples may come from OoD sets U which are not overlap with InD sets C , so we have D opentest = D closedtest ∪D unknowntest , where D unknowntest = { ( x i , y i ) } M ′ i =1 ⊂ X ×U .
Revised: In OSR and UOSR, the test samples may come from OoD sets U which are not overlap with InD sets C , so we have D opentest = D closedtest ∪D unknowntest , where D unknowntest = { ( x i , y i ) } M ′ i =1 ⊂ X ×U .

Original: The InD test samples
Revised: The InD test samples

Original: D closedtest can be divided into two splits based on whether the sample is correctly classified or wrongly classified, i.e. , D closedtest =
Revised: D closedtest can be divided into two splits based on whether the sample is correctly classified or wrongly classified, i.e. , D closedtest =

Original: D closed-ctest ∪ D closed-wtest , where D closed-ctest = { ( x i , y i ) | ˆ
Revised: D closed-ctest ∪ D closed-wtest , where D closed-ctest = { ( x i , y i ) | ˆ

Original: y i
Revised: y i

Original: =
Revised: =

Original: y i } Mi =1 , D closed-wtest = { ( x i , y i )
Revised: y i } Mi =1 , D closed-wtest = { ( x i , y i )

Original: | ˆ
Revised: | ˆ

Original: y i ̸
Revised: y i ̸

Original: = y i } Mi =1 , and ˆ y i refers to the model classification results of sample x i .
Revised: = y i } Mi =1 , and ˆ y i refers to the model classification results of sample x i .

Original: The goal of
Revised: The goal of

Original: UOSR is to reject InW and OoD samples and accept InC samples, so the ground truth uncertainty u of D closed-wtest and D unknowntest is 1 while for D closed-ctest it is 0, as shown in Table 1.
Revised: UOSR is to reject InW and OoD samples and accept InC samples, so the ground truth uncertainty u of D closed-wtest and D unknowntest is 1 while for D closed-ctest it is 0, as shown in Table 1.

Original: The key of UOSR is how to estimate the uncertainty ˆ u to be close to the ground truth uncertainty u .
Revised: The key of UOSR is how to estimate the uncertainty ˆ u to be close to the ground truth uncertainty u .

Original: [Table]
Revised: [Table]

Original: The UOSR is proposed by (Kim et al., 2021) very recently, so it has not attracted many researchers to this problem yet.
Revised: The UOSR is proposed by (Kim et al., 2021) very recently, so it has not attracted many researchers to this problem yet.

Original: SIRC (Xia & Bouganis, 2022) augments SoftMax (Hendrycks & Gimpel, 2017) baseline for the better UOSR performance.
Revised: SIRC (Xia & Bouganis, 2022) augments SoftMax (Hendrycks & Gimpel, 2017) baseline for the better UOSR performance.

Original: Build upon these existing methods, we make a deep analysis of UOSR in this work.
Revised: Build upon these existing methods, we make a deep analysis of UOSR in this work.

Original: In Table 1, we compare different settings of the related uncertainty estimation tasks, and the detailed discussions are as follows.
Revised: In Table 1, we compare different settings of the related uncertainty estimation tasks, and the detailed discussions are as follows.

Original: Selective Prediction (SP).
Revised: Selective Prediction (SP).

Original: Apart from the classical classification, SP also tries to estimate which sample is wrongly classified (Corbi ` ere et al., 2019; Moon et al., 2020; Granese et al., 2021), so the ground truth uncertainty of InW is 1.
Revised: Apart from the classical classification, SP also tries to estimate which sample is wrongly classified (Corbi ` ere et al., 2019; Moon et al., 2020; Granese et al., 2021), so the ground truth uncertainty of InW is 1.

Original: SP is constrained under the closed-set scenario and does not consider OoD samples during evaluation, as shown in Table 1, which is the key difference with
Revised: SP is constrained under the closed-set scenario and does not consider OoD samples during evaluation, as shown in Table 1, which is the key difference with

Original: UOSR.
Revised: UOSR.

Original: We involve some SP methods in the UOSR benchmark in Sec. 4.
Revised: We involve some SP methods in the UOSR benchmark in Sec. 4.

Original: Anomaly Detection (AD).
Revised: Anomaly/Outlier Detection (AD/OD).

Original: AD is to detect anomaly patches within an image or anomaly events within a video (Deng & Li, 2022; Zaheer et al., 2022).
Revised: AD is to detect anomaly patches within an image or anomaly events within a video (Deng & Li, 2022; Zaheer et al., 2022).

Original: Therefore, AD is defined within a test sample, and there is no InD  samples classification task.
Revised: OD regards a whole dataset as InD and samples from other datasets as OoD (Chauhan et al., 2022; Goodge et al., 2022).

Original: Unlike other tasks that assign uncertainty on the image or video level, AD assigns uncertainty on the pixel or frame level.
Revised: Both AD and OD do not require InD classification, so there is no InC/InW discrimination problem.

Original: Open-set Recognition (OSR) and Out-of-distribution Detection (OoDD).
Revised: Open-set Recognition (OSR) and Out-of-distribution Detection (OoDD).

Original: The task settings of
Revised: The task settings of

Original: OSR and OoDD are same, but their datasets might be different.
Revised: OSR and OoDD are same, but their datasets might be different.

Original: Both of them aim to accept all InD samples no matter they are InC or InW samples, and reject OoD samples.
Revised: Both of them aim to accept all InD samples no matter they are InC or InW samples, and reject OoD samples.

Original: OSR divides one dataset into two splits and uses one of them as InD data to train the model, while another split is regarded as
Revised: OSR divides one dataset into two splits and uses one of them as InD data to train the model, while another split is regarded as

Original: OoD samples (Scheirer et al., 2013).
Revised: OoD samples (Scheirer et al., 2013).

Original: In contrast, OoDD uses a whole dataset as InD data and regards another dataset as OoD data (Hendrycks & Gimpel, 2017).
Revised: In contrast, OoDD uses a whole dataset as InD data and regards another dataset as OoD data (Hendrycks & Gimpel, 2017).

Original: However, in the recent works about OSR in the video domain (Bao et al., 2021), it utilizes the OoDD setting rather than the OSR setting.
Revised: However, in the recent works about OSR in the video domain (Bao et al., 2021), it utilizes the OoDD setting rather than the OSR setting.

Original: In this work, we use OSR to represent the task setting and use one dataset as InD data and another dataset as OoD data.
Revised: In this work, we use OSR to represent the task setting and use one dataset as InD data and another dataset as OoD data.

Original: As mentioned before, the distinction between UOSR and OSR is InW samples, where OSR aims to accept them, and UOSR aims to reject them, so the ground truth uncertainty of
Revised: As mentioned before, the distinction between UOSR and OSR is InW samples, where OSR aims to accept them, and UOSR aims to reject them, so the ground truth uncertainty of

Original: InW samples is 0 in OSR and 1 in UOSR, as shown in Table 1.
Revised: InW samples is 0 in OSR and 1 in UOSR, as shown in Table 1.

Original: Better InC/OoD performance is beneficial for both UOSR and OSR, but higher InC/InW discrimination is preferred by UOSR but not wanted by OSR.
Revised: Better InC/OoD performance is beneficial for both UOSR and OSR, but higher InC/InW discrimination is preferred by UOSR but not wanted by OSR.

Original: Model Calibration (MC).
Revised: Model Calibration (MC).

Original: All tasks mentioned above solve the uncertainty ordinal ranking problem, i.e. , the ground truth uncertainty of each type of data is fixed, and the performance will be better if the estimated uncertainty is closer to the ground truth uncertainty (Geifman et al., 2019).
Revised: All tasks mentioned above solve the uncertainty ordinal ranking problem, i.e. , the ground truth uncertainty of each type of data is fixed, and the performance will be better if the estimated uncertainty is closer to the ground truth uncertainty (Geifman et al., 2019).

Original: In contrast, MC uses uncertainty to measure the probability of correctness (Guo et al., 2017).
Revised: In contrast, MC uses uncertainty to measure the probability of correctness (Guo et al., 2017).

Original: For a perfect calibrated model, if we consider all samples with a confidence of 0.8, then 80% of samples are correctly classified.
Revised: For a perfect calibrated model, if we consider all samples with a confidence of 0.8, then 80% of samples are correctly classified.

Original: In this case, 20% InW samples with a confidence of 0.8 are perfect, but in other uncertainty-related tasks, InW samples are supposed to have a confidence of either 1 or 0.
Revised: In this case, 20% InW samples with a confidence of 0.8 are perfect, but in other uncertainty-related tasks, InW samples are supposed to have a confidence of either 1 or 0.

Original: MC also does not consider OoD samples.
Revised: MC also does not consider OoD samples.

Original: See Appendix A for more relation between UOSR and MC. 
Revised: See Appendix A for more relation between UOSR and MC. v

Original:  OSR A PPROACHES FOR UOSR
Revised:  OSR A PPROACHES FOR UOSR

Original: In this section, we evaluate the existing OSR approaches for the UOSR problem, and show that the InW samples play a crucial role when evaluating the uncertainty quality.
Revised: In this section, we evaluate the existing OSR approaches for the UOSR problem, and show that the InW samples play a crucial role when evaluating the uncertainty quality.

Original: Specifically, simply changing the ground truth uncertainty u of InW samples from 0 to 1 can bring a large performance boost, as shown in Fig.
Revised: Specifically, simply changing the ground truth uncertainty u of InW samples from 0 to 1 can bring a large performance boost, as shown in Fig.

Original: Then we provide the comprehensive experiment results and discussion of this phenomenon.
Revised: Then we provide the comprehensive experiment results and discussion of this phenomenon.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: Applied Methods.
Revised: Applied Methods.

Original: We reproduce several classical OSR methods and evaluate their UOSR and OSR performance in our experiments, including SoftMax (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018), LC (DeVries & Taylor, 2018), OpenMax (Bendale & Boult, 2016), OLTR (Liu et al., 2019) and PROSER (Zhou et al., 2021) in the image domain, as well as DEAR (Bao et al., 2021),
Revised: We reproduce several classical OSR methods and evaluate their UOSR and OSR performance in our experiments, including SoftMax (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018), LC (DeVries & Taylor, 2018), OpenMax (Bendale & Boult, 2016), OLTR (Liu et al., 2019) and PROSER (Zhou et al., 2021) in the image domain, as well as DEAR (Bao et al., 2021),

Original: RPL (
Revised: RPL (

Original: MC Dropout (Gal & Ghahramani,
Revised: MC Dropout (Gal & Ghahramani,

Original: ., 2018) in the domain.
Revised: ., 2018) in the domain.

Original: Datasets.
Revised: Datasets.

Original: In the image domain, the InD dataset is CIFAR100 (Krizhevsky et al., 2009) and the OoD datasets are TinyImageNet (Le & Yang, 2015) and LSUN (Yu et al., 2015).
Revised: In the image domain, the InD dataset is CIFAR100 (Krizhevsky et al., 2009) and the OoD datasets are TinyImageNet (Le & Yang, 2015) and LSUN (Yu et al., 2015).

Original: In the video domain, we follow (Bao et al., 2021) and adopt UCF101 (Soomro et al., 2012) as InD dataset, while the OoD datasets are HMDB51 (Kuehne et al., 2011) and MiTv2 (Monfort et al., 2021).
Revised: In the video domain, we follow (Bao et al., 2021) and adopt UCF101 (Soomro et al., 2012) as InD dataset, while the OoD datasets are HMDB51 (Kuehne et al., 2011) and MiTv2 (Monfort et al., 2021).

Original: Experiments settings.
Revised: Experiments settings.

Original: We train the network using the InD dataset and evaluate the UOSR and OSR performance based on the ground truth in Table 1.
Revised: We train the network using the InD dataset and evaluate the UOSR and OSR performance based on the ground truth in Table 1.

Original: The evaluation metric is AUORC (Hendrycks &
Revised: The evaluation metric is AUORC (Hendrycks &

Original: Gimpel, 2017) which is a threshold-free value.
Revised: Gimpel, 2017) which is a threshold-free value.

Original: The AUROC reflects the distinction quality of two uncertainty distributions.
Revised: The AUROC reflects the distinction quality of two uncertainty distributions.

Original: We adopt VGG13 (Simonyan & Zisserman, 2015) and ResNet50 (He et al., 2016) as the network backbone in the image domain and TSM (Lin et al., 2019) and I3D (Carreira & Zisserman, 2017) in the video domain.
Revised: We adopt VGG13 (Simonyan & Zisserman, 2015) and ResNet50 (He et al., 2016) as the network backbone in the image domain and TSM (Lin et al., 2019) and I3D (Carreira & Zisserman, 2017) in the video domain.

Original: More experiment settings are in Appendix C.
Revised: More experiment settings are in Appendix E.

Original: Results.
Revised: Results.

Original: We provide the UOSR and OSR performance of different methods in Fig.
Revised: We provide the UOSR and OSR performance of different methods in Fig.

Original: It shows that all data points are above the red dotted diagonal, illustrating that the UOSR performance is significantly better than the OSR performance of the same method, and this relationship holds across different datasets, domains (image and video), and network architectures (See Appendix B for results of more network backbones).
Revised: It shows that all data points are above the red dotted diagonal, illustrating that the UOSR performance is significantly better than the OSR performance of the same method, and this relationship holds across different datasets, domains (image and video), and network architectures (See Appendix C for results of more network backbones).

Original: The performance gap between UOSR and OSR of the same method can be very large, such as 8.84% for OLTR when TinyImageNet is the OoD dataset, and 10.16% for BNN SVI when the OoD dataset is MiTv2.
Revised: The performance gap between UOSR and OSR of the same method can be very large, such as 8.84% for OLTR when TinyImageNet is the OoD dataset, and 10.16% for BNN SVI when the OoD dataset is MiTv2.

Original: Therefore, existing OSR methods have uncertainty distributions that are actually closer to the expectation of UOSR than OSR.
Revised: Therefore, existing OSR methods have uncertainty distributions that are actually closer to the expectation of UOSR than OSR.

Original: Analysis.
Revised: Analysis.

Original: To better understand our findings, we provide a detailed analysis of the uncertainty distribution relationships between InC, InW, and OoD samples in Tables 2 and 3.
Revised: To better understand our findings, we provide a detailed analysis of the uncertainty distribution relationships between InC, InW, and OoD samples in Tables 2 and 3.

Original: Note that higher
Revised: Note that higher

Original: [Figure]
Revised: [Figure]

Original: AUROC means a better distinction between two uncertainty distributions, and AUROC=50% means two distributions overlap with each other.
Revised: AUROC means a better distinction between two uncertainty distributions, and AUROC=50% means two distributions overlap with each other.

Original: From Tables 2 and 3 we can clearly see that AUROC of
Revised: From Tables 2 and 3 we can clearly see that AUROC of

Original: InC/OoD and InC/InW are significantly higher than InW/OoD, and AUROC of InW/OoD is very close to 50%.
Revised: InC/OoD and InC/InW are significantly higher than InW/OoD, and AUROC of InW/OoD is very close to 50%.

Original: Therefore, the uncertainty distribution of InC samples is distinguishable from OoD and InW samples, and there is a lot of overlap between the uncertainty distributions of InW and OoD samples.
Revised: Therefore, the uncertainty distribution of InC samples is distinguishable from OoD and InW samples, and there is a lot of overlap between the uncertainty distributions of InW and OoD samples.

Original: We provide several uncertainty distribution visualizations  in Fig.
Revised: Several uncertainty distribution visualizations are in Fig.

Original: 3, in which we can see that InW and OoD samples share a similar uncertainty distribution, while InC samples have smaller uncertainty in general.
Revised: 3, where we can see that InW and OoD samples share a similar uncertainty distribution, while InC samples have smaller uncertainty.

Revised: Importance.

Original: Therefore, the devil is in the  InW samples for OSR, as InW samples have a similar uncertainty distribution to OoD samples rather than InC samples, which is contradictory to the ground truth of OSR.
Revised: Based on the above analysis, we conclude that the false positive predictions in OSR tend to be InW samples, since they share similar uncertainty distributions with OoD samples. This conclusion is extremely important as InW samples significantly deteriorate the OSR performance.

Original: In contrast, the uncertainty distribution of InW samples is consistent with theexpectation of UOSR, so existing OSR methods are actually more suitable for  UOSR than OSR.
Revised: Without InW samples, the OSR performance increases by a large margin (75.59 to 84.69 for SoftMax method in Table 2). However, no existing OSR works mentioned and considered this phenomenon. We explicitly point out this conclusion and hope the following researchers take it into account when they design new OSR or UOSR methods.

Original: Why?
Revised: Why?

Original: We provide some empirical reasons for the finding that InW samples sharea similar uncertainty distribution with OoD samples rather than InC  samples.
Revised: To deeply understand why InW samples share similar uncertainty distribution with OoD samples instead of InC samples, we begin by analyzing the feature distributions of InC/InW/OoD samples.

Original: The model assigns low uncertainty for InC samples as they are well fitted by the model, and the cross-entropy loss encourages the SoftMax probability of  the corresponding class to be close to 1. However, InW and OoD samples are ‘hard’ samples which are not well fitted by the model, sothe cross-entropy loss cannot constrain the predictions to be confidence and their predictions tend to be more uncertain (Vaze et al., 2022;
Revised: We first find that the features of InC/InW/OoD follow the hierarchy structure where InC/InW/OoD samples are gradually far away from the training samples, so their features are separable. Please see Fig.

Original: Ranjan et al., 2017).
Revised: 10 and 11 for reference.

Original: The fact that AUROC of InW/OoD is close to 50% in Table 2 and 3 and the qualitative results in Fig.
Revised: Then we calculate the feature similarity and surprisingly find that InW features are more similar to InC features rather than OoD features in Table 9, which contradicts the uncertainty score phenomenon.

Original: 3 can prove our claims.
Revised: Therefore, the reason that InW samples have similar uncertainty scores with OoD samples is not they have similar features, but lies in the uncertainty estimation methods. We provide explanations about how the uncertainty estimation process causes the contradictory phenomenon in Appendix D.

Original: [Table]
Revised: [Table]

Original: A better closed-set model is better for UOSR.
Revised: A better closed-set model is better for UOSR.

Original: Recently, (Vaze et al., 2022) found that better closed-set performance means better OSR performance.
Revised: Recently, (Vaze et al., 2022) found that better closed-set performance means better OSR performance.

Original: We provide a deeper explanation of this finding and show that this conclusion also holds for UOSR.
Revised: We provide a deeper explanation of this finding and show that this conclusion also holds for UOSR.

Original: Table 4 shows that we improve the closed-set accuracy through data augmentation and longer training (Vaze et al., 2022).
Revised: Table 4 shows that we improve the closed-set accuracy through data augmentation and longer training (Vaze et al., 2022).

Original: The open-set method is the SoftMax baseline with ResNet50 backbone, and the OoD dataset is TinyImageNet.
Revised: The open-set method is the SoftMax baseline with ResNet50 backbone, and the OoD dataset is TinyImageNet.

Original: For OSR, we can see that AUROC of InC/OoD is significantly better than OSR, which indicates the uncertainty distribution of InW samples are contradictory with the expectation of OSR.
Revised: For OSR, we can see that AUROC of InC/OoD is significantly better than OSR, which indicates the uncertainty distribution of InW samples are contradictory with the expectation of OSR.

Original: So less InW samples and better InC/OoD performance are two reasons for better OSR performance when closed-set accuracy is higher.
Revised: So less InW samples and better InC/OoD performance are two reasons for better OSR performance when closed-set accuracy is higher.

Original: For UOSR, both the InC/InW and InC/OoD performance are improving with the growth of closed-set accuracy, which brings better UOSR performance.
Revised: For UOSR, both the InC/InW and InC/OoD performance are improving with the growth of closed-set accuracy, which brings better UOSR performance.

Original:  P RE - TRAINING AND O UTLIER E XPOSURE
Revised:  P RE - TRAINING AND O UTLIER E XPOSURE

Original: After directly applying existing OSR methods for UOSR in Sec. 3, we explore two additional training settings in this section, including pre-training (Hendrycks et al., 2019a) and outlier exposure (Hendrycks et al., 2019b), which are effective methods to improve the OSR performance because of introduced extra information beyond the training set.
Revised: After directly applying existing OSR methods for UOSR in Sec. 3, we explore two additional training settings in this section, including pre-training (Hendrycks et al., 2019a) and outlier exposure (Hendrycks et al., 2019b), which are effective methods to improve the OSR performance [Table] 85.67 DOCTOR [Table]

Original: Pre-training is to use large-scale pre-trained weights for initialization for better down-task performance.
Revised: N 72.08 378. SIRC [Table] N 73.13 358.

Revised: SIRC

Revised: [Table]

Revised: N 73.13 348.

Revised: 90.39 SIRC

Revised: [Table]

Revised: N 73.13 355.

Revised: [Table]

Revised: N 73.13 346.

Revised: SoftMax

Revised: [Table]

Revised: N 73.28 363.

Revised: 85.93 SoftMax

Revised: [Table]

Revised: (OE

Revised: [Table]

Revised: R 73.54 339.

Revised: ARPL

Revised: [Table]

Revised: N 73.03 345.

Revised: 83.49 ARPL+CS †

Revised: R 72.78 349.

Revised: MCD

Revised: [Table]

Revised: Dropout

Revised: [Table]

Revised: N 76.49 375.

Revised: 81.69 MCD

Revised: [Table]

Revised: R 70.88 365.

Revised: PROSER †

Revised: G 68.08 394.

Revised: 89.29 PROSER † (EB † )

Revised: R

Original:  Outlier exposure is to introduce some unlabeled outlier data (OD) during training and regard these outlier data as the proxy as
Revised: [Table] because of introduced extra information beyond the training set. Pre-training is to use large-scale pre-trained weights for initialization for better down-task performance. Outlier exposure is to introduce some unlabeled outlier data (OD) during training and regard these outlier data as the proxy as

Original: OoD data to improve the open-set performance.
Revised: OoD data to improve the open-set performance.

Original: We find both of them also have a positive effect on
Revised: We find both of them also have a positive effect on

Original: UOSR, but for different reasons.
Revised: UOSR, but for different reasons.

Original: [Table]

Original: Pre-training settings.
Revised: Pre-training settings.

Original: In the image domain, we use BiT pre-training weights (Kolesnikov et al., 2020) for ResNet50 and ImageNet (Deng et al., 2009) pre-training weights for VGG13.
Revised: In the image domain, we use BiT pre-training weights (Kolesnikov et al., 2020) for ResNet50 and ImageNet (Deng et al., 2009) pre-training weights for VGG13.

Original: In the video domain, Kinetics400 (Carreira & Zisserman, 2017) pre-trained weights are used for initialization.
Revised: In the video domain, Kinetics400 (Carreira & Zisserman, 2017) pre-trained weights are used for initialization.

Original: Outlier exposure settings.
Revised: Outlier exposure settings.

Original: In the image domain, we use 300K Random Images dataset (Hendrycks et al., 2019b) as outlier data.
Revised: In the image domain, we use 300K Random Images dataset (Hendrycks et al., 2019b) as outlier data.

Original: In the video domain, outlier data comes from Kinetics400, and we filter out the overlapping classes with InD and OoD datasets.
Revised: In the video domain, outlier data comes from Kinetics400, and we filter out the overlapping classes with InD and OoD datasets.

Original: We implement several outlier exposure based methods, including OE (Hendrycks et al., 2019b), EB (Thulasidasan et al., 2021),Energy (Liu et al., 2020), VOS (Du et al., 2022) and MCD (Yu & Aizawa, 2019).
Revised: We implement several outlier exposure based methods, including OE (Hendrycks et al., 2019b), EB (Thulasidasan et al., 2021), VOS (Du et al., 2022) and MCD (Yu & Aizawa, 2019).

Original: UOSR benchmark settings.
Revised: UOSR benchmark settings.

Original: We include OSR-based, SP-based, and UOSR-based methods to build a comprehensive UOSR benchmark.
Revised: We include OSR-based, SP-based, and UOSR-based methods to build a comprehensive UOSR benchmark.

Original: We implement several Selective Prediction (SP) based methods, including BCE, TCP (Corbi`ere et al., 2019), CRL (Moon et al., 2020), and DOCTOR (Granese et al., 2021).
Revised: We implement several Selective Prediction (SP) based methods, including BCE, TCP (Corbi`ere et al., 2019), CRL (Moon et al., 2020), and DOCTOR (Granese et al., 2021).

Original: Although these SP methods are originally designed to differentiate InC and InW samples, we adopt them in the UOSR setting to build a more comprehensive benchmark.
Revised: Although these SP methods are originally designed to differentiate InC and InW samples, we adopt them in the UOSR setting to build a more comprehensive benchmark.

Original: The only
Revised: The only

Original: UOSR-based method is SIRC (Xia & Bouganis, 2022), which combines two uncertainty scores for better UOSR performance.
Revised: UOSR-based method is SIRC (Xia & Bouganis, 2022), which combines two uncertainty scores for better UOSR performance.

Original: The evaluation metrics are AURC and AUROC (Kim et al., 2021).
Revised: The evaluation metrics are AURC and AUROC (Kim et al., 2021).

Original: More details can be found in Appendix C.
Revised: More details can be found in Appendix E.

Original: Results.
Revised: Results.

Original: The UOSR benchmarks for image and video domains are Table 5 and 6 respectively.
Revised: The UOSR benchmarks for image and video domains are Table 5 and 6 respectively.

Original: See
Revised: See

Original: Appendix D for the results of different model architectures.
Revised: Appendix J for the results of different model architectures.

Original: First, we can see the AUROC of UOSR is higher than OSR for almost all the methods under both settings, i.e. , pre-training and outlier exposure, which further strengthens the conclusion that the uncertainty distribution of OSR methods is closer to the ground truth of UOSR.
Revised: First, we can see the AUROC of UOSR [Table] ∥ z ∥ 1 ) ♢ N 73.59 172.

Revised: SIRC( −H , Res.)

Revised: ♢

Revised: N 73.59 175.

Revised: SoftMax †

Revised: N 73.92 173.

Revised: 91.75 SoftMax † (OE † )

Revised: R 74.42 162.

Revised: MCD † (Dropout † )

Revised: N 73.63 184.

Revised: 88.23 MCD †

Revised: R 72.49 168.

Revised: VOS †

Revised: G 74.00 187.

Revised: 93.62 VOS †

Revised: R 74.68 172.

Revised: EB †

Original:  For instance, AUROC is 87.34 for UOSR and 79.34 for OSR under the Energy method w/o pre-training in Table 5.
Revised: R 70.90 212. [Figure] set is higher than OSR for almost all the methods under both settings, i.e. , pre-training and outlier exposure, which further strengthens the conclusion that the uncertainty distribution of OSR methods is closer to the ground truth of UOSR. For instance, AUROC is 89.59 for UOSR and 83.93 for OSR under the Ensemble method w/o pre-training in Table 5.

Original: Second, pre-training and outlier exposure can effectively boost the UOSR and OSR performance , e.g. , the pre-training boost the AUROC of
Revised: Second, pre-training and outlier exposure can effectively boost the UOSR and OSR performance , e.g. , the pre-training boost the AUROC of

Revised: UOSR/OSR from 84.90/75.59 to 90.50/85.93 for the SoftMax, and outlier exposure method OE has 90.19/86.29 AUROC of UOSR/OSR compared to 84.90/75.59 of SoftMax in Table 5.

Revised: Note that to ensure outlier data is useful, we keep methods w/ and w/o outlier data as similar as possible, like

Original: UOSR/OSR from 84.90/75.59 to 90.50/85.93 for the SoftMax, and  outlier exposure method VOS has 89.96/85.78 AUROC of UOSR/OSR compared to 84.90/75.59 of SoftMax in Table 5.
Revised: SoftMax/OE, ARPL/ARPL+CS, and Dropout/MCD. Third, real outlier data is more beneficial than generated outlier data in the UOSR and OSR tasks. The UOSR/OSR AUROC of VOS method is 89.96/85.78 for real outlier data and 86.65/79.72 for generated outlier data, provided in Table 5.

Original: Analysis.
Revised: Analysis.

Original: To better understand why pre-training and outlier exposure are helpful for UOSR and
Revised: To better understand why pre-training and outlier exposure are helpful for UOSR and

Original: OSR, we provide the InC/InW and InC/OoD discrimination performance of each method in Fig. 4.
Revised: OSR, we provide the InC/InW and InC/OoD discrimination performance of each method in Fig. 4.

Original: We can see that almost all outlier exposure methods are in the Q2, which means that outlier exposure methods have lower InC/InW AUROC and higher InC/OoD AUROC than the SoftMax baseline.
Revised: We can see that almost all outlier exposure methods are in the Q2, which means that outlier exposure methods have lower InC/InW AUROC and higher InC/OoD AUROC than the SoftMax baseline.

Original: For
Revised: For

Original:  OSR, both lower InC/InW and higher InC/OoD AUROC are beneficial, but only higher InC/OoD AUROC is wanted for UOSR, while lower InC/InW AUROC is not. This can explain why some of
Revised: [Table] OSR, both lower InC/InW and higher InC/OoD AUROC are beneficial

Original: [Table]
Revised: [Table]

Original: SIRC(MSP, Res.) ♢ ✗ 73.59 174. 91.73 SIRC( −H , ∥ z ∥ 1 ) ♢ ✗ 73.59 172. SIRC( −H , Res.)
Revised: but only higher InC/OoD

Original: ♢ ✗ 73.59 175.

Original: OE ✓ 74.42 162. 93.40 EB ✓ 70.90 212. 93.06 Energy ✓ 74.00 187. 93.62 VOS ✓ 74.68 172.
Revised: AUROC is wanted for UOSR

Original: 93.72 MCD ✓ 72.49 168.

Original: [Figure]
Revised: [Table]

Original: We set the SoftMax method training from scratch as the original point and divide the coordinate into the UOSR performances with outlier exposure are comparable or even worse than the baseline, such as EB, Energy, and VOS in the video domain Table 5, but all of the OSR performances are increased.
Revised: while lower InC/InW AUROC is not. This can explain why some of the UOSR performances with outlier exposure are comparable or even worse than the baseline, such as EB and VOS in the video domain Table 6, but all of the OSR performances are increased.

Original: In contrast, pre-training is helpful for both InC/InW and InC/OoD AUROC, as methods with pretraining are in the Q1.
Revised: In contrast, pre-training is helpful for both InC/InW and InC/OoD AUROC, as methods with pretraining are in the Q1.

Revised: Outlier exposure methods also benefit from pre-training, as green marks are at the upper right location compared to orange marks.

Original: Therefore, outlier exposure and pre-training bring the UOSR performance gain for different reasons.
Revised: Therefore, outlier exposure and pre-training bring the UOSR performance gain for different reasons.

Original: InC/OoD performance is improved by both of the techniques, but pretraining is also helpful for InC/InW, but outlier exposure is not.
Revised: InC/OoD performance is improved by both of the techniques, but pre-training is also helpful for InC/InW, while outlier exposure is not.

Original: This can be explained by the fact that the model may see additional InD samples during pre-training so that the closed-set accuracy and InC/InW discrimination are improved.
Revised: This can be explained by the fact that the model may see additional InD samples during pre-training so that the closed-set accuracy and InC/InW discrimination are improved.

Original: In contrast, outlier exposure only provides OoD data but no more InD data, so only InC/OoD performance is improved.
Revised: In contrast, outlier exposure only provides OoD data but no more InD data, so only InC/OoD performance is improved.

Original:  F EW - SHOT U NIFIED O PEN - SET R ECOGNITION
Revised:  F EW - SHOT U NIFIED O PEN - SET R ECOGNITION

Original: In addition to the analysis of two useful training settings in Sec. 4, we introduce a new evaluation setting into UOSR in this section.
Revised: In addition to the analysis of two useful training settings in Sec. 4, we introduce a new evaluation setting into UOSR in this section.

Original: Inspired by the recent work SSD (Sehwag et al., 2022) which proposes the few-shot OSR, we introduce the few-shot UOSR, where 1 or 5 samples per OoD class can be introduced for reference to better distinguish OoD samples.The introduced reference OoD [Table] [Figure]
Revised: Inspired by the recent work SSD (Sehwag et al., 2022) which proposes the few-shot OSR, we introduce the few-shot UOSR, where 1 or 5 samples per OoD class can be introduced for reference to better distinguish OoD samples.

Original: (b), FS-KNN (c), and FS-KNNS (d). datasets are marked as D reftest = { ( x i , y i ) }
Revised: The introduced reference OoD datasets are marked as D reftest = { ( x i , y i ) }

Original: N ′ i =1 ⊂
Revised: N ′ i =1 ⊂

Original: X × U .
Revised: X × U .

Original: We show that few-shot UOSR has distinct challenges from few-shot OSR because of InW samples, and we propose our FS-KNNS method to achieves the state-of-the-art UOSR performance even without outlier exposure during training.
Revised: We show that few-shot UOSR has distinct challenges from few-shot OSR because of InW samples, and we propose our FS-KNNS method to achieves the state-of-the-art UOSR performance even without outlier exposure during training.

Original: Baselines and dataset settings.
Revised: Baselines and dataset settings.

Original: SSD is the only existing few-shot OSR method that utilizes the Mahalanobis distance with InD training samples D train and OoD reference samples D reftest .
Revised: SSD is the only existing few-shot OSR method that utilizes the Mahalanobis distance with InD training samples D train and OoD reference samples D reftest .

Original: KNN (Sun et al., 2022) utilizes the feature distance with samples in D train as uncertainty scores and shows it is better than Mahalanobis distance.
Revised: KNN (Sun et al., 2022) utilizes the feature distance with samples in D train as uncertainty scores and shows it is better than Mahalanobis distance.

Original: So we slightly modify KNN to adapt to the few-shot UOSR and find the modified FS-KNN can already beat SSD.
Revised: So we slightly modify KNN to adapt to the few-shot UOSR and find the modified FS-KNN can already beat SSD.

Original: D reftest comes from the validation set of TinyImageNet and training set of UCF101 in the image and video domain.
Revised: D reftest comes from the validation set of TinyImageNet and training set of UCF101 in the image and video domain.

Original: We repeat the evaluation process until all OoD reference samples are used and calculate the mean as the final few-shot result.
Revised: We repeat the evaluation process until all OoD reference samples are used and calculate the mean as the final few-shot result.

Original: FS-KNN.
Revised: FS-KNN.

Original: Given a test sample x ∗ and a feature extractor f , the feature of x ∗ is z ∗ = f
Revised: Given a test sample x ∗ and a feature extractor f , the feature of x ∗ is z ∗ = f

Original: ( x ∗ ) .
Revised: ( x ∗ ) .

Original: The cosine similarity set between z ∗ and the feature set of D train is S train =
Revised: The cosine similarity set between z ∗ and the feature set of D train is S train =

Original: { d i } Ni =1 , d i =
Revised: { d i } Ni =1 , d i =

Original: ( z ∗ · z i ) / ( ∥ z ∗ ∥ · ∥ z i ∥ ) , z i = f
Revised: ( z ∗ · z i ) / ( ∥ z ∗ ∥ · ∥ z i ∥ ) , z i = f

Original: ( x i ) , x i ∈ D train .
Revised: ( x i ) , x i ∈ D train .

Original: S reftest is similar with S train except x i comes from
Revised: S reftest is similar with S train except x i comes from

Original: D reftest .
Revised: D reftest .

Original: We believe the uncertainty should be higher if the test sample is more similar to reference
Revised: We believe the uncertainty should be higher if the test sample is more similar to reference

Original: OoD samples and not similar with InD samples.
Revised: OoD samples and not similar with InD samples.

Original: Therefore, the uncertainty score is
Revised: Therefore, the uncertainty score is

Original: [Equation]
Revised: [Equation]

Original: (1) where topK means the K th largest value.
Revised: where topK means the K th largest value.

Original: The performances of SoftMax, KNN, FS-KNN, and SSD are in Table 7 and 8.
Revised: The performances of SoftMax, KNN, FS-KNN, and SSD are in Table 7 and 19.

Original: Pre-training is used for better performance.
Revised: Pre-training is used for better performance.

Original: FS-KNN has better InC/OoD performance than KNN as reference OoD samples are introduced.
Revised: FS-KNN has better InC/OoD performance than KNN as reference OoD samples are introduced.

Original: Although the overall UOSR performance of FS-KNN is better than the SoftMax baseline, the InC/InW performance is significantly sacrificed, which is also an important aspect of UOSR.
Revised: Although the overall UOSR performance of FS-KNN is better than the SoftMax baseline, the InC/InW performance is significantly sacrificed, which is also an important aspect of UOSR.

Original: For example, InC/InW performance drops from 89.58 to 79.58 in the 5-shot results of Table 7. 
Revised: For example, InC/InW performance drops from 89.58 to 79.58 in the 5-shot results of Table 7. So we naturally ask a question: Can we improve [Figure]

Original: So we naturally ask a question: Can we improve the InC/OoD performance based on the introduced OoD reference samples while keeping similar
Revised: 𝜆 !" "# 𝜆 !" [Figure] the InC/OoD performance based on the introduced OoD reference samples while keeping similar

Original: InC/InW performance with SoftMax baseline?
Revised: InC/InW performance with SoftMax baseline?

Original: This is the key difference between few-shot OSR and few-shot UOSR, as low InC/InW performance is wanted in OSR but not preferred in UOSR.
Revised: This is the key difference between few-shot OSR and few-shot UOSR, as low InC/InW performance is wanted in OSR but not preferred in UOSR.

Original: FS-KNNS.
Revised: FS-KNNS.

Original: Inspired by SIRC (Xia & Bouganis, 2022), we aim to find a way to fuse SoftMax and
Revised: Inspired by SIRC (Xia & Bouganis, 2022), we aim to find a way to fuse SoftMax and

Original: FS-KNN scores so that the mixed score can keep the high InC/OoD performance of FS-KNN and meanwhile has the comparable InC/InW performance with SoftMax.
Revised: FS-KNN scores so that the mixed score can keep the high InC/OoD performance of FS-KNN and meanwhile has the comparable InC/InW performance with SoftMax.

Original: The uncertainty distributions of SoftMax and FS-KNNS are depicted in Fig.
Revised: The uncertainty distributions of SoftMax and FS-KNNS are depicted in Fig.

Original: We find that OoD samples have larger uncertainty than InW and InC samples in the FS-KNN method, but the uncertainty of InC samples overlaps a lot with InW samples, which explains the reason that InC/OoD performance is high but InC/InW [Table]
Revised: We find that OoD samples have larger uncertainty than InW and InC samples in the FS-KNN method, but the uncertainty of InC samples overlaps a lot with InW samples, which explains the reason that InC/OoD performance is high but InC/InW performance is low.

Original: performance is low. In contrast, InW and OoD samples share similar uncertainty in the SoftMax method, which brings higher InC/InW performance.
Revised: In contrast, InW and OoD samples share similar uncertainty in the SoftMax method, which brings higher InC/InW performance.

Original: Therefore, we want to keep the uncertainty of
Revised: Therefore, we want to keep the uncertainty of

Original: InW samples in the SoftMax method, as well as the uncertainty of OoD samples in the FS-KNN method.
Revised: InW samples in the SoftMax method, as well as the uncertainty of OoD samples in the FS-KNN method.

Original: In this way, the mixed scores obtain the high InC/OoD performance from FS-KNN while keeping the comparable InC/InW performance of SoftMax.
Revised: In this way, the mixed scores obtain the high InC/OoD performance from FS-KNN while keeping the comparable InC/InW performance of SoftMax.

Original: We call this method FS-KNN with
Revised: We call this method FS-KNN with

Original: SoftMax (FS-KNNS), and the uncertainty is
Revised: SoftMax (FS-KNNS), and the uncertainty is

Original: [Equation]
Revised: [Equation]

Original: where u 0 and u 1 refer to the uncertainty score of SoftMax and FS-KNN, respectively.
Revised: where u 0 and u 1 refer to the uncertainty score of SoftMax and FS-KNN, respectively.

Original: λ knns is a threshold to determine when the weight of u 1 becomes large, and α is a coefficient to control the change rate of the weight.
Revised: λ knns is a threshold to determine when the weight of u 1 becomes large, and α is a coefficient to control the change rate of the weight.

Original: ˆ u fs − knns will be largely influenced by u 

Revised: ˆ u fs − knns will be largely influenced by u

Original: when u 1 > λ knns .
Revised: when u 1 > λ knns .

Original: A proper λ knns should be located between the InW and OoD samples, as shown in Fig.
Revised: A proper λ knns should be located between the InW and OoD samples, as shown in Fig.

Original: 5 (a) and (c).
Revised: 5 (a) and (c).

Original: In this way, the uncertainty of InW samples is mainly controlled by SoftMax, and OoD samples are strengthened by FS-KNN.
Revised: In this way, the uncertainty of InW samples is mainly controlled by SoftMax, and OoD samples are strengthened by FS-KNN.

Original: Ablation study about hyperparameters K, α, and λ knns is in Appendix E.
Revised: Ablation study about hyperparameters K, α, and λ knns is in Appendix L.

Original: TSTPTS w/ OE TP w/ OE TP w/ FS (Ours)
Revised: Discussion.

Original: The uncertainty distribution of FSKNNS is shown in Fig.
Revised: The uncertainty distribution of FS-KNNS is shown in Fig.

Original: 5 (d), which shows the uncertainty of InC and InW samples are similar to SoftMax (b), but the uncertainty of OoD samples is larger.
Revised: 5 (d), which shows the uncertainty of InC and InW samples are similar to SoftMax (b), but the uncertainty of OoD samples is larger.

Original: From Table 7 and 8, we can see our FSKNNS has significantly better InC/InW performance than FS-KNN (87.98 and 79.58 under ResNet50), and meanwhile keeps the high InC/OoD performance, so the overall UOSR performance is better than both SoftMax and FS-KNN.
Revised: From Table 7 and 19, we can see our FS-KNNS has significantly better InC/InW performance than FS-KNN (87.98 and 79.58 under ResNet50), and meanwhile keeps the high InC/OoD performance, so the overall UOSR performance is better than both SoftMax and FS-KNN.

Original: We also try three score fusion methods, including FS-KNN+S, FSKNN*S, and SIRC, but these methods are general score fusion methods, while our method is specifically designed for UOSR, so our FS-KNNS surpass them.
Revised: We also try three score fusion methods, including FS-KNN+S, FS-KNN*S, and SIRC, but these methods are general score fusion methods, while our method is specifically designed for UOSR, so our FSKNNS surpass them.

Original: Our FS-KNNS is totally based on the existing model trained by the classical cross-entropy loss, so there is no extra effort and no outlier data during training, and our FS-KNNS still has better performance than all outlier exposure methods and achieves state-of-the-art UOSR performance, as shown in Fig.
Revised: Our FS-KNNS is totally based on the existing model trained by the classical cross-entropy loss, so there is no extra effort and no outlier data during training, and our FS-KNNS still has better performance than all outlier exposure methods and achieves state-of-the-art UOSR performance, as shown in Fig.

Original: 1 (b) and 6.
Revised: 1 (b) and 16.

Original: Note that the best choice for OSR (FS-KNN or SSD) may not be the best choice for UOSR (FS-KNNS), as their expectation of InC/InW is contradictory, which shows the necessity of our proposed few-shot UOSR.
Revised: Note that the best choice for OSR (FS-KNN or SSD) may not be the best choice for UOSR (FS-KNNS), as their expectation of InC/InW is contradictory, which shows the necessity of our proposed few-shot UOSR.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Rejecting both InW and OoD samples during evaluation has very practical value in most real-world applications, but UOSR is a newly proposed problem and lacks comprehensive analysis and research, so we look deeper into the UOSR problem in this work.
Revised: Rejecting both InW and OoD samples during evaluation has very practical value in most real-world applications, but UOSR is a newly proposed problem and lacks comprehensive analysis and research, so we look deeper into the UOSR problem in this work.

Original: We first demonstrate the uncertainty distribution of almost all existing OSR methods is actually closer to the expectation of UOSR than
Revised: We first demonstrate the uncertainty distribution of almost all existing OSR methods is actually closer to the expectation of UOSR than

Original: OSR, and then analyze two training settings including pre-training and outlier exposure.
Revised: OSR, and then analyze two training settings including pre-training and outlier exposure.

Original: Finally, we introduce a new evaluation setting into UOSR, which is few-shot UOSR, and we propose a
Revised: Finally, we introduce a new evaluation setting into UOSR, which is few-shot UOSR, and we propose a

Original: FS-KNNS method that achieves state-of-the-art performance under all settings.
Revised: FS-KNNS method that achieves state-of-the-art performance under all settings.

Original: Reproducibility Statement
Revised: Reproducibility Statement

Original: We provide the uncertainty distribution result files and corresponding codes required to reproduce
Revised: We provide the uncertainty distribution result files and corresponding codes required to reproduce

Original: Tables 2, 3, 5, 6, 7 and 8 in the supplementary material folder UOSR/UOSR eval .
Revised: Tables 2, 3, 5, 6, 7 and 19 in the supplementary material folder UOSR/UOSR eval .

Original: The users do not need to run the training or evaluation process for a deep learning model.
Revised: The users do not need to run the training or evaluation process for a deep learning model.

Original: Simply follow the
Revised: Simply follow the

Original: README UOSR eval.md in the UOSR/UOSR eval folder and all performance results can be printed in the terminal in the table form.
Revised: README UOSR eval.md in the UOSR/UOSR eval folder and all performance results can be printed in the terminal in the table form.

Original: We also provide the training codes for all methods under all settings (pre-training and outlier exposure) based on the ResNet50 backbone in the supplementary material folder UOSR/UOSR train .
Revised: We also provide the training codes for all methods under all settings (pre-training and outlier exposure) based on the ResNet50 backbone in the supplementary material folder UOSR/UOSR train .

Original: The corresponding guideline is in README UOSR train.md .
Revised: The corresponding guideline is in README UOSR train.md .

Original: The datasets and experiments details can be found in Appendix C.
Revised: The datasets and experiments details can be found in Appendix E.

Original: How to conduct few-shot evaluation to obtain the corresponding uncertainty distribution result files is also included in the supplementary material folder UOSR/UOSR few shot .
Revised: How to conduct few-shot evaluation to obtain the corresponding uncertainty distribution result files is also included in the supplementary material folder UOSR/UOSR few shot .

Original: The corresponding guideline is in README UOSR few shot.md .
Revised: The corresponding guideline is in README UOSR few shot.md .

Original: The ablation study of hyper parameters in our FSKNNS is in Appendix E.
Revised: The ablation study of hyper parameters in our FSKNNS is in Appendix L.

Original: A R ELATION BETWEEN M ODEL C ALIBRATION AND UOSR
Revised: A R ELATION BETWEEN M ODEL C ALIBRATION AND UOSR

Original: Different from Selective Prediction (SP), Anomaly Detection (AD), OSR, and UOSR, Model Calibration (MC) is not an uncertainty ordinal ranking problem.
Revised: Different from Selective Prediction (SP), Anomaly Detection (AD), OSR, and UOSR, Model Calibration (MC) is not an uncertainty ordinal ranking problem.

Original: In other words, all other settings expect the uncertainty of a test sample to be either 0 or 1, while MC is not.
Revised: In other words, all other settings expect the uncertainty of a test sample to be either 0 or 1, while MC is not.

Original: A perfect calibrated model should meet (Guo et al., 2017)
Revised: A perfect calibrated model should meet (Guo et al., 2017)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where p is the confidence or the negative version of uncertainty p = 1 − u .
Revised: where p is the confidence or the negative version of uncertainty p = 1 − u .

Original: For example, given 100 test samples whose confidence scores are all 0.8, then the model is perfectly calibrated if 80% samples are correctly classified.
Revised: For example, given 100 test samples whose confidence scores are all 0.8, then the model is perfectly calibrated if 80% samples are correctly classified.

Original: In this case, 20% samples with confidence 0.8 is consistent with the requirement of a perfect calibrated model, but the confidence is supposed to be either 0 or 1 in other settings.
Revised: In this case, 20% samples with confidence 0.8 is consistent with the requirement of a perfect calibrated model, but the confidence is supposed to be either 0 or 1 in other settings.

Original: The performance of MC is evaluated by ECE, and the readers may refer to (Guo et al., 2017) for the formal definition.
Revised: The performance of MC is evaluated by ECE, and the readers may refer to (Guo et al., 2017) for the formal definition.

Original: Smaller ECE means better calibrated model.
Revised: Smaller ECE means better calibrated model.

Original: We provideexamples to illustrate the relation between the performance of UOSR and MC in Fig.
Revised: We provideexamples to illustrate the relation between the performance of UOSR and MC in Fig.

Original: Note that
Revised: Note that

Original: MC does not consider OoD samples during evaluation, so we also do not involve OoD samples in
Revised: MC does not consider OoD samples during evaluation, so we also do not involve OoD samples in

Original: Fig.
Revised: Fig.

Original: Therefore, UOSR in Fig. 7 is also equal to SP.
Revised: Therefore, UOSR in Fig. 6 is also equal to SP.

Original: From (f) we can see that the performance of UOSR and model calibration is not perfectly positively correlated, i.e. , the best case for model calibration (c) is not the best case for UOSR and vice versa (e).
Revised: From (f) we can see that the performance of UOSR and model calibration is not perfectly positively correlated, i.e. , the best case for model calibration (c) is not the best case for UOSR and vice versa (e).

Original: [Figure]
Revised: [Figure]

Original: AU
Revised: AU

Original: P R
Revised: P R

Original: E C E
Revised: E C E

Revised: Figure 6: We provide 5 samples in (a)-(e), where we keep the confidence distribution of InC and change the confidence distribution of InW samples.

Original:  B E XISTING OSR METHODS ARE MORE SUITABLE FOR UOSR 
Revised: The evaluation metrics of UOSR are AUROC and AUPR, and ECE is for the MC. B T EMPERATURE SCALING FOR UOSR Temperature scaling is a convenient and effective method for model calibration (Guo et al., 2017). We study how this method influences the UOSR performance.

Revised: The experiments are conducted under R50 backbone while InD and OoD datasets are CIFAR100 and TinyImageNet, respectively.

Revised: The quantitative results are in Table 8.

Revised: The uncertainty distribution under different temperatures T are in Fig.

Revised: 7 and 8.

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: From Table 8 we can see that the optimal T for MC ( T = 2 ) is not the best case for UOSR.

Revised: When

Revised: T grows, the InC/OoD discrimination increases, but the InC/InW discrimination drops.

Revised: Therefore, the OSR performance keeps improving with larger T , but the UOSR may not benefit from larger T because of lower InC/InW discrimination.

Revised: For example, the best T for UOSR with pre-training israther than 20.

Revised: But in general, temperature scaling is a simple and useful technique for both MC and

Revised: UOSR, as T = 2 has better MC and UOSR performance than T = 1 (without temperature scaling).

Revised: The only drawback of temperature scaling is it needs the validation set to determine the optimal T .

Original: In addition to Fig. 2, we provide the results of VGG13 backbone in the image domain and I3D backbone in the video domain.
Revised: In addition to Fig. 2, we provide the results of VGG13 backbone in the image domain and I3D backbone in the video domain.

Original: Fig.
Revised: Fig.

Original: 8 further proves the finding that existing OSR methods are actually more suitable for UOSR holds for differnt model architectures and domains.
Revised: 9 further proves the finding that existing OSR methods are actually more suitable for UOSR holds for differnt model architectures and domains.

Original: [Figure]
Revised: [Figure]

Revised: D F EATURE AND UNCERTAINTY SCORE BEHAVIOR

Original: (72.68, 82.81) (78.32, 87.13) (81.09, 84.62) +10.
Revised: To deeply understand why the InW samples share similar uncertainty distribution with OoD samples rather than InC samples, we analyze the feature representation and uncertainty score behavior in this section. Surprisingly, we find the InW samples have more similar features with InC samples than OoD samples, which is opposite from the uncertainty score behavior. Features of InC/InW/OoD samples are separable.

Revised: We visualize the feature representations in

Revised: Fig 10.

Revised: We find the feature distribution of InC, InW and OoD samples follow a hierarchy structure.

Revised: The InW samples surround the InC samples, and OoD samples are further far away and located at the outer edge of InW samples, such as three distributions of class A and B in Fig 10 (b).

Revised: Therefore, the features of InC, InW, and OoD samples are separable from the feature representation perspective.

Revised: To further testify this idea, we calculate the similarity between InC/InW/OoD samples and training samples of each class, as shown in Fig.

Revised: We can see that the InC samples are the most similar samples with training samples, and InW samples have smaller similarity, and OoD samples have the smallest similarity.

Revised: In conclusion, features of InC/InW/OoD samples are in a hierarchy structure and distinguishable.

Revised: [Figure]

Revised: Class

Revised: [Figure]

Revised: il a r it y

Revised: [Table]

Revised: Features of InW samples are more similar with InC samples than OoD samples.

Revised: Our finding in Sec.

Revised: 3 is that InW samples share similar uncertainty scores with OoD samples rather than InC samples, so we analyze whether the feature representation also follows the same behavior.

Revised: We calculate the similarity between InW/InC samples and InW/OoD samples in the feature space and logit space.

Revised: Then we provide the mean of uncertainty scores based on the KNN method and MaxLogit method, as well as the AUROC of InW/InC and InW/OoD to illustrate the uncertainty discrimination performance.

Revised: In Table 9 we can see that the similarity of InW/InC is larger than InW/OoD (0.797-0.733), which means InW samples have more similar features with InC samples than OoD samples.

Revised: However, the uncertainty scores of InW samples are more similar with OoD scores than

Revised: InC samples (0.158/0.179-0.057), so that InW/OoD can not be distinguished very well like InW/InC (57.25-84.91).

Revised: Therefore, we draw a very interesting conclusion that the feature behavior and uncertainty score behavior of InW/InC and InW/OoD are contradictory.

Revised: InW samples are more similar to

Revised: InC samples in the feature/logit space but more similar to OoD samples from the uncertainty score perspective.

Original: +5.
Revised: Let us formulate this phenomenon mathematically. Suppose we have x c , x w , x o which represents the feature of an InC, InW, and OoD sample, respectively. From Table 9 we know that [Equation]

Revised: where sim refers to the similarity.

Revised: Then, we have an uncertainty estimation function f to measure the uncertainty u of a sample based on the features, so u c = f ( x c ) , u w = f ( x w ) , u o = f

Original: +4.32 (76.68, 81.77) +8.
Revised: ( x o ) . Based on our finding in Sec. 3 that InW samples share similar uncertainty distribution with OoD samples rather than InC samples, we have [Equation]

Revised: Comparing Eq.

Revised: 5 and Eq.

Revised: 4 we find that the uncertainty estimation function f changes the similarity relationship between InW/InC and InW/OoD.

Revised: 𝑂 𝑠 𝑡

Original:  (a) (b)
Revised: [Equation] [Equation] # (a) 𝑢 [Equation]

Revised: = −𝑠

Revised: [Equation]

Revised: (𝑥 # )

Revised: [Equation]

Revised: [Equation]

Revised: " ) 𝑓(𝑥 ! )

Revised: Uncertainty 𝑂 𝑠 𝑡 𝑥 !

Original: HMDB51MiTv (75.85, 85.68) +9.
Revised: 𝑥 " 𝑥 # (b) 𝑢 𝑓 𝑥 = 𝑡/𝑠 𝑓(𝑥 # ) 𝑓(𝑥 " ) 𝑓(𝑥 ! ) Uncertainty Let us give a toy example of how f changes the similarity relationship. Suppose the logit space is under 2 dimensions ( s, t ) , and x c = (2 , 2) , x w = (1 , 1) , x o = (1 , 3) .

Revised: The similarity is measured with Euclidean distance, and in this case x w is closer to x c ,

Original: (83.13, 90.42) +7.
Revised: so Eq. 4 holds for this example. If f ( x ) = − s like Fig. 12 (a), then u c = − 2 , u w = − 1 , u o = − 1 .

Revised: In this case, u w = u o > u c .

Revised: So the uncertainty score of InW sample is similar with OoD sample instead of InC sample.

Revised: This example illustrates why an InW sample has a similar feature to an InC sample, but has a similar uncertainty score with OoD sample.

Original: {+This is how existing uncertainty estimation methods work, as the results in Sec. 
+}
Revised: {+This is how existing uncertainty estimation methods work, as the results in Sec. 
+}

Revised: and Sec. 4 show that InW samples have similar uncertainty distribution with OoD samples.

Revised: This kind of method is suitable for

Revised: UOSR problem where InW and OoD samples are supposed to be rejected at the same time.

Revised: We provide another uncertainty estimation function f in Fig.

Original: (78.91, 80.68) +1.
Revised: 12 (b), where f ( x ) = t/s . In this case, u w = u c = 1 < u o = 3 , so the uncertainty score of the InW sample is similar to the InC sample instead of the OoD sample. This is the ideal case for the traditional OSR problem to reject OoD samples and accept InC and InW samples.

Revised: Conclusion.

Revised: In this section, we first find that the features of InC, InW and OoD samples are separable, and show that InW samples have more similar features to InC samples than OoD samples, which seems contradictory with the finding in Sec.

Revised: 3 that InW samples share more similar uncertainty scores with OoD samples than InC samples.

Revised: Then we give a toy example to illustrate that the uncertainty estimation method is the reason for this phenomenon.

Revised: Although existing uncertainty estimation methods are designed to behave like Fig.

Revised: 12 (b) for the OSR problem, they actually work as Fig.

Revised: 12 (a) which is more suitable for the UOSR problem.

Original: C I MPLEMENTATION DETAILS C. I MAGE D OMAIN
Revised: E I MPLEMENTATION DETAILS E. D EFINITION OF AURC

Original: w/o pre-training.
Revised: We follow (Kim et al., 2021) to give the definition of AURC here. Suppose we have a dataset D = { ( x i , y i ) } ni =1 , where x i and y i denote the training sample and corresponding closed-set label.

Revised: Then, we use a confidence estimation method (methods in Table 5) to assign a confidence score κ ( p i ) for each x i .

Revised: Given a threshold θ , the sample coverage is computed as

Revised: [Equation]

Revised: where | · | is the number of samples in a set.

Revised: Then the risk r ( θ ) is computed as

Revised: [Equation]

Revised: where 1 ( · ) and ˆ y i denote the indicator function the predicted class respectively.

Revised: So we have ( c ( θ ) , r ( θ )) pairs which can consist of a risk-coverage curve (RC-curve).

Original:  When we train the model from scratch, we find that setting the base learning rate as 0.1 and step-wisely decayed by 10 every 24000 steps with totally 120000 steps can achieve good enough closed-set performance.
Revised: The AURC is the area under the RC-curve. E. I MAGE D OMAIN w/o pre-training. When we train the model from scratch, we find that setting the base learning rate as 0.1 and step-wisely decayed by 10 every 24000 steps with totally 120000 steps can achieve good enough closed-set performance.

Original: We use a linear warmup strategy to warmup the training in the first 500 steps.
Revised: We use a linear warmup strategy to warmup the training in the first 500 steps.

Original: We use SGD with momentum, batch size 128 for all models.
Revised: We use SGD with momentum, batch size 128 for all models.

Original: w/ ImageNet-21k pre-training.
Revised: w/ ImageNet-21k pre-training.

Original: When we fine tune the model with ImageNet pretrained model from (Kolesnikov et al., 2020), we set the base learning rate as 0.003 and step-wisely decayed every 3000 steps with totally 10000 steps.
Revised: When we fine tune the model with ImageNet pretrained model from (Kolesnikov et al., 2020), we set the base learning rate as 0.003 and step-wisely decayed every 3000 steps with totally 10000 steps.

Original: We use a linear warmup strategy to warmup the training in the first 500 steps.
Revised: We use a linear warmup strategy to warmup the training in the first 500 steps.

Original: We use SGD with momentum, batch size 512 for all models.
Revised: We use SGD with momentum, batch size 512 for all models.

Original: For outlier exposure methods, the batch size of outlier data is set to 128.
Revised: For outlier exposure methods, the batch size of outlier data is set to 128.

Original: InD and OoD datasets.
Revised: InD and OoD datasets.

Original: We follow the datasets setting in (Liang et al., 2018).
Revised: We follow the datasets setting in (Liang et al., 2018).

Original: The training InD dataset is CIFAR-100, which contains 100 classes with 50000 training images and 10000 test images.
Revised: The training InD dataset is CIFAR-100, which contains 100 classes with 50000 training images and 10000 test images.

Original: The OoD datasets for open-set evaluation are TinyImageNet and LSUN.
Revised: The OoD datasets for open-set evaluation are TinyImageNet and LSUN.

Original: The TinyImageNet dataset contains 10000 test images from 200 different classes, we use the TinyImageNet (resize) in (Liang et al., 2018).
Revised: The TinyImageNet dataset contains 10000 test images from 200 different classes, we use the TinyImageNet (resize) in (Liang et al., 2018).

Original: The Large-scale Scene UNderstanding (LSUN) dataset consists ofimages of 10 different scenes categories like classroom, conference room, dining room, etc.
Revised: The Large-scale Scene UNderstanding (LSUN) dataset consists ofimages of 10 different scenes categories like classroom, conference room, dining room, etc.

Original: We use
Revised: We use

Original: LSUN (resize) from (Liang et al., 2018).
Revised: LSUN (resize) from (Liang et al., 2018).

Original: The size of images in both InD and OoD datasets is× 32.
Revised: The size of images in both InD and OoD datasets is× 32.

Original: We use the same strategy in image preprocessing stage as (Kolesnikov et al., 2020), if the resolution of the training image is lower than 96 × 96, we use 128 × 128 image cropping technique and random horizontal mirroring followed by 160 × 160 image resize.
Revised: We use the same strategy in image preprocessing stage as (Kolesnikov et al., 2020), if the resolution of the training image is lower than 96 × 96, we use 128 × 128 image cropping technique and random horizontal mirroring followed by 160 × 160 image resize.

Original: Test images and OoD images are directly resized to 128 × 128.
Revised: Test images and OoD images are directly resized to 128 × 128.

Original: Outlier datasets.
Revised: Outlier datasets.

Original: We use 300K Random Images dataset from (Hendrycks et al., 2019b) as outlier dataset for those outlier exposure methods.
Revised: We use 300K Random Images dataset from (Hendrycks et al., 2019b) as outlier dataset for those outlier exposure methods.

Original: The 300K Random Images dataset is a debiased dataset with real images scraped from online.
Revised: The 300K Random Images dataset is a debiased dataset with real images scraped from online.

Original: According to (Hendrycks et al., 2019b), all images that belong to CIFAR classes and images with divisive metadata have been removed.
Revised: According to (Hendrycks et al., 2019b), all images that belong to CIFAR classes and images with divisive metadata have been removed.

Original: C. V IDEO D OMAIN
Revised: E. V IDEO D OMAIN

Original: w/o pre-training.
Revised: w/o pre-training.

Original: When we train the model from scratch, we find that setting the base learning rate as 0.05 and step-wisely decayed every 160 epochs with totally 400 epochs can achieve good enough closed-set performance.
Revised: When we train the model from scratch, we find that setting the base learning rate as 0.05 and step-wisely decayed every 160 epochs with totally 400 epochs can achieve good enough closed-set performance.

Original: The batch size is 256 for all methods.
Revised: The batch size is 256 for all methods.

Original: w/ Kinetics

Revised: w/ Kinetics

Original: pre-training.
Revised: pre-training.

Original: We follow (Bao et al., 2021) to set the base learning rate as 0.
Revised: We follow (Bao et al., 2021) to set the base learning rate as 0.

Original: and step-wisely decayed every 20 epochs with totally 50 epochs.
Revised: and step-wisely decayed every 20 epochs with totally 50 epochs.

Original: For those methods without outlier exposure, we fix the parameters of all Batch Normalization layers except the first one, and set the learning rate of the fully connected layer to be 10 times of the base learning rate.
Revised: For those methods without outlier exposure, we fix the parameters of all Batch Normalization layers except the first one, and set the learning rate of the fully connected layer to be 10 times of the base learning rate.

Original: For thosemethods with outlier exposure, all parameters are updated with the same learning rate.
Revised: For those

Original: InD and OoD datasets.
Revised: InD and OoD datasets.

Original: We follow the datasets setting in (Bao et al., 2021).
Revised: We follow the datasets setting in (Bao et al., 2021).

Original: The training InD dataset is UCF101, which contains 101 classes with 9537 training samples and 3783 test samples.
Revised: The training InD dataset is UCF101, which contains 101 classes with 9537 training samples and 3783 test samples.

Original: The OoD datasets for open-set evaluation are HMDB51 and MiT-v2.
Revised: The OoD datasets for open-set evaluation are HMDB51 and MiT-v2.

Original: We use the test sets of them which contain 1530 samples and 30500 samples respectively.
Revised: We use the test sets of them which contain 1530 samples and 30500 samples respectively.

Original: For UCF101 and HMDB51, we follow the MMAction (Yue Zhao, 2019) to use the split 1 for training and evaluation, which is the same with (Bao et al., 2021).
Revised: For UCF101 and HMDB51, we follow the MMAction (Yue Zhao, 2019) to use the split 1 for training and evaluation, which is the same with (Bao et al., 2021).

Original: Note that in (Bao et al., 2021), they find some classes in HMDB51 overlap with those in UCF101 but they do not clean them.
Revised: Note that in (Bao et al., 2021), they find some classes in HMDB51 overlap with those in UCF101 but they do not clean them.

Original: We remove the overlapping classes in UCF and HMDB51 so that OoD data does not contain any samples of InD classes.
Revised: We remove the overlapping classes in UCF and HMDB51 so that OoD data does not contain any samples of InD classes.

Original: The classes we remove in HMDB51 and the corresponding same classes in UCF101 are in Table 9. [Table] up [Table]
Revised: The classes we remove in HMDB51 and the corresponding same classes in UCF101 are in Table 10.

Original: Biking

Original: [Table]

Original: Basketball

Original: [Table]
Revised: [Table]

Original: Outlier datasets.
Revised: Outlier datasets.

Original: We use Kinetics400 as our outlier datasets for those outlier exposure methods.
Revised: We use Kinetics400 as our outlier datasets for those outlier exposure methods.

Original: To ensure that the classes of outlier data is not overlapping with InD data and OoD data, we remove corresponding classes in Kinetics400.
Revised: To ensure that the classes of outlier data is not overlapping with InD data and OoD data, we remove corresponding classes in Kinetics400.

Original: The overlapping classes between Kinetics400 and
Revised: The overlapping classes between Kinetics400 and

Original: UCF101/HMDB51 are too many to be listed here (129 overlapping classes).
Revised: UCF101/HMDB51 are too many to be listed here (129 overlapping classes).

Original: The available training sample ID list of Kinetics400 and all codes will be public.
Revised: The available training sample ID list of Kinetics400 and all codes will be public.

Original: We pick up 271 classes from Kinetics and 25 samples in each class as outlier data. 
Revised: We pick up 271 classes from Kinetics and 25 samples in each class as outlier data. F D ETAILED U NCERTAINTY A NALYSIS OF T ABLE

Revised: Table 11 is the complementary analysis of Table 5, which further illustrates that outlier data can improve InC/OoD discrimination but may not be helpful for InC/InW discrimination.

Revised: [Table]

Original: D UOSR BENCHMARK AND FEW - SHOT RESULTS OF VGG
Revised: G UOSR BENCHMARK UNDER TRADITIONAL OSR SETTING

Revised: We evaluate the UOSR and OSR performance under the traditional OSR dataset setting (Vaze et al., 2022), where a part of data within one dataset is regarded as InD and the remaining data is regarded as OoD. The experiments are conducted under the most challenging TinyImageNet dataset and the results are in Table 12. We can see that the UOSR performance is still higher than OSR performance for most methods, which means InW samples share more similar uncertainty distribution with InC samples than OoD samples. Surprisingly, we find the InW/OoD AUORC of ODIN method achieves 84.32, which means InW and OoD samples can be well distinguishable.

Revised: This proves our claim in

Revised: Appendix D that the features of InC/InW/OoD samples are separable and it is possible to find a proper uncertainty estimation method to distinguish these three groups of data.

Original:  We provide the UOSR benchmark and few-shot resultsof VGG13 in Table 10 and 11 respectively. 
Revised: [Table] H UOSR BENCHMARK UNDER DIFFERENT TRAINING SET We provide the UOSR evaluation results in Table 13 when InD and OoD datasets are TinyImageNet and CIFAR100 respectively. Pre-training weights are used.

Revised: TinyImageNet has 200 classes in the training set which is more diverse than CIFAR100.

Revised: We can see that training the model on TinyImageNet is more challenging than CIFAR100, as the Acc.

Revised: of CIFAR100 is 86.44 and Acc.

Revised: of

Original: Our FS-KNNS still achieves thebest performance under all settings.
Revised: TinyImageNet only reaches 77.02. In this way, the impact of InW samples becomes further huge. For example, the performance gap between UOSR and OSR for the SoftMax method is 8.95 when InD dataset is TinyImageNet

Original: [Table]
Revised: [Table]

Revised: and this value is only 0.34 when InD dataset is CIFAR100.

Revised: So the InW samples are more important for the performance when InD dataset is difficult, as lower closed-set

Revised: Acc. means more InW samples.

Original:  PROSER † ✗ 70.95 376.
Revised: [Table] OLTR † 77.00 341. PROSER † 75.93 392.

Original: BCE ‡ ✗ 74.74 339.
Revised: BCE ‡ 76.91 339.

Original: TCP ‡ ✗ 75.09 340.
Revised: TCP ‡ 77.82 336.

Original: DOCTOR ‡ ✗ 75.07 340.
Revised: DOCTOR ‡ 77.23 339. SIRC(MSP, ∥ z ∥ 1 ) ♢ 77.03 337.

Original: SIRC(MSP, 
Revised: SIRC(MSP, Res.)

Original: [Table]
Revised: ♢ 77.03 316. SIRC( −H , ∥ z ∥ 1 ) ♢ 77.03 333.

Original: ) ♢ ✗ 75.07 343.
Revised: SIRC( −H , Res.) ♢ 77.03 311.

Original: SIRC
Revised: (Vaze et al., 2022) proposed the Semantic Shift Benchmark (SSB) which compose several finegrained datasets, including CUB, Standford Cars, FGCV-Aircraft

Original: [Table]
Revised: [Table]

Original: ♢ ✗ 75.07 336.

Original: SIRC
Revised: and a part of ImageNet. OSR in SSB is more challenging as OoD samples share the same coarse labels with InD samples, and only have some minor differences in the fine-grained properties. We evaluate the UOSR performance on

Revised: CUB and FGCV-Aircraft and provide the results of EASY and HARD modes.

Revised: Pre-training weights are used for better performance.

Revised: From Table 14 and 15 we can see the UOSR and OSR performance are higher under the EASY mode compared to HARD mode as expected, since the OoD samples are more similar with InD samples in the HARD mode.

Revised: Our conclusion that InW samples share similar uncertainty with OoD samples still holds, as the AUROC of InW/OoD is close to 50 and much lower than InC/OoD and InC/InW.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: DOCTOR

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: ✗ 75.07 343.
Revised: [Table] [Table] J UOSR BENCHMARK AND FEW - SHOT RESULTS OF VGG  We provide the UOSR benchmark and few-shot results of VGG13 in Table 16 and 17 respectively.

Original: SIRC
Revised: Our FS-KNNS still achieves the best performance under all settings [Table] [Table] K UOSR UNDER N OISY O UTLIER E XPOSURE

Revised: When we introduce outlier data into the training process

Original: [Table]
Revised: [Table]

Original: −H
Revised: the labels of some outlier data may be corrupted and become the labels of InD class. We call this kind of outlier data as noisy outlier data. We study how noisy outlier data influence the UOSR and OSR performance in this section. In addition

Original: [Table]
Revised: [Table]

Revised: we use NGC (Wu et al., 2021) to find the noisy outlier data and correct them.

Revised: The results are shown in Table 18.

Revised: We can see that the closed-set Acc.

Revised: gradually decreases with the growth of noise level when NGC is not used.

Revised: This is natural as some noisy outlier data corrupt the InD data distribution.

Revised: In contrast, the closed-set Acc. does not drop with the growth of noise level when NGC is used.

Revised: So NGC is very effective in finding corrupted samples.

Revised: When the noise level is 0%, the

Original: ♢ ✗ 75.07 335.
Revised: UOSR performance with NGC is better than the performance without NGC (lower AURC value), meaning that other parts in NGC that are not related to label corruption, such as contrastive learning, are helpful for UOSR. Surprisingly, the performance of UOSR and OSR are relatively stable under different noise levels no matter we use NGC or not, compared to the clear performance drop of closed-set Acc. when NGC is not used. So the model is robust in open-set related performance when noisy outlier data is introduced.

Revised: Our finding that InD samples share similar uncertainty scores with OoD samples still holds as AUROC of InW/OoD is close to 50.

Original: OE ✓ 71.71 312.
Revised: [Table] % ✓ [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 40 40% ✓

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: 92.77 EB ✓74.19 340.
Revised: [Table] [Table] [Table] % ✓

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: 89.79 Energy ✓ 73.30 306.
Revised: % ✓ [Table] [Table] [Table]

Original: 92.19 VOS ✓ 71.73 312.
Revised: [Table] [Table] [Table]

Original: 92.97 MCD ✓ 70.45 316.{+77.
 [Table]+}
Revised: [Table] 49 100% ✓ {+77.
 [Table]+}

Original: [Table]
Revised: [Table]

Original: E A BLATION STUDY OF FEW - SHOT UOSR
Revised: L A BLATION STUDY OF FEW - SHOT UOSR

Original: Analyze of K .
Revised: Analyze of K .

Original: Similar with (Sun et al., 2022), we find the value of K influences the performance a lot.
Revised: Similar with (Sun et al., 2022), we find the value of K influences the performance a lot.

Original: The ablation study of K is shown in Fig.
Revised: The ablation study of K is shown in Fig.

Original: It shows that the best K for InC/OoD discrimination is between 3 and 5, and drops quickly after 7.
Revised: It shows that the best K for InC/OoD discrimination is between 3 and 5, and drops quickly after 7.

Original: In contrast, the InC/InW performance keeps increasing until 20 and then drops.
Revised: In contrast, the InC/InW performance keeps increasing until 20 and then drops.

Original: The overall UOSR performance achieves the best when K = 5 .
Revised: The overall UOSR performance achieves the best when K = 5 .

Original: [Figure]
Revised: [Figure]

Original: Analyze of α and λ knns .
Revised: Analyze of α and λ knns .

Original: Two hyper parameters of FS-KNNS including α and λ knns are significant for the performance.
Revised: Two hyper parameters of FS-KNNS including α and λ knns are significant for the performance.

Original: λ knns is a threshold to determine when the weight of u 1 becomes large, i.e. , ˆ u fs − knns is important when u 1 > λ knns .
Revised: λ knns is a threshold to determine when the weight of u 1 becomes large, i.e. , ˆ u fs − knns is important when u 1 > λ knns .

Original: α is to control the change rate of the weight.
Revised: α is to control the change rate of the weight.

Original: The ideal λ knns should be located between the uncertainty of InW and OoD samples as shown in Fig.
Revised: The ideal λ knns should be located between the uncertainty of InW and OoD samples as shown in Fig.

Original: 5 (a) and (c), so that the uncertainty of InW samples is still determined by SoftMax and the uncertainty of
Revised: 5 (a) and (c), so that the uncertainty of InW samples is still determined by SoftMax and the uncertainty of

Original: OoD samples is enlarged because of FS-KNN.
Revised: OoD samples is enlarged because of FS-KNN.

Original: However, which sample is InW or OoD is unknown during test, so we cannot determine the λ knns based on the uncertainty distribution of test samples.
Revised: However, which sample is InW or OoD is unknown during test, so we cannot determine the λ knns based on the uncertainty distribution of test samples.

Original: (Xia & Bouganis, 2022) proposed to determine the hyper parameters of SIRC based on the training set, but we find that the uncertainty distribution ˆ u fs − knns of training samples significantly different from the test InC samples, as shown in Fig.
Revised: (Xia & Bouganis, 2022) proposed to determine the hyper parameters of SIRC based on the training set, but we find that the uncertainty distribution ˆ u fs − knns of training samples significantly different from the test InC samples, as shown in Fig.

Original: Therefore, we seek help from the OoD reference
Revised: Therefore, we seek help from the OoD reference

Original: [Figure]
Revised: [Figure]

Original: the λ knns when β = 1 . 5 , 1 , 0 . 5 , 0 , − 0 . 5 in Eq.
Revised: [Figure]

Original: samples, as we find their uncertainty distribution is extremely similar with OoD test samples.
Revised: samples, as we find their uncertainty distribution is extremely similar with OoD test samples.

Original: We calculate the mean u and standard deviation σ of the ˆ u fs − knns of OoD reference samples, and we aim to find a proper λ knns through:
Revised: We calculate the mean u and standard deviation σ of the ˆ u fs − knns of OoD reference samples, and we aim to find a proper λ knns through:

Original: [Equation]
Revised: [Equation]

Original: We draw several λ knns with different β in Fig.
Revised: We draw several λ knns with different β in Fig.

Original: 10 (a to e).
Revised: 14 (a to e).

Original: A smaller λ knns or a larger β means more samples will be influenced by the FS-KNN uncertainty, so that the ˆ u fs − knns of more OoD test samples will be strengthened by the FS-KNN uncertainty which brings better InC/OoD performance, but meanwhile more InW samples will also be influenced by the FS-KNN uncertainty which brings[Figure] 𝛽 𝛼 𝛽 𝛽 𝛼 worse InC/InW performance, as shown in Fig.
Revised: A smaller λ knns or a larger β means more samples will be influenced by the FS-KNN uncertainty, so that the ˆ u fs − knns of more OoD test samples will be strengthened by the FS-KNN uncertainty which brings better InC/OoD performance, but meanwhile more InW samples will also be influenced by the FS-KNN uncertainty which brings worse InC/InW performance, as shown in Fig.

Original: In other words, λ knns controls the trade-off between InC/OoD and InC/InW performance.
Revised: In other words, λ knns controls the trade-off between InC/OoD and InC/InW performance.

Original: Overall, β = 1 achieves the best UOSR performance which is a balanced result of InC/OoD and InC/InW trade-off.
Revised: Overall, β = 1 achieves the best UOSR performance which is a balanced result of InC/OoD and InC/InW trade-off.

Original: Larger α means the weight of FSKNN uncertainty grows quickly when FS-KNN uncertainty is closed to λ knns .
Revised: Larger α means the weight of FSKNN uncertainty grows quickly when FS-KNN uncertainty is closed to λ knns .

Original: We find that a smaller α makes the performance insensitive of λ knns , and vice versa.
Revised: We find that a smaller α makes the performance insensitive of λ knns , and vice versa.

Original: So finally we pick α and β as 50 and 1 as the optimal hyper parameters. 
Revised: So finally we pick α and β as 50 and 1 as the optimal hyper parameters. M F EW - SHOT UOSR IN THE VIDEO DOMAIN [Table]