Original: H OW DOES U NCERTAINTY - AWARE S AMPLE - SELECTION H ELP D ECISION AGAINST A CTION N OISE ?
Revised: H OW DOES U NCERTAINTY - AWARE S AMPLE - SELECTION H ELP D ECISION AGAINST A CTION N OISE ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning from imperfect demonstrations has become a vital problem in imitation learning (IL).
Revised: Learning from imperfect demonstrations has become a vital problem in imitation learning (IL).

Original: Since the assumption of the collected demonstrations are optimal cannot always hold in real-world tasks, many previous works considers learning from a mixture of optimal and sub-optimal demonstrations.
Revised: Since the assumption of the collected demonstrations are optimal cannot always hold in real-world tasks, many previous works considers learning from a mixture of optimal and sub-optimal demonstrations.

Original: On the other hand, video records can be hands-down demonstrations in practice.
Revised: On the other hand, video records can be hands-down demonstrations in practice.

Original: Leveraging such demonstrations requires labors to output action for each frame.
Revised: Leveraging such demonstrations requires annotators to output action for each frame.

Original: However, action noise always occurs when the labors are not domain experts, or meet confusing state frames.
Revised: However, action noise always occurs when the annotators are not domain experts, or meet confusing state frames.

Original: Previous IL methods can be vulnerable to such demonstrations with state-dependent action noise .
Revised: Previous IL methods can be vulnerable to such demonstrations with state-dependent action noise .

Original: To tackle this problem, we propose a robust learning paradigm called USN, which bridges Uncertainty-aware Sample-selection with Negative learning.
Revised: To tackle this problem, we propose a robust learning paradigm called USN, which bridges Uncertainty-aware Sample-selection with Negative learning.

Original: First, IL model feeds forward all demonstration data and estimates its predictive uncertainty.
Revised: First, IL model feeds forward all demonstration data and estimates its predictive uncertainty.

Original: Then, we select large-loss samples in the light of the uncertainty measures.
Revised: Then, we select large-loss samples in the light of the uncertainty measures.

Original: Next, we update the model parameters with additional negative learning on the selected samples.
Revised: Next, we update the model parameters with additional negative learning on the selected samples.

Original: Empirical results on Box2D tasks and
Revised: Empirical results on Box2D tasks and

Original: Atari games demonstrate that USN improves the performance of state-of-the-art IL methods by more than 10% under a large portion of action noise.
Revised: Atari games demonstrate that USN improves the performance of state-of-the-art IL methods by more than 10% under a large portion of action noise.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite the great success of reinforcement learning (RL) (Sutton & Barto, 2018) over last few years, designing hand-crafted reward functions can be extremely difficult and even impossible in many real-world tasks (Ng et al., 1999;
Revised: Despite the great success of reinforcement learning (RL) (Sutton & Barto, 2018) over last few years, designing hand-crafted reward functions can be extremely difficult and even impossible in many real-world tasks (Ng et al., 1999;

Original: Amodei et al., 2016; Brown et al., 2019a).
Revised: Amodei et al., 2016; Brown et al., 2019a).

Original: Alternatively, imitation learning (IL) (Russell, 1998; Schaal, 1999; Abbeel & Ng, 2004; Argall et al., 2009; Hussein et al., 2017) aims to train an agent to mimic the demonstrations collected from an expert, without any access to hand-crafted reward signals.
Revised: Alternatively, imitation learning (IL) (Russell, 1998; Schaal, 1999; Abbeel & Ng, 2004; Argall et al., 2009; Hussein et al., 2017) aims to train an agent to mimic the demonstrations collected from an expert, without any access to hand-crafted reward signals.

Original: However, it is expensive and difficult to collect high-quality demonstrations in real-world tasks (Silver et al., 2013).
Revised: However, it is expensive and difficult to collect high-quality demonstrations in real-world tasks (Silver et al., 2013).

Original: In practice, it is much cheaper to collect demonstrations from amateurs (Audiffren et al., 2015).
Revised: In practice, it is much cheaper to collect demonstrations from amateurs (Audiffren et al., 2015).

Original: Existing works (Tangkaratt et al., 2019; 2020; Zhang et al., 2021b) have studied imitation learning from a mixture of optimal and non-optimal demonstrations.
Revised: Existing works (Tangkaratt et al., 2019; 2020; Zhang et al., 2021b) have studied imitation learning from a mixture of optimal and non-optimal demonstrations.

Original: Specifically, Tangkaratt et al.
Revised: Specifically, Tangkaratt et al.

Original: (2019) requires that all the actions for a demonstration are drawn from the same noisy distribution with sufficiently small variance.
Revised: (2019) requires that all the actions for a demonstration are drawn from the same noisy distribution with sufficiently small variance.

Original: Following works (Tangkaratt et al., 2020) proposed robust imitation learning by optimizing a classification risk with a symmetric loss.
Revised: Following works (Tangkaratt et al., 2020) proposed robust imitation learning by optimizing a classification risk with a symmetric loss.

Original: The resulting algorithms RIL and
Revised: The resulting algorithms RIL and

Original: RIL-CO still require more optimal demonstrations than non-optimal ones in the dataset.
Revised: RIL_CO still require more optimal demonstrations than non-optimal ones in the dataset.

Original: On the other hand, in many practical activities like sport games, it is common for people to use camera to record the excellent behaviors of athletes in terms of sequences of pictures and videos.
Revised: On the other hand, in many practical activities like sport games, it is common for people to use camera to record the excellent behaviors of athletes in terms of sequences of pictures and videos.

Original: However, it usually be hard to get the exact action label for each picture.
Revised: However, it usually be hard to get the exact action label for each picture.

Original: To leverage such data for imitation learning, we need to recruit labors to output action labels for pictures in the sequence.
Revised: To leverage such data for imitation learning, we need to recruit annotators to output action labels for pictures in the sequence.

Original: Limited by the quality of the labors, action noise always occur during the action-labeling procedure.
Revised: Limited by the quality of the annotators, action noise always occur during the action-labeling procedure.

Original: An amateur labor may randomly pick an action for a picture that contains a state he never seen before.
Revised: An amateur annotator may randomly pick an action for a picture that contains a state he never seen before.

Original: In this situation, the final demonstration will contain state-independent action noise .
Revised: In this situation, the final demonstration will contain state-independent action noise .

Original: Besides, even an expert labor makes mistakes.
Revised: Besides, even an expert annotator makes mistakes.

Original: This is especially true when the labor meets some similar and confusing states.
Revised: This is especially true when the annotator meets some similar and confusing states.

Original: In this situation, the labor will output noisy actions that are dependent on the confusing states, resulting a demonstration with state-dependent action noise .
Revised: In this situation, the annotator will output noisy actions that are dependent on the confusing states, resulting a demonstration with state-dependent action noise .

Original: Previous methods (Tangkaratt et al., 2019; 2020; Zhang et al., 2021b) focus on imitation learning from a mixture of optimal and non-optimal demonstrations or noisy demonstrations with small noise on the actions.
Revised: Previous methods (Tangkaratt et al., 2019; 2020; Zhang et al., 2021b) focus on imitation learning from a mixture of optimal and non-optimal demonstrations or noisy demonstrations with small noise on the actions.

Original: They ignore the fact that both state-independent action noise and state-independent action noise widely exist in practice.
Revised: They ignore the fact that both state-independent action noise and state-independent action noise widely exist in practice.

Original: Moreover, these methods are evaluated on low-dimensional environments, and are hard scale to high-dimensional environments.
Revised: Moreover, these methods are evaluated on low-dimensional environments, and are hard scale to high-dimensional environments.

Original: Therefore, existing methods usually fail in learning a good policy with action noise, especially in high-dimensional environments.
Revised: Therefore, existing methods usually fail in learning a good policy with action noise, especially in high-dimensional environments.

Original: To tackle this challenge, we first conduct an investigation experiments to study the correlation between the dynamics of loss and predictive uncertainty metrics as the noise rate increase.
Revised: To tackle this challenge, we first conduct an investigation experiments to study the correlation between the dynamics of loss and predictive uncertainty metrics as the noise rate increase.

Original: Then, we propose a new method called uncertainty-aware sample-selection with soft negative learning (USN) based on the correlation.
Revised: Then, we propose a new method called uncertainty-aware sample-selection with soft negative learning (USN) based on the correlation.

Original: As shown in Figure 1, USN trains a policy with additional process of uncertainty-aware sample-selection for negative learning (USN).
Revised: As shown in Figure 1, USN trains a policy with additional process of uncertainty-aware sample-selection for negative learning (USN).

Original: Specifically, in positive learning (marked in blue color), we train the IL model with any noise-tolerant loss function, and then estimate the predictive uncertainty measures during training.
Revised: Specifically, in positive learning (marked in blue color), we train the IL model with any noise-tolerant loss function, and then estimate the predictive uncertainty measures during training.

Original: Then, we select large-loss samples using the estimated predictive uncertainty measure as the sampling threshold.
Revised: Then, we select large-loss samples using the estimated predictive uncertainty measure as the sampling threshold.

Original: This is motivated by recent works in the area of learning with noisy labels (Angluin & Laird, 1988; Smyth et al., 1994; Kalai & Servedio, 2005; Natarajan et al., 2013; Manwani & Sastry, 2013).
Revised: This is motivated by recent works in the area of learning with noisy labels (Angluin & Laird, 1988; Smyth et al., 1994; Kalai & Servedio, 2005; Natarajan et al., 2013; Manwani & Sastry, 2013).

Original: Since deep networks learn easy patterns first (Arpit et al., 2017), they would first memorize training data of clean labels and then those of noisy labels with the assumption that clean labels are of the majority in a noisy class.
Revised: Since deep networks learn easy patterns first (Arpit et al., 2017), they would first memorize training data of clean labels and then those of noisy labels with the assumption that clean labels are of the majority in a noisy class.

Original: Therefore, the large-loss samples can be regarded as noisy actions with high probability (Han et al., 2018; Yu et al., 2019; Wei et al., 2020; Yao et al., 2020) .
Revised: Therefore, the large-loss samples can be regarded as noisy actions with high probability (Han et al., 2018; Yu et al., 2019; Wei et al., 2020; Yao et al., 2020) .

Original: Next, we employ negative learning (marked in green color)
Revised: Next, we employ negative learning (marked in green color)

Original: (Kim et al., 2019) on the automatically selected large-loss samples, along with positive learning on the full dataset.
Revised: (Kim et al., 2019) on the automatically selected large-loss samples, along with positive learning on the full dataset.

Original: USN is a meta-algorithm that is scalable to both offline imitation learning an online imitation learning.
Revised: USN is a meta-algorithm that is scalable to both offline imitation learning an online imitation learning.

Original: Compared with existing IL methods, our method has advantages in many aspects, including imitation performance, adaptivity and scalability .
Revised: Compared with existing IL methods, our method has advantages in many aspects, including imitation performance, adaptivity and scalability .

Original: Our method can adaptively select large-loss samples for soft negative learning across different noise rates.
Revised: Our method can adaptively select large-loss samples for soft negative learning across different noise rates.

Original: Additionally, our method does not depend on any extra datasets, models or prior information about the noise model.
Revised: Additionally, our method does not depend on any extra datasets, models or prior information about the noise model.

Original: That means our method avoids the extra efforts and drawbacks of estimating noise rates and transition matrix as previous noise-robust methods.
Revised: That means our method avoids the extra efforts and drawbacks of estimating noise rates and transition matrix as previous noise-robust methods.

Revised: Empirically results show that USN is scalable to Behavioral Cloning, online imitation learning and offline imitation learning.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: In this section, we firstly discuss existing offline and online imitation learning methods.
Revised: In this section, we firstly discuss existing offline and online imitation learning methods.

Original: Then, we discussed related works of learning with noisy labels.
Revised: Then, we discussed related works of learning with noisy labels.

Original: Behavioral cloning (BC) is probably the simplest offline imitation learning algorithm.
Revised: Behavioral cloning (BC) is probably the simplest offline imitation learning algorithm.

Original: For imitation learning in environments with discrete action space, the BC policy π ( a | s ) is optimized by softmax cross-entropy loss.
Revised: For imitation learning in environments with discrete action space, the BC policy π ( a | s ) is optimized by softmax cross-entropy loss.

Original: 2. O NLINE IMITATION LEARNING
Revised: 2. O NLINE IMITATION LEARNING

Original: Generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016) is one of the state-of-theart online IL methods.
Revised: Generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016) is one of the state-of-theart online IL methods.

Original: GAIL treats imitation learning as a distribution matching problem.
Revised: GAIL treats imitation learning as a distribution matching problem.

Original: Built of top of GAN (Goodfellow et al., 2014), GAIL and its many (robust) variants (Li et al., 2017; Peng et al., 2019; Tangkaratt et al., 2019; 2020; Wang et al., 2021) has achieved great success in imitation learning in low-dimensional space even with noisy demonstrations.
Revised: Built of top of GAN (Goodfellow et al., 2014), GAIL and its many (robust) variants (Li et al., 2017; Peng et al., 2019; Tangkaratt et al., 2019; 2020; Wang et al., 2021) has achieved great success in imitation learning in low-dimensional space even with noisy demonstrations.

Original: However, GAIL fails to scale to high-dimensional imitation learning tasks (Brown et al., 2019a; Tucker et al., 2018).
Revised: However, GAIL fails to scale to high-dimensional imitation learning tasks (Brown et al., 2019a; Tucker et al., 2018).

Original: One simple solution to alleviate this issue is to initiate the actor of GAIL using behavior cloning.
Revised: One simple solution to alleviate this issue is to initiate the actor of GAIL using behavior cloning.

Original: Selective Adversarial Imitation Learning (SAIL) (Wang et al., 2021) was proposed to address imperfect demonstration issue, in which good demonstrations can be adaptively selected for training while bad demonstrations are abandoned.
Revised: Selective Adversarial Imitation Learning (SAIL) (Wang et al., 2021) was proposed to address imperfect demonstration issue, in which good demonstrations can be adaptively selected for training while bad demonstrations are abandoned.

Original: Specifically, a binary weight w ∈ { 0 , 1 } is assigned to each expert demonstration to indicate whether to select it for training.
Revised: Specifically, a binary weight w ∈ { 0 , 1 } is assigned to each expert demonstration to indicate whether to select it for training.

Original: The weight is set to be determined by the reward function in Wasserstein GAIL (Xiao et al., 2019)
Revised: The weight is set to be determined by the reward function in Wasserstein GAIL (Xiao et al., 2019)

Original: (i.e. higher reward results in higher weight).
Revised: (i.e. higher reward results in higher weight).

Original: The resulting algorithm - SAIL-hard is defined as follows:
Revised: The resulting algorithm - SAIL-hard is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: Besides hard binary weighting, they also propose a soft weighting scheme with the suggested optimal soft weight as w ∗ =
Revised: Besides hard binary weighting, they also propose a soft weighting scheme with the suggested optimal soft weight as w ∗ =

Original: [Equation]
Revised: [Equation]

Original: rϕ [Equation] s
Revised: rϕ ( s

Original: [Equation]
Revised: [Equation]

Original: The corresponding algorithm - SAIL-soft is defined as follows:
Revised: The corresponding algorithm - SAIL-soft is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where f ( w ) =
Revised: where f ( w ) =

Original: w log( w − 1 − 1) − log(1 − w ) − Kw .
Revised: w log( w − 1 − 1) − log(1 − w ) − Kw .

Original: We regards GAIL, and its variants RIL, RIL-CO (Tangkaratt et al., 2020) and SAIL as baselines, for comparing their robustness against action noise.
Revised: We regards GAIL, and its variants RIL, RIL_CO (Tangkaratt et al., 2020) and SAIL as baselines, for comparing their robustness against action noise.

Original: 2. O FFLINE IMITATION LEARNING
Revised: 2. O FFLINE IMITATION LEARNING

Original: Batch-constrained Q-learning (BCQ) (Fujimoto et al., 2019b) is the first continuous control algorithm capable of learning from arbitrary batch data, without exploration.
Revised: Batch-constrained Q-learning (BCQ) (Fujimoto et al., 2019b) is the first continuous control algorithm capable of learning from arbitrary batch data, without exploration.

Original: BCQ aims to perform
Revised: BCQ aims to perform

Original: Q-learning while constraining the action space to eliminate actions that are unlikely to be selected by the behavioral policy π b , and are therefore unlikely to be contained in the batch.
Revised: Q-learning while constraining the action space to eliminate actions that are unlikely to be selected by the behavioral policy π b , and are therefore unlikely to be contained in the batch.

Original: At its core, BCQ uses a state-conditioned generative model G : s → a to model the distribution of data in the batch,
Revised: At its core, BCQ uses a state-conditioned generative model G : s → a to model the distribution of data in the batch,

Original: G ≈ b akin to a behavioral cloning model.
Revised: G ≈ b akin to a behavioral cloning model.

Original: As it is easier to sample from π b ( a | s ) than model π b ( a | s ) exactly in a continuous action space, the policy is defined by sampling N actions a i from G ( s ) and selecting the highest valued action according to a Q-network.
Revised: As it is easier to sample from π b ( a | s ) than model π b ( a | s ) exactly in a continuous action space, the policy is defined by sampling N actions a i from G ( s ) and selecting the highest valued action according to a Q-network.

Original: Discrete BCQ (Fujimoto et al., 2019a) extends BCQ to discrete action spaces by computing the probabilities of every action G ≈ π b ( a | s ) , and instead utilizing some threshold to eliminate actions.
Revised: Discrete BCQ (Fujimoto et al., 2019a) extends BCQ to discrete action spaces by computing the probabilities of every action G ≈ π b ( a | s ) , and instead utilizing some threshold to eliminate actions.

Original: To adaptively adjust this threshold, Fujimoto et al.
Revised: To adaptively adjust this threshold, Fujimoto et al.

Original: (2019a) scale it by the maximum probability from the generative model over all actions, to only allow actions whose relative probability is above some threshold.
Revised: (2019a) scale it by the maximum probability from the generative model over all actions, to only allow actions whose relative probability is above some threshold.

Original: Specifically, Fujimoto et al. (2019a) applies Double DQN (Van Hasselt et al., 2016) to select the max valued action with the current Q-network Q θ , and evaluate with the target Q-network
Revised: Specifically, Fujimoto et al. (2019a) applies Double DQN (Van Hasselt et al., 2016) to select the max valued action with the current Q-network Q θ , and evaluate with the target Q-network

Original: Q θ ′ .
Revised: Q θ ′ .

Original: This results in an algorithm comparable to DQN (Mnih et al., 2015) where the policy is defined by a constrained argmax.
Revised: This results in an algorithm comparable to DQN (Mnih et al., 2015) where the policy is defined by a constrained argmax.

Original: The Q-network is trained by swapping the max operation with actions selected by the policy:
Revised: The Q-network is trained by swapping the max operation with actions selected by the policy:

Original: [Equation]
Revised: [Equation]

Original: where l κ defines the Huber loss (Watkins, 1989):
Revised: where l κ defines the Huber loss (Watkins, 1989):

Original: [Equation]
Revised: [Equation]

Original: With this threshold τ , Fujimoto et al. (2019a) maintain the original property of BCQ where setting τ = 0 returns Q-learning and τ = 1 returns an imitator of the actions contained in the batch.
Revised: With this threshold τ , Fujimoto et al. (2019a) maintain the original property of BCQ where setting τ = 0 returns Q-learning and τ = 1 returns an imitator of the actions contained in the batch.

Original: The generative model G , effectively a behavioral cloning network, is trained in a standard supervised learning fashion, with a cross-entropy loss.
Revised: The generative model G , effectively a behavioral cloning network, is trained in a standard supervised learning fashion, with a cross-entropy loss.

Original: In this paper, we adopt the (Discrete) BCQ method for offline IL by training a intrinsic curiosity module (ICM) Pathak et al.
Revised: In this paper, we adopt the (Discrete) BCQ method for offline IL by training a intrinsic curiosity module (ICM) Pathak et al.

Original: (2017) for generating intrinsic rewards, instead of using the true reward data in the demonstration.
Revised: (2017) for generating intrinsic rewards, instead of using the true reward data in the demonstration.

Original: Uncertainty weighted actor-critic (UWAC) (Wu et al., 2021) is a recently proposed offline reinforcement learning algorithm that is able to detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly.
Revised: Uncertainty weighted actor-critic (UWAC) (Wu et al., 2021) is a recently proposed offline reinforcement learning algorithm that is able to detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly.

Original: UWAC is closely related to our work, since it also uses the uncertainty estimation to improve the model robustness.
Revised: UWAC is closely related to our work, since it also uses the uncertainty estimation to improve the model robustness.

Original: Specifically, UWAC uses Monte
Revised: Specifically, UWAC uses Monte

Original: Carlo (MC) dropout (Gal & Ghahramani, 2016) to estimate uncertainty of Q model.
Revised: Carlo (MC) dropout (Gal & Ghahramani, 2016) to estimate uncertainty of Q model.

Original: The model uncertainty is captured by the approximate predictive variance with respect to the estimated ˆ Q for T stochastic forward passes:
Revised: The model uncertainty is captured by the approximate predictive variance with respect to the estimated ˆ Q for T stochastic forward passes:

Original: [Equation]
Revised: [Equation]

Original: Then, UWAC down-weighs the Bellman loss for the Q function by inverse the uncertainty of the
Revised: Then, UWAC down-weighs the Bellman loss for the Q function by inverse the uncertainty of the

Original: Q-target Q θ ′ ( s ′ , a ′ ) :
Revised: Q-target Q θ ′ ( s ′ , a ′ ) :

Original: [Equation]
Revised: [Equation]

Original: where Err ( s, a, s ′ , a ′ )
Revised: where Err ( s, a, s ′ , a ′ )

Original: =
Revised: =

Original: Q θ ( s, a ) − ( R [Equation] s, a ))
Revised: Q θ ( s, a ) − ( R ( s, a ))

Original: + γQ θ ′ ( s ′ , a ′ ) .
Revised: + γQ θ ′ ( s ′ , a ′ ) .

Original: This directly reduces the effect that
Revised: This directly reduces the effect that

Original: OOD backups has on the overall training process (Wu et al., 2021).
Revised: OOD backups has on the overall training process (Wu et al., 2021).

Original: We treat UWAC as a strong baseline in our paper, and we also show that our USN can scale to MC dropout for uncertainty estimation.
Revised: We treat UWAC as a strong baseline in our paper, and we also show that our USN can scale to MC dropout for uncertainty estimation.

Original: The original UWAC was built on top of BEAR (Kumar et al., 2019).
Revised: The original UWAC was built on top of BEAR (Kumar et al., 2019).

Original: To enable a fair comparison, we implement UWAC on top of BCQ as follows:
Revised: To enable a fair comparison, we implement UWAC on top of BCQ as follows:

Original: [Equation]
Revised: [Equation]

Original: 2. L EARNING WITH NOISY LABELS
Revised: 2. L EARNING WITH NOISY LABELS

Original: Learning with noisy labels aims to learn a robust classifier f by exploiting training samples with only noisy labels ( x i , ˜ y i )
Revised: Learning with noisy labels aims to learn a robust classifier f by exploiting training samples with only noisy labels ( x i , ˜ y i )

Original: Ni =1 .
Revised: Ni =1 .

Original: Generalized Cross Entropy (GCE) (Zhang & Sabuncu, 2018) is a generalization of Cross Entropy (CE) and Mean Absolute Error (MAE).
Revised: Generalized Cross Entropy (GCE) (Zhang & Sabuncu, 2018) is a generalization of Cross Entropy (CE) and Mean Absolute Error (MAE).

Original: To exploit the benefits of both the noise-robustness provided by MAE and the implicit weighting scheme of CCE, they proposed using the negative Box-Cox transformation (Box & Cox, 1964) as a loss function:
Revised: To exploit the benefits of both the noise-robustness provided by MAE and the implicit weighting scheme of CCE, they proposed using the negative Box-Cox transformation (Box & Cox, 1964) as a loss function:

Original: [Equation]
Revised: [Equation]

Original: where q ∈ (0 , 1] .
Revised: where q ∈ (0 , 1] .

Original: It is equivalent to CE for lim q → 0 L q ( f ( x ) , e j ) , and becomes MAE/unhinged loss when q = 1 .
Revised: It is equivalent to CE for lim q → 0 L q ( f ( x ) , e j ) , and becomes MAE/unhinged loss when q = 1 .

Original: Label smoothing has been commonly used to improve the performance of deep learning models (Szegedy et al., 2016;
Revised: Label smoothing has been commonly used to improve the performance of deep learning models (Szegedy et al., 2016;

Original: Chorowski & Jaitly, 2016; Vaswani et al., 2017; Zoph et al., 2018; Real et al., 2019; Huang et al., 2019).
Revised: Chorowski & Jaitly, 2016; Vaswani et al., 2017; Zoph et al., 2018; Real et al., 2019; Huang et al., 2019).

Original: It serves as a regularizer (Li et al., 2020;
Revised: It serves as a regularizer (Li et al., 2020;

Original: Pereyra et al., 2017; Müller et al., 2019; Yuan et al., 2020; Zhang et al., 2021a) that improves the generalization and model calibration (predictive uncertainty) by using smoothed labels ˜ y
Revised: Pereyra et al., 2017; Müller et al., 2019; Yuan et al., 2020; Zhang et al., 2021a) that improves the generalization and model calibration (predictive uncertainty) by using smoothed labels ˜ y

Original: that mix the original one-hot labels y with an uniform mixture over all possible labels for α ∈ [0 , 1] :
Revised: that mix the original one-hot labels y with an uniform mixture over all possible labels for α ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: (9) where K is the number of classes.
Revised: (9) where K is the number of classes.

Original: Recently, Lukasik et al.
Revised: Recently, Lukasik et al.

Original: (2020) demonstrated label smoothing is also effective when learning with symmetric label noise.
Revised: (2020) demonstrated label smoothing is also effective when learning with symmetric label noise.

Original:  H OW DOES ACTION NOISE AFFECT IMITATION LEARNING ?
Revised:  H OW DOES ACTION NOISE AFFECT IMITATION LEARNING ?

Original: In this section, we first introduce state-independent action noise and state-dependent action noise .
Revised: In this section, we first introduce state-independent action noise and state-dependent action noise .

Original: Then, we briefly introduce widely used predictive uncertainty measures, i.e. calibration errors and 
Revised: Then, we briefly summarize loss estimation and uncertainty estimation methods.

Original: MC dropout.

Original: Next, we  study  the correlation betweenthe performance (i.e. average return), loss andpredictive uncertaintyof the imitation learning model, under multiplelevel of action noise.
Revised: Next, we provide a comprehensive study on the correlations between loss and uncertainty under multiple action noise.

Original: Finally, we introduce our uncertainty-aware robust learning paradigm using the observed correlation.
Revised: 3. A CTION NOISE

Original: State-independent action noise:
Revised: State-independent action noise:

Original: In traditional imitation learning, we consider the demonstrations
Revised: In traditional imitation learning, we consider the demonstrations

Original: D =
Revised: D =

Original: { s i , a i , s i +1 } Ni =1 are collected from some expert demonstrator.
Revised: { s i , a i , s i +1 } Ni =1 are collected from some expert demonstrator.

Original: In practice, we may recruit labors to give action labels for the recorded sequences of expert behaviors.
Revised: In practice, we may recruit annotators to give action labels for the recorded sequences of expert behaviors.

Original: The state-independent action noise occurs when a amateur labor randomly pick an action for the states he never seen before.
Revised: The state-independent action noise occurs when a amateur annotator randomly pick an action for unseen states.

Original: The noise generation process is quite similar to the class-conditional label noise, where a corrupted label is randomly flipping from other classes (Symemetric noise) or its neighbor (Pairflip noise) (Natarajan et al., 2013; Han et al., 2018).
Revised: The noise generation process is quite similar to the class-conditional label noise, where a corrupted label is randomly flipping from other classes (Symmetric noise) or its neighbor (Pairflip noise) (Natarajan et al., 2013; Han et al., 2018).

Original: Therefore, we build the noise modeling of stateindependent action noise by borrowing the setting of class-conditional label noise.
Revised: Therefore, we build the noise modeling of state-independent action noise by borrowing the setting of class-conditional label noise.

Original: The resulting noisy demonstrations is noted as ˜ D
Revised: The resulting noisy demonstrations is noted as ˜ D

Original: = { s i , a i , s i +1 } Ni =1 .
Revised: = { s i , a i , s i +1 } Ni =1 .

Original: State-dependent action noise: The state-dependent action noise occurs when an expert labor meets confusing states.
Revised: State-dependent action noise: The state-dependent action noise occurs when an expert annotator meets confusing states.

Original: In this situation, we assume the noisy action is dependent on the state feature and is independent on the expert action: P (˜ a i | a i , s i ) =
Revised: In this situation, we assume the noisy action is dependent on the state feature and is independent on the expert action: P (˜ a i | a i , s i ) =

Original: P (˜ a i | s i ) .
Revised: P (˜ a i | s i ) .

Original: Figure 2 shows a graphical model for state-dependent action noise .
Revised: Figure 2 shows a graphical model for state-dependent action noise .

Original: We simulate thegeneration of state-dependent action noise following the instance-dependent label noise setting(Xia et al., 2020). The detailed generation process are summarized in the Appendix B.
Revised: We generate the state-dependent action noise following the instance-dependent label noise setting in the Appendix B.

Original: 3. C ORRELATION BETWEEN LOSS AND PREDICTIVE UNCERTAINTY UNDER ACTION NOISE
Revised: 3. C ORRELATION BETWEEN LOSS AND PREDICTIVE UNCERTAINTY UNDER ACTION NOISE

Original: In this paper, we provide a comprehensive study about the correlation between the loss, uncertainty estimation of Behavioral Cloning, online imitation learning (GAIL) and offline imitation learning (BCQ) models under diverse action noise in the demonstrations.
Revised: In this paper, we provide a comprehensive study about the correlation between the loss, uncertainty estimation of Behavioral Cloning, online imitation learning (GAIL) and offline imitation learning (BCQ) models under diverse action noise in the demonstrations.

Original: We consider two types of noisy demonstrations: the first type of demonstration is the demonstrations with state-independent action noise; and the second type of demonstrations with state-dependent action noise.
Revised: We consider two types of noisy demonstrations: the first type of demonstration is the demonstrations with state-independent action noise; and the second type of demonstrations with state-dependent action noise.

Original: To study the correlations, we pre-train an imitation learning model with the optimal demonstration, and then estimate the corresponding loss and uncertainty on multiple noisy demonstrations.
Revised: To study the correlations, we pre-train an imitation learning model with the optimal demonstration, and then estimate the corresponding loss and uncertainty on multiple noisy demonstrations.

Original: Loss estimation.
Revised: Loss estimation.

Original: For the Behavioral Cloning model, GAIL actor and BCQ generative model, we use the cross-entropy as the loss estimation method.
Revised: For the Behavioral Cloning model, GAIL actor and BCQ generative model, we use the cross-entropy as the loss estimation method.

Original: For the GAIL discriminator, we use the adversarial loss for the expert data for loss estimation.
Revised: For the GAIL discriminator, we use the adversarial loss for the expert data for loss estimation.

Original: Uncertainty estimation.
Revised: Uncertainty estimation.

Original: Uncertainty estimation has been widely used in machine learning applications, as a complement that reflect the degree of trust in the model predictions (Kotelevskii et al., 2022).
Revised: Uncertainty estimation has been widely used in machine learning applications, as a complement that reflect the degree of trust in the model predictions (Kotelevskii et al., 2022).

Original: Usually, the total uncertainty of a prediction comes from two types of uncertainty: aleatoric and epistemic (Der Kiureghian & Ditlevsen, 2009; Kendall & Gal, 2017).
Revised: Usually, the total uncertainty of a prediction comes from two types of uncertainty: aleatoric and epistemic (Der Kiureghian & Ditlevsen, 2009; Kendall & Gal, 2017).

Original: The aleatoric uncertainty, known as data uncertainty, reflects the noise and ambiguity in the data.
Revised: The aleatoric uncertainty, known as data uncertainty, reflects the noise and ambiguity in the data.

Original: The epistemic uncertainty, known as model uncertainty, is related to the lack of knowledge about model parameters.
Revised: The epistemic uncertainty, known as model uncertainty, is related to the lack of knowledge about model parameters.

Original: Quantifying both types of uncertainty is crucial for safe decision in practical applications (Filos et al., 2020).
Revised: Quantifying both types of uncertainty is crucial for safe decision in practical applications (Filos et al., 2020).

Original: One simple aleatoric uncertainty is called MaxProb, which uses maximum softmax probabilities of deep neural network as the uncertainty measure (Nguyen et al., 2015).
Revised: One simple aleatoric uncertainty is called MaxProb, which uses maximum softmax probabilities of deep neural network as the uncertainty measure (Nguyen et al., 2015).

Original: MC_dropout is based on Bayesian techniques, capturing both types of uncertainty and is known to be a more reliable uncertainty estimation method (Gal & Ghahramani, 2016).
Revised: MC_dropout is based on Bayesian techniques, capturing both types of uncertainty and is known to be a more reliable uncertainty estimation method (Gal & Ghahramani, 2016).

Original: Recently, a promissing direction of uncertainty estimation methods based on a single deterministic neural network has been developed (Lee et al., 2018;
Revised: Recently, a promissing direction of uncertainty estimation methods based on a single deterministic neural network has been developed (Lee et al., 2018;

Original: Liu et al., 2020; Kotelevskii et al., 2022).
Revised: Liu et al., 2020).

Original:  Kotelevskii et al.
Revised: More recently, Kotelevskii et al.

Original: (2022) proposes a nonparametric uncertainty quantification (NUQ) method for single deterministic neural network, which enables uncertainty disentanglement and is scalable to large dataset.
Revised: (2022) proposes a nonparametric uncertainty quantification (NUQ) method, which enables uncertainty to be disentangled into aleatoric and epistemic uncertainty, and is scalable to large dataset.

Original: Expected Calibration Error (ECE) (Naeini et al., 2015; Guo et al., 2017) is also widely used to measure the predictive uncertainty (model calibration) of a deep network.
Revised: Expected Calibration Error (ECE) (Naeini et al., 2015; Guo et al., 2017) is also widely used to measure the predictive uncertainty (model calibration) of a deep network.

Original: To approximate the calibration error in expectation, ECE discretizes the probability interval into a fixed number of bins, and assigns each predicted probability to the bin that encompasses it.
Revised: To approximate the calibration error in expectation, ECE discretizes the probability interval into a fixed number of bins, and assigns each predicted probability to the bin that encompasses it.

Original: The calibration error is the difference between the fraction of predictions in the bin that are correct (accuracy) and the mean of the probabilities in the bin (confidence).
Revised: The calibration error is the difference between the fraction of predictions in the bin that are correct (accuracy) and the mean of the probabilities in the bin (confidence).

Original: Intuitively, the accuracy estimates P ( Y =

Original: y

Original: | ˆ p = p ) , and the average confidence is a setting of p .

Original: ECE computes a weighted average of this error across bins: ECE = (cid:80) Bb =1 n b N | acc ( b ) − conf ( b ) | , where n b is the number of predictions in bin b , N is the total number of data points, and acc ( b ) and conf ( b ) are  the accuracy and confidence of bin b , respectively.
Revised: More variants of calibration errors are summarized in the Appendix G.

Original: Instead of using the uncertainty for OoD detection (Malinin & Gales, 2018), or down-weighting the
Revised: Instead of using the uncertainty for OoD detection (Malinin & Gales, 2018), or down-weighting the

Original: Bellman loss for Q function (Wu et al., 2021) in RL, we leverage uncertainty for robust imitation learning in a novel direction.
Revised: Bellman loss for Q function (Wu et al., 2021) in RL, we leverage uncertainty for robust imitation learning in a novel direction.

Original: Namely, we first study the correlation between loss and uncertainty; and then propose an robust imitation learning paradigm called uncertainty-aware sample selection with negative learning (USN) based on the correlations.
Revised: Namely, we first study the correlation between loss and uncertainty; and then propose an robust imitation learning paradigm called uncertainty-aware sample selection with negative learning (USN) based on the correlations.

Original: Our method USN is scalable to Behavioral
Revised: Our method USN is scalable to Behavioral

Original: Cloning, online imitation learning and offline imitation learning.
Revised: Cloning, online imitation learning and offline imitation learning.

Original: 3.1. A COMPREHENSIVE STUDY OF THE CORRELATIONS 
Revised: 3.2. A COMPREHENSIVE STUDY ON THE CORRELATIONS OF LOSS AND UNCERTAINTY

Original: Due to the space limit, we only provide samples of the correlations for BC, online IL and offline IL in this section.
Revised: Due to the space limit, we only provide samples of the correlations for BC, online IL and offline IL in this section.

Original: For more comprehensive results please check the Appendix C.
Revised: For more comprehensive results, please check the Appendix C.

Original: Behavioral Cloning.
Revised: Behavioral Cloning.

Original: Figure 3 shows that the loss has positive correlation with the ECE uncertainty as the noise rate increases.
Revised: Figure 3 shows that the loss has positive correlation with the ECE uncertainty as the noise rate increases.

Original: This results indicate that the loss and ECE uncertainty of BC model are good identifier of noisy actions.
Revised: This results indicate that the loss and ECE uncertainty of BC model are good identifier of noisy actions.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Online Imitation Learning.
Revised: Online Imitation Learning.

Original: Figure 5 and Figure 6 show that the loss has positive correlation with the ECE uncertainty as the noise rate increases.
Revised: Figure 5 and Figure 6 show that the loss has positive correlation with the ECE uncertainty as the noise rate increases.

Original: This results indicate that the loss and ECE uncertainty of GAIL actor and discriminator are good identifier of noisy actions
Revised: This results indicate that the loss and ECE uncertainty of GAIL actor and discriminator are good identifier of noisy actions.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Offline Imitation Learning.
Revised: Offline Imitation Learning.

Original: We train the BCQ (Algorithm 2, Appendix A) with an ICM module (Pathak et al., 2017) for generating intrinsic rewards as the base offline imitation learning method.
Revised: We train the BCQ (Algorithm 2, Appendix A) with an ICM module (Pathak et al., 2017) for generating intrinsic rewards as the base offline imitation learning method.

Original: BCQ starts by sampling a mini batch of transitions from the input demonstration data D .
Revised: BCQ starts by sampling a mini batch of transitions from the input demonstration data D .

Original: Then, we use the generative model G to eliminate actions (step 5) for updating Q network (step 6).
Revised: Then, we use the generative model G to eliminate actions (step 5) for updating Q network (step 6).

Original: The generative model G is trained using a cross-entropy loss in step 7.
Revised: The generative model G is trained using a cross-entropy loss in step 7.

Original: Figure 4 shows that the loss of  generative model Gin both BCQ and BCQ-GCE goes up first and then drops as the noise rate increases.
Revised: Figure 4 shows that the loss of the generative model G goes up first and then drops as the noise rate increases.

Original: Interestingly, the ECE show a negative correlation with the dynamics of G ’s loss. 
Revised: Interestingly, the ECE uncertainty drops at first and then increase as the noise rate increases, which shows a negative correlation with the dynamics of G ’s loss. Analysis.

Revised: From Figure 3, Figure 4, Figure 5, Figure 6 and more results in the Appendix C, we can conclude that the loss estimation can be used as a criteria for detecting noisy actions using the

Original: The predictive uncertainty, e.g. the ECE drop at first and then increase as  the noise rate increases.
Revised: BC model, BCQ’s generative model, GAIL actor and discriminator. Specifically, the ‘large-loss’ samples have high probability to contain noisy actions. For the BC model, GAIL actor and GAIL discriminator, the uncertainty estimation ECE can also be used as a good criterion for detecting noisy actions. However for the offline imitation learning method BCQ, ECE does not show a clear positive correlation with the increase of action noise.

Original: These interesting observations motivates us to design a robust IL method that uses the negative correlation between G ’s loss and  the predictive uncertainty.
Revised: Instead, the dynamics of ECE in BCQ’s generative model has a negative correlation to the loss. In this paper, we focus on proposing a general method that detects noisy actions with high probability, and leverage the selected data to improve the robustness of BC, online IL and offline IL against diverse action noise in the demonstrations. To this end, we propose a general paradigm called Uncertainty-aware Sample-selection with Negative learning (USN), for robust training IL models against action noise. USN selects large-loss samples using the uncertainty estimation, e.g. ECE as a threshold that decides the number of sample selection.

Original:  M ETHOD
Revised:  M ETHOD

Original: Based on the above observations, we propose a novel method called Uncertainty-aware sampleselection with Soft Negative learning (USN).

Original: We summarize its full procedure in Algorithm 1.
Revised: Algorithm 1 summarizes the full procedure of USN.

Original: 1: Input: A mini-batch M of B transitions ( s, a, s ′ ) from D , negative learning  loss weight as λ neg , model parameter ω . 2: Estimate predictive uncertainty, e g. ECE, and regards it as the sample-selection threshold τ u .3:
Revised: Our main goal is to develop a general robust IL paradigm against diverse type of action noise. To achieve this goal, we design USN to composite of two main steps: uncertainty-aware sample-selection and negative learning for loss correction .

Original: Sample the large-loss batch ˜ M by selecting the large-loss samples with a length of: # neg = (1 − τ u ) ∗ B . 4: Generate complementary actions for ˜ M : ¯ a = Randomly select from { 1 , .., |A|}\{ a } , resulting a complementary batch ¯ M for negative learning.
Revised: Uncertainty-aware sample-selection aims to select samples that contains noisy actions with high probability.

Original: Given a mini-batch of demonstration data, we first employ a basic IL method, for positive learning on the full-batch data.
Revised: Given a mini-batch of demonstration data with a size of B , we first employ a basic IL method with parameter ω , for positive learning on the full-batch data.

Original: For Behavioral Cloning and BC initialized GAIL, we choose the BC as the basic IL model.
Revised: Then in second step, we estimate the uncertainty, e.g. on the batch data and regards it as a threshold τ u for selecting large-loss samples.

Original: For offline imitation learning, we obtain the basic IL model - BCQ-GCE, by applying GCE on the generative model: L pos =
Revised: Next, we sample the large-loss batch ˜ M by selecting the large-loss samples with a length of: # neg =

Original: (1 − G ( a | s ) q ) q .
Revised: (1 − τ u ) ∗ B .

Original: Performing BCQ-GCE alone can be robust to a sufficient amount of action noise; however, it is weak when noise rate is high.

Original: To address this challenge, we propose to employ soft negative learning on the ‘large-loss’ samples. Without using the gradient ascent technique used in previous work (Han et al., 2020), our method uses negative learning to reduce the negative effects of high-level action noise, and thus boost the imitation learning performance across different levels of action noise. To achieve this goal, we estimate the predictive uncertainty (e.g., ECE) of the positive learning model. The small-loss criterion is widely used to select samples that can be regarded as clean samples with high probability.
Revised: This procedure is called uncertainty-aware sample-selection .

Original: Previous sample selection methods (Hafner et al., 2018; Han et al., 2020; Xia et al., 2021) usually assume that the noise rate is known, and design the sample selection threshold using the noise rate.
Revised: Previous sample selection methods (Hafner et al., 2018; Han et al., 2020; Xia et al., 2021) usually assume that the noise rate is known, and design the sample selection threshold using the noise rate.

Original: In contrast, we define the sample selection threshold τ u using theestimated calibration errors (predictive uncertainty measures).
Revised: In contrast, we define the sample selection threshold τ u using the uncertainty estimation (e.g. ECE).

Original: Since the calibration errors adaptively changes during training, the sample selection threshold is automatically updated and is adaptive to different noise rates.
Revised: Since the uncertainty estimation dynamically changes during training, the sample selection threshold is automatically updated and is adaptive to different noise rates.

Original: We can maintain less negative samples with small noise rates and more negative learning when the noise rate is high, since the uncertainty measures decreases as the noise rate increases. In this way, our method can maximally reduces the negative effects of action noise using the adaptive threshold, without requiring any prior knowledge about the noise model. 
Revised: In this way, our method can maximally reduces the negative effects of action noise using the adaptive threshold, without requiring any prior knowledge about the noise model. Negative learning for loss correction. Positive learning with full-batch data with noisy actions will results in bias in the loss training, and misguides the policy to choose the wrong actions.

Original: With the adaptive threshold, we select the remaining large-loss samples for Soft Negative Learning.
Revised: To correct the loss bias, we propose to leverage the selected large-loss sample for negative learning.

Original: Specifically, we employ the following negative log likelihood (NLL) loss for negative  learning on the ‘large-loss’ samples with label smoothing:
Revised: Intuitively, the selected large-loss batch ˜ M contains noisy actions with high probability. Its complementary set has more chances to contain true actions. Therefore, we generate complementary actions for ˜ M by randomly selecting ¯ a from { 1 , .., |A|}\{ a } , resulting a complementary bath ¯ M for negative learning. Negative learning on the complementary batch of the selected large-loss samples will correct the loss bias from action noise, and therefore improve the imitation learning performance.

Original: [Equation]
Revised: To further boost the performance, we implement negative learning with label smoothing, resulting Soft Negative learning . Specifically, we employ the following negative log likelihood (NLL) loss for negative learning on the ‘large-loss’ samples with label smoothing: L neg = NLL (cid:16) 1 − G ( a | s ) , (1 − α ) · ¯ a + α |A| · 1 (cid:17) , where ¯ a is the complementary action of the ‘large-loss’ samples, and α is the smooth rate.  E XPERIMENTS AND R ESULTS In this section, we demonstrate how does USN improves the robustness of BC, onfline IL and offline IL when learning with state-independent action noise and state-dependent action noise .

Original: where ¯ a is the complementary action of the ‘large-loss’ samples, and α is the smooth rate.
Revised: For Behavioral Cloning and BC initialized GAIL, we choose the BC as the basic IL model. For offline imitation learning, we obtain the basic IL model - BCQ-GCE, by applying GCE on the generative model: L pos = (1 − G ( a | s ) q ) q .

Original: Algorithm 3 in the Appendix A presents an instance of our proposed robust IL method, BCQ-USN.
Revised: Algorithm 3 in the Appendix A presents an instance of our proposed robust IL method, BCQ-USN.

Original: BCQ-USN improves BCQ by updating the generative model G using our USN method (Algorithm 1) in step 7.
Revised: BCQ-USN improves BCQ by updating the generative model G using our USN method (Algorithm 1) in step 7.

Original: USN improves the ability of generative model G for eliminating noisy actions for policy learning.
Revised: USN improves the ability of generative model G for eliminating noisy actions for policy learning.

Original: Thus, BCQ-USN is more robust than BCQ and BCQ-GCE, achieving better imitation learning performance across diverse levels of action noise.
Revised: Thus, BCQ-USN is more robust than BCQ and BCQ-GCE, achieving better imitation learning performance across diverse levels of action noise.

Original: E XPERIMENTS AND R ESULTS

Original: In this section, we evaluate our methods when learning with state-independent action noise and state-dependent action noise .

Original: We present empirical evaluation of BCQ-USN in a variety of settings.

Original: We start with description of our experimental setup, datasets and baseline algorithms.

Original: 5. E XPERIMENTAL S ETUP .
Revised: 5. E XPERIMENTAL SETUP

Original: We construct experiments on the widely used discrete control tasks and Atari games from OpenAI
Revised: We construct experiments on the widely used discrete control tasks and Atari games from OpenAI

Original: Gym environments.
Revised: Gym environments.

Original: To evaluate the effectiveness of our methods, we pre-train expert agent and then generate demonstrations with state-dependent noisy actions using the pre-trained agent.
Revised: To evaluate the effectiveness of our methods, we pre-train expert agent and then generate demonstrations with state-dependent noisy actions using the pre-trained agent.

Original: For each setting, we have two demonstrations, D for normal demonstration and ˜ D for demonstration with state-dependent noisy actions only, a.k.a.
Revised: For each setting, we have two demonstrations, D for normal demonstration and ˜ D for demonstration with state-dependent noisy actions only, a.k.a.

Original: the noisy demonstration .
Revised: the noisy demonstration .

Original: Note that we use ground-truth rewards only to train the expert agent, and we discard the rewards afterward.
Revised: Note that we use ground-truth rewards only to train the expert agent, and we discard the rewards afterward.

Original: In setting 1 , we use the classic control tasks from OpenAI Gym - Acrobot-v1 and LunarLander-v2.
Revised: In setting 1 , we use the classic control tasks from OpenAI Gym - Acrobot-v1 and LunarLander-v2.

Original: We pre-train a DQN policy as the expert agent, and generate demonstrations with 50K steps.
Revised: We pre-train a DQN policy as the expert agent, and generate demonstrations with 50K steps.

Original: This setting is used to evaluate how our method USN improves Behavioral Cloning and offline imitation algorithms- BCQ and CQL under state-independent action noise and state-dependent action noise.
Revised: This setting is used to evaluate how our method USN improves Behavioral Cloning and offline imitation algorithms under state-independent action noise and state-dependent action noise.

Original: In setting 2 , we use widely used Atari games.
Revised: In setting 2 , we use widely used Atari games.

Revised: For online IL, we generate one full-episode demonstration using a PPO agent.

Original: We first pre-train a QRDQN policy as the expert agent.
Revised: For offline IL, we first pre-train a QRDQN policy as the expert agent.

Original: Then, we generate a 50K-step demonstration dataset.
Revised: Then, we generate a 50K-step demonstration dataset.

Original: This setting is used for evaluation the robustness of our method USN on online imitation learning algorithm - GAIL, and offline imitation learning algorithms - BCQ and CQL against diverse action noise.
Revised: This setting is used for evaluation the robustness of our method USN on online imitation learning algorithm - GAIL, and offline imitation learning algorithms - BCQ and CQL against diverse action noise.

Original: Baseline algorithms: For Behavioral Cloning, we compare USN with BC and GCE.
Revised: Baseline algorithms: For Behavioral Cloning, we compare USN with BC and GCE.

Original: For online imitation learning, we compare our method GAIL-USN to SOTA robust imitation learning algorithms:
Revised: For online imitation learning, we compare our method GAIL-USN to SOTA robust imitation learning algorithms:

Original: RIL_CO, SAIL amd the original GAIL.
Revised: RIL_CO, SAIL amd the original GAIL.

Original: For offline imitation learning experiments, we compare
Revised: For offline imitation learning experiments, we compare

Original: BCQ-USN with BCQ in Algorithm 2 of Appendix A, which is trained with the rewards from an
Revised: BCQ-USN with BCQ in Algorithm 2 of Appendix A, which is trained with the rewards from an

Original: Intrinsic Curiosity Module (ICM).
Revised: Intrinsic Curiosity Module (ICM).

Original: (b) BCQ-GCE, which is a robust BCQ by training generative model with GCE loss.
Revised: (b) BCQ-GCE, which is a robust BCQ by training generative model with GCE loss.

Original: and (c) UWAC, a SOTA robust offline imitation learning that also uses uncertainty estimation.
Revised: and (c) UWAC, a SOTA robust offline imitation learning that also uses uncertainty estimation.

Original: We set the negative learning loss weight λ neg = 0 .
Revised: We set the negative learning loss weight λ neg = 0 .

Original: 01 for the classic control tasks and λ neg = 1 . 0 for Atari games.
Revised: 01 for the classic control tasks and λ neg = 1 . 0 for Atari games.

Original: 5. B EHAVIORAL CLONING WITH NOISY DEMONSTRATIONS
Revised: 5. B EHAVIORAL CLONING WITH NOISY DEMONSTRATIONS

Original: In the first experiment, we evaluate the robustness of our method USN on the classic control tasksAcrobot-v1 and LunarLander-v2.
Revised: In the first experiment, we evaluate the robustness of our method USN on the classic control tasksAcrobot-v1 and LunarLander-v2.

Original: Figure 7 and Figure 8 show the performance of BC, GCE and our
Revised: Figure 7 and Figure 8 show the performance of BC, GCE and our

Original: USN under multiple state-independent action noise and state-dependent action noise, respectively.
Revised: USN under multiple state-independent action noise and state-dependent action noise, respectively.

Original: In both types of action noise in the demonstrations, our method USN achieves better performance than other baselines.
Revised: In both types of action noise in the demonstrations, our method USN achieves better performance than other baselines.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 5. O NLINE IMITATION LEARNING WITH NOISY DEMONSTRATIONS
Revised: 5. O NLINE IMITATION LEARNING WITH NOISY DEMONSTRATIONS

Original: Then, we show how our method USN can improves the robustness of online imitation learning against action noise.
Revised: Then, we show how our method USN can improves the robustness of online imitation learning against action noise.

Original: Since most of the previous robust online imitation learning algorithms (Tangkaratt et al., 2020; Wang et al., 2021) are based on GAIL, we also choose GAIL as our base method.
Revised: Since most of the previous robust online imitation learning algorithms (Tangkaratt et al., 2020; Wang et al., 2021) are based on GAIL, we also choose GAIL as our base method.

Original: However, the original GAIL can not achieve good performance in high-dimensional environment like Atari games (Brown et al., 2019a;b).
Revised: However, the original GAIL can not achieve good performance in high-dimensional environment like Atari games (Brown et al., 2019a;b).

Original: Fortunately, we found that using Behavioral Cloning as an initialization for the actor, GAIL is able to achieve good performance on some Atari games, e.g. Seaquest, Q*bert and Hero.
Revised: Fortunately, we found that using Behavioral Cloning as an initialization for the actor, GAIL is able to achieve good performance on some Atari games, e.g. Seaquest, Q*bert and Hero with only one full-episode demonstration.

Original: We apply our method USN on the BC initialization and compares to SOTA robust imitation learning algorithms, RIL_CO and SAIL with soft weights.
Revised: We apply our method USN on the BC initialization and compares to SOTA robust imitation learning algorithms, RIL_CO and SAIL with soft weights.

Original: The results under multiple state-dependent action noise for three Atari games are shown in Figure 9.
Revised: The results under multiple state-dependent action noise for three Atari games are shown in Figure 9.

Original: SAIL totally fails in allof the Atari games under action noise in the demonstrations.
Revised: SAIL totally fails in all the Atari games under action noise in the demonstrations.

Original: In the Seaquest game, both our method GAIL-USN and RIL_CO achieves the best performance; while RIL_CO fails in the other two games.
Revised: In the Seaquest game, both our method GAIL-USN and RIL_CO achieves the best performance; while RIL_CO fails in the other two games.

Original: Our method GAIL-USN outperforms allof the baselines in the other two Atari games.
Revised: Our method GAIL-USN outperforms all the baselines in the other two Atari games.

Original: More details are summarized in Appendix E.
Revised: More details are summarized in Appendix E.

Original: [Figure]
Revised: [Figure]

Original: 5. O FFLINE IMITATION LEARNING WITH NOISY DEMONSTRATIONS
Revised: 5. O FFLINE IMITATION LEARNING WITH NOISY DEMONSTRATIONS

Original: Next, we evaluate the robustness of our method USN on offline imitation learning with demonstrations under both state-independent action noise and state-dependent action noise.
Revised: Next, we evaluate the robustness of our method USN on offline imitation learning with demonstrations under both state-independent action noise and state-dependent action noise.

Original: Robustness to state-independent action noise .
Revised: Robustness to state-independent action noise .

Original: In Figure 10(a)(b), we evaluate our method on two other types of state-independent action noise : symmetric noise and pairflip noise.
Revised: In Figure 10(a)(b), we evaluate our method on two other types of state-independent action noise : symmetric noise and pairflip noise.

Original: These two types of noise have been widely studied in the literature of learning with noisy labels.
Revised: These two types of noise have been widely studied in the literature of learning with noisy labels.

Original: We can see that the performance of BCQ drops when noise rate is larger than 0.45 in both Symmetric and Pairflip action noise.
Revised: We can see that the performance of BCQ drops when noise rate is larger than 0.45 in both Symmetric and Pairflip action noise.

Original: Our method BCQ-USN outperforms BCQ-GCE in both cases across diverse noise level.
Revised: Our method BCQ-USN outperforms BCQ-GCE in both cases across diverse noise level.

Revised: BCQ-USN-PC denotes our method using PC loss Ishida et al.

Revised: (2017) as a negative learning loss.

Revised: The results show that our method is scalable to multiple choices of negative learning loss.

Original: [Figure]
Revised: [Figure]

Original: Robustness to state-dependent action noise .
Revised: Robustness to state-dependent action noise .

Original: Figure 10(c) shows the average return curves of BCQ,
Revised: Figure 10(c) shows the average return curves of BCQ,

Original: BCQ-GCE and our BCQ-USN on LunarLander-v2 across diverse levels of state-dependent action noise .
Revised: BCQ-GCE and our BCQ-USN on LunarLander-v2 across diverse levels of state-dependent action noise .

Original: For both types of action noise, BCQ-USN outperforms BCQ-GCE and BCQ across diverse noise levels.
Revised: For both types of action noise, BCQ-USN outperforms BCQ-GCE and BCQ across diverse noise levels.

Original: In Figure 27 of the Appendix, we show bar charts comparison of BCQ, BCQ-GCE and BCQ-USN for four Atari games.
Revised: In Figure 34 of the Appendix, we show bar charts comparison of BCQ, BCQ-GCE and BCQ-USN for four Atari games.

Original: Our method, BCQ-USN consistently outperform BCQ-GCE and BCQ when the noise rate is 0.45.
Revised: Our method, BCQ-USN consistently outperform BCQ-GCE and BCQ when the noise rate is 0.45.

Original: The performance improvements of BCQ-USN compared to other methods are usually significant.
Revised: The performance improvements of BCQ-USN compared to other methods are usually significant.

Original: For this experiment, we compare our method with an related baseline - UWAC (Wu et al., 2021).
Revised: For this experiment, we compare our method with an related baseline - UWAC (Wu et al., 2021).

Original: In addition, we introduce a variants of our method called BCQ-USN-MC_dropout, which uses the MC dropout for uncertainty estimation in USN.
Revised: In addition, we introduce a variants of our method called BCQ-USN-MC_dropout, which uses the MC dropout for uncertainty estimation in USN.

Original: The quantitative results are summarized in Table 1.
Revised: The quantitative results are summarized in Table 1.

Original: The results show that UWAC only outperforms BCQ-GCE on the Assault and AirRaid games, while performs much worse than BCQ-GCE on two other games.
Revised: The results show that UWAC only outperforms BCQ-GCE on the Assault and AirRaid games, while performs much worse than BCQ-GCE on two other games.

Original: Our variant BCQ-USN-MC_dropout 
Revised: Our variant BCQ-USN-MC_dropout outperforms UWAC and BCQ-GCE on two games. This demonstrates that our method USN can

Original: [Table]
Revised: [Table]

Original: outperforms UWAC and BCQ-GCE on two games.

Original: This demonstrates that our method USN can use MC dropout as an uncertainty estimator, while overall it performs worse than the variant using calibration errors.
Revised: use MC dropout as an uncertainty estimator, while overall it performs worse than the variant using calibration errors.

Original: Overall, BCQ-USN outperform BCQ-GCE by 14% across the four Atari games.
Revised: Overall, BCQ-USN outperform BCQ-GCE by 14% across the four Atari games.

Original: Finally, we generalize USN to Q-learning.
Revised: Finally, we generalize USN to Q-learning.

Original: In the above experiments, USN is applied on top of
Revised: In the above experiments, USN is applied on top of

Original: Behavioral Cloning.
Revised: Behavioral Cloning.

Original: To show the scalability of our method, we apply our USN method to a more general Q-learning based IL method without a generative model.
Revised: To show the scalability of our method, we apply our USN method to a more general Q-learning based IL method without a generative model.

Original: We chose the conservative Qlearning (CQL) (Kumar et al., 2020) as our base method.
Revised: We chose the conservative Qlearning (CQL) (Kumar et al., 2020) as our base method.

Original: In Appendix F, we implement a robust CQL algorithm, CQL-GCE, by training the Q network with the original conservative loss and an additional
Revised: In Appendix F, we implement a robust CQL algorithm, CQL-GCE, by training the Q network with the original conservative loss and an additional

Original: GCE loss.
Revised: GCE loss.

Original: Similar to BCQ-USN, we apply USN on the positive learning baseline - CQL-GCE.
Revised: Similar to BCQ-USN, we apply USN on the positive learning baseline - CQL-GCE.

Original: Specifically, we estimate calibration error metrics for the Q network, and then use the calibration error to determine the large-loss samples for negative learning.
Revised: Specifically, we estimate calibration error metrics for the Q network, and then use the calibration error to determine the large-loss samples for negative learning.

Original: Empirically, we evaluate the resulting algorithm CQL-USN on the Q*bert Atari game with state-dependent action noise , symmetric action noise and pairflip action noise across different noise rates.
Revised: Empirically, we evaluate the resulting algorithm CQL-USN on the Q*bert Atari game with state-dependent action noise , symmetric action noise and pairflip action noise across different noise rates.

Original: Figure 11 shows the comparative results of
Revised: Figure 11 shows the comparative results of

Original: CQL, CQL-GCE and CQL-USN on the Q*bert game.
Revised: CQL, CQL-GCE and CQL-USN on the Q*bert game.

Original: Impressively, CQL-USN usually outperform
Revised: Impressively, CQL-USN usually outperform

Original: CQL and CQL-GCE by a large margin.
Revised: CQL and CQL-GCE by a large margin.

Original: Therefore, our USN method has good scalability to general
Revised: Therefore, our USN method has good scalability to general

Original: Q-learning based IL methods, without requiring the generative model.
Revised: Q-learning based IL methods, without requiring the generative model.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have provided a comprehensive study of correlations between loss estimation and uncertainty estimation of imitation learning models.
Revised: In this paper, we have provided a comprehensive study of correlations between loss estimation and uncertainty estimation of imitation learning models.

Original: Then, based on the correlations, we propose an novel paradigm for robust imitation learning with action noise.
Revised: Then, based on the correlations, we propose an novel paradigm for robust imitation learning with action noise.

Original: Our method, uncertainty-aware sample selection for negative learning (USN), scales to Behavioral Cloning, online imitation learning and offline imitation learning with state-independent action noise and state-dependent action noise .
Revised: Our method, uncertainty-aware sample selection for negative learning (USN), scales to Behavioral Cloning, online imitation learning and offline imitation learning with state-independent action noise and state-dependent action noise .

Original: Moreover, our USN paradigm can use diverse predictive uncertainty for robust imitation learning, showing the nice adapativity and scalability of our method.
Revised: Moreover, our USN paradigm can use diverse predictive uncertainty for robust imitation learning, showing the nice adapativity and scalability of our method.

Original: In addition, the ablation studies in the
Revised: In addition, the ablation studies in the

Original: Appendix H also demonstrate that our method is not sensitive to the adaptation of the hyperparameters.
Revised: Appendix H also demonstrate that our method is not sensitive to the adaptation of the hyperparameters.

Original: In the near future, we are interested in extending our method to continuous control tasks by studying the correlations between loss and regression uncertainty estimations of IL models.
Revised: In the near future, we are interested in extending our method to continuous control tasks by studying the correlations between loss and regression uncertainty estimations of IL models.

Original: Also, it is worthy to investigate the application of our method onmore general actor-critic learning.
Revised: Also, it is worthy to investigate the application of our method on actor-critic learning and real-world tasks.

Original: For example, the large-uncertainty samples can also be used for boost the exploration of policy

Original: A A LGORITHMS
Revised: A A LGORITHMS

Original: Algorithm 2 and Algorithm 3 shows the procedure of BCQ and BCQ-USN for offline imitation learning.
Revised: Algorithm 2 and Algorithm 3 shows the procedure of BCQ and BCQ-USN for offline imitation learning.

Original: We implement our methods using the Tianshou platform (Weng et al., 2021) for offline imitation learning, and train allof the algorithms with the default hyperparameters of BCQ.[Table] [Table] [Table]
Revised: We implement our methods using the Tianshou platform (Weng et al., 2021) for offline imitation learning, and train all the algorithms with the default hyperparameters of BCQ.

Original: [Table] [Table] [Table] [Table]
Revised: Algorithm 2 BCQ with ICM for offline imitation learning.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] Algorithm 3 BCQ-USNfor robust offline imitation learningagainst action noise mini-batch M transitions ( s from
Revised: Algorithm 3 BCQ-USN robust offline imitation learning action mini

Original: Train an ICM module to generate intrinsic reward: r =
Revised: Train an ICM module to generate intrinsic reward: r =

Original: ICM( s, a, s′ ) .
Revised: ICM( s, a, s

Original: 6: a ′ ← arg max a ′ | G ( a ′ | s ′ )
Revised: 6: a ′ ← arg max a ′ | G ( a ′ | s ′ )

Original: / max ˆ aG (ˆ a | s ′ ) >
Revised: / max ˆ aG (ˆ a | s ′ ) >

Original: τ Q θ ( s ′ , a ′ ) .
Revised: τ Q θ ( s ′ , a ′ ) .

Original: 7: θ ← arg min θ (cid:80) ( s,a,s ′ )
Revised: 7: θ ← arg min θ (cid:80) ( s,a,s ′ )

Original: ∈
Revised: ∈

Original: M k κ ( r + γQ θ ′ ( s ′ , a ′ )
Revised: M k κ ( r + γQ θ ′ ( s ′ , a ′ )

Original: − Q θ ( s, a ))
Revised: − Q θ ( s, a ))

Original: 8: ω ← USN( M ) //
Revised: 8: ω ← USN( M ) //

Original: Update G using USN approach in Algorithm 1.
Revised: Update G using USN approach in Algorithm 1.

Original: If t mod target_update_rate = 0 : θ ′ ← θ .
Revised: If t mod target_update_rate = 0 : θ ′ ← θ .

Original: 10: end for
Revised: 10: end for

Original: B S TATE -D EPENDENT A CTION N OISE G ENERATION Algorithm
Revised: B S TATE -D EPENDENT A CTION N OISE G ENERATION Algorithm

Original:  end
Revised: entries from the expert actions Let q n be probability of getting a wrong = q q end

Revised: We simulate the generation of state-dependent action noise by following the instance-dependent label noise setting (Xia et al., 2020).

Original: Note that it is more realistic that different states have different flip rates.
Revised: Note that it is more realistic that different states have different flip rates.

Original: Without constraining different states to have a same flip rate, it is more challenging to model the label noise and train robust classifiers.
Revised: Without constraining different states to have a same flip rate, it is more challenging to model the label noise and train robust classifiers.

Original: In Step 1, in order to control the global flip rate as τ but without constraining allof the states to have a same flip rate, we sample their flip rates from a truncated normal distribution N ( τ, 0 .
Revised: In Step 1, in order to control the global flip rate as τ but without constraining all the states to have a same flip rate, we sample their flip rates from a truncated normal distribution N ( τ, 0 .

Original: Specifically, this distribution limits the flip rates of states in the range [0, 1].
Revised: Specifically, this distribution limits the flip rates of states in the range [0, 1].

Original: Their mean and standard deviation are equal to the mean τ and the standard deviation 0.1 of the selected truncated normal distribution respectively.
Revised: Their mean and standard deviation are equal to the mean τ and the standard deviation 0.1 of the selected truncated normal distribution respectively.

Original: In Step 2, we sample parameters w 1 , w 2 , · · · , w c from the standard normal distribution for generating state-dependent action noise .
Revised: In Step 2, we sample parameters w 1 , w 2 , · · · , w c from the standard normal distribution for generating state-dependent action noise .

Original: The dimensionality of each parameter is d × c , where d denotes the dimensionality of the state.
Revised: The dimensionality of each parameter is d × c , where d denotes the dimensionality of the state.

Original: Learning these parameters is critical to model state-dependent action noise .
Revised: Learning these parameters is critical to model state-dependent action noise .

Original: However, it is hard to identify these parameters without any assumption.
Revised: However, it is hard to identify these parameters without any assumption.

Original: Note that an state with expert action a will be flipped only according to the a − th row of the transition matrix.
Revised: Note that an state with expert action a will be flipped only according to the a − th row of the transition matrix.

Original: Thus, in Steps 4 to 7, we only use the a i − th row of the state-dependent transition matrix for the state s i .
Revised: Thus, in Steps 4 to 7, we only use the a i − th row of the state-dependent transition matrix for the state s i .

Original: Specifically, Steps 5 and 7 are to ensure the diagonal entry of the a i − th row is 1 − q i .
Revised: Specifically, Steps 5 and 7 are to ensure the diagonal entry of the a i − th row is 1 − q i .

Original: Step 6 is to ensure that the sum of the off-diagonal entries is q i .
Revised: Step 6 is to ensure that the sum of the off-diagonal entries is q i .

Original: C T HE CORRELATION BETWEEN LOSS , UNCERTAINTY AND NOISE RATES OR OPTIMALITY OF DEMONSTRATIONS IN THE A TARI DOMAIN Results
Revised: C T HE CORRELATION BETWEEN LOSS , UNCERTAINTY AND NOISE RATES OR OPTIMALITY OF DEMONSTRATIONS IN THE A TARI DOMAIN C. R ESULTS OF BC ON CLASSIC CONTROL TASKS .

Original: of BC on classic control tasks.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] C. R ESULTS OF GAIL

Original: Reusults of BCQ_ICM on Atari games.
Revised: ON A TARI GAMES

Original: [Figure]
Revised: C.2.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: R ESULTS OF GAIL ON A SINGLE A TARI GAME .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Results of GAIL on a single Atari game.

Original: For the noisy demonstration with state-dependent action noise, we study correlation between loss, uncertainty estimation of GAIL and noise rate of the demonstrations.
Revised: For the noisy demonstration with state-dependent action noise, we study correlation between loss, uncertainty estimation of GAIL and noise rate of the demonstrations.

Original: The results of GAIL actor and discriminator in Q*bert game are shown in Figure 22 and Figure 23.
Revised: The results of GAIL actor and discriminator in Q*bert game are shown in Figure 15 and Figure 16.

Original: As expected, with the increasing of noise rates in the demonstrations, the loss estimation of the GAIL actor increases accordingly.
Revised: As expected, with the increasing of noise rates in the demonstrations, the loss estimation of the GAIL actor increases accordingly.

Original: For the uncertainty estimations, ECE and MCE show the positive correlation with the loss estimation and the increase of noise rate.
Revised: For the uncertainty estimations, ECE and MCE show the positive correlation with the loss estimation and the increase of noise rate.

Original: For the GAIL discriminator, the loss estimation and three uncertainty estimations (i.e. ECE, MCE and NUQ) hold the postive correlation with the increase of noise rate in the demonstrations.
Revised: For the GAIL discriminator, the loss estimation and three uncertainty estimations (i.e. ECE, MCE and NUQ) hold the positive correlation with the increase of noise rate in the demonstrations.

Original: The Bald and MC_dropout uncertainty estimations also show light positive correlations.
Revised: The Bald and MC_dropout uncertainty estimations also show light positive correlations.

Original: The above observations hold for many Atari games.
Revised: The above observations hold for many Atari games.

Original: The results can be found bellow.
Revised: The results can be found bellow.

Original: Therefore, the loss estimation, the uncertainty estimations - ECE and MCE, can be used as creteria for detecting noisy actions in the demonstrations using the GAIL actor.
Revised: Therefore, the loss estimation, the uncertainty estimations - ECE and MCE, can be used as criteria for detecting noisy actions in the demonstrations using the GAIL actor.

Original: Similarly, the loss estimation, the uncertainty estimationsECE, MCE, NUQ and even Bald and MC_dropout, can be used as cretia for detecting noisy actions in the demonstrations using the GAIL discriminator.
Revised: Similarly, the loss estimation, the uncertainty estimations - ECE, MCE, NUQ and even Bald and MC_dropout, can be used as criteria for detecting noisy actions in the demonstrations using the GAIL discriminator.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: For the noisy demonstration with suboptimal, we study correlation between loss, uncertainty estimation of GAIL and optimality of the demonstrations.
Revised: For the noisy demonstration with suboptimal demonstrations, we study correlation between loss, uncertainty estimation of GAIL and optimality of the demonstrations.

Original: The results of GAIL actor and discriminator in Q*bert game are shown in Figure 24 and Figure 25.
Revised: The results of GAIL actor and discriminator in Q*bert game are shown in Figure 17 and Figure 18.

Original: The lower load iter denotes worse optimality.
Revised: The lower load iter denotes worse optimality.

Original: The results show that with the decreasing of the optimality in the demonstrations, the loss estimation of the GAIL actor increases accordingly.
Revised: The results show that with the decreasing of the optimality in the demonstrations, the loss estimation of the GAIL actor increases accordingly.

Original: For the uncertainty estimations, ECE and MCE show the positive correlation with the loss estimation and the increase of noise rate.
Revised: For the uncertainty estimations, ECE and MCE show the positive correlation with the loss estimation and the increase of noise rate.

Original: Entropy, MaxProb and MC_dropout show light positive correlations.
Revised: Entropy, MaxProb and MC_dropout show light positive correlations.

Original: For the GAIL discriminator, the loss estimation and many uncertainty estimations (i.e. ECE, MCE, Bald, MC_dropout and NUQ) hold the postive correlation with the decrease of optimality in the demonstrations.
Revised: For the GAIL discriminator, the loss estimation and many uncertainty estimations (i.e. ECE, MCE, Bald, MC_dropout and NUQ) hold the positive correlation with the decrease of optimality in the demonstrations.

Original: The above observations hold for many Atari games.
Revised: The above observations hold for many Atari games.

Original: The results can be found bellow.
Revised: The results can be found bellow.

Original: Therefore, the loss estimation, the uncertainty estimations - ECE, MCE and even MC_dropout, can be used as creteria for detecting suboptimal data in the demonstrations using the GAIL actor.
Revised: Therefore, the loss estimation, the uncertainty estimations - ECE, MCE and even MC_dropout, can be used as creteria for detecting suboptimal data in the demonstrations using the GAIL actor.

Original: Similarly, the loss estimation, the uncertainty estimations - ECE, MCE, Bald, MC_dropout and NUQ, can be used as cretia for detecting suboptimal data in the demonstrations using the GAIL discriminator. 
Revised: Similarly, the loss estimation, the uncertainty estimations - ECE, MCE, Bald, MC_dropout and NUQ, can be used as criteria for detecting suboptimal data in the demonstrations using the GAIL discriminator. C.2.  GAIL

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: C.2.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: D A DDITIONAL RESULTS . D. C OMPARATIVE RESULTS ACROSS MULTIPLE NOISE RATES .
Revised: D A DDITIONAL RESULTS . D. C OMPARATIVE RESULTS ACROSS MULTIPLE NOISE RATES .

Original: Table 2 shows the comparative results of BCQ, BCQ-GCE and our BCQ-USN on LunarLander-v across different levels of state-dependent action noise .
Revised: Table 2 shows the comparative results of BCQ, BCQ-GCE and our BCQ-USN on LunarLander-v across different levels of state-dependent action noise .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: E O NLINE I MITATION L EARNING E. G ENERATIVE A DVERSARIAL
Revised: E O NLINE I MITATION L EARNING E. G ENERATIVE A DVERSARIAL

Original: [Table]
Revised: [Table]

Original: Generative adversarial imitation learning (GAIL) was proposed based on the general framework of Generative Adversarial Networks (GANs) Goodfellow et al.
Revised: Generative adversarial imitation learning (GAIL) was proposed based on the general framework of Generative Adversarial Networks (GANs) Goodfellow et al.

Original: GAIL regards the classical imitation learning problem as an occupancy measure matching between the expert policy and the agent policy.
Revised: GAIL regards the classical imitation learning problem as an occupancy measure matching between the expert policy and the agent policy.

Original: Specifically, GAIL uses a discriminator D ϕ to distinguish expert transitions from agent transitions, while the agent is to “fool” the discriminator into treating agent transitions as expert data.
Revised: Specifically, GAIL uses a discriminator D ϕ to distinguish expert transitions from agent transitions, while the agent is to “fool” the discriminator into treating agent transitions as expert data.

Original: Formally, the objective function of GAIL is written as
Revised: Formally, the objective function of GAIL is written as

Original: [Equation]
Revised: [Equation]

Original: where ρ θ and ρ E denote the occupancy measures of agent policies π θ and the expert π E , respectively.
Revised: where ρ θ and ρ E denote the occupancy measures of agent policies π θ and the expert π E , respectively.

Original: GAIL and its many variants (Peng et al., 2019; Li et al., 2017) has achieved great success in imitation learning in low-dimensional tasks.
Revised: GAIL and its many variants (Peng et al., 2019; Li et al., 2017) has achieved great success in imitation learning in low-dimensional tasks.

Original: In imitation learning with imperfect demonstrations / noisy demonstrations, many previous methods are also based the framework of GAIL.
Revised: In imitation learning with imperfect demonstrations / noisy demonstrations, many previous methods are also based the framework of GAIL.

Original: However, GAIL had been demonstrated that does not scale well to high-dimensional tasks like Atari games (Tucker et al., 2018; Brown et al., 2019a;b).
Revised: However, GAIL had been demonstrated that does not scale well to high-dimensional tasks like Atari games (Tucker et al., 2018; Brown et al., 2019a;b).

Original: To boost the performance of GAIL in high-dimensional task, we take a simple step by initializing the actor with behavior cloning.
Revised: To boost the performance of GAIL in high-dimensional task, we take a simple step by initializing the actor with behavior cloning.

Original: We found that GAIL is able to achieve demonstration-lelve performance on some games by using the BC initialization on the actor.
Revised: We found that GAIL is able to achieve demonstration-lelve performance on some games by using the BC initialization on the actor.

Original: Our implementation is based on the ‘gail_atari’ repository 1 .
Revised: Our implementation is based on the ‘gail_atari’ repository 1 .

Original: E. GAIL-USN
Revised: E. GAIL-USN

Original: To extend our USN paradigm to GAIL, we directly apply USN on the behavior cloning model in the initialization step.
Revised: To extend our USN paradigm to GAIL, we directly apply USN on the behavior cloning model in the initialization step.

Original: E. E XPERIMENTS AND R ESULTS
Revised: E. E XPERIMENTS AND R ESULTS

Original: Experiment setup.
Revised: Experiment setup.

Original: For online imitation learning, we consider learning from the noisy environments.
Revised: For online imitation learning, we consider learning from the noisy environments.

Original: Namely, besides to the noisy demonstration data, the data collected from the environment are full of action noise.
Revised: Namely, besides to the noisy demonstration data, the data collected from the environment are full of action noise.

Original: This happens in practice when the environments suffers from system error or the data collecting and transferring protocol under attack.
Revised: This happens in practice when the environments suffers from system error or the data collecting and transferring protocol under attack.

Original: This task is more challenging than previous imitation learning problem with noisy demonstrations only.
Revised: This task is more challenging than previous imitation learning problem with noisy demonstrations only.

Original: Baselines.
Revised: Baselines.

Original: In this experiment, we compare our method GAIL-USN to GAIL and the following state-of-the-art robust imitation learning methods:
Revised: In this experiment, we compare our method GAIL-USN to GAIL and the following state-of-the-art robust imitation learning methods:

Original: • RIL and RIL-CO (Tangkaratt et al., 2020) • SAIL-soft and SAIL-hard (Wang et al., 2021)
Revised: • RIL and RIL_CO (Tangkaratt et al., 2020) • SAIL-soft and SAIL-hard (Wang et al., 2021)

Original: Since allof the baselines are variants of GAIL, we use the same settings including behavior cloning initialization forall of the methods
Revised: Since all the baselines are variants of GAIL, we use the same settings including behavior cloning initialization for

Original: Results.
Revised: Results.

Original: As shown in Table 4, SOTA imitation learning method like GAIL and SAILs total fails in this task.
Revised: As shown in Table 4, SOTA imitation learning method like GAIL and SAILs total fails in this task.

Original: Our method GAIL-USN outperforms allof the baselines in the Atari game.
Revised: Our method GAIL-USN outperforms all the baselines in the Atari game.

Original: RIL-CO also performs closely to the optimal performance GAIL*.
Revised: RIL_CO also performs closely to the optimal performance GAIL*.

Original: However, it requires more computation cost due to the usage of co-training.
Revised: However, it requires more computation cost due to the usage of co-training.

Original: Table 5 shows the performance of GAIL-USN and other baselines on
Revised: Table 5 shows the performance of GAIL-USN and other baselines on

Original: Seaquestgame across multiple state-dependent actionnoise.
Revised: Seaquest action

Original: [Table]
Revised: [Table]

Original: F E XTENSION TO GENERAL Q- LEARNING METHOD
Revised: F E XTENSION TO GENERAL Q- LEARNING METHOD

Original: USN is a general method for robust training of imitation learning against action noise.
Revised: USN is a general method for robust training of imitation learning against action noise.

Original: BCQ-USN can easily apply USN on top of the generative model of BCQ methods.
Revised: BCQ-USN can easily apply USN on top of the generative model of BCQ methods.

Original: Because the generative model is similar to a classifier in label-noise learning tasks.
Revised: Because the generative model is similar to a classifier in label-noise learning tasks.

Original: Therefore, BCQ-USN can naturally employ the noise-tolerant methods for positive learning and negative learning.
Revised: Therefore, BCQ-USN can naturally employ the noise-tolerant methods for positive learning and negative learning.

Original: However, many IL methods
Revised: However, many IL methods

Original: [Table]
Revised: [Table]

Original: do not have such generative model.
Revised: do not have such generative model.

Original: How to extend USN to such IL methods is an interesting and essential problem.
Revised: How to extend USN to such IL methods is an interesting and essential problem.

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we show how to apply USN on some general Q-learning based IL methods, e
Revised: we show how to apply USN on some general Q-learning based IL methods, e

Original: [Table]
Revised: [Table]

Original: g. the conservative Q-learning (CQL) (Kumar et al., 2020)
Revised: g. the conservative Q-learning (CQL) (Kumar et al., 2020)

Original: [Table]
Revised: [Table]

Original: F. C ONSERVATIVE Q- LEARNING (CQL)
Revised: F. C ONSERVATIVE Q- LEARNING (CQL)

Original: Offline RL algorithms promise to learn effective policies from previously-collected, static datasets without further interaction.
Revised: Offline RL algorithms promise to learn effective policies from previously-collected, static datasets without further interaction.

Original: However, in practice, offline RL presents a major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions.
Revised: However, in practice, offline RL presents a major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions.

Original: Kumar et al.
Revised: Kumar et al.

Original: (2020) proposed conservative Q-learning (CQL), which aims to address these limitations by learning a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value.
Revised: (2020) proposed conservative Q-learning (CQL), which aims to address these limitations by learning a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value.

Original: They theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees.
Revised: They theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees.

Original: In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward to implement on top of existing deep Q-learning and actor-critic implementations.
Revised: In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward to implement on top of existing deep Q-learning and actor-critic implementations.

Original: Eq.
Revised: Eq.

Original: 12 addresses the impact of out-ofdistribution actions and obtain conservative value estimates.
Revised: 11 addresses the impact of out-ofdistribution actions and obtain conservative value estimates.

Original: Algorithm 5 presents the procedures of
Revised: Algorithm 5 presents the procedures of

Original: CQL.
Revised: CQL.

Original: To use CQL for offline imitation learning, we train an intrinsic curiosity module (ICM) (Pathak et al., 2017) for generating intrinsic rewards.
Revised: To use CQL for offline imitation learning, we train an intrinsic curiosity module (ICM) (Pathak et al., 2017) for generating intrinsic rewards.

Original: [Equation]
Revised: [Equation]

Original: where ˆ B π k ˆ Q k ( s, a ) =
Revised: where ˆ B π k ˆ Q k ( s, a ) =

Original: r ( s, a ) + γ E s ′ ∼ P
Revised: r ( s, a ) + γ E s ′ ∼ P

Original: ( s ′ | s,a ) [max a ′
Revised: ( s ′ | s,a ) [max a ′

Original: ˆ Q ( s ′ , a ′ )] is the Bellman optimality operator.
Revised: ˆ Q ( s ′ , a ′ )] is the Bellman optimality operator.

Original: Algorithm
Revised: Algorithm

Original: F. CQL-USN
Revised: F. CQL-USN

Original: Since there is no generative model in CQL, we apply USN on top of the Q network.
Revised: Since there is no generative model in CQL, we apply USN on top of the Q network.

Original: We first introduce a robust IL baseline algorithm, CQL-GCE, by training CQL with additional GCE loss on Q:
Revised: We first introduce a robust IL baseline algorithm, CQL-GCE, by training CQL with additional GCE loss on Q:

Original: [Equation]
Revised: [Equation]

Original: We use CQL-GCE for positive learning on the full-batch data.
Revised: We use CQL-GCE for positive learning on the full-batch data.

Original: Then, we estimate the predictive uncertainty measures (calibration errors) on Q. Next, we set the sample-selection threshold using the calibration errors, and select large-loss samples for soft negative learning.
Revised: Then, we estimate the predictive uncertainty measures (calibration errors) on Q. Next, we set the sample-selection threshold using the calibration errors, and select large-loss samples for soft negative learning.

Original: This simple extension is called CQL-USN.
Revised: This simple extension is called CQL-USN.

Original: In CQL-USN (Algorithm 6), the Q network is trained in a multi-task manner, predicting values and noise-tolerant action prediction.
Revised: In CQL-USN (Algorithm 6), the Q network is trained in a multi-task manner, predicting values and noise-tolerant action prediction.

Original: Algorithm 6 Robust CQL (e.g. CQL-USN) 1: Initialize Q-function Q θ , and optionally a policy π ϕ .
Revised: Algorithm 6 Robust CQL (e.g. CQL-USN) 1: Initialize Q-function Q θ , and optionally a policy π ϕ .

Original: 2: for t = 1 , · · · , T do 3:
Revised: 2: for t = 1 , · · · , T do 3:

Original: Update Q-function parameters by minimizing Eq.
Revised: Update Q-function parameters by minimizing Eq.

Original: (12) and USN in Algorithm 1.
Revised: (11) and USN in Algorithm 1.

Original: 4: (only with actor-critic)
Revised: 4: (only with actor-critic)

Original: Improve policy π ϕ via with SAC-style entropy regularization.
Revised: Improve policy π ϕ via with SAC-style entropy regularization.

Original: 5: end for
Revised: 5: end for

Original: F. E XPERIMENTS AND R ESULTS
Revised: F. E XPERIMENTS AND R ESULTS

Original: Settings.
Revised: Settings.

Original: We evaluate the effectiveness of CQL-USN on offline datasets of Atari games from the d4rl benchmark (Fu et al., 2020).
Revised: We evaluate the effectiveness of CQL-USN on offline datasets of Atari games from the d4rl benchmark (Fu et al., 2020).

Original: We build our method on top of d3rlpy library (Takuma Seno, 2021), and use its default hyperparameter settings.
Revised: We build our method on top of d3rlpy library (Takuma Seno, 2021), and use its default hyperparameter settings.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Results.
Revised: Results.

Original: We compare the average return of CQL, CQL-GCE and CQL-USN with state-dependent action noise (Table 6), symmetric action noise (Table 7) and pairflip action noise (Table 8).
Revised: We compare the average return of CQL, CQL-GCE and CQL-USN with state-dependent action noise (Table 6), symmetric action noise (Table 7) and pairflip action noise (Table 8).

Original: CQLUSN usually outperforms CQL-GCE and CQL on the three types of action noise significantly across multiple noise rates.
Revised: CQLUSN usually outperforms CQL-GCE and CQL on the three types of action noise significantly across multiple noise rates.

Original: G M ORE PREDICTIVE UNCERTAINTY DETAILS
Revised: G M ORE PREDICTIVE UNCERTAINTY DETAILS

Original: Expected Calibration Error (ECE) (Naeini et al., 2015; Guo et al., 2017) is widely used to measure the predictive uncertainty (model calibration) of a deep network.
Revised: Expected Calibration Error (ECE) (Naeini et al., 2015; Guo et al., 2017) is widely used to measure the predictive uncertainty (model calibration) of a deep network.

Original: To approximate the calibration error in expectation, ECE discretizes the probability interval into a fixed number of bins, and assigns each predicted probability to the bin that encompasses it.
Revised: To approximate the calibration error in expectation, ECE discretizes the probability interval into a fixed number of bins, and assigns each predicted probability to the bin that encompasses it.

Original: The calibration error is the difference between the fraction of predictions in the bin that are correct (accuracy) and the mean of the probabilities in the bin (confidence).
Revised: The calibration error is the difference between the fraction of predictions in the bin that are correct (accuracy) and the mean of the probabilities in the bin (confidence).

Original: Intuitively, the accuracy estimates P ( Y =
Revised: Intuitively, the accuracy estimates P ( Y =

Original: y
Revised: y

Original: | ˆ p = p ) , and the average confidence is a setting of p .
Revised: | ˆ p = p ) , and the average confidence is a setting of p .

Original: ECE computes a weighted average of this error across bins:
Revised: ECE computes a weighted average of this error across bins:

Original: [Equation]
Revised: [Equation]

Original: (14) where n b is the number of predictions in bin b , N is the total number of data points, and acc ( b ) and conf ( b ) are the accuracy and confidence of bin b , respectively.
Revised: (13) where n b is the number of predictions in bin b , N is the total number of data points, and acc ( b ) and conf ( b ) are the accuracy and confidence of bin b , respectively.

Original: ECE as framed in (Naeini et al., 2015) leaves ambiguity in both its binning implementation and how to compute calibration
Revised: ECE as framed in (Naeini et al., 2015) leaves ambiguity in both its binning implementation and how to compute calibration

Original: [Table]
Revised: [Table]

Original: for multiple classes.
Revised: for multiple classes.

Original: In Guo et al.
Revised: In Guo et al.

Original: , they bin the probability interval [0; 1] into equally spaced subintervals, and they take the maximum probability output for each datapoint (i.e., the predicted class’s probability).
Revised: , they bin the probability interval [0; 1] into equally spaced subintervals, and they take the maximum probability output for each datapoint (i.e., the predicted class’s probability).

Original: We use this for our ECE implementation.
Revised: We use this for our ECE implementation.

Original: Adaptive Calibration Error (ACE) improves ECE by using adaptive calibration ranges.
Revised: Adaptive Calibration Error (ACE) improves ECE by using adaptive calibration ranges.

Original: The motivation is that in order to get the best estimate of the overall calibration error the metric should focus on the regions where the predictions are made (and focus less on regions with few predictions)
Revised: The motivation is that in order to get the best estimate of the overall calibration error the metric should focus on the regions where the predictions are made (and focus less on regions with few predictions)

Original: (Nixon et al., 2019).
Revised: (Nixon et al., 2019).

Original: ACE uses an adaptive scheme which spaces the bin intervals so that each contains an equal number of predictions.
Revised: ACE uses an adaptive scheme which spaces the bin intervals so that each contains an equal number of predictions.

Original: In detail, ACE takes as input the predictions P (usually out of a softmax), correct labels, and a number of ranges R :
Revised: In detail, ACE takes as input the predictions P (usually out of a softmax), correct labels, and a number of ranges R :

Original: [Equation]
Revised: [Equation]

Original: where acc ( r, k ) and conf ( r, k ) are the accuracy and confidence of adaptive calibration range r for class label k , respectively.
Revised: where acc ( r, k ) and conf ( r, k ) are the accuracy and confidence of adaptive calibration range r for class label k , respectively.

Original: Thresholded Adaptive Calibration Error (TACE) is identical to ACE, with the only difference being that TACE is only evaluated on values above ϵ .
Revised: Thresholded Adaptive Calibration Error (TACE) is identical to ACE, with the only difference being that TACE is only evaluated on values above ϵ .

Original: Static Calibration Error (SCE) is a simple extension of Expected Calibration Error to every probability in the multiclass setting.
Revised: Static Calibration Error (SCE) is a simple extension of Expected Calibration Error to every probability in the multiclass setting.

Original: SCE bins predictions separately for each class probability, computes the calibration error within the bin, and averages across bins (Nixon et al., 2019):
Revised: SCE bins predictions separately for each class probability, computes the calibration error within the bin, and averages across bins (Nixon et al., 2019):

Original: [Equation]
Revised: [Equation]

Original: where acc ( b, k ) and conf ( b, k ) are the accuracy and confidence of bin b for class label k , respectively; n bk is the number of predictions in bin b for class label k ; and N is the total number of data points.
Revised: where acc ( b, k ) and conf ( b, k ) are the accuracy and confidence of bin b for class label k , respectively; n bk is the number of predictions in bin b for class label k ; and N is the total number of data points.

Original: Unlike ECE, assuming infinite data and infinite bins, SCE is guaranteed to be zero if and only if the model is calibrated.
Revised: Unlike ECE, assuming infinite data and infinite bins, SCE is guaranteed to be zero if and only if the model is calibrated.

Original: Root Mean Square Calibration Error (RMSCE) (Hendrycks et al., 2018) measures the square root of the expected squared difference between confidence and accuracy at a confidence level.
Revised: Root Mean Square Calibration Error (RMSCE) (Hendrycks et al., 2018) measures the square root of the expected squared difference between confidence and accuracy at a confidence level.

Original: The
Revised: The

Original: RMSCE is estimated with the numerically stable formula:
Revised: RMSCE is estimated with the numerically stable formula:

Original: [Equation]
Revised: [Equation]

Original: H A BLATION S TUDIES
Revised: H A BLATION S TUDIES

Original: In this subsection, we study the effects of several essential hyper-parameters and components in
Revised: In this subsection, we study the effects of several essential hyper-parameters and components in

Original: LunarLander-2 task with state-dependent action noise ( ϵ = 0 . 45 ).
Revised: LunarLander-2 task with state-dependent action noise ( ϵ = 0 . 45 ).

Original: This ablation follows the rule of ‘study one, fix the others’.
Revised: This ablation follows the rule of ‘study one, fix the others’.

Original: The ablation study results are illustrated in Figure 30 and Figure 28.
Revised: The ablation study results are illustrated in Figure 37 and Figure 35.

Original: Effect of negative learning loss.
Revised: Effect of negative learning loss.

Original: We evaluate the performance of our method using the classic complementary-label learning approach, pairwise-comparison (PC) loss (Ishida et al., 2017) for negative learning.
Revised: We evaluate the performance of our method using the classic complementary-label learning approach, pairwise-comparison (PC) loss (Ishida et al., 2017) for negative learning.

Original: We denote this instance as BCQ-USN-PC.
Revised: We denote this instance as BCQ-USN-PC.

Original: Figure 10(c) and Figure 10 show that
Revised: Figure 10(c) and Figure 10 show that

Original: BCQ-USN-PC is comparable to the original BCQ-USN when learning with state-dependent action noise , symmetric action noise and pairflip action noise.
Revised: BCQ-USN-PC is comparable to the original BCQ-USN when learning with state-dependent action noise , symmetric action noise and pairflip action noise.

Original: This implies that our method can generate many variants by replacing the negative learning loss.
Revised: This implies that our method can generate many variants by replacing the negative learning loss.

Original: Effect of sample-selection threshold.
Revised: Effect of sample-selection threshold.

Original: Figure 30 shows that BCQ-USN outperforms BCQ-GCE significantly with diverse calibration error metrics when λ neg = 1 . 0 .
Revised: Figure 37 shows that BCQ-USN outperforms BCQ-GCE significantly with diverse calibration error metrics when λ neg = 1 . 0 .

Original: We can set sample-selection threshold τ u as any types of calibration error metric, which demonstrates the good scalability of our method.
Revised: We can set sample-selection threshold τ u as any types of calibration error metric, which demonstrates the good scalability of our method.

Original: Effect of unlikely action threshold τ of generative model G .
Revised: Effect of unlikely action threshold τ of generative model G .

Original: Figure 28(a) shows that BCQ-USN outperforms BCQ-GCE significantly with five calibration error metrics and one fixed value.
Revised: Figure 35(a) shows that BCQ-USN outperforms BCQ-GCE significantly with five calibration error metrics and one fixed value.

Original: Effect of the weight of negative learning loss λ neg .
Revised: Effect of the weight of negative learning loss λ neg .

Original: We select λ neg from [0.01, 0.1, 1.0], and find our method BCQ-USN achieves the best performance with λ neg = 1 . 0 , and all three instances outperform BCQ-GCE (the blue dash dot line in Figure 28(b)).
Revised: We select λ neg from [0.01, 0.1, 1.0], and find our method BCQ-USN achieves the best performance with λ neg = 1 . 0 , and all three instances outperform BCQ-GCE (the blue dash dot line in Figure 35(b)).

Original: Effect of smooth rate α .
Revised: Effect of smooth rate α .

Original: To avoid tuning of the smooth rate in soft negative learning loss, we set the smooth rate as the estimated calibration errors.
Revised: To avoid tuning of the smooth rate in soft negative learning loss, we set the smooth rate as the estimated calibration errors.

Original: Figure 28(c) shows that ECE and TACE are the worst, and the other four instances all outperform BCQ-GCE with a large margin.
Revised: Figure 35(c) shows that ECE and TACE are the worst, and the other four instances all outperform BCQ-GCE with a large margin.

Original: The above ablations show that our method is not sensitive to the choices of hyper-parameters.
Revised: The above ablations show that our method is not sensitive to the choices of hyper-parameters.

Original: More importantly, the calibration error metrics can be of multiple use, which maintains the robustness of BCQ-USN while avoiding the effort of tuning hyper-parameters.
Revised: More importantly, the calibration error metrics can be of multiple use, which maintains the robustness of BCQ-USN while avoiding the effort of tuning hyper-parameters.

Original: [Figure]
Revised: [Figure]

Original: D ETAILED ABLATION STUDY RESULTS
Revised: D ETAILED ABLATION STUDY RESULTS

Original: Effect of sample-selection threshold.
Revised: Effect of sample-selection threshold.

Original: Table 9 and Table 10 shows the ablation results on diverse sample-selection threshold when λ neg = 0 .
Revised: Table 9 and Table 10 shows the ablation results on diverse sample-selection threshold when λ neg = 0 .

Original: 01 and λ neg = 1 . 0 , respectively.
Revised: 01 and λ neg = 1 . 0 , respectively.

Original: BCQ-USN with λ neg = 1 . 0 outperforms that trained with λ neg = 0 .
Revised: BCQ-USN with λ neg = 1 . 0 outperforms that trained with λ neg = 0 .

Original: no mater the choice of sample-selection threshold.
Revised: no mater the choice of sample-selection threshold.

Original: The corresponding bar chats comparison are shown in Figure 29.
Revised: The corresponding bar chats comparison are shown in Figure 36.

Original: BCQ-USN usually outperforms the BCQ-GCE by a large margin when using the estimated calibration error metrics (predictive uncertainty) as the small-loss sample-selection threshold.
Revised: BCQ-USN usually outperforms the BCQ-GCE by a large margin when using the estimated calibration error metrics (predictive uncertainty) as the small-loss sample-selection threshold.

Original: Therefore, predictive uncertainty is a good alternative to the manual tuning sample-selection threshold, since it requires no prior knowledge about noise and it has good adapbility to diverse noise types and noise rates.
Revised: Therefore, predictive uncertainty is a good alternative to the manual tuning sample-selection threshold, since it requires no prior knowledge about noise and it has good adapbility to diverse noise types and noise rates.

Original: Effect of threshold τ .
Revised: Effect of threshold τ .

Original: The ablation results on the unlikely action threshold τ are shown in Table 11.
Revised: The ablation results on the unlikely action threshold τ are shown in Table 11.

Original: BCQ-USN significantly outperforms BCQ-GCE when setting τ as different predictive uncertainty
Revised: BCQ-USN significantly outperforms BCQ-GCE when setting τ as different predictive uncertainty

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: measures except ECE.
Revised: measures except ECE.

Original: This means that the predictive uncertainty measures are usually reliable choices for eliminating the unlikely action for policy training in BCQ and its variants.
Revised: This means that the predictive uncertainty measures are usually reliable choices for eliminating the unlikely action for policy training in BCQ and its variants

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Effect of negative learning weight λ neg .
Revised: Effect of negative learning weight λ neg .

Original: Table 13 shows the ablation results on the multiple choices of negative learning loss weight λ neg .
Revised: Table 13 shows the ablation results on the multiple choices of negative learning loss weight λ neg .

Original: Effect of smooth rate α .
Revised: Effect of smooth rate α .

Original: Table 14 shows the ablation results on using different calibration error metrics as the smooth rate α in soft negative learning loss.
Revised: Table 14 shows the ablation results on using different calibration error metrics as the smooth rate α in soft negative learning loss.

Original: Effect of negative learning loss.
Revised: Effect of negative learning loss.

Original: we compare the performance of the original BCQ-USN with the one using a classic complementary label learning method, e.g. Pairwise-comparison (PC) loss (Ishida et al., 2017) for negative learning:
Revised: we compare the performance of the original BCQ-USN with the one using a classic complementary label learning method, e.g. Pairwise-comparison (PC) loss (Ishida et al., 2017) for negative learning:

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Ishida et al.
Revised: Ishida et al.

Original: (2017) first formulated the problem of complementary-label learning (CLL) and proposed a risk minimization framework.
Revised: (2017) first formulated the problem of complementary-label learning (CLL) and proposed a risk minimization framework.

Original: The goal of CLL is to learn a classifier f that minimizes the classification risk (18), from only the complementary labeled samples { ( x i , ¯ y i ) } ni =1 .
Revised: The goal of CLL is to learn a classifier f that minimizes the classification risk (17), from only the complementary labeled samples { ( x i , ¯ y i ) } ni =1 .

Original: { ( x i , ¯ y i ) } ni =1 are drawn independently from an unknown probability distribution with density: ¯ p ( x, ¯ y ) =
Revised: { ( x i , ¯ y i ) } ni =1 are drawn independently from an unknown probability distribution with density: ¯ p ( x, ¯ y ) =

Original: [Equation]
Revised: [Equation]

Original: where K is the number of classes.
Revised: where K is the number of classes.

Original: [Equation]
Revised: [Equation]

Original: To achieve this goal
Revised: To achieve this goal

Original: [Table]
Revised: [Table]

Original: Ishida et al.
Revised: Ishida et al.

Original: (2017) proved that the risk (18) can be approximated in an unbiased fashion:
Revised: (2017) proved that the risk (17) can be approximated in an unbiased fashion:

Original: [Equation]
Revised: [Equation]

Original: where ¯ L is a complementary loss.
Revised: where ¯ L is a complementary loss.

Original: They considered the one-versus-all (OVA) loss ¯ L OV A ( f ( x ) , ¯ y ) =
Revised: They considered the one-versus-all (OVA) loss ¯ L OV A ( f ( x ) , ¯ y ) =

Original: K − 1 (cid:80) y ̸
Revised: K − 1 (cid:80) y ̸

Original: =¯ y ℓ ( g y ( x ))
Revised: =¯ y ℓ ( g y ( x ))

Original: + ℓ ( − g ¯ y ( x )) and pairwise comparison (PC) loss:
Revised: + ℓ ( − g ¯ y ( x )) and pairwise comparison (PC) loss:

Original: [Equation]
Revised: [Equation]

Original: where ℓ ( z ) is a binary loss
Revised: where ℓ ( z ) is a binary loss

Original: which satisfy ℓ ( z ) + ℓ ( − z ) = 1 , such as the sigmoid loss ℓ S ( z ) = 1 

Revised: which satisfy ℓ ( z ) + ℓ ( − z ) = 1 , such as the sigmoid loss ℓ S ( z ) = 1

Original: + e z .
Revised: + e z .

Original: ¯ L
Revised: ¯ L

Original: OV A satisfies this condition with M 1 = K and M 2 = 2 , and ¯ L PC satisfies the condition with
Revised: OV A satisfies this condition with M 1 = K and M 2 = 2 , and ¯ L PC satisfies the condition with

Original: M 1 =
Revised: M 1 =

Original: K ( K − 1) / 2 and M 2 = K − 1 .
Revised: K ( K − 1) / 2 and M 2 = K − 1 .

Original: [Table]
Revised: [Table]

Original: Results on state-independent action noise .
Revised: Results on state-independent action noise .

Original: In Table 16 and Table 17, we show the comparative results of BCQ, BCQ-GCE, BCQ-USN and BCQ-USN-PC on LunarLander-v2 with symmetric action noise and pairflip action noise.
Revised: In Table 16 and Table 17, we show the comparative results of BCQ, BCQ-GCE, BCQ-USN and BCQ-USN-PC on LunarLander-v2 with symmetric action noise and pairflip action noise.

Original: BCQ-USN is comparable to BCQ-USN-PC, and are usually better than BCQ-GCE and BCQ across diverse noise rates.
Revised: BCQ-USN is comparable to BCQ-USN-PC, and are usually better than BCQ-GCE and BCQ across diverse noise rates.

Original: [Table]
Revised: [Table]

Original: Results on state-dependent action noise .
Revised: Results on state-dependent action noise .

Original: Training with PC loss, BCQ-USN-PC achieves comparable results to the original BCQ-USN in LunarLander-v2 across multiple levels of state-dependent action noise (Table 18).
Revised: Training with PC loss, BCQ-USN-PC achieves comparable results to the original BCQ-USN in LunarLander-v2 across multiple levels of state-dependent action noise (Table 18).

Original: [Table]
Revised: [Table]