Original: Ensemble-based Uncertainty Estimation with overlapping alternative Predictions
Revised: Ensemble-based Uncertainty Estimation with overlapping alternative Predictions

Original: Abstract
Revised: Abstract

Original: A reinforcement learning model will predict an action in whatever state it is - even if there is no distinct outcome due to unseen states the model may not indicate that.
Revised: A reinforcement learning model will predict an action in whatever state it is. Even if there is no distinct outcome due to unseen states the model may not indicate that.

Original: Different methods for uncertainty estimation can be used to indicate this.
Revised: Methods for uncertainty estimation can be used to indicate this.

Original: Although a known approach in AI, most of the available uncertainty estimation methods are not able to deal with the choice overlap that happens in states where multiple actions can be taken by a reinforcement learning agent with a similar performance outcome.
Revised: Although a known approach in Machine Learning, most of the available uncertainty estimation methods are not able to deal with the choice overlap that happens in states where multiple actions can be taken by a reinforcement learning agent with a similar performance outcome.

Original: In this work, we investigate uncertainty estimation on simplified scenarios in a gridworld environment.
Revised: In this work, we investigate uncertainty estimation on simplified scenarios in a gridworld environment.

Original: Using ensemble-based uncertainty estimation we propose an algorithm based on action count variance  to deal with discrete action spaces and  delta to ID action  variancecalculation to handle overlapping alternative predictions.
Revised: Using ensemble-based uncertainty estimation we propose an algorithm based on action count variance (ACV) to deal with discrete action spaces and a calculation based on the in-distribution delta (IDD) of the action count variance to handle overlapping alternative predictions.

Original: To visualize the expressiveness of the model uncertainty we create heatmaps for different in-distribution  and out-of-distribution (OOD) scenarios and propose an indicator for uncertainty.
Revised: To visualize the expressiveness of the model uncertainty we create heatmaps for different in-distribution (ID) and out-of-distribution (OOD) scenarios and propose an indicator for uncertainty.

Original: We can show that the method is able to indicate potentially unsafe states when the agent is facing novel elements in the OOD scenarios while capable to distinguish uncertainty resulting from OOD instances from uncertainty caused by the overlapping of alternative predictions.
Revised: We can show that the method is able to indicate potentially unsafe states when the agent is facing novel elements in the OOD scenarios while capable to distinguish uncertainty resulting from OOD instances from uncertainty caused by the overlapping of alternative predictions.

Original:  Introduction
Revised:  Introduction

Original: In order to apply Machine Learning in today’s and future’s real-world applications it is necessary to takeinto account the given system boundaries of an application, e g. resource and/or timing requirements and safety constraints.
Revised: In order to apply Machine Learning in today’s and future’s real-world applications it is necessary to take the given system boundaries of an application into account, e g. resource and/or timing requirements and safety constraints.

Original: There is a high incentive to use Reinforcement Learning (RL) in parts of these system approaches, as to the high flexibility and self-handling in decision-making tasks.
Revised: There is a high incentive to use Reinforcement Learning (RL) in parts of these system approaches, as to the high flexibility and self-handling in decision-making tasks.

Original: However, this also comes with a high risk in safety-critical applications.
Revised: However, this also comes with a high risk in safety-critical applications.

Original: As an RL agent will predict an action in whatever state it will find itself, it is of high importance for the application controller to be aware of the certainty and confidence of its own decisions in all situations it could face.
Revised: As an RL agent will predict an action in whatever state it will find itself, it is of high importance for the application controller to be aware of the certainty and confidence of its own decisions in all situations it could face.

Original: This includes situations from inside as from outside the training data.
Revised: This includes situations from inside as from outside the training data distribution.

Original: To overcome this problem in the larger field of safe RL this paper focuses on estimating uncertainty from data during deployment (test data) that is out of the distribution of the training data(distributional shift) as one aspect of the problem solution.
Revised: To overcome this problem in the larger field of safe RL this paper focuses on estimating uncertainty from data during deployment (test data) that is out of the distribution of the training data as one aspect of the problem solution.

Original: Distributional shift in data science is widely understood as the distributional difference between the training and test data of an algorithm.
Revised: Distributional shift in data science is widely understood as the distributional difference between the training and test data of a problemHendrycks and Gimpel [2018]Lütjens et al. [2019]Lee et al. [2018]Postels et al.

Original: This distributional shift originates from different sources, such as perturbations to the data-set due to system complexity and a lack of training data.
Revised: This distributional shift can originate from different sources, such as perturbations to the data-set due to system complexity and a lack of training data.

Original: In machine learning, a distributional shift often leads to degraded performance during testing, because the probability distribution over data (or state-action pairs in RL) is shifted and a classifier or an agent may therefore predict wrong or sub-optimal classes (or actions).
Revised: In machine learning, a distributional shift often leads to degraded performance during test time, because the probability distribution over data (or state-action pairs in RL) is shifted and a classifier or an agent may therefore predict wrong or sub-optimal classes (or actions).

Original: In general, when the testing distribution differs from the training distribution, machine learning systems may not only demonstrate poor performance but also wrongly assume that their performance is good.
Revised: In general, when the testing distribution differs from the training distribution, machine learning systems may not only demonstrate poor performance but also wrongly assume that their performance is good.

Original: This issue is also present in system solutions based on Reinforcement Learning.
Revised: This issue is also present in system solutions based on Reinforcement Learning.

Original: To overcome this limitation, safe Reinforcement Learning (Safe RL) solutions must be capable of detecting and handling the uncertainty in the decision-making process.
Revised: To overcome this limitation, safe Reinforcement Learning (Safe RL) solutions must be capable of detecting and handling the uncertainty in the decision-making process.

Original: Therefore, it is necessary to deal with uncertainty and potentially improve given algorithms to optimize these with respect to safety constraints.
Revised: Therefore, it is necessary to deal with uncertainty and potentially improve given algorithms to optimize these with respect to safety constraints.

Original: For instance, uncertainty estimation can detect a lack of generalization due to insufficient training and unseen states during training (OOD, epistemic uncertainty) as well as uncertainty resulting from randomness in the environment (aleatoric uncertainty).
Revised: For instance, uncertainty estimation can detect a lack of generalization due to insufficient training and unseen states during training (epistemic uncertainty) as well as uncertainty resulting from randomness in the environment (aleatoric uncertainty).

Original: For epistemic uncertainty, estimates can be obtained from the variance over the prediction of a set of agents trained independently.
Revised: For epistemic uncertainty, estimates can be obtained from the variance over the prediction of a set of agents trained independently.

Original: In states where the agents are unsure due to insufficient training, different agents are prone to diverge in their action predictions.
Revised: In states where the agents are unsure due to insufficient training data, different agents are prone to diverge in their action predictions.

Original: This can be utilized to indicate uncertainty, however, in some optimization problems, multiple paths could be taken to get to the goal and the agents can deviate within the possible alternative predictions.
Revised: This can be utilized to indicate uncertainty, however, in some optimization problems, multiple paths could be taken to get to the goal and the agents can deviate within the possible alternative predictions.

Original: Therefore, it is necessary to differentiate between the two effects.
Revised: Therefore, it is necessary to differentiate between these two effects.

Original:  Related work
Revised:  Related work

Original: There is a lot of work in recent years with respect to OOD in the classical image classification domain as well as some work in the RL domain.
Revised: There is a lot of work in recent years with respect to OOD in the classical image classification domain as well as some work in the RL domain.

Original: Pimentel et al.
Revised: Pimentel et al.

Original: [2014] define novelty as samples that differ in some respect from the data used during training.
Revised: [2014] define novelty as samples that differ in some respect from the data used during training.

Original: Insufficient training data will make the model less capable to generalize to data that conceptually belong to the same distribution but were not experienced during training.
Revised: Insufficient training data will make the model less capable to generalize to data that conceptually belong to the same distribution but were not experienced during training.

Original: Hendrycks and Gimpel [2018] define OOD as testing samples drawn from a distribution that differs from the training data and describe OOD detection as a threshold–based process.
Revised: Hendrycks and Gimpel [2018] define OOD as testing samples drawn from a distribution that differs from the training data and describe OOD detection as a threshold–based process.

Original: Furthermore, a differentiation is posed between OOD instances that are near or far from the training data distribution.
Revised: Furthermore, a differentiation is posed between OOD instances that are near or far from the training data distribution.

Original: The closer the OOD sample is to the training data distribution, the easiest it is to mitigate the issue by training with more data, whereas the further away it gets it will most likely represent conceptual or semantic gaps that are not easily overcome, e.g., samples which are not included in the known classes.
Revised: The closer the OOD sample is to the training data distribution, the easiest it is to mitigate the issue by training with more data, whereas the further away it gets it will most likely represent conceptual or semantic gaps that are not easily overcome, e.g., samples which are not included in the known classes.

Original: In this regard, DNNs tend to be overconfident in predictions on unseen data and can give unpredictable results for far-from-distribution test data Lütjens et al.
Revised: In this regard, DNNs tend to be overconfident in predictions on unseen data and can give unpredictable results for far-from-distribution test data Lütjens et al.

Original: A lot of prior work focuses on OOD as a concept of samples from outside the class set.
Revised: A lot of prior work focuses on OOD as a concept of samples from outside the class set.

Original: If samples are outside the set, classifications for that cannot be learned, even with more/unlimited training.
Revised: If samples are outside the set, classifications for that cannot be learned, even with more/unlimited training.

Original: A recurrent approach is to specify a separate OOD class to train the model on Pimentel et al.
Revised: A recurrent approach is to specify a separate OOD class to train the model on Pimentel et al.

Original: Similarly, DeVries and Taylor [2018], Mohseni et al.
Revised: Similarly, DeVries and Taylor [2018], Mohseni et al.

Original: [2020] define OOD samples as examples of classes different from those in the ID dataset.
Revised: [2020] define OOD samples as examples of classes different from those in the ID data set.

Original: Lee et al.
Revised: Lee et al.

Original: [2018] describe ID  as a trained distribution  by a classifier and OOD  as sufficiently different from it.
Revised: [2018] describe ID data as a data distribution trained by a classifier and OOD data as sufficiently different from it.

Original: Also, Schwaiger et al.
Revised: Also, Schwaiger et al.

Original: [2020] follow the approach to consider strong conceptual differences between training and test data to beOOD.
Revised: [2020] follow the approach to consider strong conceptual differences between training and test data to be

Original: Yu and
Revised: OOD.

Original:  Aizawa [2019] go as far as to define OOD by the gap in between classified ID data.
Revised: Yu and Aizawa [2019] go as far as to define OOD by the gap in between classified ID data.

Original: They propose to maximize the discrepancy between the decision boundaries of e.g. two classifiers to push OOD samples outside.
Revised: They propose to maximize the discrepancy between the decision boundaries of e.g. two classifiers to push OOD samples outside.

Original: They also follow the concept of contrasting samples being near and far from the known distribution.
Revised: They also follow the concept of contrasting samples being near and far from the known distribution.

Original: Furthermore, it is important to understand the difference between epistemic and aleatoric uncertainty for uncertainty estimation as a proxy for OOD detectors.
Revised: Furthermore, it is important to understand the difference between epistemic and aleatoric uncertainty for uncertainty estimation as a proxy for OOD detectors.

Original: Epistemic uncertainty arises out of a lack of sufficient data to exactly infer the underlying system Sedlmeier et al.
Revised: Epistemic uncertainty arises out of a lack of sufficient data to exactly infer the underlying system Sedlmeier et al.

Original: Clements et al.
Revised: Clements et al.

Original: [2020] claim epistemic uncertainty to stem from limited data.
Revised: [2020] claim epistemic uncertainty to stem from limited data.

Original: Epistemic uncertainty can indicate samples that reside far away from the data distribution, as well as the capabilities of a model to generalize to data close to it Postels et al.
Revised: Epistemic uncertainty can indicate samples that reside far away from the data distribution, as well as the capabilities of a model to generalize to data close to it Postels et al.

Original: In contrast, aleatoric uncertainty arises from  stochastic  environments and must be accounted for in risk-sensitive applications Clements et al.
Revised: In contrast, aleatoric uncertainty arises from the stochastic of the environments and must be accounted for in risk-sensitive applications Clements et al.

Original: [2020], Chua et al.
Revised: Chua et al.

Original: Aleatoric uncertainty cannot be solved just by more training.
Revised: Aleatoric uncertainty cannot be reduced just by more training.

Original: For safety reasons, aleatoric uncertainty is a significant indicator in RL and should also be considered.
Revised: For safety reasons, aleatoric uncertainty is a significant indicator in RL and should also be considered.

Original: However, in this paper, we focus on the detection of epistemic uncertainty, as we focus on distributional shift and OOD detection.
Revised: However, in this paper, we focus on the detection of epistemic uncertainty, as we focus on distributional shift and OOD detection.

Original: Kahn et al.
Revised: Kahn et al.

Original: [2017] present an uncertainty-aware model-based learning algorithm that estimates the probability of  collision together with a statistical estimate of uncertainty.
Revised: [2017] present an uncertainty-aware model-based learning algorithm that estimates the probability of a collision together with a statistical estimate of uncertainty.

Original: The predictive model is based on bootstrapped neural networks using dropout.
Revised: The predictive model is based on bootstrapped neural networks using dropout.

Original: In regions of high uncertainty, their risk-averse cost function causes the robot to revert to a cautious low-speed strategy.
Revised: In regions of high uncertainty, their risk-averse cost function causes the robot to revert to a cautious low-speed strategy.

Original: In Da Silva et al.
Revised: In Da Silva et al.

Original: [2020] they propose an action-advising framework where the agent asks for advice when its epistemic uncertainty is high for a certain state to accelerate reinforcement learning.
Revised: [2020] they propose an action-advising framework where the agent asks for advice when its epistemic uncertainty is high for a certain state to accelerate reinforcement learning.

Original: They addas a last layer multiple heads estimating separately expected values for each action, as done in Bootstrapped DQN.
Revised: They add multiple heads estimating separately expected values for each action as a last layer, as done in Bootstrapped DQN.

Original: As the learning algorithm updates the network, their predictions get closer to the real underlying function, and close tothe each other.
Revised: As the learning algorithm updates the network, their predictions get closer to the real underlying function, and closer to each other.

Original: Sedlmeier et al.
Revised: Sedlmeier et al.

Original: [2019] use uncertainty-based OOD, using Q-value uncertainty in DQN Algorithm.
Revised: [2019] use uncertainty-based OOD detection, using Q-value uncertainty in DQN Algorithm.

Original: They compare MC-Dropout, Bootstrapped and Bootstrapped with prior functions.
Revised: They compare MC-Dropout, Bootstrapped and Bootstrapped with prior functions.

Original: They also address the problem of alternative valid predictions of the model agent.
Revised: They also address the problem of alternative valid predictions of the model agent.

Original: Unfortunately, they do not dig into details when it comes to the uncertainty estimation in those cases but rather calculate an overall estimate for the epoch.
Revised: Unfortunately, they do not further investigate the uncertainty estimation in those cases but rather calculate an overall estimate for the epoch.

Original: Hoel et al.
Revised: Hoel et al.

Original: [2020] estimate uncertainty for RL based on ensembles with randomized prior functions (RPF).
Revised: [2020] estimate uncertainty for RL based on ensembles with randomized prior functions (RPF).

Original: They are based on Osband et al.
Revised: They are based on Osband et al.

Original: [2018] and propose a criterion function.
Revised: [2018] and propose a criterion function.

Original: They choose safe actions in unknown situations far from the training distribution.
Revised: They choose safe actions in unknown situations far from the training distribution.

Original: In Hoel et al.
Revised: In Hoel et al.

Original: [2021] they also utilize an ensemble of DQN agents to estimate Q-value uncertainty to switch back to a fallback policy in uncertain situations given a certain threshold.
Revised: [2021] they also utilize an ensemble of DQN agents to estimate Q-value uncertainty to switch back to a fallback policy in uncertain situations given a dedicated threshold.

Original: An ensemble is trained on bootstrapped data, which provides a distribution over the estimated Q-values to provide a Bayesian estimation of the epistemic uncertainty.
Revised: An ensemble is trained on bootstrapped data, which provides a distribution over the estimated Q-values to provide a Bayesian estimation of the epistemic uncertainty.

Original: The epistemic uncertainty estimate is then used to choose less risky actions in unknown situations.
Revised: The epistemic uncertainty estimate is then used to choose less risky actions in unknown situations.

Original: However, they don’t take into account a potential overlapping uncertainty due to possible alternative actions.
Revised: However, they do not take into account a potential overlapping uncertainty due to possible alternative actions.

Original: Rotman et al.
Revised: Rotman et al.

Original: [2020] investigate different UE methods for detection during deployment, which they call the online safety assurance problem (OSAP).
Revised: [2020] investigate different UE methods for detection during deployment, which they call the online safety assurance problem (OSAP).

Original: The method for policy-based UE via agent ensembles is also comparable to our baseline UE method.
Revised: The method for policy-based UE via agent ensembles is also comparable to our baseline UE method.

Original:  Background 3.1 Reinforcement Learning and MDP
Revised:  Background 3.1 Reinforcement Learning and MDP

Original: In Reinforcement Learning, we consider an agent that sequentially interacts with an environment modeled as a Markov Decision Process (MDP).
Revised: In Reinforcement Learning, we consider an agent that sequentially interacts with an environment modeled as a Markov Decision Process (MDP).

Original: An MDP is a tuple M :=
Revised: An MDP is a tuple M :=

Original: ( S , A , R, P, µ 0 ) composed by the set of states S , the set of actions A , the reward function R, the reward function R : S×A×S (cid:55)→ R , the transition probability function P :( S ×
Revised: ( S , A , R, P, µ 0 ) composed by the set of states S , the set of actions A , the reward function R : S × A × S (cid:55)→ R , the transition probability function P : S ×

Original: A × S (cid:55)→ [0 , 1] which describes the system dynamics, and the starting state distribution µ 0 .
Revised: A × S (cid:55)→ [0 , 1] which describes the system dynamics, and the starting state distribution µ 0 .

Original: The transition probability function P ( s t +1 | s t , a t ) models the probability of transitioning to a state s t +1 given a previous state s t and taking the action a t .
Revised: The transition probability function P ( s t +1 | s t , a t ) models the probability of transitioning to a state s t +1 given a previous state s t and taking the action a t .

Original: The reward function represents the return as sum of the discounted reward with γ k being the discount factor at time steps k , given by
Revised: The reward function represents the return as sum of the discounted reward with γ k being the discount factor at time steps k , given by

Original: [Equation]
Revised: [Equation]

Original: In the MDP framework, at each timestep, the agent observes the current state, takes an action, transitions to the next state drawn from the distribution, and receives a reward.
Revised: In the MDP framework, at each timestep, the agent observes the current state, takes an action, transitions to the next state drawn from the distribution, and receives a reward.

Original: The action-value function, also known as the Q-value function, where Q π represents the expected return when following a policy π (which basically maps states into actions, as shown below.
Revised: The action-value function, also known as the Q-value function, where Q π represents the expected return when following a policy π which maps states into actions, as shown in equation 2.

Original: [Equation]
Revised: [Equation]

Original: Q-learning, in which a policy is learned using Q-values, is a popular model-free method.
Revised: Q-learning, in which a policy is learned using Q-values, is a popular model-free method.

Original: Deep
Revised: Deep

Original: Q-networks (DQNs) are an extension of Q-learning that work with neural networks and optimization algorithms commonly used in deep learning, such as gradient descent.
Revised: Q-networks (DQNs) are an extension of Q-learning that work with neural networks and optimization algorithms commonly used in deep learning, such as gradient descent.

Original: To do so, the temporaldifferent error δ t can be derived from the Q-value function using the Bellman operator, resulting inthe equation below, whereas θ − and θ are the DQN parameters from the target and the prediction network, respectively.
Revised: To do so, the temporaldifference error δ t can be derived from the Q-value function using the Bellman operator, resulting in equation 3, whereas θ − and θ are the DQN parameters from the target and the prediction network, respectively.

Original: [Equation]
Revised: [Equation]

Original: 3.2 Distributional shift and OOD
Revised: 3.2 Distributional shift and OOD

Original: Distributional shift and OOD are two concepts that are closely related.
Revised: Distributional shift and OOD are two concepts that are closely related.

Original: For instance, Hendrycks and
Revised: For instance, Hendrycks and

Original: Gimpel [2018] correlate distributional shift to OOD by describing the latter as samples drawn from a different distribution than the training data.
Revised: Gimpel [2018] correlate distributional shift to OOD by describing the latter as samples drawn from a different distribution than the training data.

Original: When transferring this definition to RL problems, it is important to distinguish distributions that are closer or further away from the training distribution.
Revised: When transferring this definition to RL problems, it is important to distinguish distributions that are closer or further away from the training distribution.

Original: It is expected that an RL agent would be able to perform well in scenarios that are slightly different from the training ones, i.e., it should be able to generalize, but when the situation is too dissimilar (perhaps at a semantic level) the agent might have its ability to make proper decisions severely affected.
Revised: It is expected that an RL agent would be able to perform well in scenarios that are slightly different from the training ones, i.e., it should be able to generalize, but when the distributions are too dissimilar (perhaps at a semantic level) the agent might have its ability to make proper decisions severely affected.

Original: A model or agent which is not able to detect a distributional shift may tend to be overconfident in predictions on unseen data and can give unpredictable resultsfor far-from-distribution test data Lütjens et al.
Revised: A model or agent which is not able to detect a distributional shift may tend to be overconfident in predictions on unseen data and can give unpredictable results Lütjens et al.

Original: Some authors also show that epistemic uncertainty can be used as a proxy for detecting distributional shifts, e.g., Sedlmeier et al.
Revised: Some authors also show that epistemic uncertainty can be used as a proxy for detecting distributional shifts, e.g., Sedlmeier et al.

Original: Epistemic uncertainty is usually associated with a lack of sufficient data to better infer the underlying system.
Revised: Epistemic uncertainty is usually associated with a lack of sufficient data to better infer the underlying system.

Original: Clements et al.
Revised: Clements et al.

Original: [2020] claim that epistemic uncertainty stems from limited data.
Revised: [2020] claim that epistemic uncertainty stems from limited data.

Original: However, besides detecting a model’s lack of generalization to data close to it epistemic uncertainty can also point to samples that reside far away from the data distribution known to the trained model, as shown in Postels et al.
Revised: However, besides detecting a model’s lack of generalization to data close to it, epistemic uncertainty can also point to samples that reside far away from the data distribution known to the trained model, as shown in Postels et al.

Original: Defining distributional shift within the RL domain is not trivial Haider et al.
Revised: Defining distributional shift within the RL domain is not trivial Haider et al.

Original: In this paper we assume that distributional shift can be characterized by changes in the system dynamics, more specifically the shift of the distribution over the state transitions given state action pairs between training and test in MDPs, as shown below:
Revised: In this paper we assume that distributional shift can be characterized by changes in the system dynamics, more specifically the shift of the distribution over the state transitions given state action pairs between training and test in MDPs, as shown in equation 4.

Original: [Equation]
Revised: [Equation]

Original: Additionally, when considering partially observable MDPs (POMDPs) where the system’s state cannot be assessed but rather an observation o t is available to the agent, the shift of the distribution over observations given states has to be taken into account:
Revised: Additionally, when considering partially observable MDPs (POMDPs) where the system’s state cannot be assessed but rather an observation o t is available to the agent, the shift of the distribution over observations given states has to be taken into account (equation 5).

Original: [Equation]
Revised: [Equation]

Original: 3.3 Ensemble based uncertainty estimation
Revised: 3.3 Ensemble based uncertainty estimation

Original: As discussed previously, uncertainty estimation often is the utilized method to detect a distributional shift.
Revised: As discussed previously, uncertainty estimation often is the utilized method to detect a distributional shift.

Original: Detecting a distributional shift for the sake of exploration during training overlaps with the concept of novelty detection.
Revised: Detecting a distributional shift for the sake of exploration during training overlaps with the concept of novelty detection.

Original: In contrast, applying distributional shift detection during test time is considered as OOD detection in the classical image classification domain, where OOD describes the distributional shift of the data distribution between training and test.
Revised: In contrast, applying distributional shift detection during test time is considered as OOD detection in the classical image classification domain, where OOD describes the distributional shift of the data distribution between training and test.

Original: This paper focuses on ensemble-based epistemic uncertainty estimation to detect distributional shift and OOD data during test time.
Revised: This paper focuses on ensemblebased epistemic uncertainty estimation to detect distributional shift and OOD data during test time.

Original: An ensemble oftrained agents  ona subset of the available data will estimate with little variance  in well-trained states.
Revised: An ensemble of agents trained on the available data will estimate with little variance between the ensemble members in well-trained states.

Original: When the ensemble members face little trained states, the estimates vary naturally across the members and give a distribution over the estimated Q-values.
Revised: When the ensemble members face too few trained states, the estimates vary naturally across the members and give a distribution over the estimated Q-values.

Original: The variance of the estimated Q-values can be used to quantify the epistemic uncertainty of a decision.
Revised: The variance of the estimated Q-values can be used to quantify the epistemic uncertainty of a decision.

Original: An ensemble of bootstrapped data over DQNs provides a distribution over the estimated Q-values to provide a Bayesian estimation of the epistemic uncertainty.
Revised: An ensemble of bootstrapped data over DQNs provides a distribution over the estimated Q-values to provide a Bayesian estimation of the epistemic uncertainty.

Original: The Q-values will converge to the real values  in situations the agent learned sufficiently.
Revised: The Q-values will converge to the real values of the underlying function in situations the agent learned sufficiently well.

Original: In untrained situations, the Q-value estimates will still diverge and the variance will therefore give an estimate of the epistemic uncertainty.
Revised: In untrained situations, the Q-value estimates will still diverge and the variance will therefore give an estimate of the epistemic uncertainty.

Original: In Osband et al.
Revised: In Osband et al.

Original: [2018]they use random prior functions  to introduce diversity in an ensemble of agents trained on bootstrapped data.
Revised: [2018] random prior functions are used to introduce diversity in an ensemble of agents trained on bootstrapped data.

Original: The expected return is then given by
Revised: The expected return is then given by

Original: [Equation]
Revised: [Equation]

Original: where Q k is the Q-function of the k th ensemble member, ˆ θ are the parameters of the prior function and β is a factor to weight the impact of the prior function.
Revised: where Q k is the Q-function of the k th ensemble member, ˆ θ are the parameters of the prior function and β is a factor to weight the impact of the prior function.

Original: In Hoel et al.
Revised: In Hoel et al.

Original: [2020] and Hoel et al.
Revised: [2020] and Hoel et al.

Original: [2021] they use the variance of the Q-values of the ensemble estimate to derive an uncertainty estimation threshold to invoke a backup policy.
Revised: [2021] they use the variance of the Q-values of the ensemble estimate to derive an uncertainty estimation threshold to invoke a backup policy.

Original: With the variance
Revised: With the variance

Original: V ar k
Revised: V ar k

Original: [ Q k ( s, a )] < σ 2 the policy with threshold can be calculated by
Revised: [ Q k ( s, a )] < σ 2 the policy with threshold can be calculated by

Original: [Equation]
Revised: [Equation]

Original:  E k [ Q k ( s, a )] if V ar k [ Q k ( s, a )] < σ 2 , π backup ( s ) otherwise . 4.1 Action count variance uncertainty estimation
Revised:  Ensemble Uncertainty estimation based on action count variance and delta to ID 4.1 Action count variance uncertainty estimation

Original: The Q-value is a continuous variable where high variance in the predictions means high uncertainty of the ensemble.
Revised: The Q-value is a continuous variable where high variance in the predictions means high uncertainty of the ensemble.

Original: However, when given encapsulated agents or due to other reasons, where the Q-values are not accessible, it is possible to use the action prediction result from the ensemble members and derive an uncertainty estimate.
Revised: However, when given encapsulated agents or when the Q-values are not accessible due to other reasons, it is possible to take the deviation over the proposed actions of the ensemble members, to indicate uncertainty.

Original: Instead, it is possible to take the deviation over the proposed actions of an ensemble, to indicate uncertainty.

Original: In cases where the action space is continuous, the variance can be directly calculated as action variance like with the Q-values.
Revised: In cases where the action space is continuous, the variance can be directly calculated as action variance like with the Q-values.

Original: However, with discrete action spaces, this will lead to false results, as the actions themselves are orthogonal and e.g. a mean action can not be calculated.
Revised: However, with discrete action spaces, this will lead to false results, as the actions themselves are orthogonal and e.g. a mean action can not be calculated.

Original: Therefore, in cases where the action space is discrete, we propose to calculate an action count on each action over the ensemble given a certain state and then calculate the variance of that action count (ACV - action count variance).
Revised: Therefore, in cases where the action space is discrete, we propose to calculate an action count on each action over the ensemble given a certain state and then calculate the variance of that action count (ACV - action count variance).

Original: When the ACV is low,there is a balance in the proposed different actions  over the ensemble and the uncertainty is therefore high.
Revised: When the ACV is low, the proposed different actions are balanced equally over the ensemble and the uncertainty is therefore high.

Original: In contrast, when the action count variance is high, there is a concentration of one or more actions in the ensemble and the uncertainty is low.
Revised: In contrast, when the action count variance is high, there is a concentration of one or more actions in the ensemble and the uncertainty is low.

Original: The higher the ACV gets, the lower the uncertainty.
Revised: The higher the ACV gets, the lower the uncertainty.

Original: Given N actions as the number of alternative actions and K ensemble as the number of ensemble members and  8 the minimum of the ACV gets to equation 9 and the maximum to equation 10.
Revised: Given N actions as the number of alternative actions and K ensemble as the number of ensemble members and equation 8 the minimum of the ACV gets to equation 9 and the maximum to equation 10.

Original: [Equation]
Revised: [Equation]

Original: A backup policy can then be chosen based on the ACV calculation like given in equation 11.
Revised: A backup policy can then be chosen based on the ACV calculation as given in equation 11.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 4.2 Delta to ID uncertainty estimation
Revised: 4.2 Delta to ID uncertainty estimation

Original: One problem with uncertainty estimation within reinforcement learning is the fact, that thereoften are  alternative decisions to take in an MDP.
Revised: One problem with uncertainty estimation in reinforcement learning is the fact, that there are often alternative decisions to take in an MDP.

Original: These can be called alternative possible actions.
Revised: These can be called alternative possible actions - or more generally alternative predictions.

Original: When an agent is in a state with alternative possible actions, the ensemble may already deviate in its prediction, although  trained sufficiently in this state (ID - in-distribution).
Revised: When an agent is in a state with alternative possible actions, the ensemble may already deviate in its prediction, although it might be trained sufficiently in this state.

Original: This means alternative possible actions will sort of overlay the uncertainty from a lack of training (OOD).
Revised: This means alternative possible actions will sort of overlay the uncertainty from a lack of training (OOD).

Original: Given an overlay of such kind will make it hard to decide if real uncertainty, only alternative possible actions or both are in place.
Revised: Given an overlap of such kind will make it hard to decide if real uncertainty, only alternative possible actions or both are present.

Original: Therefore, we propose to "compare" the given (potentially OOD) situation to the nearest ID situation (IDD - ID delta), to identify "normal" deviation versus uncertainty deviation.
Revised: Therefore, we propose to compare the given (potentially OOD) situation to a nearest ID situation (IDD for ID Delta), to identify normal deviation versus uncertainty deviation.

Original: To get a comparison, we propose to subtract ID ACV from the given AVC and use the result as a cleaned version of the ACV for uncertainty indication.
Revised: To get a comparison, we propose to subtract ID ACV from the given ACV and use the result as a cleaned version of the ACV for uncertainty indication.

Revised: Because of the (1-x) characteristic of the ACV to the uncertainty, we actually subtract ( Const maxV ar − ACV OOD ) − ( Const maxV ar − ACV ID ) which inverses the ACV characteristic to match the uncertainty’s and results for the subtraction in equation 12.

Original: [Equation]
Revised: [Equation]

Original: There can be different approaches to obtaining the nearest ID  from a given OOD scenario.
Revised: There can be different approaches to obtain a nearest ID scenario from a given OOD scenario.

Original: To simplify, we stick to an OOD scenario with one dedicated OOD Object. We propose two variations as a straightforward approach.
Revised: To simplify, we stick to an OOD scenario with one dedicated OOD object and propose two variations as a straightforward approach.

Original: At first, the OOD object in the given OOD scenario(observation) will be exchanged with a corresponding IDobject - like a known obstacle.
Revised: On the one hand, the OOD obstacle in a given OOD scenario will be exchanged with an ID obstacle.

Original: Then, the given OOD object willjust be removed from the scenario without replacement - or with zero replacement, so to say. [Equation]
Revised: On the other hand, the given OOD obstacle will be removed from the scenario without a replacement. The observation functions Obs () change as given in equationsand 14.

Original: [Equation]
Revised: [Equation]

Original: A high delta of the ACV will indicate high uncertainty and a low delta is associated with low uncertainty in both cases, respectively.
Revised: A high delta of the ACV will indicate high uncertainty and a low delta is associated with low uncertainty in both cases, respectively.

Original: To decide on an uncertain situation in a given state, we propose to use a threshold to mask out insignificant variance-delta to ID.
Revised: To decide on an uncertain situation in a given state, we propose to use a threshold to mask out insignificant variance-delta to ID.

Original: This threshold can be used in future work to switch to a backup policy.
Revised: This threshold can be used in future work to switch to a backup policy.

Original:  Experimental results 5.1 Setup and Training
Revised:  Experimental results 5.1 Setup and training

Original: We trained complete agents in parallel witha randomly placed set of 10 hazards singly placed in a gridwold of 10x10 positions and training runs of 1 million steps for each agent.
Revised: We trained complete agents in parallel with 10 obstacles randomly placed in a gridworld of 10x positions and training runs of 1 million steps for each agent.

Original: For testing, we set up different scenarios with hazards used for training as ID and added a single novel obstacle as ODD.
Revised: For testing, we set up different scenarios with obstacles used for training as ID and added a single novel obstacle as ODD.

Original: For the results shown below, we focused solely on ID scenarios with a line of known hazards in the middle of the grid and the goal at the end of the line.
Revised: For the results shown below, we focused solely on ID scenarios with a line of known obstacles in the middle of the grid and the goal at the end of the line.

Original: For the visualization of the uncertainty estimation, we calculated heatmaps over the grid showing each resulting uncertainty estimation for each position of the agent in the grid given the overall scenario.
Revised: For the visualization of the uncertainty estimation, we calculated heatmaps over the grid showing each resulting uncertainty estimation for each position of the agent in the grid given the overall scenario.

Original: 5.2 Uncertainty heatmaps
Revised: 5.2 Uncertainty heatmaps

Original: For the depicted results the uncertainty calculation based on the action count variance of the ensemble members is used.
Revised: For the shown results the uncertainty calculation based on the action count variance of the ensemble members is used.

Original: Figure 1 is the base scenario with the known ID hazard line in blue and the goal in green.
Revised: Figure 1 shows the base scenario with the known ID obstacle line in blue and the goal in green.

Original: As we use variance in the action count, ahigher brighter color means more concentration on fewer actions (and therefore more certainty) and darker color means less concentration in the actions or more equally distributed actions (and therefore higher uncertainty.
Revised: As we use variance in the action count, a brighter color means more concentration on fewer actions and therefore more certainty and darker color means less concentration in the actions or more equally distributed actions and therefore higher uncertainty.

Original: Asone can see in the base scenario- due to possible alternative action predictions there are some "uncertainties" along the diagonals to the goal, as these coordinates have equal probabilities vertically and horizontally to approach the goal.
Revised: As can be seen in the base scenario there are some "uncertainties" along the diagonals to the goal due to possible alternative action predictions, as these coordinates have equal probabilities vertically and horizontally to approach the goal.

Original: This applies e.g. also for the point on the left of the hazard line, as the probabilities for up and down are equally distributed.
Revised: This applies e.g. also for the point on the left of the obstacle line, as the probabilities for up and down are equally distributed.

Original: Figure 2a shows the predictions with one unknown obstacle inserted in the middle direct on top of the line shown in purple.
Revised: Figure 2a shows the predictions with one unknown obstacle inserted in the middle direct on top of the line shown in purple.

Original: One can see the tendency ofmany of the areas, and especially the area around the unknown obstacle, to shown an increase in uncertainty.
Revised: One can see the tendency of areas, and especially the area around the unknown obstacle, to shown an increase in uncertainty.

Original: Nevertheless, the uncertainty indication is superposed by the already given "uncertainty" of the possible alternative predictions from the baseline ID scenario.
Revised: Nevertheless, the uncertainty indication is superposed by the already given "uncertainty" of the possible alternative predictions from the baseline ID scenario.

Original: In contrast, figure 2b shows the predictions with a known obstacle inserted in the middle direct on top of the line shown in blue, instead of the OOD obstacle.
Revised: In contrast, figure 2b shows the predictions with a known obstacle inserted in the middle direct on top of the line shown in blue, instead of the OOD obstacle.

Original: Now, the uncertainty indication is nearer to the baseline ID scenario.
Revised: Now, the uncertainty indication is closer to the base ID scenario.

Original: Our idea presented in 4.2 proposes to subtract the baseline variance from the OOD variance and therefore try to eliminate the baseline variance resulting from the possible alternative predictions.
Revised: Our idea presented in 4.2 proposes to subtract the baseline variance from the OOD variance and therefore try to eliminate the baseline variance resulting from the possible alternative predictions.

Original: In Figure 3 the results are depicted for the delta to the ID baseline without and with a threshold (3a and  3b) and the delta to ID with the known obstacle (3c and 3d).
Revised: In figure 3 the results are depicted for the delta to the ID baseline without and with a threshold (3a and [Figure] 3b) and the delta to ID with the known obstacle (3c and 3d).

Original: It seems feasible to indicate the given OOD hotspot when masking out a dedicated threshold, like given in 3b and 3d, although the indication is not totally sharp.[Figure]
Revised: It seems feasible to indicate the given OOD hotspot when masking out a dedicated threshold, as given in 3b and 3d, although the indication is not totally sharp.

Original: [Figure]
Revised: [Figure]

Original: In the given scenario, the OOD hotspot is lying direct in an area of low uncertainty (the yellow area on top of the blue line).
Revised: In the given scenario, the OOD hotspot is lieing direct in an area of low uncertainty (the yellow area on top of the blue line).

Original: One can think of a special situation, where the ID subtraction is convenient to isolate the hotspot.
Revised: One can think of a special situation, where the ID subtraction is convenient to isolate the hotspot.

Original: Therefore, we also ran setups, where the hotspot is lying in an area of already given "uncertainty" from possible alternative predictions- like in the upper middle section.
Revised: Therefore, we also ran setups, where the hotspot is lieing in an area of already given "uncertainty" from possible alternative predictions like in the upper middle section.

Original: See Figure 4.[Figure] [Figure]
Revised: The results are depicted in figure 4.

Original: And Figure 5 shows the resulting indication for the delta OOD to ID baseline in 5a and 5b- and the delta to ID with known hazard in 5c and 5d.
Revised: Figure 5 shows the resulting indication for the delta OOD to ID without obstacle in 5a and 5b and the delta to ID with known obstacle in 5c and 5d.

Original: Again, the given threshold seems to isolate the hotspot from the OOD obstacle quite well.
Revised: The given threshold seems to isolate the hotspot from the OOD obstacle quite well.

Original: But here, the delta to ID baseline fails, as the top and left fields from the obstacle are not indicated.
Revised: But here, the delta to ID without obstacle fails, as the top and left fields from the obstacle are not indicated as uncertain.

Original: The already higher "uncertainty" from the possible alternative predictions in that area subtracts out the indication.
Revised: The already given "uncertainty" from the possible alternative predictions in that area subtracts out the indication.

Original: But actually, these fields should be indicated - because they lie in the diagonal path to the goal and would therefore be conflicting due to the unknown obstacle.
Revised: Indeed, these fields should be indicated - because they lie in the diagonal path to the goal and would therefore be conflicting due to the unknown obstacle.

Original: In contrast, the delta to ID with known obstacle replacement performs better and indicates the respective fields. 
Revised: In contrast, the delta to ID with known obstacle replacement performs better and indicates the respective fields. [Figure] [Figure] [Figure]

Revised: [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: Within Reinforcement Learning for a decision-making agent aleatoric and epistemic uncertainty can be distinguished. Aleatoric uncertainty refers to the inherent randomness of an outcome whereas epistemic uncertainty arises due to a lack of knowledge.
Revised: Within Reinforcement Learning for a decision-making agent aleatoric and epistemic uncertainty can be distinguished, whereas epistemic uncertainty arises due to a lack of knowledge.

Original: In this paper, we investigated ensemble-based epistemic uncertainty estimation on gridworld scenarios with discrete action spaces and overlapping alternative predictions.
Revised: In this paper, we investigated ensemble-based epistemic uncertainty estimation on gridworld scenarios with discrete action spaces and overlapping alternative predictions.

Original: We exhibited the problem obstacle with discrete action spaces and variance calculation and proposed to use action count variance (ACV) instead of action variance.
Revised: We exhibited the problem obstacle with discrete action spaces and variance calculation and proposed to use action count variance (ACV) instead of action variance.

Original: We depicted the conflict of uncertainty estimation due to a lack of training with overlapping alternative solutions.
Revised: We depicted the conflict of uncertainty estimation due to a lack of training and due to overlapping alternative solutions.

Original: As a viable solution, we proposed to calculate the delta to ID (IDD) for the action count variance and utilized a threshold to indicate uncertainty due to a lack of training, only.
Revised: As a viable solution, we proposed to calculate the delta to ID (IDD) for the action count variance and utilized a threshold to indicate uncertainty due to a lack of training, only.

Original: With experiments utilizing representative gridworld test scenarios, we could show that action count variance with delta to ID is able to indicate uncertain states based on a threshold calculation with high probability.
Revised: With experiments utilizing representative gridworld test scenarios, we showed that action count variance with delta to ID is able to indicate uncertain states based on a threshold calculation with high probability.

Original: Therefore, a decision for a backup policy based on that indication seems to be a feasible solution.
Revised: Therefore, a decision for a backup policy based on that indication can be a feasible solution.

Original: In future work, we want to investigate feasible methods to determine a sufficient near ID scenario for a given OOD scenario and extend the approach to more general and realistic environments.
Revised: Future work can investigate feasible methods to determine a sufficient near ID scenario for a given OOD scenario and extend the approach to more general and realistic environments.

Original: We alsowant to focus on strategies to deal with an uncertainty indication and compare the resulting performance to other given approaches.
Revised: It can also focus on strategies to deal with an uncertainty indication and compare the resulting performance to other given approaches.

Revised: Acknowledgement: This work was funded by the Bavarian Ministry for Economic Affairs, Regional Development and Energy as part of a project to support the thematic development of the

Revised: Institute for Cognitive Systems.