Information-Theoretic Safe Exploration with Gaussian Processes
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We consider a sequential decision making task where we are not allowed to evaluate parameters that violate an a priori unknown (safety) constraint.
A commonapproach is to place a Gaussian process prior on the unknown constraint and allow evaluations only in regions that are safe with high probability.
Most currentmethods rely on a discretization of the domain and cannot be directly extendedto the continuous case.
Moreover, the way in which they exploit regularity assumptions about the constraint introduces an additional critical hyperparameter.
In this paper, we propose an information-theoretic safe exploration criterion thatdirectly exploits the GP posterior to identify the most informative safe parametersto evaluate.
Our approach is naturally applicable to continuous domains and doesnot require additional hyperparameters.
We theoretically analyze the method andshow that we do not violate the safety constraint with high probability and thatwe explore by learning about the constraint up to arbitrary precision.
Empiricalevaluations demonstrate improved data-efﬁciency and scalability.
Introduction
In sequential decision making problems, we iteratively select parameters in order to optimize a givenperformance criterion.
However, real-world applications such as robotics (Berkenkamp et al., 2021),mechanical systems (Schillinger et al., 2017) or medicine (Sui et al., 2015) are often subject toadditional safety constraints that we are not allowed to violate during the exploration process (DulacArnold et al., 2019).
Since it is a priori unknown which parameters lead to constraint violations, weneed to actively and carefully learn about the constraints without ever violating them.
That is, weneed to learn about the safety of parameters by only evaluating parameters that are currently knownto be safe.
Existing methods by Schreiter et al. (2015);
Sui et al.
(2015) tackle this problem by placing a Gaussianprocess (GP) prior over the constraint and only evaluate parameters that do not violate the constraintwith high probability.
To learn about the safety of parameters, they evaluate the constraint at theparameter with the largest posterior variance.
This process is made more efﬁcient by S AFE O PT ,which restricts exploration to parameters that are close to the boundary of the current set of safeparameters (Sui et al., 2015) at the cost of an additional tuning hyperparameter (Lipschitz constant).
However, uncertainty about the constraint is only a proxy objective, which only indirectly learnsabout the safety of parameters.
Consequently, data-efﬁciency could be improved with an explorationcriterion that directly maximizes the information gained about the safety of other parameters.
Our contribution
In this paper, we propose Information-Theoretic Safe Exploration (ISE), a safeexploration algorithm that directly exploits the information gain about the safety of parameters inorder to expand the region of the parameter space that we can classify as safe with high conﬁdence.
By directly optimizing for safe information gain, ISE is more data-efﬁcient than existing approaches without manually restricting evaluated parameters to be on the boundary of the safe set.
Thisexploration criterion also means that we do not require additional hyperparameters beyond the GPposterior and that ISE is directly applicable to continuous domains.
We theoretically analyze ourmethod and prove that it learns about the safety of reachable parameters to arbitrary precision.
Related work
Information-based selection criteria with Gaussian processes models are successfully used in the context of unconstrained Bayesian optimization (BO, Shahriari et al.
; Bubeckand Cesa-Bianchi (2012)), where the goal is to ﬁnd the parameters that maximize an a priori unknownfunction.
Hennig and Schuler (2012); Henrández-Lobato et al.
; Wang and Jegelka (2017)select parameters that provide the most information about the optimal parameters, while Fröhlich et al.(2020) consider the information under noisy parameters.
The success of these information-basedapproaches also relies in the superior data efﬁciency that they demonstrated.
We draw inspirationfrom these methods when deﬁning an information-based criterion w.r.t.
the safety of parameters toguide safe exploration.
In the presence of constraints that the ﬁnal solution needs to satisfy, but which we can violate duringexploration, Gelbart et al.
(2014) propose to combine typical BO acquisition functions with theprobability of satisfying the constraint.
Instead, Gotovos et al.
(2013) propose an uncertainty-basedcriterion that learns about the feasible region of parameters.
When we are not allowed to ever evaluateunsafe parameters, safe exploration is a necessary sub-routine of BO algorithms to learn about thesafety of parameters.
To safely explore, Schreiter et al.
(2015) globally learn about the constraintby evaluating the most uncertain parameters.
S AFE O PT by Sui et al.
(2015) extends this to jointexploration and optimization and makes it more efﬁcient by explicitly restricting safe exploration tothe boundary of the safe set.
Sui et al.
(2018) proposes S TAGE O PT , which additionally separates theexploration and optimization phases.
Both of these algorithms assume access to a Lipschitz constantto deﬁne parameters close to the boundary of the safe set, which is a difﬁcult tuning parameter inpractice.
These methods have been extended to multiple constraints by Berkenkamp et al.
(2021),while Kirschner et al.
(2019) scale them to higher dimensions with L INE BO, which explores inlow-dimensional sub-spaces.
To improve computational costs, Duivenvoorden et al.
(2017) suggesta continuous approximation to S AFE O PT without providing exploration guarantees.
All of thesemethods rely on function uncertainty to drive exploration, while we directly maximize the informationgained about the safety of parameters.
Safe exploration also arises in the context of Markov decision processes (MDP), (Moldovan and
Abbeel, 2012; Hans et al., 2008).
In particular, Turchetta et al.
(2019) traverse the MDP to learn aboutthe safety of parameters selected by some oracle. At its core, their method explores using the sameideas as S AFE O PT and S TAGE O PT to select parameters to evaluate.
Consequently, our proposedmethod for safe exploration is also directly applicable to their setting.
Problem Statement
In this section, we introduce the problem and notation that we use throughout the paper.
We aregiven an unknown and expensive to evaluate safety constraint f : X → R s.t. parameters that satisfyf ( x ) ≥ 0 are classiﬁed as safe, while others are unsafe.
To start exploring safely, we also have accessto an initial safe parameter x 0 that satisﬁes the safety constraint, f ( x 0 ) ≥ 0 .
We sequentially selectsafe parameters x n ∈ X where to evaluate f in order to learn about the safety of parameters beyondx 0 .
At each iteration n , we obtain a noisy observation of y n := f ( x n ) + ν n that is corrupted byadditive homoscedastic Gaussian noise ν n ∼ N (cid:0) 0 , σ 2 ν (cid:1) .
We illustrate the task in Figure 1a, wherestarting from x 0 we aim to safely explore the domain so that we ultimately classify as safe all thesafe parameters that are reachable from x 0 .
As f is unknown and the evaluations y n are noisy, it is not feasible to select parameters that are safewith certainty and we provide high-probability safety guarantees instead.
To this end, we assumethat the safety constraint f has bounded norm in the Reproducing Kernel Hilbert Space (RKHS)(Schölkopf and Smola, 2002)
H k associated to some kernel k : X × X
→ R with k ( x , x (cid:48) ) ≤ 1 .
Thisassumption allows us to to model f as a Gaussian process (GP) (Srinivas et al., 2010).
Gaussian Processes
A GP is a stochastic process speciﬁed by a mean function µ : X → R anda kernel k (Rasmussen and Williams, 2006).
It deﬁnes a probability distribution over real-valuedfunctions on X , such that any ﬁnite collection of function values at parameters [ x 1 , . . .
, x n ] isdistributed as a multivariate normal distribution.
The GP prior can then be conditioned on (noisy) [Figure] function evaluations D n = { ( x i , y i ) } ni =1 .
If the noise is Gaussian, then the resulting posterior is alsoa GP and with posterior mean and variance
[Equation]
where µ :=
[ µ ( x 1 ) , . . .
µ ( x n )] is the mean vector at parameters x i ∈ D n and [ y ] i :=
y ( x i ) thecorresponding vector of observations.
We have (cid:2) k
( x ) (cid:3) i :
= k ( x , x i ) , the kernel matrix has entries[ K ] ij :
= k
( x i , x j ) , and I is the identity matrix.
Safe set
Using the previous assumptions, we can construct high-probability conﬁdence intervalson the function values f ( x ) .
Concretely, for any δ > 0 it is possible to ﬁnd a sequence of positivenumbers { β n } such that f ( x ) ∈ (cid:2) µ n ( x ) ± β n σ n ( x ) (cid:3) with probability at least 1 − δ , jointly for allx ∈ X and n ≥ 1 .
For a proof and more details see (Chowdhury and Gopalan, 2017).
We use theseconﬁdence intervals to deﬁne a safe set
[Equation]
which contains all parameters whose β n -lower conﬁdence bound is above the safety threshold andthe initial safe parameter x 0 .
Consequently, we know that all parameters in S n are safe, f ( x ) ≥ 0for all x ∈ S n , with probability at least 1 − δ jointly over all iterations n .
Safe exploration
Given the safe set S n , the next question is which parameters in S n to evaluate inorder to efﬁciently expand it.
Most existing safe exploration methods rely on uncertainty samplingover subsets of S n .
S AFE O PT -like approaches, for example, use the Lipschitz assumption on f toidentify parameters in S n that could expand the safe set, and then select the parameter that has thebiggest uncertainty among those.
In the next sections, we present and analyze our safe explorationstrategy, ISE, that instead uses an information gain measure to identify the parameters that allow usto efﬁciently learn about the safety of parameters outside of S n .
Information-Theoretic Safe Exploration
We present Information-Theoretic Safe Exploration (ISE), which guides the safe exploration by usingan information-theoretic criterion.
Our goal is to design an exploration strategy that directly exploitsthe properties of GPs to learn about the safety of parameters outside of S n .
We draw inspiration from
Hennig and Schuler (2012); Wang and Jegelka (2017), who exploit information-theoretic insights todesign data-efﬁcient BO acquisition functions for their respective optimization objectives.
Information gain measure
In our case, we want to evaluate f at safe parameters that are maximally informative about the safety of other parameters, in particular of those where we are uncertainabout whether they are safe or not.
To this end, we need a corresponding measure of information
1: Input: GP prior ( µ 0 , k , σ ν ), Safe seed x2: for n = 0 , ... do 3: x
[Equation]
← arg
[Equation]
z ) [Equation] 4: y n +1 ← f ( x n +1 ) + ν 5:
Update GP posterior with ( x n +1 , y n +1 ) gain.
We deﬁne such a measure using the binary variable Ψ( x ) =
I { f ( x ) ≥ 0 } , which is equal to oneiff f ( x ) ≥ 0 .
Its entropy is given by
[Equation]
3) where p − n ( z ) is the probability of z being unsafe: p − n ( z ) =
12 + 12 erf (cid:16) − [Equation] ( z ) (cid:17) .
The randomvariable Ψ( z ) has high-entropy when we are uncertain whether a parameter is safe or not; that is, itsentropy decreases monotonically as |
µ n ( z ) | increases and the GP posterior moves away from thesafety threshold, and also decreases monotonically as σ n ( z ) decreases and we become more certainabout the constraint.
This behaviour also implies that the entropy goes to zero as the conﬁdence aboutthe safety of z increases, as desired.
Given our deﬁnition of Ψ , we consider the mutual information
I (cid:0) { x , y } ; Ψ( z ) (cid:1) between an observation y at a parameter x and the value of Ψ at another parameter z .
Since Ψ is the indicatorfunction of the safe regions of the parameter space, the quantity I (cid:0) { x , y } ; Ψ( z ) (cid:1) measures howmuch information about the safety of z we gain by evaluating the safety constraint f at x , averagedover all possible observed values y .
This interpretation follows directly from the deﬁnition of mutualinformation: I (cid:0) { x , y } ; Ψ( z ) (cid:1)
=
H n (cid:2) Ψ( z ) (cid:3) − E y (cid:104) H n +1 (cid:2) Ψ( z ) (cid:12)(cid:12) { x , y } (cid:3)(cid:105) , where H n [Ψ( z )] is theentropy of Ψ( z ) according to the GP posterior at iteration n , while H n +1 (cid:2) Ψ( z ) (cid:12)(cid:12) { x , y } (cid:3) is its entropyat iteration n + 1 , conditioned on a measurement y at x at iteration n .
Intuitively, I (cid:0) { x , y } ; Ψ( z ) (cid:1)is negligible whenever the conﬁdence about the safety of z is high or, more generally, whenever anevaluation at x does not have the potential to substantially change our belief about the safety of z .
The mutual information is large whenever an evaluation at x on average causes the conﬁdence aboutthe safety of z to increase signiﬁcantly.
As an example, in Figure 1 we plot I (cid:0) { x , y } ; Ψ( z ) (cid:1) as afunction of x ∈ S n for a speciﬁc choice of z and for an RBF kernel.
As one would expect, we seethat the closer it gets to z , the bigger the mutual information becomes, and that it vanishes in theneighborhood of previously evaluated parameters, where the posterior variance is negligible.
To compute I (cid:0) { x , y } ; Ψ( z ) (cid:1) , we need to average (3) conditioned on an evaluation y over all possiblevalues of y .
However, the resulting integral is intractable given the expression of H n [Ψ( z )] in (3).
Inorder to get a tractable result, we derive a close approximation of (3), which recovers almost exactlyits true behavior (see Appendix B for details),
[Equation]
Since the posterior mean at z after an evaluation at x depends linearly on µ n ( x ) , and since theprobability density of y depends exponentially on − µ 2 n ( x ) , using (4) reduces the conditional entropy
E y (cid:104) H n +1 (cid:2)
Ψ( z ) (cid:12)(cid:12) { x , y } (cid:3)(cid:105) to a Gaussian integral with the exact solution
[Equation]
where ρ n ( x , z ) is the linear correlation coefﬁcient between f ( x ) and f ( z ) , and with c 1 and c 2 givenby c 1 := 1 / ln(2) π and c 2 :
= 2 c 1 − 1 .
In the following, I n (cid:0) { x , y } ; Ψ( z ) (cid:1) refers to the mutualinformation based on (4) and (5) and all our theoretical results in Section 4 are proven for theseexpressions.
[Figure]
ISE acquisition function
Now that we have deﬁned a way to measure and compute the informationgain about the safety of parameters, we can use it to design an exploration strategy that selects thenext parameters to evaluate.
The natural choice for such selection criterion is to select the parameterthat maximizes the information gain; that is, we select x n +1 according to [Equation]
[Equation]
where we jointly optimize over x in the safe set S n and an unconstrained second parameter z .
Evaluating f at x n +1 according to (6) maximizes the information gained about the safety of someparameter z ∈ X , so that it allows us to efﬁciently learn about parameters that are not yet known tobe safe.
While z can lie in the whole domain, the parameters where we are the most uncertain aboutthe safety constraint lie outside the safe set.
By leaving z unconstrained, we show in our theoreticalanalysis in Section 4 that once we have learned about the safety of parameters outside the safe set, (6)resorts to learning about the constraint function also inside S n .
An overview of ISE can be foundin Algorithm 1 and we show an example run of a one-dimensional illustration of the algorithm in
Figure 2.
Theoretical Results
In this section, we study the expression for I n (cid:0) { x , y } ; Ψ( z ) (cid:1) obtained using (4) and (5) and analyzethe properties of the ISE exploration criterion (6).
By construction of S n in (2) and the assumptionson f in Section 2, we know that any parameter selected according to (6) is safe with high probability,see Appendix A for details.
To show that we also learn about the safe set, we ﬁrst need to deﬁnewhat it means to successfully explore starting from x 0 .
The main challenge is that it is difﬁcult toanalyze how a GP generalizes based on noisy observations, so that it is difﬁcult to deﬁne a notionof convergence that is not dependent on the speciﬁc run.
In S AFE O PT , they avoid this issue byexpanding the safe set not based on the GP, but only using the Lipschitz constant L .
Contrary to theirapproach, we depend on the GP to generalize from the safe set.
In this case, the equivalent naturalnotion of convergence is provided by the the posterior variance.
In particular, we say that at iterationn we have explored the safe set up to ε -accuracy if σ 2 n ( x ) ≤ ε for all parameters x in S n .
In thefollowing, we show that ISE asymptotically leads either to convergence or to indeﬁnite expansion ofthe safe set.
The proofs can be found in Appendix A.
Theorem 1.
Assume that x n +1 is chosen according to (6) , and that there exists ˆ n such that for alln ≥ ˆ n S n +1 ⊆ S n .
Moreover, assume that for all n ≥ ˆ n , | µ n ( x ) | ≤ M for some M > 0 for allx ∈ S n .
Then, for all ε > 0 there exists N ε such that σ 2 n ( x ) ≤ ε for every x ∈ S n if n ≥ ˆ n + N ε .
The smallest of such N ε is given by
[Equation]
where F ( η ) : = ln(2) exp (cid:110) − c 1 M 2 η (cid:111)(cid:20) 1 − (cid:113)
[Equation]
(cid:21) , γ N = max D
[Equation]
; | D | = N I (cid:0) f
( D ); y ( D ) (cid:1) isthe maximum information capacity of the chosen kernel (Srinivas et al., 2010; Contal et al., 2014),and C = ln(2) /σ 2 ν ln (cid:0) 1 + σ − 2 ν (cid:1) .
Theorem 1 tells us that if at some point the set of safe parameters S n stops expanding, then theposterior variance over the safe set vanishes eventually.
The intuition behind Theorem 1 is that if therewere a parameter x in the safe set whose posterior mean remained ﬁnite and whose posterior varianceremained bounded from below, then an evaluation of f at such x would yield a non negligible averageinformation gain about the safety of x , so that, since x is in the safe set, at some point ISE will beforced to choose to evaluate x , reducing its posterior variance.
This intuition also offers an insightabout how to interpret the result: although Theorem 1 guarantees that ISE asymptotically exploresthe safe set up to an arbitrary ε -accuracy, in practice it will ﬁrst focus on reducing the uncertainty inareas of the safe set that are most informative about parameters whose classiﬁcation is still uncertain(e.g. areas close to the boundary of the safe set), and only eventually turns to learning about the insideof the safe set.
This behaviour is what ultimately leads to the posterior variance to decay over thewhole S n .
We can therefore read Theorem 1 as an exploration guarantee for ISE, as it cannot happenthat it gets stuck in some subset of S n if that causes σ 2 n to stay ﬁnite over the safe set.
Theorem 1 requires a bound on the GP posterior mean function, which is always satisﬁed withhigh probability based on our assumptions about f .
Speciﬁcally, we have that | µ n ( x ) | ≤ 2 β n withprobability of at least 1 − δ , for all n (see Appendix A for details).
Therefore, it does not representan additional restrictive assumption for f .
Finally, we also note that the the constant N ε deﬁnedby (7) always exists since F is monotonically increasing, as long as γ N grows sublinearly in N .
Srinivas et al.
(2010) prove that this is the case for commonly-used kernel and, more generally, it is aprerequisite for data-efﬁcient learning with GP models.
Discussion and Limitations
ISE drives exploration of the parameter space by selecting the parameters to evaluate accordingto (6).
An alternative but conceptually similar approach to this criterion would be to consider theparameter that yields the biggest information gain on average over the domain, i.e., substituting theinner max in (6) with an average over X .
The resulting integral, however, is intractable and wouldrequire further approximations.
Moreover, the parameter found by solving (6) will also yield a highaverage information gain over the domain, due to the regularity of all involved objects.
Being able to work in a continuous domain, ISE scales better than algorithms requiring a discreteparameter space.
However, as noted in Section 4, ﬁnding x n +1 as in (6) means to solve a non convexoptimization problem with twice the dimension of the the parameter space, which can also becomea computationally challenging problem as the dimension grows.
In a high-dimensional setting, wefollow L INE BO by Kirschner et al.
(2019), which at each iteration selects a random one-dimensionalsubspace to which it restricts the optimization of the acquisition function.
Lastly, we reiterate that the theoretical safety guarantees offered by ISE are derived under theassumption that f is a bounded norm element of the RKHS space associated with the GP’s kernel.
Inapplications, therefore, the choice of the kernel function becomes even more crucial when safety isan issue.
For details on how to construct and choose kernels see (Garnett, 2022).
Experiments
In this section we empirically evaluate ISE.
Additional details about the experiments and setup canbe found in Appendix C.
GP samples
For the ﬁrst part of the experiments, we evaluate ISE on samples from a GP.
Thisallows us to test ISE under the assumptions of the theory and we compare its performance to that ofthe exploration part of S TAGE O PT (Sui et al., 2018).
S TAGE O PT is a modiﬁed version of S AFE O PT , inwhich the exploration and optimization parts are performed separately: ﬁrst the S AFE O PT explorationexploration strategy is used to expand the safe set as much as possible, then the objective function isoptimized within the discovered safe set.
We select 100 samples from a two-dimensional GP with RBF kernel, deﬁned in [ − 2 .
5 , 2 . 5] × [ − 2 . 5 , 2 . 5] and run ISE and S TAGE O PT for 100 iterations for [Figure]
each sample. As S TAGE O PT requires a discretization of the domain, we use this discretization tocompare the sample efﬁciency of the two methods, by computing, at each iteration, what percentageof the discretized domain is classiﬁed as safe.
The results of this comparison can be seen in Figure 3a.
For the S TAGE O PT run, the kernel metric has been used to compute the set of potential expanders,for different values of the Lipschitz constant L .
From Figure 3a we see not only that ISE performsas well or better than all tested instances of S TAGE O PT , but also how the choice of L affects theperformance of the latter.
This plot makes also evident how crucial the choice of the Lipschitzconstant is for S TAGE O PT and S AFE O PT -like algorithms in general.
Moreover, we use GP samplesalso to test our theoretical results, according to which ISE eventually leads to the discovery of themaximally reachable safe set.
This can be seen in Figure 3b, where we plot the average percentage ofthe maximally reachable safe set classiﬁed as safe by ISE: we can see that eventually the whole set isexplored and classiﬁed as safe.
OpenAI Gym Control
After investigating the performance of ISE under the hypothesis of thetheory, we apply it to two popular classic control tasks from the OpenAI Gym framework (Brockmanet al., 2016), with the goal of ﬁnding the set of parameters of a controller that satisfy some safetyconstraint.
In particular we considered linear controllers for the inverted pendulum and cart poletasks.
For the inverted pendulum task, the linear controller is given by u t = α 1 θ t + α 2 ˙ θ t , where u t is thecontrol signal at time t , while θ t and ˙ θ t are, respectively, the angular position and the angular velocityof the pendulum.
Starting from a position close to the upright equilibrium, the considered controller’stask is the stabilization of the pendulum, subject to a safety constraint on the maximum velocity inone episode.
For some given initial controller conﬁguration α 0 :
= ( α 01 , α 02 ) , we want to explore thecontroller’s parameter space avoiding conﬁgurations that lead the pendulum to swing with a too highvelocity.
We apply ISE to explore the α -space with x 0 =
α 0 and the safety constraint being themaximum angular velocity reached by the pendulum in an episode of ﬁxed length.
In this case thesafety threshold is not at zero, but some ﬁnite value ˙ θ M , and the safe parameters are those for whichthe maximum velocity is below ˙
θ M .
The formalism developed in the previous sections can be easilyapplied to this scenario if we consider f ( α ) =
− (max t ˙ θ t ( α ) − ˙ θ M ) .
in Figure 4a we show the truesafe for this problem, while in Figure 4b to Figure 4d one can see ISE in action and the safe set itdiscovers.
The setting for the cart pole task is similar, but the parameter space in this case has three dimensions.
The controller we consider is given by u t = α 1 θ t + α 2 ˙ θ t + α 3 ˙
s t , where θ t and ˙ θ t are, respectively, [Figure] [Figure] the angular position and angular velocity of the pole at time t , while ˙
s t is the cart’s velocity.
In thiscase, we set the initial state as one having zero angular and linear velocity and with the pole closeto the vertical position, with the controller’s goal being of keeping the pole stable in the uprightposition.
A combination of the three parameters α 1 , α 2 and α 3 is considered safe if the angle of thepole does not exceed a given threshold.
Again, we can easily cast this safety constraint in terms ofthe formalism developed in the paper: f ( α ) =
− (max t θ t ( α ) − θ M ) where θ M is the maximumallowed angle.
Figure 5a shows the expansion of the cart pole α space promoted by ISE.
[Figure]
High dimensional domains
Many interesting applications have a high dimensional parameterspace.
While S AFE O PT -like methods are difﬁcult to apply already with dimension > 3 due to thediscretization of the domain, ISE can perform well also in four or ﬁve dimensions.
To see this, weapplied ISE to the constraint function f ( x ) =
e − x 2 + 2 e − ( x − x 1 ) 2 + 5 e − ( x − x 2 ) 2 − 0 .
2 , which hasthree local optima of different heights at 0 , x 1 and x 2 , with (cid:107) x 1 (cid:107) <
(cid:107) x 2 (cid:107) . Starting a safe seedclose to the border of the safe set in the vicinity of the origin, ISE was able to explore the safe setuntil also x 2 was classiﬁed as safe. Figure 6b shows the percentage of the true safe set discovered by ISE as function of the iteration.
As discussed in Section 5, for even higher dimensions, we can followa similar approach to L INE BO, limiting the optimization to a randomly selected one-dimensionalsubspace.
We applied this method to the same constraint function used for the ﬁve dimensional caseand found that indeed we were able to explore the domain until the discovery of the location of theglobal optimum.
The corresponding results can be seen in Figure 6.
Conclusion and Societal Impact
We have introduced Information-Theoretic Safe Exploration (ISE), a novel approach to safely explorea space in a sequential decision task where the safety constraint is a priori unknown.
ISE efﬁcientlyand safely explores by evaluating only parameters that are safe with high probability and by choosingthose parameters that yield the greatest information gain about the safety of other parameters.
Wetheoretically analyzed ISE and showed that it leads to arbitrary reduction of the uncertainty inthe largest reachable safe set containing the starting parameter.
Our experiments support thesetheoretical results and demonstrate an increased sample efﬁciency and scalability of ISE compared to
S AFE O PT -based approaches.
In many safety sensitive applications the shape of the safety constraints is unknown, so that animportant prerequisite for any kind of process is to identify what parameters are safe to evaluate.
Byproviding a principled way to do this, the contributions of this paper allow to deal with safety in abroad range of applications, which can favor the usage of ML approaches also in safety sensitivesettings.
On the other hand, misuse of the proposed method cannot be prevented in general.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Limitations are discussed inSection 5(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 7(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We will include
the url to the code with the camera-ready version of the paper.
Experiment details to
reproduce our results can be found in Appendix C.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] They are in the respective plots.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 6 andAppendix C
(b) Did you mention the license of the assets?
[Yes] See Appendix C
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
No new assets have been used nor introduced.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]