Movement Penalized Bayesian Optimization with Application to Wind Energy Systems
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Contextual Bayesian optimization (CBO) is a powerful framework for sequentialdecision-making given side information, with important applications, e.g., in windenergy systems.
In this setting, the learner receives context (e.g., weather conditions) at each round, and has to choose an action (e.g., turbine parameters).
Standardalgorithms assume no cost for switching their decisions at every round.
However,in many practical applications, there is a cost associated with such changes, whichshould be minimized.
We introduce the episodic CBO with movement costs problem and, based on the online learning approach for metrical task systems of Coesterand Lee [17], propose a novel randomized mirror descent algorithm that makes useof Gaussian Process conﬁdence bounds.
We compare its performance with the ofﬂine optimal sequence for each episode and provide rigorous regret guarantees.
Wefurther demonstrate our approach on the important real-world application of altitudeoptimization for Airborne Wind Energy Systems.
In the presence of substantialmovement costs, our algorithm consistently outperforms standard CBO algorithms.
Introduction
Bayesian optimization (BO) is a well-established framework for sequential black-box function optimization that relies on Gaussian Process (GP) models [39] to sequentially learn and optimize theunknown objective.
In many practical scenarios, however, one wants to additionally use availablecontextual information when making decisions.
In this setting, at each round, the learner receives acontext from the environment and has to choose an action based upon it.
Previous works have developed contextual BO algorithms [29, 15, 28, 37] and applied them to various important applications,e.g., vaccine design, nuclear fusion, database tuning, crop recommender systems, etc.
A potential practical issue with these standard algorithms is that they assume no explicit costs forswitching between their actions at every round.
Frequent action changes can be extremely costlyin many real-world applications.
This work is motivated by the problem of real-time control of thealtitude of an airborne wind energy (AWE) system.
In AWE systems, the wind speed is often onlymeasurable at the system’s altitude, and determining the optimal operating altitude of an AWE systemas the wind speed varies represents a challenging problem.
Another fundamental challenge is thatadditional energy is required for adjusting the altitude, which makes the frequent altitude changescostly.
Consequently, this work is motivated by the following question: How can we efﬁciently learnto optimize the AWE system’s operating altitude despite varying wind conditions while minimizingthe energy cost associated with turbine altitude changes?
In this work, we formalize the movement penalized contextual BO problem.
When the switching costis a metric (distance function), we propose a novel algorithm that effectively combines ideas from
BO with the online learning strategies proposed in [17] for solving the so-called metrical task system(MTS) problem [11].
Furthermore, our algorithm relies solely on noisy point evaluations (i.e., banditfeedback), allows for arbitrary context sequences, and besides the standard exploration-exploitationtrade-off, it also balances the movement costs.
As a result, it outperforms the standard movementcost-agnostic contextual BO algorithms as well as movement-conservative baselines (see Fig. 1).
Related Work.
Bayesian optimization (BO)refers to a sequential approach for optimizing anunknown objective (cost function) from noisypoint evaluations.
A great number of BO methods have been developed over the years (e.g.,[35, 47, 16]).
While the focus of standard BO approaches is mainly on optimizing the unknownobjective cost function, in this work, we additionally focus on penalizing frequent actionchanges.
This problem makes the most sense inthe contextual BO setting [29], where the mainobjective changes with the observed contextualinformation and the learner also seeks to minimize the cost associated with frequent changesof its actions.
Several works have tried to incorporate such and similar cost functions in the
BO setting.
[32] explicitly consider switchingcosts based on how deep into the pipeline thechange in variable occurs.
Other works include cost-aware and multi-objective sampling strategies invarious settings such as batch [26, 24], multi-ﬁdelity [10, 25, 46], multi-objective optimization [2]and dynamic programming [31, 30].
Similarly, we consider movement/switching costs, but unlikeprevious work, we speciﬁcally focus on minimizing the movement costs in the contextual setting.
The Metrical Task Systems (MTS) problem [11] is a sequential decision-making problem widelystudied in the online learning literature.
At each round, the learner observes a service cost function,chooses an action, and incurs the corresponding cost together with a movement cost penalizing thedistance (according to some metric) between the current action and the one chosen at the previousround.
The MTS problem is directly related to our problem setting (see Section 2.2).
After a longseries of works, in [14] and [17], an O  (log n ) -competitive algorithm for MTS on any ﬁnite metricspace was established.
The approach of Coester and Lee [17] relies on a tree representation of thedecision space and an action randomization scheme via a mirror descent procedure.
In contrast to oursetting, these works assume that the service costs are known (i.e., they assume the full-informationfeedback).
Moreover, in the online optimization literature, other related works study the online convexoptimization with switching costs [23, 45, 22, 33] and convex body chasing problems [12, 5, 40, 13].
We make no use of convexity and take a model-based (GP) approach to learn about the unknownservice costs.
[34] consider non-convex objective functions, however, they impose certain conditionson the objective that are not easily modeled via GP as done in our work.
The use of GP conﬁdence bounds in online learning settings has been previously explored, e.g., inrepeated multi-agent [41, 44] and sequential games [43], and to discover randomized max-min strategies [42].
However, none of these works has considered movement costs in the objective.
This makesour problem signiﬁcantly different from the aforementioned ones, and requires a suitable action randomization scheme that can trade-off exploration, exploitation, and movement costs simultaneously.
Various works have applied Bayesian Optimization in the context of wind energy systems before.[38, 19, 4] use BO techniques to maximize the total energy yield in a wind farm in a cooperative or aclosed-loop framework.
[36] use it to tune the parameters of the wind turbine to learn the effectivewind speed.
[6] use BO for plant design (i.e., physical system design) of airborne wind energy systems.[48] apply BO and other regression techniques to predict short-term wind energy production to makeinformed production offers.
Finally, [3] survey several different methods for efﬁcient wind-powerprediction using machine-learning methods including BO.
Our problem formulation differs from theabove mentioned works, since we explicitly consider movement energy loss caused by the altitudechange of the system.
Contributions.
We formally introduce Bayesian optimization with movement costs and proposea novel GP-MD algorithm (in Algorithm 1).
GP-MD combines the online mirror descent (MD)algorithm with shrinking Gaussian Process (GP) conﬁdence bound to decide on which point toevaluate next.
In our theoretical analysis, we establish rigorous sublinear regret guarantees forour algorithm by combining techniques from Bayesian optimization and metrical task systemsapproaches [17].
Finally, we demonstrate that GP-MD is able to successfully outperform previouscontextual Bayesian optimization approaches on both synthetic and real-world data in the presenceof movement costs.
In particular, we consider the application to airborne wind energy systems anddemonstrate that GP-MD can effectively operate such a system by considering movement costsand varying environmental conditions.
Problem Statement
Let f : X ⇥ E ! R + be an unknown cost function deﬁned over X ⇥ E ⇢ R p , where X is a 101 ﬁnite set of actions, i.e., |X| = n , and E represents convex and compact space of contexts.
We 102 denote the known metric (i.e., distance function) of
X as d ( · , · ) , and similarly to other works in 103 Bayesian optimization (e.g., [47, 16]) assume that the target cost function f belongs to a reproducingkernel Hilbert space (RKHS)
H k of functions (deﬁned on X ⇥ E ), that corresponds to a known 105 kernel k : (
X ⇥ E ) ⇥
( X ⇥ E ) ! R + with k (( x, e ) , ( x 0 , e 0 ))  1 for any action-context pair.
In 106 particular, we assume that for some known B > 0 , the target cost f has a bounded RKHS norm, i.e.,f 2 F k = { f 2 H k : k f k k  B } .
Also, we assume that the diameter of X ( max x,x 0 2 X d ( x, x 0 ) ) is 108 bounded and denote it by .
We consider an episodic setting, wherein each episode runs over a ﬁnite time horizon H .
Let theinitial state of the system in the ﬁrst episode correspond to action x 0 , 1 2 X .
At the end of everyepisode, the system resets to a new given initial action x 0 ,m 2 X where m 2 { 1 , 2 , . . .
, N ep } denotes 112 the episode index.
In each episode m and at every time step h 2 { 1 , 2 , . . .
, H } , the environment 113 reveals the context e h,m 2 E to the learner.
We make no assumptions on the context sequence 114 provided by the environment (i.e., it can be arbitrary and different across episodes).
The learnerthen chooses x h,m 2 X and observes the noisy function value:
[Equation]
where ⇠ h,m ⇠ N (0 ,  2 ) with known  , and independence over time steps.
The goal of the learneris to minimize the cost incurred over the rounds in every episode, but at the same time to minimizethe distance between its subsequent decisions as measured by d ( x h  1 ,m , x h,m ) .
Let D m = { x 1 ,m , x 2 ,m , . . .
, x H,m } denote the set of actions chosen by the learner over H roundsin episode m .
We recall that each action x h,m 2 D m is chosen after observing the correspondingcontext e h,m .
The objective is to minimize the cumulative episodic cost for each episode m ,
[Equation]
where we refer to the two terms in Eq.
(2) as service cost S m and movement cost M m .
When f is known, the problem can be seen as a MTS instance as detailed in Section 2.2.
Even in sucha case, we cannot hope to solve this problem optimally, and nearly-optimal approximate algorithmswere recently proposed (see Coester and Lee [17]).
Hence, the learner’s performance in episode m ismeasured via ( ↵ ,  ) -approximate regret:
[Equation]
where D ⇤ m := arg min D ⇢ X , | D | =
H cost m ( D ) is the ofﬂine optimal action sequence obtained assuming the knowledge of the true sequence of contexts { e h,m } Hh =1 in advance , and ↵ and  are 129 approximation constants (independent of N ep ).
In contrast, in our setting, the learner only gets to seethe current context when making a decision and has no knowledge about the future ones.
After N e episodes, the total cumulative regret is deﬁned as
[Equation]
We seek an algorithm whose total cumulative regret grows sublinearly in N ep , so thatlim N ep ! 1 R ↵ ,  H,N ep /N
ep
= 0 , for any set of initial states { x 0 ,m } N ep m =1 ⇢ X .
2.1 Gaussian Process Model
In standard Bayesian optimization, a surrogate Gaussian Process model is typically used to model thetarget cost function.
A Gaussian Process GP ( µ ( · ) , k ( · , · )) over the input domain X ⇥ E , is a collection 137 of random variables ( f ( x, e ))
x 2 X ,e 2 E where every ﬁnite number of them ( f ( x i , e i ))
ni =1 , n 2 N , is 138 jointly Gaussian with mean E [ f ( x i )]
= µ ( x i , e i ) and covariance E [( f ( x i , e i )  µ ( x i , e i ))
( f ( x j , e j )  139 µ ( x j , e j ))]
=
k
(( x i , e i ) , ( x j , e j )) for every 1  i, j  n .
BO algorithms typically use zero-mean GP priors to model uncertainty in f , i.e., f ⇠ GP (0 , k ( · , · )) ,and Gaussian likelihood models for the observed data.
As more data points are observed, GP(Bayesian) posterior updates are performed in which noise variables are assumed to be drawnindependently across t from N (0 ,  ) .
Here,  is a hyperparameter that might be different from thetrue noise variance  2 .
More precisely, given a sequence of previously queried points and their noisyobservations the posterior is again Gaussian, with the posterior mean and variance given by:
[Equation]
) where Y t :=
[ y 1 , . . .
, y t ] denotes a vector of observations, K t =
[ k (( x s , e s ) , ( x s 0 , e s 0 )] s,s 0  t is thecorresponding kernel matrix, and k t ( x, e ) =
[ k (( x 1 , e 1 ) , ( x, e )) , . . .
, k (( x t , e t ) , ( x, e ))]
T 2 R t ⇥ 1 .
Maximum Information Gain.
In the standard Bayesian optimization, the main quantity thatcharacterizes the complexity of optimizing the target cost function is the maximum information gain[47] deﬁned at time t as:
[Equation]
where I ( Y t ; f ) denotes the mutual information between random observations Y t and GP model f .
The mutual information for the GP model is given as:
[Equation]
This quantity is kernel-speciﬁc and for compact and convex domains  t is sublinear in t for variousclasses of kernel functions [47] as well as for kernel compositions (e.g., additive kernels in [29]).
Conﬁdence Bounds.
We also use the following result ([47, 1, 16]) that is frequently used in Bayesianoptimization to provide conﬁdence bounds around the unknown function.
Lemma 1.
Assume the  -sub-Gaussian noise model as in Eq.
(1) , and let f belong to F k .
Then, thefollowing holds with probability at least 1   simultaneously over all t  1 and x 2 X , e 2 E :
[Equation]
(9) where  t =  1 /p 2 ln(1 /  ) + 2  t + B , and µ t and  t are deﬁned in Eqs.
(5) and (6) with  > 0 .
Based on the previous, we also deﬁne the lower conﬁdence bound for every x 2 X , e 2 E as:
[Equation]
We use lcb m ( x, e ) when it is computed based on data collected before episode m .
2.2 Relation to Metrical Task Systems (MTS)
When f is known, our optimization objective in Eq.
(2) can be seen as a particular type of MTSproblem, where f ( · , e h,m ) is the MTS service cost that changes for every h and m .
Compared toa standard MTS (see Appendix A), our problem formulation is more challenging since the learnercan only learn about f from previously observed data.
The approach proposed in this paper buildson the algorithm by Coester and Lee [17] for standard MTS problems.
However, to cope with theaforementioned challenge, our approach exploits the regularity assumptions regarding f and utilizesthe constructed lower conﬁdence bounds Eq.
(10) to hallucinate information about the unavailable
1: Require: Action space X , kernel function k ( · , · ) , metric d ( · , · ) 2: Run FRT ( X , d ( · , · )) and obtain ⌧ -HST T =
( V, E ) with leaves L =
X 3: for m = 1 , . .
. , N ep do 4:
Receive x 0 ,m and initialize z 0 ,m (Eq. (27)), conditional prob.
q 0 =   1 ( z 0 ,m ) as in Eq.
(15) 5: for h = 1 , . .
, H do 6:
Observe context e h,m and initialize costs: lcb m ( v, e h,m ) = 0 , 8 v 2 V \L 7: for u 2 OD ( V \L ) do 8:
Update vertex prob.
q ( u ) h from q ( u ) h  1 and lcb m ( · , e h,m ) via Mirror Descent (Eq.
Update cost for vertex u : lcb m ( u, e h,m )
= h q ( u ) h , lcb m ( · , e h,m ) i =
X ⌫ 2 C ( u ) q h, ⌫ · lcb m ( ⌫ , e h,m ) 10: end for 11:
Compute prob.
vector z h,m =  ( q h ) (Eq.
and leaves’ prob.
l ( z h,m ) (Eq.
Estimate optimal coupling ⇣ h  1 ,h,m between l ( z h  1 ,m ) and l ( z h,m ) as in Eq.
Sample action x h,m ⇠ ⇣ h  1 ,h,m ( ·| x h  1 ,m ) and observe y h,m =
f ( x h,m , e h,m ) + ⇠ h,m 14: end for 15:
Update µ m +1 ( · , · ) and  m +1 ( · , · ) as per Eq.
(5) and Eq.
(6) 16: end for service cost at each round.
Before presenting our overall approach, we describe a preliminarystep proposed by [17], which consists of representing our metric space ( X , d ) by a Hierarchically
Separated Tree (HST) metric space.
HST metric space.
Consider a tree T
= ( V, E ) with root r , leaves L ⇢ V and non-negative weights 174 w v , for each v 2 V , which are non-increasing along root-leaf paths.
Let d T ( l, l 0 ) denote a distancemetric between any two leaves l, l 0 2 L given as the sum of the encountered weights on the pathfrom l to l 0 (see Fig. 4).
( L , d T ) is a HST metric space, and ⌧ -HST metric space if the weights areexponentially decreasing, i.e., w u  w v / ⌧ , with v being the parent of u .
Similarly to [17], we use the algorithm from [21] (which we name via the authors’ surnames as
FRT) to approximate the given metric space ( X , d ) by a ⌧ -HST one.
In particular, we use FRTin Algorithm 1 as a computationally efﬁcient preprocessing step to create a tree T with leaves Lcorresponding to actions in X , distance metric d T , and root node r .
We explain the intrinsic MTSmotivation for this preprocessing step in Appendix B.2, and defer additional details to Appendix B.3.
The GP-MD Algorithm
In this section, we introduce GP-MD, a novel algorithm for the contextual BO problem withmovement costs deﬁned in Section 2.
At each episode m and round h , the state of GP-MD can besummarized by a vector of probabilities z h,m 2 K T over the vertices of T , where K T := {+n z 
+}
R | V | + : z r = 1 , z u = P ⌫ 2 C ( u ) z ⌫ 8 u 2 V \L o , and C ( u ) denotes the children of u .
Each entry z ⌫represents the probability that the selected action x h,m belongs to the leaves of the subtree rooted at⌫ , i.e., z ⌫ = P ( x h,m 2 L ( ⌫ )) .
Below, we specify how z h,m is computed at each round.
Moreover,given any z 2 K T , the vector
[Equation]
deﬁnes a probability distribution over the leaves L , and hence the actions X .
As in [17], givenprobability vectors z h,m and z h  1 ,m , GP-MD computes the minimal distance distribution [Equation]
[Equation]
where U h  1 ,m and U h are random variables having marginals l ( z h  1 ,m ) and l ( z h,m ) respec- 194 tively.
Finally, action x h,m is sampled from the conditional minimal distance distribution x h,m ⇠ ⇣ h  1 ,h,m ( ·| x h  1 ,m ) (Line 13 in Algorithm 1).
At the end of each episode m , the newly observed 196 data are then used to update posterior mean and standard deviation about the cost function.
Finally, we describe how probability vectors
z h,m are computed at each round, a key step of GP-MD(Lines 8–12 in Algorithm 1).
We follow the recursive Mirror Descent (MD) procedure proposedby [17], with the important difference that we are dealing with an unknown context-dependent costfunction.
Hence, we make use of the Gaussian process model and corresponding conﬁdence estimatesas deﬁned in Section 2.1.
To obtain probabilities z h,m , we consider conditional probability vectors q 2 Q T , where Q T isthe set of valid conditional probabilities Q T :=
[Equation]
Foreach vertex ⌫ with parent u , q ⌫ represents the conditional probability P ( x h,m 2 L ( ⌫ ) | x h,m 2 L ( u )) .
Moreover, given q h 2 Q T we deﬁne the vector q ( u ) h :=
[ q h, ⌫ , ⌫ 2 C ( u )] as the conditionaldistribution over children of u , and let Q ( u ) T be the set of all valid distributions q ( u ) h .
In each episode m , conditional probability vector q h for round h is obtained recursively, from leavesto root, as a function of q h  1 , the observed context e h,m , and the current estimate about the costassociated to each particular vertex.
More precisely, let OD ( V \L ) be a topological ordering of theinternal vertices V \L so that every child in T occurs before its parent.
Then, for each u 2 OD ( V \L )conditional probabilities q ( u ) h are obtained via the Mirror Descent update:
[Equation]
[Equation]
Function D ( u ) is the Bregman divergence with respect to a suitable potential function (see Appendix D.1), while lcb ( u ) m ( · , e h,m ) :
= [ lcb m ( ⌫ , e h,m ) , 8 ⌫ 2 C ( u )] is a lower conﬁdence boundestimate of the costs corresponding to children of vertex u .
For v 2 L , lcb m ( ⌫ , e h,m ) are obtained bythe GP-regression techniques outlined in Section 2.1, while for internal vertices these are computedrecursively from their children nodes as:
[Equation]
The movement cost is primarily controlled by this usage of Bregman divergence based mirror descent.
Also, sampling from the conditional minimal distance distribution further restricts movement betweenalternate actions.
Once the vector of conditional probabilities q h 2 Q T has been updated, we canobtain the corresponding probability vector z h,m via the mapping  : Q T ! K T such that:
[Equation] [Equation]
3.1 Theoretical Guarantees
Our main theorem bounds the cumulative regret of GP-MD.
Theorem 1.
Let X be represented by a ⌧ -HST space with ⌧ > 4 (Line 2 of Algorithm 1), and set 2 (0 , 1) .
Then, with probability at least 1   , the regret of GP-MD over N ep episodes is bounded by
[Equation]
↵ ,
[Equation]
with approximation factors ↵ =
O  (log n ) and  = O (1) .
Here, H is the episodes’ length,  N epis the conﬁdence level from Lemma 1, and  HN ep is the maximum information gain deﬁned in Eq.
For most of the popularly used kernels, Thm. 1 can be made more explicit by substituting bounds on HN ep (e.g., in the case of linear kernel and compact domain, we have  t
=
O
( p log t ) , while forsquared-exponential kernel it holds  t
= O ((log t ) p +1 )
[47]; see also Section 2.1).
In such cases, wemake the following two important observations regarding our result: i)
The obtained regret boundis sublinear in the number of episodes N ep and hence lim N ep !
1 R ↵ ,  N ep /N
ep
= 0 ; ii)
The boundis independent of the input space size, i.e., the number of actions n (although the approximationfactor ↵ depends logarithmically on n , similarly to [17]).
These imply that GP-MD approaches↵ -competitive ratio performance of the MTS algorithm by [17], while learning about the service
[Figure]
cost from noisy point evaluations (i.e., bandit feedback) only. Finally, in our analysis, we treat H asconstant and note that a similar dependence on H is also present in the batch Bayesian optimizationresults and analysis that we also build upon (see [18, Theorem 2]).
Improving this dependency in ourbound is an interesting direction for future work.
Proof of Thm.
1 is detailed in
Appendix E. Next, we outline some main steps.
We make use ofthe competitive ratio guarantees for the used Mirror Descent algorithm from [17, Corollary 4] tobound the expected hallucinated service cost
P Hh =1 h lcb m ( · , e h,m ) , l ( z h,m ) i .
Here, we use the fact 242 that lcb m ( · , · ) does not change within an episode.
Since this is not the actual service cost, we bound
P Hh =1 h f ( · , e h,m ) , l ( z h,m ) i by P Hh =1 h lcb m ( · , e h,m ) , l ( z h,m ) i with an additional learning error.
Thesum of the learning errors over all episodes can be rewritten as
P [Equation] We use the concentration of the conditional mean result from [27, Lemma 3] to upper bound it by theactual realizations P
N ep m = P Hh =1  2 m ( x h,m , e h,m ) , and use the result of [18, Proposition 1], to furtherupper bound it with the maximum information gain quantity  N ep H .
Finally, the movement cost can also be bounded similar to that of the previously mentioned expectedhallucinated service cost P Hh =1 h lcb m ( · , e h,m ) , l ( z h,m ) i with an extra ↵ = O  (log n ) factor using[17, Corollary 4].
Experiments
This section provides numerical results on synthetic and real-world data.
We compare the performanceof our GP-MD algorithm with the following baselines:
• S TATIONARY selects the stationary strategy x
h = x 0 for all h ,
• CGP-LCB [29] neglects the movement cost and sets
x h = arg min x lcb h ( x, e h ) for all h ,• M IN C-K NOWN assumes f ( · ) is known and chooses x h = arg min x f ( x, e h ) , and• MD-K NOWN assumes f ( · ) is known and runs mirror descent from [17] on f ( · , e h ) .
MD-K NOWN and M IN C-K NOWN unrealistically assume that f ( · ) is known and can be seen asupper-bound for the achievable performance of GP-MD and CGP-LCB, respectively.
We use thesame constant value  = 2 . 0 for the exploration parameter in both GP-MD and CGP-LCB (since thetheoretical worst-case bounds are found to be overly pessimistic and can impede performance [47]).
We run the algorithms over a single episode (as done in CGP-LCB).
We also discover that updatingthe conﬁdence bounds after every timestep in GP-MD leads to better performance in practice.
Synthetic experiments.
We consider synthetic experiments, where the objective function is a random
GP sample.
The considered action space X is a subset of [0 , 1] 2 consisting of 400 points that form theuniform grid, while the context space E consists of 40 contexts that are uniformly sampled from (0 , 1) .
We sample objective function (i.e., actual cost)
f : X ⇥ E ! R from a GP (0 , k ) , where k is a squaredexponential kernel with lengthscale parameter set to l = 0 .
We use the Euclidean distance betweenthe domain points as the movement cost, calculate the distance matrix (between every pair of points)and the average movement cost.
We subtract the minimum value from f and scale it such that the average function value is equal to the average movement cost.
We also set the noise parameter to 1% of thefunction range.
We introduce the trade-off parameter ⇢ 2 { 0 . 25 , 0 .
5 , 1 , 2 , 4 } that only multiplies theservice cost, i.e, ⇢ f ( x, e ) , but not the movement cost.
This is to test the performance of the algorithmsfor varying importance of the service/movement costs.
For each ⇢ we sample 25 different functionsand run the algorithms for 500 timesteps wherein at each step the contexts are randomly sampled.
In Fig.
2a-Fig.
2e, we show the total cumulative cost as a function of timesteps for different ⇢ values.
Then, in Fig.
2f, we show the performance of the algorithms (for known kernel parameters) whenrun for 800 timesteps for varying importance of the service and movement costs.
In particular, weconsider a convex combination of the service and movement costs, where we set the respectivecoefﬁcients multiplying these two objectives as ⇢ / (1 + ⇢ ) and 1 / (1 + ⇢ ) .
As shown in Fig.
2a-Fig.
2c, the performance of GP-MD is generally close to the one of the idealized,unrealistic benchmark MD-K NOWN , which, as expected, performs the best.
The stationary baselineperforms comparably when ⇢ is small, while its performance deteriorates for larger values.
Asexpected, both M IN C-K NOWN and CGP-LCB incur higher total costs than GP-MD when themovement cost is of the higher or same relative importance as the service cost (i.e., ⇢ 2 { 0 . 5 , 1 . 0 } ),while the performance gap slowly decreases when the service cost becomes dominant ( ⇢ = 2 . 0 ).
In Figs. 2d and 2e, we also show the corresponding movement costs, and observe that movement costignorant CGP-LCB incurs signiﬁcant movement costs, while our GP-MD successfully minimizesthe movement costs.
Finally, in Fig.
2f, we observe that the performance of GP-MD is robust, i.e.,it clearly outperforms CGP-LCB whenever the movement cost dominates the total cost objective,while its performance remains comparable to the one of CGP-LCB (that is built to maximize servicecost) when the movement cost becomes dominated by the service cost.
4.1 Altitude Optimization in AWE Systems
In airborne wind energy (AWE) systems, the turbine’s operating altitude can be changed dependingon the wind pattern.
We follow the setup of Baheri et al.
[7] that applied CGP-LCB [29] whichignores movement-costs, to learn this control task.
In this section, we use a dataset from [9] whichcontains wind-speed information over various locations in Europe for a period ranging from 2011to 2017, and also includes measurements at different altitudes per location.
We consider the windspeed data from the second half of 2016.
Our goal is to maximize the generated energy, while takinginto account the energy loss due to moving the turbine from one altitude to another.
We consider 25 different altitudes (ranging from 10 m to 1600 m) as the action space and the contextspace to be hours in the day (i.e., 24 values).
We deﬁne our unknown service objective function to bef ( x, t ) = max x 0 ( E S ( x 0 , t ))
E S ( x, t ) where E S ( x, t ) denotes the energy generated based on thewindspeed at altitude x and time t .
Based on the discrete-time power generation formula from Baheriet al.
[7] (Eq.
(10)), we have
[Equation]
[Equation]
Here V w ( x, t ) denotes the windspeed at altitude x and time t , and V r denotes the rated windspeedof the turbine.
The constants c 1 = 0 .
0579 and c 2 = 0 .
09 are system dependent.
This correspondsto the energy generated at a particular altitude x for  t time.
Similarly to Baheri et al.
[7], we use t = 60 since we consider intervals of one hour length.
Next, we deﬁne the movement cost to bethe energy lost in changing altitude (from x to x 0 ):
[Equation]
[Equation]
where c 3 = 0 .
15 (see Appendix F for more details). [Figure]
We assume that a wind speed gauge is attached to the turbine, and the operator knows the windspeed at the current altitude.
Hence, instead of directly learning f ( x, t ) , we learn V w ( x, t ) and use itsconﬁdence bounds to calculate the conﬁdence bounds of f ( x, t ) .
To learn V w ( x, t ) , we normalize theinputs, and ﬁt a GP with RBF kernel (lengthscale=3.67, outputscale=6.85 and noise parameter=2.73).
We run the algorithms for different ⇢ for 960 timesteps, where again ⇢ is used to multiply E S .
Foreach ⇢ , the algorithms were initiated with every possible starting point (25 different altitudes), and ranfor 3 iterations.
Based on this we plot the total energy generated w.r.t.
varying ⇢ in Fig.
In Figs.
3aand 3f, we show the performance of the algorithms at two different locations (we also consideradditional locations and time periods in Appendix F).
We use different values of ⇢ > 1 to show therobustness of our algorithm (as ⇢ increases, the importance of the service cost w.r.t.
the movementcost in the overall objective increases).
Our algorithm outperforms CGP-LCB for a range of ⇢ values.
As ⇢ keeps increasing, we observe that M IN C-K NOWN closes the performance gap to MD-K NOWN ,and the same is happening with CGP-LCB w.r.t.
GP-MD.
In Fig.
3b, we focus on a particular⇢ = 4 , and notice that GP-MD performs better than CGP-LCB and S TATIONARY algorithm at thislocation.
In Fig.
3c, we plot the service cost and observe that both learning algorithms GP-MD and
CGP-LCB have lower service cost than the S TATIONARY baseline.
We also note that due to theimplicit service cost deﬁnition, the M IN C-K NOWN baseline achieves zero service cost.
In Figs.
3dand 3e, we compare the movement energy loss for ⇢ = 4 and ⇢ = 1 .
As expected, ⇢ = 1 results inslightly lower GP-MD movement energy loss due to the tradeoff shifting towards the movement cost.
Conclusions
We have considered the problem of optimizing an unknown cost function subject to time-varyingcontextual information, as well as movement costs of changing the selected action from round to round.
Our problem formulation is motivated by Airborne Wind Energy systems, where one seeks to optimizethe operating altitude of the wind turbine to maximize the amount of generated energy.
We propose anovel algorithm, GP-MD, which makes use of GP conﬁdence bounds and employs the mirror descenttechniques from [17] for solving MTS problems.
We analyze the theoretical performance of ouralgorithm by providing a rigorous regret bound.
Moreover, we demonstrate its performance in synthetic experiments and on an AWE application by using real-world data.
GP-MD carefully trades offservice and movement costs while at the same time learning about the unknown objective function andyielding improved performance (i.e., generating more energy) compared to the considered baselines.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Added insupplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Internal Cluster was used
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N /A](c)
Did you include the estimated
[Table]
wage paid to participants and the total amountspent on participant compensation
[Table] [N/A]