Movement Penalized Bayesian Optimization with Application to Wind Energy Systems
Abstract
Contextual Bayesian optimization (CBO) is a powerful framework for sequential decision-making given side information, with important applications, e.g., in wind energy systems.
In this setting, the learner receives context (e.g., weather conditions) at each round, and has to choose an action (e.g., turbine parameters).
Standard algorithms assume no cost for switching their decisions at every round.
However, in many practical applications, there is a cost associated with such changes, which should be minimized.
We introduce the episodic CBO with movement costs problem and, based on the online learning approach for metrical task systems of Coester and Lee [19], propose a novel randomized mirror descent algorithm that makes use of Gaussian Process conﬁdence bounds.
We compare its performance with the ofﬂine optimal sequence for each episode and provide rigorous regret guarantees.
We further demonstrate our approach on the important real-world application of altitude optimization for Airborne Wind Energy Systems.
In the presence of substantial movement costs, our algorithm consistently outperforms standard CBO algorithms.
Introduction
Bayesian optimization (BO) is a well-established framework for sequential black-box function optimization that relies on Gaussian Process (GP) models [42] to sequentially learn and optimize the unknown objective.
In many practical scenarios, however, one wants to additionally use available contextual information when making decisions.
In this setting, at each round, the learner receives a context from the environment and has to choose an action based upon it.
Previous works have developed contextual BO algorithms [32, 16, 31, 40] and applied them to various important applications, e.g., vaccine design, nuclear fusion, database tuning, crop recommender systems etc.
A potential practical issue with these standard algorithms is that they assume no explicit costs for switching between their actions at every round.
Frequent action changes can be extremely costly in many real-world applications.
This work is motivated by the problem of real-time control of the altitude of an airborne wind energy (AWE) system.
In AWE systems, the wind speed is often only measurable at the system’s altitude, and determining the optimal operating altitude of an AWE system as the wind speed varies represents a challenging problem.
Another fundamental challenge is that additional energy is required for adjusting the altitude, which makes the frequent altitude changes costly.
Consequently, this work is motivated by the following question: How can we efﬁciently learn to optimize the AWE system’s operating altitude despite varying wind conditions while minimizing the energy cost associated with turbine altitude changes?
In this work, we formalize the movement penalized contextual BO problem.
When the switching cost is a metric (distance function), we propose a novel algorithm that effectively combines ideas from BO with the online learning strategies proposed in [19] for solving the so-called metrical task system (MTS) problem [11].
Furthermore, our algorithm relies solely on noisy point evaluations (i.e., bandit feedback), allows for arbitrary context sequences, and besides the standard exploration-exploitation trade-off, it also balances the movement costs.
As a result, it outperforms the standard movement-costagnostic contextual BO algorithms as well as movement-conservative baselines (see Fig. 1).
refers to a sequential approach for optimizing an unknown objective (cost function) from noisy point evaluations.
A great number of BO methods have been developed over the years (e.g., [38, 50, 17]).
While the focus of standard BO approaches is mainly on optimizing the unknown objective cost function, in this work, we additionally focus on penalizing frequent action changes.
This problem makes the most sense in the contextual BO setting [32], where the main objective changes with the observed contextual information and the learner also seeks to minimize the cost associated with frequent changes of its actions.
Several works have tried to incorporate such and similar cost functions in the BO setting.
[35] explicitly consider switching costs based on how deep into the pipeline the change in variable occurs.
But they do not consider the contextual setting that is essential for our wind energy application and hence our work is not directly comparable with theirs.
Moreover, it assumes that the system of interest has a modular structure (as detailed in [35, Section-3]).
In such a modular setting, the cost at each round is the number of modules that has to be changed rather than the amount of change of each variable.
Other works include cost-aware and multi-objective sampling strategies in various settings such as batch [29, 27], multi-ﬁdelity [10, 28, 49], multi-objective optimization [2] and dynamic programming [34, 33].
Finally, two recent works [24, 15] consider the problem of switching cost minimization in Bayesian optimization.
They both consider the non-contextual setting, and while [24] lacks theoretical guarantees, [15] does not explicitly consider the cost in the regret deﬁnition.
Similarly, we consider movement/switching costs, but unlike these previous work, we speciﬁcally focus on minimizing the movement costs in the contextual setting.
The Metrical Task Systems (MTS) problem [11] is a sequential decision-making problem widely studied in the online learning literature.
At each round, the learner observes a service cost function, chooses an action, and incurs the corresponding cost together with a movement cost penalizing the distance (according to some metric) between the current action and the one chosen at the previous round.
The MTS problem is directly related to our problem setting (see Section 2.2).
After a long series of works, in [14] and [19], an O  (log n ) -competitive algorithm for MTS on any ﬁnite metric space was established.
The approach of Coester and Lee [19] relies on a tree representation of the decision space and an action randomization scheme via a mirror descent procedure.
In contrast to our setting, these works assume that the service costs are known (i.e., they assume the full-information feedback).
Moreover, in the online optimization literature, other related works study the online convex optimization with switching costs [26, 48, 25, 36] and convex body chasing problems [12, 5, 43, 13].
We make no use of convexity and take a model-based (GP) approach to learn about the unknown service costs.
[37] consider non-convex objective functions, however, they impose certain conditions on the objective that are not easily modeled via GP as done in our work.
The use of GP conﬁdence bounds in online learning settings has been previously explored, e.g., in repeated multi-agent [44, 47] and sequential games [46], and to discover randomized max-min strategies [45].
However, none of these works has considered movement costs in the objective.
This makes our problem signiﬁcantly different from the aforementioned ones, and requires a suitable action randomization scheme that can trade-off exploration, exploitation, and movement costs simultaneously.
Various works have applied Bayesian Optimization in the context of wind energy systems before. [41, 21, 4] use BO techniques to maximize the total energy yield in a wind farm in a cooperative or a closed-loop framework.
[39] use it to tune the parameters of the wind turbine to learn the effective wind speed.
[6] use BO for plant design (i.e., physical system design) of airborne wind energy systems. [51] apply BO and other regression techniques to predict short-term wind energy production to make informed production offers.
Finally, [3] survey several different methods for efﬁcient wind-power prediction using machine-learning methods including BO.
Our problem formulation differs from the above mentioned works, since we explicitly consider movement energy loss caused by the altitude change of the system.
Contributions.
We formally introduce Bayesian optimization with movement costs and propose a novel GP-MD algorithm (in Algorithm 1).
GP-MD combines the online mirror descent (MD) algorithm with shrinking Gaussian Process (GP) conﬁdence bound to decide on which point to evaluate next.
In our theoretical analysis, we establish rigorous sublinear regret guarantees for our algorithm by combining techniques from Bayesian optimization and metrical task systems approaches [19].
Finally, we demonstrate that GP-MD is able to successfully outperform previous contextual Bayesian optimization approaches on both synthetic and real-world data in the presence of movement costs.
In particular, we consider the application to airborne wind energy systems and demonstrate that GP-MD can effectively operate such a system by considering movement costs and varying environmental conditions.
Problem Statement
Let f : X ⇥ E ! R + be an unknown cost function deﬁned over X ⇥ E ⇢ R p , where X is a ﬁnite set of actions, i.e., |X| = n , and E represents convex and compact space of contexts.
We denote the known metric (i.e., distance function) of X as d ( · , · ) , and similarly to other works in Bayesian optimization (e.g., [50, 17]) assume that the target cost function f belongs to a reproducing kernel Hilbert space (RKHS)
H k of functions (deﬁned on X ⇥ E ), that corresponds to a known kernel k : (
X ⇥ E ) ⇥
( X ⇥ E ) ! R + with k (( x, e ) , ( x 0 , e 0 ))  1 for any action-context pair.
In particular, we assume that for some known B > 0 , the target cost f has a bounded RKHS norm, i.e., f 2 F k = { f 2 H k : k f k k  B } .
Also, we assume that the diameter of X ( max x,x 0 2 X d ( x, x 0 ) ) is bounded and denote it by .
We consider an episodic setting, wherein each episode runs over a ﬁnite time horizon H .
Let the initial state of the system in the ﬁrst episode correspond to action x 0 , 1 2 X .
At the end of every episode, the system resets to a new given initial action x 0 ,m 2 X where m 2 { 1 , 2 , . . .
, N ep } denotes the episode index.
In each episode m and at every time step h 2 { 1 , 2 , . . .
, H } , the environment reveals the context e h,m 2 E to the learner.
We make no assumptions on the context sequence provided by the environment (i.e., it can be arbitrary and different across episodes).
The learner then chooses x h,m 2 X and observes the noisy function value:
[Equation]
where ⇠ h,m ⇠ N (0 ,  2 ) with known  , and independence over time steps.
The goal of the learner is to minimize the cost incurred over the rounds in every episode, but at the same time to minimize the distance between its subsequent decisions as measured by d ( x h  1 ,m , x h,m ) .
Let D m = { x 1 ,m , x 2 ,m , . . .
, x H,m } denote the set of actions chosen by the learner over H rounds in episode m .
We recall that each action x h,m 2 D m is chosen after observing the corresponding context e h,m .
The objective is to minimize the cumulative episodic cost for each episode m , cost m ( D m [Equation] =
[Equation]
where we refer to the two terms in Eq.
(2) as service cost S m and movement cost M m .
When f is known, the problem can be seen as a MTS instance as detailed in Section 2.2.
Even in such a case, we cannot hope to solve this problem optimally, and nearly-optimal approximate algorithms were recently proposed (see Coester and Lee [19]).
Hence, the learner’s performance in episode m is measured via ( ↵ ,  ) -approximate regret:
[Equation]
where D ⇤ m := arg min D ⇢ X , | D | =
H cost m ( D ) is the ofﬂine optimal action sequence obtained assuming the knowledge of the true sequence of contexts { e h,m } Hh =1 in advance , and ↵ and  are approximation constants (independent of N ep ).
In contrast, in our setting, the learner only gets to see the current context when making a decision and has no knowledge about the future ones.
After N ep episodes, the total cumulative regret is deﬁned as
[Equation]
We seek an algorithm whose total cumulative regret grows sublinearly in N ep , so that lim N ep ! 1 R ↵ ,  H,N ep /N
ep
= 0 , for any set of initial states { x 0 ,m } N ep m =1 ⇢ X .
2.1 Gaussian Process Model
In standard Bayesian optimization, a surrogate Gaussian Process model is typically used to model the target cost function.
A Gaussian Process GP ( µ ( · ) , k ( · , · )) over the input domain X ⇥ E , is a collection of random variables ( f ( x, e ))
x 2 X ,e 2 E where every ﬁnite number of them ( f ( x i , e i ))
ni =1 , n 2 N , is jointly Gaussian with mean E [ f ( x i )]
= µ ( x i , e i ) and covariance E [( f ( x i , e i )  µ ( x i , e i ))
( f ( x j , e j )  µ ( x j , e j ))]
=
k
(( x i , e i ) , ( x j , e j )) for every 1  i, j  n .
BO algorithms typically use zero-mean GP priors to model uncertainty in f , i.e., f ⇠ GP (0 , k ( · , · )) , and Gaussian likelihood models for the observed data.
As more data points are observed, GP (Bayesian) posterior updates are performed in which noise variables are assumed to be drawn independently across t from N (0 ,  ) .
Here,  is a hyperparameter that might be different from the true noise variance  2 .
More precisely, given a sequence of previously queried points and their noisy observations the posterior is again Gaussian, with the posterior mean and variance given by:
[Equation]
where Y t :=
[ y 1 , . . .
, y t ] denotes a vector of observations, K t =
[ k (( x s , e s ) , ( x s 0 , e s 0 )] s,s 0  t is the corresponding kernel matrix, and k t ( x, e ) =
[ k (( x 1 , e 1 ) , ( x, e )) , . . .
, k (( x t , e t ) , ( x, e ))]
T 2 R t ⇥ 1 .
Maximum Information Gain.
In the standard Bayesian optimization, the main quantity that characterizes the complexity of optimizing the target cost function is the maximum information gain [50] deﬁned at time t as: [Equation]
[Equation]
where I ( Y t ; f ) denotes the mutual information between random observations Y t and GP model f .
The mutual information for the GP model is given as:
[Equation]
This quantity is kernel-speciﬁc and for compact and convex domains  t is sublinear in t for various classes of kernel functions [50] as well as for kernel compositions (e.g., additive kernels in [32]).
Conﬁdence Bounds.
We also use the following result ([50, 1, 17]) that is frequently used in Bayesian optimization to provide conﬁdence bounds around the unknown function.
Lemma 1.
Assume the  -sub-Gaussian noise model as in Eq.
(1) , and let f belong to F k .
Then, the following holds with probability at least 1   simultaneously over all t  1 and x 2 X , e 2 E :
[Equation]
where  t =  1 /p 2 ln(1 /  ) + 2  t + B , and µ t and  t are deﬁned in Eqs.
(5) and (6) with  > 0 .
Based on the previous, we also deﬁne the lower conﬁdence bound for every x 2 X , e 2 E as:
[Equation]
[Equation]
We use lcb m ( x, e ) when it is computed based on data collected before episode m .
1: Require: Action space X , kernel function k ( · , · ) , metric d ( · , · ) 2: Run FRT (
X , d ( · , · )) and obtain ⌧ -HST T =
( V, E ) with leaves L =
X 3: for m = 1 , . .
. , N ep do 4:
Receive x 0 ,m and initialize z 0 ,m (Eq.
(29)), conditional prob.
q 0 =   1 ( z 0 ,m ) as in Eq.
(15) 5: for h = 1 , . .
, H do 6:
Observe context e h,m and initialize costs: lcb m ( v, e h,m ) = 0 , 8 v 2 V \L 7: for u 2 OD ( V \L ) do 8:
Update vertex prob.
q ( u ) h from q ( u ) h  1 and lcb m ( · , e h,m ) via Mirror Descent (Eq.
Update cost for vertex u : lcb m ( u, e h,m )
= h q ( u ) h , lcb m ( · , e h,m ) i =
X ⌫ 2 C ( u ) q h, ⌫ · lcb m ( ⌫ , e h,m ) 10: end for 11:
Compute prob.
vector z h,m =  ( q h ) (Eq.
and leaves’ prob.
l ( z h,m ) (Eq.
Estimate optimal coupling ⇣ h  1 ,h,m between l ( z h  1 ,m ) and l ( z h,m ) as in Eq.
Sample action x h,m ⇠ ⇣ h  1 ,h,m ( ·| x h  1 ,m ) and observe y h,m =
f ( x h,m , e h,m ) + ⇠ h,m 14: end for 15:
Update µ m +1 ( · , · ) and  m +1 ( · , · ) as per Eq.
(5) and
(6) 16: end for 2.2 Relation to Metrical Task Systems (MTS)
When f is known, our optimization objective in Eq.
(2) can be seen as a particular type of MTS problem, where f ( · , e h,m ) is the MTS service cost that changes for every h and m .
Compared to a standard MTS (see Appendix A), our problem formulation is more challenging since the learner can only learn about f from previously observed data.
The approach proposed in this paper builds on the algorithm by Coester and Lee [19] for standard MTS problems. However, to cope with the aforementioned challenge, our approach exploits the regularity assumptions regarding f and utilizes the constructed lower conﬁdence bounds Eq. (10) to hallucinate information about the unavailable service cost at each round. Before presenting our overall approach, we describe a preliminary step proposed by [19], which consists of representing our metric space ( X , d ) by a Hierarchically
Separated Tree (HST) metric space.
HST metric space.
Consider a tree T
= ( V, E ) with root r , leaves L ⇢ V and non-negative weights w v , for each v 2 V , which are non-increasing along root-leaf paths.
Let d T ( l, l 0 ) denote a distance metric between any two leaves l, l 0 2 L given as the sum of the encountered weights on the path from l to l 0 (see Fig. 4).
( L , d T ) is a HST metric space, and ⌧ -HST metric space if the weights are exponentially decreasing, i.e., w u  w v / ⌧ , with v being the parent of u .
Similarly to [19], we use the algorithm from [23] (which we name via the authors’ surnames as
FRT) to approximate the given metric space ( X , d ) by a ⌧ -HST one.
In particular, we use FRT in Algorithm 1 as a computationally efﬁcient preprocessing step to create a tree T with leaves L corresponding to actions in X , distance metric d T , and root node r .
We explain the intrinsic MTS motivation for this preprocessing step in Appendix B.1, and defer additional details to Appendix B.2.
The GP-MD Algorithm
In this section, we introduce GP-MD, a novel algorithm for the contextual BO problem with movement costs deﬁned in Section 2.
At each episode m and round h , the state of GP-MD can be summarized by a vector of probabilities z h,m 2 K T over the vertices of T , where K T := n z
R | V | + : z r = 1 , z u = P ⌫ 2 C ( u ) z ⌫ 8 u 2 V \L o , and C ( u ) denotes the children of u .
Each entry z ⌫ represents the probability that the selected action x h,m belongs to the leaves of the subtree rooted at ⌫ , i.e., z ⌫ = P ( x h,m 2 L ( ⌫ )) .
Below, we specify how z h,m is computed at each round.
Moreover, given any z 2 K T , the vector
[Equation]
deﬁnes a probability distribution over the leaves L , and hence the actions X .
As in [19], given probability vectors z h,m and z h  1 ,m , GP-MD computes the minimal distance distribution [Equation]
[Equation]
where U h  1 ,m and U h are random variables having marginals l ( z h  1 ,m ) and l ( z h,m ) respec- tively
Finally, action x h,m is sampled from the conditional minimal distance distribution x h,m ⇠ ⇣ h  1 ,h,m ( ·| x h  1 ,m ) (Line 13 in Algorithm 1).
At the end of each episode m , the newly observed data are then used to update posterior mean and standard deviation about the cost function.
Finally, we describe how probability vectors
z h,m are computed at each round, a key step of GP-MD (Lines 8–12 in Algorithm 1).
We follow the recursive Mirror Descent (MD) procedure proposed by [19], with the important difference that we are dealing with an unknown context-dependent cost function.
Hence, we make use of the Gaussian process model and corresponding conﬁdence estimates as deﬁned in Section 2.1.
To obtain probabilities z h,m , we consider conditional probability vectors q 2 Q T , where Q T is the set of valid conditional probabilities Q T :=
[Equation]
o .
For each vertex ⌫ with parent u , q ⌫ represents the conditional probability P ( x h,m 2 L ( ⌫ ) | x h,m 2 L ( u )) .
Moreover, given q h 2 Q T we deﬁne the vector q ( u ) h
[Equation] 2 C ( u )] as the conditional distribution over children of u , and let Q ( u ) T be the set of all valid distributions q ( u ) h .
In each episode m , conditional probability vector q h for round h is obtained recursively, from leaves to root, as a function of q h  1 , the observed context e h,m , and the current estimate about the cost associated to each particular vertex.
More precisely, let OD ( V \L ) be a topological ordering of the internal vertices V \L so that every child in T occurs before its parent.
Then, for each u 2 OD ( V \L ) conditional probabilities q ( u ) h are obtained via the Mirror Descent update:
[Equation]
Function D ( u ) is the Bregman divergence with respect to a suitable potential function (see Appendix D.1), while lcb ( u ) m ( · , e h,m ) :
= [ lcb m ( ⌫ , e h,m ) , 8 ⌫ 2 C ( u )] is a lower conﬁdence bound estimate of the costs corresponding to children of vertex u .
For v 2 L , lcb m ( ⌫ , e h,m ) are obtained by the GP-regression techniques outlined in Section 2.1, while for internal vertices these are computed recursively from their children nodes as:
[Equation]
The movement cost is primarily controlled by this usage of Bregman divergence based mirror descent.
Also, sampling from the conditional minimal distance distribution further restricts movement between alternate actions.
Once the vector of conditional probabilities q h 2 Q T has been updated, we can obtain the corresponding probability vector z h,m via the mapping  : Q T ! K T such that:
[Equation]
3.1 Theoretical Guarantees
Our main theorem bounds the cumulative regret of GP-MD.
Theorem 1.
Let X be represented by a ⌧ -HST space with ⌧ > 4 (Line 2 of Algorithm 1), and set  2 (0 , 1) .
Then, with probability at least 1   , the regret of GP-MD over N ep episodes is bounded by
[Equation]
⌘
[Equation]
with approximation factors ↵ =
O  (log n ) and  = O (1) .
Here, H is the episodes’ length,  N ep is the conﬁdence level from Lemma 1, and  HN ep is the maximum information gain deﬁned in Eq.
[Figure]
For most of the popularly used kernels, Thm. 1 can be made more explicit by substituting bounds on  HN ep (e.g., in the case of linear kernel and compact domain, we have  t
=
O
( p log t ) , while for squared-exponential kernel it holds  t
= O ((log t ) p +1 )
[50]; see also Section 2.1).
In such cases, we make the following two important observations regarding our result: i)
The obtained regret bound is sublinear in the number of episodes N ep and hence lim N ep !
1 R ↵ ,  N ep /N
ep
= 0 ; ii)
The bound is independent of the input space size, i.e., the number of actions n (although the approximation factor ↵ depends logarithmically on n , similarly to [19]).
These imply that GP-MD approaches ↵ -competitive ratio performance of the MTS algorithm by [19], while learning about the service cost from noisy point evaluations (i.e., bandit feedback) only.
Finally, in our analysis, we treat H as constant.
Proof of Thm.
1 is detailed in
Appendix E. Next, we outline some main steps.
We make use of the competitive ratio guarantees for the used Mirror Descent algorithm from [19, Corollary 4] to bound the expected hallucinated service cost
[Equation]
Here, we use the fact that lcb m ( · , · ) does not change within an episode.
Since this is not the actual service cost, we bound
P Hh =1 h f ( · , e h,m ) , l ( z h,m ) i by P [Equation]
[Equation]
i with an additional learning error.
The sum of the learning errors over all episodes can be rewritten as P [Equation] , l
[Equation] z h, [Equation] ) i .
We use the concentration of the conditional mean result from [30, Lemma 3] to upper bound it by the actual realizations P [Equation] x
[Equation]
[Equation]
and use the result of [18, Lemma-2], to further upper bound it with the maximum information gain quantity  N ep H . Finally, the movement cost can also be bounded similar to that of the previously mentioned expected hallucinated service cost [Equation] with an
[Equation]
factor using [19, Corollary 4].
Experiments
This section provides numerical results on synthetic and real-world data.
We compare the performance of our GP-MD algorithm with the following baselines:
• S TATIONARY selects the stationary strategy x
h = x 0 for all h , • CGP-LCB
[32] neglects the movement cost and sets
x h = arg min x lcb h ( x, e h ) for all h , • M IN C-K NOWN assumes f ( · ) is known and chooses x h = arg min x f ( x, e h ) , and • MD-K NOWN assumes f ( · ) is known and runs mirror descent from [19] on f ( · , e h ) .
MD-K NOWN and M IN C-K NOWN unrealistically assume that f ( · ) is known and can be seen as upper-bound for the achievable performance of GP-MD and CGP-LCB, respectively.
We use the same constant value  = 2 . 0 for the exploration parameter in both GP-MD and CGP-LCB (since the theoretical worst-case bounds are found to be overly pessimistic and can impede performance [50]).
We run the algorithms over a single episode (as done in CGP-LCB).
We also discover that updating the conﬁdence bounds after every timestep in GP-MD leads to better performance in practice.
Synthetic experiments.
We consider synthetic experiments, where the objective function is a random
GP sample.
The considered action space X is a subset of [0 , 1] 2 consisting of 400 points that form the uniform grid, while the context space E consists of 40 contexts that are uniformly sampled from (0 , 1) .
We sample objective function (i.e., actual cost)
f : X ⇥ E ! R from a GP (0 , k ) , where k is a squared exponential kernel with lengthscale parameter set to l = 0 .
We use the Euclidean distance between the domain points as the movement cost, calculate the distance matrix (between every pair of points) and the average movement cost.
We subtract the minimum value from f and scale it such that the average function value is equal to the average movement cost.
We also set the noise parameter to 1% of the function range.
We introduce the trade-off parameter ⇢ 2 { 0 . 25 , 0 .
5 , 1 , 2 , 4 } that only multiplies the service cost, i.e, ⇢ f ( x, e ) , but not the movement cost.
This is to test the performance of the algorithms for varying importance of the service/movement costs.
For each ⇢ we sample 25 different functions and run the algorithms for 500 timesteps wherein at each step the contexts are randomly sampled.
In Fig.
2a-Fig.
2e, we show the total cumulative cost as a function of timesteps for different ⇢ values.
Then, in Fig.
2f, we show the performance of the algorithms (for known kernel parameters) when run for 800 timesteps for varying importance of the service and movement costs.
In particular, we consider a convex combination of the service and movement costs, where we set the respective coefﬁcients multiplying these two objectives as ⇢ / (1 + ⇢ ) and 1 / (1 + ⇢ ) .
As shown in Fig.
2a-Fig.
2c, the performance of GP-MD is generally close to the one of the idealized, unrealistic benchmark MD-K NOWN , which, as expected, performs the best.
The stationary baseline performs comparably when ⇢ is small, while its performance deteriorates for larger values.
As expected, both M IN C-K NOWN and CGP-LCB incur higher total costs than GP-MD when the movement cost is of the higher or same relative importance as the service cost (i.e., ⇢ 2 { 0 . 5 , 1 . 0 } ), while the performance gap slowly decreases when the service cost becomes dominant ( ⇢ = 2 . 0 ).
In Figs. 2d and 2e, we also show the corresponding movement costs, and observe that movement cost ignorant CGP-LCB incurs signiﬁcant movement costs, while our GP-MD successfully minimizes the movement costs.
Finally, in Fig.
2f, we observe that the performance of GP-MD is robust, i.e., it clearly outperforms CGP-LCB whenever the movement cost dominates the total cost objective, while its performance remains comparable to the one of CGP-LCB (that is built to minimize service cost) when the movement cost becomes dominated by the service cost.
4.1 Altitude Optimization in AWE Systems
In airborne wind energy (AWE) systems, the turbine’s operating altitude can be changed depending on the wind pattern.
We follow the setup of Baheri et al.
[7] that applied CGP-LCB [32] which ignores movement-costs, to learn this control task.
In this section, we use a dataset from [9] which contains wind-speed information over various locations in Europe for a period ranging fromto 2017, and also includes measurements at different altitudes per location.
We consider the wind speed data from the second half of 2016.
Our goal is to maximize the generated energy, while taking into account the energy loss due to moving the turbine from one altitude to another.
We consider 25 different altitudes (ranging from 10 m to 1600 m) as the action space and the context space to be hours in the day (i.e., 24 values).
We deﬁne our unknown service objective function to be f ( x, t ) = max x 0 ( E S ( x 0 , t ))
E S ( x, t ) where E S ( x, t ) denotes the energy generated based on the windspeed at altitude x and time t .
Based on the discrete-time power generation formula from Baheri et al.
[7] (Eq.
(10)), we have
[Equation]
Here V w ( x, t ) denotes the windspeed at altitude x and time t , and V r denotes the rated windspeed of the turbine.
The constants c 1 = 0 .
0579 and c 2 = 0 .
09 are system dependent.
This corresponds to the energy generated at a particular altitude x for  t time.
Similarly to Baheri et al.
[7], we use  t = 60 since we consider intervals of one hour length.
Next, we deﬁne the movement cost to be [Figure] the energy lost in changing altitude (from x to x 0 ):
[Equation]
(17) where c 3 = 0 .
15 (see Appendix F for more details).
We assume that a wind speed gauge is attached to the turbine, and the operator knows the wind speed at the current altitude.
Hence, instead of directly learning f ( x, t ) , we learn V w ( x, t ) and use its conﬁdence bounds to calculate the conﬁdence bounds of f ( x, t ) .
To learn V w ( x, t ) , we normalize the inputs, and ﬁt a GP with RBF kernel (lengthscale=3.67, outputscale=6.85 and noise parameter=2.73).
We run the algorithms for different ⇢ for 960 timesteps, where again ⇢ is used to multiply E S .
For each ⇢ , the algorithms were initiated with every possible starting point (25 different altitudes), and ran for 3 iterations.
Based on this we plot the total energy generated w.r.t.
varying ⇢ in Fig.
In Figs. 3a and 3f, we show the performance of the algorithms at two different locations (we also consider additional locations and time periods in Appendix F).
We use different values of ⇢ > 1 to show the robustness of our algorithm (as ⇢ increases, the importance of the service cost w.r.t.
the movement cost in the overall objective increases).
Our algorithm outperforms CGP-LCB for a range of ⇢ values.
As ⇢ keeps increasing, we observe that M IN C-K NOWN closes the performance gap to MD-K NOWN , and the same is happening with CGP-LCB w.r.t.
GP-MD.
In Fig.
3b, we focus on a particular ⇢ = 4 , and notice that GP-MD performs better than CGP-LCB and S TATIONARY algorithm at this location.
In Fig.
3c, we plot the service cost and observe that both learning algorithms GP-MD and
CGP-LCB have lower service cost than the S TATIONARY baseline.
We also note that due to the implicit service cost deﬁnition, the M IN C-K NOWN baseline achieves zero service cost.
In Figs.
3d and 3e, we compare the movement energy loss for ⇢ = 4 and ⇢ = 1 .
As expected, ⇢ = 1 results in slightly lower GP-MD movement energy loss due to the tradeoff shifting towards the movement cost.
Conclusions
We have considered the problem of optimizing an unknown cost function subject to time-varying contextual information, as well as movement costs of changing the selected action from round to round.
Our problem formulation is motivated by Airborne Wind Energy systems, where one seeks to optimize the operating altitude of the wind turbine to maximize the amount of generated energy.
We propose a novel algorithm, GP-MD, which makes use of GP conﬁdence bounds and employs the mirror descent techniques from [19] for solving MTS problems.
We analyze the theoretical performance of our algorithm by providing a rigorous regret bound.
Moreover, we demonstrate its performance in synthetic experiments and on an AWE application by using real-world data.
GP-MD carefully trades off service and movement costs while at the same time learning about the unknown objective function and yielding improved performance (i.e., generating more energy) compared to the considered baselines.
Our setup and analysis open up multiple interesting directions for further exploration.
For instance, an extension to continuous action spaces via discretization arguments is an immediate direction for future work.
Another interesting direction is to analyze the single-episode setting and obtain general sublinear regret guarantees.
Acknowledgements
The authors would like to thank James R. Lee and Christian Coester for the various discussions regarding their paper [19] during the course of this work.
This project has received funding from the
European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme grant agreement
No