The on-line shortest path problem under partial monitoring
András György Tamás Linder Gábor Lugosi
György Ottucsák ∗
October 27, 2018
Abstract
The on-line shortest path problem is considered under various models of partial monitor-
ing. Given a weighted directed acyclic graph whose edge weights can change in an arbitrary
(adversarial) way, a decision maker has to choose in each round of a game a path between two
distinguished vertices such that the loss of the chosen path (defined as the sum of the weights of
its composing edges) be as small as possible. In a setting generalizing the multi-armed bandit
problem, after choosing a path, the decision maker learns only the weights of those edges that
belong to the chosen path. For this problem, an algorithm is given whose average cumulative
loss in n rounds exceeds that of the best path, matched off-line to the entire sequence of the
edge weights, by a quantity that is proportional to 1/
n and depends only polynomially on the
number of edges of the graph. The algorithm can be implemented with linear complexity in the
number of rounds n and in the number of edges. An extension to the so-called label efficient
setting is also given, in which the decision maker is informed about the weights of the edges
corresponding to the chosen path at a total of m ≪ n time instances. Another extension is
shown where the decision maker competes against a time-varying path, a generalization of the
problem of tracking the best expert. A version of the multi-armed bandit setting for shortest
path is also discussed where the decision maker learns only the total weight of the chosen path
but not the weights of the individual edges on the path. Applications to routing in packet
switched networks along with simulation results are also presented.
Index Terms: On-line learning, shortest path problem, multi-armed bandit problem.
∗A. György is with the Machine Learning Research Group, Computer and Automation Research Institute of the
Hungarian Academy of Sciences, Kende u. 11-13, Budapest, Hungary, H-1111 (email: gya@szit.bme.hu). T. Linder
is with the Department of Mathematics and Statistics, Queen’s University, Kingston, Ontario, Canada K7L 3N6
(email: linder@mast.queensu.ca). G. Lugosi is with ICREA and Department of Economics, Universitat Pompeu
Fabra, Ramon Trias Fargas 25-27, 08005 Barcelona, Spain (email: gabor.lugosi@gmail.com). Gy. Ottucsák is
with Department of Computer Science and Information Theory, Budapest University of Technology and Economics,
Magyar Tudósok Körútja 2., Budapest, Hungary, H-1117. He is also with the Machine Learning Research Group,
Computer and Automation Research Institute of the Hungarian Academy of Sciences (email: oti@szit.bme.hu).
This research was supported in part by the János Bolyai Research Scholarship of the Hungarian Academy of Sciences,
the Mobile Innovation Center of Hungary, by the Natural Sciences and Engineering Research Council (NSERC) of
Canada, by the Spanish Ministry of Science and Technology grant MTM2006-05650, by the PASCAL Network of
Excellence under EC grant no. 506778 and by the High Speed Networks Laboratory, Department of Telecommuni-
cations and Media Informatics, Budapest University of Technology and Economics. Parts of this paper have been
presented at COLT’06.
http://arxiv.org/abs/0704.1020v1
1 Introduction
In a sequential decision problem, a decision maker (or forecaster) performs a sequence of actions.
After each action the decision maker suffers some loss, depending on the response (or state) of the
environment, and its goal is to minimize its cumulative loss over a certain period of time. In the
setting considered here, no probabilistic assumption is made on how the losses corresponding to
different actions are generated. In particular, the losses may depend on the previous actions of the
decision maker, whose goal is to perform well relative to a set of reference forecasters (the so-called
“experts”) for any possible behavior of the environment. More precisely, the aim of the decision
maker is to achieve asymptotically the same average (per round) loss as the best expert.
Research into this problem started in the 1950s (see, for example, Blackwell [5] and Hannan
[18] for some of the basic results) and gained new life in the 1990s following the work of Vovk [29],
Littlestone and Warmuth [24], and Cesa-Bianchi et al. [7]. These results show that for any bounded
loss function, if the decision maker has access to the past losses of all experts, then it is possible
to construct on-line algorithms that perform, for any possible behavior of the environment, almost
as well as the best of N experts. More precisely, the per round cumulative loss of these algorithms
is at most as large as that of the best expert plus a quantity proportional to
lnN/n for any
bounded loss function, where n is the number of rounds in the decision game. The logarithmic
dependence on the number of experts makes it possible to obtain meaningful bounds even if the
pool of experts is very large.
In certain situations the decision maker has only limited knowledge about the losses of all
possible actions. For example, it is often natural to assume that the decision maker gets to know
only the loss corresponding to the action it has made, and has no information about the loss it would
have suffered had it made a different decision. This setup is referred to as the multi-armed bandit
problem, and was considered, in the adversarial setting, by Auer et al. [1] who gave an algorithm
whose normalized regret (the difference of the algorithm’s average loss and that of the best expert)
is upper bounded by a quantity which is proportional to
N lnN/n. Note that, compared to the
full information case described above where the losses of all possible actions are revealed to the
decision maker, there is an extra
N factor in the performance bound, which seriously limits the
usefulness of the bound if the number of experts is large.
Another interesting example for the limited information case is the so-called label efficient
decision problem (see Helmbold and Panizza [22]) in which it is too costly to observe the state of
the environment, and so the decision maker can query the losses of all possible actions for only a
limited number of times. A recent result of Cesa-Bianchi, Lugosi, and Stoltz [9] shows that in this
case, if the decision maker can query the losses m times during a period of length n, then it can
achieve O(
lnN/m) average excess loss relative to the best expert.
In many applications the set of experts has a certain structure that may be exploited to construct
efficient on-line decision algorithms. The construction of such algorithms has been of great interest
in computational learning theory. A partial list of works dealing with this problem includes Herbster
and Warmuth [19], Vovk [30], Bousquet and Warmuth [6], Helmbold and Schapire [27], Takimoto
and Warmuth [28], Kalai and Vempala [23], György at al. [13, 14, 15]. For a more complete survey,
we refer to Cesa-Bianchi and Lugosi [8, Chapter 5].
In this paper we study the on-line shortest path problem, a representative example of structured
expert classes that has received attention in the literature for its many applications, including,
among others, routing in communication networks; see, e.g., Takimoto andWarmuth [28], Awerbuch
et al. [3], or György and Ottucsák [17], and adaptive quantizer design in zero-delay lossy source
coding; see, György et al. [13, 14, 16]. In this problem, a weighted directed (acyclic) graph is given
whose edge weights can change in an arbitrary manner, and the decision maker has to pick in each
round a path between two given vertices, such that the weight of this path (the sum of the weights
of its composing edges) be as small as possible.
Efficient solutions, with time and space complexity proportional to the number of edges rather
than to the number of paths (the latter typically being exponential in the number of edges), have
been given in the full information case, where in each round the weights of all the edges are revealed
after a path has been chosen; see, for example, Mohri [26], Takimoto and Warmuth [28], Kalai and
Vempala [23], and György et al. [15].
In the bandit setting only the weights of the edges or just the sum of the weights of the edges
composing the chosen path are revealed to the decision maker. If one applies the general bandit
algorithm of Auer et al. [1], the resulting bound will be too large to be of practical use because
of its square-root-type dependence on the number of paths N . On the other hand, using the
special graph structure in the problem, Awerbuch and Kleinberg [4] and McMahan and Blum [25]
managed to get rid of the exponential dependence on the number of edges in the performance bound.
They achieved this by extending the exponentially weighted average predictor and the follow-the-
perturbed-leader algorithm of Hannan [18] to the generalization of the multi-armed bandit setting
for shortest paths, when only the sum of the weights of the edges is available for the algorithm.
However, the dependence of the bounds obtained in [4] and [25] on the number of rounds n is
significantly worse than the O(1/
n) bound of Auer et al. [1]. Awerbuch and Kleinberg [4] consider
the model of “non-oblivious” adversaries for shortest path (i.e., the losses assigned to the edges can
depend on the previous actions of the forecaster) and prove an O(n−1/3) bound for the expected
per-round regret. McMahan and Blum [25] give a simpler algorithm than in [4] however obtain a
bound of the order of O(n−1/4) for the expected regret.
In this paper we provide an extension of the bandit algorithm of Auer et al. [1] unifying the
advantages of the above approaches, with a performance bound that is polynomial in the number
of edges, and converges to zero at the right O(1/
n) rate as the number of rounds increases. We
achieve this bound in a model which assumes that the losses of all edges on the path chosen by the
forecaster are available separately after making the decision. We also discuss the case (considered
by [4] and [25]) in which only the total loss (i.e., the sum of the losses on the chosen path) is known
to the decision maker. We exhibit a simple algorithm which achieves an O(n−1/3) per-round regret
with high probability against “non-oblivious” adversary. In this case it remains an open problem
to find an algorithm whose cumulative loss is polynomial in the number of edges of the graph and
decreases as O(n−1/2) with the number of rounds.
In Section 2 we formally define the on-line shortest path problem, which is extended to the
multi-armed bandit setting in Section 3. Our new algorithm for the shortest path problem in
the bandit setting is given in Section 4 together with its performance analysis. The algorithm
is extended to solve the shortest path problem in a combined label efficient multi-armed bandit
setting in Section 5. Another extension, when the algorithm competes against a time-varying path
is studied in Section 6. An algorithm for the “restricted” multi-armed bandit setting (when only the
sums of the losses of the edges are available) is given in Section 7. Simulation results are presented
in Section 8.
2 The shortest path problem
Consider a network represented by a set of vertices connected by edges, and assume that we have
to send a stream of packets from a distinguished vertex, called source, to another distinguished
vertex, called destination. At each time slot a packet is sent along a chosen route connecting source
and destination. Depending on the traffic, each edge in the network may have a different delay, and
the total delay the packet suffers on the chosen route is the sum of delays of the edges composing
the route. The delays may change from one time slot to the next one in an arbitrary way, and
our goal is to find a way of choosing the route in each time slot such that the sum of the total
delays over time is not significantly more than that of the best fixed route in the network. This
adversarial version of the routing problem is most useful when the delays on the edges can change
dynamically, even depending on our previous routing decisions. This is the situation in the case
of ad-hoc networks, where the network topology can change rapidly, or in certain secure networks,
where the algorithm has to be prepared to handle denial of service attacks, that is, situations where
willingly malfunctioning vertices and links increase the delay; see, e.g., Awerbuch et al. [3].
This problem can be cast naturally as a sequential decision problem in which each possible
route is represented by an action. However, the number of routes is typically exponentially large
in the number of edges, and therefore computationally efficient algorithms are called for. Two
solutions of different flavor have been proposed. One of them is based on a follow-the-perturbed-
leader forecaster, see Kalai and Vempala [23], while the other is based on an efficient computation
of the exponentially weighted average forecaster, see, for example, Takimoto and Warmuth [28].
Both solutions have different advantages and may be generalized in different directions.
To formalize the problem, consider a (finite) directed acyclic graph with a set of edges E =
{e1, . . . , e|E|} and a set of vertices V . Thus, each edge e ∈ E is an ordered pair of vertices (v1, v2).
Let u and v be two distinguished vertices in V . A path from u to v is a sequence of edges e(1), . . . , e(k)
such that e(1) = (u, v1), e
(j) = (vj−1, vj) for all j = 2, . . . , k − 1, and e(k) = (vk−1, v). Let
P = {i1, . . . , iN} denote the set of all such paths. For simplicity, we assume that every edge in
E is on some path from u to v and every vertex in V is an endpoint of an edge (see Figure 1 for
examples).
PSfrag replacements
Figure 1: Two examples of directed acyclic graphs for the shortest path problem.
(a) (b)
In each round t = 1, . . . , n of the decision game, the decision maker chooses a path It among
all paths from u to v. Then a loss ℓe,t ∈ [0, 1] is assigned to each edge e ∈ E. We write e ∈ i if the
edge e ∈ E belongs to the path i ∈ P, and with a slight abuse of notation the loss of a path i at
time slot t is also represented by ℓi,t. Then ℓi,t is given as
ℓi,t =
and therefore the cumulative loss up to time t of each path i takes the additive form
Li,t =
ℓi,s =
where the inner sum on the right-hand side is the loss accumulated by edge e during the first t
rounds of the game. The cumulative loss of the algorithm is
L̂t =
ℓIs,s =
ℓe,s .
It is well known that for a general loss sequence, the decision maker must be allowed to use
randomization to be able to approximate the performance of the best expert (see, e.g., Cesa-Bianchi
and Lugosi [8]). Therefore, the path It is chosen randomly according to some distribution pt over
all paths from u to v. We study the normalized regret over n rounds of the game
L̂n −min
where the minimum is taken over all paths i from u to v.
For example, the exponentially weighted average forecaster ([29], [24], [7]), calculated over all
possible paths, has regret
L̂n −min
ln(1/δ)
with probability at least 1− δ, where N is the total number of paths from u to v in the graph and
K is the length of the longest path.
3 The multi-armed bandit setting
In this section we discuss the “bandit” version of the shortest path problem. In this setup, which is
more realistic in many applications, the decision maker has only access to the losses corresponding
to the paths it has chosen. For example, in the routing problem this means that information is
available on the delay of the route the packet is sent on, and not on other routes in the network.
We distinguish between two types of bandit problems, both of which are natural generalizations
of the simple bandit problem to the shortest path problem. In the first variant, the decision maker
has access to the losses of those edges that are on the path it has chosen. That is, after choosing a
path It at time t, the value of the loss ℓe,t is revealed to the decision maker if and only if e ∈ It.
We study this case and its extensions in Sections 4, 5, and 6.
The second variant is a more restricted version in which the loss of the chosen path is observed,
but no information is available on the individual losses of the edges belonging to the path. That
is, after choosing a path It at time t, only the value of the loss of the path ℓIt,t is revealed to the
decision maker. Further on we call this setting as the restricted bandit problem for shortest path.
We consider this restricted problem in Section 7.
Formally, the on-line shortest path problem in the multi-armed bandit setting is described as
follows: at each time instance t = 1, . . . , n, the decision maker picks a path It ∈ P from u to v.
Then the environment assigns loss ℓe,t ∈ [0, 1] to each edge e ∈ E, and the decision maker suffers
loss ℓIt,t =
ℓe,t. In the unrestricted case the losses ℓe,t are revealed for all e ∈ It, while in
the restricted case only ℓIt,t is revealed. Note that in both cases ℓe,t may depend on I1, . . . , It−1,
the earlier choices of the decision maker.
For the basic multi-armed bandit problem, Auer et al. [1] gave an algorithm, based on exponen-
tial weighting with a biased estimate of the gains (defined, in our case, as gi,t = K− ℓi,t), combined
with uniform exploration. Applying their algorithm to the on-line shortest path problem in the
bandit setting results in a performance that can be bounded, for any 0 < δ < 1 and fixed time
horizon n, with probability at least 1− δ, by
L̂n −min
≤ 11K
N ln(N/δ)
K lnN
(The constants follow from a slightly improved version; see Cesa-Bianchi and Lugosi [8].)
However, for the shortest path problem this bound is unacceptably large because, unlike in the
full information case, here the dependence on the number of all paths N is not merely logarithmic,
while N is typically exponentially large in the size of the graph (as in the two simple examples
of Figure 1). In order to achieve a bound that does not grow exponentially with the number of
edges of the graph, it is imperative to make use of the dependence structure of the losses of the
different actions (i.e., paths). Awerbuch and Kleinberg [4] and McMahan and Blum [25] do this
by extending low complexity predictors, such as the follow-the-perturbed-leader forecaster [18],
[23] to the restricted bandit setting. However, in both cases the price to pay for the polynomial
dependence on the number of edges is a worse dependence on the length n of the game.
4 A bandit algorithm for shortest paths
In this section we describe a variant of the bandit algorithm of [1] which achieves the desired
performance for the shortest path problem. The new algorithm uses the fact that when the losses
of the edges of the chosen path are revealed, then this also provides some information about the
losses of each path sharing common edges with the chosen path.
For each edge e ∈ E, and t = 1, 2, . . ., introduce the gain ge,t = 1− ℓe,t, and for each path i ∈ P,
let the gain be the sum of the gains of the edges on the path, that is,
gi,t =
ge,t .
The conversion from losses to gains is done in order to facilitate the subsequent performance
analysis. To simplify the conversion, we assume that each path i ∈ P is of the same length K
for some K > 0. Note that although this assumption may seem to be restrictive at the first
glance, from each acyclic directed graph (V,E) one can construct a new graph by adding at most
(K−2)(|V |−2)+1 vertices and edges (with constant weight zero) to the graph without modifying
the weights of the paths such that each path from u to v will be of length K, where K denotes the
length of the longest path of the original graph. If the number of edges is quadratic in the number
of vertices, the size of the graph is not increased substantially.
A main feature of the algorithm below is that the gains are estimated for each edge and not
for each path. This modification results in an improved upper bound on the performance with
the number of edges in place of the number of paths. Moreover, using dynamic programming as
in Takimoto and Warmuth [28], the algorithm can be computed efficiently. Another important
ingredient of the algorithm is that one needs to make sure that every edge is sampled sufficiently
often. To this end, we introduce a set C of covering paths with the property that for each edge
e ∈ E there is a path i ∈ C such that e ∈ i. Observe that one can always find such a covering set
of cardinality |C| ≤ |E|.
We note that the algorithm of [1] is a special case of the algorithm below: For any multi-
armed bandit problem with N experts, one can define a graph with two vertices u and v, and N
directed edges from u to v with weights corresponding to the losses of the experts. The solution
of the shortest path problem in this case is equivalent to that of the original bandit problem with
choosing expert i if the corresponding edge is chosen. For this graph, our algorithm reduces to the
original algorithm of [1].
A BANDIT ALGORITHM FOR SHORTEST PATHS
Parameters: real numbers β > 0, 0 < η, γ < 1.
Initialization: Set we,0 = 1 for each e ∈ E, wi,0 = 1 for each i ∈ P, and W 0 = N .
For each round t = 1, 2, . . .
(a) Choose a path It at random according to the distribution pt on P, defined by
pi,t =
(1− γ)wi,t−1
W t−1
if i ∈ C
(1− γ)wi,t−1
W t−1
if i 6∈ C.
(b) Compute the probability of choosing each edge e as
qe,t =
i:e∈i
pi,t = (1− γ)
i:e∈iwi,t−1
W t−1
|{i ∈ C : e ∈ i}|
(c) Calculate the estimated gains
g′e,t =
ge,t+β
if e ∈ It
otherwise.
(d) Compute the updated weights
we,t = we,t−1e
ηg′e,t
wi,t =
we,t = wi,t−1e
where g′
i,t =
e∈i g
e,t, and the sum of the total weights of the paths
W t =
wi,t.
The main result of the paper is the following performance bound for the shortest-path bandit
algorithm. It states that the per round regret of the algorithm, after n rounds of play, is, roughly,
of the order of K
|E| lnN/n where |E| is the number of edges of the graph, K is the length of the
paths, and N is the total number of paths.
Theorem 1 For any δ ∈ (0, 1) and parameters 0 ≤ γ < 1/2, 0 < β ≤ 1, and η > 0 satisfying
2ηK|C| ≤ γ, the performance of the algorithm defined above can be bounded, with probability at
least 1− δ, as
L̂n −min
≤ Kγ + 2ηK2|C|+
+ |E|β.
In particular, choosing β =
, γ = 2ηK|C|, and η =
4nK2|C|
yields for all n ≥
, 4|C| lnN
L̂n −min
4K|C| lnN +
|E| ln |E|
The proof of the theorem is based on the analysis of the original algorithm of [1] with necessary
modifications required to transform parts of the argument from paths to edges, and to use the
connection between the gains of paths sharing common edges.
For the analysis we introduce some notation:
Gi,n =
gi,t and G
i,n =
g′i,t
for each i ∈ P and
Ge,n =
ge,t and G
e,n =
g′e,t
for each e ∈ E, and
Ĝn =
gIt,t.
The following lemma, shows that the deviation of the true cumulative gain from the estimated
cumulative gain is of the order of
n. The proof is a modification of [8, Lemma 6.7].
Lemma 1 For any δ ∈ (0, 1), 0 ≤ β < 1 and e ∈ E we have
Ge,n > G
e,n +
Proof. Fix e ∈ E. For any u > 0 and c > 0, by the Chernoff bound we have
P[Ge,n > G
e,n + u] ≤ e−cuEec(Ge,n−G
e,n) . (1)
Letting u = ln(|E|/δ)/β and c = β, we get
e−cuEec(Ge,n−G
e,n) = e− ln(|E|/δ)Eeβ(Ge,n−G
e,n) =
Eeβ(Ge,n−G
e,n) ,
so it suffices to prove that Eeβ(Ge,n−G
e,n) ≤ 1 for all n. To this end, introduce
Zt = e
β(Ge,t−G
e,t) .
Below we show that Et[Zt] ≤ Zt−1 for t ≥ 2 where Et denotes the conditional expectation
E[·|I1, . . . , It−1] . Clearly,
Zt = Zt−1 exp
ge,t −
1{e∈It}ge,t + β
Taking conditional expectations, we obtain
Et[Zt]
= Zt−1Et
ge,t −
1{e∈It}ge,t + β
= Zt−1e
qe,t Et
ge,t −
1{e∈It}ge,t
≤ Zt−1e
qe,t Et
1 + β
ge,t −
1{e∈It}ge,t
ge,t −
1{e∈It}ge,t
= Zt−1e
qe,t Et
1 + β2
ge,t −
1{e∈It}ge,t
≤ Zt−1e
qe,t Et
1 + β2
1{e∈It}ge,t
≤ Zt−1e
≤ Zt−1. (4)
Here (2) holds since β ≤ 1, ge,t −
1{e∈It}
≤ 1 and ex ≤ 1 + x + x2 for x ≤ 1. (3) follows from
1{e∈It}
= ge,t. Finally, (4) holds by the inequality 1 + x ≤ ex. Taking expectations on both
sides proves E[Zt] ≤ E[Zt−1]. A similar argument shows that E[Z1] ≤ 1, implying E[Zn] ≤ 1 as
desired. ✷
Proof of Theorem 1. As usual in the analysis of exponentially weighted average forecasters, we
start with bounding the quantity ln Wn
. On the one hand, we have the lower bound
i,n − lnN ≥ ηmax
G′i,n − lnN . (5)
To derive a suitable upper bound, first notice that the condition η ≤ γ
2K|C|
implies ηg′
i,t ≤ 1 for
all i and t, since
ηg′i,t = η
g′e,t ≤ η
1 + β
≤ ηK(1 + β)|C|
where the second inequality follows because qe,t ≥ γ/|C| for each e ∈ E.
Therefore, using the fact that ex ≤ 1 + x+ x2 for all x ≤ 1, for all t = 1, 2, . . . we have
W t−1
wi,t−1
W t−1
pi,t − γ|C|1{i∈C}
pi,t − γ|C|1{i∈C}
1 + ηg′i,t + η
ηg′i,t + η
pi,tg
i,t +
pi,tg
i,t (7)
where (6) follows form the definition of pi,t, and (7) holds by the inequality ln(1 + x) ≤ x for all
x > −1.
Next we bound the sums in (7). On the one hand,
pi,tg
i,t =
g′e,t =
g′e,t
i∈P:e∈i
g′e,tqe,t = gIt,t + |E|β.
On the other hand,
pi,tg
i,t =
g′e,t
pi,tK
i∈P:e∈i
e,tqe,t
qe,tg
β + 1{e∈It}ge,t
≤ K(1 + β)
g′e,t
where the first inequality is due to the inequality between the arithmetic and quadratic mean, and
the second one holds because ge,t ≤ 1. Therefore,
W t−1
(gIt,t + |E|β) +
η2K(1 + β)
g′e,t .
Summing for t = 1, . . . , n, we obtain
Ĝn + n|E|β
η2K(1 + β)
G′e,n
Ĝn + n|E|β
η2K(1 + β)
|C|max
G′i,n (8)
where the second inequality follows since
e∈E G
e,n ≤
i∈C G
i,n. Combining the upper bound
with the lower bound (5), we obtain
Ĝn ≥ (1− γ − ηK(1 + β)|C|)max
G′i,n −
lnN − n|E|β. (9)
Now using Lemma 1 and applying the union bound, for any δ ∈ (0, 1) we have that, with probability
at least 1− δ,
Ĝn ≥ (1− γ − ηK(1 + β)|C|)
Gi,n −
− 1− γ
lnN − n|E|β ,
where we used 1− γ − ηK(1 + β)|C| ≥ 0 which follows from the assumptions of the theorem.
Since Ĝn = Kn− L̂n and Gi,n = Kn− Li,n for all i ∈ P, we have
L̂n ≤ Kn (γ + η(1 + β)K|C|) + (1− γ − η(1 + β)K|C|)min
+ (1− γ − η(1 + β)K|C|)
lnN + n|E|β
with probability at least 1− δ. This implies
L̂n −min
Li,n ≤ Knγ + η(1 + β)nK2|C|+
lnN + n|E|β
≤ Knγ + 2ηnK2|C|+ K
+ n|E|β
with probability at least 1− δ, which is the first statement of the theorem. Setting
and γ = 2ηK|C|
results in the inequality
L̂n −min
Li,n ≤ 4ηnK2|C|+
nK|E| ln
which holds with probability at least 1 − δ if n ≥ (K/|E|) ln(|E|/δ) (to ensure β ≤ 1). Finally,
setting
4nK2|C|
yields the last statement of the theorem (n ≥ 4 lnN |C| is required to ensure γ ≤ 1/2). ✷
Next we analyze the computational complexity of the algorithm. The next result shows that
the algorithm is feasible as its complexity is linear in the size (number of edges) of the graph.
Theorem 2 The proposed algorithm can be implemented efficiently with time complexity O(n|E|)
and space complexity O(|E|).
Proof. The two complex steps of the algorithm are steps (a) and (b), both of which can be
computed, similarly to Takimoto and Warmuth [28], using dynamic programming. To perform
these steps efficiently, first we order the vertices of the graph. Since we have an acyclic directed
graph, its vertices can be labeled (in O(|E|) time) from 1 to |V | such that u = 1, v = |V |, and if
(v1, v2) ∈ E, then v1 < v2. For any pair of vertices u1 < v1 let Pu1,v1 denote the set of paths from
u1 to v1, and for any vertex s ∈ V , let
Ht(s) =
i∈Ps,v
Ĥt(s) =
i∈Pu,s
we,t .
Given the edge weights {we,t}, Ht(s) can be computed recursively for s = |V | − 1, . . . , 1, and Ĥt(s)
can be computed recursively for s = 2, . . . , |V | in O(|E|) time (letting Ht(v) = Ĥt(u) = 1 by
definition). In step (a), first one has to decide with probability γ whether It is generated according
to the graph weights, or it is chosen uniformly from C. If It is to be drawn according to the graph
weights, it can be shown that its vertices can be chosen one by one such that if the first k vertices of
It are v0 = u, v1, . . . , vk−1, then the next vertex of It can be chosen to be any vk > vk−1, satisfying
(vk−1, vk) ∈ E, with probability w(vk−1,vk),t−1Ht−1(vk)/Ht−1(vk−1). The other computationally
demanding step, namely step (b), can be performed easily by noting that for any edge (v1, v2),
q(v1,v2),t = (1− γ)
Ĥt−1(v1)w(v1,v2),t−1Ht−1(v2)
Ht−1(u)
|{i ∈ C : (v1, v2) ∈ i}|
as desired. ✷
5 A combination of the label efficient and bandit settings
In this section we investigate a combination of the multi-armed bandit and the label efficient
problems. This means that the decision maker only has access to the loss of the chosen path upon
request and the total number of requests must be bounded by a constant m. This combination is
motivated by some applications, in which feedback information is costly to obtain.
In the general label efficient decision problem, after taking an action, the decision maker has the
option to query the losses of all possible actions. For this problem, Cesa-Bianchi et al. [9] proved an
upper bound on the normalized regret of order O(K
ln(4N/δ)/(m)) which holds with probability
at least 1− δ.
Our model of the label-efficient bandit problem for shortest paths is motivated by an application
to a particular packet switched network model. This model, called the cognitive packet network,
was introduced by Gelenbe et al. [11, 12]. In these networks a particular type of packets, called
smart packets, are used to explore the network (e.g., the delay of the chosen path). These packets
do not carry any useful data; they are merely used for exploring the network. The other type of
packets are the data packets, which do not collect any information about their paths. The task of
the decision maker is to send packets from the source to the destination over routes with minimum
average transmission delay (or packet loss). In this scenario, smart packets are used to query the
delay (or loss) of the chosen path. However, as these packets do not transport information, there
is a tradeoff between the number of queries and the usage of the network. If data packets are on
the average α times larger than smart packets (note that typically α ≫ 1) and ǫ is the proportion
of time instances when smart packets are used to explore the network, then ǫ/(ǫ+ α(1− ǫ)) is the
proportion of the bandwidth sacrificed for well informed routing decisions.
We study a combined algorithm which, at each time slot t, queries the loss of the chosen path
with probability ǫ (as in the solution of the label efficient problem proposed in [9]), and, similarly
to the multi-armed bandit case, computes biased estimates g′
i,t of the true gains gi,t. Just as in the
previous section, it is assumed that each path of the graph is of the same length K.
The algorithm differs from our bandit algorithm of the previous section only in step (c), which
is modified in the spirit of [9]. The modified step is given below:
MODIFIED STEP FOR THE LABEL EFFICIENT BANDIT ALGORITHM FOR
SHORTEST PATHS
(c’) Draw a Bernoulli random variable St with P(St = 1) = ǫ, and compute the
estimated gains
g′e,t =
ge,t+β
ǫqe,t
St if e ∈ It
ǫqe,t
St if e /∈ It .
The performance of the algorithm is analyzed in the next theorem, which can be viewed as a
combination of Theorem 1 in the preceding section and Theorem 2 of [9].
Theorem 3 For any δ ∈ (0, 1), ǫ ∈ (0, 1], parameters η =
ǫ lnN
4nK2|C|
, γ =
2ηK|C|
≤ 1/2, and
n|E|ǫ
≤ 1, and for all
K2 ln2(2|E|/δ)
|E| lnN
|E| ln(2|E|/δ)
, 4|C| lnN
the performance of the algorithm defined above can be bounded, with probability at least 1− δ, as
L̂n −min
K|C| lnN + 5
|E| ln
8K ln
|E| ln 2N
If ǫ is chosen as (m −
2m ln(1/δ))/n then, with probability at least 1 − δ, the total number
of queries is bounded by m (see [8, Lemma 6.1]) and the performance bound above is of the order
|E| ln(N/δ)/m.
Similarly to Theorem 1, we need a lemma which reveals the connection between the true and the
estimated cumulative losses. However, here we need a more careful analysis because the “shifting
term”
ǫqe,t
St, is a random variable.
Lemma 2 For any 0 < δ < 1, 0 < ǫ ≤ 1, for any
n ≥ 1
K2 ln2(2|E|/δ)
|E| lnN
K ln(2|E|/δ)
parameters
2ηK|C|
≤ γ, η =
ǫ lnN
4nK2|C|
and β =
n|E|ǫ
≤ 1 ,
and e ∈ E, we have
Ge,n > G
e,n +
Proof. Fix e ∈ E. Using (1) with u = 4
and c =
, it suffices to prove for all n that
ec(Ge,n−G
≤ 1 .
Similarly to Lemma 1 we introduce Zt = e
c(Ge,t−G
e,t) and we show that Z1, . . . , Zn is a super-
martingale, that is Et[Zt] ≤ Zt−1 for t ≥ 2 where Et denotes E[·|(I1, S1), . . . , (I t−1, St−1)]. Taking
conditional expectations, we obtain
Et[Zt] = Zt−1Et
ge,t−
1{e∈It}
Stge,t+Stβ
qe,tǫ
≤ Zt−1Et
1 + c
ge,t −
1{e∈It}Stge,t + Stβ
qe,tǫ
ge,t −
1{e∈It}Stge,t + Stβ
qe,tǫ
. (10)
Since
ge,t −
1{e∈It}Stge,t + Stβ
qe,tǫ
= − β
ge,t −
1{e∈It}Stge,t
qe,tǫ
1{e∈It}Stge,t
qe,tǫ
qe,tǫ
we get from (10) that
Et[Zt]
≤ Zt−1Et
1− cβ
qe,tǫ
21{e∈It}Stge,tβ
q2e,tǫ
2ge,tStβ
qe,tǫ
q2e,tǫ
≤ Zt−1
qe,tǫ
. (11)
Since c = βǫ/4 we have
− β + c
qe,tǫ
= −3β
qe,tǫ
4qe,t
4qe,t
β3|C|
≤ 0, (13)
where (12) follows from qe,t ≥ γ|C| and (13) holds by
β2|C|
≤ 1 ,
and the last inequality is ensured by n ≥ K
2 ln2(2|E|/δ)
ǫ|E| lnN
, the assumption of the lemma.
Combining (11) and (13) we get that Et[Zt] ≤ Zt−1. Taking expectations on both sides of the
inequality, we get E[Zt] ≤ E[Zt−1] and since E[Z1] ≤ 1, we obtain E[Zn] ≤ 1 as desired. ✷
Proof of Theorem 3. The proof of the theorem is a generalization of that of Theorem 1, and
follows the same lines with some extra technicalities to handle the effects of the modified step (c’).
Therefore, in the following we emphasize only the differences. First note that (5) and (7) also hold
in this case. Bounding the sums in (7), one obtains
pi,tg
i,t =
(gIt,t + |E|β)
and ∑
pi,tg
i,t ≤
K(1 + β)
g′e,t .
Plugging these bounds into (7) and summing for t = 1, . . . , n, we obtain
(gIt,t + |E|β ) +
η2K(1 + β)
(1− γ)ǫ
|C|max
G′i,n .
Combining the upper bound with the lower bound (5), we obtain
(gIt,t + |E|β ) ≥
1−γ− ηK(1 + β)|C|
G′i,n−
. (14)
To relate the left-hand side of the above inequality to the real gain
t=1 gIt,t, notice that
(gIt,t + |E|β) − (gIt,t + |E|β)
is a martingale difference sequence with respect to (I1, S1), (I2, S2), . . .. Now for all t = 1, . . . , n,
we have the bound
X2t |(I1, S1), . . . , (It−1, St−1)
(gIt,t + |E|β)2
∣∣∣∣(I1, S1), . . . , (I t−1, St−1)
≤ (K + |E|β)
= σ2, (15)
where (15) holds by n ≥ |E| ln(2|E|/δ)
(to ensure β|E| ≤ K). We know that
for all t. Now apply Bernstein’s inequality for martingale differences (see Lemma 9 in the Appendix)
to obtain
Xt > u
, (16)
where
From (16) we get
(gIt,t + |E|β) ≥ Ĝn + βn|E|+ u
. (17)
Now Lemma 2, the union bound, and (17) combined with (14) yield, with probability at least
1− δ,
Ĝn ≥
1− γ −
ηK(1 + β)|C|
Gi,n −
− βn|E| − u
since the coefficient of G′i,n is greater than zero by the assumptions of the theorem.
Since Ĝn = Kn− L̂n and Gi,n = Kn− Li,n, we have
L̂n ≤
1− γ − K(1 + β)η|C|
Li,n +Kn
K(1 + β)η|C|
1− γ −
K(1 + β)η|C|
+ βn|E|+
≤ min
Li,n +Kn
K(1 + β)η|C|
+ 5βn|E|+
+ u ,
where we used the fact that K
= βn|E|.
Substituting the value of β, η and γ, we have
L̂n −min
Li,n ≤Kn
2Kη|C|
2Kη|C|
+ 5βn|E|+ u
n|C| lnN
n|E|K ln(2|E|/δ)
K|C| lnN + 5
|E| ln(2|E|/δ) +
8K ln (2/δ)
ln (2/δ)
as desired. ✷
6 A bandit algorithm for tracking the shortest path
Our goal in this section is to extend the bandit algorithm so that it is able to compete with time-
varying paths under small computational complexity. This is a variant of the problem known
as tracking the best expert ; see, for example, Herbster and Warmuth [19], Vovk [30], Auer and
Warmuth [2], Bousquet and Warmuth [6], Herbster and Warmuth [20].
To describe the loss the decision maker is compared to, consider the following “m-partition”
prediction scheme: the sequence of paths is partitioned into m+1 contiguous segments, and on each
segment the scheme assigns exactly one of the N paths. Formally, an m-partition Part(n,m, t, i) of
the n paths is given by an m-tuple t = (t1, . . . , tm) such that t0 = 1 < t1 < · · · < tm < n+1 = tm+1,
and an (m+ 1)-vector i = (i0, . . . , im) where ij ∈ P. At each time instant t, tj ≤ t < tj+1, path ij
is used to predict the best path. The cumulative loss of a partition Part(n,m, t, i) is
L(Part(n,m, t, i)) =
tj+1−1∑
ℓij ,t =
tj+1−1∑
ℓe,t.
The goal of the decision maker is to perform as well as the best time-varying path (partition),
that is, to keep the normalized regret
L̂n −min
L(Part(n,m, t, i))
as small as possible (with high probability) for all possible outcome sequences.
In the “classical” tracking problem there is a relatively small number of “base” experts and the
goal of the decision maker is to predict as well as the best “compound” expert (i.e., time-varying
expert). However in our case, base experts correspond to all paths of the graph between source and
destination whose number is typically exponentially large in the number of edges, and therefore we
cannot directly apply the computationally efficient methods for tracking the best expert. György,
Linder, and Lugosi [15] develop efficient algorithms for tracking the best expert for certain large
and structured classes of base experts, including the shortest path problem. The purpose of the
following algorithm is to extend the methods of [15] to the bandit setting when the forecaster only
observes the losses of the edges on the chosen path.
A BANDIT ALGORITHM FOR TRACKING SHORTEST PATHS
Parameters: real numbers β > 0, 0 < η, γ < 1, 0 ≤ α ≤ 1.
Initialization: Set we,0 = 1 for each e ∈ E, wi,0 = 1 for each i ∈ P, and W 0 = N .
For each round t = 1, 2, . . .
(a) Choose a path It according to the distribution pt defined by
pi,t =
(1− γ)wi,t−1
W t−1
if i ∈ C;
(1− γ)wi,t−1
W t−1
if i 6∈ C.
(b) Compute the probability of choosing each edge e as
qe,t =
i:e∈i
pi,t = (1− γ)
i:e∈iwi,t−1
W t−1
|{i ∈ C : e ∈ i}|
(c) Calculate the estimated gains
g′e,t =
ge,t+β
if e ∈ It;
otherwise.
(d) Compute the updated weights
vi,t = wi,t−1e
wi,t = (1− α)vi,t +
where g′
i,t =
e∈i g
e,t and W t is the sum of the total weights of the paths, that
W t =
vi,t =
wi,t.
The following performance bounds shows that the normalized regret with respect to the best
time-varying path which is allowed to switch pathsm times is roughly of the order ofK
(m/n)|C| lnN .
Theorem 4 For any δ ∈ (0, 1) and parameters 0 ≤ γ < 1/2, α, β ∈ [0, 1], and η > 0 satisfying
2ηK|C| ≤ γ, the performance of the algorithm defined above can be bounded, with probability at
least 1− δ, as
L̂n −min
L(Part(n,m, t, i))
≤ Kn (γ + η(1 + β)K|C|) +
K(m+ 1)
|E|(m+ 1)
+ βn|E|+ 1
αm(1− α)n−m−1
In particular, choosing
K(m+ 1)
|E|(m+ 1)
, γ = 2ηK|C|, α =
(m+ 1) lnN +m ln
e(n−1)
4nK2|C|
we have, for all n ≥ max
K(m+1)
|E|(m+1)
, 4|C|D
L̂n −min
L(Part(n,m, t, i)) ≤ 2
4K|C|D +
|E|(m + 1) ln
|E|(m + 1)
where
D = (m+ 1) lnN +m
1 + ln
The proof of the theorem is a combination of that of our Theorem 1 and Theorem 1 of [15]. We
will need the following three lemmas.
Lemma 3 For any 1 ≤ t ≤ t′ ≤ n and any i ∈ P,
vi,t′
wi,t−1
≥ eηG
i,[t,t′](1− α)t′−t
where G′
i,[t,t′]
τ=t g
i,τ .
Proof. The proof is a straightforward modification of the one in Herbster and Warmuth [19]. From
the definitions of vi,t and wi,t (see step (d) of the algorithm) it is clear that for any τ ≥ 1,
wi,τ = (1− α)vi,τ +
W τ ≥ (1− α)eηg
i,τwi,τ−1 .
Applying this equation iteratively for τ = t, t+1, . . . , t′ − 1, and the definition of vi,t (step (d)) for
τ = t′, we obtain
vi,t′ = wi,t′−1e
i,t′ ≥ eηg
t′−1∏
(1− α)eηg
wi,t−1
i,[t,t′](1− α)t′−twi,t−1
which implies the statement of the lemma. ✷
Lemma 4 For any t ≥ 1 and i, j ∈ P, we have
Proof. By the definition of wi,t we have
wi,t = (1− α)vi,t +
W t ≥
W t ≥
vj,t .
This completes the proof of the lemma. ✷
The next lemma is a simple corollary of Lemma 1.
Lemma 5 For any δ ∈ (0, 1), 0 ≤ β ≤ 1, t ≥ 1 and e ∈ E we have
Ge,t > G
e,t +
|E|(m+ 1)
|E|(m+ 1)
Proof of Theorem 4. The theorem is proved the same way as Theorem 1 until (8), that is,
Ĝn + n|E|β
η2K(1 + β)
|C|max
G′i,n . (18)
Let Part(n,m, t, i) be an arbitrary partition. Then the lower bound is obtained as
vim,n
(recall that im denotes the path used in the last segment of the partition). Now vim,n can be
rewritten in the form of the following telescoping product
vim,n = wi0,t0−1
vi0,t1−1
wi0,t0−1
wij ,tj−1
vij−1,tj−1
vij ,tj+1−1
wij ,tj−1
Therefore, applying Lemmas 3 and 4, we have
vim,n ≥ wi0,t0−1
)m m∏
(1− α)tj+1−1−tjeηG
ij ,[tj ,tj+1−1]
′(Part(n,m,t,i))(1− α)n−m−1.
Combining the lower bound with the upper bound (18), we have
αm(1− α)n−m−1
ηG′(Part(n,m, t, i))
Ĝn + n|E|β
η2K(1+β)
|C|maxi∈P G′i,n ,
where we used the fact that Part(n,m, t, i) is an arbitrary partition. After rearranging and using
maxi∈P G
i,n ≤ maxt,iG′(Part(n,m, t, i)) we get
Ĝn ≥ (1− γ − ηK(1 + β)|C|) max
G′(Part(n,m, t, i))
−n|E|β − 1− γ
αm(1− α)n−m−1
Now since 1 − γ − ηK(1 + β)|C| ≥ 0, by the assumptions of the theorem and from Lemma 5 with
an application of the union bound we obtain that, with probability at least 1− δ,
Ĝn ≥ (1− γ − ηK(1 + β)|C|)
G(Part(n,m, t, i))−
K(m+ 1)
|E|(m+ 1)
− n|E|β − 1− γ
αm(1− α)n−m−1
Since Ĝn = Kn− L̂n and G(Part(n,m, t, i)) = Kn− L(Part(n,m, t, i)), we have
L̂n ≤ (1− γ − ηK(1 + β)|C|) min
L(Part(n,m, t, i)) +Kn (γ + η(1 + β)K|C|)
+ (1− γ − η(1 + β)K|C|) K(m+ 1)
|E|(m+ 1)
+ n|E|β
αm(1− α)n−m−1
This implies that, with probability at least 1− δ,
L̂n −min
L(Part(n,m, t, i))
≤ Kn (γ + η(1 + β)K|C|) +
K(m+ 1)
|E|(m+ 1)
+ n|E|β + 1
αm(1− α)n−m−1
. (20)
To prove the second statement, let H(p) = −p ln p − (1 − p) ln(1 − p) and D(p ‖ q) = p ln p
(1− p) ln 1−p
. Optimizing the value of α in the last term of (20) gives
αm(1− α)n−m−1
(m+ 1) ln (N) +m ln
+ (n−m− 1) ln
(m+ 1) ln (N) + (n − 1)(Db(α∗ ‖ α) +Hb(α∗))
where α∗ = m
. For α = α∗ we obtain
αm(1− α)n−m−1
((m+ 1) ln (N) + (n− 1)(Hb(α∗)))
((m+ 1) ln (N) +m ln((n − 1)/m)
+(n−m− 1) ln(1 +m/(n−m− 1)))
((m+ 1) ln (N) +m ln((n − 1)/m) +m)
(m+ 1) ln (N) +m ln
e(n− 1)
where the inequality follows since ln(1 + x) ≤ x for x > 0. Therefore
L̂n −min
L(Part(n,m, t, i))
≤ Kn (γ + η(1 + β)K|C|) +
K(m+ 1)
|E|(m+ 1)
+ n|E|β +
which is the first statement of the theorem. Setting
K(m+ 1)
|E|(m+ 1)
, γ = 2ηK|C|, and η =
4nK2|C|
results in the second statement of the theorem, that is,
L̂n −min
L(Part(n,m, t, i))
4K|C|D +
|E|(m+ 1) ln
|E|(m+ 1)
Similarly to [15], the proposed algorithm has an alternative version, which is efficiently com-
putable:
AN ALTERNATIVE BANDIT ALGORITHM FOR TRACKING SHORTEST
PATHS
For t = 1, choose I1 uniformly from the set P. For t ≥ 2,
(a) Draw a Bernoulli random variable Γt with P(Γt = 1) = γ.
(b) If Γt = 1, then choose It uniformly from C.
(c) If Γt = 0,
(c1) choose τt randomly according to the distribution
P{τt = t′} =
(1−α)t−1Z1,t−1
for t′ = 1
α(1−α)t−t
Wt′Zt′,t−1
for t′ = 2, . . . , t
where Zt′,t−1 =
i∈P e
i,[t′,t−1] for t′ = 1, . . . , t− 1, and Zt,t−1 = N ;
(c2) given τt = t
′, choose It randomly according to the probabilities
P{It = i|τt = t′} =
i,[t′,t−1]
Zt′,t−1
for t′ = 1, . . . , t− 1
for t′ = t.
In a way completely analogous to [15], in this alternative formulation of the algorithm one can
compute the probabilities P{It = i|τt = t′} and the normalization factors Zt′,t−1 efficiently. Using
the fact that the baseline bandit algorithm for shortest paths has an O(n|E|) time complexity by
Theorem 2, it follows from Theorem 3 of [15] that the time complexity of the alternative bandit
algorithm for tracking the shortest path is O(n2|E|).
7 An algorithm for the restricted multi-armed bandit problem
In this section we consider the situation where the decision maker receives information only about
the performance of the whole chosen path, but the individual edge losses are not available. That
is, the forecaster has access to the sum ℓIt,t of losses over the chosen path It but not to the losses
{ℓe,t}e∈It of the edges belonging to It.
This is the problem formulation considered by McMahan and Blum [25] and Awerbuch and
Kleinberg [4]. McMahan and Blum provided a relatively simple algorithm whose regret is at
most of the order of n−1/4, while Awerbuch and Kleinberg gave a more complex algorithm to
achieve O(n−1/3) regret. In this section we combine the strengths of these papers, and propose
a simple algorithm with regret at most of the order of n−1/3. Moreover, our bound holds with
high probability, while the above-mentioned papers prove bounds for the expected regret only. The
proposed algorithm uses ideas very similar to those of McMahan and Blum [25]. The algorithm
alternates between choosing a path from a “basis” B to obtain unbiased estimates of the loss
(exploration step), and choosing a path according to exponential weighting based on these estimates.
A simple way to describe a path i ∈ P is a binary row vector with |E| components which are
indexed by the edges of the graph such that, for each e ∈ E, the eth entry of the vector is 1 if e ∈ i
and 0 otherwise. With a slight abuse of notation we will also denote by i the binary row vector
representing path i. In the previous sections, where the loss of each edge along the chosen path
is available to the decision maker, the complexity stemming from the large number of paths was
reduced by representing all information in terms of the edges, as the set of edges spans the set of
paths. That is, the vector corresponding to a given path can be expressed as the linear combination
of the unit vectors associated with the edges (the eth component of the unit vector representing
edge e is 1, while the other components are 0). While the losses corresponding to such a spanning
set are not observable in the restricted setting of this section, one can choose a subset of P that
forms a basis, that is, a collection of b paths which are linearly independent and each path in P
can be expressed as a linear combination of the paths in the basis. We denote by B the b × |E|
matrix whose rows b1, . . . , bb represent the paths in the basis. Note that b is equal to the maximum
number of linearly independent vectors in {i : i ∈ P}, so b ≤ |E|.
Let ℓ
t denote the (column) vector of the edge losses {ℓe,t}e∈E at time t, and let ℓ
(ℓb1,t, . . . , ℓbb,t)
T be a b-dimensional column vector whose components are the losses of the paths
in the basis B at time t. If α
(i,B)
, . . . , α
(i,B)
are the coefficients in the linear combination of the
basis paths expressing path i ∈ P, that is, i =
j=1 α
(i,B)
j, then the loss of path i ∈ P at time t
is given by
ℓi,t = 〈i, ℓ
t 〉 =
(i,B)
〈bj, ℓ(E)t 〉 =
(i,B)
bj ,t (21)
where 〈·, ·〉 denotes the standard inner product in R|E|. In the algorithm we obtain estimates ℓ̃
bj ,t
of the losses of the basis paths and use (21) to estimate the loss of any i ∈ P as
ℓ̃i,t =
(i,B)
bj ,t . (22)
It is algorithmically advantageous to calculate the estimated path losses ℓ̃i,t from an intermediate
estimate of the individual edge losses. LetB+ denote the the Moore-Penrose inverse ofB defined by
B+ = BT (BBT )−1, where BT denotes the transpose of B and BBT is invertible since the rows of
B are linearly independent. (Note thatB+ = B−1 if b = |E|). Then letting ℓ̃(B)t = (ℓ̃b1,t, . . . , ℓ̃bb,t)
t = B
it is easy to see that ℓ̃i,t in (22) can be obtained as ℓ̃i,t = 〈i, ℓ̃
t 〉, or equivalently
ℓ̃i,t =
ℓ̃e,t.
This form of the path losses allows for an efficient implementation of exponential weighting via
dynamic programming [28].
To analyze the algorithm we need an upper bound on the magnitude of the coefficients α
(i,B)
For this, we invoke the definition of a barycentric spanner from [4]: the basis B is called a C-
barycentric spanner if |α(i,B)
| ≤ C for all i ∈ P and j = 1, . . . , b. Awerbuch and Kleinberg [4] show
that a 1-barycentric spanner exists if B is a square matrix (i.e., b = |E|) and give a low-complexity
algorithm which finds a C-barycentric spanner for C > 1. We use their technique to show that a
1-barycentric spanner also exists in case of a non-square B, when the basis is chosen to maximize
the absolute value of the determinant of BBT . As before, b denotes the maximum number of
linearly independent vectors (paths) in P.
Lemma 6 For a directed acyclic graph, the set of paths P between two dedicated nodes has a 1-
barycentric spanner. Moreover, let B be a b×|E| matrix with rows from P such that det[BBT ] 6= 0.
If B−j,i is the matrix obtained from B by replacing its jth row by i ∈ P and
∣∣det
B−j,iB
]∣∣ ≤ C2
∣∣det
]∣∣ (23)
for all j = 1, . . . , b and i ∈ P, then B is a C-barycentric spanner.
Proof. Let B be a basis of P with rows b1, . . . , bb ∈ P that maximizes |det[BBT ]|. Then, for
any path i ∈ P, we have i =
j=1 α
(i,B)
j for some coefficients {α(i,B)
}. Now for the matrix
B−1,i = [i
T , (b2)T , . . . , (bb)T ]T we have
∣∣det
B−1,iB
∣∣∣det
B−1,ii
T ,B−1,i(b
2)T ,B−1,i(b
3)T , . . . ,B−1,i(b
∣∣∣∣∣∣∣
(i,B)
B−1,ib
,B−1,i(b
2)T ,B−1,i(b
3)T , . . . ,B−1,i(b
∣∣∣∣∣∣∣
∣∣∣∣∣∣
(i,B)
B−1,i(b
j)T ,B−1,i(b
2)T ,B−1,i(b
3)T , . . . ,B−1,i(b
∣∣∣∣∣∣
= |α(i,B)
∣∣det
B−1,iB
(i,B)
)2 ∣∣det
where last equality follows by the same argument the penultimate equality was obtained. Repeating
the same argument for B−j,i, j = 2, . . . , b we obtain
∣∣det
B−j,iB
]∣∣ =
(i,B)
)2 ∣∣det
]∣∣ . (24)
Thus the maximal property of |det[BBT ]| implies |α(i,B)
| ≤ 1 for all j = 1, . . . , b. The second
statement follows trivially from (23) and (24). ✷
Awerbuch and Kleinberg [4] also present an iterative algorithm to find a C-barycentric spanner
if B is a square matrix. Starting from the identity matrix, their algorithm replaces a row of the
matrix in each step by maximizing the determinant with respect to the given row. This is done by
calling an oracle function, and it is shown that the oracle is called O(b logC b) times. In case B is
not a square matrix, the algorithm carries over if we have access to an alternative oracle that can
maximize |det[BBT ]|: Starting from an arbitrary basis B we can iteratively replace one row in
each step, using the oracle, to maximize the determinant |det[BBT ]| until (23) is satisfied for all j
and i. By Lemma 6, this results in a C-barycentric spanner. Similarly to [4], it can be shown that
the oracle is called O(b logC b) times for C > 1.
For simplicity (to avoid carrying the constant C), assume that we have a 2-barycentric spanner
B. Based on the ideas of label efficient prediction, the next algorithm gives a simple solution to
the restricted shortest path problem. The algorithm is very similar to that of the algorithm in the
label efficient case, but here we cannot estimate the edge losses directly. Therefore, we query the
loss of a (random) basis vector from time to time, and create unbiased estimates ℓ̃
bj ,t of the losses
of basis paths ℓ
bj ,t, which are then transformed into edge-loss estimates.
A BANDIT ALGORITHM FOR THE RESTRICTED SHORTEST PATH
PROBLEM
Parameters: 0 < ǫ, η ≤ 1.
Initialization: Set we,0 = 1 for each e ∈ E, wi,0 = 1 for each i ∈ P, W 0 = N . Fix a
basis B, which is a 2-barycentric spanner. For each round t = 1, 2, . . .
(a) Draw a Bernoulli random variable St such that P(St = 1) = ǫ;
(b) If St = 1, then choose the path It uniformly from the basis B. If St = 0, then
choose It according to the distribution {pi,t}, defined by
pi,t =
wi,t−1
W t−1
(c) Calculate the estimated loss of all edges according to
t = B
where ℓ̃
t = {ℓ̃
e,t }e∈E , and ℓ̃
t = (ℓ̃
, . . . , ℓ̃
) is the vector of the estimated
losses
bj ,t =
bj ,t1{It=b
for j = 1, . . . , b.
(d) Compute the updated weights
we,t = we,t−1e
−ηℓ̃e,t ,
wi,t =
we,t = wi,t−1e
e∈i ℓ̃e,t ,
and the sum of the total weights of the paths
W t =
wi,t .
The performance of the algorithm is analyzed in the next theorem. The proof follows the argu-
ment of Cesa-Bianchi et al. [9], but we also have to deal with some additional technical difficulties.
Note that in the theorem we do not assume that all paths between u and v have equal length.
Theorem 5 Let K denote the length of the longest path in the graph. For any δ ∈ (0, 1), parameters
0 < ǫ ≤ 1
and η > 0 satisfying η ≤ ǫ2, and n ≥ 8b
ln 4bN
, the performance of the algorithm defined
above can be bounded, with probability at least 1− δ, as
L̂n −min
Li,n ≤ K
+ nǫ+
2nǫ ln 4
In particular, choosing
and η = ǫ2
we obtain
L̂n −min
Li,n ≤ 9.1K2b (Kb ln(4bN/δ))1/3 n2/3 .
The theorem is proved using the following two lemmas. The first one is an easy consequence of
Bernstein’s inequality:
Lemma 7 Under the assumptions of Theorem 5, the probability that the algorithm queries the basis
more than nǫ+
2nǫ ln 4
times is at most δ/4.
Using the estimated loss of a path i ∈ P given in (22), we can estimate the cumulative loss of
i up to time n as
L̃i,n =
ℓ̃i,t .
The next lemma demonstrates the quality of these estimates.
Lemma 8 Let 0 < δ < 1 and assume n ≥ 8b
ln 4bN
. For any i ∈ P, with probability at least
1− δ/4,
pi,tℓi,t −
pi,tℓ̃i,t ≤
Furthermore, with probability at least 1− δ/(4N),
L̃i,n − Li,n ≤
Proof. We may write
pi,tℓi,t −
pi,tℓ̃i,t =
(i,B)
bj ,t − ℓ̃bj ,t
pi,tα
(i,B)
bj ,t − ℓ̃bj ,t
bj ,t . (25)
Note that for any bj, X
bj ,t, t = 1, 2, . . . is a martingale difference sequence with respect to (It, St),
t = 1, 2, . . . as Etℓ̃b,t = ℓb,t. Also,
bj ,t
pi,tα
(i,B)
bj ,t
(i,B)
)2 K2b
bj ,t| ≤
∣∣∣∣∣
pi,tα
(i,B)
∣∣∣∣∣
∣∣∣ℓbj ,t − ℓ̃bj ,t
∣∣∣ ≤
∣∣∣α(i,B)
where the last inequalities in both cases follow from the fact that B is a 2-barycentric spanner.
Then, using Bernstein’s inequality for martingale differences (Lemma 9), we have, for any fixed bj ,
bj ,t ≥
where we used (26), (27) and the assumption of the lemma on n. The proof of the first statement
is finished with an application of the union bound and its combination with (25).
For the second statement we use a similar argument, that is,
(ℓ̃i,t − ℓi,t) =
(i,B)
bj ,t − ℓbj ,t) ≤
∣∣∣α(i,B)
∣∣∣∣∣
bj ,t − ℓbj ,t)
∣∣∣∣∣
∣∣∣∣∣
bj ,t − ℓbj ,t)
∣∣∣∣∣ . (29)
Now applying Lemma 9 for a fixed bj we get
bj ,t − ℓbj ,t) ≥
because of Et[(ℓ̃bj ,t − ℓbj ,t)2] ≤
and −K ≤ ℓ̃
bj ,t − ℓbj ,t ≤ K
. The proof is completed by
applying the union bound to (30) and combining the result with (29). ✷
Proof of Theorem 5. Similarly to earlier proofs, we follow the evolution of the term ln Wn
the same way as we obtained (5) and (7), we have
≥ −ηmin
L̃i,n − lnN
pi,tℓ̃i,t +
pi,tℓ̃
Combining these bounds, we obtain
L̃i,n −
pi,tℓ̃i,t +
pi,tℓ̃
−1 + ηKb
pi,tℓ̃i,t ,
because 0 ≤ ℓ̃i,t ≤ 2Kbǫ . Applying the results of Lemma 8 and the union bound, we have, with
probability 1− δ/2,
Li,n −
−1 + ηKb
)( n∑
pi,tℓi,t −
pi,tℓi,t +
. (31)
Introduce the sets
= {t : 1 ≤ t ≤ n and St = 0} and T n
= {t : 1 ≤ t ≤ n and St = 1}
of “exploitation” and “exploration” steps, respectively. Then, by the Hoeffding-Azuma inequality
[21] we obtain that, with probability at least 1− δ/4,
pi,tℓi,t ≥
ℓIt,t −
|Tn|K2
Note that for the exploration steps t ∈ T n, as the algorithm plays according to a uniform distribu-
tion instead of pi,t, we can only use the trivial lower bound zero on the losses, that is,
t∈T n
pi,tℓi,t ≥
t∈T n
ℓIt,t −K|T n| .
The last two inequalities imply
pi,tℓi,t ≥ L̂n −
|Tn|K2
−K|T n| . (32)
Then, by (31), (32) and Lemma 7 we obtain, with probability at least 1− δ,
L̂n −min
+ nǫ+
2nǫ ln 4
where we used L̂n ≤ Kn and |Tn| ≤ n. Substituting the values of ǫ and η gives
L̂n −min
Li,n ≤ K2bnǫ+
Knǫ+Knǫ+
Knǫ+ nǫ
≤ 9.1K2bnǫ
where we used
2nǫ ln 4
ln 4N
= nǫ, and lnN
≤ nǫ (from the
assumptions of the theorem). ✷
8 Simulation results
To further investigate our new algorithms, we have conducted some simple simulations. As the main
motivation of this work is to improve earlier algorithms in case the number of paths is exponentially
large in the number of edges, we tested the algorithms on the small graph shown in Figure 1 (b),
which has one of the simplest structures with exponentially many (namely 2|E|/2) paths.
The losses on the edges were generated by a sequence of independent and uniform random
variables, with values from [0, 1] on the upper edges, and from [0.32, 1] on the lower edges, resulting
in a (long-term) optimal path consisting of the upper edges. We ran the tests for n = 10000 steps,
with confidence value δ = 0.001. To establish baseline performance, we also tested the EXP3
algorithm of Auer et al. [1] (note that this algorithm does not need edge losses, only the loss of the
chosen path). For the version of our bandit algorithm that is informed of the individual edge losses
(edge-bandit), we used the simple 2-element cover set of the paths consisting of the upper and
lower edges, respectively (other 2-element cover sets give similar performance). For our restricted
shortest path algorithm (path-bandit) the basis {uuuuu, uuuul, uuull, uulll, ullll} was used, where
u (resp. l) in the kth position denotes the upper (resp. lower) edge connecting vk−1 and vk. In
this example the performance of the algorithm appeared to be independent of the actual choice of
the basis; however, in general we do not expect this behavior. Two versions of the algorithm of
Awerbuch and Kleinberg [4] were also simulated. With its original parameter setting (AwKl), the
algorithm did not perform well. However, after optimizing its parameters off-line (AwKl tuned),
substantially better performance was achieved. The normalized regret of the above algorithms,
averaged over 30 runs, as well as the regret of the fixed paths in the graph are shown in Figure 2.
Although all algorithms showed better performance than the bound for the edge-bandit algo-
rithm, the latter showed the expected superior performance in the simulations. Furthermore, our
algorithm for the restricted shortest path problem outperformed Awerbuch and Kleinberg’s (AwKl)
algorithm, while being inferior to its off-line tuned version (AwKl tuned). It must be noted that
similar parameter optimization did not improve the performance of our path-bandit algorithm,
which showed robust behavior with respect to parameter tuning.
9 Conclusions
We considered different versions of the on-line shortest path problem with limited feedback. These
problems are motivated by realistic scenarios, such as routing in communication networks, where
the vertices do not have all the information about the state of the network. We have addressed the
problem in the adversarial setting where the edge losses may vary in an arbitrary way; in particular,
they may depend on previous routing decisions of the algorithm. Although this assumption may
neglect natural correlation in the loss sequence, it suits applications in mobile ad-hoc networks,
where the network topology changes dynamically in time, and also in certain secure networks that
has to be able to handle denial of service attacks.
Efficient algorithms have been provided for the multi-armed bandit setting and in a combined
label efficient multi-armed bandit setting, provided the individual edge losses along the chosen
path are revealed to the algorithms. The normalized regrets of the algorithms, compared to the
performance of the best fixed path, converge to zero at an O(1/
n) rate as the time horizon n
grows to infinity, and increases only polynomially in the number of edges (and vertices) of the
graph. Earlier methods for the multi-armed bandit problem either do not have the right O(1/
convergence rate, or their regret increase exponentially in the number of edges for typical graphs.
 0  2000  4000  6000  8000  10000
Number of packets
edge-bandit
path-bandit
AwKl tuned
bound for edge-bandit
Figure 2: Normalized regret of several algorithms for the shortest path problem. The gray dotted
lines show the normalized regret of fixed paths in the graph.
The algorithm has also been extended so that it can compete with time varying paths, that is, to
handle situations when the best path can change from time to time (for consistency, the number
of changes must be sublinear in n).
In the restricted version of the shortest path problem, where only the losses of the whole paths
are revealed, an algorithm with a worse O(n−1/3) normalized regret was provided. This algorithm
has comparable performance to that of the best earlier algorithm for this problem [4], however,
our algorithm is significantly simpler. Simulation results are also given to assess the practical
performance and compare it to the theoretical bounds as well as other competing algorithms.
It should be noted that the results are not entirely satisfactory in the restricted version of the
problem, as it remains an open question whether the O(1/
n) regret can be achieved without the
exponential dependence on the size of the graph. Although we expect that this is the case, we have
not been able to construct an algorithm with such a proven performance bound.
10 Appendix
Lemma 9 (Bernstein’s inequality for martingale differences [10].) Let X1, . . . ,Xn be a martingale
difference sequence such that Xt ∈ [a, b] with probability one (t = 1, . . . , n). Assume that, for all t,
X2t |Xt−1, . . . ,X1
≤ σ2 a.s.
Then, for all ǫ > 0,
Xt > ǫ
2nσ2+2ǫ(b−a)/3
and therefore
2nσ2 ln δ−1 + 2 ln δ−1(b− a)/3
References
[1] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. Schapire. The non-stochastic multi-armed bandit
problem. SIAM Journal on Computing, 32(1):48–77, 2002.
[2] P. Auer and M. K. Warmuth, “Tracking the best disjunction,” Machine Learning, vol. 32,
no. 2, pp. 127–150, 1998.
[3] B. Awerbuch, D. Holmer, H. Rubens, and R. Kleinberg. Provably competitive adaptive routing.
In Proceedings of IEEE INFOCOM 2005, volume 1, pages 631–641, March 2005.
[4] B. Awerbuch and R. D. Kleinberg. Adaptive routing with end-to-end feedback: distributed
learning and geometric approaches. In Proceedings of the 36th Annual ACM Symposium on the
Theory of Computing, STOC 2004, pages 45–53, Chicago, IL, USA, Jun. 2004. ACM Press.
[5] D. Blackwell. An analog of the minimax theorem for vector payoffs. Pacific Journal of Math-
ematics, 6:1–8, 1956.
[6] O. Bousquet and M. K. Warmuth. Tracking a small set of experts by mixing past posteriors.
Journal of Machine Learning Research, 3:363–396, Nov. 2002.
[7] N. Cesa-Bianchi, Y. Freund, D. P. Helmbold, D. Haussler, R. Schapire, and M. K. Warmuth.
How to use expert advice. Journal of the ACM, 44(3):427–485, 1997.
[8] N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University
Press, Cambridge, 2006.
[9] N. Cesa-Bianchi, G. Lugosi, and G. Stoltz. Minimizing regret with label efficient prediction.
IEEE Trans. Inform. Theory, IT-51:2152–2162, June 2005.
[10] D. A. Freedman. On tail probabilities for martingales. Annals of Probability, 3:100–118, Feb.
1975.
[11] E. Gelenbe, M. Gellman, R. Lent, P. Liu, and P. Su. Autonomous smart routing for network
QoS. In Proceedings of First International Conference on Autonomic Computing, pages 232–
239, New York, May 2004. IEEE Computer Society.
[12] E. Gelenbe, R. Lent, and Z. Xhu. Measurement and performance of a cognitive packet network.
Journal of Computer Networks, 37:691–701, 2001.
[13] A. György, T. Linder, and G. Lugosi. Efficient algorithms and minimax bounds for zero-delay
lossy source coding. IEEE Transactions on Signal Processing, 52:2337–2347, Aug. 2004.
[14] A. György, T. Linder, and G. Lugosi. A ”follow the perturbed leader”-type algorithm for
zero-delay quantization of individual sequences. In Proc. Data Compression Conference, pages
342–351, Snowbird, UT, USA, Mar. 2004.
[15] A. György, T. Linder, and G. Lugosi. Tracking the best of many experts. In Proceedings of
the 18th Annual Conference on Learning Theory, COLT 2005, pages 204–216, Bertinoro, Italy,
Jun. 2005. Springer.
[16] A. György, T. Linder, and G. Lugosi. Tracking the best quantizer. In Proceedings of the IEEE
International Symposium on Information Theory, pages 1163–1167, Adelaide, Australia, June-
July 2005.
[17] A. György and Gy. Ottucsák. Adaptive routing using expert advice. The Computer Journal,
49(2):180–189, 2006.
[18] J. Hannan. Approximation to Bayes risk in repeated plays. In M. Dresher, A. Tucker, and
P. Wolfe, editors, Contributions to the Theory of Games, volume 3, pages 97–139. Princeton
University Press, 1957.
[19] M. Herbster and M. K. Warmuth. Tracking the best expert. Machine Learning, 32(2):151–178,
1998.
[20] M. Herbster and M. K. Warmuth, “Tracking the best linear predictor,” Journal of Machine
Learning Research, vol. 1, pp. 281–309, 2001.
[21] W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the
American Statistical Association, 58:13–30, 1963.
[22] D.P. Helmbold and S. Panizza. Some label efficient learning results. In Proceedings of the 10th
Annual Conference on Computational Learning Theory, pages 218–230. ACM Press, 1997.
[23] A. Kalai and S Vempala. Efficient algorithms for the online decision problem. In B. Schölkopf
and M. Warmuth, editors, Proceedings of the 16th Annual Conference on Learning Theory
and the 7th Kernel Workshop, COLT-Kernel 2003, pages 26–40, New York, USA, Aug. 2003.
Springer.
[24] N. Littlestone and M. K. Warmuth. The weighted majority algorithm. Information and
Computation, 108:212–261, 1994.
[25] H. B. McMahan and A. Blum. Online geometric optimization in the bandit setting against an
adaptive adversary. In Proceedings of the 17th Annual Conference on Learning Theory, COLT
2004, pages 109–123, Banff, Canada, Jul. 2004. Springer.
[26] M. Mohri. General algebraic frameworks and algorithms for shortest distance problems. Tech-
nical Report 981219-10TM, AT&T Labs Research, 1998.
[27] R. E. Schapire and D. P. Helmbold. Predicting nearly as well as the best pruning of a decision
tree. Machine Learning, 27:51–68, 1997.
[28] E. Takimoto and M. K. Warmuth. Path kernels and multiplicative updates. Journal of Machine
Learning Research, 4:773–818, 2003.
[29] V. Vovk. Aggregating strategies. In Proceedings of the Third Annual Workshop on Computa-
tional Learning Theory, pages 372–383, Rochester, NY, Aug. 1990. Morgan Kaufmann.
[30] V. Vovk. Derandomizing stochastic prediction strategies. Machine Learning, 35(3):247–282,
Jun. 1999.
	Introduction
	The shortest path problem
	The multi-armed bandit setting
	A bandit algorithm for shortest paths
	A combination of the label efficient and bandit settings
	A bandit algorithm for tracking the shortest path
	An algorithm for the restricted multi-armed bandit problem
	Simulation results
	Conclusions
	Appendix

  The on-line shortest path problem is considered under various models of
partial monitoring. Given a weighted directed acyclic graph whose edge weights
can change in an arbitrary (adversarial) way, a decision maker has to choose in
each round of a game a path between two distinguished vertices such that the
loss of the chosen path (defined as the sum of the weights of its composing
edges) be as small as possible. In a setting generalizing the multi-armed
bandit problem, after choosing a path, the decision maker learns only the
weights of those edges that belong to the chosen path. For this problem, an
algorithm is given whose average cumulative loss in n rounds exceeds that of
the best path, matched off-line to the entire sequence of the edge weights, by
a quantity that is proportional to 1/\sqrt{n} and depends only polynomially on
the number of edges of the graph. The algorithm can be implemented with linear
complexity in the number of rounds n and in the number of edges. An extension
to the so-called label efficient setting is also given, in which the decision
maker is informed about the weights of the edges corresponding to the chosen
path at a total of m << n time instances. Another extension is shown where the
decision maker competes against a time-varying path, a generalization of the
problem of tracking the best expert. A version of the multi-armed bandit
setting for shortest path is also discussed where the decision maker learns
only the total weight of the chosen path but not the weights of the individual
edges on the path. Applications to routing in packet switched networks along
with simulation results are also presented.

Introduction
In a sequential decision problem, a decision maker (or forecaster) performs a sequence of actions.
After each action the decision maker suffers some loss, depending on the response (or state) of the
environment, and its goal is to minimize its cumulative loss over a certain period of time. In the
setting considered here, no probabilistic assumption is made on how the losses corresponding to
different actions are generated. In particular, the losses may depend on the previous actions of the
decision maker, whose goal is to perform well relative to a set of reference forecasters (the so-called
“experts”) for any possible behavior of the environment. More precisely, the aim of the decision
maker is to achieve asymptotically the same average (per round) loss as the best expert.
Research into this problem started in the 1950s (see, for example, Blackwell [5] and Hannan
[18] for some of the basic results) and gained new life in the 1990s following the work of Vovk [29],
Littlestone and Warmuth [24], and Cesa-Bianchi et al. [7]. These results show that for any bounded
loss function, if the decision maker has access to the past losses of all experts, then it is possible
to construct on-line algorithms that perform, for any possible behavior of the environment, almost
as well as the best of N experts. More precisely, the per round cumulative loss of these algorithms
is at most as large as that of the best expert plus a quantity proportional to
lnN/n for any
bounded loss function, where n is the number of rounds in the decision game. The logarithmic
dependence on the number of experts makes it possible to obtain meaningful bounds even if the
pool of experts is very large.
In certain situations the decision maker has only limited knowledge about the losses of all
possible actions. For example, it is often natural to assume that the decision maker gets to know
only the loss corresponding to the action it has made, and has no information about the loss it would
have suffered had it made a different decision. This setup is referred to as the multi-armed bandit
problem, and was considered, in the adversarial setting, by Auer et al. [1] who gave an algorithm
whose normalized regret (the difference of the algorithm’s average loss and that of the best expert)
is upper bounded by a quantity which is proportional to
N lnN/n. Note that, compared to the
full information case described above where the losses of all possible actions are revealed to the
decision maker, there is an extra
N factor in the performance bound, which seriously limits the
usefulness of the bound if the number of experts is large.
Another interesting example for the limited information case is the so-called label efficient
decision problem (see Helmbold and Panizza [22]) in which it is too costly to observe the state of
the environment, and so the decision maker can query the losses of all possible actions for only a
limited number of times. A recent result of Cesa-Bianchi, Lugosi, and Stoltz [9] shows that in this
case, if the decision maker can query the losses m times during a period of length n, then it can
achieve O(
lnN/m) average excess loss relative to the best expert.
In many applications the set of experts has a certain structure that may be exploited to construct
efficient on-line decision algorithms. The construction of such algorithms has been of great interest
in computational learning theory. A partial list of works dealing with this problem includes Herbster
and Warmuth [19], Vovk [30], Bousquet and Warmuth [6], Helmbold and Schapire [27], Takimoto
and Warmuth [28], Kalai and Vempala [23], György at al. [13, 14, 15]. For a more complete survey,
we refer to Cesa-Bianchi and Lugosi [8, Chapter 5].
In this paper we study the on-line shortest path problem, a representative example of structured
expert classes that has received attention in the literature for its many applications, including,
among others, routing in communication networks; see, e.g., Takimoto andWarmuth [28], Awerbuch
et al. [3], or György and Ottucsák [17], and adaptive quantizer design in zero-delay lossy source
coding; see, György et al. [13, 14, 16]. In this problem, a weighted directed (acyclic) graph is given
whose edge weights can change in an arbitrary manner, and the decision maker has to pick in each
round a path between two given vertices, such that the weight of this path (the sum of the weights
of its composing edges) be as small as possible.
Efficient solutions, with time and space complexity proportional to the number of edges rather
than to the number of paths (the latter typically being exponential in the number of edges), have
been given in the full information case, where in each round the weights of all the edges are revealed
after a path has been chosen; see, for example, Mohri [26], Takimoto and Warmuth [28], Kalai and
Vempala [23], and György et al. [15].
In the bandit setting only the weights of the edges or just the sum of the weights of the edges
composing the chosen path are revealed to the decision maker. If one applies the general bandit
algorithm of Auer et al. [1], the resulting bound will be too large to be of practical use because
of its square-root-type dependence on the number of paths N . On the other hand, using the
special graph structure in the problem, Awerbuch and Kleinberg [4] and McMahan and Blum [25]
managed to get rid of the exponential dependence on the number of edges in the performance bound.
They achieved this by extending the exponentially weighted average predictor and the follow-the-
perturbed-leader algorithm of Hannan [18] to the generalization of the multi-armed bandit setting
for shortest paths, when only the sum of the weights of the edges is available for the algorithm.
However, the dependence of the bounds obtained in [4] and [25] on the number of rounds n is
significantly worse than the O(1/
n) bound of Auer et al. [1]. Awerbuch and Kleinberg [4] consider
the model of “non-oblivious” adversaries for shortest path (i.e., the losses assigned to the edges can
depend on the previous actions of the forecaster) and prove an O(n−1/3) bound for the expected
per-round regret. McMahan and Blum [25] give a simpler algorithm than in [4] however obtain a
bound of the order of O(n−1/4) for the expected regret.
In this paper we provide an extension of the bandit algorithm of Auer et al. [1] unifying the
advantages of the above approaches, with a performance bound that is polynomial in the number
of edges, and converges to zero at the right O(1/
n) rate as the number of rounds increases. We
achieve this bound in a model which assumes that the losses of all edges on the path chosen by the
forecaster are available separately after making the decision. We also discuss the case (considered
by [4] and [25]) in which only the total loss (i.e., the sum of the losses on the chosen path) is known
to the decision maker. We exhibit a simple algorithm which achieves an O(n−1/3) per-round regret
with high probability against “non-oblivious” adversary. In this case it remains an open problem
to find an algorithm whose cumulative loss is polynomial in the number of edges of the graph and
decreases as O(n−1/2) with the number of rounds.
In Section 2 we formally define the on-line shortest path problem, which is extended to the
multi-armed bandit setting in Section 3. Our new algorithm for the shortest path problem in
the bandit setting is given in Section 4 together with its performance analysis. The algorithm
is extended to solve the shortest path problem in a combined label efficient multi-armed bandit
setting in Section 5. Another extension, when the algorithm competes against a time-varying path
is studied in Section 6. An algorithm for the “restricted” multi-armed bandit setting (when only the
sums of the losses of the edges are available) is given in Section 7. Simulation results are presented
in Section 8.
2 The shortest path problem
Consider a network represented by a set of vertices connected by edges, and assume that we have
to send a stream of packets from a distinguished vertex, called source, to another distinguished
vertex, called destination. At each time slot a packet is sent along a chosen route connecting source
and destination. Depending on the traffic, each edge in the network may have a different delay, and
the total delay the packet suffers on the chosen route is the sum of delays of the edges composing
the route. The delays may change from one time slot to the next one in an arbitrary way, and
our goal is to find a way of choosing the route in each time slot such that the sum of the total
delays over time is not significantly more than that of the best fixed route in the network. This
adversarial version of the routing problem is most useful when the delays on the edges can change
dynamically, even depending on our previous routing decisions. This is the situation in the case
of ad-hoc networks, where the network topology can change rapidly, or in certain secure networks,
where the algorithm has to be prepared to handle denial of service attacks, that is, situations where
willingly malfunctioning vertices and links increase the delay; see, e.g., Awerbuch et al. [3].
This problem can be cast naturally as a sequential decision problem in which each possible
route is represented by an action. However, the number of routes is typically exponentially large
in the number of edges, and therefore computationally efficient algorithms are called for. Two
solutions of different flavor have been proposed. One of them is based on a follow-the-perturbed-
leader forecaster, see Kalai and Vempala [23], while the other is based on an efficient computation
of the exponentially weighted average forecaster, see, for example, Takimoto and Warmuth [28].
Both solutions have different advantages and may be generalized in different directions.
To formalize the problem, consider a (finite) directed acyclic graph with a set of edges E =
{e1, . . . , e|E|} and a set of vertices V . Thus, each edge e ∈ E is an ordered pair of vertices (v1, v2).
Let u and v be two distinguished vertices in V . A path from u to v is a sequence of edges e(1), . . . , e(k)
such that e(1) = (u, v1), e
(j) = (vj−1, vj) for all j = 2, . . . , k − 1, and e(k) = (vk−1, v). Let
P = {i1, . . . , iN} denote the set of all such paths. For simplicity, we assume that every edge in
E is on some path from u to v and every vertex in V is an endpoint of an edge (see Figure 1 for
examples).
PSfrag replacements
Figure 1: Two examples of directed acyclic graphs for the shortest path problem.
(a) (b)
In each round t = 1, . . . , n of the decision game, the decision maker chooses a path It among
all paths from u to v. Then a loss ℓe,t ∈ [0, 1] is assigned to each edge e ∈ E. We write e ∈ i if the
edge e ∈ E belongs to the path i ∈ P, and with a slight abuse of notation the loss of a path i at
time slot t is also represented by ℓi,t. Then ℓi,t is given as
ℓi,t =
and therefore the cumulative loss up to time t of each path i takes the additive form
Li,t =
ℓi,s =
where the inner sum on the right-hand side is the loss accumulated by edge e during the first t
rounds of the game. The cumulative loss of the algorithm is
L̂t =
ℓIs,s =
ℓe,s .
It is well known that for a general loss sequence, the decision maker must be allowed to use
randomization to be able to approximate the performance of the best expert (see, e.g., Cesa-Bianchi
and Lugosi [8]). Therefore, the path It is chosen randomly according to some distribution pt over
all paths from u to v. We study the normalized regret over n rounds of the game
L̂n −min
where the minimum is taken over all paths i from u to v.
For example, the exponentially weighted average forecaster ([29], [24], [7]), calculated over all
possible paths, has regret
L̂n −min
ln(1/δ)
with probability at least 1− δ, where N is the total number of paths from u to v in the graph and
K is the length of the longest path.
3 The multi-armed bandit setting
In this section we discuss the “bandit” version of the shortest path problem. In this setup, which is
more realistic in many applications, the decision maker has only access to the losses corresponding
to the paths it has chosen. For example, in the routing problem this means that information is
available on the delay of the route the packet is sent on, and not on other routes in the network.
We distinguish between two types of bandit problems, both of which are natural generalizations
of the simple bandit problem to the shortest path problem. In the first variant, the decision maker
has access to the losses of those edges that are on the path it has chosen. That is, after choosing a
path It at time t, the value of the loss ℓe,t is revealed to the decision maker if and only if e ∈ It.
We study this case and its extensions in Sections 4, 5, and 6.
The second variant is a more restricted version in which the loss of the chosen path is observed,
but no information is available on the individual losses of the edges belonging to the path. That
is, after choosing a path It at time t, only the value of the loss of the path ℓIt,t is revealed to the
decision maker. Further on we call this setting as the restricted bandit problem for shortest path.
We consider this restricted problem in Section 7.
Formally, the on-line shortest path problem in the multi-armed bandit setting is described as
follows: at each time instance t = 1, . . . , n, the decision maker picks a path It ∈ P from u to v.
Then the environment assigns loss ℓe,t ∈ [0, 1] to each edge e ∈ E, and the decision maker suffers
loss ℓIt,t =
ℓe,t. In the unrestricted case the losses ℓe,t are revealed for all e ∈ It, while in
the restricted case only ℓIt,t is revealed. Note that in both cases ℓe,t may depend on I1, . . . , It−1,
the earlier choices of the decision maker.
For the basic multi-armed bandit problem, Auer et al. [1] gave an algorithm, based on exponen-
tial weighting with a biased estimate of the gains (defined, in our case, as gi,t = K− ℓi,t), combined
with uniform exploration. Applying their algorithm to the on-line shortest path problem in the
bandit setting results in a performance that can be bounded, for any 0 < δ < 1 and fixed time
horizon n, with probability at least 1− δ, by
L̂n −min
≤ 11K
N ln(N/δ)
K lnN
(The constants follow from a slightly improved version; see Cesa-Bianchi and Lugosi [8].)
However, for the shortest path problem this bound is unacceptably large because, unlike in the
full information case, here the dependence on the number of all paths N is not merely logarithmic,
while N is typically exponentially large in the size of the graph (as in the two simple examples
of Figure 1). In order to achieve a bound that does not grow exponentially with the number of
edges of the graph, it is imperative to make use of the dependence structure of the losses of the
different actions (i.e., paths). Awerbuch and Kleinberg [4] and McMahan and Blum [25] do this
by extending low complexity predictors, such as the follow-the-perturbed-leader forecaster [18],
[23] to the restricted bandit setting. However, in both cases the price to pay for the polynomial
dependence on the number of edges is a worse dependence on the length n of the game.
4 A bandit algorithm for shortest paths
In this section we describe a variant of the bandit algorithm of [1] which achieves the desired
performance for the shortest path problem. The new algorithm uses the fact that when the losses
of the edges of the chosen path are revealed, then this also provides some information about the
losses of each path sharing common edges with the chosen path.
For each edge e ∈ E, and t = 1, 2, . . ., introduce the gain ge,t = 1− ℓe,t, and for each path i ∈ P,
let the gain be the sum of the gains of the edges on the path, that is,
gi,t =
ge,t .
The conversion from losses to gains is done in order to facilitate the subsequent performance
analysis. To simplify the conversion, we assume that each path i ∈ P is of the same length K
for some K > 0. Note that although this assumption may seem to be restrictive at the first
glance, from each acyclic directed graph (V,E) one can construct a new graph by adding at most
(K−2)(|V |−2)+1 vertices and edges (with constant weight zero) to the graph without modifying
the weights of the paths such that each path from u to v will be of length K, where K denotes the
length of the longest path of the original graph. If the number of edges is quadratic in the number
of vertices, the size of the graph is not increased substantially.
A main feature of the algorithm below is that the gains are estimated for each edge and not
for each path. This modification results in an improved upper bound on the performance with
the number of edges in place of the number of paths. Moreover, using dynamic programming as
in Takimoto and Warmuth [28], the algorithm can be computed efficiently. Another important
ingredient of the algorithm is that one needs to make sure that every edge is sampled sufficiently
often. To this end, we introduce a set C of covering paths with the property that for each edge
e ∈ E there is a path i ∈ C such that e ∈ i. Observe that one can always find such a covering set
of cardinality |C| ≤ |E|.
We note that the algorithm of [1] is a special case of the algorithm below: For any multi-
armed bandit problem with N experts, one can define a graph with two vertices u and v, and N
directed edges from u to v with weights corresponding to the losses of the experts. The solution
of the shortest path problem in this case is equivalent to that of the original bandit problem with
choosing expert i if the corresponding edge is chosen. For this graph, our algorithm reduces to the
original algorithm of [1].
A BANDIT ALGORITHM FOR SHORTEST PATHS
Parameters: real numbers β > 0, 0 < η, γ < 1.
Initialization: Set we,0 = 1 for each e ∈ E, wi,0 = 1 for each i ∈ P, and W 0 = N .
For each round t = 1, 2, . . .
(a) Choose a path It at random according to the distribution pt on P, defined by
pi,t =
(1− γ)wi,t−1
W t−1
if i ∈ C
(1− γ)wi,t−1
W t−1
if i 6∈ C.
(b) Compute the probability of choosing each edge e as
qe,t =
i:e∈i
pi,t = (1− γ)
i:e∈iwi,t−1
W t−1
|{i ∈ C : e ∈ i}|
(c) Calculate the estimated gains
g′e,t =
ge,t+β
if e ∈ It
otherwise.
(d) Compute the updated weights
we,t = we,t−1e
ηg′e,t
wi,t =
we,t = wi,t−1e
where g′
i,t =
e∈i g
e,t, and the sum of the total weights of the paths
W t =
wi,t.
The main result of the paper is the following performance bound for the shortest-path bandit
algorithm. It states that the per round regret of the algorithm, after n rounds of play, is, roughly,
of the order of K
|E| lnN/n where |E| is the number of edges of the graph, K is the length of the
paths, and N is the total number of paths.
Theorem 1 For any δ ∈ (0, 1) and parameters 0 ≤ γ < 1/2, 0 < β ≤ 1, and η > 0 satisfying
2ηK|C| ≤ γ, the performance of the algorithm defined above can be bounded, with probability at
least 1− δ, as
L̂n −min
≤ Kγ + 2ηK2|C|+
+ |E|β.
In particular, choosing β =
, γ = 2ηK|C|, and η =
4nK2|C|
yields for all n ≥
, 4|C| lnN
L̂n −min
4K|C| lnN +
|E| ln |E|
The proof of the theorem is based on the analysis of the original algorithm of [1] with necessary
modifications required to transform parts of the argument from paths to edges, and to use the
connection between the gains of paths sharing common edges.
For the analysis we introduce some notation:
Gi,n =
gi,t and G
i,n =
g′i,t
for each i ∈ P and
Ge,n =
ge,t and G
e,n =
g′e,t
for each e ∈ E, and
Ĝn =
gIt,t.
The following lemma, shows that the deviation of the true cumulative gain from the estimated
cumulative gain is of the order of
n. The proof is a modification of [8, Lemma 6.7].
Lemma 1 For any δ ∈ (0, 1), 0 ≤ β < 1 and e ∈ E we have
Ge,n > G
e,n +
Proof. Fix e ∈ E. For any u > 0 and c > 0, by the Chernoff bound we have
P[Ge,n > G
e,n + u] ≤ e−cuEec(Ge,n−G
e,n) . (1)
Letting u = ln(|E|/δ)/β and c = β, we get
e−cuEec(Ge,n−G
e,n) = e− ln(|E|/δ)Eeβ(Ge,n−G
e,n) =
Eeβ(Ge,n−G
e,n) ,
so it suffices to prove that Eeβ(Ge,n−G
e,n) ≤ 1 for all n. To this end, introduce
Zt = e
β(Ge,t−G
e,t) .
Below we show that Et[Zt] ≤ Zt−1 for t ≥ 2 where Et denotes the conditional expectation
E[·|I1, . . . , It−1] . Clearly,
Zt = Zt−1 exp
ge,t −
1{e∈It}ge,t + β
Taking conditional expectations, we obtain
Et[Zt]
= Zt−1Et
ge,t −
1{e∈It}ge,t + β
= Zt−1e
qe,t Et
ge,t −
1{e∈It}ge,t
≤ Zt−1e
qe,t Et
1 + β
ge,t −
1{e∈It}ge,t
ge,t −
1{e∈It}ge,t
= Zt−1e
qe,t Et
1 + β2
ge,t −
1{e∈It}ge,t
≤ Zt−1e
qe,t Et
1 + β2
1{e∈It}ge,t
≤ Zt−1e
≤ Zt−1. (4)
Here (2) holds since β ≤ 1, ge,t −
1{e∈It}
≤ 1 and ex ≤ 1 + x + x2 for x ≤ 1. (3) follows from
1{e∈It}
= ge,t. Finally, (4) holds by the inequality 1 + x ≤ ex. Taking expectations on both
sides proves E[Zt] ≤ E[Zt−1]. A similar argument shows that E[Z1] ≤ 1, implying E[Zn] ≤ 1 as
desired. ✷
Proof of Theorem 1. As usual in the analysis of exponentially weighted average forecasters, we
start with bounding the quantity ln Wn
. On the one hand, we have the lower bound
i,n − lnN ≥ ηmax
G′i,n − lnN . (5)
To derive a suitable upper bound, first notice that the condition η ≤ γ
2K|C|
implies ηg′
i,t ≤ 1 for
all i and t, since
ηg′i,t = η
g′e,t ≤ η
1 + β
≤ ηK(1 + β)|C|
where the second inequality follows because qe,t ≥ γ/|C| for each e ∈ E.
Therefore, using the fact that ex ≤ 1 + x+ x2 for all x ≤ 1, for all t = 1, 2, . . . we have
W t−1
wi,t−1
W t−1
pi,t − γ|C|1{i∈C}
pi,t − γ|C|1{i∈C}
1 + ηg′i,t + η
ηg′i,t + η
pi,tg
i,t +
pi,tg
i,t (7)
where (6) follows form the definition of pi,t, and (7) holds by the inequality ln(1 + x) ≤ x for all
x > −1.
Next we bound the sums in (7). On the one hand,
pi,tg
i,t =
g′e,t =
g′e,t
i∈P:e∈i
g′e,tqe,t = gIt,t + |E|β.
On the other hand,
pi,tg
i,t =
g′e,t
pi,tK
i∈P:e∈i
e,tqe,t
qe,tg
β + 1{e∈It}ge,t
≤ K(1 + β)
g′e,t
where the first inequality is due to the inequality between the arithmetic and quadratic mean, and
the second one holds because ge,t ≤ 1. Therefore,
W t−1
(gIt,t + |E|β) +
η2K(1 + β)
g′e,t .
Summing for t = 1, . . . , n, we obtain
Ĝn + n|E|β
η2K(1 + β)
G′e,n
Ĝn + n|E|β
η2K(1 + β)
|C|max
G′i,n (8)
where the second inequality follows since
e∈E G
e,n ≤
i∈C G
i,n. Combining the upper bound
with the lower bound (5), we obtain
Ĝn ≥ (1− γ − ηK(1 + β)|C|)max
G′i,n −
lnN − n|E|β. (9)
Now using Lemma 1 and applying the union bound, for any δ ∈ (0, 1) we have that, with probability
at least 1− δ,
Ĝn ≥ (1− γ − ηK(1 + β)|C|)
Gi,n −
− 1− γ
lnN − n|E|β ,
where we used 1− γ − ηK(1 + β)|C| ≥ 0 which follows from the assumptions of the theorem.
Since Ĝn = Kn− L̂n and Gi,n = Kn− Li,n for all i ∈ P, we have
L̂n ≤ Kn (γ + η(1 + β)K|C|) + (1− γ − η(1 + β)K|C|)min
+ (1− γ − η(1 + β)K|C|)
lnN + n|E|β
with probability at least 1− δ. This implies
L̂n −min
Li,n ≤ Knγ + η(1 + β)nK2|C|+
lnN + n|E|β
≤ Knγ + 2ηnK2|C|+ K
+ n|E|β
with probability at least 1− δ, which is the first statement of the theorem. Setting
and γ = 2ηK|C|
results in the inequality
L̂n −min
Li,n ≤ 4ηnK2|C|+
nK|E| ln
which holds with probability at least 1 − δ if n ≥ (K/|E|) ln(|E|/δ) (to ensure β ≤ 1). Finally,
setting
4nK2|C|
yields the last statement of the theorem (n ≥ 4 lnN |C| is required to ensure γ ≤ 1/2). ✷
Next we analyze the computational complexity of the algorithm. The next result shows that
the algorithm is feasible as its complexity is linear in the size (number of edges) of the graph.
Theorem 2 The proposed algorithm can be implemented efficiently with time complexity O(n|E|)
and space complexity O(|E|).
Proof. The two complex steps of the algorithm are steps (a) and (b), both of which can be
computed, similarly to Takimoto and Warmuth [28], using dynamic programming. To perform
these steps efficiently, first we order the vertices of the graph. Since we have an acyclic directed
graph, its vertices can be labeled (in O(|E|) time) from 1 to |V | such that u = 1, v = |V |, and if
(v1, v2) ∈ E, then v1 < v2. For any pair of vertices u1 < v1 let Pu1,v1 denote the set of paths from
u1 to v1, and for any vertex s ∈ V , let
Ht(s) =
i∈Ps,v
Ĥt(s) =
i∈Pu,s
we,t .
Given the edge weights {we,t}, Ht(s) can be computed recursively for s = |V | − 1, . . . , 1, and Ĥt(s)
can be computed recursively for s = 2, . . . , |V | in O(|E|) time (letting Ht(v) = Ĥt(u) = 1 by
definition). In step (a), first one has to decide with probability γ whether It is generated according
to the graph weights, or it is chosen uniformly from C. If It is to be drawn according to the graph
weights, it can be shown that its vertices can be chosen one by one such that if the first k vertices of
It are v0 = u, v1, . . . , vk−1, then the next vertex of It can be chosen to be any vk > vk−1, satisfying
(vk−1, vk) ∈ E, with probability w(vk−1,vk),t−1Ht−1(vk)/Ht−1(vk−1). The other computationally
demanding step, namely step (b), can be performed easily by noting that for any edge (v1, v2),
q(v1,v2),t = (1− γ)
Ĥt−1(v1)w(v1,v2),t−1Ht−1(v2)
Ht−1(u)
|{i ∈ C : (v1, v2) ∈ i}|
as desired. ✷
5 A combination of the label efficient and bandit settings
In this section we investigate a combination of the multi-armed bandit and the label efficient
problems. This means that the decision maker only has access to the loss of the chosen path upon
request and the total number of requests must be bounded by a constant m. This combination is
motivated by some applications, in which feedback information is costly to obtain.
In the general label efficient decision problem, after taking an action, the decision maker has the
option to query the losses of all possible actions. For this problem, Cesa-Bianchi et al. [9] proved an
upper bound on the normalized regret of order O(K
ln(4N/δ)/(m)) which holds with probability
at least 1− δ.
Our model of the label-efficient bandit problem for shortest paths is motivated by an application
to a particular packet switched network model. This model, called the cognitive packet network,
was introduced by Gelenbe et al. [11, 12]. In these networks a particular type of packets, called
smart packets, are used to explore the network (e.g., the delay of the chosen path). These packets
do not carry any useful data; they are merely used for exploring the network. The other type of
packets are the data packets, which do not collect any information about their paths. The task of
the decision maker is to send packets from the source to the destination over routes with minimum
average transmission delay (or packet loss). In this scenario, smart packets are used to query the
delay (or loss) of the chosen path. However, as these packets do not transport information, there
is a tradeoff between the number of queries and the usage of the network. If data packets are on
the average α times larger than smart packets (note that typically α ≫ 1) and ǫ is the proportion
of time instances when smart packets are used to explore the network, then ǫ/(ǫ+ α(1− ǫ)) is the
proportion of the bandwidth sacrificed for well informed routing decisions.
We study a combined algorithm which, at each time slot t, queries the loss of the chosen path
with probability ǫ (as in the solution of the label efficient problem proposed in [9]), and, similarly
to the multi-armed bandit case, computes biased estimates g′
i,t of the true gains gi,t. Just as in the
previous section, it is assumed that each path of the graph is of the same length K.
The algorithm differs from our bandit algorithm of the previous section only in step (c), which
is modified in the spirit of [9]. The modified step is given below:
MODIFIED STEP FOR THE LABEL EFFICIENT BANDIT ALGORITHM FOR
SHORTEST PATHS
(c’) Draw a Bernoulli random variable St with P(St = 1) = ǫ, and compute the
estimated gains
g′e,t =
ge,t+β
ǫqe,t
St if e ∈ It
ǫqe,t
St if e /∈ It .
The performance of the algorithm is analyzed in the next theorem, which can be viewed as a
combination of Theorem 1 in the preceding section and Theorem 2 of [9].
Theorem 3 For any δ ∈ (0, 1), ǫ ∈ (0, 1], parameters η =
ǫ lnN
4nK2|C|
, γ =
2ηK|C|
≤ 1/2, and
n|E|ǫ
≤ 1, and for all
K2 ln2(2|E|/δ)
|E| lnN
|E| ln(2|E|/δ)
, 4|C| lnN
the performance of the algorithm defined above can be bounded, with probability at least 1− δ, as
L̂n −min
K|C| lnN + 5
|E| ln
8K ln
|E| ln 2N
If ǫ is chosen as (m −
2m ln(1/δ))/n then, with probability at least 1 − δ, the total number
of queries is bounded by m (see [8, Lemma 6.1]) and the performance bound above is of the order
|E| ln(N/δ)/m.
Similarly to Theorem 1, we need a lemma which reveals the connection between the true and the
estimated cumulative losses. However, here we need a more careful analysis because the “shifting
term”
ǫqe,t
St, is a random variable.
Lemma 2 For any 0 < δ < 1, 0 < ǫ ≤ 1, for any
n ≥ 1
K2 ln2(2|E|/δ)
|E| lnN
K ln(2|E|/δ)
parameters
2ηK|C|
≤ γ, η =
ǫ lnN
4nK2|C|
and β =
n|E|ǫ
≤ 1 ,
and e ∈ E, we have
Ge,n > G
e,n +
Proof. Fix e ∈ E. Using (1) with u = 4
and c =
, it suffices to prove for all n that
ec(Ge,n−G
≤ 1 .
Similarly to Lemma 1 we introduce Zt = e
c(Ge,t−G
e,t) and we show that Z1, . . . , Zn is a super-
martingale, that is Et[Zt] ≤ Zt−1 for t ≥ 2 where Et denotes E[·|(I1, S1), . . . , (I t−1, St−1)]. Taking
conditional expectations, we obtain
Et[Zt] = Zt−1Et
ge,t−
1{e∈It}
Stge,t+Stβ
qe,tǫ
≤ Zt−1Et
1 + c
ge,t −
1{e∈It}Stge,t + Stβ
qe,tǫ
ge,t −
1{e∈It}Stge,t + Stβ
qe,tǫ
. (10)
Since
ge,t −
1{e∈It}Stge,t + Stβ
qe,tǫ
= − β
ge,t −
1{e∈It}Stge,t
qe,tǫ
1{e∈It}Stge,t
qe,tǫ
qe,tǫ
we get from (10) that
Et[Zt]
≤ Zt−1Et
1− cβ
qe,tǫ
21{e∈It}Stge,tβ
q2e,tǫ
2ge,tStβ
qe,tǫ
q2e,tǫ
≤ Zt−1
qe,tǫ
. (11)
Since c = βǫ/4 we have
− β + c
qe,tǫ
= −3β
qe,tǫ
4qe,t
4qe,t
β3|C|
≤ 0, (13)
where (12) follows from qe,t ≥ γ|C| and (13) holds by
β2|C|
≤ 1 ,
and the last inequality is ensured by n ≥ K
2 ln2(2|E|/δ)
ǫ|E| lnN
, the assumption of the lemma.
Combining (11) and (13) we get that Et[Zt] ≤ Zt−1. Taking expectations on both sides of the
inequality, we get E[Zt] ≤ E[Zt−1] and since E[Z1] ≤ 1, we obtain E[Zn] ≤ 1 as desired. ✷
Proof of Theorem 3. The proof of the theorem is a generalization of that of Theorem 1, and
follows the same lines with some extra technicalities to handle the effects of the modified step (c’).
Therefore, in the following we emphasize only the differences. First note that (5) and (7) also hold
in this case. Bounding the sums in (7), one obtains
pi,tg
i,t =
(gIt,t + |E|β)
and ∑
pi,tg
i,t ≤
K(1 + β)
g′e,t .
Plugging these bounds into (7) and summing for t = 1, . . . , n, we obtain
(gIt,t + |E|β ) +
η2K(1 + β)
(1− γ)ǫ
|C|max
G′i,n .
Combining the upper bound with the lower bound (5), we obtain
(gIt,t + |E|β ) ≥
1−γ− ηK(1 + β)|C|
G′i,n−
. (14)
To relate the left-hand side of the above inequality to the real gain
t=1 gIt,t, notice that
(gIt,t + |E|β) − (gIt,t + |E|β)
is a martingale difference sequence with respect to (I1, S1), (I2, S2), . . .. Now for all t = 1, . . . , n,
we have the bound
X2t |(I1, S1), . . . , (It−1, St−1)
(gIt,t + |E|β)2
∣∣∣∣(I1, S1), . . . , (I t−1, St−1)
≤ (K + |E|β)
= σ2, (15)
where (15) holds by n ≥ |E| ln(2|E|/δ)
(to ensure β|E| ≤ K). We know that
for all t. Now apply Bernstein’s inequality for martingale differences (see Lemma 9 in the Appendix)
to obtain
Xt > u
, (16)
where
From (16) we get
(gIt,t + |E|β) ≥ Ĝn + βn|E|+ u
. (17)
Now Lemma 2, the union bound, and (17) combined with (14) yield, with probability at least
1− δ,
Ĝn ≥
1− γ −
ηK(1 + β)|C|
Gi,n −
− βn|E| − u
since the coefficient of G′i,n is greater than zero by the assumptions of the theorem.
Since Ĝn = Kn− L̂n and Gi,n = Kn− Li,n, we have
L̂n ≤
1− γ − K(1 + β)η|C|
Li,n +Kn
K(1 + β)η|C|
1− γ −
K(1 + β)η|C|
+ βn|E|+
≤ min
Li,n +Kn
K(1 + β)η|C|
+ 5βn|E|+
+ u ,
where we used the fact that K
= βn|E|.
Substituting the value of β, η and γ, we have
L̂n −min
Li,n ≤Kn
2Kη|C|
2Kη|C|
+ 5βn|E|+ u
n|C| lnN
n|E|K ln(2|E|/δ)
K|C| lnN + 5
|E| ln(2|E|/δ) +
8K ln (2/δ)
ln (2/δ)
as desired. ✷
6 A bandit algorithm for tracking the shortest path
Our goal in this section is to extend the bandit algorithm so that it is able to compete with time-
varying paths under small computational complexity. This is a variant of the problem known
as tracking the best expert ; see, for example, Herbster and Warmuth [19], Vovk [30], Auer and
Warmuth [2], Bousquet and Warmuth [6], Herbster and Warmuth [20].
To describe the loss the decision maker is compared to, consider the following “m-partition”
prediction scheme: the sequence of paths is partitioned into m+1 contiguous segments, and on each
segment the scheme assigns exactly one of the N paths. Formally, an m-partition Part(n,m, t, i) of
the n paths is given by an m-tuple t = (t1, . . . , tm) such that t0 = 1 < t1 < · · · < tm < n+1 = tm+1,
and an (m+ 1)-vector i = (i0, . . . , im) where ij ∈ P. At each time instant t, tj ≤ t < tj+1, path ij
is used to predict the best path. The cumulative loss of a partition Part(n,m, t, i) is
L(Part(n,m, t, i)) =
tj+1−1∑
ℓij ,t =
tj+1−1∑
ℓe,t.
The goal of the decision maker is to perform as well as the best time-varying path (partition),
that is, to keep the normalized regret
L̂n −min
L(Part(n,m, t, i))
as small as possible (with high probability) for all possible outcome sequences.
In the “classical” tracking problem there is a relatively small number of “base” experts and the
goal of the decision maker is to predict as well as the best “compound” expert (i.e., time-varying
expert). However in our case, base experts correspond to all paths of the graph between source and
destination whose number is typically exponentially large in the number of edges, and therefore we
cannot directly apply the computationally efficient methods for tracking the best expert. György,
Linder, and Lugosi [15] develop efficient algorithms for tracking the best expert for certain large
and structured classes of base experts, including the shortest path problem. The purpose of the
following algorithm is to extend the methods of [15] to the bandit setting when the forecaster only
observes the losses of the edges on the chosen path.
A BANDIT ALGORITHM FOR TRACKING SHORTEST PATHS
Parameters: real numbers β > 0, 0 < η, γ < 1, 0 ≤ α ≤ 1.
Initialization: Set we,0 = 1 for each e ∈ E, wi,0 = 1 for each i ∈ P, and W 0 = N .
For each round t = 1, 2, . . .
(a) Choose a path It according to the distribution pt defined by
pi,t =
(1− γ)wi,t−1
W t−1
if i ∈ C;
(1− γ)wi,t−1
W t−1
if i 6∈ C.
(b) Compute the probability of choosing each edge e as
qe,t =
i:e∈i
pi,t = (1− γ)
i:e∈iwi,t−1
W t−1
|{i ∈ C : e ∈ i}|
(c) Calculate the estimated gains
g′e,t =
ge,t+β
if e ∈ It;
otherwise.
(d) Compute the updated weights
vi,t = wi,t−1e
wi,t = (1− α)vi,t +
where g′
i,t =
e∈i g
e,t and W t is the sum of the total weights of the paths, that
W t =
vi,t =
wi,t.
The following performance bounds shows that the normalized regret with respect to the best
time-varying path which is allowed to switch pathsm times is roughly of the order ofK
(m/n)|C| lnN .
Theorem 4 For any δ ∈ (0, 1) and parameters 0 ≤ γ < 1/2, α, β ∈ [0, 1], and η > 0 satisfying
2ηK|C| ≤ γ, the performance of the algorithm defined above can be bounded, with probability at
least 1− δ, as
L̂n −min
L(Part(n,m, t, i))
≤ Kn (γ + η(1 + β)K|C|) +
K(m+ 1)
|E|(m+ 1)
+ βn|E|+ 1
αm(1− α)n−m−1
In particular, choosing
K(m+ 1)
|E|(m+ 1)
, γ = 2ηK|C|, α =
(m+ 1) lnN +m ln
e(n−1)
4nK2|C|
we have, for all n ≥ max
K(m+1)
|E|(m+1)
, 4|C|D
L̂n −min
L(Part(n,m, t, i)) ≤ 2
4K|C|D +
|E|(m + 1) ln
|E|(m + 1)
where
D = (m+ 1) lnN +m
1 + ln
The proof of the theorem is a combination of that of our Theorem 1 and Theorem 1 of [15]. We
will need the following three lemmas.
Lemma 3 For any 1 ≤ t ≤ t′ ≤ n and any i ∈ P,
vi,t′
wi,t−1
≥ eηG
i,[t,t′](1− α)t′−t
where G′
i,[t,t′]
τ=t g
i,τ .
Proof. The proof is a straightforward modification of the one in Herbster and Warmuth [19]. From
the definitions of vi,t and wi,t (see step (d) of the algorithm) it is clear that for any τ ≥ 1,
wi,τ = (1− α)vi,τ +
W τ ≥ (1− α)eηg
i,τwi,τ−1 .
Applying this equation iteratively for τ = t, t+1, . . . , t′ − 1, and the definition of vi,t (step (d)) for
τ = t′, we obtain
vi,t′ = wi,t′−1e
i,t′ ≥ eηg
t′−1∏
(1− α)eηg
wi,t−1
i,[t,t′](1− α)t′−twi,t−1
which implies the statement of the lemma. ✷
Lemma 4 For any t ≥ 1 and i, j ∈ P, we have
Proof. By the definition of wi,t we have
wi,t = (1− α)vi,t +
W t ≥
W t ≥
vj,t .
This completes the proof of the lemma. ✷
The next lemma is a simple corollary of Lemma 1.
Lemma 5 For any δ ∈ (0, 1), 0 ≤ β ≤ 1, t ≥ 1 and e ∈ E we have
Ge,t > G
e,t +
|E|(m+ 1)
|E|(m+ 1)
Proof of Theorem 4. The theorem is proved the same way as Theorem 1 until (8), that is,
Ĝn + n|E|β
η2K(1 + β)
|C|max
G′i,n . (18)
Let Part(n,m, t, i) be an arbitrary partition. Then the lower bound is obtained as
vim,n
(recall that im denotes the path used in the last segment of the partition). Now vim,n can be
rewritten in the form of the following telescoping product
vim,n = wi0,t0−1
vi0,t1−1
wi0,t0−1
wij ,tj−1
vij−1,tj−1
vij ,tj+1−1
wij ,tj−1
Therefore, applying Lemmas 3 and 4, we have
vim,n ≥ wi0,t0−1
)m m∏
(1− α)tj+1−1−tjeηG
ij ,[tj ,tj+1−1]
′(Part(n,m,t,i))(1− α)n−m−1.
Combining the lower bound with the upper bound (18), we have
αm(1− α)n−m−1
ηG′(Part(n,m, t, i))
Ĝn + n|E|β
η2K(1+β)
|C|maxi∈P G′i,n ,
where we used the fact that Part(n,m, t, i) is an arbitrary partition. After rearranging and using
maxi∈P G
i,n ≤ maxt,iG′(Part(n,m, t, i)) we get
Ĝn ≥ (1− γ − ηK(1 + β)|C|) max
G′(Part(n,m, t, i))
−n|E|β − 1− γ
αm(1− α)n−m−1
Now since 1 − γ − ηK(1 + β)|C| ≥ 0, by the assumptions of the theorem and from Lemma 5 with
an application of the union bound we obtain that, with probability at least 1− δ,
Ĝn ≥ (1− γ − ηK(1 + β)|C|)
G(Part(n,m, t, i))−
K(m+ 1)
|E|(m+ 1)
− n|E|β − 1− γ
αm(1− α)n−m−1
Since Ĝn = Kn− L̂n and G(Part(n,m, t, i)) = Kn− L(Part(n,m, t, i)), we have
L̂n ≤ (1− γ − ηK(1 + β)|C|) min
L(Part(n,m, t, i)) +Kn (γ + η(1 + β)K|C|)
+ (1− γ − η(1 + β)K|C|) K(m+ 1)
|E|(m+ 1)
+ n|E|β
αm(1− α)n−m−1
This implies that, with probability at least 1− δ,
L̂n −min
L(Part(n,m, t, i))
≤ Kn (γ + η(1 + β)K|C|) +
K(m+ 1)
|E|(m+ 1)
+ n|E|β + 1
αm(1− α)n−m−1
. (20)
To prove the second statement, let H(p) = −p ln p − (1 − p) ln(1 − p) and D(p ‖ q) = p ln p
(1− p) ln 1−p
. Optimizing the value of α in the last term of (20) gives
αm(1− α)n−m−1
(m+ 1) ln (N) +m ln
+ (n−m− 1) ln
(m+ 1) ln (N) + (n − 1)(Db(α∗ ‖ α) +Hb(α∗))
where α∗ = m
. For α = α∗ we obtain
αm(1− α)n−m−1
((m+ 1) ln (N) + (n− 1)(Hb(α∗)))
((m+ 1) ln (N) +m ln((n − 1)/m)
+(n−m− 1) ln(1 +m/(n−m− 1)))
((m+ 1) ln (N) +m ln((n − 1)/m) +m)
(m+ 1) ln (N) +m ln
e(n− 1)
where the inequality follows since ln(1 + x) ≤ x for x > 0. Therefore
L̂n −min
L(Part(n,m, t, i))
≤ Kn (γ + η(1 + β)K|C|) +
K(m+ 1)
|E|(m+ 1)
+ n|E|β +
which is the first statement of the theorem. Setting
K(m+ 1)
|E|(m+ 1)
, γ = 2ηK|C|, and η =
4nK2|C|
results in the second statement of the theorem, that is,
L̂n −min
L(Part(n,m, t, i))
4K|C|D +
|E|(m+ 1) ln
|E|(m+ 1)
Similarly to [15], the proposed algorithm has an alternative version, which is efficiently com-
putable:
AN ALTERNATIVE BANDIT ALGORITHM FOR TRACKING SHORTEST
PATHS
For t = 1, choose I1 uniformly from the set P. For t ≥ 2,
(a) Draw a Bernoulli random variable Γt with P(Γt = 1) = γ.
(b) If Γt = 1, then choose It uniformly from C.
(c) If Γt = 0,
(c1) choose τt randomly according to the distribution
P{τt = t′} =
(1−α)t−1Z1,t−1
for t′ = 1
α(1−α)t−t
Wt′Zt′,t−1
for t′ = 2, . . . , t
where Zt′,t−1 =
i∈P e
i,[t′,t−1] for t′ = 1, . . . , t− 1, and Zt,t−1 = N ;
(c2) given τt = t
′, choose It randomly according to the probabilities
P{It = i|τt = t′} =
i,[t′,t−1]
Zt′,t−1
for t′ = 1, . . . , t− 1
for t′ = t.
In a way completely analogous to [15], in this alternative formulation of the algorithm one can
compute the probabilities P{It = i|τt = t′} and the normalization factors Zt′,t−1 efficiently. Using
the fact that the baseline bandit algorithm for shortest paths has an O(n|E|) time complexity by
Theorem 2, it follows from Theorem 3 of [15] that the time complexity of the alternative bandit
algorithm for tracking the shortest path is O(n2|E|).
7 An algorithm for the restricted multi-armed bandit problem
In this section we consider the situation where the decision maker receives information only about
the performance of the whole chosen path, but the individual edge losses are not available. That
is, the forecaster has access to the sum ℓIt,t of losses over the chosen path It but not to the losses
{ℓe,t}e∈It of the edges belonging to It.
This is the problem formulation considered by McMahan and Blum [25] and Awerbuch and
Kleinberg [4]. McMahan and Blum provided a relatively simple algorithm whose regret is at
most of the order of n−1/4, while Awerbuch and Kleinberg gave a more complex algorithm to
achieve O(n−1/3) regret. In this section we combine the strengths of these papers, and propose
a simple algorithm with regret at most of the order of n−1/3. Moreover, our bound holds with
high probability, while the above-mentioned papers prove bounds for the expected regret only. The
proposed algorithm uses ideas very similar to those of McMahan and Blum [25]. The algorithm
alternates between choosing a path from a “basis” B to obtain unbiased estimates of the loss
(exploration step), and choosing a path according to exponential weighting based on these estimates.
A simple way to describe a path i ∈ P is a binary row vector with |E| components which are
indexed by the edges of the graph such that, for each e ∈ E, the eth entry of the vector is 1 if e ∈ i
and 0 otherwise. With a slight abuse of notation we will also denote by i the binary row vector
representing path i. In the previous sections, where the loss of each edge along the chosen path
is available to the decision maker, the complexity stemming from the large number of paths was
reduced by representing all information in terms of the edges, as the set of edges spans the set of
paths. That is, the vector corresponding to a given path can be expressed as the linear combination
of the unit vectors associated with the edges (the eth component of the unit vector representing
edge e is 1, while the other components are 0). While the losses corresponding to such a spanning
set are not observable in the restricted setting of this section, one can choose a subset of P that
forms a basis, that is, a collection of b paths which are linearly independent and each path in P
can be expressed as a linear combination of the paths in the basis. We denote by B the b × |E|
matrix whose rows b1, . . . , bb represent the paths in the basis. Note that b is equal to the maximum
number of linearly independent vectors in {i : i ∈ P}, so b ≤ |E|.
Let ℓ
t denote the (column) vector of the edge losses {ℓe,t}e∈E at time t, and let ℓ
(ℓb1,t, . . . , ℓbb,t)
T be a b-dimensional column vector whose components are the losses of the paths
in the basis B at time t. If α
(i,B)
, . . . , α
(i,B)
are the coefficients in the linear combination of the
basis paths expressing path i ∈ P, that is, i =
j=1 α
(i,B)
j, then the loss of path i ∈ P at time t
is given by
ℓi,t = 〈i, ℓ
t 〉 =
(i,B)
〈bj, ℓ(E)t 〉 =
(i,B)
bj ,t (21)
where 〈·, ·〉 denotes the standard inner product in R|E|. In the algorithm we obtain estimates ℓ̃
bj ,t
of the losses of the basis paths and use (21) to estimate the loss of any i ∈ P as
ℓ̃i,t =
(i,B)
bj ,t . (22)
It is algorithmically advantageous to calculate the estimated path losses ℓ̃i,t from an intermediate
estimate of the individual edge losses. LetB+ denote the the Moore-Penrose inverse ofB defined by
B+ = BT (BBT )−1, where BT denotes the transpose of B and BBT is invertible since the rows of
B are linearly independent. (Note thatB+ = B−1 if b = |E|). Then letting ℓ̃(B)t = (ℓ̃b1,t, . . . , ℓ̃bb,t)
t = B
it is easy to see that ℓ̃i,t in (22) can be obtained as ℓ̃i,t = 〈i, ℓ̃
t 〉, or equivalently
ℓ̃i,t =
ℓ̃e,t.
This form of the path losses allows for an efficient implementation of exponential weighting via
dynamic programming [28].
To analyze the algorithm we need an upper bound on the magnitude of the coefficients α
(i,B)
For this, we invoke the definition of a barycentric spanner from [4]: the basis B is called a C-
barycentric spanner if |α(i,B)
| ≤ C for all i ∈ P and j = 1, . . . , b. Awerbuch and Kleinberg [4] show
that a 1-barycentric spanner exists if B is a square matrix (i.e., b = |E|) and give a low-complexity
algorithm which finds a C-barycentric spanner for C > 1. We use their technique to show that a
1-barycentric spanner also exists in case of a non-square B, when the basis is chosen to maximize
the absolute value of the determinant of BBT . As before, b denotes the maximum number of
linearly independent vectors (paths) in P.
Lemma 6 For a directed acyclic graph, the set of paths P between two dedicated nodes has a 1-
barycentric spanner. Moreover, let B be a b×|E| matrix with rows from P such that det[BBT ] 6= 0.
If B−j,i is the matrix obtained from B by replacing its jth row by i ∈ P and
∣∣det
B−j,iB
]∣∣ ≤ C2
∣∣det
]∣∣ (23)
for all j = 1, . . . , b and i ∈ P, then B is a C-barycentric spanner.
Proof. Let B be a basis of P with rows b1, . . . , bb ∈ P that maximizes |det[BBT ]|. Then, for
any path i ∈ P, we have i =
j=1 α
(i,B)
j for some coefficients {α(i,B)
}. Now for the matrix
B−1,i = [i
T , (b2)T , . . . , (bb)T ]T we have
∣∣det
B−1,iB
∣∣∣det
B−1,ii
T ,B−1,i(b
2)T ,B−1,i(b
3)T , . . . ,B−1,i(b
∣∣∣∣∣∣∣
(i,B)
B−1,ib
,B−1,i(b
2)T ,B−1,i(b
3)T , . . . ,B−1,i(b
∣∣∣∣∣∣∣
∣∣∣∣∣∣
(i,B)
B−1,i(b
j)T ,B−1,i(b
2)T ,B−1,i(b
3)T , . . . ,B−1,i(b
∣∣∣∣∣∣
= |α(i,B)
∣∣det
B−1,iB
(i,B)
)2 ∣∣det
where last equality follows by the same argument the penultimate equality was obtained. Repeating
the same argument for B−j,i, j = 2, . . . , b we obtain
∣∣det
B−j,iB
]∣∣ =
(i,B)
)2 ∣∣det
]∣∣ . (24)
Thus the maximal property of |det[BBT ]| implies |α(i,B)
| ≤ 1 for all j = 1, . . . , b. The second
statement follows trivially from (23) and (24). ✷
Awerbuch and Kleinberg [4] also present an iterative algorithm to find a C-barycentric spanner
if B is a square matrix. Starting from the identity matrix, their algorithm replaces a row of the
matrix in each step by maximizing the determinant with respect to the given row. This is done by
calling an oracle function, and it is shown that the oracle is called O(b logC b) times. In case B is
not a square matrix, the algorithm carries over if we have access to an alternative oracle that can
maximize |det[BBT ]|: Starting from an arbitrary basis B we can iteratively replace one row in
each step, using the oracle, to maximize the determinant |det[BBT ]| until (23) is satisfied for all j
and i. By Lemma 6, this results in a C-barycentric spanner. Similarly to [4], it can be shown that
the oracle is called O(b logC b) times for C > 1.
For simplicity (to avoid carrying the constant C), assume that we have a 2-barycentric spanner
B. Based on the ideas of label efficient prediction, the next algorithm gives a simple solution to
the restricted shortest path problem. The algorithm is very similar to that of the algorithm in the
label efficient case, but here we cannot estimate the edge losses directly. Therefore, we query the
loss of a (random) basis vector from time to time, and create unbiased estimates ℓ̃
bj ,t of the losses
of basis paths ℓ
bj ,t, which are then transformed into edge-loss estimates.
A BANDIT ALGORITHM FOR THE RESTRICTED SHORTEST PATH
PROBLEM
Parameters: 0 < ǫ, η ≤ 1.
Initialization: Set we,0 = 1 for each e ∈ E, wi,0 = 1 for each i ∈ P, W 0 = N . Fix a
basis B, which is a 2-barycentric spanner. For each round t = 1, 2, . . .
(a) Draw a Bernoulli random variable St such that P(St = 1) = ǫ;
(b) If St = 1, then choose the path It uniformly from the basis B. If St = 0, then
choose It according to the distribution {pi,t}, defined by
pi,t =
wi,t−1
W t−1
(c) Calculate the estimated loss of all edges according to
t = B
where ℓ̃
t = {ℓ̃
e,t }e∈E , and ℓ̃
t = (ℓ̃
, . . . , ℓ̃
) is the vector of the estimated
losses
bj ,t =
bj ,t1{It=b
for j = 1, . . . , b.
(d) Compute the updated weights
we,t = we,t−1e
−ηℓ̃e,t ,
wi,t =
we,t = wi,t−1e
e∈i ℓ̃e,t ,
and the sum of the total weights of the paths
W t =
wi,t .
The performance of the algorithm is analyzed in the next theorem. The proof follows the argu-
ment of Cesa-Bianchi et al. [9], but we also have to deal with some additional technical difficulties.
Note that in the theorem we do not assume that all paths between u and v have equal length.
Theorem 5 Let K denote the length of the longest path in the graph. For any δ ∈ (0, 1), parameters
0 < ǫ ≤ 1
and η > 0 satisfying η ≤ ǫ2, and n ≥ 8b
ln 4bN
, the performance of the algorithm defined
above can be bounded, with probability at least 1− δ, as
L̂n −min
Li,n ≤ K
+ nǫ+
2nǫ ln 4
In particular, choosing
and η = ǫ2
we obtain
L̂n −min
Li,n ≤ 9.1K2b (Kb ln(4bN/δ))1/3 n2/3 .
The theorem is proved using the following two lemmas. The first one is an easy consequence of
Bernstein’s inequality:
Lemma 7 Under the assumptions of Theorem 5, the probability that the algorithm queries the basis
more than nǫ+
2nǫ ln 4
times is at most δ/4.
Using the estimated loss of a path i ∈ P given in (22), we can estimate the cumulative loss of
i up to time n as
L̃i,n =
ℓ̃i,t .
The next lemma demonstrates the quality of these estimates.
Lemma 8 Let 0 < δ < 1 and assume n ≥ 8b
ln 4bN
. For any i ∈ P, with probability at least
1− δ/4,
pi,tℓi,t −
pi,tℓ̃i,t ≤
Furthermore, with probability at least 1− δ/(4N),
L̃i,n − Li,n ≤
Proof. We may write
pi,tℓi,t −
pi,tℓ̃i,t =
(i,B)
bj ,t − ℓ̃bj ,t
pi,tα
(i,B)
bj ,t − ℓ̃bj ,t
bj ,t . (25)
Note that for any bj, X
bj ,t, t = 1, 2, . . . is a martingale difference sequence with respect to (It, St),
t = 1, 2, . . . as Etℓ̃b,t = ℓb,t. Also,
bj ,t
pi,tα
(i,B)
bj ,t
(i,B)
)2 K2b
bj ,t| ≤
∣∣∣∣∣
pi,tα
(i,B)
∣∣∣∣∣
∣∣∣ℓbj ,t − ℓ̃bj ,t
∣∣∣ ≤
∣∣∣α(i,B)
where the last inequalities in both cases follow from the fact that B is a 2-barycentric spanner.
Then, using Bernstein’s inequality for martingale differences (Lemma 9), we have, for any fixed bj ,
bj ,t ≥
where we used (26), (27) and the assumption of the lemma on n. The proof of the first statement
is finished with an application of the union bound and its combination with (25).
For the second statement we use a similar argument, that is,
(ℓ̃i,t − ℓi,t) =
(i,B)
bj ,t − ℓbj ,t) ≤
∣∣∣α(i,B)
∣∣∣∣∣
bj ,t − ℓbj ,t)
∣∣∣∣∣
∣∣∣∣∣
bj ,t − ℓbj ,t)
∣∣∣∣∣ . (29)
Now applying Lemma 9 for a fixed bj we get
bj ,t − ℓbj ,t) ≥
because of Et[(ℓ̃bj ,t − ℓbj ,t)2] ≤
and −K ≤ ℓ̃
bj ,t − ℓbj ,t ≤ K
. The proof is completed by
applying the union bound to (30) and combining the result with (29). ✷
Proof of Theorem 5. Similarly to earlier proofs, we follow the evolution of the term ln Wn
the same way as we obtained (5) and (7), we have
≥ −ηmin
L̃i,n − lnN
pi,tℓ̃i,t +
pi,tℓ̃
Combining these bounds, we obtain
L̃i,n −
pi,tℓ̃i,t +
pi,tℓ̃
−1 + ηKb
pi,tℓ̃i,t ,
because 0 ≤ ℓ̃i,t ≤ 2Kbǫ . Applying the results of Lemma 8 and the union bound, we have, with
probability 1− δ/2,
Li,n −
−1 + ηKb
)( n∑
pi,tℓi,t −
pi,tℓi,t +
. (31)
Introduce the sets
= {t : 1 ≤ t ≤ n and St = 0} and T n
= {t : 1 ≤ t ≤ n and St = 1}
of “exploitation” and “exploration” steps, respectively. Then, by the Hoeffding-Azuma inequality
[21] we obtain that, with probability at least 1− δ/4,
pi,tℓi,t ≥
ℓIt,t −
|Tn|K2
Note that for the exploration steps t ∈ T n, as the algorithm plays according to a uniform distribu-
tion instead of pi,t, we can only use the trivial lower bound zero on the losses, that is,
t∈T n
pi,tℓi,t ≥
t∈T n
ℓIt,t −K|T n| .
The last two inequalities imply
pi,tℓi,t ≥ L̂n −
|Tn|K2
−K|T n| . (32)
Then, by (31), (32) and Lemma 7 we obtain, with probability at least 1− δ,
L̂n −min
+ nǫ+
2nǫ ln 4
where we used L̂n ≤ Kn and |Tn| ≤ n. Substituting the values of ǫ and η gives
L̂n −min
Li,n ≤ K2bnǫ+
Knǫ+Knǫ+
Knǫ+ nǫ
≤ 9.1K2bnǫ
where we used
2nǫ ln 4
ln 4N
= nǫ, and lnN
≤ nǫ (from the
assumptions of the theorem). ✷
8 Simulation results
To further investigate our new algorithms, we have conducted some simple simulations. As the main
motivation of this work is to improve earlier algorithms in case the number of paths is exponentially
large in the number of edges, we tested the algorithms on the small graph shown in Figure 1 (b),
which has one of the simplest structures with exponentially many (namely 2|E|/2) paths.
The losses on the edges were generated by a sequence of independent and uniform random
variables, with values from [0, 1] on the upper edges, and from [0.32, 1] on the lower edges, resulting
in a (long-term) optimal path consisting of the upper edges. We ran the tests for n = 10000 steps,
with confidence value δ = 0.001. To establish baseline performance, we also tested the EXP3
algorithm of Auer et al. [1] (note that this algorithm does not need edge losses, only the loss of the
chosen path). For the version of our bandit algorithm that is informed of the individual edge losses
(edge-bandit), we used the simple 2-element cover set of the paths consisting of the upper and
lower edges, respectively (other 2-element cover sets give similar performance). For our restricted
shortest path algorithm (path-bandit) the basis {uuuuu, uuuul, uuull, uulll, ullll} was used, where
u (resp. l) in the kth position denotes the upper (resp. lower) edge connecting vk−1 and vk. In
this example the performance of the algorithm appeared to be independent of the actual choice of
the basis; however, in general we do not expect this behavior. Two versions of the algorithm of
Awerbuch and Kleinberg [4] were also simulated. With its original parameter setting (AwKl), the
algorithm did not perform well. However, after optimizing its parameters off-line (AwKl tuned),
substantially better performance was achieved. The normalized regret of the above algorithms,
averaged over 30 runs, as well as the regret of the fixed paths in the graph are shown in Figure 2.
Although all algorithms showed better performance than the bound for the edge-bandit algo-
rithm, the latter showed the expected superior performance in the simulations. Furthermore, our
algorithm for the restricted shortest path problem outperformed Awerbuch and Kleinberg’s (AwKl)
algorithm, while being inferior to its off-line tuned version (AwKl tuned). It must be noted that
similar parameter optimization did not improve the performance of our path-bandit algorithm,
which showed robust behavior with respect to parameter tuning.
9 Conclusions
We considered different versions of the on-line shortest path problem with limited feedback. These
problems are motivated by realistic scenarios, such as routing in communication networks, where
the vertices do not have all the information about the state of the network. We have addressed the
problem in the adversarial setting where the edge losses may vary in an arbitrary way; in particular,
they may depend on previous routing decisions of the algorithm. Although this assumption may
neglect natural correlation in the loss sequence, it suits applications in mobile ad-hoc networks,
where the network topology changes dynamically in time, and also in certain secure networks that
has to be able to handle denial of service attacks.
Efficient algorithms have been provided for the multi-armed bandit setting and in a combined
label efficient multi-armed bandit setting, provided the individual edge losses along the chosen
path are revealed to the algorithms. The normalized regrets of the algorithms, compared to the
performance of the best fixed path, converge to zero at an O(1/
n) rate as the time horizon n
grows to infinity, and increases only polynomially in the number of edges (and vertices) of the
graph. Earlier methods for the multi-armed bandit problem either do not have the right O(1/
convergence rate, or their regret increase exponentially in the number of edges for typical graphs.
 0  2000  4000  6000  8000  10000
Number of packets
edge-bandit
path-bandit
AwKl tuned
bound for edge-bandit
Figure 2: Normalized regret of several algorithms for the shortest path problem. The gray dotted
lines show the normalized regret of fixed paths in the graph.
The algorithm has also been extended so that it can compete with time varying paths, that is, to
handle situations when the best path can change from time to time (for consistency, the number
of changes must be sublinear in n).
In the restricted version of the shortest path problem, where only the losses of the whole paths
are revealed, an algorithm with a worse O(n−1/3) normalized regret was provided. This algorithm
has comparable performance to that of the best earlier algorithm for this problem [4], however,
our algorithm is significantly simpler. Simulation results are also given to assess the practical
performance and compare it to the theoretical bounds as well as other competing algorithms.
It should be noted that the results are not entirely satisfactory in the restricted version of the
problem, as it remains an open question whether the O(1/
n) regret can be achieved without the
exponential dependence on the size of the graph. Although we expect that this is the case, we have
not been able to construct an algorithm with such a proven performance bound.
10 Appendix
Lemma 9 (Bernstein’s inequality for martingale differences [10].) Let X1, . . . ,Xn be a martingale
difference sequence such that Xt ∈ [a, b] with probability one (t = 1, . . . , n). Assume that, for all t,
X2t |Xt−1, . . . ,X1
≤ σ2 a.s.
Then, for all ǫ > 0,
Xt > ǫ
2nσ2+2ǫ(b−a)/3
and therefore
2nσ2 ln δ−1 + 2 ln δ−1(b− a)/3
References
[1] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. Schapire. The non-stochastic multi-armed bandit
problem. SIAM Journal on Computing, 32(1):48–77, 2002.
[2] P. Auer and M. K. Warmuth, “Tracking the best disjunction,” Machine Learning, vol. 32,
no. 2, pp. 127–150, 1998.
[3] B. Awerbuch, D. Holmer, H. Rubens, and R. Kleinberg. Provably competitive adaptive routing.
In Proceedings of IEEE INFOCOM 2005, volume 1, pages 631–641, March 2005.
[4] B. Awerbuch and R. D. Kleinberg. Adaptive routing with end-to-end feedback: distributed
learning and geometric approaches. In Proceedings of the 36th Annual ACM Symposium on the
Theory of Computing, STOC 2004, pages 45–53, Chicago, IL, USA, Jun. 2004. ACM Press.
[5] D. Blackwell. An analog of the minimax theorem for vector payoffs. Pacific Journal of Math-
ematics, 6:1–8, 1956.
[6] O. Bousquet and M. K. Warmuth. Tracking a small set of experts by mixing past posteriors.
Journal of Machine Learning Research, 3:363–396, Nov. 2002.
[7] N. Cesa-Bianchi, Y. Freund, D. P. Helmbold, D. Haussler, R. Schapire, and M. K. Warmuth.
How to use expert advice. Journal of the ACM, 44(3):427–485, 1997.
[8] N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University
Press, Cambridge, 2006.
[9] N. Cesa-Bianchi, G. Lugosi, and G. Stoltz. Minimizing regret with label efficient prediction.
IEEE Trans. Inform. Theory, IT-51:2152–2162, June 2005.
[10] D. A. Freedman. On tail probabilities for martingales. Annals of Probability, 3:100–118, Feb.
1975.
[11] E. Gelenbe, M. Gellman, R. Lent, P. Liu, and P. Su. Autonomous smart routing for network
QoS. In Proceedings of First International Conference on Autonomic Computing, pages 232–
239, New York, May 2004. IEEE Computer Society.
[12] E. Gelenbe, R. Lent, and Z. Xhu. Measurement and performance of a cognitive packet network.
Journal of Computer Networks, 37:691–701, 2001.
[13] A. György, T. Linder, and G. Lugosi. Efficient algorithms and minimax bounds for zero-delay
lossy source coding. IEEE Transactions on Signal Processing, 52:2337–2347, Aug. 2004.
[14] A. György, T. Linder, and G. Lugosi. A ”follow the perturbed leader”-type algorithm for
zero-delay quantization of individual sequences. In Proc. Data Compression Conference, pages
342–351, Snowbird, UT, USA, Mar. 2004.
[15] A. György, T. Linder, and G. Lugosi. Tracking the best of many experts. In Proceedings of
the 18th Annual Conference on Learning Theory, COLT 2005, pages 204–216, Bertinoro, Italy,
Jun. 2005. Springer.
[16] A. György, T. Linder, and G. Lugosi. Tracking the best quantizer. In Proceedings of the IEEE
International Symposium on Information Theory, pages 1163–1167, Adelaide, Australia, June-
July 2005.
[17] A. György and Gy. Ottucsák. Adaptive routing using expert advice. The Computer Journal,
49(2):180–189, 2006.
[18] J. Hannan. Approximation to Bayes risk in repeated plays. In M. Dresher, A. Tucker, and
P. Wolfe, editors, Contributions to the Theory of Games, volume 3, pages 97–139. Princeton
University Press, 1957.
[19] M. Herbster and M. K. Warmuth. Tracking the best expert. Machine Learning, 32(2):151–178,
1998.
[20] M. Herbster and M. K. Warmuth, “Tracking the best linear predictor,” Journal of Machine
Learning Research, vol. 1, pp. 281–309, 2001.
[21] W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the
American Statistical Association, 58:13–30, 1963.
[22] D.P. Helmbold and S. Panizza. Some label efficient learning results. In Proceedings of the 10th
Annual Conference on Computational Learning Theory, pages 218–230. ACM Press, 1997.
[23] A. Kalai and S Vempala. Efficient algorithms for the online decision problem. In B. Schölkopf
and M. Warmuth, editors, Proceedings of the 16th Annual Conference on Learning Theory
and the 7th Kernel Workshop, COLT-Kernel 2003, pages 26–40, New York, USA, Aug. 2003.
Springer.
[24] N. Littlestone and M. K. Warmuth. The weighted majority algorithm. Information and
Computation, 108:212–261, 1994.
[25] H. B. McMahan and A. Blum. Online geometric optimization in the bandit setting against an
adaptive adversary. In Proceedings of the 17th Annual Conference on Learning Theory, COLT
2004, pages 109–123, Banff, Canada, Jul. 2004. Springer.
[26] M. Mohri. General algebraic frameworks and algorithms for shortest distance problems. Tech-
nical Report 981219-10TM, AT&T Labs Research, 1998.
[27] R. E. Schapire and D. P. Helmbold. Predicting nearly as well as the best pruning of a decision
tree. Machine Learning, 27:51–68, 1997.
[28] E. Takimoto and M. K. Warmuth. Path kernels and multiplicative updates. Journal of Machine
Learning Research, 4:773–818, 2003.
[29] V. Vovk. Aggregating strategies. In Proceedings of the Third Annual Workshop on Computa-
tional Learning Theory, pages 372–383, Rochester, NY, Aug. 1990. Morgan Kaufmann.
[30] V. Vovk. Derandomizing stochastic prediction strategies. Machine Learning, 35(3):247–282,
Jun. 1999.
	Introduction
	The shortest path problem
	The multi-armed bandit setting
	A bandit algorithm for shortest paths
	A combination of the label efficient and bandit settings
	A bandit algorithm for tracking the shortest path
	An algorithm for the restricted multi-armed bandit problem
	Simulation results
	Conclusions
	Appendix
