Sequential Information Design: Learning to Persuade in the Dark
Abstract
We study a repeated information design problem faced by an informed sender who tries to inﬂuence the behavior of a self-interested receiver .
We consider settings where the receiver faces a sequential decision making (SDM) problem.
At each round, the sender observes the realizations of random events in the SDM problem.
This begets the challenge of how to incrementally disclose such information to the receiver to persuade them to follow (desirable) action recommendations.
We study the case in which the sender does not know random events probabilities, and, thus, they have to gradually learn them while persuading the receiver.
We start by providing a non-trivial polytopal approximation of the set of sender’s persuasive information structures.
This is crucial to design efﬁcient learning algorithms.
Next, we prove a negative result: no learning algorithm can be persuasive.
Thus, we relax persuasiveness requirements by focusing on algorithms that guarantee that the receiver’s regret in following recommendations grows sub-linearly.
In the full-feedback setting—where the sender observes all random events realizations—, we provide an algorithm with ˜ O ( p T ) regret for both the sender and the receiver.
Instead, in the bandit-feedback setting—where the sender only observes the realizations of random events actually occurring in the SDM problem—, we design an algorithm that, given an ↵ 2 [1 / 2 , 1] as input, ensures ˜ O
( T ↵ ) and ˜ O
( T max { ↵ , 1  ↵ 2 } ) regrets, for the sender and the receiver respectively.
This result is complemented by a lower bound showing that such a regrets trade-off is essentially tight.
Introduction
Bayesian persuasion [Kamenica and Gentzkow, 2011] (a.k.a. information design ) is the problem faced by an informed sender who wants to inﬂuence the behavior of a self-interested receiver via the provision of payoff-relevant information.
This captures the problem of “who gets to know what”, which is fundamental in all economic interactions.
Thus, Bayesian persuasion is ubiquitous in realworld problems, such as, e.g. , online advertising [Bro Miltersen and Sheffet, 2012], voting [Alonso and Câmara, 2016, Castiglioni et al., 2020a, Castiglioni and Gatti, 2021], trafﬁc routing
[Bhaskar et al., 2016, Castiglioni et al., 2021a], security [Rabinovich et al., 2015, Xu et al., 2016], and marketing [Babichenko and Barman, 2017, Candogan, 2019].
We study Bayesian persuasion in settings where the receiver plays in a sequential decision making (SDM) problem.
An SDM problem is characterized by a tree structure made by: decision nodes, where the receiver takes actions, and chance nodes, in which partially observable random events occur.
The sender perfectly observes the realizations of random events, and their goal is to incrementally disclose the acquired information to induce the receiver towards desirable outcomes for the sender.
This is not an easy feat when the sender and the receiver have different utilities.
In order to do so, the sender commits to a signaling scheme specifying a probability distribution over action recommendations for the receiver at each decision node.
Speciﬁcally, the sender commits to a persuasive signaling scheme, meaning that the receiver is incentivized to follow recommendations.
We consider the case of a farsighted receiver, meaning that they take into account all the possible future events when deciding whether to deviate or not from recommendations at each decision node.
With some notable exceptions (such as, e.g. , [Zu et al., 2021]), Bayesian persuasion models in the literature make the stringent assumption that both the sender and the receiver know the prior , which, in our setting, is deﬁned by the probabilities associated with random events in the SDM problem.
We relax such an assumption by considering an online learning framework in which the sender, without any knowledge of the prior, repeatedly interacts with the receiver to gradually learn the prior while still being persuasive.
A concrete example of a setting that ﬁts our model is that of a navigation app sending route recommendations to a driver.
The app (sender) gets to know information about trafﬁc congestion sequentially, since it might change over time, especially for long routes.
Moreover, at any point in time, the app can send recommendations to the driver (receiver), who in turn has to take decisions on which routes to choose sequentially.
The app and the driver might have different utilities.
For example, the app may want to maximize overall trafﬁc congestion, while the driver has to minimize their travel time.
Moreover, the receiver interacts with the app multiple times and, thus, the application must provide good recommendations to the receiver, otherwise they would switch to another navigation app.
Original contributions.
Our goal is to design online learning algorithms that are no-regret for the sender, while being persuasive for the receiver.
We start by providing a non-trivial polytopal approximation of the set of sender’s persuasive signaling schemes.
This will be crucial in designing efﬁcient ( i.e. , polynomial-time) learning algorithms, and it also shows how a sender-optimal signaling scheme can be found in polynomial time in the ofﬂine version of our problem, which may be of independent interest.
Next, we prove a negative result: without knowing the prior, no algorithm can be persuasive at each round with high probability.
Thus, we relax persuasiveness requirements by focusing on learning algorithms that guarantee that the receiver’s regret in following recommendations grows sub-linearly, while guaranteeing the same for sender’s regret.
First, we study the full-feedback case, where the sender observes the realizations of all the random events that may potentially happen in the SDM problem.
In such a setting, we provide an algorithm with ˜ O ( p T ) regret for both the sender and the receiver.
Then, we focus on the bandit-feedback setting, where the sender only observes the realizations of random events on the path in the tree traversed during the SDM problem.
In this case, we design an algorithm that achieves
˜ O
( T ↵ ) sender’s regret and ˜ O
( T max { ↵ , 1  ↵ 2 } ) receiver’s regret, for any ↵ 2 [1 / 2 , 1] given as input.
The crucial component of the algorithm is a non-trivial exploration phase that uniformly explores the tree deﬁning the SDM problem to build suitable estimators of the prior.
This is needed since, with bandit feedback, playing a signaling scheme may provide insufﬁcient information about its persuasiveness.
Finally, we provide a lower bound showing that the regrets trade off achieved by our algorithm is tight for ↵ 2 [1 / 2 , 2 / 3] .
Related works.
Some works addressed Bayesian persuasion in Markov decision processes (MDPs).
Gan et al.
[2022] and Wu et al.
[2022] show how to efﬁciently ﬁnd a sender-optimal policy when the receiver is myopic ( i.e. , it only optimizes one-step rewards) in MDPs with inﬁnite and ﬁnite horizon, respectively.
Moreover, the former assume that the environment is known, while the latter do not .
These works considerably differ from ours, since we assume a farsighted receiver and also model partial observability of random events.
Another work close to ours is [Zu et al., 2021], which studies a (non-sequential) persuasion problem in which the sender and the receiver do not know the prior and interact online.
Zu et al.
[2021] provide a persuasive learning algorithm, while, in our model, we show that the ignorance of the prior precludes the possibility of committing to persuasive signaling schemes, and, thus, we need to resort to new techniques to circumvent the issue.
Another line of research, that uses similar techniques as the one employed in this work, studies learning in SDM problems while satisfying unknown constraints [Bernasconi-de-Luca et al., 2021, Bernasconi et al., 2022].
Finally, Celli et al.
[2020a] study Bayesian persuasion with multiple receivers interacting in an study a model with farsighted
, where they show that the problem of ﬁnding a such imperfect-information sequential game.
Differently from ours, their model adopts a different notion of persuasiveness, known as ex ante persuasiveness, and it assumes that the prior is known.
Other works study learning problems in which the sender does not know the receivers’ payoffs (but knows the prior); see, e.g. , [Castiglioni et al., 2020b, 2021b, 2022].
Preliminaries 2.1 Sequential decision making problems
An instance of an SDM problem is deﬁned by a tree structure, utilities, and random events probabilities.
The tree structure has a set of nodes H :
= Z [ H d [ H c , where: Z contains all the terminal nodes in which the problem ends (corresponding to the leaves of the tree), H d is the set of decision nodes in which the agent acts, while H c is the set of chance nodes where random events occur.
Given any non-terminal node h 2 H \ Z , we let A ( h ) be the set of arcs outgoing from h .
If h 2 H d , then A ( h ) is the set of receiver’s actions available at h , while, if h 2 H c , then A ( h ) encodes the possible outcomes of the random event occurring at h .
Furthermore, the utility function u : Z ! [0 , 1] deﬁnes the agent’s payoff u ( z ) when the problem ends in terminal node z 2 Z .
Finally, each chance node h 2 H c is characterized by a probability distribution µ h 2  A ( h ) over the possible outcomes of the corresponding random event, with µ h ( a ) denoting the probability of action a 2 A ( h ) .
In an SDM problem, the agent has imperfect information , since they do not perfectly observe the outcomes of random events.
Thus, the set of decision nodes H d is partitioned into information sets (infosets for short), where an infoset I ✓ H d is a subset of decision nodes that are indistinguishable for the agent.
We denote the set of infosets as I .
For every infoset I 2 I and pair of nodes h, h [-(cid:
-]{+0 
+}
I , it must be the case that A ( h ) =
A ( h 0 )
=:
A ( I ) , otherwise the agent could distinguish between the two nodes.
We assume that the agent has perfect recall , which means that they never forget information once acquired.
Formally, this is equivalent to assume that, for every infoset I 2 I , all the paths from the root of the tree to a node h {+
+}
I identify the same ordered sequence of agent’s actions.
2.2 Bayesian persuasion in sequential decision making problems
We study Bayesian persuasion in SDM (BPSDM) problems.
These extend the Bayesian persuasion framework [Kamenica and Gentzkow, 2011] to SDM problems by introducing an exogenous agent that acts as a sender by issuing signals to the decision-making agent (the receiver ).
By following the Bayesian persuasion terminology, the probability distributions µ h for each chance node h are collectively referred to as the prior .
Thus, the sender observes the realizations of random events occurring in the SDM problem and can partially disclose information to inﬂuence the receiver’s behavior.
Moreover, the sender has their own utility function deﬁned over terminal nodes, denoted as f : Z ! [0 , 1] , and their goal is to commit to a publicly known signaling scheme that maximizes their utility in expectation with respect to the prior, the signaling scheme, and the receiver’s strategy.
Formally, a signaling scheme for the sender deﬁnes a probability distribution  h 2  S ( h ) at each decision node h 2 H d , where S ( h ) is a ﬁnite set of signals available at h .
During the SDM problem, when the receiver reaches a node h 2 H d belonging to an infoset I 2 I , the sender draws a signal s ⇠  h and communicates it to the receiver.
Then, based on the history of signals observed from the beginning of the SDM problem ( s included), the receiver computes a posterior belief over the nodes belonging to the infoset I and plays so as to maximize their expected utility in the SDM sub-problem that starts from I , taking into account the just acquired information.
As customary in these settings, a simple revelation-principle-style argument allows us to focus on signaling schemes that are direct and persuasive [Kamenica and Gentzkow, 2011, Arieli and
Babichenko, 2019].
In particular, a signaling scheme is direct if signals correspond to action recommendations, namely S ( h ) =
A ( h ) for all h 2 H d .
A direct signaling scheme is persuasive if the receiver is incentivized to follow action recommendations issued by the sender.
Moreover, we assume that, if the receiver does not follow action recommendations at some decision node, then the sender stops issuing recommendations at nodes later reached during the SDM problem.
This is without loss of generality.
We refer to [Von Stengel and Forges, 2008, Morrill et al., 2021] for a discussion on a similar problem in the ﬁeld of correlation in sequential games.
2.3 The sequence-form representation
The sequence form is a commonly-used, compact way of representing (mixed) strategies in SDM problems [Koller et al., 1996].
In this work, the sequence-form representation will be employed for receiver’s strategies, and to encode the signaling schemes and priors, as we describe in the following.
Receiver’s strategies.
Given any h 2 H , we let  r ( h ) be the ordered sequence of receiver’s actions on the path from the root of the tree to node h .
By the perfect recall assumption, given any infoset
I 2 I , it holds that  r ( h ) =
r ( h 0 )
=:
r ( I ) for every pair of nodes h, h 0 2 I .
Thus, we can identify sequences with infoset-action pairs, with
= ( I, a ) encoding the sequence of actions obtained by appending action a 2 A ( I ) at the end of  r ( I ) , for any infoset I 2 I .
Moreover, ? denotes the empty sequence .
Hence, the receiver’s sequences are ⌃ r :=
{ ( I, a ) | I 2 I , a 2 A ( I ) }
In the sequence-form representation, mixed strategies are deﬁned by specifying the probability of playing each sequence of actions.
Thus, a receiver’s strategy is represented by a vector x 2 [0 , 1] | ⌃ r | , where x [  ] encodes the realization probability of sequence  2 ⌃ r .
Furthermore, a sequence-form strategy is well-deﬁned if and only if it satisﬁes the following linear constraints:
[Equation]
We denote by X r the polytope of all receiver’s sequence-form strategies.
We will also need to work with the sets of receiver’s strategies in the SDM sub-problem that starts from an infoset I 2 I , formally deﬁned as X r,
I := { x 2 X r | x [  r ( I )]
= 1 } .
Signaling schemes.
We represent signaling schemes in sequence form by leveraging the fact that the sender can be thought of as a perfect-information agent who plays at the decision nodes of the SDM problem, since their actions correspond to recommendations for the receiver.
Thus, since sender’s infosets correspond to decision nodes, their sequences ⌃ s :=
{ ( h, a ) | h 2 H d , a 2 A ( h ) } [ { ? } .
Then, we denote the polytope of (sequence-form) signaling schemes as  ✓ [0 , 1] | ⌃ s | , where each signaling scheme is represented as a vector  2 [0 , 1] | ⌃ s | satisfying:
[Equation]
where, similarly to  r ( h ) for the receiver,  s ( h ) denotes the sender’s sequence identiﬁed by h 2 H .
We also deﬁne ⇧ :=
\ { 0 , 1 } | ⌃ s | as the set of deterministic signaling schemes, which are those that recommend a single action with probability one at each decision node.
Priors.
We also encode prior probability distributions µ h by means of the sequence form.
Indeed, these can be though of as elements of a ﬁxed strategy played by a (ﬁctitious) perfect-information agent that acts at chance nodes.
Thus, for such a chance agent, we deﬁne ⌃ c , X c , and  c ( h ) as their counterparts previously introduced for the receiver.
Moreover, in the following, we denote by µ ?
2 X c the (sequence-form) prior, recursively deﬁned as follows:
[Equation]
Ordering of sequences.
For the sake of presentation, we introduce a partial ordering relation among sequences.
Given two sequences  = ( I, a ) 2 ⌃ r and  0 =
( J, b ) 2 ⌃ r , we write   (read as  precedes  0 ), whenever there exists a path in the tree connecting a node in I to a node in J , and such a path includes action a .
We adopt analogous deﬁnitions for sequences in ⌃ s and ⌃ c .
Learning to persuade
In this work, we relax the strong assumption that both the sender and the receiver know the prior µ ? by casting the BPSDM problem into an online learning framework in which the sender repeatedly interacts with the receiver over a time horizon of length T .
At each round t 2 [ T ] , the interaction goes as follows: 6 (i) the sender commits to a signaling scheme  t 2  ; (ii) a vector y t 2 { 0 , 1 } | ⌃ c | encoding realizations of random events is drawn according to µ ? ; (iii) the sender and the receiver play an instance of the (one-shot) BPSDM problem (detailed in Section 2.2), in which the sender commits to  t , random events at chance nodes are realized as deﬁned by y t , and the receiver sticks to the recommendations issued by the sender; and (iv) the sender observes a feedback on the realization of random events at chance nodes, which can be of two types: full feedback when the sender observes y t , which speciﬁes the realizations of all the random events at chance nodes that are possibly reachable during the SDM problem; bandit feedback when the sender observes the terminal node z t 2 Z reached at the end of the SDM problem.
The latter is equivalent to observing the realizations of random events at the chance nodes that are actually reached during the SDM problem, namely  c ( z t ) .
By letting  ⇧ ( µ ? ) be the set of persuasive signaling schemes, i.e. , such that the receiver is incentivized to following recommendations (a formal deﬁnition is provided in Deﬁnition 2), the goal of the sender is to select a sequence of signaling schemes, namely  1 , . . .
,  T , which maximizes their expected utility, while guaranteeing that each signaling scheme  t is persuasive, namely  t 2  ⇧ ( µ ? ) .
We measure the performance of a sequence  1 , . . .
,  T of signaling schemes by comparing it with an optimal (ﬁxed) persuasive signaling scheme.
Formally, given a signaling scheme  2  , we ﬁrst deﬁne U (  , µ ? ) , respectively F (  , µ ? ) , as the expected utility achieved by the receiver, respectively the sender, whenever the former follows action recommendations.
These can be expressed as linear functions of  , which, for any µ 2 X c , are deﬁned as follows:
[Equation]
Finally, by letting  ? 2 argmax  2  ⇧ ( µ ? ) F (  , µ ? ) be an optimal (ﬁxed) persuasive signaling scheme, the sender’ performance over T rounds is measured by the (cumulative) sender’s regret :
[Equation]
The goal is to design learning algorithms (for the sender) which select sequences of persuasive signaling schemes such that R T grows asymptotically sub-linearly in T , namely R T
= o ( T ) .
On the characterization of persuasive signaling schemes 4.1 A local decomposition of persuasiveness
In this section, we formally introduce the set of persuasive signaling schemes  ⇧ ( µ ? ) as the set of signaling schemes for which the receiver’s expected utility by following recommendations is greater than the one provided by an optimal deviation policy (DP).
In addition, we show how to decompose any DP into components deﬁned locally at each infoset, which will be crucial in the following Section 4.2.
Intuitively, a DP for the receiver is speciﬁed by two elements: (i) a set of deviation points in which the DP prescribes to stop following action recommendations; and (ii) the continuation strategies to be adopted after deviating from recommendations.
We represent deviation points by vectors ! 2 { 0 , 1 } | ⌃ r | , which are deﬁned so that ! [  ] = 1 if and only if the DP prescribes to deviate upon observing the sequence of action recommendations  2 ⌃ r .
Moreover, by leveraging the w.l.o.g. assumption that the sender stops issuing recommendations after the receiver deviated from them, we focus on DPs such that each path from the root of the tree to a terminal node involves only one deviation point.
As a result, the set of all valid vectors ! 2 { 0 , 1 } | ⌃ r | is formally deﬁned as ⌦ :=
[Equation]
We represent the continuation strategies of DPs by introducing the set of continuation strategy proﬁles , denoted as P := ⇥
[Equation]
=
[Equation]
I,
[Equation]
⌃ [Equation]
X r,I .
A continuation strategy proﬁle ⇢ 2 P , with ⇢ = ( ⇢  )  2 ⌃ r , deﬁnes a strategy ⇢  2 X r, I for every receiver’s sequence  =
( I, a ) 2 ⌃ r .
Intuitively, ⇢  is the strategy for the SDM sub-problem starting from infoset I that is used by the receiver after deviating upon observing sequence  2 ⌃ r .
As a result, any pair ( ! , ⇢ ) 2 ⌦ ⇥ P speciﬁes a valid DP; formally:
Deﬁnition 1 (Deviation policy) .
Given a vector ! 2 ⌦ and a proﬁle ⇢ 2 P , the ( ! , ⇢ ) -DP prescribes to follow sender’s recommendations until action a is recommended at infoset I for some sequence  =
( I, a ) such that ! [  ] = 1 ; from that point on, it prescribes to play according to strategy ⇢  .
We denote by U ! !
⇢ (  , µ ? )
the receiver’s expected utility obtained with a ( ! , ⇢ ) -DP, so that we can state the following formal deﬁnition of persuasive signaling schemes.
Deﬁnition 2 (Persuasiveness) .
A signaling scheme  2  is ✏ -persuasive , namely  2  ⇧ ✏ ( µ ? ) , if
[Equation]
Moreover, a signaling scheme  2  is persuasive , namely  2  ⇧ ( µ ? ) , if it is 0 -persuasive.
Intuitively, the above deﬁnition states that a signaling scheme is ✏ -persuasive if the receiver’s expected utility by following recommendations is at most ✏ less than the one obtained by an optimal DP, which is a DP maximizing receiver’s expected utility.
Our local decomposition of DPs is based on suitably-deﬁned, simple deviation policies, which we call single-point DPs (SPDPs).
These are a special case of DPs that stop following sender’s action recommendations only when a speciﬁc single infoset is reached and a particular action is recommended therein.
SPDPs are formally deﬁned as follows:
Deﬁnition 3 (Single-point deviation strategy) .
Given a receiver’s sequence  = ( I, a ) 2 ⌃ r and a receiver’s strategy ⇢  2 X r,I for the SDM sub-problem starting from infoset I , the (  , ⇢  ) -SPDP prescribes to follow sender’s recommendations until action a is recommended at infoset I ; from that point on, the strategy prescribes to play according to ⇢  .
We denote by U  ! ⇢  (  , µ ? )
the receiver’s expected utility obtained by following an (  , ⇢  ) -SPDP.
The following theorem provides the key result underlying our decomposition.
It shows that the difference between the utility achieved by a ( ! , ⇢ ) -DP and that obtained by following recommendations can be decomposed into the sum over all the sequences  2 ⌃ r of analogous differences deﬁned for the (  , ⇢  ) -SPDPs, where each difference is weighted by ! [  ] .
Theorem 1.
Given a signaling scheme  2  and a ( ! , ⇢ ) -DP, it holds:
[Equation]
4.2 A polytopal approximation of the set of persuasive signaling schemes
In the following, we show how to exploit Theorem 1 to provide an approximate characterization of the set  ⇧ ✏ ( µ ? ) using a polynomially-sized polytope.
First, we state a corollary of Theoremshowing that persuasiveness can be bounded by suitably deﬁned SPDPs.
Formally:
Corollary 1.
Given a signaling scheme  2  , the following holds:
[Equation]
By exploiting Corollary 1, we introduce the following deﬁnition of ✏ -persuasive polytope (Lemmajustiﬁes the term polytope), as the set of signaling schemes for which there is no (  , ⇢  ) -SPDP that achieves a receiver’s utility that exceeds by more than ✏ / | ⌃ r | that of following recommendations.
Deﬁnition 4 (Persuasive polytope) .
The ✏ -persuasive polytope is deﬁned as:
[Equation]
Moreover, we denote by ⇤ ( µ ? )
the 0 -persuasive polytope.
As we show in the following lemma, ⇤ ✏ ( µ ? ) is an efﬁciently-representable polytope.
Lemma 1.
The set ⇤ ✏ ( µ ? ) can be described by means of a polynomial number of linear constraints.
The following lemma shows that the ✏ -persuasive polytope is contained in  ⇧ ✏ ( µ ? ) , and that the set of persuasive signaling schemes is contained in the former.
Formally:
Lemma 2.
It is always the case that  ⇧ ( µ ? ) ⌘ ⇤ ( µ ? )
✓ ⇤ ✏ ( µ ? ) ✓  ⇧ ✏ ( µ ? ) .
Lemma 2 also implies that the polytope ⇤ ( µ ? ) exactly characterizes the set of persuasive signaling schemes  ⇧ ( µ ? ) .
Thus, by adding the maximization of the sender’s expected utility F (  , µ ? ) on top of the linear constraints describing ⇤ ( µ ? ) , we obtain a polynomially-sized linear program for ﬁnding an optimal sender’s signaling scheme in any instance of the BPSDM problem in which µ ? is known.
Theorem 2.
The BPSDM problem can be solved in polynomial time when the prior µ ? is known.
Always being persuasive is impossible: a relaxation is needed
In this section, we prove that it is impossible to design an algorithm that returns a sequence of persuasive signaling schemes for a generic BPSDM problem.
Theorem 3 (Impossibility of persuasiveness) .
There exists a constant  2 (0 , 1) such that no algorithm can guarantee to output a sequence  1 , . . .
,  T of signaling schemes such that, with probability al least  , all the signaling schemes  t are persuasive.
Notice that this result is in contrast with what happens in non-sequential Bayesian persuasion (see the work by Zu et al.
[2021]), where it is possible to design no-regret algorithms that output sequences of signaling schemes that are guaranteed to be persuasive with high probability.
Theorem 3 motivates the introduction of a less restrictive requirement on the signaling schemes.
In particular, we look for algorithms that output signaling schemes  1 , . . .
,  T , such that the expected utility loss incurred by the receiver by following recommendations rather than playing an optimal DP is small.
To capture such a requirement, we introduce the following deﬁnition of (cumulative) receiver’s regret :
[Equation]
Therefore our goal becomes that of designing algorithms guaranteeing that the cumulative receiver’s regret grows sub-linearly in T , namely V T
= o ( T ) , while continuing to ensure that R T
= o ( T ) .
In Sections 6 and 7, we design algorithms achieving sub-linear V T and R T for the learning problem described in Section 3.
The algorithms implement two functions: (i) S ELECT S TRATEGY () , which, at each t 2 [ T ] , draws a signaling scheme  t 2  on the basis of the internal state of the algorithm; and (ii) U PDATE ( o t ) , which modiﬁes the internal state on the basis of the observation o t received as feedback.
Each algorithm alternates these two functions as the interaction between the sender and the receiver unfolds as described in Section 3.
Speciﬁcally, under full feedback the sender observes y t and calls U PDATE ( y t ) , while in the bandit feedback it observes z t and calls U PDATE ( z t ) .
Learning with full feedback
Algorithm 1 Full-feedback algorithm function S ELECT S TRATEGY ():  t arg max  2 ⇤  t ( b µ t ) F (  , b µ t ) return  t function U PDATE ( y t ): b µ t
+1 [  ] t P ⌧ = y ⌧ [  ] / ⌧ 8  2 ⌃ c ✏ t + q log(2 T | ⌃ c | /  ) 2 t  t + 2 |Z|| ⌃ r | ✏ t +
In this section, we will discuss the online problem faced by the sender that wants to optimize online its utility
F (  , µ ? ) while learning the unknown prior µ ? .
We start by providing a learning algorithm (Algorithm 1) working with full feedback, i.e. , when the sender observes the realizations of all the possible random events.
The main idea of the algorithm is to choose signaling schemes  t that belong to suitable sets ⇤  t ( b µ t ) which are designed to be “close” to the set  ⇧ ( µ ? ) of persuasive signaling schemes.
At each round t 2 [ T ] , Algorithm 1 deﬁnes the desired set as follows.
First, it maintains an estimate b µ t of µ ? ; formally, it deﬁnes a radius ✏ t such that the event E : =
{ k b µ t  µ ?
k 1  ✏ t 8 t 2 [ T ] } holds with probability at least  .
Second, it deﬁnes a parameter  t such that, conditionally to the realization of the event E , the following two conditions hold: (i) the decision space ⇤  t ( b µ t ) contains the optimal signaling scheme  ? ; (ii) ⇤ 2  t ( µ ? ) contains the signaling scheme  t .
Intuitively, the ﬁrst condition is needed to have low sender’s regret, while the second one yields signaling schemes that are approximately persuasive.
The polytopal approximation that we provide in Section 4.2 plays a crucial role in the complexity of
Algorithm 1.
Speciﬁcally, it allows it to select the desired  t in polynomial time by optimizing over the set ⇤  t ( b µ t ) , which can be done efﬁciently.
The use of the set ⇤  t ( b µ t ) over  ⇧  t ( b µ t ) is necessary due to the fact that the latter is not known to admit an efﬁcient representation.
Formally:
Theorem 4.
Given any  2 (0 , 1) , with probability at least 1   , Algorithm 1 guarantees:
[Equation]
Moreover, the algorithm runs in polynomial time.
Learning with bandit feedback
Algorithm 2 Bandit-feedback algorithm function S ELECT S TRATEGY (): if t  N then .
First Phase  =
( h, a ) arg min
2 ⌃ c C t [  ] ⌃ s
3  s ( h ) Choose  t 2  :  t [  0 ]
= 1 else .
Second Phase  t arg max  2 ⇤  N ( b µ N ) max µ 2 C t (  ) F (  , µ ) return  t function U PDATE ( z t ): Build path p t 2 { 0 , 1 } | ⌃ c | from  c ( z t ) Sample ⇡ t ⇠  t s.t. p t [  ] = 1 )  2 ⌃ # ( ⇡ t ) for  2 ⌃ # ( ⇡ t ) do C t +1 [  ]
{+C t [  ] +b µ t +
+}
[  ] 1 C t +1 [  ]
P
C t +1 [  ] ⌧ = p ⌧ [  ] ✏ t +1 [  ] q log(4 T | ⌃ c | /  ) 2 C t +1 [  ]
C t +1 (  ) n µ  | µ [  ] {+ b µ t +
+}
[  ] |  ✏ t +1 [  ] 8  2 ⌃ c o  t + 2 |Z|| ⌃ c | q | ⌃ c | log(4 T | ⌃ c | /  ) 2( t +1)
In this section, we build on Algorithm 1 to deal with bandit feedback, i.e. , when at each round t 2 [ T ] the sender only observes the terminal node z t reached at the end of the SDM problem.
The main difﬁculties of such a setting can be summarized by the following observations.
First, the feedback z t only reveals partial information about the prior, and such information also depends on the selected signaling scheme  t .
Second, even if the sender plays a signaling scheme  2  for an arbitrarily large number of rounds, there is no guarantee that they collect enough information to tell whether  2  ⇧ ✏ ( µ ? ) or not for some ✏ > 0 .
Indeed, the persuasiveness of a signaling scheme depends on all receiver’s utilities in the SDM problem, and some parts of the tree may not be reached during a sufﬁciently large number of rounds by committing to  .
Thus, any algorithm for the bandit-feedback setting must guarantee a suitable level of exploration over the entire tree, so as to keep track of the entity of the violation of persuasiveness constraints.
We design a two-phase algorithm, whose pseudo-code is provided in Algorithm 2.
The algorithm takes as input the number N 2 [ T ] of rounds devoted to the ﬁrst phase guaranteeing the necessary amount of exploration, as detailed in Section 7.1.
During this phase, the S ELECT S TRATEGY () procedure implements an efﬁcient deterministic uniform exploration policy, which builds an unbiased estimator b µ N of µ ? .
This allows to restrict the space of feasible signaling schemes used in the subsequent phase to those that are approximately persuasive, i.e. , those in the set ⇤  N ( b µ N ) .
In
Section 7.2, we discuss the second phase of the the algorithm, composed by the rounds t > N , during which the algorithm focuses on the minimization of sender’s regret by exploiting the optimism in face of uncertainty principle.
Finally, in Section 7.3, we provide a lower bound on the trade-off between sender’s and receiver’s regrets, matching the upper bounds achieved by Algorithm 2 for a large portion of the trade-off frontier.
This result formally motivates the necessity of the uniform exploration which is performed in the ﬁrst phase of the algorithm.
7.1 Minimizing the receiver’s regret
At each round t 2 [ T ] , the sender observes a terminal node z t 2 Z that uniquely determines a path in the tree deﬁning the SDM problem.
We encode such a path by means of a vector p t 2 { 0 , 1 } | ⌃ c | such that p t [  ] = 1 if and only if the chance sequence  2 ⌃ c lies on the path from the root of the tree to z t , namely    c ( z t ) .
If the sender commits to a signaling scheme  t 2  , then it is easy to see that, for every
= ( h, a ) 2 ⌃ c , the element p t [  ] is distributed as a Bernoulli of parameter  t [  s ( h )] µ ?
[  ] .
The crucial observation behind the design of our estimator is that, if the sender commits to a deterministic signaling schemes ⇡ t 2 ⇧ at some round t 2 [ T ] , then for all the chance sequences  2 ⌃ c that are compatible with ⇡ t , i.e. , that can be observed when ⇡ t is played, we have that p t [  ] is distributed as a Bernoulli of parameter µ ? [  ] .
Formally, a sequence  2 ⌃ c is compatible with ⇡ t if there exists a chance node h 2 H c and an outcome a 2 A ( h ) satisfying  =
( h, a ) and ⇡ t [  s ( h )]
= 1 .
This observation leads to the following result:
Lemma 3.
For every deterministic signaling scheme ⇡ 2 ⇧ , let ⌃
[Equation]
Then, during each round t  N of Algorithm 2, it holds E [ p t [  ]] = µ ?
[  ] for every  2 ⌃ # ( ⇡ t ) .
Thus, during the ﬁrst phase, Algorithm 2 builds the desired estimator b µ N of µ ? as follows.
At each round t  N , after observing the feedback z t , the algorithm samples a deterministic signaling scheme ⇡ t 2 ⇧ according to  t (the one actually selected at t ), so that all the sequences  2 ⌃ c such that p t [  ] = 1 (or, equivalently,    c ( z t ) ) belong to ⌃ # ( ⇡ t ) .
Then, for every  2 ⌃ # ( ⇡ t ) , the algorithm updates the estimator component
[Equation]  ] according to p t [  ] . Since the probability of visiting a sequence  2 ⌃ c depends on  t (and, thus, can be arbitrarily small), the ﬁrst N rounds must be carefully used to ensure that each sequence is explored at least N/ | ⌃ c | times.
To explore a speciﬁc sequence  2 ⌃ c , we choose a signaling scheme  t such that  2 ⌃ # ( ⇡ t ) for every deterministic ⇡ t ⇠  t .
The procedure described above is needed for minimizing the receiver’s regret, since, in the second phase, the algorithm selects signaling schemes  t from ⇤  N ( b µ N ) .
In particular, as shown by the following lemma, Algorithm 2 guarantees that the receiver’s regret is upper bounded by 2  N at each round t > N , since it deﬁnes ✏ t [  ] for each sequence  2 ⌃ c so that the event ˜
E := {| µ ? [  ]  [Equation]
with probability at [Equation]
Lemma 4.
Under the event ˜ E , Algorithm 2 guarantees that  t 2 ⇤ 2  N ( µ ? ) at each round t > N .
7.2 Minimizing the sender’s regret
Algorithm 2 also needs to guarantee small sender’s regret.
To do so, we would like that  ? is a valid pick for the algorithm, i.e. , it belongs to ⇤  N ( b µ t ) .
However, differently from the full-feedback setting, stopping exploration after the ﬁrst N round does not guarantee optimal rates.
In order to ﬁx this issue, in the second phase, the algorithm selects  t optimistically by maximizing the sender’s expected utility F (  , µ ) over both  2 ⇤  N ( b µ N ) and µ 2 C t (  ) , where C t (  ) is a suitably-deﬁned conﬁdence set centered around b µ t such that { µ ? 2 C t (  ) } ⌘ ˜ E , and, thus, it holds with high probability.
This guarantees that max [Equation]  ) F (  ? , µ )  F (  ? , µ ? ) .
Formally:
Lemma 5.
If the event ˜ E holds, then, for every round t > N , it holds that  ? 2 ⇤  N ( b µ t ) and max µ 2 C t (  ) F (  ? , µ )  F (  ? , µ ? ) . Thus, F (  t , µ ? )
⇡ F (  t ,
[Equation]
µ t )  [Equation] µ 2 C t (  ) F (  ? , b µ )  F (  ? , µ ? ) holds in the limit, implying that F (  t , µ ? ) converges to F (  ? , µ ? ) after sufﬁciently many rounds.
Formally:
Theorem 5.
Given any  2 (0 , 1) and N 2 [ T ] , Algorithm 2 guarantees:
[Equation]
with probability at least 1   .
Moreover, the algorithm runs in polynomial time.
In contrast to the case with full feedback, the optimization problem solved by Algorithm 2 belongs to the class of bilinear problems, which are NP -hard in general [Hillar and Lim, 2013].
However, in
Theorem 5 we prove that our speciﬁc problem can be solved in polynomial time.
Furthermore, notice that Theorem 5 takes as input the number N of rounds devoted to the ﬁrst phase.
Given an ↵  1 / 2 , by choosing any N = b T ↵ c we get bounds of R T =
˜ O
( T ↵ ) and V T =
˜ O
( T max { ↵ , 1  ↵ 2 } ) .
7.3 The lower bound frontier
Order of T in R T
O r d e r o f T i n V T
LowerBoundAlgorithm
We conclude by showing that the trade offs between V T and R T achieved by Algorithm 2 are essentially tight.
Previously, we provided an intuition as to why the algorithm needs to uniformly explore the entire tree of the SDM problem.
Here, we provide a lower bound that corroborates such a statement.
In particular, the following theorem shows that, for any ↵ 2 [1 / 2 , 1] , in order to guarantee a sender’s regret of the order of O ( T ↵ ) , it is necessary to suffer a receiver’s regret of the order of ⌦ ( T 1  ↵ / 2 ) .
Theorem 6.
For any ↵ 2 [1 / 2 , 1] , there exists a constant (0 , 1) such that no algorithm guarantees both R T
=
o ( T ↵ ) and
V T
=
o ( T 1  ↵ / 2 ) with probability greater than  .
Figure 1 shows on the horizontal axis the order of the T term in R T , while, on the vertical axis, it shows the order of the T in V T .
The shaded area over the blue line shows the achievable trade offs, while the marked red line shows the performances proved in Theorem 5.
Thus, we show that Algorithm 2 matches the lower bound for ↵ 2 [1 / 2 , 2 / 3] .
However, when ↵ 2 [2 / 3 , 1] , the guarantees proved in Theorem 5 diverge from the ones proved in the lower bound.
This is due to the
N = b T ↵ c component in the receiver’s regret that becomes dominant when ↵  2 / 3 .
We conjecture that it is possible to reduce this term to p N , hence matching the lower bound of Theorem 6.
The reason for such a gap between the lower and upper bounds is that, during the ﬁrst phase, Algorithmutilizes signaling schemes without taking into account their persuasiveness, thus incurring in large receiver’s regret during the ﬁrst steps.
We leave as future work addressing the question of whether it is possible to design exploration strategies by only using approximately-persuasive signaling schemes.
[-= 
-]
[-= 0 for all sequences σ (cid:
-]
[-Given any φ ∈ Φ and µ , µ (cid:
-]