Interaction-Grounded Learning with Action-Inclusive Feedback
Anonymous Author(s)
AffiliationAddress email
Abstract
Consider the problem setting of Interaction-Grounded Learning (IGL), in which alearner’s goal is to optimally interact with the environment with no explicit rewardto ground its policies.
The agent observes a context vector, takes an action, andreceives a feedback vector—using this information to effectively optimize a policywith respect to a latent reward function.
Prior analyzed approaches fail whenthe feedback vector contains the action, which significantly limits IGL’s successin many potential scenarios such as Brain-computer interface (BCI) or Humancomputer interface (HCI) applications.
We address this by creating an algorithmand analysis which allows IGL to work even when the feedback vector contains theaction, encoded in any fashion.
We provide theoretical guarantees and large-scaleexperiments based on supervised datasets to demonstrate the effectiveness of thenew approach
Introduction
Most real-world learning problems, such as BCI and HCI problems, are not tagged with rewards.
Consequently, (biological and artificial) learners must infer rewards based on interactions with theenvironment, which reacts to the learner’s actions by generating feedback, but does not provideany explicit reward signal.
This paradigm has been previously studied by researchers [e.g., 1, 2],including a recent formalization [3] that proposed the term Interaction-Grounded Learning (IGL).
In IGL, the learning algorithm discovers a grounding for the feedback which implicitly discovers areward function.
An information-theoretic impossibility argument indicates additional assumptionsare necessary to succeed.
Xie et al.
[3] proceed by assuming the action is conditionally independentof the feedback given the reward.
However, this is unnatural in many settings such as neurofeedbackin BCI [4, 5, 6, 7, 8, 9, 10] and multimodal interactive feedback in HCI [11, 12, 13, 14, 15, 16, 17, 18,19, 20] where the action proceeds and thus influences the feedback.
For example, if the feedback isan fMRI recording from a human brain implant in the parietal cortex, and the action of the algorithmis to display a number being imagined by the human (Fig. 1), the parietal cortex feedback will containinformation about both perceiving the number being displayed (action) and human’s reaction to it formatch or mismatch (reward).
Another example is determining if a patient in a coma is happy or not,and using their brain signals (feedback) to dissociate the fact that they were asked about being happy(action) and the actual thought that they are happy (reward).
An HCI example is a self-calibrating eyetracker used for typing on a screen by an ALS patient [14, 20], where the automated agent calibratesthe eye gaze to a certain point on a screen (action) and the human’s gaze patterns (feedback) convey aresponse to the tracker’s offset (action) as well as their intention to type a certain key (reward).
If youapply the existing IGL approach to such settings, it will fail catastrophically because the requirementof conditional independence is essential to its function.
This motivates the question:
Is it possible to do interaction-ground learning when the feedback has the full information of the action embedded in it?
We propose a new approach to solve IGL, which we call action-inclusive IGL (AI-IGL), that allowsthe action to be incorporated into the feedback in arbitrary ways.
We consider latent reward as playingthe role of latent states, which can be further separated via a contrastive learning method (Section 3.1).
Different from the typical latent state discovery in rich-observation reinforcement learning [e.g.,21, 22, 23], the IGL setting also requires identifying the semantic meaning of the latent reward states,which is addressed by a symmetry breaking procedure (Section 3.2).
We analyze the theoreticalproperties of the proposed approach, and we prove that it is guaranteed to learn a near-optimal policyas long as the feedback satisfies a weaker context conditional independence assumption.
We alsoempirically evaluate the proposed AI-IGL approach using large-scale experiments on Open-ML’ssupervised classification datasets [24], as well as a simulated BCI experiment using real human fMRIdata [25], demonstrating the effectiveness of the proposed approach (Section 5).
Thus, our findingsbroaden the scope of applicability for IGL.
The paper proceeds as follows.
In Section 2, we present the mathematical formulation for IGL.
In
Section 3, we present a contrastive learning perspective for grounding latent reward which helps toexpand the applicability of IGL.
In Section 4, we state the resulting algorithm AI-IGL.
We provideexperimental support for the technique in Section 5 using a diverse set of supervised learning datasetsand a simulated BCI dataset.
We conclude with discussion in Section 6.
Background
Interaction-Grounded Learning This paper studies the Interaction-Grounded Learning (IGL)setting [3], where the learner optimizes for a latent reward by interacting with the environment andassociating (“grounding”) observed feedback with the latent reward.
At each time step, the learnerreceives an i.i.d.
context x from context set X and distribution d 0 .
The learner then selects an actiona ∈
A from a finite action set |A| =
K .
The environment generates a latent binary reward r ∈ { 0 , 1 }(can be either deterministic or stochastic) and a feedback vector y ∈ Y conditional on ( x, a ) , but onlythe feedback vector y is revealed to the learner.
In this paper, we use R ( x, a ) :
= E x,a [ r ] to denote theexpected (latent) reward after executing action a on context x .
The space of context X and feedbackvector Y can be arbitrarily large.
Throughout this paper, we use π ∈ Π : X → ∆( A ) to denote a (stochastic) policy.
The expectedreturn of policy π is defined by V ( π ) =
E ( x,a ) ∼ d 0 × π [ r ] .
The learning goal of IGL is to find theoptimal policy in the policy class, π ⋆ = argmax π ∈ Π V ( π ) , only from the observations of contextaction-feedback tuples, ( x, a, y ) .
This paper mainly considers the batch setting, and we use µ todenote the behavior policy.
In this paper, we also introduce value function classes and decoder classes.
We assume the learner has access to a value function class F where f ∈ F : X ×
A → [0 , 1] andreward decoder class ψ ∈ Ψ :
Y × A → [0 , 1] .
We defer the assumptions we made on these classesto Section 3 for clarity.
We may hope to solve IGL without any additional assumptions.
However, it is informationtheoretically impossible without additional assumptions, even if the latent reward is decodablefrom ( x, a, y ) , as demonstrated by the following example.
Example 1 (Hardness of assumption-free IGL) .
Suppose y
=
( x, a ) and suppose the reward isdeterministic in ( x, a ) .
In this case, the latent reward r can be perfectly decoded from y .
However,the learner receives no more information than ( x, a ) from the ( x, a, y ) tuple.
Thus if Π contains atleast 2 policies, for any environment where any IGL algorithm succeeds, we can construct anotherenvironment with the same observable statistics where that algorithm must fail.
IGL with full conditional independence Example 1 demonstrates the need for further assumptionsto succeed at IGL.
Xie et al.
[3] proposed an algorithm that leverages the following conditionalindependence assumption to facilitate grounding the feedback in the latent reward.
Assumption 1 (Full conditional independence) .
For arbitrary ( x, a, r, y ) tuples where r and y aregenerated conditional on the context x and action a , we assume the feedback vector y is conditionallyindependent of context x and action a given the latent reward r , i.e. x, a ⊥⊥ y | r .
Xie et al.
[3] introduce a reward decoder class ψ ∈ Ψ :
Y → [0 , 1] for estimating E [ r | y ] , which leadsto the decoded return V ( π, ψ ) :
=
E ( x,a ) ∼ d 0 × π [ ψ ( y )] .
They proved that it is possible to learn the bestπ and ψ jointly by optimizing the following proxy learning objective:
[Equation]
where π base is a policy known to have low expected return.
Over this paper, we use IGL (full CI) todenote the proposed algorithm in [3].
A Contrastive-learning Perspective for Grounding Latent Reward
The existing work of interaction-grounded learning leverages the assumption of full conditionalindependence (Assumption 1), where the feedback vector only contains information from the latentreward.
The goal of this paper is to relax these constraining assumptions and broaden the scope ofapplicability for IGL.
This paper focuses on the scenario where the action information is possiblyembedded in the feedback vector, which is formalized by the following assumption.
Assumption 2 (Context Conditional Independence) .
For arbitrary ( x, a, r, y ) tuple where r and y aregenerated conditional on the context x and action a , we assume the feedback vector y is conditionallyindependent of context x given the latent reward r and action a .
That is, x ⊥⊥ y | a, r .
Assumption 2 allows the feedback vector y to be generated from both latent reward r and action a ,differing from Assumption 1 which constrains the feedback vector y to be generated based only onthe latent reward r .
We discuss the implications at the end of this section.
3.1 Grounding Latent Reward via Contrastive Learning
In this section, we propose a contrastive learning objective for interaction-grounded learning, whichfurther guides the design of our algorithm.
We perform derivations with exact expectations for clarityand intuitions and provide finite-sample guarantees in Section 4.
Assumption 3 (Separability) .
For each ¯ a ∈ A , there exists an ( f ⋆ ¯ a , ψ ⋆ ¯ a ) ∈ F × Ψ ,such that: 1) E
[ ψ ⋆ ¯ a ( y, ¯ a ) | ¯ a, r = 1] − E [ ψ ⋆ ¯ a ( y, ¯ a ) | ¯ a, r = 0] = 1 ; 2)
| E µ [ f ⋆ ¯ a ( x, ¯ a ) | ¯ a, r = 1] − 109 E µ [ f ⋆ ¯ a ( x, ¯ a ) | ¯ a, r = 0] | ≥ ∆ F .
We also assume
1 − Ψ ⊆ Ψ , where 1 − Ψ :
= { 1 − ψ ( · , · ) : ψ ∈ Ψ } .
Assumption 3 consists of two components.
For Ψ , it is a realizability assumption that ensures thata perfect reward decoder is included in the function classes.
Although this superficially appears unreasonably strong, note y is generated based upon a and the realization of r ; therefore this iscompatible with stochastic rewards.
For F , it ensures the expected predicted reward conditionedon the latent reward, having value r ∈ { 0 , 1 } and the action being a , is separable.
When µ = π unif ,∆ F can be lower bounded by max f ∈F 4 | Cov π ¯ a ( f, R ) | ( π ¯ a denotes the constant policy with action¯ a ).
Detailed proof of this argument can be found in Appendix A. One sufficient condition is that theexpected latent reward R ( · , ¯ a ) has enough variance and R ∈ F .
The condition of 1 − Ψ ⊆
Ψ can beconstructed easily via standard realizable classes.
That is, if ψ ⋆ ¯ a ∈ Ψ ′ for some classes Ψ ′ , simplysetting Ψ ← Ψ ′ (cid:83) (1 − Ψ ′ )
satisfies Assumption 3.
Note that, this construction of Ψ only amplifiesthe size of Ψ ′ by a factor of 2.
Reward Prediction via Contrastive Learning We now formulate the following contrastivelearning objective for solving IGL.
Suppose µ ( a | x ) is the behavior policy.
We also abuse µ ( x, a, y )to denote the data distribution, and µ a ( x, y ) , µ a ( x ) , µ a ( y ) to denote the marginal distribution underaction a ∈ A .
We construct an augmented data distribution for each a ∈ A : (cid:101) µ a ( x, y ) :
= µ a ( x ) · µ a ( y )(i.e., sampling x and y independently from µ a ).
Conceptually, for each action a ∈ A , we consider tuples ( x, y ) ∼ µ a ( x, y ) and ( (cid:101) x, (cid:101) y ) ∼ (cid:101) µ a ( x, y ) .
From Assumption 3, conditioned on ( x, a ) the optimal feedback decoder ψ ⋆ ( y, a ) has mean equal tothe optimal reward predictor f ⋆ ( x, a ) .
Therefore we might seek an ( f, ψ ) pair which minimizes anyconsistent loss function, e.g., squared loss.
However this is trivially achievable, e.g., by having bothalways predict 0.
Therefore we formulate a contrastive-learning objective, where we maximize theloss between the predictor and decoder on the augmented data distribution.
For each a ∈ A , we solvethe following objective
[Equation]
In the notation of equation (2), the a subscript indicates the ( f a , ψ a ) pair are optimal for action a .
Note they are always evaluated at a , which we retain as an input for compatibility with the originalfunction classes.
Note that L is also similar to many popular contrastive losses [e.g., 26, 27] especially for the spectralcontrastive loss [28].
[Equation]
Below we show that minimizing L a ( f a , ψ a ) decodes the latent reward under Assumptions 2 and 3 upto a sign ambiguity.
For simplicity, we introduce the notation of f a,r and ψ a,r for any ( f, ψ ) ∈ F × Ψ ,
[Equation]
f a,r and ψ a,r are the expected predicted reward of f ( x, a ) and decoded reward ψ ( y, a ) (under thebehavior policy µ for f a,r ) conditioned on the latent reward having value r and the action being a .
Proposition 1.
For any action a ∈ A , if µ ( a | x ) > 0 for all x ∈ X and Assumption 2 and 3hold, and let ( (cid:98)
f a , (cid:98) ψ a ) be the solution of Eq.
Then, | (cid:98) f a, 1 − (cid:98) f
a, 0 | =
max f ∈F | f a, 1 − f a, 0 | and| (cid:98) ψ a, 1 − (cid:98) ψ a, 0 | = max ψ ∈ Ψ | ψ a, 1 − ψ a, 0 | .
Proof Sketch.
For any policy π , we use d πa := (cid:80) x d 0 ( x ) π ( a | x ) to denote the visitation occupancyof action a under policy π , and ρ πa := 1 / d πa (cid:80) x
d 0 ( x ) π ( a | x ) R ( x, a ) to denote the average rewardreceived under executing action a .
Then, by the context conditional independent assumption x ⊥⊥ y | r, a(Assumption 2), we know
[Equation]
Therefore, separately maximizing | f a, 1 − f a, 0 | and | ψ a, 1 − ψ a, 0 | maximizes L a ( f a , ψ a ) .
3.2 Symmetry Breaking
In the last section, we demonstrated that the latent reward could be decoded in a contrastive-learningmanner up to a sign ambiguity.
The following example demonstrates the ambiguity.
Example 2 (Why do we need extra information to identify the latent reward?) .
In the optimal solutionof objective Eq.
(2) , both (cid:98) ψ a and (cid:98) ψ ′ a :=
1 − (cid:98) ψ a yield the same value.
It is information-theoreticallydifficult to distinguish which one of them is the correct solution without extra information.
That is because, for any environment ENV1 , there always exists a “symmetric” environment ENV2 , where: 1)
R ( x, a ) of ENV1 is identical to (1 − R ( x, a )) of ENV2 for all ( x, a ) ∈
X
×
A ; 2) the the conditionaldistribution of y | r, a in ENV1 is identical to the conditional distribution of y | 1 − r, a in the ENV2for all a ∈ A .
In this example, ENV1 and ENV2 will always generate the identical distribution offeedback vector y after any ( x, a ) ∈
X × A .
However, ENV1 and ENV2 have the exactly oppositelatent reward information.
As we demonstrate in Example 2, the learner decoder from Eq.(2) could be corresponding to asymmetric pair of semantic meanings, and identifying them without extra information is informationtheoretically impossible.
The symmetry breaking procedure is one of the key challenges of interactiongrounded learning.
To achieve symmetry breaking, we make the following assumption to ensure theidentifiability of the latent reward.
Assumption 4 (Baseline Policies) .
For each a ∈ A , there exists a baseline policy π a base , such that,( a)
π a base satisfies (cid:80)
[Equation] [Equation] [Equation]
| ≥ η , where ρ π a base a
[Equation]
To instantiate Assumption 4 in practice, we provide the following simple example of π base thatsatisfies Assumption 4.
Suppose π base =
π unif (uniformly random policy), and we have “all constantpolicies are bad”, i.e., V ( π ¯ a = 1 ( a = ¯ a )) < 1 / 2 − η for all ¯ a ∈ A .
Then it is easy to verify thatc m = 1 / K and ρ π base ¯ a ≤ 1 / 2 − η for all ¯ a ∈ A .
Note that π a base can be different over actions.
Intuitively, Assumption 4(a) is saying that the totalprobability of π a base selecting action a (over all context x ∈ X ) is at least c m .
This condition ensuresthat π a base has enough visitation to action a and makes symmetry breaking possible.
Assumption 4(b)states that if we only consider the reward obtained from taking action a , π a base is known to be either“sufficiently bad” or “sufficiently good”.
Note the directionality of the extremeness of π a base must beknown, e.g., a policy which has a unknown reward of either 0 or 1 is not usable.
This condition followsa similar intuition as the identifiability assumption of [3, Assumption 2] and breaks the symmetry.
For example, consider the ENV1 and ENV2 introduced in Example 2, ρ πa, ENV1 = 1 − ρ πa, ENV2 for anypolicy π .
To separating ENV1 and ENV2 using some policy π , ρ πa, ENV1 and ρ πa, ENV2 require to have anon-zero gap, which leads to Assumption 4(b).
The effectiveness of symmetry breaking under Assumption 4 can be summarized as below: weconduct the following estimation of ρ π base a , using the learned (cid:98) ψ
[Equation]
If (cid:98) ψ a can efficiently decode the latent reward, then (cid:98) ρ π a base a converges to either ρ π base a or 1 − ρ π base a .
Therefore, applying Assumption 4(b) breaks the symmetry.
3.3 Comparison to Full CI
When we have the context conditional independence, it is easy to verify the failure of optimizing theoriginal IGL objective Eq.(1) by the following example.
Example 3 (Failure of the original IGL objective under Assumption 2) .
Let X
=
A = { 1 , 2 , . .
. , 10 }and feedback vector is generated by y =
( a + R ( x, a ))
mod 10 (we use % to denote mod inthe following part).
We also assume d 0 ( x ) =
π base ( a | x ) = 1 / 10 for any ( x, a ) ∈ X × A and
R ( x, a ) =
π ⋆ ( a | x ) :
= 1 ( x = a ) .
Then, we have, for any ψ : Y → [0 , 1] (approach proposed by Xieet al.
[3] assumes the reward decoder only takes feedback vector y as the input), L
[Equation]
π [Equation] [Equation] [Equation]
On the other hand, consider the constant policy π 1 ( a | x ) :
= 1 ( a = 1) for all x ∈ X and decoderψ 2 ( y ) :
=
( y = 2) for all y ∈ Y , then,
L ( π 1 , ψ 2 ) =
110 (cid:80) 10 x =
[Equation]
(cid:80)
[Equation]
=
[Equation]
( a = 1)
[Equation]
a
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
=
[Equation]
(cid:80)
[Equation]
=
[Equation]
[Equation]
a +
[Equation]
x
[Equation]
a ))%10) =
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
Ψ .
This implies that maximizing the original IGL objective Eq.
(1) could not always converge to π ⋆ whenwe only have the context conditional independence.
This example indicates optimizing a combined contrastive objective with a single symmetry-breakingpolicy is insufficient to succeed in our x ⊥⊥ y | r, a case.
Our current approach corresponds to optimizinga contrastive objective and breaking symmetry for each action separately rather than simultaneously. [Figure]
3.4 Viewing Latent Reward as a Latent State
[Figure] Our approach is motivated by latent state discovery in Rich-Observation RL [23].
Figure 2 comparesthe causal graphs of Rich-Observation RL, IGL with context conditional independence, and IGLwith full conditional independence.
In Rich-Observation RL, a contrastive learning objective is usedto discover latent states; whereas in IGL a contrastive learning objective is used to discover latentrewards.
In this manner we view latent rewards analogously to latent states.
Identifying latent states up to a permutation is completely acceptable in Rich-Observations RL, asthe resulting imputed MDP orders policies identically to the true MDP.
However in IGL the latentstates have scalar values associated with them (rewards), and identifying up to a permutation is notsufficient to order policies correctly.
Thus, we require the additional step of symmetry breaking.
Main Algorithm
This section instantiates the algorithm for IGL with action-inclusive feedback, following the conceptwe introduced in Section 3.
For simplicity, we select the uniformly random policy π unif as behaviorpolicy µ in this section.
That choice of µ can be further relaxed using an appropriate importanceweight.
We first introduce the empirical estimation of L a (defined in Eq.(2)) as follows.
For any¯ a ∈ A , we define the following empirical estimation of the spectral contrastive loss:
[Equation]
Using this definition, Algorithm 1 instantiates a version of the IGL algorithm with action-inclusivefeedback.
Without loss of generality, we also assume ρ π a base a ≤ 1 / 2 for Assumption 4(b) for all a ∈ Ain this section.
The case of ρ π a base a > 1 / 2 for some action a can be addressed by modifying thesymmetry-breaking step properly in Algorithm 1.
Algorithm do 3:
Obtain (
) by ▷ Latent State (Reward) Discovery
[Equation]
) where L ¯ a, D ( f, ψ ) is defined in Eq.(4) 4:
Compute ϱ π
[Equation]
by
[Equation]
Generate At a high level, Algorithm 1 has two separate components, latent state (reward) discovery (line 3) andsymmetry breaking (line 4-7), for each action in A . Theoretical guarantees In Algorithm 1, the output policy (cid:98) π is obtained by calling an offlinecontextual bandits oracle ( CB ).
We now formally define this oracle and its expected property. Definition 1 (Offline contextual bandits oracle) . An algorithm CB is called an offline contextualbandit oracle if for any dataset D
= { ( x i , a i , r i , µ ( a i | x i )) } |D| i =1 ( x i ∼ d 0 , a i ∼ µ , and r i is thereward determined by ( x i , a i ) ) and any policy class Π , the policy (cid:98) π produced by CB ( D ) satisfies
[Equation]
The notion in Definition 1 corresponds to the standard policy learning approaches in the contextualbandits literature [e.g., 29, 30, 31], and typically leads to ε CB = (cid:112) K log | Π | / δ / |D| .
We now providethe theoretical analysis of Algorithm 1.
In this paper, we use d F , Ψ to denote the joint statisticalcomplexity of the class of F and Ψ .
For example, if the function classes are finite, we haved F , Ψ
= O (log |F|| Ψ | / δ ) , and δ is the failure probability.
The infinite function classes can be addressedby some advanced methods such as covering number or Rademacher complexity [see, e.g., 32].
Thefollowing theorem provides the performance guarantee of the output policy of Algorithm 1.
Theorem 2.
Suppose Assumptions 2, 3 and 4 hold.
Let (cid:98) π be the output policy of Algorithm 1 and∆ F = max f ∈F | f a, 1 − f a, 0 | .
If we have |D| ≥ O (cid:16)
[Equation]
, then, with high probability,
V
[Equation]
⋆
[Equation] Similar to the performance of [3], the learned is guaranteed to converge in the right direction onlyafter we have sufficient data for the symmetric breaking.
The dependence on K in Theorem 2 canbe improved as different action has a separate learning procedure.
For example, if we consider
F = F 1 × F 2 × · · · × F K and Ψ =
Ψ 1 × Ψ 2 × · · · × Ψ K , where F a and Ψ a are independentcomponents that is only corresponding to action a (this is a common setup for linear approximatedreinforcement learning approaches with discrete action space).
If F a and Ψ a are identical copiesof K separate classes, we know log |F| =
K log |F a | and log | Ψ | =
K log | Ψ a | , which leads a √ Kimprovement.
We now provide the proof sketch of Theorem 2 and we defer the detailed proof to Appendix A.
Proof Sketch.
The proof of Theorem 2 consists of two different components—discovering latentreward and breaking the symmetry, which are formalized by the following lemma.
Lemma 3 (Discovering latent reward) .
Suppose Assumptions 2 and 3 hold, and let ( f ¯ a , ψ ¯ a ) beobtained by Eq.
Then, with high probability, we have | ψ ¯ a, 1 − ψ ¯ a, 0 | ≥ (cid:16) 1 − O (cid:16) 1∆ F (cid:113) K 3 d F , Ψ |D| (cid:17)(cid:17) .
Lemma 3 ensures that the learned decoder on Eq.(5) correctly separates the latent reward.
In particularsince ψ ranges over [0 , 1] , Lemma 3 ensures max (Pr( ψ ( y, a ) = r ) , Pr( ψ ( y, a )
= 1 − r ))
> 1 − o
(1)under the behaviour policy.
Thus, if we can break symmetry, we can use ψ to generate a reward signaland reduce to ordinary contextual bandit learning.
The following lemma guarantees the correctnessof the symmetry-breaking step.
Lemma 4 (Breaking symmetry) .
Suppose Assumption 4 holds.
For any ¯ a ∈ A , if we have |D| ≥
O (cid:16)
K 3 d F , Ψ (min { η ∆ F ,Kc m } ) 2 (cid:17) , then, with high probability, ψ ′ ¯ a, 1 ≥ ψ ′ ¯ a, 0 .
Combining these two lemmas above as well as the CB oracle establishes the proof of Theorem 2, andthe detailed proof can be found in Appendix A.
Empirical Evaluations
In this section, we provide empirical evaluations in simulated environments created using supervisedclassification datasets (Sec. 5.1) and an open source fMRI simulator (Sec. 5.2).
We evaluate ourapproach by comparing: (1) CB: Contextual Bandits with exact reward, (2) IGL (full CI):
The methodproposed by Xie et al.
[3] which assumes the feedback vector contains no information about thecontext and reward, and (3) AI-IGL: The proposed method which assumes that the feedback vectorcould contain information about the action but is conditionally independent of the context given the reward.
Note that contextual bandits (CB) is a skyline compared to both IGL (full CI) and AI-IGL,since it need not disambiguate the latent reward.
All methods use logistic regression with a linearrepresentation.
At test time, each method takes the argmax of the policy.
Following the practicalinstantiation of Assumption 4 (Section 3.2), we know if the dataset has a balanced action distribution(no action belongs to more than 50% of the samples), selecting uniformly random policy π unif asπ a base for all a ∈ A satisfies Assumption 4.
Therefore, in this section, our experiments are all basedon the dataset with a balanced action distribution, and we select π a base = π unif for all a ∈ A .
5.1 Large-scale Experiments with OpenML CC-18 Datasets
To verify that our proposed algorithm scales to a variety of tasks, we evaluate performance onmore than 200 datasets from the publicly available OpenML
Curated Classification Benchmarking
Suite [33, 34, 35, 24].
OpenML CC-18 datasets are licensed under CC-BY license 1 and the platformand library are licensed under the BSD (3-Clause) license 2 .
At each time step, the context x t isgenerated uniformly at random.
The learner selects an action a t ∈ { 0 , . .
, K − 1 } as the predictedlabel of x t (where K is the total number of actions available in the environment).
The binary rewardr t is the correctness of the prediction label a t .
The feedback y t is a two dimensional vector ( a t , r t ) .
Each dataset has a different sample size ( N ) and a different set of available actions ( K ).
We sampledatasets with 3 or more actions, and with a balanced action distribution (no action belongs to morethan 50% of the samples) to satisfy Assumption 4.
We use 90% of the data for training and theremaining 10% for evaluation.
Additional details of setting up the experiment are in Appendix C.
The results are averaged over 20 trials and shown in Table 1.
[Figure]
[Table]
5.2 Experiments on Realistic fMRI Simulation
We also evaluate our approach on a simulated BCI experiment for a number-guessing task (Fig. 1).
We simulate a brain implant, placed in the parietal cortex of a human brain, since both numericalprocessing [36] and match-mismatch processing are reported to involve this region [37].
The taskrequires an intelligent agent to decode the numbers the human imagines.
This task is motivated byreal-world examples where an implant assists a person, user, or patient by communicating their mentalstates to the outside world, whether for therapeutic purposes or to control hardware or software.
The task setup consists of the following steps—(1) Human Imagination: A human participantimagines a number from a set of digits (i.e., 7 , 8 , 9 ), (2) Agent Decoding: an agent (Full CI IGLor AI-IGL) decodes the number being imagined using fMRI signals of the human, with no labelsand no supervised pretraining, (3) Human Perception: the agent displays the number it decodedto the human, the human brain generates patterns related to the perception reflected in their fMRIsignals, (4) Human Judgment: The fMRI signal reflects the human participant’s judgment as towhether the agent’s number matched their imagined number or not.
This simple setup mimics manyrelevant applications of BCI with a realistic simulation of fMRI signals, using a third party simulationsoftware, offering a strong test bed for the algorithms compared here.
We use an open source simulator [25] to simulate fMRI signals for imagination, perception, andjudgment.
This simulator leverages real human fMRI data [38] to simulate signals for differentregions of the brain.
From prior neuroscience studies, it is known that the parietal cortices in thehuman brain (in particular the posterior parietal cortex) process numbers [36], while the frontoparietalcortices process visual attention as well as match-mismatch judgments [37].
These studies, togetherwith the low temporal resolution of fMRI/fNIRS [39], warrant the assumption that number perceptionand judgments signals can both be decoded from the same parietal region at the same time.
Therefore,per AI-IGL’s theoretical assumptions, the two signals can be mixed in the feedback fMRI signal inarbitrary ways.
Thus, we selected a region of interest (ROI) in the parietal cortex to cover the mixed feedback of perceiving the numbers and making a match-mismatch judgments.
This is precisely arealistic condition that our proposed AI-IGL solution is designed to handle well.
Moreover, it is quitepossible to conduct this experiment in principle using either fMRI or fNIRS (which is portable andcan focus on specific patch on the surface of the brain).
Each event (perception, imagination, judgment) evokes different activity in the human brain, withinthe same set of 4 × 4 × 4 voxels, generating a 64-dimensional activity vector.
We generate a timecourse of 666 trials, each of them including a sequence of events where an imagination event isfollowed by a response from the agent, then a perception event, and a judgment event.
Thus, the 666trials include 1998 simulated brain events total.
Each event is 2 seconds long, with an inter-stimulusinterval of 7 seconds, and a temporal resolution of 10 seconds.
For each event, the simulator combinesspatiotemporal noise (trial noise) and voxel activation patterns to simulate activity signals.
We usethe imagination signal as context (Fig. 1, Context) and a weighted average of the perception andjudgement signals as the feedback (Fig. 1, Feedback).
We evaluate the performance of both full CI IGL and AI-IGL on this number-guessing task undervarying levels of feedback noise.
The results are shown in Table 2.
We found that full CI IGLfailed at the task, remaining at chance level guessing.
However, the novel algorithm proposed here,
AI-IGL, was capable of learning to dissociate the human judgment from a mixed feedback signal.
Thesignificance of these findings are two-fold.
First, they offer a realistic approach to simulating BCIexperiments in order to test theoretical advances before conducting real world experiments.
Second,our results further validate the theoretical advances of AI-IGL compared to the previous IGL approachfor a realistic interactive brain-computer interface.
[Table]
Discussion
We have presented a new approach to solving Interaction-Grounded Learning, in which an agent learnsto interact with the environment in the absence of any explicit reward signals.
Compared to a priorsolution [3], the proposed AI-IGL approach removes the assumption of conditional independenceof actions and the feedback vector by treating the latent reward as a latent state.
It thereby provablysolves IGL for action-inclusive feedback vectors.
By viewing the feedback as containing an action-(latent) reward pair which is an unobserved latentspace, we propose latent reward learning using a contrastive learning approach.
This solutionconcept naturally connects to latent state discovery in rich-observation reinforcement learning [e.g.,21 , 22, 23].
On the other hand, different from rich-observation RL, the problem of IGL also containsa unique challenge in identifying the semantic meaning of the decoded class, which is addressed by asymmetry-breaking procedure.
In this work, we focus on binary latent rewards, for which symmetrybreaking is possible using one policy (per action).
Breaking symmetry in more general latent rewardspaces is a topic for future work.
A possible negative societal impact of this work can be performanceinstability, especially with inappropriate use of the techniques in risk-sensitive applications.
Barring intentional misuse, we envision several potential benefits of the proposed approach.
Theproposed algorithm broadens the scope of IGL’s feasibility for real-world applications. Imagine anagent being trained to interpret the brain signals of a user to control a prosthetic arm. The brain’sresponse (feedback vector) to an action is a neural signal that may contain information about the actionitself.
This is so prevalent in neuroimaging that fMRI studies routinely use specialized techniques toorthogonalize different information (e.g. action and reward) within the same signal [40, 41, 42, 43].
Another example is a continually self-calibrating eye tracker, used by people with motor disabilitiessuch as ALS [44, 45, 14, 20].
A learning agent adapting to the ability of such users can encounterfeedback directly influenced by the calibration correction action.
The proposed approach is a steppingstone on the path to solving IGL for complex interactive settings, overcoming the need for explicitrewards as well as explicit separation of action and feedback information.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative social impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assump-
tion 2, 3, and 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 4 and
Appendix A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 5 and Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5 andAppendix C.(b)
Did you mention the license of the assets?
[Yes] See Section 5.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]