Interaction-Grounded Learning with Action-Inclusive Feedback
Abstract
Consider the problem setting of Interaction-Grounded Learning (IGL), in which a learner’s goal is to optimally interact with the environment with no explicit reward to ground its policies.
The agent observes a context vector, takes an action, and receives a feedback vector—using this information to effectively optimize a policy with respect to a latent reward function.
Prior analyzed approaches fail when the feedback vector contains the action, which significantly limits IGL’s success in many potential scenarios such as Brain-computer interface (BCI) or Humancomputer interface (HCI) applications.
We address this by creating an algorithm and analysis which allows IGL to work even when the feedback vector contains the action, encoded in any fashion.
We provide theoretical guarantees and large-scale experiments based on supervised datasets to demonstrate the effectiveness of the new approach.
Introduction
Most real-world learning problems, such as BCI and HCI problems, are not tagged with rewards.
Consequently, (biological and artificial) learners must infer rewards based on interactions with the environment, which reacts to the learner’s actions by generating feedback, but does not provide any explicit reward signal.
This paradigm has been previously studied by researchers [e g., Grizou et al., 2014; Nguyen et al., 2021], including a recent formalization [Xie et al., 2021b] that proposed the term Interaction-Grounded Learning (IGL).
In IGL, the learning algorithm discovers a grounding for the feedback which implicitly discovers a reward function.
An information-theoretic impossibility argument indicates additional assumptions are necessary to succeed.
Xie et al.
[2021b] proceed by assuming the action is conditionally independent of the feedback given the reward.
However, this is unnatural in many settings such as neurofeedback in BCI [Katyal et al., 2014; Mishra and Gazzaley, 2015; Debettencourt et al., 2015;
Muñoz-Moldes and Cleeremans, 2020; Akinola et al., 2020; Poole and Lee, 2021;
Xu et al., 2021] and multimodal interactive feedback in HCI [Pantic and Rothkrantz, 2003; Vitense et al., 2003;
Freeman et al., 2017; Mott et al., 2017; Duchowski, 2018; Saran et al., 2018, 2020; Zhang et al.,
2020; Cui et al., 2021; Gao et al., 2021] where the action proceeds and thus influences the feedback.
For example, if the feedback is an fMRI recording from a human brain implant in the parietal cortex, and the action of the algorithm is to display a number being imagined by the human (Fig. 1), the parietal cortex feedback will contain information about both perceiving the number being displayed (action) and human’s reaction to it for match or mismatch (reward).
Another example is determining if a patient in a coma is happy or not, and using their brain signals (feedback) to dissociate the fact that they were asked about being happy (action) and the actual thought that they are happy (reward).
An HCI example is a self-calibrating eye tracker used for typing on a screen by an ALS patient [Mott et al., 2017; Gao et al., 2021], where the automated agent calibrates the eye gaze to a certain point on a screen (action) and the human’s gaze patterns (feedback) convey a response to the tracker’s offset (action) as well as their satisfaction for being able to type a certain key (reward).
If you apply the prior
IGL approach to such settings, it will fail catastrophically because the requirement of conditional independence is essential to its function.
This motivates the question:
Is it possible to do interaction-ground learning when the feedback has the full information of the action embedded in it?
We propose a new approach to solve IGL, which we call action-inclusive IGL (AI-IGL), that allows the action to be incorporated into the feedback in arbitrary ways.
We consider latent reward as playing the role of latent states, which can be further separated via a contrastive learning method (Section 3.1).
Different from the typical latent state discovery in rich-observation reinforcement learning [e.g.,
Dann et al., 2018; Du et al., 2019; Misra et al., 2020], the IGL setting also requires identifying the semantic meaning of the latent reward states, which is addressed by a symmetry breaking procedure (Section 3.2).
We analyze the theoretical properties of the proposed approach, and we prove that it is guaranteed to learn a near-optimal policy as long as the feedback satisfies a weaker context conditional independence assumption.
We also empirically evaluate the proposed AI-IGL approach using large-scale experiments on Open-ML’s supervised classification datasets [Bischl et al., 2021], as well as a simulated BCI experiment using real human fMRI data [Ellis et al., 2020], demonstrating the effectiveness of the proposed approach (Section 5).
Thus, our findings broaden the scope of applicability for IGL.
The paper proceeds as follows.
In Section 2, we present the mathematical formulation for IGL.
In
Section 3, we present a contrastive learning perspective for grounding latent reward which helps to expand the applicability of IGL.
In Section 4, we state the resulting algorithm AI-IGL.
We provide experimental support for the technique in Section 5 using a diverse set of supervised learning datasets and a simulated BCI dataset.
We conclude with discussion in Section 6.
Background
Interaction-Grounded Learning This paper studies the Interaction-Grounded Learning (IGL) setting [Xie et al., 2021b], where the learner optimizes for a latent reward by interacting with the environment and associating (“grounding”) observed feedback with the latent reward.
At each time step, the learner receives an i.i.d.
context x from context set X and distribution d 0 .
The learner then selects an action a ∈ A from a finite action set |A| =
K .
The environment generates a latent binary reward r ∈ { 0 , 1 } (can be either deterministic or stochastic) and a feedback vector y ∈ Y conditional on ( x, a ) , but only the feedback vector y is revealed to the learner.
In this paper, we use
R ( x, a ) :=
E x,a [ r ] to denote the expected (latent) reward after executing action a on context x .
The space of context X and feedback vector Y can be arbitrarily large.
Throughout this paper, we use π ∈ Π : X → ∆( A ) to denote a (stochastic) policy.
The expected return of policy π is defined by V ( π ) =
E ( x,a ) ∼ d 0 × π [ r ] .
The learning goal of IGL is to find the optimal policy in the policy class, π ⋆ = argmax π ∈ Π V ( π ) , only from the observations of contextaction-feedback tuples, ( x, a, y ) .
This paper mainly considers the batch setting, and we use µ to denote the behavior policy.
In this paper, we also introduce value function classes and decoder classes.
We assume the learner has access to a value function class F where f ∈ F : X ×
A → [0 , 1] and reward decoder class ψ ∈ Ψ : Y ×
A → [0 , 1] .
We defer the assumptions we made on these classes to Section 3 for clarity.
We may hope to solve IGL without any additional assumptions.
However, it is informationtheoretically impossible without additional assumptions, even if the latent reward is decodable from ( x, a, y ) , as demonstrated by the following example.
Example 1 (Hardness of assumption-free IGL) .
Suppose y
=
( x, a ) and suppose the reward is deterministic in ( x, a ) .
In this case, the latent reward r can be perfectly decoded from y .
However, the learner receives no more information than ( x, a ) from the ( x, a, y ) tuple.
Thus if Π contains at least 2 policies, for any environment where any IGL algorithm succeeds, we can construct another environment with the same observable statistics where that algorithm must fail.
IGL with full conditional independence Example 1 demonstrates the need for further assumptions to succeed at IGL.
Xie et al.
[2021b] proposed an algorithm that leverages the following conditional independence assumption to facilitate grounding the feedback in the latent reward.
Assumption 1 (Full conditional independence) .
For arbitrary ( x, a, r, y ) tuples where r and y are generated conditional on the context x and action a , we assume the feedback vector y is conditionally independent of context x and action a given the latent reward r , i.e. x, a ⊥⊥ y | r .
Xie et al.
[2021b] introduce a reward decoder class ψ ∈ Ψ :
Y → [0 , 1] for estimating E [ r | y ] , which leads to the decoded return V ( π, ψ ) :
=
E ( x,a ) ∼ d 0 × π [ ψ ( y )] .
They proved that it is possible to learn the best π and ψ jointly by optimizing the following proxy learning objective:
[Equation]
where π base is a policy known to have low expected return.
Over this paper, we use IGL (full CI) to denote the proposed algorithm of Xie et al.
[2021b].
A Contrastive-learning Perspective for Grounding Latent Reward
The existing work of interaction-grounded learning leverages the assumption of full conditional independence (Assumption 1), where the feedback vector only contains information from the latent reward.
The goal of this paper is to relax these constraining assumptions and broaden the scope of applicability for IGL.
This paper focuses on the scenario where the action information is possibly embedded in the feedback vector, which is formalized by the following assumption.
Assumption 2 (Context Conditional Independence) .
For arbitrary ( x, a, r, y ) tuple where r and y are generated conditional on the context x and action a , we assume the feedback vector y is conditionally independent of context x given the latent reward r and action a .
That is, x ⊥⊥ y | a, r .
Assumption 2 allows the feedback vector y to be generated from both latent reward r and action a , differing from Assumption 1 which constrains the feedback vector y to be generated based only on the latent reward r .
We discuss the implications at the end of this section.
3.1 Grounding Latent Reward via Contrastive Learning
In this section, we propose a contrastive learning objective for interaction-grounded learning, which further guides the design of our algorithm.
We perform derivations with exact expectations for clarity and intuitions and provide finite-sample guarantees in Section 4.
Assumption 3 (Separability) .
For each ¯ a ∈ A , there exists an ( f ⋆ ¯ a , ψ ⋆ ¯ a ) ∈ F × Ψ , such that: 1) E
[ ψ ⋆ ¯ a ( y, ¯ a ) | ¯ a, r = 1] − E [ ψ ⋆ ¯ a ( y, ¯ a ) | ¯ a, r = 0] = 1 ; 2)
| E µ [ f ⋆ ¯ a ( x, ¯ a ) | ¯ a, r = 1] − E µ [ f ⋆ ¯ a ( x, ¯ a ) | ¯ a, r = 0] | ≥ ∆ F .
We also assume
1 − Ψ ⊆ Ψ , where 1 − Ψ :
= { 1 − ψ ( · , · ) : ψ ∈ Ψ } .
Assumption 3 consists of two components.
For Ψ , it is a realizability assumption that ensures that a perfect reward decoder is included in the function classes.
Although this superficially appears unreasonably strong, note y is generated based upon a and the realization of r ; therefore this is compatible with stochastic rewards.
For F , it ensures the expected predicted reward conditioned on the latent reward, having value r ∈ { 0 , 1 } and the action being a , is separable.
When µ = π unif , ∆ F can be lower bounded by max f ∈F 4 | Cov π ¯ a ( f, R ) | ( π ¯ a denotes the constant policy with action ¯ a ).
Detailed proof of this argument can be found in Appendix A. One sufficient condition is that the expected latent reward R ( · , ¯ a ) has enough variance and R ∈ F .
The condition of 1 − Ψ ⊆
Ψ can be constructed easily via standard realizable classes.
That is, if ψ ⋆ ¯ a ∈ Ψ ′ for some classes Ψ ′ , simply setting Ψ ← Ψ ′ (cid:83) (1 − Ψ ′ )
satisfies Assumption 3.
Note that, this construction of Ψ only amplifies the size of Ψ ′ by a factor of 2.
Reward Prediction via Contrastive
Learning We now formulate the following contrastivelearning objective for solving IGL.
Suppose µ ( a | x ) is the behavior policy.
We also abuse µ ( x, a, y ) to denote the data distribution, and µ a ( x, y ) , µ a ( x ) , µ a ( y ) to denote the marginal distribution under action a ∈ A .
We construct an augmented data distribution for each a ∈ A : (cid:101) µ a ( x, y ) :
= µ a ( x ) · µ a ( y ) (i.e., sampling x and y independently from µ a ).
Conceptually, for each action a ∈ A , we consider tuples ( x, y ) ∼ µ a ( x, y ) and ( (cid:101) x, (cid:101) y ) ∼ (cid:101) µ a ( x, y ) .
From Assumption 3, conditioned on ( x, a ) the optimal feedback decoder ψ ⋆ ( y, a ) has mean equal to the optimal reward predictor f ⋆ ( x, a ) .
Therefore we might seek an ( f, ψ ) pair which minimizes any consistent loss function, e.g., squared loss.
However this is trivially achievable, e.g., by having both always predict 0.
Therefore we formulate a contrastive-learning objective, where we maximize the loss between the predictor and decoder on the augmented data distribution.
For each a ∈ A , we solve the following objective
[Equation]
In the notation of equation (2), the a subscript indicates the ( f a , ψ a ) pair are optimal for action a .
Note they are always evaluated at a , which we retain as an input for compatibility with the original function classes.
Note that L is also similar to many popular contrastive losses [e.g., Wu et al., 2018; Chen et al., 2020] especially for the spectral contrastive loss [HaoChen et al., 2021].
[Equation]
Below we show that minimizing L a ( f a , ψ a ) decodes the latent reward under Assumptions 2 and 3 up to a sign ambiguity.
For simplicity, we introduce the notation of f a,r and ψ a,r for any ( f, ψ ) ∈ F × Ψ ,
[Equation]
f a,r and ψ a,r are the expected predicted reward of f ( x, a ) and decoded reward ψ ( y, a ) (under the behavior policy µ for f a,r ) conditioned on the latent reward having value r and the action being a .
Proposition 1.
For any action a ∈ A , if µ ( a | x ) > 0 for all x ∈ X and Assumption 2 andhold, and let ( (cid:98)
f
a , (cid:98) ψ a ) be the solution of Eq.
Then, | (cid:98) f a, 1 − (cid:98) f
a, 0 | =
max f ∈F | f a, 1 − f a, 0 | and | (cid:98) ψ a, 1 − (cid:98) ψ a, 0 | = max ψ ∈ Ψ | ψ a, 1 − ψ a, 0 | .
Proof Sketch.
For any policy π , we use d πa := (cid:80) x d 0 ( x ) π ( a | x ) to denote the visitation occupancy of action a under policy π , and ρ πa := 1 / d πa (cid:80) x
d 0 ( x ) π ( a | x ) R ( x, a ) to denote the average reward received under executing action a .
Then, by the context conditional independent assumption x ⊥⊥ y | r, a (Assumption 2), we know
[Equation]
Therefore, separately maximizing | f a, 1 − f a, 0 | and | ψ a, 1 − ψ a, 0 | maximizes L a ( f a , ψ a ) .
3.2 Symmetry Breaking
In the last section, we demonstrated that the latent reward could be decoded in a contrastive-learning manner up to a sign ambiguity.
The following example demonstrates the ambiguity.
Example 2 (Why do we need extra information to identify the latent reward?) .
In the optimal solution of objective Eq.
(2) , both (cid:98) ψ a and (cid:98) ψ ′ a :=
1 − (cid:98) ψ a yield the same value.
It is information-theoretically difficult to distinguish which one of them is the correct solution without extra information.
That is because, for any environment ENV1 , there always exists a “symmetric” environment ENV2 , where: 1)
R ( x, a ) of ENV1 is identical to (1 − R ( x, a )) of ENV2 for all ( x, a ) ∈
X
×
A ; 2) the the conditional distribution of y | r, a in ENV1 is identical to the conditional distribution of y | 1 − r, a in the ENV for all a ∈ A .
In this example, ENV1 and ENV2 will always generate the identical distribution of feedback vector y after any ( x, a ) ∈
X × A .
However, ENV1 and ENV2 have the exactly opposite latent reward information.
As we demonstrate in Example 2, the learner decoder from Eq.(2) could be corresponding to a symmetric pair of semantic meanings, and identifying them without extra information is informationtheoretically impossible.
The symmetry breaking procedure is one of the key challenges of interactiongrounded learning.
To achieve symmetry breaking, we make the following assumption to ensure the identifiability of the latent reward.
Assumption 4 (Baseline Policies) .
For each a ∈ A , there exists a baseline policy π a base , such that,
(a) π a base satisfies (cid:80) x
d 0 ( x ) π ( a | x ) ≥ c m > 0 .
(b) | 12 − ρ π a base a | ≥ η , where ρ π a base a
[Equation]
To instantiate Assumption 4 in practice, we provide the following simple example of π base that satisfies Assumption 4.
Suppose π base =
π unif (uniformly random policy), and we have “all constant policies are bad”, i.e., V ( π ¯ a = 1 ( a = ¯ a )) < 1 / 2 − η for all ¯ a ∈ A .
Then it is easy to verify that c m = 1 / K and ρ π base ¯ a ≤ 1 / 2 − η for all ¯ a ∈ A .
Note that π a base can be different over actions.
Intuitively, Assumption 4(a) is saying that the total probability of π a base selecting action a (over all context x ∈ X ) is at least c m .
This condition ensures that π a base has enough visitation to action a and makes symmetry breaking possible.
Assumption 4(b) states that if we only consider the reward obtained from taking action a , π a base is known to be either “sufficiently bad” or “sufficiently good”.
Note the directionality of the extremeness of π a base must be known, e.g., a policy which has a unknown reward of either 0 or 1 is not usable.
This condition follows a similar intuition as the identifiability assumption of Xie et al.
[2021b, Assumption 2] and breaks the symmetry.
For example, consider the ENV1 and ENV2 introduced in Example 2, ρ πa, ENV1 = 1 − ρ πa, ENV2 for any policy π .
To separating ENV1 and ENV2 using some policy π , ρ πa, ENV and ρ πa, ENV2 require to have a non-zero gap, which leads to Assumption 4(b).
The effectiveness of symmetry breaking under Assumption 4 can be summarized as below: we conduct the following estimation of ρ π base a , using the learned (cid:98)
[Equation]
If (cid:98) ψ a can efficiently decode the latent reward, then (cid:98) ρ π a base a converges to either ρ π base a or 1 − ρ π base a .
Therefore, applying Assumption 4(b) breaks the symmetry.
[Figure]
3.3 Comparison to Full CI
When we have the context conditional independence, it is easy to verify the failure of optimizing the original IGL objective Eq.(1) by the following example.
Example 3 (Failure of the original IGL objective under Assumption 2) .
Let X
=
A = { 1 , 2 , . .
. , 10 } and feedback vector is generated by y =
( a + R ( x, a ))
mod 10 (we use % to denote mod in the following part).
We also assume d 0 ( x ) =
π base ( a | x ) = 1 / 10 for any ( x, a ) ∈ X × A and
R ( x, a ) =
π ⋆ ( a | x ) :
= 1 ( x = a ) .
Then, we have, for any ψ : Y → [0 , 1] (approach proposed by Xie et al.
[2021b] assumes the reward decoder only takes feedback vector y as the input), L
[Equation]
On the other hand, consider the constant policy π 1 ( a | x ) :
= 1 ( a = 1) for all x ∈ X and decoder ψ 2 ( y ) :
=
( y = 2) for all y ∈ Y , then,
L
[Equation]
[Equation]
[Table]
[Equation]
This implies that maximizing the original IGL objective Eq.
(1) could not always converge to π ⋆ when we only have the context conditional independence.
This example indicates optimizing a combined contrastive objective with a single symmetry-breaking policy is insufficient to succeed in our x ⊥⊥ y | r, a case.
Our current approach corresponds to optimizing a contrastive objective and breaking symmetry for each action separately rather than simultaneously.
3.4 Viewing Latent Reward as a Latent State
Our approach is motivated by latent state discovery in Rich-Observation RL [Misra et al., 2020].
Figure 2 compares the causal graphs of Rich-Observation RL, IGL with context conditional independence, and IGL with full conditional independence.
In Rich-Observation RL, a contrastive learning objective is used to discover latent states; whereas in IGL a contrastive learning objective is used to discover latent rewards.
In this manner we view latent rewards analogously to latent states.
Identifying latent states up to a permutation is completely acceptable in Rich-Observations RL, as the resulting imputed MDP orders policies identically to the true MDP.
However in IGL the latent states have scalar values associated with them (rewards), and identifying up to a permutation is not sufficient to order policies correctly.
Thus, we require the additional step of symmetry breaking.
Main Algorithm
This section instantiates the algorithm for IGL with action-inclusive feedback, following the concept we introduced in Section 3.
For simplicity, we select the uniformly random policy π unif as behavior policy µ in this section.
That choice of µ can be further relaxed using an appropriate importance weight.
We first introduce the empirical estimation of L a as follows ( L a is defined in Eq.(2).
J ¯ a, D estimates the spectral contrastive loss, which corresponds to −L a ).
For any ¯ a ∈ A , we define the following empirical estimation of the spectral contrastive loss:
[Equation]
Using this definition, Algorithm 1 instantiates a version of the IGL algorithm with action-inclusive feedback.
Without loss of generality, we also assume ρ π a base a ≤ 1 / 2 for Assumption 4(b) for all a ∈ A in this section.
The case of ρ π a base a > 1 / 2 for some action a can be addressed by modifying the symmetry-breaking step properly in Algorithm 1.
do 3:
Obtain ( f ¯ a , ψ ¯ a ) by ▷ Latent State (Reward) Discovery (
[Equation]
←
[Equation] [Equation]
where J ¯ a, D ( f, ψ ) is defined in Eq.(4) 4:
Compute (cid:98) ϱ π ¯ a base ¯ a by (cid:98) ϱ π ¯ a base ¯ a = (cid:80) ( x,a,y ) ∈D π ¯ a base ( a | x )
ψ ¯ a ( x, ¯ a ) 1 ( a =¯ a ) (cid:80) ( x,a,y ) ∈D π ¯ a base ( a | x ) 1 ( a =¯ a ) .
▷ Symmetry Breaking 5: if (cid:98) ϱ π base ¯ a > 12 then ψ ′ ¯ a ← (1 − ψ ¯ a ) .
6: else ψ ′ ¯ a ← ψ ¯ a .
7: end if 8: end for
At a high level, Algorithm 1 has two separate components, latent state (reward) discovery (line 3) and symmetry breaking (line 4-7), for each action in A .
Theoretical guarantees In Algorithm 1, the output policy (cid:98) π is obtained by calling an offline contextual bandits oracle ( CB ).
We now formally define this oracle and its expected property.
Definition 1 (Offline contextual bandits oracle) .
An algorithm CB is called an offline contextual bandit oracle if for any dataset D
= { ( x i , a i , r i , µ ( a i | x i )) } |D| i =1 ( x i ∼ d 0 , a i ∼ µ , and r i is the reward determined by ( x i , a i ) ) and any policy class Π , the policy (cid:98) π produced by CB ( D ) satisfies ε CB :
[Equation]
max
[Equation]
The notion in Definition 1 corresponds to the standard policy learning approaches in the contextual bandits literature [e.g., Langford and Zhang, 2007; Dudik et al., 2011; Agarwal et al., 2014], and typically leads to ε CB = (cid:112) K log | Π | / δ / |D| .
We now provide the theoretical analysis of Algorithm 1.
In this paper, we use d F , Ψ to denote the joint statistical complexity of the class of F and Ψ .
For example, if the function classes are finite, we have d F , Ψ
= O (log |F|| Ψ | / δ ) , and δ is the failure probability.
The infinite function classes can be addressed by some advanced methods such as covering number or Rademacher complexity [see, e.g., Mohri et al., 2018].
The following theorem provides the performance guarantee of the output policy of Algorithm 1.
Theorem 2.
Suppose Assumptions 2, 3 and 4 hold, ∆ F is defined in Assumption 3, and (cid:98) π be the output policy of Algorithm 1.
If we have |D| ≥ O (cid:16)
[Equation]
, then, with high probability,
[Equation]
Similar to the performance of Xie et al.
[2021b], the learned is guaranteed to converge in the right direction only after we have sufficient data for the symmetric breaking.
The dependence on K in
Theorem 2 can be improved as different action has a separate learning procedure.
For example, if we consider F = F 1 ×F 2 ×· · ·×F K and Ψ = Ψ 1 × Ψ 2 ×· · ·× Ψ K , where F a and Ψ a are independent components that is only corresponding to action a (this is a common setup for linear approximated reinforcement learning approaches with discrete action space).
If F a and Ψ a are identical copies of K separate classes, we know log |F| =
K log |F a | and log | Ψ | =
K log | Ψ a | , which leads a √ K improvement.
We now provide the proof sketch of Theorem 2 and we defer the detailed proof to Appendix A.
Proof Sketch.
The proof of Theorem 2 consists of two different components—discovering latent reward and breaking the symmetry, which are formalized by the following lemma.
Lemma 3 (Discovering latent reward) .
Suppose Assumptions 2 and 3 hold, and let ( f ¯ a , ψ ¯ a ) be obtained by Eq.
Then, with high probability, we have | ψ ¯ a, 1 − ψ ¯ a, 0 | ≥ (cid:16) 1 − O (cid:16) 1∆ F (cid:113) K 3 d F , Ψ |D| (cid:17)(cid:17) .
Lemma 3 ensures that the learned decoder on Eq.(5) correctly separates the latent reward.
In particular since ψ ranges over [0 , 1] , Lemma 3 ensures max (Pr( ψ ( y, a ) = r ) , Pr( ψ ( y, a )
= 1 − r ))
> 1 − o (1) under the behaviour policy.
Thus, if we can break symmetry, we can use ψ to generate a reward signal and reduce to ordinary contextual bandit learning.
The following lemma guarantees the correctness of the symmetry-breaking step.
Lemma 4 (Breaking symmetry) .
Suppose Assumption 4 holds.
For any ¯ a ∈ A , if we have |D| ≥
O (cid:16)
[Equation]
, then, with high probability, ψ
[Equation]
Combining these two lemmas above as well as the CB oracle establishes the proof of Theorem 2, and the detailed proof can be found in Appendix A.
Empirical Evaluations
In this section, we provide empirical evaluations in simulated environments created using supervised classification datasets (Sec. 5.1) and an open source fMRI simulator (Sec. 5.2).
We evaluate our approach by comparing: (1) CB: Contextual Bandits with exact reward, (2) IGL (full CI): The method proposed by Xie et al.
[2021b] which assumes the feedback vector contains no information about the context and reward, and (3) AI-IGL: The proposed method which assumes that the feedback vector could contain information about the action but is conditionally independent of the context given the reward.
Note that contextual bandits (CB) is a skyline compared to both IGL (full CI) and AI-IGL, since it need not disambiguate the latent reward.
All methods use logistic regression with a linear representation.
At test time, each method takes the argmax of the policy.
Following the practical instantiation of Assumption 4 (Section 3.2), we know if the dataset has a balanced action distribution (no action belongs to more than 50% of the samples), selecting uniformly random policy π unif as π a base for all a ∈ A satisfies Assumption 4.
Therefore, in this section, our experiments are all based on the dataset with a balanced action distribution, and we select π a base = π unif for all a ∈ A .
5.1 Large-scale Experiments with OpenML CC-18 Datasets
To verify that our proposed algorithm scales to a variety of tasks, we evaluate performance on more than 200 datasets from the publicly available OpenML
Curated Classification Benchmarking
Suite [Vanschoren et al., 2015; Casalicchio et al., 2019; Feurer et al., 2021; Bischl et al., 2021].
OpenML CC-18 datasets are licensed under CC-BY license 2 and the platform and library are licensed under the BSD (3-Clause) license 3 .
At each time step, the context x t is generated uniformly at random.
The learner selects an action a t ∈ { 0 , . .
, K − 1 } as the predicted label of x t (where K is the total number of actions available in the environment).
The binary reward r t is the correctness of the prediction label a t .
The feedback y t is a two dimensional vector ( a t , r t ) .
Each dataset has a different sample size ( N ) and a different set of available actions ( K ).
We sample datasets with 3 or more actions, and with a balanced action distribution (no action belongs to more than 50% of the samples) to satisfy Assumption 4.
We use 90% of the data for training and the remaining 10% for evaluation.
Additional details of setting up the experiment are in Appendix C.
The results are averaged over 20 trials and shown in Table 1.
[Table]
5.2 Experiments on Realistic fMRI Simulation
We also evaluate our approach on a simulated BCI experiment, where a human imagines a number and the agent faces a number-decoding task (Fig. 1).
We simulate an fMRI based BCI setup, recording the fMRI BOLD (Blood Oxygenation Level Dependent) response in the parietal cortex of a human brain.
We have selected the parietal cortex since both numerical processing [Santens et al., 2010] and match-mismatch processing are reported to involve this region [Male et al., 2020], and in principle it could feasibly be recorded by fNIRS (with similar spatial resolution to fMRI)
[Naseer and Hong, 2015].
The task requires an intelligent agent to decode the numbers the human imagines.
This task is motivated by real-world examples where an implant could assist a person, user, or patient by communicating their mental states to the outside world, whether for therapeutic purposes or to control hardware or software.
The task setup consists of the following steps—(1) Human Imagination: A human participant imagines a number from a set of digits (i.e., 7 , 8 , 9 ), (2) Agent Decoding: an agent (Full CI IGL or
AI-IGL) decodes the number being imagined using only the fMRI signals of the human participant, with no labels and no supervised pretraining, (3) Human Perception: the agent visually displays the number it decoded to the human, which the human brain perceives, and hence the fMRI signals would contain patterns related to the perception of the agent
’s guess, (4) Human Judgment: Once the human judges whether the agent-guessed number yields a match or a mismatch to the number they imagined, fMRI signals reflect the participant’s judgment.
This simple setup mimics many relevant applications of BCI with a realistic simulation of fMRI signals (generalizable to the more cost-efficient fNIRS).
We used a third party simulation software, offering a strong test bed for the algorithms compared here.
We use an open source simulator
[Ellis et al., 2020] to simulate BCI-like, interactive fMRI signals for the imagination, perception, and judgment phases described above.
This simulator leverages real human fMRI data [Bejjanki et al., 2017] to simulate signals for different regions of the brain.
Prior neuroscience studies have shown that the parietal cortices in the human brain (in particular the posterior parietal cortex) process numbers [Santens et al., 2010], while the frontoparietal cortices process visual attention as well as match-mismatch judgments [Male et al., 2020].
These studies, together with the low temporal resolution of fMRI/fNIRS
[Glover, 2011], warrant the assumption we make here: that number perception and judgments signals can both be decoded from the same region, i.e., the parietal cortex.
Therefore, per AI-IGL’s theoretical assumptions, the two signals can be mixed in the brain’s implicit feedback signal, here simulated as an fMRI or fNIRS signal, in arbitrary ways.
Thus, we selected a region of interest (ROI) in the parietal cortex to cover the mixed feedback of perceiving the numbers and making a match-mismatch judgments.
This simulated BCI setting is precisely the sort of realistic condition that our proposed AI-IGL solution is designed to handle well.
Moreover, it is quite possible to conduct this experiment in principle using fMRI, or more realistically using fNIRS–which is portable, noninvasive, easy to use, and can be focused on a specific patch on the surface of the brain.
Each event (perception, imagination, judgment) evokes different activity in the human brain, within the same set of 4 × 4 × 4 voxels, generating a 64-dimensional activity vector.
We generate a time course of 666 trials, each of them including a sequence of events where an imagination event is followed by a response from the agent, then a perception event, and a judgment event.
Thus, thetrials include 1998 simulated brain events total.
Each event is 2 seconds long, with an inter-stimulus (ISI) of 7 seconds, and a temporal resolution of 10 seconds.
For each event, the simulator combines spatiotemporal noise (trial noise) and voxel activation patterns to simulate activity signals.
We use the imagination signal as context (Fig. 1, Context) and a weighted average of the perception and judgement signals to simulate the brain’s mixed feedback signal (Fig. 1, Feedback).
We evaluate the performance of both full CI IGL and AI-IGL on this simulated and interactive number-decoding task, under varying levels of feedback noise.
The results are shown in Table 2.
We found that full CI IGL failed at the task, remaining at chance level guessing.
However, the novel algorithm proposed here, AI-IGL, was capable of learning to dissociate the human judgment from a mixed feedback signal.
The significance of these findings are two-fold.
First, they offer a realistic approach to simulating BCI experiments in order to test theoretical advances before conducting real world experiments.
Second, our results further validate the theoretical advances of AI-IGL compared to the previous IGL approach for a realistic interactive brain-computer interface.
[Table]
Discussion
We have presented a new approach to solving Interaction-Grounded Learning, in which an agent learns to interact with the environment in the absence of any explicit reward signals.
Compared to a prior solution [Xie et al., 2021b], the proposed AI-IGL approach removes the assumption of conditional independence of actions and the feedback vector by treating the latent reward as a latent state.
It thereby provably solves IGL for action-inclusive feedback vectors.
By viewing the feedback as containing an action-(latent) reward pair which is an unobserved latent space, we propose latent reward learning using a contrastive learning approach.
This solution concept naturally connects to latent state discovery in rich-observation reinforcement learning [e.g., Dann et al., 2018; Du et al., 2019; Misra et al., 2020].
On the other hand, different from rich-observation
RL, the problem of IGL also contains a unique challenge in identifying the semantic meaning of the decoded class, which is addressed by a symmetry-breaking procedure.
In this work, we focus on binary latent rewards, for which symmetry breaking is possible using one policy (per action).
Breaking symmetry in more general latent reward spaces is a topic for future work.
A possible negative societal impact of this work can be performance instability, especially with inappropriate use of the techniques in risk-sensitive applications.
Barring intentional misuse, we envision several potential benefits of the proposed approach.
The proposed algorithm broadens the scope of IGL’s feasibility for real-world applications.
Imagine an agent being trained to interpret the brain signals of a user to control a prosthetic arm.
The brain’s response (feedback vector) to an action is a neural signal that may contain information about the action itself.
This is so prevalent in neuroimaging that fMRI studies routinely use specialized techniques to orthogonalize different information (e.g. action and reward) within the same signal [Momennejad and
Haynes, 2012, 2013; Belghazi et al., 2018; Shah and Peters, 2020].
Another example is a continually self-calibrating eye tracker, used by people with motor disabilities such as ALS [Hansen et al., 2004;
Liu et al., 2010; Mott et al., 2017; Gao et al., 2021].
A learning agent adapting to the ability of such users can encounter feedback directly influenced by the calibration correction action.
The proposed approach is a stepping stone on the path to solving IGL for complex interactive settings, overcoming the need for explicit rewards as well as explicit separation of action and feedback information.
Acknowledgments
The authors would like to thank Mark Rucker for sharing methods and code for computing and comparing diagnostic features of OpenML datasets.
NJ acknowledges funding support from ARL
Cooperative Agreement W911NF-17-2-0196, NSF IIS-2112471, NSF CAREER award, and Adobe
Data Science Research Award.
Appendix A Detailed Proofs
Proof of Proposition 1.
Over this proof, we follow the definition of Eq.(3) for each ( f, ψ ) ∈ F × Ψ .
For any π , let d πa := (cid:80) x
d 0 ( x ) π ( a | x ) and ρ πa := 1 / d πa (cid:80) x
d 0 ( x ) π ( a | x ) Pr( r = 1 | x, a ) , then
E µ a [ f ( x ) ψ ( y )]
= 1 d µa (cid:88) x,y d 0 ( x ) µ ( a | x ) f
( x ) Pr( y | a, r = 0) ψ
( y ) Pr( r =
| x, a ) +d µa (cid:88) x,y d 0 ( x ) µ ( a | x ) f
( x ) Pr( y | a, r = 1) ψ
( y ) Pr( r = 1 | x, a ) (by x ⊥⊥ y | r, a ) (a) =
(1 − ρ µa ) f a,
ψ a, 0 + ρ µa f a, 1 ψ a, 1 , and
E µ a [ f ( x )]
E µ a [ ψ ( y )]
= (cid:32)d µa (cid:88) x
d 0 ( x ) µ ( a | x ) f ( x ) (cid:33) (cid:32)d µa (cid:88) x,y d 0 ( x ) µ ( a | x ) Pr( y | a ) ψ ( y ) (cid:33) (b) =
((1 − ρ µa )
f a,
+ ρ µa f a, 1 ) ((1 − ρ µa )
ψ a,
+ ρ µa ψ a, 1 ) =
(1 − ρ µa ) 2 f a, 0 ψ a, 0 + ( ρ µa ) 2 f a, 1 ψ a, 1 + (1 − ρ µa ) ρ µa ( f a, 0 ψ a, 1 + f a, 1 ψ a, 0 ) .
To see why (a) and (b) hold, we use the following argument.
By definition of f a,r and ψ a,r , we have
[Equation]
Then
[Equation]
we have
[Equation]
Similar procedure also induces the remaining terms of (a) and (b)
[Equation]
Therefore
[Equation]
combining the two equalities above
[Equation]
we obtain
[Equation]
This completes the proof
[Equation]
Lower bound of ∆ F when µ = π unif .
For any f ∈ F , let V f ( π ) :
= (cid:80) x
E d 0 × π [ f ( x, a )] ∈ [0 , 1] .
Then,
[Equation]
denotes the constant policy with action
[Equation]
= ⇒ | E µ [ f ( x, ¯ a ) | ¯ a,
[Equation]
= 1]
[Equation]
E µ
[Equation]
f ( x, ¯ a ) |
[Equation]
a
[Equation]
0] | ≥ 4 |
[Equation]
π ¯
[Equation]
f,
[Equation]
) | , where (a) follows from
[Equation]
This completes the proof.
Proof of Lemma 3.
Let,
[Equation]
Over this proof, we define
[Equation]
Now, for J ¯ a, D ( f, ψ ) with any ( f, ψ ) , we also have with probability at least 1 − δ
[Equation]
This means
[Equation]
This implies
[Equation]
This completes the proof.
Proof of Lemma 4.
We now provide the proof for any fixed ¯ a ∈ A .
We define d π ¯ a := (cid:80) x d 0 ( x ) π
(¯ a | x ) , and R ψ ( x, a ) =
E [ ψ ( y, a ) | x, a ] as the decoded reward by ψ for all ( x, a, ψ ) ∈
X × A × Ψ .
Also, let ρ π ¯ a base ¯ a , (cid:98) ρ π
[Equation]
a
[Equation]
a
[Equation]
be
[Equation]
Over this section
[Equation]
we define ε stat , ¯ a as,
[Equation]
(by [Xie et al., 2021a, Lemma A.1])
Then, we know with probability at least 1 − δ ,
[Equation]
by a standard concentration argument.
We now show when line 5-6 in Algorithm 1 correctly break the symmetry.
By Assumption 4, we have ρ π ¯ a base ¯ a ≤ 1 2 − η .
In addition, by Eq.(10),
[Equation]
Recall the definition of ρ π ¯ a base ¯ a and ϱ π ¯ a base ¯ a (Eq.(8) and Eq.(9)), we have
[Equation]
According to Eq.(7), if ψ ¯ a, 1 > ψ ¯ a, 0 , we have
[Equation]
(by Eq.(11))
otherwise,
[Equation]
To guarantee the correctness of the symmetry breaking step (line 5-6 in Algorithm 1), we need 1.
If f ¯ a, 1 > f ¯ a, 0 , (cid:98) ϱ π ¯ a base ¯ a ≤ 12 , 2.
Otherwise, (cid:98)
ϱ π ¯ a base ¯ a > 12 .
That requires,
[Equation]
This can be induced by
[Equation]
Proof of Theorem 2.
Combining Lemma 3 and Lemma 4, we know for any ¯ a ∈ A
[Equation]
Then,
[Equation]
property of CB oracle in Definition 1)
This completes the proof
[Equation]
B Experiments on the MNIST dataset
The environment for this experiment is generated from the supervised classification MNIST dataset [LeCun et al., 1998] which is licensed under Attribution-Share Alike 3.0 license 4 .
At each time step, the context x t is generated uniformly at random.
Then the learner selects an action a t ∈ { 0 , . .
, 9 } as the predicted label of x t .
The binary reward r is the correctness of the prediction label a t .
The high-dimensional feedback vector y t is an image of the digit ( a t + 6 r − 3) mod 10 .
An example is shown in Figure 3.
Our results are averaged over 20 trials.
[Figure]
[Table]
To highlight that the proposed algorithm still operates well under conditions where the feedback vector does not include action information, we also perform experiments under the setting introduced by Xie et al.
[2021b] on the MNIST environment.
This setting is similar to the one described in
Section B except the feedback vector is the image of the digit r instead of ( a t +6 r − 3) mod 10 .
We find that under this action-exclusive feedback setting, the new proposed algorithm AI-IGL works as well as the IGL (full CI).
This signifies that our algorithm, which incorporates the presence of action information in the feedback vector, does not hurt performance in cases when the action information is missing from the feedback vector.
C Additional Experimental Details
This section provides additional details on our implementation.
The experiments on the MNIST dataset were conducted using a Google Colab, which was based on Intel Xeon CPU (2.30GHz) and 12 GB memory.
Large scale experiments on the OpenML datasets were conducted on CPU instances of an internal cluster.
No GPU was used.
The prototype codes were built over Python, PyTorch 5 , and Vowpal Wabbit 6 .
With the single process in the setup above, a single trial of each experiment took less than 30 minutes to finish.
Each experiment runs for 10 epochs over the dataset it is being evaluated on.
The data is shuffled only once before the training begins so each algorithms views the data in the same order in each epoch.
The experiment is established based on the Brain Imaging Analysis Kit (brainiak) 7 , which provides the realistic simulation of functional Magnetic Resonance Imaging (fMRI) based on the real human data [Bejjanki et al., 2017].
Our detailed setup, including the choice of the most of experiment parameters, follows from the demo provided in brainiak 8 .
C.1 Results for Individual OpenML datasets
For each individual OpenML dataset, we report the mean performance and standard errors for the best contact action policy, contextual bandit (CB), IGL assuming conditional independence of feedback on context and action given the latent reward (full CI)
[Xie et al., 2021b]
[Table]
and IGL with conditional independence of feedback on context given reward (AI-IGL) in Table 4.
The best constant action policy is computed based on the highest percentage of samples in the entire dataset with the same action.
We do not report the standard error for it.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
C.2 Ablation Analysis on OpenML Datasets
This section provides an ablation study based on the results of OpenML datasets in Table 1 and
Figure 4.
Both Table 1 and Figure 4 compares the performance of AI-IGL, CB, and IGL (full CI) on all OpenML datasets with balanced action distributions, and that with large sample size (sample size ≥ MNIST).
We can observe that AI-IGL almost always outperforms IGL (full CI).
We now demonstrate that the performance of AI-IGL is affected by: 1) the label noise; 2) the size of datasets (expected from theory), by the following observation.
At first, it is easy to notice that the performance of AI-IGL is significantly benefited from the sample size, by comparing with the performance of AI-IGL between all datasets and datasets with a large sample size (also suggested in Table 1), which follows the prediction of Theorem 2.
On the other hand, we study how the label noise affects the performance of AI-IGL.
To ablate the effect from the sample size, we only study the results in the dataset with large sample size.
In this case, we
[Figure]
consider the performance of CB to indicate the label noise, i.e., datasets with high CB performance means small label noise, and vice versa.
We can observe from Figure 4(a) that the performance gap between CB and AI-IGL increases considerably with the label noise increasing (i.e., CB performance decreasing).
This suggests that AI-IGL is more sensitive to the label noise compared with CB.
It is unclear if this is an information-theoretical difficulty due to the setting without explicit reward, and we leave further investigation along this line as future work.
C.3 Analyzing OpenML Dataset Meta-Properties with respect to AI-IGL’s Performance
In order to better understand which features of a dataset make them amenable to high IGL performance, we systematically analyzed the features of OpenML datasets.
Specifically, we measured a total offeatures/meta-properties for all OpenML datasets and compared the accuracy of AI-IGL’s success relative to CB.
We first explain the first 12, and explanation for the additional 3 measures follow.
We collected a total of 49 features following the findings of Lorena et al.
[2019]; Torra et al.
Reif et al.
[2014]; Abdelmessih et al.
Based on preliminary analysis.
We narrowed these features down to the following 12 features for further investigation of AI-IGL on OpenML datasets, because they had the least correlations with one another:
• Accuracy of the 1-nearest neighbor classifier on the dataset (1nn_accuracy)
[Reif et al., 2014; Abdelmessih et al., 2010] • Best single decision node accuracy created using the feature attribute with the highest information gain (best_node_accuracy) [Reif et al., 2014; Abdelmessih et al., 2010] • The number of 0/1 features when the dataset is one-hot encoded (feature_onehot_count) [Lorena et al., 2019] • Ratio of sample dimensionality by sample count (instance_per_feature) [Lorena et al., 2019] • The ratio of the class distribution entropy and the maximum entropy for the uniform distribution over classes (class_entropy_N)
• Maximum Fisher discriminant ratio (max_fisher_discrim)
[Lorena et al., 2019]
• The percentage of values in the feature matrix that are non-zero (max_single_feature_eff)
[Lorena et al., 2019] • Mutual information mean (mutual_XY_info_mean)
[Torra et al., 2008; Reif et al., 2014]
• Naive Bayes accuracy (naive_bayes_accuracy)
[Reif et al., 2014; Abdelmessih et al., 2010] • Noise to signal ratio (noise_signal_ratio)
[Torra et al., 2008]
• Number of principal components needed to represent 95% of data variability (pca_dims_95) [Lorena et al., 2019] • Percent of data variance explained by the top principal component (pca_top_1_percent)
[Lorena et al., 2019]
In addition to these 12 features, we added three additional features as follows.
Compared to the typical CB guarantee, AI-IGL needs one more K factor in its theoretical guarantees (Theorem 2).
Since K factors can be improved under some specific choice of function class (see discussion in
Section 4), four additional features of the dataset were used to predict the relative performance of
AI-IGL:
• N (n) • N/ √ K (n_by_sqrt_k) • N/K (n_by_k)
We used a binary random forest classifier to predict the success of AI-IGL’s performance relative to CB.
If the relative performance is ≥ 0 .
7 , we label it as a success.
In Table 5, we report the F scores for the success and failure classes defined in this way for a binary random forest classifier with 100 trees (evaluated using 10-fold cross validation on all 271 OpenML datasets).
Based on further analysis of the importance weight for each feature computed using the information gain metric (Figure 5(a)), we created smaller subsets of the datasets to further analyze datasets representative of realistic interaction datasets with small sample sizes.
We also report the average F1 scores for random forest classifiers trained and evaluated in a similar manner in rows 2,3 in Table 5.
Dataset
[Table]
We also computed the feature importance for each of the random forest classifiers to determine which feature is most predictive of AI-IGL’s success relative to CB.
Feature importance was computed using the information gain metric.
We used python’s scikit-learn library to implement the random forest classifiers as well as feature importance values.
We present feature importance plots for three different subsets of datasets in Figure 5.
We find N/K to be the most predictive feature of AI-IGL’s relative performance (Figure 5(a)).
It can alone predict its performance with an average F1 score of 0.79 under the same experimental setup.
However, for datasets with a small value of N/K ( ≤ 1000 , ≤ 200 ), there is high variability in relative performance.
Using such a subset of datasets, we find maximum Fisher discriminant [Lorena et al., 2019] (a measure of classification complexity that quantifies the informativeness of a given sample) to be the most predictive of relative performance (Figure 5(b), Figure 5(c)).
This finding identifies clear measures of small and large datasets that can predict whether AI-IGL can match CB performance for any given dataset.
It can also help researchers improve the design of novel applications of IGL, e.g., in HCI and BCI, by ensuring the resulting dataset’s features are amenable to high performance.
For the features considered relatively more important than others as shown in Figure 5(a), 5(b) and 5(c)), we also present how the relative performance of AI-IGL varies for different values of such features.
Visualizations for different features across all 271 datasets are shown in Figure 6, across the subset of 155 datasets with N/K ≤ 1000 are shown in Figure 7, and across the subset ofdatasets with N/K ≤ 200 are shown in Figure 8.
[Figure] [Figure]
[Figure] [Figure]