Interaction-Grounded Learning with Action-inclusive Feedback
Anonymous Author(s)
AffiliationAddress email
Abstract
Consider the problem setting of Interaction-Grounded Learning (IGL), in which alearner’s goal is to optimally interact with the environment with no explicit rewardto ground its policies.
The agent observes a context vector, takes an action, andreceives a feedback vector, using this information to effectively optimize a policywith respect to a latent reward function.
Prior analyzed approaches fail whenthe feedback vector contains the action, which significantly limits IGL’s successin many potential scenarios such as Brain-computer interface (BCI) or Humancomputer interface (HCI) applications.
We address this by creating an algorithmand analysis which allows IGL to work even when the feedback vector contains theaction, encoded in any fashion.
We provide theoretical guarantees and large-scaleexperiments based on supervised datasets to demonstrate the effectiveness of thenew approach.
Introduction
Most real-world learning problems, such as BCI and HCI problems, are not tagged with rewards.
Consequently, (biological and artificial) learners must infer rewards based on interactions with theenvironment, which reacts to the learner’s actions by generating feedback, but does not provideany explicit reward signal.
This paradigm has been previously studied by researchers [e.g., 1, 2],including a recent formalization [3] that proposed the term Interaction-Grounded Learning (IGL).
In IGL, the learning algorithm discovers a grounding for the feedback which implicitly discovers areward function.
An information-theoretic impossibility argument indicates additional assumptionsare necessary to succeed.
Xie et al.
[3] proceed by assuming the action is conditionally independent ofthe feedback given the reward.
However, this is unnatural in many settings such as neurofeedback in BCI [4, 5, 6, 7, 8] and multimodal interactive feedback in HCI [9, 10, 11, 12, 10, 13, 14, 15, 16, 17],where the action proceeds and thus influences the feedback.
If you apply this prior approach to thesesettings, it will fail catastrophically because the requirement of conditional independence is essentialto its function.
This motivates the question:
Is it possible to do interaction-ground learning when the feedback has the full
information of the action embedded in it
We propose a new approach to solve IGL, which we call action-inclusive IGL (AI-IGL), that allowsthe action to be incorporated into the feedback in arbitrary ways.
We consider latent reward as playingthe role of latent states, which can be further separated via a contrastive learning method (Section 3.1).
Different from the typical latent state discovery in rich-observation reinforcement learning [e.g.,18, 19, 20], the IGL setting also requires identifying the semantic meaning of the latent reward states,which is addressed by a symmetry breaking procedure (Section 3.2).
We analyze the theoreticalproperties of the proposed approach, and we prove that it is guaranteed to learn a near-optimal policy as long as the feedback satisfies a weaker context conditional independence assumption.
We alsoevaluate the proposed AI-IGL approach using large-scale experiments on Open-ML’s supervisedclassification datasets [21] and demonstrate the effectiveness of the proposed approach (Sec. 5).
Thus,our findings broaden the scope of applicability for IGL.
The paper proceeds as follows.
In Section 2, we present the mathematical formulation for IGL.
In
Section 3, we present a contrastive learning perspective for grounding latent reward which helps toexpand the applicability of IGL.
In Section 4, we state the resulting algorithm AI-IGL.
We provideexperimental support for the technique in Section 5 using a diverse set of datasets.
We conclude withdiscussion in Section 6.
Background
Interaction-Grounded Learning This paper studies the Interaction-Grounded Learning (IGL)setting [3], where the learner optimizes for a latent reward by interacting with the environment andassociating (“grounding”)
observed feedback with the latent reward.
At each time step, the learnerreceives an i.i.d.
context x from context set X and distribution d 0 .
The learner then selects an actiona ∈
A from a finite action set |A| =
K .
The environment generates a latent binary reward r ∈ { 0 , 1 }(can be either deterministic or stochastic) and a feedback vector y ∈ Y conditional on ( x, a ) , but onlythe feedback vector y is revealed to the learner.
In this paper, we use R ( x, a ) :
= E x,a [ r ] to denote theexpected (latent) reward after executing action a on context x .
The space of context X and feedbackvector Y can be arbitrarily large.
Throughout this paper, we use π ∈ Π : X → ∆( A ) to denote a (stochastic) policy.
The expectedreturn of policy π is defined by V ( π ) =
E ( x,a ) ∼ d 0 × π [ r ] .
The learning goal of IGL is to find theoptimal policy in the policy class, π ⋆ = argmax π ∈ Π V ( π ) , only from the observations of contextaction-feedback tuples, ( x, a, y ) .
This paper mainly considers the batch setting, and we use µ todenote the behavior policy.
In this paper, we also introduce value function classes and decoder classes.
We assume the learner has access to a value function class F where f ∈ F : X ×
A → [0 , 1] andreward decoder class ψ ∈ Ψ :
Y × A → [0 , 1] .
We defer the assumptions we made on these classesto Section 3 for clarity.
We may hope to solve IGL without any additional assumptions.
However, it is informationtheoretically impossible without additional assumptions, even if the latent reward is decodablefrom ( x, a, y ) , as demonstrated by the following example.
Example 1 (Hardness of assumption-free IGL) .
Suppose y
=
( x, a ) and suppose the reward isdeterministic in ( x, a ) .
In this case, the latent reward r can be perfectly decoded from y .
However,the learner receives no more information than ( x, a ) from the ( x, a, y ) tuple.
Thus if Π contains atleast 2 policies, for any environment where any IGL algorithm succeeds, we can construct anotherenvironment with the same observable statistics where that algorithm must fail.
IGL with full conditional independence Example 1 demonstrates the need for further assumptionsto succeed at IGL.
Xie et al.
[3] proposed an algorithm that leverages the following conditionalindependence assumption to facilitate grounding the feedback in the latent reward.
Assumption 1 (Full conditional independence) .
For arbitrary ( x, a, r, y ) tuples where r and y aregenerated conditional on the context x and action a , we assume the feedback vector y is conditionallyindependent of context x and action a given the latent reward r , i.e. x, a ⊥⊥ y | r .
Xie et al.
[3] introduce a reward decoder class ψ ∈ Ψ :
Y → [0 , 1] for estimating E [ r | y ] , which leadsto the decoded return V ( π, ψ ) :
=
E ( x,a ) ∼ d 0 × π [ ψ ( y )] .
They proved that it is possible to learn the bestπ and ψ jointly by optimizing the following proxy learning objective:
[Equation]
where π base is a policy known to have low expected return.
Over this paper, we use IGL (full CI) todenote the proposed algorithm in [3].
A Contrastive-learning Perspective for Grounding Latent Reward
The existing work of interaction-grounded learning leverages the assumption of full conditionalindependence (Assumption 1), where the feedback vector only contains information from the latent reward.
The goal of this paper is to relax these constraining assumptions and broaden the scope ofapplicability for IGL.
This paper focuses on the scenario where the action information is possiblyembedded in the feedback vector, which is formalized by the following assumption.
Assumption 2 (Context Conditional Independence) .
For arbitrary ( x, a, r, y ) tuple where r and y aregenerated conditional on the context x and action a , we assume the feedback vector y is conditionallyindependent of context x given the latent reward r and action a .
That is, x ⊥⊥ y | a, r .
Assumption 2 allows the feedback vector y to be generated from both latent reward r and action a ,differing from Assumption 1 which constrains the feedback vector y to be generated based only onthe latent reward r .
We discuss the implications at the end of this section.
3.1 Grounding Latent Reward via Contrastive Learning
In this section, we propose a contrastive learning objective for interaction-grounded learning, whichfurther guides the design of our algorithm.
We perform derivations with exact expectations for clarityand intuitions and provide finite-sample guarantees in Section 4.
Assumption 3 (Separability) .
For each ¯ a ∈ A , there exists an ( f ⋆ ¯ a , ψ ⋆ ¯ a ) ∈ F × Ψ ,such that: 1) E
[ ψ ⋆ ¯ a ( y, ¯ a ) | ¯ a, r = 1] − E [ ψ ⋆ ¯ a ( y, ¯ a ) | ¯ a, r = 0] = 1 ; 2)
| E µ [ f ⋆ ¯ a ( x, ¯ a ) | ¯ a, r = 1] − 99 E µ [ f ⋆ ¯ a ( x, ¯ a ) | ¯ a, r = 0] | ≥ ∆ F .
We also assume
1 − Ψ ⊆ Ψ , where 1 − Ψ :
= { 1 − ψ ( · , · ) : ψ ∈ Ψ } .
Assumption 3 consists of two components.
For Ψ , it is a realizability assumption that ensures thata perfect reward decoder is included in the function classes.
Although this superficially appearsunreasonably strong, note y is generated based upon a and the realization of r ; therefore this iscompatible with stochastic rewards.
For F , it ensures the expected predicted reward conditionedon the latent reward, having value r ∈ { 0 , 1 } and the action being a , is separable.
When µ = π unif ,∆ F can be lower bounded by max f ∈F 4 | Cov π ¯ a ( f, R ) | ( π ¯ a denotes the constant policy with action¯ a ).
Detailed proof of this argument can be found in Appendix A. One sufficient condition is that theexpected latent reward R ( · , ¯ a ) has enough variance and R ∈ F .
The condition of 1 − Ψ ⊆
Ψ can beconstructed easily via standard realizable classes.
That is, if ψ ⋆ ¯ a ∈ Ψ ′ for some classes Ψ ′ , simplysetting Ψ ← Ψ ′ (cid:83) (1 − Ψ ′ )
satisfies Assumption 3.
Note that, this construction of Ψ only amplifiesthe size of Ψ ′ by a factor of 2.
Reward Prediction via Contrastive Learning We now formulate the following contrastivelearning objective for solving IGL.
Suppose µ ( a | x ) is the behavior policy.
We also abuse µ ( x, a, y )to denote the data distribution, and µ a ( x, y ) , µ a ( x ) , µ a ( y ) to denote the marginal distribution underaction a ∈ A .
We construct an augmented data distribution for each a ∈ A : (cid:101) µ a ( x, y ) :
= µ a ( x ) · µ a ( y )(i.e., sampling x and y independently from µ a ).
Conceptually, for each action a ∈ A , we consider tuples ( x, y ) ∼ µ a ( x, y ) and ( (cid:101) x, (cid:101) y ) ∼ (cid:101) µ a ( x, y ) .
From Assumption 3, conditioned on ( x, a ) the optimal feedback decoder ψ ⋆ ( y, a ) has mean equal tothe optimal reward predictor f ⋆ ( x, a ) .
Therefore we might seek an ( f, ψ ) pair which minimizes anyconsistent loss function, e.g., squared loss.
However this is trivially achievable, e.g., by having bothalways predict 0.
Therefore we formulate a contrastive-learning objective, where we maximize theloss between the predictor and decoder on the augmented data distribution.
For each a ∈ A , we solvethe following objective
[Equation]
In the notation of equation (2), the a subscript indicates the ( f a , ψ a ) pair are optimal for action a .
Note they are always evaluated at a , which we retain as an input for compatibility with the originalfunction classes.
We now show that L is equivalent to a spectral contrastive loss [e.g., 22, 23, 24].
[Equation]
Below we show that minimizing L a ( f a , ψ a ) decodes the latent reward under Assumptions 2 and 3 upto a sign ambiguity.
For simplicity, we introduce the notation of f a,r and ψ a,r for any ( f, ψ ) ∈ F × Ψ ,
[Equation]
f a,r and ψ a,r are the expected predicted reward of f ( x, a ) and decoded reward ψ ( y, a ) (under thebehavior policy µ for f a,r ) conditioned on the latent reward having value r and the action being a .
Proposition 1.
For any action a ∈ A , if µ ( a | x ) > 0 for all x ∈ X and Assumption 2 and 3hold, and let ( (cid:98)
f a , (cid:98) ψ a ) be the solution of Eq.
Then, | (cid:98) f a, 1 − (cid:98) f
a, 0 | =
max f ∈F | f a, 1 − f a, 0 | and| (cid:98) ψ a, 1 − (cid:98) ψ a, 0 | = max ψ ∈ Ψ | ψ a, 1 − ψ a, 0 | .
Proof Sketch.
For any policy π , we use d πa := (cid:80) x d 0 ( x ) π ( a | x ) to denote the visitation occupancyof action a under policy π , and ρ πa := 1 / d πa (cid:80) x
d 0 ( x ) π ( a | x ) R ( x, a ) to denote the average rewardreceived under executing action a .
Then, by the context conditional independent assumption x ⊥⊥ y | r, a(Assumption 2), we know [Equation] ψ
[Equation]
y
[Equation]
=
[Equation] = ⇒ L a ( f a , ψ a ) ∝ − ( f a [Equation] − f a
[Equation]
)( ψ a
[Equation]
1 − ψ a, 0 ) .
Therefore, separately maximizing | f a, 1 − f a, 0 | and | ψ a, 1 − ψ a, 0 | maximizes L a ( f a , ψ a ) .
3.2 Symmetry Breaking
In the last section, we demonstrated that the latent reward could be decoded in a contrastive-learningmanner up to a sign ambiguity.
The following example demonstrates the ambiguity.
Example 2 (Why do we need extra information to identify the latent reward?) .
In the optimal solutionof objective Eq.
(2) , both (cid:98) ψ a and (cid:98) ψ ′ a :=
1 − (cid:98) ψ a yield the same value.
It is information-theoreticallydifficult to distinguish which one of them is the correct solution without extra information.
That isbecause, for any environment ENV1 , there always exists a “symmetric” environment ENV2 , where: 1)
R ( x, a ) of ENV1 is identical to (1 − R ( x, a )) of ENV2 for all ( x, a ) ∈
X
×
A ; 2) the the conditionaldistribution of y | r, a in ENV1 is identical to the conditional distribution of y | 1 − r, a in the ENV2for all a ∈ A .
In this example, ENV1 and ENV2 will always generate the identical distribution offeedback vector y after any ( x, a ) ∈
X × A .
However, ENV1 and ENV2 have the exactly oppositelatent reward information.
As we demonstrate in Example 2, the learner decoder from Eq.(2) could be corresponding to asymmetric pair of semantic meanings, and identifying them without extra information is informationtheoretically impossible.
The symmetry breaking procedure is one of the key challenges of interactiongrounded learning.
To achieve symmetry breaking, we make the following assumption to ensure theidentifiability of the latent reward.
Assumption 4 (Baseline Policies) .
For each a ∈ A , there exists a baseline policy π a base , such that,
(a) π a base satisfies (cid:80) x
d 0 ( x ) π ( a | x ) ≥ c m > 0 .
(b) | 12 − ρ π a base a | ≥ η , where ρ π a base a
[Equation]
To instantiate Assumption 4 in practice, we provide the following simple example of π base thatsatisfies Assumption 4.
Suppose π base =
π unif (uniformly random policy), and we have “all constantpolicies are bad”, i.e., V ( π ¯ a = 1 ( a = ¯ a )) < 1 / 2 − η for all ¯ a ∈ A .
Then it is easy to verify thatc m = 1 / K and ρ π base ¯ a ≤ 1 / 2 − η for all ¯ a ∈ A .
Note that π a base can be different over actions.
Intuitively, Assumption 4(a) is saying that the totalprobability of π a base selecting action a (over all context x ∈ X ) is at least c m .
This condition ensuresthat π a base has enough visitation to action a and makes symmetry breaking possible.
Assumption 4(b)states that if we only consider the reward obtained from taking action a , π a base is known to be either“sufficiently bad” or “sufficiently good”.
Note the directionality of the extremeness of π a base must beknown, e.g., a policy which has a unknown reward of either 0 or 1 is not usable.
This condition follows [Figure] a similar intuition as the identifiability assumption of [3, Assumption 2] and breaks the symmetry.
For example, consider the ENV1 and ENV2 introduced in Example 2, ρ πa, ENV1 = 1 − ρ πa, ENV2 for anypolicy π .
To separating ENV1 and ENV2 using some policy π , ρ πa, ENV1 and ρ πa, ENV2 require to have anon-zero gap, which leads to Assumption 4(b).
The effectiveness of symmetry breaking under Assumption 4 can be summarized as below: weconduct the following estimation of ρ π base a , using the learned (cid:98) ψ a , (cid:98) ρ
[Equation]
If (cid:98) ψ a can efficiently decode the latent reward, then (cid:98) ρ π a base a converges to either ρ π base a or 1 − ρ π base a .
Therefore, applying Assumption 4(b) breaks the symmetry.
3.3 Comparison to Full CI
When we have the context conditional independence, it is easy to verify the failure of optimizing theoriginal IGL objective Eq.(1) by the following example.
Example 3 (Failure of the original IGL objective under Assumption 2) .
Let X
=
A = { 1 , 2 , . .
. , 10 }and feedback vector is generated by y =
( a + R ( x, a ))
mod 10 (we use % to denote mod inthe following part).
We also assume d 0 ( x ) =
π base ( a | x ) = 1 / 10 for any ( x, a ) ∈ X × A and
R ( x, a ) =
π ⋆ ( a | x ) :
= 1 ( x = a ) .
Then, we have, for any ψ : Y → [0 , 1] (approach proposed by Xieet al.
[3] assumes the reward decoder only takes feedback vector y as the input),
[Equation]
On the other hand, consider the constant policy π 1 ( a | x ) :
= 1 ( a = 1) for all x ∈ X and decoderψ 2 ( y ) :
=
( y = 2) for all y ∈ Y , then,
[Equation]
This implies that maximizing the original IGL objective Eq.
(1) could not always converge to π ⋆ whenwe only have the context conditional independence.
This example indicates optimizing a combined contrastive objective with a single symmetry-breakingpolicy is insufficient to succeed in our x ⊥⊥ y | r, a case.
Our current approach corresponds to optimizinga contrastive objective and breaking symmetry for each action separately rather than simultaneously.
3.4 Viewing Latent Reward as a Latent State
Our approach is motivated by latent state discovery in Rich-Observation RL [20].
Figure 1 comparesthe causal graphs of Rich-Observation RL, IGL with context conditional independence, and IGLwith full conditional independence.
In Rich-Observation RL, a contrastive learning objective is usedto discover latent states; whereas in IGL a contrastive learning objective is used to discover latentrewards.
In this manner we view latent rewards analogously to latent states.
Identifying latent states up to a permutation is completely acceptable in Rich-Observations RL, asthe resulting imputed MDP orders policies identically to the true MDP.
However in IGL the latent states have scalar values associated with them (rewards), and identifying up to a permutation is notsufficient to order policies correctly.
Thus, we require the additional step of symmetry breaking.
Main Algorithm
This section instantiates the algorithm for IGL with action-inclusive feedback, following the conceptwe introduced in Section 3.
For simplicity, we select the uniformly random policy π unif as behaviorpolicy µ in this section.
That choice of µ can be further relaxed using an appropriate importanceweight.
We first introduce the empirical estimation of L a (defined in Eq.(2)) as follows.
For any¯ a ∈ A , we define the following empirical estimation of the spectral contrastive loss:
[Equation]
Using this definition, Algorithm 1 instantiates a version of the IGL algorithm with action-inclusivefeedback.
Without loss of generality, we also assume ρ π a base a ≤ 1 / 2 for Assumption 4(b) for all a ∈ Ain this section.
The case of ρ π a base a > 1 / 2 for some action a can be addressed by modifying thesymmetry-breaking step properly in Algorithm 1.
Algorithm
Obtain ( f ¯ a , ψ ¯ a ) by ▷ Latent State (Reward) Discovery
[Equation]
[Equation]
where L ¯ a, D ( f, ψ ) is defined in Eq.(4) 4:
Compute (cid:98) ϱ
[Equation]
by (cid:98) ϱ
[Equation]
[Table]
Breaking 5: if base a > then 6: else ψ 7: end if 8: end for 9: Generate decoded contextual bandits dataset D CB ← { ( x, a, ψ ′ a ( y, a ) , µ ( a | x )) : ( x, a, y ) ∈ D} . 10: Output policy (cid:98) π ( x ) ← CB ( D CB ) , where CB denotes an offline contextual bandit oracle.
At a high level, Algorithm 1 has two separate components, latent state (reward) discovery (line 3) andsymmetry breaking (line 4-7), for each action in A .
Theoretical guarantees In Algorithm 1, the output policy (cid:98) π is obtained by calling an offlinecontextual bandits oracle ( CB ).
We now formally define this oracle and its expected property.
Definition 1 (Offline contextual bandits oracle) .
An algorithm CB is called an offline contextualbandit oracle if for any dataset D
= { ( x i , a i , r i , µ ( a i | x i )) } |D| i =1 ( x i ∼ d 0 , a i ∼ µ , and r i is thereward determined by ( x i , a i ) ) and any policy class Π , the policy (cid:98) π produced by CB ( D ) satisfiesε CB :=
max π ∈ Π E d 0 × π [ r ] − E d 0 × (cid:98) π [ r ] ≤ o (1) .
The notion in Definition 1 corresponds to the standard policy learning approaches in the contextualbandits literature [e.g., 25, 26, 27], and typically leads to ε CB = (cid:112) K log | Π | / δ / |D| .
We now providethe theoretical analysis of Algorithm 1.
In this paper, we use d F , Ψ to denote the joint statisticalcomplexity of the class of F and Ψ .
For example, if the function classes are finite, we haved F , Ψ
= O (log |F|| Ψ | / δ ) , and δ is the failure probability.
The infinite function classes can be addressedby some advanced methods such as covering number or Rademacher complexity [see, e.g., 28].
Thefollowing theorem provides the performance guarantee of the output policy of Algorithm 1.
Theorem 2.
Suppose Assumptions 2, 3 and 4 hold.
Let (cid:98) π be the output policy of Algorithm 1 and∆ F = max f ∈F | f a, 1 − f a, 0 | .
If we have |D| ≥ O (cid:16)
[Equation]
, then, with high probability,
[Equation]
[Figure]
Similar to the performance of [3], the learned is guaranteed to converge in the right direction onlyafter we have sufficient data for the symmetric breaking.
The dependence on K in Theorem 2 canbe improved as different action has a separate learning procedure.
For example, if we consider
F = F 1 × F 2 × · · · × F K and Ψ =
Ψ 1 × Ψ 2 × · · · × Ψ K , where F a and Ψ a are independentcomponents that is only corresponding to action a (this is a common setup for linear approximatedreinforcement learning approaches with discrete action space).
If F a and Ψ a are identical copiesof K separate classes, we know log |F| =
K log |F a | and log | Ψ | =
K log | Ψ a | , which leads a √ Kimprovement.
We now provide the proof sketch of Theorem 2 and we defer the detailed proof to Appendix A.
Proof Sketch.
The proof of Theorem 2 consists of two different components—discovering latentreward and breaking the symmetry, which are formalized by the following lemma.
Lemma 3 (Discovering latent reward) .
Suppose Assumptions 2 and 3 hold, and let ( f ¯ a , ψ ¯ a ) beobtained by Eq.
Then, with high probability, we have | ψ ¯ a, 1 − ψ ¯ a, 0 | ≥ (cid:16) 1 − O (cid:16) 1∆ F (cid:113) K 3 d F , Ψ |D| (cid:17)(cid:17) .
Lemma 3 ensures that the learned decoder on Eq.(5) correctly separates the latent reward.
In particularsince ψ ranges over [0 , 1] , Lemma 3 ensures max (Pr( ψ ( y, a ) = r ) , Pr( ψ ( y, a )
= 1 − r ))
> 1 − o
(1)under the behaviour policy.
Thus, if we can break symmetry, we can use ψ to generate a reward signaland reduce to ordinary contextual bandit learning.
The following lemma guarantees the correctnessof the symmetry-breaking step.
Lemma 4 (Breaking symmetry) .
Suppose Assumptions 4 hold.
For any ¯ a ∈ A , if we have |D| ≥
O (cid:16)
[Equation] (cid:17) , then, with high probability, ψ ′ ¯ a, 1 ≥ ψ ′ ¯ a, 0 .
Combining these two lemmas above as well as the CB oracle establishes the proof of Theorem 2, andtheir detailed proof can be found in Appendix A.
Empirical Evaluations
In this section, we provide empirical evaluations in simulated environments created using supervisedclassification datasets. The task is depicted in Figure 2.
We evaluate our approach by comparing: (1) CB: Contextual Banditswith exact reward, (2) IGL (full CI): The method proposed by Xie et al.
[3] which assumes thefeedback vector contains no information about the context and reward, and (3) AI-IGL: The proposedmethod which assumes that the feedback vector could contain information about the action but isconditionally independent of the context given the reward.
Note that contextual bandits (CB) is askyline compared to both IGL (full CI) and AI-IGL, since it need not disambiguate the latent reward.
All methods use logistic regression with a linear representation.
At test time, each method takes theargmax of the policy.
We provide details of setting up the experiment in Appendix B.
We evaluate our approach on different environments with action-inclusive feedback generated usingsupervised classification datasets.
We generate high-dimensional feedback vectors for the MNISTclassification dataset (Sec. 5.1) and low-dimensional feedback vectors for more than 200 OpenML
CC-18 datasets (Sec. 5.2). Following the practical instantiation of Assumption 4 (see Section 3.2),we know if the dataset has a balanced action distribution (no action belongs to more than 50% of thesamples), selecting random policy π unif as π a base for all a ∈ A satisfies Assumption 4.
Therefore, inthis section, our experiments are all based on the dataset with a balanced action distribution, and weselect π a base = π unif for all a ∈ A .
5.1 Experiments on the MNIST dataset
The environment for this experiment is generated from the supervised classification MNISTdataset [29] which is licensed under Attribution-Share Alike 3.0 license 1 . At each time step, thecontext x t is generated uniformly at random. Then the learner selects an action a t ∈ { 0 , . .
, 9 }as the predicted label of x t .
The binary reward r is the correctness of the prediction label a t .
Thehigh-dimensional feedback vector y t is an image of the digit ( a t + 6 r − 3) mod 10 .
An example isshown in Figure 2.
Our results are averaged over 20 trials.
[Table]
To highlight that the proposed algorithm still operates well under conditions where the feedbackvector does not include action information, we also perform experiments under the setting introducedby Xie et al.
[3] on the MNIST environment.
This setting is similar to the one described in Sec.
5.1except the feedback vector is the image of the digit r instead of a + r .
We find that under thisaction-exclusive feedback setting, the new proposed algorithm AI-IGL works as well as the IGL (full
CI).
This signifies that our algorithm, which incorporates the presence of action information in thefeedback vector, does not hurt performance in cases when the action information is missing from thefeedback vector.
5.2 Large-scale Experiments with OpenML CC-18 Datasets
To verify that our proposed algorithm scales to a variety of tasks, we evaluate performance on datasetsfrom the publicly available OpenML
Curated Classification Benchmarking Suite [30, 31, 32, 21].
OpenML datasets are licensed under CC-BY license 2 and the platform and library are licensed underthe BSD (3-Clause) license 3 . Similar to Sec.
5.1, at each time step, the context x t is generateduniformly at random.
Again, the learner selects an action a t ∈ { 0 , . . .
, K − 1 } as the predicted labelof x t (where K is the total number of actions available in the environment).
The binary reward r tis the correctness of the prediction label a t .
The feedback y t is a two dimensional vector ( a t , r t ) .
Each dataset has a different sample size ( N ) and a different set of available actions ( K ).
We sampledatasets with 3 or more actions, and with a balanced action distribution (no action belongs to morethan 50% of the samples) to satisfy Assumption 4.
We use 90% of the data for training and theremaining 10% for evaluation.
The results are averaged over 20 trials.
Analysis of Dataset Properties in relation to AI-IGL’s performance To further understand underwhich conditions AI-IGL succeeds, we analyzed its performance against 12 different features (basedon findings from Lorena et al. [33], Torra et al. [34], Reif et al. [35], Abdelmessih et al.
[36]) for
[Table]
each OpenML dataset (more details in Appendix B.3). We consider 3 additional features based onthe following criteria. Compared to the typical CB guarantee, AI-IGL needs one more K factor in itstheoretical guarantees (Theorem 2).
Since K factors can be improved under some specific choiceof function class (see discussion in Section 4), we select all of N , N/ √ K , N/K as the additionalfeatures of the dataset for predicting the relative performance of AI-IGL. We use a binary random forest classifier to predict the success of AI-IGL’s performance relative to
CB.
If the relative performance is ≥ 0 .
7 , we label it as a success. We find that with all 271 datasetswith K ≥ 3 , 10-fold cross-validation with 100 trees gives an average F1-score of 0 . 85 .
We find N/Kto be the most predictive feature of AI-IGL’s relative performance (Fig. 4(a)).
It can alone predictit’s performance with an average F1 score of 0.79 under the same experimental setup.
However, fordatasets with a small value of N/K ( ≤ 1000 ), there is high variability in relative performance.
Usingsuch a subset of datasets, we find maximum Fisher discriminant [33] (a measure of classificationcomplexity that quantifies the informativeness of a given sample) to be the most predictive of relativeperformance (Fig 4(b)).
Such datasets are representative of realistic interaction datasets with smallsample sizes.
More details on the experimental analysis are in Appendix B.3 (Fig. 5 and Fig. 6).
Thisfinding makes it possible to predict, for a given dataset, whether AI-IGL can match CB performance.
It can also help researchers improve the design of novel applications of IGL, e.g., in HCI and BCI, byensuring the resulting dataset’s features are amenable to high performance.
Discussion
We have presented a new approach to solving Interaction-Grounded Learning, in which an agent learnsto interact with the environment in the absence of any explicit reward signals.
Compared to a priorsolution [3], the proposed AI-IGL approach removes the assumption of conditional independenceof actions and the feedback vector by treating the latent reward as a latent state.
It thereby provablysolves IGL for action-inclusive feedback vectors.
By viewing the feedback as containing an action-(latent) reward pair which is an unobserved latentspace, we propose latent reward learning using a contrastive learning approach.
This solutionconcept naturally connects to latent state discovery in rich-observation reinforcement learning [e.g.,18 , 19, 20].
On the other hand, different from rich-observation RL, the problem of IGL also containsa unique challenge in identifying the semantic meaning of the decoded class, which is addressed by asymmetry-breaking procedure.
In this work, we focus on binary latent rewards, for which symmetrybreaking is possible using one policy (per action).
Breaking symmetry in more general latent rewardspaces is a topic for future work.
A possible negative societal impact of this work can be performanceinstability, especially with inappropriate use of the techniques in risk-sensitive applications.
Barring intentional misuse, we envision several potential benefits of the proposed approach.
Theproposed algorithm broadens the scope of IGL’s feasibility for real-world applications (where actionsignals are included in the feedback).
Imagine an agent being trained to interpret the brain signalsof a user to control a prosthetic arm.
The brain’s response (feedback vector) to an action is a neuralsignal that may contain information about the action itself.
This is so prevalent in neuroimaging thatfMRI studies routinely use specialized techniques to orthogonalize different information (e.g. actionand reward) within the same signal [37, 38, 39, 40].
Another example is a continually self-calibratingeye tracker, used by people with motor disabilities such as ALS [15, 17, 41, 42].
A learning agentadapting to the ability of such users can encounter feedback directly influenced by the calibrationcorrection action.
The proposed approach is a stepping stone on the path to solving IGL in suchcomplex settings, overcoming the need for explicit rewards as well as explicit separation of actionand feedback information.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative social impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assump-
tion 2, 3, and 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 4 and
Appendix A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 5 and Appendix B.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5 and Appendix B.
(b) Did you mention the license of the assets?
[Yes] See Section 5.
(c) Did you include any new assets either in the supplemental material or as a URL?
Yes](d)
Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A
](e ) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A
b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]