Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
In many real-world imitation learning tasks, the demonstrator and the learner haveto act under totally different observation spaces.
This situation brings significantobstacles to existing imitation learning approaches, since most of them learn policies under homogeneous observation spaces .
On the other hand, previous studiesunder different observation spaces have strong assumptions that these two observation spaces coexist during the entire learning process .
However, in reality, theobservation coexistence will be limited due to the high cost of acquiring expertobservations.
In this work, we study this challenging problem with limited observation coexistence under heterogeneous observations: Heterogeneously Observable
Imitation Learning (HOIL).
We identify two underlying issues in HOIL, i.e., thedynamics mismatch and the support mismatch, and further propose the Importance Weighting with REjection (IWRE)
algorithm based on importance-weightingand learning with rejection to solve HOIL problems.
Experimental results showthat IWRE can successfully solve various HOIL tasks, including the challengingtasks of transforming the vision-based demonstrations to random access memory(RAM)-based policies in the Atari domain, even with limited visual observations.
Introduction
Imitation Learning (IL) studies how to learn a good policy by imitating the given expert demonstrations [16, 1] and has achieved great success in many domains such as autonomous driving [8], videogames [7] and continuous control [19].
In real-world IL applications, the expert and the learnerusually have their own observations of the same underlying states from the environment.
For example,in Figure 1, an autonomous agent is learning to drive by imitating a human expert.
The expert takesher actions mainly based on auditory and visual observations, which are familiar to human beings.
However, the learning agent does not necessarily use the same way to observe: it can utilize moremachine-capable sensors such as a LiDAR, radar, and bird-eye view (BEV) map to generate itsobservations [20].
The key features behind this example are two-fold: First, both the expert andthe learner have their totally different observations of the same state of the environment.
Thus theyessentially have to choose the same action if acting optimally.
Second, the observation space of theexpert is often of high cost for the learner to utilize [6, 10].
We call this problem Heterogeneously
Observable Imitation Learning (HOIL).
There are two lines of research studying the related problems.
The first line relates to domainadaptation: the observation space of the expert and the learner are the homogeneous, while sometypical mismatches of distributions could exist: morphological mismatch, viewpoint mismatch, anddynamics mismatch [30, 17, 26].
However, these approaches are invalid when the observation spacesfor experts and learners are completely different as in HOIL.
The second line studied IL under different observations similar to HOIL, and some representativeworks include Partially Observable Imitation Learning (POIL) [14, 36] and Learning by Cheating
[Figure] [Figure]
(LBC) [8], as depicted in Figure 2.
Both POIL and LBC assume that the expert’s observations canbe easily accessed by the learner without any budget limit.
However in practice, different from thelearner observations, the access to expert’s observations might be of high cost and invasive [6, 10],hindering the wide application of these methods.
In this paper, we initialize the study of the HOIL problem.
We propose a learning process acrossobservation spaces of experts and learners for solving this problem, and analyze the underlying issuesof HOIL, i.e., the dynamics mismatch and the support mismatch.
To tackle both two issues, we resortto the techniques of importance-weighting [12] and learning with rejection [9, 15] for active queryingto propose the Importance Weighting with REjection (IWRE) approach.
We evaluate the effectivenessof the IWRE algorithm in continuous control tasks of MuJoCo [33], and the challenging tasks oflearning random access memory (RAM)-based policies given vision-based expert demonstrationsin Atari [3] games.
The results demonstrate that IWRE can significantly outperform existing ILalgorithms in HOIL tasks, with limited access to expert observations.
Related Work
Domain-Shifted IL.
For the standard IL process, where the learner and the expert share the sameobservation space, current state-of-the-art methods tend to learn the policy in an adversarial style [16,7].
When considering the domain mismatch problem, i.e., Domain-Shifted IL (DSIL), the researchaims at addressing the static distributional shift of the optimal policies resulted from the environmentaldifferences but still under homogeneous observation spaces.
Stadie et al.
[30], Sermanet et al.
[29],and Liu et al.
[23] studied the situation where the demonstrations are in view of a third person.
Kimet al.
[19] and Kim et al.
[18] addressed the IL problem with morphological mismatch between theexpert’s and learner’s environment.
Stadie et al.
[30], Tirinzoni et al.
[32], and Desai et al.
[11]focused on the calibration for the mismatch between simulators and the real world through sometransfer learning styles.
There are two major differences between HOIL and DSIL:
One is that HOIL considers heterogeneous observation spaces instead of homogeneous ones; another is thatin HOIL, instead of only considering the difference between two fixed policies, dealing with the dynamic change of the learner’s policy during learning is essential.
Thus HOIL is a significantlymore challenging problem than DSIL.
Besides, Chen et al.
[8] learned a vision-based agent from aprivileged expert.
But it can obtain expert’s observations throughout the whole learning process, so itcannot handle the problem of the support mismatch under HOIL.
POMDP.
The problem of POMDPs, in which only partial observations are available for the agent(s),has been studied in the context of multi-agent [25, 36] and imitation learning [14, 36] problems.
But distinct from HOIL, in a POMDP, the learner only have partial observations and share a sameunderlying observation space with the expert, which would become an obstacle for them to makedecisions correctly.
For example, Warrington et al.
[36] assumed that the observation of the learner ispartial than that of the expert.
While in HOIL, expert’s and learner’s observations are totally differentfrom each other, in the sense that the observations themselves are not the obstacle for acting optimally.
Instead, the main challenge for HOIL is to deal with the mismatches between the observation spaces,especially when the access to expert’s observations is strictly limited.
The HOIL Problem
In this section, we first give a formal definition of the HOIL setting, and then introduce the learningprocess for solving the HOIL problem.
3. Setting Definition
A HOIL problem is defined within a Markov decision process with mutiple observation spaces, i.e.,⟨S , {O} , A , P , γ ⟩ , where S denotes the state space, {O} denotes a set of observation spaces, Adenotes the action space, P : S ×
A × S → R denotes the transition probability distribution ofthe state and action, and γ ∈ (0 , 1] denotes the discount factor.
Furthermore, a policy π over anobservation space O is defined as a function mapping from O to A , and we denote by Π O the setof all policies over O .
In HOIL, both the expert and the learner have their own observation spaces,which are denoted as O E and O L respectively.
Both O E and O L are assumed to be produced bytwo bijective mappings f E :
S → O E , f L : S → O L , which are unknown functions mapping theunderlying true states to the observations.
It is obvious to see that by this assumption, any policy over
O E has a unique correspondence over O L .
This makes HOIL possible since the target of HOIL is tofind the corresponding policy of the expert policy under O L .
An state-action pair ( s, a ) , denoted by x , is called an instance .
Also, a trajectory T = { x } is aset of instances.
For each observation space, (cid:101) x ∈ (cid:101) T ⊆
O E × A and x ∈ T ⊆
O L × A , where
O E = f E ( S ) and O L = f L ( S ) .
Furthermore, we define the occupancy measure of a policy π underthe state space S as ρ π : O ×
A → R such that
ρ π ( x ) =
π ( a | o )
Pr( o | s ) (cid:80) ∞ t =0 γ t Pr( s t = s | π ) .
Under HOIL, the learner accesses the expert demonstrations (cid:101) T π E , a set of instances sampled from ρ π E .
The goal of HOIL is to learn a policy ˆ π as the corresponding policy of π E over O L .
If O E = O L ,
HOIL degenerates to standard IL .
GAIL [16] is one of the state-of-the-art IL approaches under thissituation, which tries to minimize the divergence between the learner’s and the expert’s occupancymeasures d ( ρ ˆ π , ρ π E ) .
The objective of GAIL is
[Equation]
where H (ˆ π ) is the causal entropy performed as a regularization term, and D w : O E ×
A → [0 , 1] isthe discriminator of π E and ˆ π .
GAIL solved Equation (1) by alternatively taking a gradient ascentstep to train the discriminator D w , and a minimization step to learn policy ˆ π based on an off-the-shelf
RL algorithm with the pseudo reward − log D w ( (cid:101) x ) .
3.2 The Learning Process for Solving HOIL
In HOIL, we need to cope with the absence of the learner’s observations in demonstrations and thehigh cost of collecting the expert’s observations while learning.
So we introduce a learning processwith pretraining across two different observation spaces for solving HOIL, as abstracted in Figure 3.
Pretraining.
Same to LBC [8], we assume that we can obtain an auxiliary policy π 1 based on O E atthe beginning.
π 1 can be directly provided by any sources, or trained by GAIL or behavior cloningas did in LBC.
Besides, we use this π 1 to sample some data T π 1 , which contain both observationunder O E (i.e., (cid:101) T π 1 ) and O L (i.e., T π 1 ), in order to connect these two different observation spaces.
We name T π 1 = { (cid:101) T π 1 , T π 1 } the evolving data .
[Figure]
Training.
Here we learn a policy π 2 from the evolving data T π 1 and the collected data T π 2 , under
O L only.
Besides, the learner is allowed for some operation of observation coexistence (OC): Atsome steps of learning, besides the observations O L , the learner could also request (cid:101) T π 2 from thecorresponding observations O E (e.g., from the human-understandable sensors).
The final objective of
HOIL is to learn a good policy π 2 under O L .
In practical applications, the auxiliary policy π 1 can also come from simulation training or directimitation.
But since π 1 is additionally provided, it is more practical to consider π 1 as a non-optimalpolicy.
During training, OC is an essential operation for solving HOIL, which helps the learneraddress the issues of the dynamics mismatch and the support mismatch (especially the latter one).
Also, in reality, OC just provides observations information without corresponding action signals as inmany active querying research [4, 8], so its cost will be much lower.
Besides, the related work [8]also required an initialized policy π 1 to solve their problem, which act as a teacher under privileged O E in the pretraining and then learned a vision-based student from the guidance of the teacher underboth O L and O E .
Their setting can be viewed as a variety of HOIL with optimal π 1 , unlimited O E ,and unlimited OC operations, so HOIL is actually a more practical learning framework.
Imitation Learning with Importance-Weighting and Rejection
In HOIL, the access frequency to O E is strictly limited, so it is unrealistic to learn π 2 in a Dataset
Aggregation (DAgger) style [27] as in LBC.
Therefore, we resort to learning π 2 with a learned rewardfunction by inverse reinforcement learning [1] in an adversarial learning style [16, 13].
In addition, both O E and O L are assumed to share the same latent state space S as introduced in
Section 3.1, so the following analysis will be based on S , while the algorithm will handle the problembased on O E and O L specifically.
4.1 Dynamics Mismatch and Importance-Weighting
To analyze the learning process, we let ρ π E , ρ π 1 , and ρ π 2 be the occupancy measure distributionsof the expert demonstrations, the evolving data, and the data during training respectively.
Since weneed to consider the sub-optimality of π 1 , ρ π 1 should be a mixture distribution of the expert ρ π E andnon-expert ρ π NE , i.e., there exists some δ ∈ (0 , 1) such that
[Equation]
as depicted in Figure 4a.
During training, the original objective of π 2 is to imitate π E throughdemonstrations.
To this end, the original objective of reward function D w 2 for π 2 is to optimize
[Equation]
But the expert demonstrations are only available under O E .
While during training, we can only utilizethe evolving data T π 1 ∼ ρ π 1 to learn π 2 and D w 2 .
Besides, as π 1 is sub-optimal, directly imitating T π 1 could reduce the performance of the optimal π 2 to that of π 1 .
So we use the importance-weightingto calibrate this dynamics mismatch, i.e.,
[Equation]
[Figure]
where α ( x ) ≜ ρ π E ( x ) ρ π 1 ( x ) is an importance-weighting factor [12].
So the current issue lies in how toestimate ρ π E ρ π 1 under O E .
To achieve this purpose, we need to bridge the expert demonstrations andthe evolving data.
Therefore, here we use these two data sets to train an adversarial model D w 1 in thesame way as D w 2 in the pretraining:
[Equation]
If we write the training criterion (5) in the form of integral, i.e.,
[Equation]
(6) then, by setting the derivative of the objective (6) to 0 ( ∂ L ∂D w 1 = 0 ), we can obtain the optimum D w 1 :
[Equation]
(7) in which the order of differentiation and integration was changed by the Leibniz rule.
Besides, wecan sufficiently train D w 1 using the evolving data (cid:101) T π 1 and the expert demonstrations (cid:101) T π E .
Then D w 1 154 will be good enough to estimate the importance-weighting factor, i.e.,
[Equation]
In this way, we can use D w 1 , which can connect demonstrations and evolving data, to calibrate thelearning process of D w 2 .
The final optimization objective for D w 2 is
[Equation]
In this way, D w 2 can effectively dig out the expert part of ρ π 1 and produce efficient rewards for π 2 .
4.2 Support Mismatch
So far the challenges have still been similar to homogeneously observable imitation learning.
However,our preliminary experiments demonstrated that merely importance-weighting is not enough to fixthe problem that occurred by the absence of interactions under O E .
So there exist some other issuesbetween the expert demonstrations and the evolving data.
To find out the underlying issues, we plotthe t-Distributed Stochastic Neighbor
Embedding (t-SNE) [34] visualizations of these two empiricaldistributions under O E on Hopper and Walker2d , as shown in Figure 5.
Twenty trajectories werecollected for both the expert demonstrations and the evolving data.
We can observe that there exist some high-density regions of demonstrations in which the evolving data do not cover; that is, thereexist some regions of the demonstrations that π 1 did not explore .
Wang et al.
[35] found a similarphenomenon in the standard IL setting.
On the other hand, the importance-weighting α cannotcalibrate this situation where ρ π E ρ π 1 = ∞ .
Expert π(a) Hopper (b) Walker2d
To formulate this problem, here we introduce the Support
Set of the occupancy measure:
Definition 1 (Support Set) .
The support set of a occupancy measure ρ is the subset of the domain containingthe elements which are not mapped to zero:
[Equation]
Due to the sub-optimality of π 1 , supp( ρ π E ) \ supp( ρ π 1 ) ̸
=ϕ (see Figure 4b).
We call this part the Latent Demonstration , defined as:
Definition 2 (Latent Demonstration) .
The latent demonstration H is the set of those x ∈ S ×
A that belong to therelative complement of supp( ρ π 1 ) in supp( ρ π E ) :
[Equation]
Also, another part of the demonstration is named the Observed Demonstration , defined as:
Definition 3 (Observed Demonstration) .
The observed demonstration O is the set of those x ∈ S ×Athat belong to the complement of H in supp( ρ π E ) :
[Equation]
Besides, the data outside of demonstrations should be non-expert data:
Definition 4 (Non-Expert Data) .
The non-expert data N is the set of those x ∈ S
×
A that out ofsupp( ρ π E ) :
[Equation]
In other words, the sub-optimality of π 1 will cause not only the dynamics mismatch, but also theappearance of the latent demonstration H .
We call the latter one the problem of Support Mismatch .
Intuitively, when π 2 → π E , we have H → ϕ , monotonously .
So in order to fix the support mismatchbetween ρ π E and ρ π 1 , guiding π 2 to find out H is the key .
In addition, the support mismatch problem can be viewed as an inverse problem of the Out Of
Distribution (OOD) problem that frequently occurred in offline RL setting [21], in which they tried toavoid supp( ρ π 1 ) \ supp( ρ π E ) instead.
4.3 Imitation Learning with Rejection
We can observe that H ∪ O ∪ N = S ×A .
So it is desirable to filter out H from O and N .
Meanwhile,
D w 1 and D w 2 can only classify O ∪ H and N , under O E and O L respectively.
Therefore, here wedesign two models g 1 : O E × A → { 0 , 1 } and g 2 :
O L × A → { 0 , 1 } (Output 0: x ∈ O and output1: otherwise), so that given x ∼ T (corresponding (cid:101) x ∼ (cid:101) T and x ∼ T ) they can satisfy:
[Equation]
[Equation] [Equation] [Equation]
[Equation]
respectively, where ⟨· , ·⟩ denotes the dot product and I ( · ) takes +1 if · > 0 . 5 , and − 1 otherwise.
Thetarget combined model [-
-]⟨ I ( D ∗ w ( x ) , g ∗ ( x ) ⟩ is depicted in Figure4c.
To this end, both g 1 and g 2 should be able to cover O , meanwhile g 2 can be adaptive to continuouslychange of ρ π 2 due to the update of π 2 .
Here we learn g 1 and g 2 in a rejection form, to reject O from
O ∪ H (where I ( D w ) = +1 ).
Concretely, the rejection setting is the same as that in Cortes et al.
Also inspired by Geifman et al.
[15], the optimization objective of the combination of D w and g is
[Equation]
17) where c > 0 denotes the target coverage, and λ denotes the factor for controlling the relativeimportance of rejection.
Besides, the empirical coverage ˆ ϕ ( g ) is defined as
[Equation]
where X = { x i } , i ∈ [ m ] .
The empirical rejection risk ˆ l ( D w , g ) is the ratio between the covered riskof the discriminator and the empirical coverage:
[Equation]
Meanwhile, both D w 1 and g 1 can access ρ π E under O E directly.
So given x ∼ T π 2 under O L , once 212 ⟨ I ( D w 2 ( x ) , g 2 ( x ) ⟩ = +1 , we can query the corresponding observations (cid:101) x of x through OC operationand use ⟨ I ( D w 1 ( (cid:101) x ) , g 1 ( (cid:101) x ) ⟩ to calibrate the output of g 2 and D w 2 .
In this way, g 2 and D w 2 can beentangled together and adaptively guide π 2 to find out the latent demonstrations H under O L .
4.4 IWRE
Here we combine the importance-weighting and rejection into a unified whole, to propose a novelalgorithm named Importance Weighting with REjection (IWRE).
Concretely, in a HOIL process:
Pretraining.
We train a discriminator D w 1 by Equation (5) and its corresponding rejection model g 1by Equation (17) using the evolving data and the expert demonstrations.
Training.
We train a discriminator D w 2 by the combination of Equation (9) and Equation (17), aswell as its corresponding rejection model g 2 by Equation (17), using the evolving data, the datacollected by π 2 , and the output of D w 1 with g 1 through OC operation.
Also, π 2 will be updated with D w 2 and g 2 asymmetrically as in GAIL.
The pseudo-code of our algorithm is provided in the supplementary material.
Experiment
In this section, we validate our algorithm in Atari 2600 [3] (GPL License) and MuJoCo [33](Academic License) environments.
The experiments were designed to investigate:
1) Can IWRE achieve significant performance under HOIL tasks?
2) Can IWRE deal with the support mismatch problem?
3) During training, is active querying for HOIL indeed necessary?
Below we first introduce the experimental setup and then investigate the above questions.
Moreresults and experimental details are included in the supplementary material.
5.1 Experimental Setup
Environments.
We choose three pixel-memory based games in Atari and five continuous controlobjects in MuJoCo on OpenAI platform [5] (MIT License).
Details as below:
Pixel-memory Atari games .
O E : 84 × 84 × 4 raw pixels; O L : 128 -byte random access memories
RAM).
Expert: converged DQN-based agents [24].
Atari games contain twototally isolated views: raw pixels and RAM, under the same state.
Through these environments, we want to investigate whether the agent can learn an effective policy fromdemonstrations under completely different observation spaces.
Moreover, IL with visualobservations only is already very difficult [7], while learning a RAM-based policy can beeven more challenging [3, 31], so few IL research reported desirable results on this task.
Continuous control MuJoCo objects .
O E : half of original observation features; O L : an-
other half of original observation features
Expert: converged DDPG-based agents [22].
The
[Figure]
features of MuJoCo contain monotonous information like the direction, position, velocity,etc., of an object.
Through these environments, we want to investigate whether the agentcan learn from demonstrations with complementary signals under observations with missinginformation.
Meanwhile, we make sure RL algorithms can obtain comparable performancesunder O E and O L .
More details are reported in the supplementary material.
Besides, twenty expert trajectories were collected for each environment.
Each result contains fivetrials with different random seeds.
All experiments were conducted on server clusters with NVIDIA
Tesla V100 GPUs.
The summary of the environments is gathered in the supplementary material.
Baselines.
Six basic contenders were included in the experiments: Vanilla GAIL [16], GAILwith importance-weighting [12] ( IW ), third-person IL [30] ( TPIL ), generative adversarial MDPalignment [19] ( GAMA ), behavioral cloning [2] ( BC ), and learning by cheating [8] ( LBC ).
For
IW, we utilized the discriminator D w 1 trained in the pretraining to calculate the importance weight;also
the optimization objective for D w 2 during training is the same as Equation (9); TPIL learns thethird-person demonstrations by leading the cross-entropy loss into the update of the feature extractor;
GAMA learns a mapping function ϕ in view of adversarial training to align the observation of thetarget domain into the source domain, and thereby can utilize the policy in the source domain forzero-shot imitation.
For fairness, we allowed the interaction between the policy and the environmentfor GAMA under HOIL; LBC uses π 1 learned from privileged states as a teacher to train π 2 in a
DAgger [27] style, so here we allowed LBC to access O E during the whole IL process.
In Atari, toinvestigate whether our method could achieve good performance for RAM-based control, we furtherincluded a contender PPO-RAM , which uses proximal policy optimization (PPO) [28] to perform
RL directly with environmental true rewards under the RAM-based observations.
More detailedsetup including query strategies for TPIL and GAMA, network architecture, and hyper-parametersare reported in the supplementary material.
Learning process.
To simulate the situation that O E is costly, the steps for training π 1 was set as1/4 of that for training π 2 , using GAIL [16]/HashReward [7] under the O E space for MuJoCo/Atarienvironments.
The learning steps were 10 7 for MuJoCo and 5 × 10 6 for Atari environments.
In thepretraining, we sampled 20 trajectories from π 1 , and the data from each trajectory had both O E and
O L observations.
In the training, each method learned 4 × 10 7 steps for MuJoCo and 2 × 10 7 stepsfor Atari under the O L space to obtain π 2 .
5.2 Results
Experimental results are reported in Figure 6.
Since the mapping function is hard to learn wheninput is RAM and output is raw images, we omit the results of GAMA in Atari.
We can observe thatwhile IW is better than GAIL in most environments, both GAIL and IW can hardly outperform π 1 .
Because they just imitated the performance of π 1 instead of π E , even with importance-weightingfor calibration.
For TPIL, its learning process was extremely unstable on Hopper , Swimmer , and
Walker2d due to the continuous distribution shift.
Furthermore, the performance of GAMA wasnot satisfactory in Hopper and Walker2d because its mapping function is hard to learn well whenthe support mismatch appears.
The results of TPIL and GAMA demonstrate that DSIL methodswill be invalid under heterogeneous observations as in HOIL tasks.
On Atari environments, O Econtains more privileged information than O L , so LBC can achieve good performance.
But when O Eis not more privileged than O L , like in most environments of MuJoCo, its performance will decreasedue to the support mismatch, which would make it even worse than BC.
Finally, IWRE obtainedthe best performance on 6/8 environments, and comparable performance with LBC on Reacher ,which shows the effectiveness of our method even with limited access to O E (LBC can access to
O E all the time).
Besides, we can see that the performance differences between the GAIL/IW and
IWRE/TPIL/GAMA/LBC are huge (especially on Reacher ) because of the absence of queries, whichdemonstrates that the query operation is indeed necessary for HOIL problems.
Moreover, even learned with true rewards, PPO-RAM surprisingly failed to achieve comparableperformance to IWRE, which shows that IWRE could possibly learn more effective rewards thantrue environmental rewards in RAM-input tasks.
The results verify that, IWRE provides a powerfulapproach for tackling HOIL problems, even under the situation that the demonstrations are gatheredfrom such a different observation space, meanwhile O E is strictly limited during training.
[Figure]
under O E .
The high-density regions of the expert demonstrations were covered by the collected data of t-SNE visualization of ρ π 2 and ρ π E under O E .
In Sec- tion 4.2, we point that the sub-optimality of π 1 will causethe problem of support mismatch, which is embodied asthe appearance of the latent demonstration H during training.
Also the empirical results in Figure 5 on Hopper and
Walker2d verify the existence of this problem.
So we wantto investigate whether the superiority of IWRE indeedcomes from successfully tackling the support mismatchproblem.
To this end, we plotted the t-SNE visualizationof the same expert demonstrations as in Section 4.2 andthe collected data of π 2 by IWRE under O E ( O E is hiddento π 2 ).
All setups are the same as in Section 4.2.
From theresults shown in Figure 7, we can see that even under O E ,which cannot be obtained by π 2 , almost all high-densityregions of the demonstrations were covered by the collected data.
Meanwhile, the latent demonstration H is dugout nearly.
The results demonstrate that IWRE basically solves the problem of support mismatch andthereby performs well in these environments.
Besides, some collected data of π 2 of IWRE were out of the distribution of the demonstrations,which means π 2 slightly overly explored the environment.
Since O E is hidden to π 2 , the rewardfunction will encourage π 2 to explore more areas to fix the support mismatch problem.
Meanwhile,the out-of-distribution problem in HOIL is not as severe as in the offline RL settings [21], so thisover-exploration phenomenon makes sense.
Conclusion
In this paper, we proposed a new learning framework named Heterogeneously Observable Imitation
Learning (HOIL), to formulate the situations where the observation space of demonstrations isdifferent from that of the imitator while learning.
We formally modeled a learning process of HOIL,in which the access to the observations of an expert is limited due to the high cost.
Furthermore,we analyzed underlying challenges during training, i.e., the dynamics mismatch and the supportmismatch, on the occupancy distributions between the demonstrations and the policy.
To tackle thesechallenges, we proposed a new algorithm named Importance Weighting with REjection (IWRE),using importance-weighting and learning with rejection.
Experimental results showed that the directimitation and domain adaptive methods could not solve this problem, while our approach obtainedpromising results.
In the future, we hope to involve the theoretical guarantee for our algorithm
IWRE and investigate how many O E do we need to query to learn a promising π 2 .
Furthermore,we hope to use the learning framework of HOIL and IWRE to tackle more learning scenarios withdemonstrations in different spaces.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seesupplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi
-mental results (either the supplemental material or as a URL)
Section 5.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 5.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]