Original: S EEING D IFFERENTLY , A CTING S IMILARLY : H ETEROGENEOUSLY O BSERVABLE I MITATION L EARNING
Revised: S EEING D IFFERENTLY , A CTING S IMILARLY : H ETEROGENEOUSLY O BSERVABLE I MITATION L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In many real-world imitation learning tasks, the demonstrator and the learner have to act under different observation spaces .
Revised: In many real-world imitation learning tasks, the demonstrator and the learner have to act under different observation spaces .

Original: This situation brings significant obstacles to existing imitation learning approaches, since most of them learn policies under homogeneous observation spaces .
Revised: This situation brings significant obstacles to existing imitation learning approaches, since most of them learn policies under homogeneous observation spaces .

Original: On the other hand, previous studies under different observation spaces have strong assumptions that these two observation spaces coexist during the entire learning process .
Revised: On the other hand, previous studies under different observation spaces have strong assumptions that these two observation spaces coexist during the entire learning process .

Original: However, in reality, the observation coexistence will be limited due to the high cost of acquiring expert observations.
Revised: However, in reality, the observation coexistence will be limited due to the high cost of acquiring expert observations.

Original: In this work, we study this challenging problem with limited observation coexistence under heterogeneous observations: Heterogeneously Observable Imitation Learning (HOIL).
Revised: In this work, we study this challenging problem with limited observation coexistence under heterogeneous observations: Heterogeneously Observable Imitation Learning (HOIL).

Original: We identify two underlying issues in HOIL: the dynamics mismatch and the support mismatch, and further propose the Importance Weighting with REjection (IWRE)
Revised: We identify two underlying issues in HOIL: the dynamics mismatch and the support mismatch, and further propose the Importance Weighting with REjection (IWRE)

Original: algorithm based on importance weighting and learning with rejection to solve HOIL problems.
Revised: algorithm based on importance weighting and learning with rejection to solve HOIL problems.

Original: Experimental results show that IWRE can solve various HOIL tasks, including the challenging tasks of transforming the vision-based demonstrations to random access memory (RAM)-based policies in the Atari domain, even with limited visual observations.
Revised: Experimental results show that IWRE can solve various HOIL tasks, including the challenging tasks of transforming the vision-based demonstrations to random access memory (RAM)-based policies in the Atari domain, even with limited visual observations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Imitation Learning (IL), which studies how to learn a good policy by imitating the given expert demonstrations (Ho & Ermon, 2016; Abbeel & Ng, 2010), has made significant progress in realworld applications such as autonomous driving (Chen et al., 2019), health care (Iyer et al., 2021), and continuous control (Kim et al., 2020).
Revised: Imitation Learning (IL), which studies how to learn a good policy by imitating the given expert demonstrations (Ho & Ermon, 2016; Abbeel & Ng, 2010), has made significant progress in realworld applications such as autonomous driving (Chen et al., 2019), health care (Iyer et al., 2021), and continuous control (Kim et al., 2020).

Original: In tradition, the expert and the learner are assumed to use the same observation space.
Revised: In tradition, the expert and the learner are assumed to use the same observation space.

Original: However, nowadays, many real-world IL tasks demand to remove this assumption (Chen et al., 2019; Warrington et al., 2021), such as in autonomous driving (Chen et al., 2019), recommendation system (Wu et al., 2019), and medical decision making (Wang et al., 2021).
Revised: However, nowadays, many real-world IL tasks demand to remove this assumption (Chen et al., 2019; Warrington et al., 2021), such as in autonomous driving (Chen et al., 2019), recommendation system (Wu et al., 2019), and medical decision making (Wang et al., 2021).

Original: Taking AI for medical diagnosis as an example in Figure 1: A medical AI is learning to make medical decisions based on expert doctor demonstrations.
Revised: Taking AI for medical diagnosis as an example in Figure 1: A medical AI is learning to make medical decisions based on expert doctor demonstrations.

Original: To ensure demonstration quality, the expert may use high-cost observations such as CT, MRI, and B-ultrasound.
Revised: To ensure demonstration quality, the expert may use high-cost observations such as CT, MRI, and B-ultrasound.

Original: In contrast, the AI learner is ideal to use only low-cost observations from cheaper devices, which could be newly designed ones that have not been used previously by the expert.
Revised: In contrast, the AI learner is ideal to use only low-cost observations from cheaper devices, which could be newly designed ones that have not been used previously by the expert.

Original: Meanwhile, to ensure reliability, it is also reasonable to allow the learner to access the high-cost observations during training under a limited budget (Yu et al., 2019).
Revised: Meanwhile, to ensure reliability, it is also reasonable to allow the learner to access the high-cost observations during training under a limited budget (Yu et al., 2019).

Original: The above examples share three characteristics: (1) Even though a pair of expert and learner observations can be different, they are under the same state of the environment, leading to similar policies; (2) The learner’s new observations are not available to the expert when generating demonstrations; (3) During training, the learner can only access expert observations under a limited budget, in special the high-cost ones, since it is also important to minimize the usage of the expert observations during training to avoid unnecessary costs.
Revised: The above examples share three characteristics: (1) Even though a pair of expert and learner observations can be different, they are under the same state of the environment, leading to similar policies; (2) The learner’s new observations are not available to the expert when generating demonstrations; (3) During training, the learner can only access expert observations under a limited budget, in special the high-cost ones, since it is also important to minimize the usage of the expert observations during training to avoid unnecessary costs.

Original: We name such IL tasks Heterogeneously
Revised: We name such IL tasks Heterogeneously

Original: Observable Imitation Learning (HOIL).
Revised: Observable Imitation Learning (HOIL).

Original: Among them, we focus on the most challenging HOIL setting in which the expert and learner observation spaces have no overlap .
Revised: Among them, we focus on the most challenging HOIL setting in which the expert and learner observation spaces have no overlap .

Original: There are two lines of research studying the related problems, summarized in Table 1 and Figure 2.
Revised: There are two lines of research studying the related problems, summarized in Table 1 and Figure 2.

Original: The first one relates to Domain-shifted IL (DSIL): the observation spaces of experts and learners are the homogeneous , while some typical mismatches of distributions could exist: morphological mismatch, viewpoint mismatch, and dynamics mismatch (Stadie et al., 2017; Raychaudhuri et al.,
Revised: The first one relates to Domain-shifted IL (DSIL): the observation spaces of experts and learners are the homogeneous , while some typical mismatches of distributions could exist: morphological mismatch, viewpoint mismatch, and dynamics mismatch (Stadie et al., 2017; Raychaudhuri et al.,

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: However, the approaches for DSIL are invalid when the observation spaces of experts and learners are heterogeneous as in HOIL.
Revised: However, the approaches for DSIL are invalid when the observation spaces of experts and learners are heterogeneous as in HOIL.

Original: The second line studied IL under different observations similar to HOIL.
Revised: The second line studied IL under different observations similar to HOIL.

Original: Some representative works include Partially Observable IL (POIL) (Gangwani et al., 2019;
Revised: Some representative works include Partially Observable IL (POIL) (Gangwani et al., 2019;

Original: Warrington et al., 2021) and Learning by Cheating (LBC) (Chen et al., 2019).
Revised: Warrington et al., 2021) and Learning by Cheating (LBC) (Chen et al., 2019).

Original: Both POIL and LBC assume that the expert’s observations can be easily accessed by the learner without any budget limit.
Revised: Both POIL and LBC assume that the expert’s observations can be easily accessed by the learner without any budget limit.

Original: However, in practice, different from the learner observations, the access to expert’s observations might be of high cost, invasive, and even unavailable (Yu et al., 2019), which hinder the wide application of these methods.
Revised: However, in practice, different from the learner observations, the access to expert’s observations might be of high cost, invasive, and even unavailable (Yu et al., 2019), which hinder the wide application of these methods.

Original: In this paper, we initialize the study of the HOIL problem.
Revised: In this paper, we initialize the study of the HOIL problem.

Original: We propose a learning process across observation spaces of experts and learners to solve this problem.
Revised: We propose a learning process across observation spaces of experts and learners to solve this problem.

Original: We further analyze the underlying issues of HOIL, i.e., the dynamics mismatch and the support mismatch.
Revised: We further analyze the underlying issues of HOIL, i.e., the dynamics mismatch and the support mismatch.

Original: To tackle both two issues, we use the techniques of importance weighting (Shimodaira, 2000; Fang et al., 2020) and learning with rejection (Cortes et al., 2016; Geifman & El-Yaniv, 2019) for active querying to propose the
Revised: To tackle both two issues, we use the techniques of importance weighting (Shimodaira, 2000; Fang et al., 2020) and learning with rejection (Cortes et al., 2016; Geifman & El-Yaniv, 2019) for active querying to propose the

Original: Importance Weighting with REjection (IWRE) approach.
Revised: Importance Weighting with REjection (IWRE) approach.

Original: We evaluate the effectiveness of the IWRE algorithm in continuous control tasks of MuJoCo (Todorov et al., 2012), and the challenging tasks of learning random access memory (RAM)-based policies from vision-based expert demonstrations in Atari (Bellemare et al., 2013) games.
Revised: We evaluate the effectiveness of the IWRE algorithm in continuous control tasks of MuJoCo (Todorov et al., 2012), and the challenging tasks of learning random access memory (RAM)-based policies from vision-based expert demonstrations in Atari (Bellemare et al., 2013) games.

Original: The results demonstrate that IWRE can significantly outperform existing IL algorithms in HOIL tasks, with limited access to expert observations.
Revised: The results demonstrate that IWRE can significantly outperform existing IL algorithms in HOIL tasks, with limited access to expert observations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: DSIL.
Revised: DSIL.

Original: For the standard IL process, where the learner and the expert share the same observation space, current state-of-the-art methods tend to learn the policy in an adversarial style (Cai et al., 2021), like Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016).
Revised: For the standard IL process, where the learner and the expert share the same observation space, current state-of-the-art methods tend to learn the policy in an adversarial style (Cai et al., 2021), like Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016).

Original: When considering the domain mismatch problem, i.e., DSIL, the research aims at addressing the static distributional shift of the optimal policies resulted from the environmental differences but still under homogeneous observation spaces.
Revised: When considering the domain mismatch problem, i.e., DSIL, the research aims at addressing the static distributional shift of the optimal policies resulted from the environmental differences but still under homogeneous observation spaces.

Original: Stadie et al.
Revised: Stadie et al.

Original: (2017), Sermanet et al.
Revised: (2017), Sermanet et al.

Original: (2018), and Liu et al.
Revised: (2018), and Liu et al.

Original: (2018) studied the situation where the demonstrations are in view of a third person.
Revised: (2018) studied the situation where the demonstrations are in view of a third person.

Original: Kim et al.
Revised: Kim et al.

Original: (2020) and
Revised: (2020) and

Original: Kim et al.
Revised: Kim et al.

Original: (2019) addressed the IL problem with morphological mismatch between the expert’s and learner’s environment.
Revised: (2019) addressed the IL problem with morphological mismatch between the expert’s and learner’s environment.

Original: Stadie et al.
Revised: Stadie et al.

Original: (2017), Tirinzoni et al.
Revised: (2017), Tirinzoni et al.

Original: (2018), and Desai et al.
Revised: (2018), and Desai et al.

Original: (2020) focused on the calibration for the mismatch between simulators and the real world through some transfer learning styles.
Revised: (2020) focused on the calibration for the mismatch between simulators and the real world through some transfer learning styles.

Original: There are two major differences between HOIL and DSIL:
Revised: There are two major differences between HOIL and DSIL:

Original: One is that HOIL considers heterogeneous observation spaces instead of homogeneous ones; another is that without observation heterogeneity, DSIL can directly align two fixed domains, which may not be realistic for solving HOIL when two observation spaces are totally different.
Revised: One is that HOIL considers heterogeneous observation spaces instead of homogeneous ones; another is that without observation heterogeneity, DSIL can directly align two fixed domains, which may not be realistic for solving HOIL when two observation spaces are totally different.

Original: Thus HOIL is a significantly more challenging problem than DSIL.
Revised: Thus HOIL is a significantly more challenging problem than DSIL.

Original: Besides, Chen et al.
Revised: Besides, Chen et al.

Original: (2019) learned a vision-based agent from a privileged expert.
Revised: (2019) learned a vision-based agent from a privileged expert.

Original: But it can obtain expert’s observations throughout the whole learning process, so it cannot handle the problem of the support mismatch under HOIL.
Revised: But it can obtain expert’s observations throughout the whole learning process, so it cannot handle the problem of the support mismatch under HOIL.

Original: POMDP.
Revised: POMDP.

Original: The problem of POMDPs, in which only partial observations are available for the agent(s), has been studied in the context of multi-agent (Omidshafiei et al., 2017;
Revised: The problem of POMDPs, in which only partial observations are available for the agent(s), has been studied in the context of multi-agent (Omidshafiei et al., 2017;

Original: Warrington et al., 2021) and imitation learning (Gangwani et al., 2019; Warrington et al., 2021) problems.
Revised: Warrington et al., 2021) and imitation learning (Gangwani et al., 2019; Warrington et al., 2021) problems.

Original: But distinct from
Revised: But distinct from

Original: HOIL, in a POMDP, the learner only have partial observations and share a same underlying observation space with the expert, which would become an obstacle for them to make decisions correctly.
Revised: HOIL, in a POMDP, the learner only have partial observations and share a same underlying observation space with the expert, which would become an obstacle for them to make decisions correctly.

Original: For example, Warrington et al.
Revised: For example, Warrington et al.

Original: (2021) assumed that the observation of the learner is partial than that
Revised: (2021) assumed that the observation of the learner is partial than that

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: of the expert.
Revised: of the expert.

Original: Instead, in HOIL, expert’s and learner’s observations are totally different from each other, while the learner’s observations are not belong to a part of the expert’s.
Revised: Instead, in HOIL, expert’s and learner’s observations are totally different from each other, while the learner’s observations are not belong to a part of the expert’s.

Original: For HOIL, the main challenge is to deal with the mismatches between the observation spaces, especially when the access to expert’s observations is strictly limited.
Revised: For HOIL, the main challenge is to deal with the mismatches between the observation spaces, especially when the access to expert’s observations is strictly limited.

Original:  T HE HOIL P ROBLEM
Revised:  T HE HOIL P ROBLEM

Original: In this section, we first give a formal definition of the HOIL setting, and then introduce the learning process for solving the HOIL problem.
Revised: In this section, we first give a formal definition of the HOIL setting, and then introduce the learning process for solving the HOIL problem.

Original: 3. S ETTING D EFINITION
Revised: 3. S ETTING D EFINITION

Original: A HOIL problem is defined within a Markov decision process with mutiple observation spaces, i.e., ⟨S , {O} , A , P , γ ⟩ , where S denotes the state space, {O} denotes a set of observation spaces, A denotes the action space, P : S ×
Revised: A HOIL problem is defined within a Markov decision process with mutiple observation spaces, i.e., ⟨S , {O} , A , P , γ ⟩ , where S denotes the state space, {O} denotes a set of observation spaces, A denotes the action space, P : S ×

Original: A × S → R denotes the transition probability distribution of the state and action, and γ ∈ (0 , 1] denotes the discount factor.
Revised: A × S → R denotes the transition probability distribution of the state and action, and γ ∈ (0 , 1] denotes the discount factor.

Original: Furthermore, a policy π over an observation space O is defined as a function mapping from O to A , and we denote by Π O the set of all policies over O .
Revised: Furthermore, a policy π over an observation space O is defined as a function mapping from O to A , and we denote by Π O the set of all policies over O .

Original: In HOIL, both the expert and the learner have their own observation spaces, which are denoted as O E and O L respectively.
Revised: In HOIL, both the expert and the learner have their own observation spaces, which are denoted as O E and O L respectively.

Original: Both O E and O L are assumed to be produced by two bijective mappings
Revised: Both O E and O L are assumed to be produced by two bijective mappings

Original: f E : S → O E , f L : S → O L , which are unknown functions mapping the underlying true states to the observations.
Revised: f E : S → O E , f L : S → O L , which are unknown functions mapping the underlying true states to the observations.

Original: It is obvious to see that by this assumption, any policy over
Revised: It is obvious to see that by this assumption, any policy over

Original: O E has a unique correspondence over O L .
Revised: O E has a unique correspondence over O L .

Original: This makes HOIL possible since the target of HOIL is to find the corresponding policy of the expert policy under O L .
Revised: This makes HOIL possible since the target of HOIL is to find the corresponding policy of the expert policy under O L .

Original: A state-action pair ( s, a ) , denoted by x , is called an instance .
Revised: A state-action pair ( s, a ) , denoted by x , is called an instance .

Original: Also, a trajectory T = { x i } , i ∈
Revised: Also, a trajectory T = { x i } , i ∈

Original: [ m ] is a set of m instances.
Revised: [ m ] is a set of m instances.

Original: For each observation space, (cid:101) x ∈ (cid:101) T ⊆
Revised: For each observation space, (cid:101) x ∈ (cid:101) T ⊆

Original: O E × A and x ∈ T ⊆
Revised: O E × A and x ∈ T ⊆

Original: O L × A , where
Revised: O L × A , where

Original: O E = f E ( S ) and O L = f L ( S ) .
Revised: O E = f E ( S ) and O L = f L ( S ) .

Original: Furthermore, we define the occupancy measure of a policy π under the state space S as ρ π : S×A → R such that ρ π ( x ) =
Revised: Furthermore, we define the occupancy measure of a policy π under the state space S as ρ π : S×A → R such that ρ π ( x ) =

Original: π ( a | o )
Revised: π ( a | o )

Original: Pr( o | s ) (cid:80) ∞ t =0 γ t Pr( s t = s | π ) .
Revised: Pr( o | s ) (cid:80) ∞ t =0 γ t Pr( s t = s | π ) .

Original: Under
Revised: Under

Original: HOIL, the learner accesses the expert demonstrations (cid:101) T π E , a set of instances sampled from ρ π E .
Revised: HOIL, the learner accesses the expert demonstrations (cid:101) T π E , a set of instances sampled from ρ π E .

Original: The goal of HOIL is to learn a policy ˆ π as the corresponding policy of π E over O L .
Revised: The goal of HOIL is to learn a policy ˆ π as the corresponding policy of π E over O L .

Original: If O E = O L ,
Revised: If O E = O L ,

Original: HOIL degenerates to standard IL .
Revised: HOIL degenerates to standard IL .

Original: GAIL (Ho & Ermon, 2016) is one of the state-of-the-art IL approaches under this situation, which tries to minimize the divergence between the learner’s and the expert’s occupancy measures d ( ρ ˆ π , ρ π E ) .
Revised: GAIL (Ho & Ermon, 2016) is one of the state-of-the-art IL approaches under this situation, which tries to minimize the divergence between the learner’s and the expert’s occupancy measures d ( ρ ˆ π , ρ π E ) .

Original: The objective of GAIL is
Revised: The objective of GAIL is

Original: [Equation]
Revised: [Equation]

Original: where H (ˆ π ) is the causal entropy performed as a regularization term, and D w : O E ×
Revised: where H (ˆ π ) is the causal entropy performed as a regularization term, and D w : O E ×

Original: A → [0 , 1] is the discriminator of π E and ˆ π .
Revised: A → [0 , 1] is the discriminator of π E and ˆ π .

Original: GAIL solved Equation equation 1 by alternatively taking a gradient ascent step to train the discriminator D w , and a minimization step to learn policy ˆ π based on an off-the-shelf RL algorithm with the pseudo reward − log D w ( (cid:101) x ) .
Revised: GAIL solved Equation equation 1 by alternatively taking a gradient ascent step to train the discriminator D w , and a minimization step to learn policy ˆ π based on an off-the-shelf RL algorithm with the pseudo reward − log D w ( (cid:101) x ) .

Original: 3. T HE L EARNING P ROCESS FOR S OLVING HOIL
Revised: 3. T HE L EARNING P ROCESS FOR S OLVING HOIL

Original: In HOIL, we need to cope with the absence of the learner’s observations in demonstrations and the high cost of collecting the expert’s observations while learning.
Revised: In HOIL, we need to cope with the absence of the learner’s observations in demonstrations and the high cost of collecting the expert’s observations while learning.

Original: So we introduce a learning process with pretraining across two different observation spaces for solving HOIL, as abstracted in Figure 3.
Revised: So we introduce a learning process with pretraining across two different observation spaces for solving HOIL, as abstracted in Figure 3.

Original: Pretraining.
Revised: Pretraining.

Original: Same to LBC (Chen et al., 2019), we assume that we can obtain an auxiliary policy π 1 based on O E at the beginning.
Revised: Same to LBC (Chen et al., 2019), we assume that we can obtain an auxiliary policy π 1 based on O E at the beginning.

Original: π 1 can be directly provided by any sources, or trained by GAIL or behavior cloning as did in LBC.
Revised: π 1 can be directly provided by any sources, or trained by GAIL or behavior cloning as did in LBC.

Original: Besides, we use this π 1 to sample some data T π 1 , which contain both observation under O E (i.e., (cid:101) T π 1 ) and O L (i.e., T π 1 ), in order to connect these two different observation spaces.
Revised: Besides, we use this π 1 to sample some data T π 1 , which contain both observation under O E (i.e., (cid:101) T π 1 ) and O L (i.e., T π 1 ), in order to connect these two different observation spaces.

Original: We name T π 1 = { (cid:101) T π 1 , T π 1 } the initial data .
Revised: We name T π 1 = { (cid:101) T π 1 , T π 1 } the initial data .

Original: [Figure]
Revised: [Figure]

Original: Training.
Revised: Training.

Original: Here we learn a policy π 2 from the initial data T π 1 and the collected data T π 2 , under
Revised: Here we learn a policy π 2 from the initial data T π 1 and the collected data T π 2 , under

Original: O L only.
Revised: O L only.

Original: Besides, the learner is allowed for some operation of observation coexistence (OC): At some steps of learning, besides the observations
Revised: Besides, the learner is allowed for some operation of observation coexistence (OC): At some steps of learning, besides the observations

Original: O L , the learner could also request (cid:101) T π 2 from the corresponding observations O E (e.g., from the human-understandable sensors).
Revised: O L , the learner could also request (cid:101) T π 2 from the corresponding observations O E (e.g., from the human-understandable sensors).

Original: The final objective of HOIL is to learn a good policy π 2 under O L .
Revised: The final objective of HOIL is to learn a good policy π 2 under O L .

Original: In practical applications, the auxiliary policy π 1 can also come from simulation training or direct imitation.
Revised: In practical applications, the auxiliary policy π 1 can also come from simulation training or direct imitation.

Original: But since π 1 is additionally provided, it is more practical to consider π 1 as a non-optimal policy.
Revised: But since π 1 is additionally provided, it is more practical to consider π 1 as a non-optimal policy.

Original: During training, OC is an essential operation for solving HOIL, which helps the learner address the issues of the dynamics mismatch and the support mismatch (especially the latter one).
Revised: During training, OC is an essential operation for solving HOIL, which helps the learner address the issues of the dynamics mismatch and the support mismatch (especially the latter one).

Original: Also, in reality, we do not need an oracle for actions, which still needs OC for obtaining expert observations first, as in many active querying research (Brantley et al., 2020; Chen et al., 2019), so its cost will be relatively lower.
Revised: Also, in reality, we do not need an oracle for actions, which still needs OC for obtaining expert observations first, as in many active querying research (Brantley et al., 2020; Chen et al., 2019), so its cost will be relatively lower.

Original: Besides, the related work (Chen et al., 2019) also required an initialized policy π 1 to solve their problem, which act as a teacher under privileged O E in the pretraining and then learned a visionbased student from the guidance of the teacher under both O L and O E .
Revised: Besides, the related work (Chen et al., 2019) also required an initialized policy π 1 to solve their problem, which act as a teacher under privileged O E in the pretraining and then learned a visionbased student from the guidance of the teacher under both O L and O E .

Original: Their setting can be viewed as a variety of HOIL with optimal π 1 , unlimited O E , and unlimited OC operations, so HOIL is actually a more practical learning framework.
Revised: Their setting can be viewed as a variety of HOIL with optimal π 1 , unlimited O E , and unlimited OC operations, so HOIL is actually a more practical learning framework.

Original:  I MITATION L EARNING WITH I MPORTANCE W EIGHTING AND R EJECTION
Revised:  I MITATION L EARNING WITH I MPORTANCE W EIGHTING AND R EJECTION

Original: In HOIL, the access frequency to O E is strictly limited, so it is unrealistic to learn π 2 in a Dataset
Revised: In HOIL, the access frequency to O E is strictly limited, so it is unrealistic to learn π 2 in a Dataset

Original: Aggregation (DAgger) style (Ross et al., 2011) as in LBC.
Revised: Aggregation (DAgger) style (Ross et al., 2011) as in LBC.

Original: Therefore, we resort to learning π 2 with a learned reward function by inverse reinforcement learning (Abbeel & Ng, 2010) in an adversarial learning style (Ho & Ermon, 2016; Fu et al., 2018).
Revised: Therefore, we resort to learning π 2 with a learned reward function by inverse reinforcement learning (Abbeel & Ng, 2010) in an adversarial learning style (Ho & Ermon, 2016; Fu et al., 2018).

Original: In addition, both O E and O L are assumed to share the same latent state space S as introduced in Section 3.1, so the following analysis will be based on S , while the algorithm will handle the problem based on O E and O L specifically.
Revised: In addition, both O E and O L are assumed to share the same latent state space S as introduced in Section 3.1, so the following analysis will be based on S , while the algorithm will handle the problem based on O E and O L specifically.

Original: 4. D YNAMICS M ISMATCH AND I MPORTANCE W EIGHTING
Revised: 4. D YNAMICS M ISMATCH AND I MPORTANCE W EIGHTING

Original: To analyze the learning process, we let ρ π E , ρ π 1 , and ρ π 2 be the occupancy measure distributions of the expert demonstrations, the initial data, and the data during training respectively.
Revised: To analyze the learning process, we let ρ π E , ρ π 1 , and ρ π 2 be the occupancy measure distributions of the expert demonstrations, the initial data, and the data during training respectively.

Original: Since we need to consider the sub-optimality of π 1 , ρ π 1 should be a mixture distribution of the expert ρ π E and non-expert ρ π NE , i.e., there exists some δ ∈ (0 , 1) such that
Revised: Since we need to consider the sub-optimality of π 1 , ρ π 1 should be a mixture distribution of the expert ρ π E and non-expert ρ π NE , i.e., there exists some δ ∈ (0 , 1) such that

Original: [Equation]
Revised: [Equation]

Original: as depicted in Figure 4a.
Revised: as depicted in Figure 4a.

Original: During training, the original objective of π 2 is to imitate π E through demonstrations.
Revised: During training, the original objective of π 2 is to imitate π E through demonstrations.

Original: To this end, the original objective of reward function D w 2 for π 2 is to optimize
Revised: To this end, the original objective of reward function D w 2 for π 2 is to optimize

Original: [Equation]
Revised: [Equation]

Original: But the expert demonstrations are only available under O E .
Revised: But the expert demonstrations are only available under O E .

Original: While during training, we can only utilize the initial data T π 1 ∼ ρ π 1 to learn π 2 and D w 2 .
Revised: While during training, we can only utilize the initial data T π 1 ∼ ρ π 1 to learn π 2 and D w 2 .

Original: Besides, as π 1 is sub-optimal, directly imitating T π 1 could reduce the performance of the optimal π 2 to that of π 1 .
Revised: Besides, as π 1 is sub-optimal, directly imitating T π 1 could reduce the performance of the optimal π 2 to that of π 1 .

Original: So we use the importance weighting to calibrate this dynamics mismatch, i.e.,
Revised: So we use the importance weighting to calibrate this dynamics mismatch, i.e.,

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: where α ( x ) ≜ ρ π E ( x ) ρ π 1 ( x ) is an importance weighting factor (Fang et al., 2020).
Revised: where α ( x ) ≜ ρ π E ( x ) ρ π 1 ( x ) is an importance weighting factor (Fang et al., 2020).

Original: So the current issue lies in how to estimate ρ π E ρ π 1 under O E .
Revised: So the current issue lies in how to estimate ρ π E ρ π 1 under O E .

Original: To achieve this purpose, we need to bridge the expert demonstrations and the initial data.
Revised: To achieve this purpose, we need to bridge the expert demonstrations and the initial data.

Original: Therefore, here we use these two data sets to train an adversarial model
Revised: Therefore, here we use these two data sets to train an adversarial model

Original: D w 1 in the same way as D w 2 in the pretraining:
Revised: D w 1 in the same way as D w 2 in the pretraining:

Original: [Equation]
Revised: [Equation]

Original: If we write the training criterion equation 5 in the form of integral, i.e.,
Revised: If we write the training criterion equation 5 in the form of integral, i.e.,

Original: [Equation]
Revised: [Equation]

Original: then, by setting the derivative of the objective to 0 ( ∂ L ∂D w 1 = 0 ), we can obtain the optimum D w 1 :
Revised: then, by setting the derivative of the objective to 0 ( ∂ L ∂D w 1 = 0 ), we can obtain the optimum D w 1 :

Original: [Equation]
Revised: [Equation]

Original: in which the order of differentiation and integration was changed by the Leibniz rule.
Revised: in which the order of differentiation and integration was changed by the Leibniz rule.

Original: Besides, we can sufficiently train D w 1 using the initial data (cid:101) T π 1 and the expert demonstrations (cid:101) T π E .
Revised: Besides, we can sufficiently train D w 1 using the initial data (cid:101) T π 1 and the expert demonstrations (cid:101) T π E .

Original: Then D wwill be good enough to estimate the importance weighting factor, i.e.,
Revised: Then D wwill be good enough to estimate the importance weighting factor, i.e.,

Original: [Equation]
Revised: [Equation]

Original: In this way, we can use D w 1 , which can connect demonstrations and initial data, to calibrate the learning process of D w 2 .
Revised: In this way, we can use D w 1 , which can connect demonstrations and initial data, to calibrate the learning process of D w 2 .

Original: The final optimization objective for D w 2 is
Revised: The final optimization objective for D w 2 is

Original: [Equation]
Revised: [Equation]

Original: In this way, D w 2 can effectively dig out the expert part of ρ π 1 and produce efficient rewards for π 2 .
Revised: In this way, D w 2 can effectively dig out the expert part of ρ π 1 and produce efficient rewards for π 2 .

Original: 4. S UPPORT M ISMATCH
Revised: 4. S UPPORT M ISMATCH

Original: So far the challenges have still been similar to homogeneously observable imitation learning.
Revised: So far the challenges have still been similar to homogeneously observable imitation learning.

Original: However, our preliminary experiments demonstrated that merely importance weighting is not enough to fix the problem that occurred by the absence of interactions under O E .
Revised: However, our preliminary experiments demonstrated that merely importance weighting is not enough to fix the problem that occurred by the absence of interactions under O E .

Original: So there exist some other issues between the expert demonstrations and the initial data.
Revised: So there exist some other issues between the expert demonstrations and the initial data.

Original: To find out the underlying issues, we plot the t-Distributed Stochastic Neighbor Embedding
Revised: To find out the underlying issues, we plot the t-Distributed Stochastic Neighbor Embedding

Original: ) (van der Maaten & Hinton, 2008) visualizations of these two empirical distributions under O E on Hopper and Walker2d , as shown in
Revised: ) (van der Maaten & Hinton, 2008) visualizations of these two empirical distributions under O E on Hopper and Walker2d , as shown in

Original: Figure 5.
Revised: Figure 5.

Original: Twenty trajectories were collected for both the expert demonstrations and the initial data.
Revised: Twenty trajectories were collected for both the expert demonstrations and the initial data.

Original: We can observe that there exist some high-density regions of demonstrations in which the initial data do not cover; that is, there exist some regions of the demonstrations that π 1 did not explore .
Revised: We can observe that there exist some high-density regions of demonstrations in which the initial data do not cover; that is, there exist some regions of the demonstrations that π 1 did not explore .

Original: Wang et al.
Revised: Wang et al.

Original: (2019) found a similar phenomenon in the standard IL setting.
Revised: (2019) found a similar phenomenon in the standard IL setting.

Original: On the other hand, the importance weighting α cannot calibrate this situation where ρ
Revised: On the other hand, the importance weighting α cannot calibrate this situation where ρ

Original: [Figure]
Revised: [Figure]

Original: To formulate this problem, here we introduce the Support
Revised: To formulate this problem, here we introduce the Support

Original: Set of the occupancy measure:
Revised: Set of the occupancy measure:

Original: Definition 1 (Support Set) .
Revised: Definition 1 (Support Set) .

Original: The support set of a occupancy measure ρ is the subset of the domain containing the elements which are not mapped to zero:
Revised: The support set of a occupancy measure ρ is the subset of the domain containing the elements which are not mapped to zero:

Original: [Equation]
Revised: [Equation]

Original: Due to the sub-optimality of π 1 ,
Revised: Due to the sub-optimality of π 1 ,

Original: [Equation]
Revised: [Equation]

Original: supp( ρ π 1 ) ̸
Revised: supp( ρ π 1 ) ̸

Original: = ∅ (see Figure 4b).
Revised: = ∅ (see Figure 4b).

Original: We call this part the Latent Demonstration , defined as:
Revised: We call this part the Latent Demonstration , defined as:

Original: Definition 2 (Latent Demonstration) .
Revised: Definition 2 (Latent Demonstration) .

Original: The latent demonstration H is the set of the domain that belong to the relative complement of supp( ρ π 1 ) in supp( ρ π E ) :
Revised: The latent demonstration H is the set of the domain that belong to the relative complement of supp( ρ π 1 ) in supp( ρ π E ) :

Original: [Equation]
Revised: [Equation]

Original: Also, another part of the demonstration is named the Observed Demonstration , defined as:
Revised: Also, another part of the demonstration is named the Observed Demonstration , defined as:

Original: Definition 3 (Observed Demonstration) .
Revised: Definition 3 (Observed Demonstration) .

Original: The observed demonstration O is the set of the domain that belong to the complement of H in supp( ρ π E ) :
Revised: The observed demonstration O is the set of the domain that belong to the complement of H in supp( ρ π E ) :

Original: [Equation]
Revised: [Equation]

Original: Besides, the data outside of demonstrations should be non-expert data:
Revised: Besides, the data outside of demonstrations should be non-expert data:

Original: Definition 4 (Non-Expert Data) .
Revised: Definition 4 (Non-Expert Data) .

Original: The non-expert data N is the set of the domain out of supp( ρ π E ) :
Revised: The non-expert data N is the set of the domain out of supp( ρ π E ) :

Original: [Equation]
Revised: [Equation]

Original: In other words, the sub-optimality of π 1 will cause not only the dynamics mismatch, but also the appearance of the latent demonstration H .
Revised: In other words, the sub-optimality of π 1 will cause not only the dynamics mismatch, but also the appearance of the latent demonstration H .

Original: We call the latter one the problem of Support Mismatch .
Revised: We call the latter one the problem of Support Mismatch .

Original: Intuitively, when π 2 → π E , we have H → ∅ , monotonously .
Revised: Intuitively, when π 2 → π E , we have H → ∅ , monotonously .

Original: So in order to fix the support mismatch between ρ π E and ρ π 1 , guiding π 2 to find out H is the key .
Revised: So in order to fix the support mismatch between ρ π E and ρ π 1 , guiding π 2 to find out H is the key .

Original: In addition, the support mismatch problem can be viewed as an inverse problem of the Out Of
Revised: In addition, the support mismatch problem can be viewed as an inverse problem of the Out Of

Original: Distribution (OOD) problem that frequently occurred in offline RL setting (Levine et al., 2020), in which they tried to avoid supp( ρ π 1 ) \ supp( ρ π E ) instead.
Revised: Distribution (OOD) problem that frequently occurred in offline RL setting (Levine et al., 2020), in which they tried to avoid supp( ρ π 1 ) \ supp( ρ π E ) instead.

Original: 4. I MITATION L EARNING WITH R EJECTION
Revised: 4. I MITATION L EARNING WITH R EJECTION

Original: We can observe that H ∪ O ∪ N = S×A .
Revised: We can observe that H ∪ O ∪ N = S×A .

Original: So it is desirable to filter out H from O and N .
Revised: So it is desirable to filter out H from O and N .

Original: Meanwhile,
Revised: Meanwhile,

Original: D w 1 and D w 2 can only classify O ∪ H and N , under O E and O L respectively.
Revised: D w 1 and D w 2 can only classify O ∪ H and N , under O E and O L respectively.

Original: Therefore, here we design two models g 1 : O E × A → { 0 , 1 } and g 2 :
Revised: Therefore, here we design two models g 1 : O E × A → { 0 , 1 } and g 2 :

Original: O L × A → { 0 , 1 } (Output 0: x ∈ O and output 1: otherwise), so that given x ∼ T (corresponding (cid:101) x ∼ (cid:101) T and x ∼ T ) they can satisfy:
Revised: O L × A → { 0 , 1 } (Output 0: x ∈ O and output 1: otherwise), so that given x ∼ T (corresponding (cid:101) x ∼ (cid:101) T and x ∼ T ) they can satisfy:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: respectively, where I [ · ] takes +1 if · > 0 . 5 , and − 1 otherwise.
Revised: respectively, where I [ · ] takes +1 if · > 0 . 5 , and − 1 otherwise.

Original: The target combined model I [ D ∗ w ( x )]
Revised: The target combined model I [ D ∗ w ( x )]

Original: g ∗ ( x ) is depicted in Figure4c.
Revised: g ∗ ( x ) is depicted in Figure4c.

Original: To this end, both g 1 and g 2 should be able to cover O , meanwhile g 2 can be adaptive to continuously change of ρ π 2 due to the update of π 2 .
Revised: To this end, both g 1 and g 2 should be able to cover O , meanwhile g 2 can be adaptive to continuously change of ρ π 2 due to the update of π 2 .

Original: Here we learn g 1 and g 2 in a rejection form, to reject O from
Revised: Here we learn g 1 and g 2 in a rejection form, to reject O from

Original: O ∪ H (where I ( D w ) = +1 ).
Revised: O ∪ H (where I ( D w ) = +1 ).

Original: Concretely, the rejection setting is the same as that in Cortes et al.
Revised: Concretely, the rejection setting is the same as that in Cortes et al.

Original: Also inspired by Geifman & El-Yaniv (2019), the optimization objective of the combination of D w and g is
Revised: Also inspired by Geifman & El-Yaniv (2019), the optimization objective of the combination of D w and g is

Original: [Equation]
Revised: [Equation]

Original: where c > 0 denotes the target coverage, and λ denotes the factor for controlling the relative importance of rejection.
Revised: where c > 0 denotes the target coverage, and λ denotes the factor for controlling the relative importance of rejection.

Original: Besides, the empirical coverage ˆ ϕ ( g ) is defined as
Revised: Besides, the empirical coverage ˆ ϕ ( g ) is defined as

Original: [Equation]
Revised: [Equation]

Original: with a batch of data X = { x i } , i ∈ [ m ] .
Revised: with a batch of data X = { x i } , i ∈ [ m ] .

Original: The empirical rejection risk ˆ l ( D w , g ) is the ratio between the covered risk of the discriminator and the empirical coverage:
Revised: The empirical rejection risk ˆ l ( D w , g ) is the ratio between the covered risk of the discriminator and the empirical coverage:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, both D w 1 and g 1 can access ρ π E under O E directly.
Revised: Meanwhile, both D w 1 and g 1 can access ρ π E under O E directly.

Original: So given x ∼ T π 2 under O L , once ⟨ I ( D w 2 ( x )) , g 2 ( x ) ⟩ = +1 , we can query the corresponding observations (cid:101) x of x through OC operation and use ⟨ I ( D w 1 ( (cid:101) x )) , g 1 ( (cid:101) x ) ⟩ to calibrate the output of g 2 and D w 2 .
Revised: So given x ∼ T π 2 under O L , once ⟨ I ( D w 2 ( x )) , g 2 ( x ) ⟩ = +1 , we can query the corresponding observations (cid:101) x of x through OC operation and use ⟨ I ( D w 1 ( (cid:101) x )) , g 1 ( (cid:101) x ) ⟩ to calibrate the output of g 2 and D w 2 .

Original: In this way, g 2 and D wcan be entangled together and adaptively guide π 2 to find out the latent demonstrations H under O L .
Revised: In this way, g 2 and D wcan be entangled together and adaptively guide π 2 to find out the latent demonstrations H under O L .

Original: 4. IWRE
Revised: 4. IWRE

Original: Here we combine the importance weighting and rejection into a unified whole, to propose a novel algorithm named Importance Weighting with REjection (IWRE).
Revised: Here we combine the importance weighting and rejection into a unified whole, to propose a novel algorithm named Importance Weighting with REjection (IWRE).

Original: Concretely, in a HOIL process:
Revised: Concretely, in a HOIL process:

Original: Pretraining.
Revised: Pretraining.

Original: We train a discriminator D w 1 by Equation equation 5 and its corresponding rejection model g 1 by Equation equation 17 using the initial data and the expert demonstrations.
Revised: We train a discriminator D w 1 by Equation equation 5 and its corresponding rejection model g 1 by Equation equation 17 using the initial data and the expert demonstrations.

Original: Training.
Revised: Training.

Original: We train a discriminator D w 2 by the combination of Equation equation 9 and Equation equation 17, as well as its corresponding rejection model g 2 by Equation equation 17, using the initial data, the data collected by π 2 , and the output of D w 1 with g 1 through OC operation.
Revised: We train a discriminator D w 2 by the combination of Equation equation 9 and Equation equation 17, as well as its corresponding rejection model g 2 by Equation equation 17, using the initial data, the data collected by π 2 , and the output of D w 1 with g 1 through OC operation.

Original: Also, πwill be updated with D w 2 and g 2 asymmetrically as in GAIL.
Revised: Also, πwill be updated with D w 2 and g 2 asymmetrically as in GAIL.

Original: The pseudo-code of our algorithm is provided in the supplementary material.
Revised: The pseudo-code of our algorithm is provided in the supplementary material.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In this section, we validate our algorithm in Atari 2600 (Bellemare et al., 2013) (GPL License) and
Revised: In this section, we validate our algorithm in Atari 2600 (Bellemare et al., 2013) (GPL License) and

Original: MuJoCo (Todorov et al., 2012) (Academic License) environments.
Revised: MuJoCo (Todorov et al., 2012) (Academic License) environments.

Original: The experiments were designed to investigate:
Revised: The experiments were designed to investigate:

Original: 1) Can IWRE achieve significant performance under HOIL tasks?
Revised: 1) Can IWRE achieve significant performance under HOIL tasks?

Original: 2) Can IWRE deal with the support mismatch problem?
Revised: 2) Can IWRE deal with the support mismatch problem?

Original: During training, is active querying for HOIL indeed necessary?
Revised: During training, is active querying for HOIL indeed necessary?

Original: Below we first introduce the experimental setup and then investigate the above questions.
Revised: Below we first introduce the experimental setup and then investigate the above questions.

Original: More results and experimental details are included in the supplementary material.
Revised: More results and experimental details are included in the supplementary material.

Original: 5. E XPERIMENTAL S ETUP
Revised: 5. E XPERIMENTAL S ETUP

Original: Environments.
Revised: Environments.

Original: We choose three pixel-memory based games in Atari and five continuous control objects in MuJoCo on OpenAI platform (Brockman et al., 2016) (MIT License).
Revised: We choose three pixel-memory based games in Atari and five continuous control objects in MuJoCo on OpenAI platform (Brockman et al., 2016) (MIT License).

Original: Details as below:
Revised: Details as below:

Original: Pixel-memory Atari games .
Revised: Pixel-memory Atari games .

Original: O E : 84 × 84 × 4 raw pixels; O L : 128 -byte random access memories (RAM).
Revised: O E : 84 × 84 × 4 raw pixels; O L : 128 -byte random access memories (RAM).

Original: Expert: converged DQN-based agents (
Revised: Expert: converged DQN-based agents (

Original: Mnih et al., 2013).
Revised: Mnih et al., 2013).

Original: Atari games contain two totally isolated views: raw pixels and RAM, under the same state.
Revised: Atari games contain two totally isolated views: raw pixels and RAM, under the same state.

Original: Through these environments, we want to investigate whether the agent can learn an effective policy from demonstrations under completely different observation spaces.
Revised: Through these environments, we want to investigate whether the agent can learn an effective policy from demonstrations under completely different observation spaces.

Original: Moreover, IL with visual observations only is already very difficult (Cai et al., 2021), while learning a RAM-based policy can be even more challenging (Bellemare et al., 2013; Sygnowski &
Revised: Moreover, IL with visual observations only is already very difficult (Cai et al., 2021), while learning a RAM-based policy can be even more challenging (Bellemare et al., 2013; Sygnowski &

Original: Michalewski, 2016), so few IL research reported desirable results on this task.
Revised: Michalewski, 2016), so few IL research reported desirable results on this task.

Original: [Figure]
Revised: [Figure]

Revised: , velocity, etc., of an object.

Original:  to investigate whether the agent can learn from demonstrations with complementary signals under observations with missing information.
Revised: Here we want to investigate whether the agent can learn from demonstrations with complementary signals under observations with missing information.

Original: Meanwhile, we make sure RL algorithms can obtain comparable performances under O E and O L .
Revised: Meanwhile, we make sure RL algorithms can obtain comparable performances under O E and O L .

Original: More details are reported in the supplementary material.
Revised: More details are reported in the supplementary material.

Original: Besides, twenty expert trajectories were collected for each environment.
Revised: Besides, twenty expert trajectories were collected for each environment.

Original: Each result contains five trials with different random seeds.
Revised: Each result contains five trials with different random seeds.

Original: All experiments were conducted on server clusters with NVIDIA
Revised: All experiments were conducted on server clusters with NVIDIA

Original: Tesla V100 GPUs.
Revised: Tesla V100 GPUs.

Original: The summary of the environments is gathered in the supplementary material.
Revised: The summary of the environments is gathered in the supplementary material.

Original: Baselines.
Revised: Baselines.

Original: Six basic contenders were included in the experiments: Vanilla GAIL (Ho & Ermon, 2016), GAIL with importance weighting (Fang et al., 2020) ( IW ), third-person IL (Stadie et al., 2017) ( TPIL ), generative adversarial MDP alignment (Kim et al., 2020) ( GAMA ), behavioral cloning (Bain & Sammut, 1996) ( BC ), and learning by cheating (Chen et al., 2019)
Revised: Six basic contenders were included in the experiments: Vanilla GAIL (Ho & Ermon, 2016), GAIL with importance weighting (Fang et al., 2020) ( IW , an ablation study of IWRE), thirdperson IL (Stadie et al., 2017) ( TPIL ), generative adversarial MDP alignment (Kim et al., 2020) ( GAMA ), behavioral cloning (Bain & Sammut, 1996) ( BC ), and learning by cheating (Chen et al., 2019)

Original: ( LBC ).
Revised: ( LBC ).

Original: For IW, we utilized the discriminator D w 1 trained in the pretraining to calculate the importance weight; also the optimization objective for D w 2 during training is the same as Equation equation 9; TPIL learns the third-person demonstrations by leading the cross-entropy loss into the update of the feature extractor; GAMA learns a mapping function ψ in view of adversarial training to align the observation of the target domain into the source domain, and thereby can utilize the policy in the source domain for zero-shot imitation.
Revised: For IW, we utilized the discriminator D w 1 trained in the pretraining to calculate the importance weight; also the optimization objective for D w 2 during training is the same as Equation equation 9; TPIL learns the third-person demonstrations by leading the cross-entropy loss into the update of the feature extractor; GAMA learns a mapping function ψ in view of adversarial training to align the observation of the target domain into the source domain, and thereby can utilize the policy in the source domain for zero-shot imitation.

Original: For fairness, we allowed the interaction between the policy and the environment for GAMA under HOIL; LBC uses π 1 learned from privileged states as a teacher to train π 2 in a DAgger (Ross et al., 2011) style, so here we allowed LBC to access O E during the whole IL process.
Revised: For fairness, we allowed the interaction between the policy and the environment for GAMA under HOIL; LBC uses π 1 learned from privileged states as a teacher to train π 2 in a DAgger (Ross et al., 2011) style, so here we allowed LBC to access O E during the whole IL process.

Original: In Atari, to investigate whether our method could achieve good performance for RAM-based control , we further included a contender PPO-RAM , which uses proximal policy optimization (PPO) (Schulman et al., 2017) to perform RL directly with environmental true rewards under the RAM-based observations.
Revised: In Atari, to investigate whether our method could achieve good performance for RAM-based control, we further included a contender PPO-RAM , which uses proximal policy optimization (PPO) (Schulman et al., 2017) to perform RL directly with environmental true rewards under the RAM-based observations.

Original: More detailed setup including query strategies forTPIL and
Revised: More detailed setup including query strategies for

Original:  GAMA, network architecture, and hyper-parameters are reported in the supplementary material.
Revised: TPIL and GAMA, network architecture, and hyper-parameters are reported in the supplementary material.

Original: Learning process.
Revised: Learning process.

Original: To simulate the situation that O E is costly, the steps for training π 1 was set as 1/ of that for training π 2 , using GAIL (Ho & Ermon, 2016)/HashReward (Cai et al., 2021) under the
Revised: To simulate the situation that O E is costly, the steps for training π 1 was set as 1/ of that for training π 2 , using GAIL (Ho & Ermon, 2016)/HashReward (Cai et al., 2021) under the

Original: O E space for MuJoCo/Atari environments.
Revised: O E space for MuJoCo/Atari environments.

Original: The learning steps were 10 7 for MuJoCo and 5 × 10 6 for
Revised: The learning steps were 10 7 for MuJoCo and 5 × 10 6 for

Original: Atari environments.
Revised: Atari environments.

Original: In the pretraining, we sampled 20 trajectories from π 1 , and the data from each trajectory had both O E and O L observations.
Revised: In the pretraining, we sampled 20 trajectories from π 1 , and the data from each trajectory had both O E and O L observations.

Original: In the training, each method learned 4 × 10 7 steps for
Revised: In the training, each method learned 4 × 10 7 steps for

Original: MuJoCo and 2 × 10 7 steps for Atari under the O L space to obtain π 2 .
Revised: MuJoCo and 2 × 10 7 steps for Atari under the O L space to obtain π 2 .

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Experimental results are reported in Figure 6.
Revised: Experimental results are reported in Figure 6.

Original: Since the mapping function is hard to learn when input is RAM and output is raw images, we omit the results of GAMA in Atari.
Revised: Since the mapping function is hard to learn when input is RAM and output is raw images, we omit the results of GAMA in Atari.

Original: We can observe that while IW is better than GAIL in most environments, both GAIL and IW can hardly outperform π 1 .
Revised: We can observe that while IW is better than GAIL in most environments, both GAIL and IW can hardly outperform π 1 .

Original: Because they just imitated the performance of π 1 instead of π E , even with importance weighting for calibration.
Revised: Because they just imitated the performance of π 1 instead of π E , even with importance weighting for calibration.

Original: For TPIL, its learning process was extremely unstable on Hopper , Swimmer , and
Revised: For TPIL, its learning process was extremely unstable on Hopper , Swimmer , and

Original: Walker2d due to the continuous distribution shift.
Revised: Walker2d due to the continuous distribution shift.

Original: Furthermore, the performance of GAMA was not satisfactory in Hopper and Walker2d because its mapping function is hard to learn well when the support mismatch appears.
Revised: Furthermore, the performance of GAMA was not satisfactory in Hopper and Walker2d because its mapping function is hard to learn well when the support mismatch appears.

Original: The results of TPIL and GAMA demonstrate that DSIL methods will be invalid under heterogeneous observations as in HOIL tasks.
Revised: The results of TPIL and GAMA demonstrate that DSIL methods will be invalid under heterogeneous observations as in HOIL tasks.

Original: On Atari environments, O E contains more privileged information than O L , so LBC can achieve good performance.
Revised: On Atari environments, O E contains more privileged information than O L , so LBC can achieve good performance.

Original: But when O E is not more privileged than O L , like in most environments of MuJoCo, its performance will decrease due to the support mismatch, which would make it even worse than BC.
Revised: But when O E is not more privileged than O L , like in most environments of MuJoCo, its performance will decrease due to the support mismatch, which would make it even worse than BC.

Original: Finally, IWRE obtained the best performance on 6/8 environments, and comparable performance with LBC on Reacher , which shows the effectiveness of our method even with limited access to O E (LBC can access to O E all the time).
Revised: Finally, IWRE obtained the best performance on 6/8 environments, and comparable performance with LBC on Reacher , which shows the effectiveness of our method even with limited access to O E (LBC can access to O E all the time).

Original: Besides, we can see that the performance differences between the GAIL/IW and IWRE/TPIL/GAMA/LBC are huge (especially on Reacher ) because of the absence of queries, which demonstrates that the query operation is indeed necessary for HOIL problems.
Revised: Besides, we can see that the performance differences between the GAIL/IW and IWRE/TPIL/GAMA/LBC are huge (especially on Reacher ) because of the absence of queries, which demonstrates that the query operation is indeed necessary for HOIL problems.

Original: Moreover, even learned with true rewards, PPO-RAM surprisingly failed to achieve comparable performance to IWRE, which shows that IWRE could possibly learn more effective rewards than true environmental rewards in RAM-input tasks.
Revised: Moreover, even learned with true rewards, PPO-RAM surprisingly failed to achieve comparable performance to IWRE, which shows that IWRE could possibly learn more effective rewards than true environmental rewards in RAM-input tasks.

Original: The results verify that, IWRE provides a powerful approach for tackling HOIL problems, even under the situation that the demonstrations are gathered from such a different observation space, meanwhile O E is strictly limited during training.
Revised: The results verify that, IWRE provides a powerful approach for tackling HOIL problems, even under the situation that the demonstrations are gathered from such a different observation space, meanwhile O E is strictly limited during training.

Original: [Figure]
Revised: [Figure]

Original: under O E . The high-density regions of the expert demonstrations were covered by the collected data of π 2 of IWRE t-SNE visualization of ρ π 2 and ρ π E under O E .
Revised: t-SNE visualization of ρ π 2 and ρ π E under O E .

Original: In Section 4.2, we point that the sub-optimality of π 1 will cause the problem of support mismatch, which is embodied as the appearance of the latent demonstration H during training.
Revised: In Section 4.2, we point that the sub-optimality of π 1 will cause the problem of support mismatch, which is embodied as the appearance of the latent demonstration H during training.

Original: Also the empirical results in Figure 5 on Hopper and Walker2d verify the existence of this problem.
Revised: Also the empirical results in Figure 5 on Hopper and Walker2d verify the existence of this problem.

Original: So we want to investigate whether the superiority of IWRE indeed comes from successfully tackling the support mismatch problem.
Revised: So we want to investigate whether the superiority of IWRE indeed comes from successfully tackling the support mismatch problem.

Original: To this end, we plotted the t-SNE visualization of the same expert demonstrations as in Section 4.2 and the collected data of π 2 by IWRE under O E ( O E is hidden to π 2 ).
Revised: To this end, we plotted the t-SNE visualization of the same expert demonstrations as in Section 4.2 and the collected data of π 2 by IWRE under O E ( O E is hidden to π 2 ).

Original: All setups are the same as in Section 4.2.
Revised: All setups are the same as in Section 4.2.

Original: From the results shown in Figure 7, we can see that even under O E , which cannot be obtained by π 2 , almost all high-density regions of the demonstrations were covered by the collected data.
Revised: From the results shown in Figure 7, we can see that even under O E , which cannot be obtained by π 2 , almost all high-density regions of the demonstrations were covered by the collected data.

Original: Meanwhile, the latent demonstration H is dug out nearly.
Revised: Meanwhile, the latent demonstration H is dug out nearly.

Original: The results demonstrate that IWRE basically solves the problem of support mismatch and thereby performs well in these environments.
Revised: The results demonstrate that IWRE basically solves the problem of support mismatch and thereby performs well in these environments.

Original: Besides, some collected data of π 2 of IWRE were out of the distribution of the demonstrations, which means π 2 slightly overly explored the environment.
Revised: Besides, some collected data of π 2 of IWRE were out of the distribution of the demonstrations, which means π 2 slightly overly explored the environment.

Original: Since O E is hidden to π 2 , the reward function will encourage π 2 to explore more areas to fix the support mismatch problem.
Revised: Since O E is hidden to π 2 , the reward function will encourage π 2 to explore more areas to fix the support mismatch problem.

Original: Meanwhile, the out-of-distribution problem in HOIL is not as severe as in the offline RL settings (Levine et al., 2020), so this over-exploration phenomenon makes sense.
Revised: Meanwhile, the out-of-distribution problem in HOIL is not as severe as in the offline RL settings (Levine et al., 2020), so this over-exploration phenomenon makes sense.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed a new learning framework named Heterogeneously Observable Imitation
Revised: In this paper, we proposed a new learning framework named Heterogeneously Observable Imitation

Original: Learning (HOIL), to formulate the situations where the observation space of demonstrations is different from that of the imitator while learning.
Revised: Learning (HOIL), to formulate the situations where the observation space of demonstrations is different from that of the imitator while learning.

Original: We formally modeled a learning process of HOIL, in which the access to the observations of an expert is limited due to the high cost.
Revised: We formally modeled a learning process of HOIL, in which the access to the observations of an expert is limited due to the high cost.

Original: Furthermore, we analyzed underlying challenges during training, i.e., the dynamics mismatch and the support mismatch, on the occupancy distributions between the demonstrations and the policy.
Revised: Furthermore, we analyzed underlying challenges during training, i.e., the dynamics mismatch and the support mismatch, on the occupancy distributions between the demonstrations and the policy.

Original: To tackle these challenges, we proposed a new algorithm named Importance Weighting with REjection (IWRE), using importance weighting and learning with rejection.
Revised: To tackle these challenges, we proposed a new algorithm named Importance Weighting with REjection (IWRE), using importance weighting and learning with rejection.

Original: Experimental results showed that the direct imitation and domain adaptive methods could not solve this problem, while our approach obtained promising results.
Revised: Experimental results showed that the direct imitation and domain adaptive methods could not solve this problem, while our approach obtained promising results.

Original: In the future, we hope to involve the theoretical guarantee for our algorithm
Revised: In the future, we hope to involve the theoretical guarantee for our algorithm

Original: IWRE and investigate how many O E do we need to query to learn a promising π 2 .
Revised: IWRE and investigate how many O E do we need to query to learn a promising π 2 .

Original: Furthermore, we hope to use the learning framework of HOIL and IWRE to tackle more learning scenarios with demonstrations in different spaces.
Revised: Furthermore, we hope to use the learning framework of HOIL and IWRE to tackle more learning scenarios with demonstrations in different spaces.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To help with reproducibility we reported extensive details as follows:
Revised: To help with reproducibility we reported extensive details as follows:

Original: The basic setup of the environment, contenders, and the learning process in Section 5.1; 2.
Revised: The basic setup of the environment, contenders, and the learning process in Section 5.1; 2.

Original: Pseudo-code for the algorithm in Appendix B; 3.
Revised: Pseudo-code for the algorithm in Appendix B; 3.

Original: Details of the environments and the expert performances in Table 3; 4.
Revised: Details of the environments and the expert performances in Table 3; 4.

Original: Hyper-parameters for each method in Appendix D.
Revised: Hyper-parameters for each method in Appendix D.

Original: A N OTATIONS
Revised: A N OTATIONS

Original: The notations of the main paper are gathered in Table 2.
Revised: The notations of the main paper are gathered in Table 2.

Original: [Table]
Revised: [Table]

Original: Target
Revised: Target

Original: [Table]
Revised: [Table]

Original: Mapping
Revised: Mapping

Original: [Table]
Revised: [Table]

Original: →
Revised: →

Original: [Table]
Revised: [Table]

Original: Mapping function S → O
Revised: Mapping function S → O

Original: [Table]
Revised: [Table]

Original: Trajectory sampled by π E under O E (demonstrations) (cid:101)
Revised: Trajectory sampled by π E under O E (demonstrations) (cid:101)

Original: [Table]
Revised: [Table]

Original: Trajectory sampled by π 1 under O E
Revised: Trajectory sampled by π 1 under O E

Original: [Table]
Revised: [Table]

Original: Trajectory sampled by π 2 under O E
Revised: Trajectory sampled by π 2 under O E

Original: [Table]
Revised: [Table]

Original: Trajectory sampled by π 1 under O L
Revised: Trajectory sampled by π 1 under O L

Original: [Table]
Revised: [Table]

Original: Trajectory sampled by π 2 under O L
Revised: Trajectory sampled by π 2 under O L

Original: [Table]
Revised: [Table]

Original: An instance of state-action pair
Revised: An instance of state-action pair

Original: [Table]
Revised: [Table]

Original: An instance of observation-action pair under O E
Revised: An instance of observation-action pair under O E

Original: [Table]
Revised: [Table]

Original: An instance of observation-action pair under O L
Revised: An instance of observation-action pair under O L

Original: [Table]
Revised: [Table]

Original: Occupancy measure of the expert policy π E
Revised: Occupancy measure of the expert policy π E

Original: [Table]
Revised: [Table]

Original: Occupancy measure of the auxiliary policy π
Revised: Occupancy measure of the auxiliary policy π

Original: [Table]
Revised: [Table]

Original: Occupancy measure of the target policy π
Revised: Occupancy measure of the target policy π

Original: [Table]
Revised: [Table]

Original: Adversarial model on (cid:101) T π E and (cid:101) T π
Revised: Adversarial model on (cid:101) T π E and (cid:101) T π

Original: D
Revised: D

Original: [Table]
Revised: [Table]

Original: Adversarial model on T π 1 and T πα
Revised: Adversarial model on T π 1 and T πα

Original: Importance weighting factor
Revised: Importance weighting factor

Original: [Table]
Revised: [Table]

Original: Latent demonstration
Revised: Latent demonstration

Original: [Table]
Revised: [Table]

Original: Observed demonstration
Revised: Observed demonstration

Original: N
Revised: N

Original: Non-expert data
Revised: Non-expert data

Original: [Table]
Revised: [Table]

Original: rejection model under O E
Revised: rejection model under O E

Original: [Table]
Revised: [Table]

Original: rejection model under O L
Revised: rejection model under O L

Original: B A LGORITHM
Revised: B A LGORITHM

Original: The pseudo codes of our algorithm are illustrated in Algorithms 1 and 2.
Revised: The pseudo codes of our algorithm are illustrated in Algorithms 1 and 2.

Original: Update π 2 by RL algorithms (such as PPO (Schulman et al., 2017)) using instances { x π 2 } and pseudo rewards { log D w 2 ( x π 2 ) }
Revised: Update π 2 by RL algorithms (such as PPO (Schulman et al., 2017)) using instances { x π 2 } and pseudo rewards { log D w 2 ( x π 2 ) }

Original: { x π
Revised: { x π

Original: ( D w 2 ( x π 2 )) , g 2 ( x π
Revised: ( D w 2 ( x π 2 )) , g 2 ( x π

Original: Query the O E observation of x π 2 , i.e., (cid:101) x π 2 , through OC operation.
Revised: Query the O E observation of x π 2 , i.e., (cid:101) x π 2 , through OC operation.

Original: Update
Revised: Update

Original: Equation
Revised: Equation

Original: 17) using the instance x π 2 and the corresponding label ⟨ I
Revised: 17) using the instance x π 2 and the corresponding label ⟨ I

Original: 12: end
Revised: 12: end

Original: 13: end
Revised: 13: end

Original: 14: return π15: end function
Revised: 14: return π15: end function

Original: C D EFINITIONS
Revised: C D EFINITIONS

Original: The core challenges of HOIL, i.e., dynamics mismatch and support mismatch, are illustrated as below.
Revised: The core challenges of HOIL, i.e., dynamics mismatch and support mismatch, are illustrated as below.

Original: Definition 5 (Dynamics Mismatch) .
Revised: Definition 5 (Dynamics Mismatch) .

Original: The dynamics mismatch between the demonstrations and the initial data denotes the situation that:
Revised: The dynamics mismatch between the demonstrations and the initial data denotes the situation that:

Original: [Equation]
Revised: [Equation]

Original: Definition 6 (Support Mismatch) .
Revised: Definition 6 (Support Mismatch) .

Original: The support mismatch between the demonstrations and the initial data denotes the situation that:
Revised: The support mismatch between the demonstrations and the initial data denotes the situation that:

Original: [Equation]
Revised: [Equation]

Original: D D ETAILED S ETUP FOR THE E XPERIMENTS
Revised: D D ETAILED S ETUP FOR THE E XPERIMENTS

Original: The details of the environments are reported in Table 3.
Revised: The details of the environments are reported in Table 3.

Original: Also, the detailed comparisons of the contenders (both in the main paper and the supplementary material) and IWRE are gathered in
Revised: Also, the detailed comparisons of the contenders (both in the main paper and the supplementary material) and IWRE are gathered in

Original: Table 4.
Revised: Table 4.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: About query strategies, for TPIL and GAMA
Revised: About query strategies, for TPIL and GAMA

Original: [Table]
Revised: [Table]

Original: if the output of the domain invariant discriminator is larger than 0.
Revised: if the output of the domain invariant discriminator is larger than 0.

Original: [Table]
Revised: [Table]

Original: which means the encoder fails to generate proper features to confuse its discriminator
Revised: which means the encoder fails to generate proper features to confuse its discriminator

Original: [Table]
Revised: [Table]

Original: then we would query O E of this data to update the encoder.
Revised: then we would query O E of this data to update the encoder.

Original: For IWRE, the threshold of the rejection model g and the discriminator D w 2 was also 0.5, which means that if g 2 ( x ) > 0 .
Revised: For IWRE, the threshold of the rejection model g and the discriminator D w 2 was also 0.5, which means that if g 2 ( x ) > 0 .

Original: 5 meanwhile D w 2 ( x ) > 0 .
Revised: 5 meanwhile D w 2 ( x ) > 0 .

Original: 5 , O E of this data would be queried.
Revised: 5 , O E of this data would be queried.

Original: D w 2 , π 2 , and the encoder (for TPIL/GAMA) were pretrained for 100 epochs for all methods using evolving data during pretraining.
Revised: D w 2 , π 2 , and the encoder (for TPIL/GAMA) were pretrained for 100 epochs for all methods using evolving data during pretraining.

Original: The basic
Revised: The basic

Original: RL algorithm is PPO
Revised: RL algorithm is PPO

Original: [Table]
Revised: [Table]

Original: and the reward signals of all methods were normalized into [0 , 1] to enhance the performance of RL (Dhariwal et al., 2017).
Revised: and the reward signals of all methods were normalized into [0 , 1] to enhance the performance of RL (Dhariwal et al., 2017).

Original: The buffer size for TPIL and IWRE was set as 5000.
Revised: The buffer size for TPIL and IWRE was set as 5000.

Original: Each time the buffer is full, the encoder and the rejection model will be updated for 4 epochs; also
Revised: Each time the buffer is full, the encoder and the rejection model will be updated for 4 epochs; also

Original: LBC will update π 2 for 100 epochs with batch size 256 using the cross-entropy loss for Atari and the mean-square loss for MuJoCo.
Revised: LBC will update π 2 for 100 epochs with batch size 256 using the cross-entropy loss for Atari and the mean-square loss for MuJoCo.

Original: We set all hyper-parameters, update frequency, and network architectures of the policy part the same as Dhariwal et al.
Revised: We set all hyper-parameters, update frequency, and network architectures of the policy part the same as Dhariwal et al.

Original: Besides, the hyper-parameters of the discriminator for all methods were the same: The rejection model and discriminator were updated using Adam with a decayed learning rate of 3 × 10 − 4 ; the batch size was 256.
Revised: Besides, the hyper-parameters of the discriminator for all methods were the same: The rejection model and discriminator were updated using Adam with a decayed learning rate of 3 × 10 − 4 ; the batch size was 256.

Original: The ratio of update frequency between the learner and discriminator was 3: 1.
Revised: The ratio of update frequency between the learner and discriminator was 3: 1.

Original: The target coverage c in Equation (17) was set as 0 . 8 .
Revised: The target coverage c in Equation (17) was set as 0 . 8 .

Original: λ in Equation (17) was 1.0.
Revised: λ in Equation (17) was 1.0.

Original: E RL P ERFORMANCE UNDER THE D IVISIONS OF M U J O C O
Revised: E RL P ERFORMANCE UNDER THE D IVISIONS OF M U J O C O

Original: Here we report the performance under the division of O E and O L in MuJoCo.
Revised: Here we report the performance under the division of O E and O L in MuJoCo.

Original: The details of the division are reported in Table 5.
Revised: The details of the division are reported in Table 5.

Original: We use DDPG-based (Lillicrap et al., 2016) agent with 10 7 training steps and repeat 10 times with different random seeds.
Revised: We use DDPG-based (Lillicrap et al., 2016) agent with 10 7 training steps and repeat 10 times with different random seeds.

Original: The results are shown in Figure 8.
Revised: The results are shown in Figure 8.

Original: We can see that the agent can obtain comparable performance under O E and O L .
Revised: We can see that the agent can obtain comparable performance under O E and O L .

Original: So for MuJoCo environments, the fairness of the division in HOIL can be guaranteed, and O E is not more or less privileged than
Revised: So for MuJoCo environments, the fairness of the division in HOIL can be guaranteed, and O E is not more or less privileged than

Original: O L .
Revised: O L .

Original: F E STIMATION OF H , O , AND N BY I [ D w 2 ] g
Revised: F E STIMATION OF H , O , AND N BY I [ D w 2 ] g

Original: To investigate the ability of IWRE to distinguish the areas of latent demonstrations H , observed demonstrations O , and non-expert data N during policy learning, we recorded the accuracy and estimated ratio of each part on Hopper and Walker2d .
Revised: To investigate the ability of IWRE to distinguish the areas of latent demonstrations H , observed demonstrations O , and non-expert data N during policy learning, we recorded the accuracy and estimated ratio of each part on Hopper and Walker2d .

Original: The calculations of each curve are shown as below:
Revised: The calculations of each curve are shown as below:

Original: [Table]
Revised: [Table]

Original: − − − − −
Revised: − − − − −

Original: R e w a r d s (a) Reacher  E
Revised: R e w a r d s (a) Reacher  E

Original: [Figure]
Revised: [Figure]

Original: Accuracy H =
Revised: Accuracy H =

Original: [Equation]
Revised: [Equation]

Original: Accuracy
Revised: Accuracy

Original: [Equation]
Revised: [Equation]

Original: Accuracy N =
Revised: Accuracy N =

Original: [Equation]
Revised: [Equation]

Original: Ratio O = (cid:80)
Revised: Ratio O = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: m , (26)
Revised: m , (26)

Original: Ratio N =
Revised: Ratio N =

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: } m , (27) in which { x i , (cid:101) x i } ∼ ρ π 2 denotes a batch of data sampled by π 2 .
Revised: } m , (27) in which { x i , (cid:101) x i } ∼ ρ π 2 denotes a batch of data sampled by π 2 .

Original: The results are shown in Figure 9.
Revised: The results are shown in Figure 9.

Original: The results depicted not only the accuracies of I [ D w 2 ] g 2 , but also the changes of these three areas during the policy learning.
Revised: The results depicted not only the accuracies of I [ D w 2 ] g 2 , but also the changes of these three areas during the policy learning.

Original: We can see that the accuracies in each area and the ratio of O will decrease at first.
Revised: We can see that the accuracies in each area and the ratio of O will decrease at first.

Original: While at the same time, the ratio of H will increase.
Revised: While at the same time, the ratio of H will increase.

Original: This is because the successful detection of H will decrease the estimated ratio of O and reduce the accuracy of I [ D w 2 ] g 2 .
Revised: This is because the successful detection of H will decrease the estimated ratio of O and reduce the accuracy of I [ D w 2 ] g 2 .

Original: With the help of query operations, the accuracy of I [ D w 2 ] g 2 will gradually increase.
Revised: With the help of query operations, the accuracy of I [ D w 2 ] g 2 will gradually increase.

Original: Also, followed by the learning procedure of the policy π 2 , more and more H will be recognized as O , with less and less N .
Revised: Also, followed by the learning procedure of the policy π 2 , more and more H will be recognized as O , with less and less N .

Original: This is why in the following period, the ratios of H and N will decrease while that of O will increase.
Revised: This is why in the following period, the ratios of H and N will decrease while that of O will increase.

Original: These results also verify that our algorithm IWRE can indeed detect H , O , and N successfully as the learning process of the policy.
Revised: These results also verify that our algorithm IWRE can indeed detect H , O , and N successfully as the learning process of the policy.

Original: [Figure]
Revised: [Figure]

Original: u
Revised: u

Original: [Figure]
Revised: [Figure]

Original: a c
Revised: a c

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: R a t i o
Revised: R a t i o

Original: [Figure]
Revised: [Figure]

Revised: G I NVESTIGATIONS ON ChopperCommand P ERFORMANCE

Revised: We observed some interesting phenomenons on ChopperCommand performance.

Revised: Here we investigate the behavior of a random policy, the IWRE policy, and the expert policy, shown in Figure 10.

Revised: On ChopperCommand , the agent needs to imitate the expert to manipulate a helicopter (noted by the black circle) to avoid attacks from enemy aircraft meanwhile shooting them.

Revised: So the policy in this environment can be split into two parts:

Revised: The movement of the helicopter; 2.

Revised: The shoot of the helicopter.

Revised: While capturing the image-level semantic information with only RAM input is quite difficult.

Revised: Even with RAM input, IWRE still successfully learned the expert’s movement of the helicopter (see the rows of “IWRE 2e7 steps” and “Expert Policy” in Figure 10).

Revised: But it is much harder to capture the shot bullet in the image, not to mention in the RAM.

Revised: So IWRE policy did not shoot enemy aircraft successfully.

Revised: However, the environmental reward of ChopperCommand is only related to the number of enemy aircraft shot down, regardless of the distance the helicopter has flown.

Revised: Therefore, in Figure 6a, IWRE policy can obtain higher rewards than the random policy at the beginning of the training (1e5 steps).

Revised: Then the environmental reward fluctuates within a specific range.

Revised: On the other hand, IWRE indeed learned some expert’s policy of manipulating the helicopter in view of the trajectory shown in Figure 10.

Revised: So we believe that only considering the reward value as an evaluation is insufficient to reflect the degree of imitation on ChopperCommand .

Revised: This interesting phenomenon reveals the mismatch between the evaluations of IL methods and the that of environmental rewards on this environment.

Revised: Figure 10 can also be used as a metric.

Revised: We also provide the full videos of these four policies on ChopperCommand in this anonymous link: https://www.dropbox.

Revised: com/sh/fz7xpbt4y8t3umz/AACw5cMq5eG8sCl04x9_OOnha?dl=0 .

Original: [Figure]
Revised: [Figure]

Original: G Q UERY E FFICIENCY
Revised: H Q UERY E FFICIENCY

Original: We also investigate whether our query strategy is efficient.
Revised: We also investigate whether our query strategy is efficient.

Original: To this end, we allocate the query budget, i.e., limiting the query ratio for each method.
Revised: To this end, we allocate the query budget, i.e., limiting the query ratio for each method.

Original: For TPIL, it preferentially queries those data with low
Revised: For TPIL, it preferentially queries those data with low

Original: D w ϕ output; for our method IWRE, it preferentially queries those data with high ⟨ D w 2 , g 2 ⟩ output.
Revised: D w ϕ output; for our method IWRE, it preferentially queries those data with high ⟨ D w 2 , g 2 ⟩ output.

Original: Besides, since GAIL and IW cannot directly perform queries, we design a random-selection strategy  for them as GAIL-Rand and IW-Rand: for each batch of data, we randomly select data and input the
Revised: Besides, since GAIL and IW cannot directly perform queries, we design a random-selection strategy [Figure] for them as GAIL-Rand and IW-Rand: for each batch of data, we randomly select data and input the

Original: O E observations of these data to D w 1 .
Revised: O E observations of these data to D w 1 .

Original: If D w 1 ( x ) > 0 . 5 , which means D w 1 regard this data being belonging to the expert demonstrations, then we would label this data as the expert data to update
Revised: If D w 1 ( x ) > 0 . 5 , which means D w 1 regard this data being belonging to the expert demonstrations, then we would label this data as the expert data to update

Original: D w 2 .
Revised: D w 2 .

Original: The results are depicted in Figure 10.
Revised: The results are depicted in Figure 11.

Original: We can observe that the random strategy does not always improve the performance of GAIL and
Revised: We can observe that the random strategy does not always improve the performance of GAIL and

Original: IW.
Revised: IW.

Original: For GAIL-Rand, without importance weighting to calibrate the learning process of the reward function, its performance become even worse on Hopper , Swimmer , and Walker2d , because the queried information enhances the discrimination ability of reward function, making it even more impossible for the agent to obtain effective feedbacks; for IW-Rand, its performance is better than
Revised: For GAIL-Rand, without importance weighting to calibrate the learning process of the reward function, its performance become even worse on Hopper , Swimmer , and Walker2d , because the queried information enhances the discrimination ability of reward function, making it even more impossible for the agent to obtain effective feedbacks; for IW-Rand, its performance is better than

Original: GAIL-Rand on most environments, and is reinforced on Hopper , Reacher , and Walker2d , which further demonstrate that the query operation is indeed necessary for HOIL problem, but still fails compared with our method; for TPIL, it is comparable with IW-Rand, however, its performance improvement is very limited as the budget increases, and on Swimmer and Walker2d there even exist performance degradations, which suggests that its query strategy is very unstable; for GAMA, it has a good start point, but the performance gain is very limited while the budget increases; for our method, its performance is almost the same as that of IW-Rand without query on most environments.
Revised: GAIL-Rand on most environments, and is reinforced on Hopper , Reacher , and Walker2d , which further demonstrate that the query operation is indeed necessary for HOIL problem, but still fails compared with our method; for TPIL, it is comparable with IW-Rand, however, its performance improvement is very limited as the budget increases, and on Swimmer and Walker2d there even exist performance degradations, which suggests that its query strategy is very unstable; for GAMA, it has a good start point, but the performance gain is very limited while the budget increases; for our method, its performance is almost the same as that of IW-Rand without query on most environments.

Original: When it is allowed to query O E observation, our method outperforms other methods with a large gap, which shows that the query strategy of our method is indeed more efficient.
Revised: When it is allowed to query O E observation, our method outperforms other methods with a large gap, which shows that the query strategy of our method is indeed more efficient.

Original: H I MITATION WITH D IFFERENT N UMBER OF E XPERT T RAJECTORIES
Revised: I I MITATION WITH D IFFERENT N UMBER OF E XPERT T RAJECTORIES

Original: The performances of different numbers of expert trajectories of all contenders are reported in Figure 11.
Revised: The performances of different numbers of expert trajectories of all contenders are reported in Figure 12.

Original: Each experiment is conducted 5 trials with different random seeds.
Revised: Each experiment is conducted 5 trials with different random seeds.

Original: We can observe that even with a very limited number of trajectories, our algorithm achieves better performance than other algorithms in most environments.
Revised: We can observe that even with a very limited number of trajectories, our algorithm achieves better performance than other algorithms in most environments.

Original: [Figure]
Revised: [Figure]