From: auai.org/UAI/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept (Poster)
comment: Meta Review: The paper presents an interesting use of marginal MAP to inverse reinforcement learning extended to the case of partially observable demonstrations. All reviewers agree that the paper has merit as it presents a sound solution to the general and difficult problem of IRL in a partially observable environment. The reviewers also recommended a number of improvements, in particular in the experimental section, which I strongly recommend the authors to address to improve the current version.

From: auai.org/UAI/2022/Conference/Paper84/Reviewer_qKfv
Q1 Summary and contributions: This paper presents a generalised approach for the maximum-a-posteriori inverse RL algorithm that handles the noisy information collected while observing and learning with an expert. The proposal marginalises and estimates occluded portions of information using the Bayesian version of MAP-IRL. The marginalised MAP-Bayesian IRL (MMAP-BIRL) were tested on two different domains and against 2 main baselines (MAP-BIRL and the HiddenDataEM). 
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 4: Excellent: The paper is well-organized and clearly written.
Q3 Main strengths: Novelty: The paper presents a novel algorithm that composes well-known approaches into a fine and robust solution.

Impact: The application of the marginal approximation through the Bayesian framework will likely influence the AI community as a possible solution while handling partially observable domains. 

Technical Quality: The paper presents an excellent background that justifies and supports the solution presentation. I have not found any issues, but I have not carefully checked the mathematical details.

Quality Of Experiments: The experiments are adequate and the results are enough to support the authors' contribution.

Reproducibility: This work delivers most of the details necessary to reproduce the experiments. I think there is only one piece of information missing: the number of experiments performed.

Writing: The paper is well-organised and clearly written, presenting a good motivation, detailed methodology and clear discussion about the results and development.
Q4 Main weakness: Novelty: Even considering the solution non-trivial and well-made, I would say it follows the current community ideas to advance the state-of-the-art solutions. This point doesn't disqualify the work.

Impact: Perhaps the paper would have a bigger impact if more complex scenarios were included in the experiments. I think future works, based on these current results, could have an even higher impact.

Quality Of Experiments: I missed the presentation of more results in similar and different scenarios. Even the performance of an ablation study would be appreciated and interesting to see in this work. Presenting more baselines would also corroborate a better evaluation of the current results.

Reproducibility: I missed the number of experiments and the information about which test or method the authors used to calculate the confidence/deviation of their results.

The source code is not included in the supplementary material, but the authors promise to make it available after acceptance. 
Q5 Detailed comments to the authors: I really appreciate the reading of this work and think it presented a great solution for the community with good results that supports the authors' claim. I kindly suggested improvements above, however it can be addressed in futures works.
I only found two minors while reading:

(1) in the introduction, the emphasis (italics) on the text is somewhat long:

"As such, the learner’s perception of the task performance may be both incomplete and imperfect. This is different from a typical POMDP setup [Choi and Kim, 2011b] as in our case the expert can perfectly observe the environment and the noisy observations are due to the learner’s imperfect sensors."

I think I would highlight only the citation from Choi and Kim's work or the phrase about this work. In the current text, I missed the detail that I should maintain my focus.

(2) in Section 4.2:

"The value ‘hover location’ is a region of space just on top of the conveyor and ’in bin’ indicates just inside a bin that holds (...)"

The apostrophe should be replaced by quotation marks. Moreover, I recommend using ``in bin'' instead of ’in bin’ to guarantee that both quotation marks will be correctly sided.

Additionally, I would like to see an extensive and deeper discussion about the results. I think Fig. 4 could be smaller to save space.
Q6 Overall score: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
Q7 Justification for your score: The paper presents a fine and robust solution for the target domain using relevant state-of-the-art methods to compound the proposed algorithm. The results well-support the authors' claims and the paper is well-organised/clearly written.
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.
Q10 Ethical concerns (Optional): No.

From: auai.org/UAI/2022/Conference/Paper84/Reviewer_cSDR
Q1 Summary and contributions: This is a nice application of MMAP (marginal MAP) to inverse reinforcement learning.
The IRL task (learning the reward function of a MDP from demonstrations) is extended to the case of partially observable (and not only noisy) samples. Experiments on a toy model and on a real application in robotics are promising. The proposed technique is basically a MMAP procedure for continuous "MAR" variables based on gradient ascent.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: The method proposed in this paper extends the class of setups to be used for (Bayesian) inverse RL.
The new setups are of high interest for applications in robotics.
The experiments show the advantages of the new approach (wrt existing methods)
Q4 Main weakness: The baseline method is designed for a less general setup and, as a matter of fact, the comparison is not fair. Yet, as the authors claim, the method proposed here extend the scope of BIRL so such improved performance is expected.
Q5 Detailed comments to the authors: My overall opinion about the paper is positive. As a non-specialist of IRL I see the advantages related to mapping the inference to a MMAP instead of a standard MAP. 

Regarding the advantages to not ignoring the "occluded" data, I understand the rationale of this point, but an empirical analysis would make this claim stronger. I also miss a dedicated analysis of the complexity of the overall procedure (and in the empirical part discussing the execution times would be also valuable).

Finally, I could imagine situations where the variables related to the occlusion are discrete. In would be interesting to discuss how to adapt the proposed procedure to such alternative setup.
Q6 Overall score: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
Q7 Justification for your score: I am not a specialist, but I see a clear improvement of the SOTA about (B)IRL provided by this paper. This also establishes a nice link between (MMAP) inference and RL and this sounds very much in the spirit of UAI.
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.
Q10 Ethical concerns (Optional): Nothing specific to consider.

From: auai.org/UAI/2022/Conference/Paper84/Reviewer_kWBV
Q1 Summary and contributions: Bayesian formulation of inversement reinforcement learning that allows for state occlusion and lerner's imperfect sensing; gradient-based solution to learn reward function. Experiments with a toyset example a realistic robotic task show the proposed method's advantage over competitors.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 2: Fair: The paper contributes some new ideas.
Q2(2) Significance/Impact: 2: Fair: The paper is likely to have moderate impact within a subfield of AI.
Q2(3) Correctness/Technical quality: 4: Excellent: I am confident that the paper is technically sound, and I have carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 4: Excellent: Key resources (e.g., proofs, code, data) are available and key details (e.g., proof sketches, experimental setup) are comprehensively described for competent researchers to confidently and easily reproduce the main results.
Q2(6) Clarity of writing: 4: Excellent: The paper is well-organized and clearly written.
Q3 Main strengths: - Relevant and scientifically challenging problem
- Simple, principled and potentially effective solution
- Clear writing
Q4 Main weakness: - Proposed approach is straightforward (limited impact)
- Requires handcrafted feature functions
- No discussion on how to extend approach to continuous state and actions
- Experiments consider a single use case and do not test
Q5 Detailed comments to the authors: The whole discussion assumes discrete states and actions, however that assumption is not explicitly stated in sections 1 and 2. The discussion in Section 2 and 3 could be summarized and better illustrated with a plate model describing the statistical assumptions of the model, so that most equation would follow directly and could be omitted. The gained space could then be used to elaborate on the experimental results with the robot arm. 

There is a hidden assumption that the missing data (occlusion) are missing at random (MAR). For example, occlusion could be associated with a state (e.g. when the human demonstrator behaves in some particular way that obscures the learner's camera). Authors should briefly discuss why such an assumption is reasonable (or consider MAR as an approximation/simplificaiton).

How are the state transition function in the robotic experiment specified/learned?
Q6 Overall score: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
Q7 Justification for your score: The paper pushes the state-of-the-art in inverse reinforcement learning from expert's demonstrations with discrete state-action spaces and known transition function. The proposed method relaxes some of the previous work's limitations (complete observation and perfect sensing) and thus steps towards more realistic settings. The proposed method is a straightforward application of Bayesian inference to the problem, with gradient-based optimization. Experiments show promising results.
Q8 Confidence in your score: 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper84/Reviewer_c25Z
Q1 Summary and contributions: This paper focuses on the problem of inverse reinforcement learning. It is motivated by partial observations are occluded and noisy perception in real-world scenes observations are unavoidable in real-world scenes while existed works cannot well handle these situations. This paper proposes the MMAP-BIRL method to improve IRL which allows learning from both occlusions and noisy perception. 
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 2: Fair: The paper contributes some new ideas.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: - This paper proposes the MMAP-BIRL method to improve IRL in a more realistic and challenging scenario whose motivation is clear and sufficient. 
- This paper provides a relatively clear description of the proposed method. The effect of each part of the method is also clarified.
- This paper is well organized and no obvious syntax errors are found. 
Q4 Main weakness: - The figure and font are not well self-restrained. For example, in Figure 4, the indices are not aligned and the size of subfigures are not consistent. 
- The experiment evaluations are not adequate to verify the effectiveness of the proposed methods. More baselines or more evaluation matrices need to be considered for comparison.
- The proposed method relies on somewhat strong assumptions. 
Q5 Detailed comments to the authors: This paper is well-motivated and the description of the proposed method is clear. However, there are some details worth noting. 

- The typesetting of the paper could be improved. 
- Some figures are not well self-restrained. For example, the color of each setting is not easy to distinguish which reduce the readability of this paper.
- I have some questions/concerns for this paper:  
(1) The rationality for using a Gaussian distribution as a prior could be further explained.   
(2) How do you select the hyperparameters (described at the beginning of Section 4)?

Q6 Overall score: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
Q7 Justification for your score: The motivation of this paper is clear and the logic is well organized which is what I weigh most. Moreover, they improve the MMAP model based on MAP and develop a gradient-based method for the MMAP inference problem which makes this paper contributes to this community to some extent. But the experiment evaluations are not adequate to support their claims. The details of this paper also need to be improved.
Q8 Confidence in your score: 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
Q9 Complying with reviewing instructions: 1: Yes.