From: auai.org/UAI/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept (Poster)
comment: Meta Review: The paper incorporates prototype learning in offline policy evaluation for better interpretability.

The reviewers found the idea interesting and useful for interpretability, and the paper is well written. There are concerns that the performance of the method is not competitive enough, so it would be better if the authors could explain a little about this and try to improve the performance, since both performance and interpretability are important. I hope the authors consider the comments of the reviewers when polishing the paper.

From: auai.org/UAI/2022/Conference/Paper289/Reviewer_5ABX
Q1 Summary and contributions: This paper proposes the use of prototype learning as a diagnostic tool for off-policy evaluation. In particular, the authors propose using prototypes as a way to provide human interpretable diagnostics around where two policies differ, and where the rewards of the two policies may differ. It argue IS weights are inappropriate for this task due to their lack of explainability, then describe the use of prototype learning. Experimental results show interpretability, and the relative efficacy.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: 1. Compelling problem--interpretability is an important aspect of off-policy evaluation when humans are likely to be in the loop.
2. Simple, practical solution which is well motivated
3. Strong empirical results.
Q4 Main weakness: 1. No theoretical contribution/examination of proposed approach
2. No clear statement of necessary assumptions
Q5 Detailed comments to the authors: Overall, I liked this paper and think it is an interesting approach to an important problem. The solution is compelling both in its simplicity and its apparent efficacy. The one thing I would like to have seen is some kind of theoretical examination of the properties of the proposed estimator. Because the task is inherently unsupervised, it is important for practitioners to be able to reason over the assumptions required for the proposed approach and also possible worst case behavior. This, arguably, is one of the strengths of IS which has been incredibly well studied for decades. 
Q6 Overall score: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
Q7 Justification for your score: Overall, I think this paper is interesting and would be happy to see it published, but I would have liked to also have seen an examination beyond empirical results. 
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper289/Reviewer_AJdH
Q1 Summary and contributions: Importance sampling(IS) has been commonly used in OPE literature due to its simplicity. However, it can be problematic when the behavior policy is unknown. The paper proposes to improve IS by prototype learning, which has been developed in previous literature. Experiments on sepsis management have been conducted. The learned prototypes improve interpretability and provide new insights. The paper also studies the bias by using prototype learning and how that affects the IS estimator.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 2: Fair: The paper contributes some new ideas.
Q2(2) Significance/Impact: 2: Fair: The paper is likely to have moderate impact within a subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: -While most previous literature has been focused on improving the performance of off-policy evaluation estimators, interpretability has not been studied a lot in this setting, which makes the setting of this paper rather novel.

-The paper provided a thorough go-through of how to interpret the result from the prototypical network in the experiment setting, with systematic studying of different design choices of their method, how increased sequence length could introduce bias, etc.

Q4 Main weakness: The paper applies the prototypical network to the task of off-policy evaluation. The main concern I have about this paper is how to justify this combination. Prototypical networks as in [Li 2018b] are originally proposed to improve the interpretability of classification tasks in the iid setting without taking distribution shifting into consideration. While one of the most important issues about off-policy evaluation is that this is a distribution shifting problem. For example, IS method is famous for suffering from high variance, especially in the setting when behavior policy is unknown, how could using a prototypical network instead relieve the high-variance issue? Also, using prototypical networks will introduce new hyper-parameters, such as the number of prototypes and the number of prediction prototypes to use, which could also be problematic in practice. 
Q5 Detailed comments to the authors: -In the experiment part on Page 6, the paper says for all models except the RNN and prototype model, we make the Markov assumption,  could this be elaborated more? Will this create an unfair comparison?
Q6 Overall score: 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Q7 Justification for your score: While I agree interpretability is under-explored in the ope setting, some more explanations need to be made to justify the use of prototypical networks in the ope setting.
Q8 Confidence in your score: 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper289/Reviewer_n4At
Q1 Summary and contributions: The paper presents a method for evaluating a target policy based on a set of prototype trajectories that are learned from observations collected according to a behavioral policy. The main advantage of this approach is its interpretability although performance may be reduced compared to black box models. 
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 2: Fair: Key resources (e.g., proofs, code, data) are unavailable but key details (e.g., proof sketches, experimental setup) are sufficiently well-described for an expert to confidently reproduce the main results.
Q2(6) Clarity of writing: 4: Excellent: The paper is well-organized and clearly written.
Q3 Main strengths: The paper is well written and technically sound. The idea presented are an interesting contribution to explainable AI. 
Experiments illustrate well the methods, discussing both to its strengths and weaknesses.

Q4 Main weakness: The application to the AI Clinician shows that the possibility to apply the method only if the behavioral and target policy are similar is quite a strong limitation. Also, the loss in accuracy, compared to plain neural networks is not neglectable. 
Q5 Detailed comments to the authors: At page 2, just after eq. (3), R^{k x n} --> What is k? 
At page 5 the short description of the AI Clinician approach was not clear to me. Besides, you mention 25 possible actions, but in figure 2 I see only 7.

Q6 Overall score: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
Q7 Justification for your score: The interpretability of the method makes it very interesting, despite the reduce performance, in some fields such a the medical one. 
Q8 Confidence in your score: 2: Not very confident. I am able to defend my evaluation of some aspects of the paper, but it is quite likely that I missed or did not understand some key details, or can't be sure about the novelty of the work.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper289/Reviewer_ix6i
Q1 Summary and contributions: This paper use prototype learning to model the behavior policy and estimate the policy value of the target policy. The employment of prototype learning improve the interpretability of the method and answer the two question: 1. Where do $\pi$ and $\mu$ differ? 2. What give $\pi$ the edge. The experimental results on sepsis management gives some inspiring insight on the analysis of target policy.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 4: Excellent: The paper is well-organized and clearly written.
Q3 Main strengths: 1. The paper incorporates the prototype learning to offline policy evaluation, which brings impressive interpretability.
2. The experimental results is sufficient. The author also conduct experiments to analyze the influence of the number of prototypes, prediction-prototypes on the results.
3. The paper is well written and convenient to follow. The information about the algorithm is described in a detailed way.
Q4 Main weakness: 1. The method in the paper somehow lacks novelty. It seems to be a direct combination of offline policy evaluation and prototype learning.
2. The performance of the method is not competive enough. The baselines performs better than it.
3. The section of experiments does not describe all the necessary information of the experimental settup. I think the authors could improve it and give more detailed information.
Q5 Detailed comments to the authors: 1. I think the policy value is a continuous value rather than a binary variable. Therefore, I am confused about how to compute the accuracy and AUC of the method.
Q6 Overall score: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
Q7 Justification for your score: I carefully read the paper for over 5 hours and obtain a comprehensive understand of the paper.
Q8 Confidence in your score: 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
Q9 Complying with reviewing instructions: 1: Yes.