Paper:
Off-policy estimation for long-horizon problems is important in many real-life applications such as healthcare and robotics, where high-fidelity simulators may not be available and on-policy evaluation is expensive or impossible. Recently, Liu et al. (2018) proposed an approach that avoids the curse of horizon suffered by typical importance-sampling-based methods. While showing promising results, this approach is limited in practice as it requires data be drawn from the stationary distribution of a known behavior policy. In this work, we propose a novel approach that eliminates such limitations. In particular, we formulate the problem as solving for the fixed point of a certain operator, and develop a new estimator that computes importance ratios of stationary distributions, without knowledge of how the off-policy data are collected. We analyze its asymptotic consistency and finite-sample generalization. Experiments on benchmarks verify the effectiveness of our approach.

Review:
Summary : 
This paper proposes an approach for long horizon off-policy evaluation, for the case where the behaviour policy may not be known. The main idea of the paper is to propose an estimator for off-policy evaluation, that computes the ratio of the stationary state distributions (similar to recent previous work Liu et al.,) but when the behaviour policy may be unknown. This is an important contribution since in most practical off-policy settings, the behaviour policy used to collect the data may not necessarily be known. 
Comments and Questions : 
	- This is an important and novel contribution where the authors propose an approach for OPE that comes down to solving a fixed point operator. This is simlar to solving for fixed point Bellman operators, but where their problem can be formulated as a backward recursive problem. 
	- The operator defined as in equation 3 is in terms of the backward view of time - this allows the operator to capture the visitation from the previous s,a to the current s. This is the backward flow operator with which a fixed point equation can be described. Although the authors note that similar operators have appeared in the literature before - their main contribution is in terms of using such operators for the OPE problem, which seems novel to me and is an interesting approach with potential benefits as demonstrated in this paper. 
	- The core idea comes from equation 9 which tries to minimize the discrepancy between the empirical distribution and the stationary distribution. This can be formulated as an optimization problem, and the paper uses blackbox estimators, as described in section 4.2 for solving this problem.
	- The next interesting part of the paper comes from solving the optimization problem in equation 9 with Maximum Mean Discrepancy (MMD) - this is a popular approach that has recently become well-known, and the authors make use of it minimize the differences between the empirical and the stationary state distribution. 
	- Section 4.2 appears a bit confusiing to me with some details missing - it would perhaps be useful if the authors could include more details for 4.2, especially with some explanations of how they arrived at the expression in equation 10. This would also make the paper more self-contained, for readers in the RL community perhaps not so well-read with literature on MMD.  Appendix C contains the detailed derivation, but more intuitive explanations might be useful for the context of the paper. 
	- The proposed black box estimator seems quite useful as demonstrated in figures 2 and 3. Although the authors evaluate their approach of few simple domains - it would have been useful if there were more extensive experiments performed for OPE problems. This would be useful since from fig 2, it appears that the proposed method only outperforms in 3 out of 4 evaluated problems. 
	- For experiments, it would also be useful to demonstrate the significance of not knowing the behaviour policy and what are the usefulness of it. The paper is motived in terms of unknown behaviour policies that generated the data - so few experiments that explicitly shows the benefit of it would perhaps strengthen the paper more. 
	- I am curious to know more about the bias-variance trade-off of the proposed OPE estimator as well. Ie, does the proposed method introduce any bias, or has significance in terms of lower variance for the long horizon problem? Experimentally, would it be possible to demosntrate whether the approach has lower variance compared to existing baselines?
Score : 
- Overall, I think the paper has useful contributions. It is a well written paper, but some additonal details in section 4.2 might be useful, especially on the appriach with MMD. Experimentally, I think there are some experiments missing and doing those can significantly strengthen the paper as well. The proposed method seems simple and elegant, and I would recommend for a weak acceptance of the paper.