From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This paper analyzes multiple POMDP algorithms on numerous environments with different aspects. While their experimental and discussion sections could benefit from more extensive comparisons and deeper analysis, the paper has tested and compared various algorithms on different environments. This benchmark of POMDPs is actually quite important and timely -- many real worlds problems almost always require working with partial observation, thus understanding the current states of algorithms adds an important value to the community. I however personally found their results not super surprising, but it is also good to share with the community that many results follow our intuition or word-of-mouth. So, I give a weak accept. Should the paper be rejected, I highly recommend performing more analysis/ablation studies.
justification_for_why_not_higher_score: The paper is clearly missing deep analysis on understanding each given result.
justification_for_why_not_lower_score: With the same reason above, I would be OK if the paper is rejected -- and give one more iteration to deeply analyze and polish.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper1223/Reviewer_X3ta
title: Thanks for the response!
comment: I hope that the library will be maintained and improved in the future to support the study of partially observed problems in RL.

From: ICLR.cc/2023/Conference/Paper1223/Reviewer_HVVV
title: Re: Weaknesses
comment: > I would expect the analysis part of a benchmark paper would take the largest portion of the paper and will have a very detailed analysis of the benchmarks and experiments. However, the major parts of the paper are about the task description and baseline description;

This is maybe a matter of personal taste, but I didn't have any issues with the structure of the submission. The tasks and baselines are important parts of benchmarking papers.

> Though the authors have run very extensive experiments, the results do not seem dramatically different than what people already understood; I found limited insights from the benchmark results;

I'm not sure that I agree that this is a weakness. The implicit insinuation here is that that authors would have benefitted from fudging their results into looking more surprising.

That said, I actually did find some of their results surprising. For example, I did not anticipate an MLP to perform as well as it did on tasks that clearly require memory; also, I did not necessary anticipate a GRU to outperform an LSTM to the extent to which it did.

> The authors should point out potential directions for tackling the partially observable tasks after the benchmarking.

I do not mind benchmark papers doing this if the experimenters feel they have gained insight into promising future directions from the process of their experiments. However, I do not think benchmarkers ought to be compelled to offer such ideas if they do not have any. Benchmarking by itself is both an important and under-pursued contribution.

From: ICLR.cc/2023/Conference/Paper1223/Reviewer_HVVV
title: Thanks for your reply!
comment: > We intend to add more environments as time goes on. After submission, we implemented, tested, and benchmarked the Count Recall environment. We posted a call for new POMDP suggestions on our public repo.

That's great to hear.

> We are not sure we understand what you mean by ”strategically interesting”, could you clarify?

One example of a POMDP that I would consider strategically interesting is computing a best response to a distribution of partners in Hanabi. This setting is "strategically interesting" in the sense that, to do well, it is necessary to (at least implicitly) do inference about the likelihood of partners and private hands, given the observed behavior, in order to optimize the forward-looking expected return. This inference involves complexity beyond simply remembering an old observation.

> The review period is not enough to implement, test, and benchmark new environments, but if you have any POMDP suggestions, we would be happy to start implementing them.

Agreed. I also don't think the Hanabi-based POMDP I described is a suitable fit for the submission's benchmark, given how expensive it would be.

> We started new differentiable neural computer (DNC) experiments the weekend the reviews were released, but please note that in Figure 2 these are orders of magnitude slower than other memory models, regardless of how fast our environments are. The DNC runs are roughly 50% done at the time of writing (Thursday Nov. 10th). We will do our best to update the paper with these runs before the review closes, otherwise we will add them to the final release, if accepted.

Awesome!

> We agree that more experiments would be ideal, however we have already run over 1700 separate trials and our compute is fully tied up with the requested DNC experiments. We specifically chose conservative hyperparameters for our experiments, so the standard deviations and confidence intervals reported in the appendix are quite small in most cases. Furthermore, performance over a number of separate environments (now 15) should provide good signal, even if a single environment result might contain some uncertainty. We are unfortunately limited in what we can do during the review period.

Makes sense. I hope the authors will consider running a larger number of seeds at some point, even if not during the review period.

> We now provide the interquartile range, mean, and median in the summary plot (Fig. 3) and removed the summary table that only provided means. In the appendix, we report the mean, standard deviation (table), and the 95% bootstrapped confidence interval (bar chart) for all environments and all models. We also report the reward curves with a 95% bootstrapped confidence interval in the appendix.

Thanks for doing this. I hope the authors will consider including additional metrics suggested by the statistical precipice paper, even if not during the review period.

---

As stated in my original review, I think the value of the submission is strictly positive. I do not see any good reason for gatekeeping it from publication at ICLR.

From: ICLR.cc/2023/Conference/Paper1223/Authors
title: Summary Response
comment: We thank all the reviewers and chairs for taking the time to review our paper, and we are very positively encouraged  that others find POMDPs as meaningful and exciting as us. We are excited to hear that we provide a “very good and in-depth summary (of RL benchmarks)” and an “extensive set of experiments” (Reviewer eVQA). Reviewer X3TA writes that “partially observed systems are a crucial challenge”, that we provide, a “clear and nice contribution”, and that our benchmark “is very much worthwhile and will hopefully encourage the community to study these type of problems in more depth.” Reviewer HVVV notes that “the value [the submission] provides to the community is strictly positive.”

Reviewer eVQA has concerns that our discussion section is lacking. We address eVQA’s analysis concerns with a complete rewrite of the discussion and conclusion sections. Our analytical contributions should now be much more clear.

X3TA and HVVV share concerns about our distribution of provided POMDPs. After reading papers provided by X3TA, we explicitly categorize our available environments as overcomplete POMDPs [1,2], and now correctly point users to Ni et al. [3] which contains a library of latent MDPs. We published a public issue on our GitHub repo requesting environment suggestions for types of POMDPs we do not yet represent in our library. We will make a good-faith effort to implement any suggested environments that adhere to the efficiency principles of POPGym. We stress that our library will be updated and actively maintained for at least the next two years – while our submission underwent review we have already added, tested, and benchmarked a new environment (Count Recall) across all memory baselines.

[1] https://proceedings.neurips.cc/paper/2017/hash/6aca97005c68f1206823815f66102863-Abstract.html

[2] https://arxiv.org/abs/2006.12484

[3] https://proceedings.mlr.press/v162/ni22a.html

From: ICLR.cc/2023/Conference/Paper1223/Authors
title: Response to HVVV
comment: >While (again) I agree that there is a dearth of good POMDP benchmarks, I am not sure the submission does a great job addressing the problem:
>* Six of the 14 games are small modifications of fully observable settings
>* Four are described as "diagnostic", meaning useful as "memory introspection and debugging tools"
>* Two are relatively simple card games,
>* Two are navigation based, which, as the submission points out, are one of the few types of POMDPs for which there are suitable existing benchmarks.
>What I am getting at is that the partial observability in POPGym largely doesn't feel _strategically interesting_.”

We have edited the description of diagnostic tasks to be more clear. The diagnostic tests are still quite difficult environments, they just tend to focus on quantifiable aspects of memory, e.g. being able to reconstruct a sequence of observations, determining if memory can forget information after it becomes useless, etc. 

We intend to add more environments as time goes on. After submission, we implemented, tested, and benchmarked the Count Recall environment. We posted a call for new POMDP suggestions on our public repo. 

We are not sure we understand what you mean by __”strategically interesting”__, could you clarify? The review period is not enough to implement, test, and benchmark new environments, but if you have any POMDP suggestions, we would be happy to start implementing them.

>It seems disappointing that it implements but does not benchmark differentiable neural computers. The submission states that this is because computation cost, despite also indicating that POPGym environments induce relatively quick convergence.

We started new differentiable neural computer (DNC) experiments the weekend the reviews were released, but please note that in Figure 2 these are orders of magnitude slower than other memory models, regardless of how fast our environments are. The DNC runs are roughly 50% done at the time of writing (Thursday Nov. 10th). We will do our best to update the paper with these runs before the review closes, otherwise we will add them to the final release, if accepted.

>It is well documented that three seeds may be insufficient to accurately reflect performance. Given that it is a benchmark paper, I'd hope for at least 5 or 10.

We agree that more experiments would be ideal, however we have already run over 1700 separate trials and our compute is fully tied up with the requested DNC experiments. We specifically chose conservative hyperparameters for our experiments, so the standard deviations and confidence intervals reported in the appendix are quite small in most cases. Furthermore, performance over a number of separate environments (now 15) should provide good signal, even if a single environment result might contain some uncertainty. We are unfortunately limited in what we can do during the review period.

>Additionally, it would be good to include the metrics endorsed by the paper, such as interquartile mean, optimality gap, performance profiles, and stratified bootstrap confidence intervals.”

We now provide the interquartile range, mean, and median in the summary plot (Fig. 3) and removed the summary table that only provided means. In the appendix, we report the mean, standard deviation (table), and the 95% bootstrapped confidence interval (bar chart) for all environments and all models. We also report the reward curves with a 95% bootstrapped confidence interval in the appendix.

From: ICLR.cc/2023/Conference/Paper1223/Authors
title: Response to X3TA
comment: > First, the problems (in my opinion) are quite basic and still seem quite far from practical applications

We agree that our environments are far from practical application. Environments mentioned in the related work, such as DMLab, are closer to real-world application than our proposed environments. However, there is a trade off – more realistic and practical environments have higher latency and require training larger models. The point of POPGym is to provide __fast__ and __simple__ environments, similar to those in the original OpenAI Gym library. POPGym should serve as a proving ground: new memory models can be quickly validated and tuned on POPGym before training on a more expensive and complex target environment. As we note in the related work, most POPGym environments can be solved in a few hours while more complex simulators take weeks.

>The theoretical RL community recently studied several POMDP models that would be interesting to explore from practical perspective (in my opinion):
>1. Weakly revealing POMDPs (see https://arxiv.org/pdf/2204.08967.pdf).
>2. MDPs with latent context (see https://arxiv.org/abs/2102.04939).
>3. \gamma observable POMDP (see https://arxiv.org/abs/2206.03446)

We appreciate your suggestions. We now clarify that our environments are overcomplete POMDPs in the POPGym Environments section. The rebuttal time is too short for us to come up with, implement, test, and benchmark new environments. Nonetheless, we have published an issue to our public repo asking for suggestions and we will make a good-faith effort to implement any suggestions. Promises are worth little on the internet, but while the paper was under review, we implemented, tested, and benchmarked a new environment (Count Recall). Hopefully, this provides evidence that we intend to maintain and incorporate new POMDPs into our library as time goes on. 

Moving on, the work from Ni et al. provide a number of latent MDPs, but they call it “Meta RL” so we didn’t make the connection at first. We now accurately describe their environments as “Latent MDPs”, pointing potentially interested users to them in the Related Work section.

With regards to gamma POMDPs: Assigning value to an observation is a very powerful tool that we can use to make strong claims about POMDPs. Unfortunately, it seems that this measure is conditioned on the belief state, so it measures the observability of an environment with respect to a model. It seems quite difficult to measure the observational value when our belief states are uninterpretable, which is the case for all of our benchmarked models. 

> Minor comments

Thank you for the minor comments. We have integrated all of them into the paper.

From: ICLR.cc/2023/Conference/Paper1223/Authors
title: Response to eVQA
comment: >I would expect the analysis part of a benchmark paper would take the largest portion of the paper and will have a very detailed analysis of the benchmarks and experiments. However, the major parts of the paper are about the task description and baseline description

Our original submission followed the tradition of prior literature on RL benchmarks, which focuses on the code/library and quantitative results, rather than analysis (e.g. ProcGen [1], DMLab [2], and Arcade Learning Environment [3]). That said, we have completely rewritten the discussion and conclusion sections to clarify and emphasize our findings.

[1] https://proceedings.mlr.press/v119/cobbe20a/cobbe20a.pdf

[2] https://arxiv.org/pdf/1612.03801.pdf

[3] https://www.jair.org/index.php/jair/article/view/10819/25823

>Though the authors have run very extensive experiments, the results do not seem dramatically different than what people already understood; I found limited insights from the benchmark results

We have emphasized our non-intuitive findings in our revised discussion section. To summarize a few of our less-intuitive findings:
1. Modern POMDP benchmarks rely heavily on navigation tasks. We hypothesize and then show that MLPs routinely outperform all memory models on nearly all navigation tasks. As far as we know, nobody else has made this connection explicit in RL. This has far-reaching implications for how we measure progress on partially observable RL.
2. Time-dependent stateless policies (positional MLP) are much more effective than they have any right to be, nearly solving Stateless Cartpole and outperforming MLPs in many cases.
3. We show that supervised learning results are a poor predictor of memory in RL, providing a need for POMDP benchmarks. Despite the explosion of attention in RL, we show it exhibits relatively poor performance when used as memory.
4. Before our study, there were few/no RL-based studies on the IndRNN, FART, LMU, and S4D baselines. It was not clear how these models would perform against other baselines.
5. Elman RNNs, discarded decades ago for their vanishing/exploding gradients, work surprisingly well in RL – on-par with LSTM in fact, while being more efficient in every sense of the word. 

>The authors should point out potential directions for tackling the partially observable tasks after the benchmarking.

This is a good point. We have revised our text so that nearly every paragraph in the discussion section now provides avenues for future research.

From: ICLR.cc/2023/Conference/Paper1223/Reviewer_eVQA
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper performs benchmarks on partially observable RL environments and a bunch of network architectures on memory. The authors make sure the learning of the POPGym environments is affordable with consumer-grade GPU to ensure the experiments are feasible for researchers. They conclude that GRU is the best general-purpose memory model. 
strength_and_weaknesses: Strength
+ A very good and in-depth summary of the current RL benchmarks, especially on the partially observable environments. After analysis of the shortcomings of the current benchmarks, they motivate the work;
+ The author conducted an extensive set of experiments with 14 partially observable environments and with 13 memory model baselines;
+ Considerations have been made for wide adoption of the proposed benchmark, including low compute cost for a wide community, compatible APIs with the popular training library, etc;

Weaknesses
- I would expect the analysis part of a benchmark paper would take the largest portion of the paper and will have a very detailed analysis of the benchmarks and experiments. However, the major parts of the paper are about the task description and baseline description; 
- Though the authors have run very extensive experiments, the results do not seem dramatically different than what people already understood;
- I  found limited insights from the benchmark results;
- The authors should point out potential directions for tackling the partially observable tasks after the benchmarking. 
clarity,_quality,_novelty_and_reproducibility: The clarity is high, the quality is decent, novelty is on a borderline level. The reproducibility is good.
summary_of_the_review: I found unsatisfactory that a benchmark paper does not provide much content on the in-depth analysis of experiment results, even though the authors conduct an extensive set of experiments.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1223/Reviewer_X3ta
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors introduced a set of new environments to test RL algorithms on partially observable environments. The authors introduced a diverse set of environments in which different aspects of the hardness of partially observed system was tested. Further, the authors conduced quite extensive study and tested the performance of previously suggested algorithms for the partially observed domain. 

strength_and_weaknesses: Pros:
Partially observed system are a crucial challenge to tackle for applying RL in real-world applications. I tend to agree with the authors claim that currently there are not enough benchmarks to test different RL algorithms for the partially observable settings. I hope that this work will encourage further research into the partially observable setting, and lead to the design of new and practical algorithms for this domain. 

Cons:
Although the authors introduced several new environments there are several aspects of partially observed systems that are not being investigated. First, the problems (in my opinion) are quite basic and still seem quite far from practical applications. Second, the theoretical RL community recently studied several POMDP models that would be interesting to explore from practical perspective (in my opinion):
1. Weakly revealing POMDPs (see https://arxiv.org/pdf/2204.08967.pdf).
2. MDPs with latent context (see https://arxiv.org/abs/2102.04939).
3. \gamma observable POMDP (see https://arxiv.org/abs/2206.03446)


clarity,_quality,_novelty_and_reproducibility: The writing is quite clear and in good quality in my opinion. Below are some minor comments.

1. This it would be worthwhile to add that current benchmarks test the high dimensional observations.
2. Figure 3. Can you normalize the reward such that 1 would indicate the optimal value? if I understand correctly it seems that the normalisation might make it seems the performance is high although it is extremely sub optimal (this can happen due to the normalization).
3. End of section 1.1. should put period "." at the end of each line.
4. page 3, "wall following behavior", what does that mean?
5. page 5 "Like Repeat Fist" -> "Like Repeat First"
6. Page 5. Reference to additional bandit book: https://tor-lattimore.com/downloads/book/book.pdf.
summary_of_the_review: Overall, I think this paper has a clear and nice contribution: it introduced new benchmarks for study partially observed RL problems, This, in my opinion, is very much worthwhile and will hopefully encourage the community to study these type of problems in more depth. Further, I hope that the authors/additional contributers will expand the set of task in the future.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper1223/Reviewer_HVVV
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The submission introduces a library containing 14 POMDPs, each with three levels of difficulty, and 13 memory model baselines. The submission benchmarks 12 of these memory baselines.
strength_and_weaknesses: ### Strengths

- I agree with the submission's characterization that there is a lack of suitable partially observable testbeds.
- Because the submission compares existing algorithms, rather than introducing new ones, the experiments are less likely to suffer from experimenter bias.
- From a brief perusal, the code appears easy to read and well organized.

### Weaknesses

- While (again) I agree that there is a dearth of good POMDP benchmarks, I am not sure the submission does a great job addressing the problem:
    - Six of the 14 games are small modifications of fully observable settings
    - Four are described as "diagnostic", meaning useful as "memory introspection and debugging tools"
    - Two are relatively simple card games,
    - Two are navigation based, which, as the submission points out, are one of the few types of POMDPs for which there are suitable existing benchmarks.

    What I am getting at is that the partial observability in POPGym largely doesn't feel *strategically interesting*.
- The other gripe I have is with algorithm benchmarking:
    - First, given that the submission is a benchmarking paper, it seems disappointing that it implements but does not benchmark differentiable neural computers. The submission states that this is because computation cost, despite also indicating that POPGym environments induce relatively quick convergence.
    - Second, the metrics that were reported are not up to date with modern deep RL benchmarking -- see *Deep RL at the Edge of the Statistical Precipice* (NeurIPS 2021). 
        - The most obvious criticism here is that each algorithm is over only three seeds. It is well documented that three seeds may be insufficient to accurately reflect performance. Given that it is a benchmark paper, I'd hope for at least 5 or 10. 
        - Additionally, it would be good to include the metrics endorsed by the statistical precipice paper, such as interquartile mean, optimality gap, performance profiles, and stratified bootstrap confidence intervals.
clarity,_quality,_novelty_and_reproducibility: The clarity of the work is good.

The quality is fair; however, as articulated in the weaknesses section, there is substantial room for improvement.

The originality is good in the respect that there has not been much work organizing POMDP benchmarks for deep RL.

The reproducibility is high in the sense that code is available and appears to be well organized but less high in the sense that outlier-sensitive results were reported over a small number of runs.
summary_of_the_review: While the idea motivating the submission (providing good POMDP benchmarks for deep RL) is one that is very valuable to the community, I think the submission falls short of its potential both in terms of the proposed environments and the benchmarking of existing algorithms.

Still, I think the submission ought to be accepted, as the value it provides to the community is strictly positive. 

I hope the authors will consider addressing the weaknesses listed above in future revisions.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper