Paper: Explore Outworld Knowledge in Large Language Models: A Case Study in Pokemon Game (ICLR-2024)

Reviewer comment (Experiments):
The paper focuses solely on the Pokemon world, so it's unclear how generalizable the findings are to other outworld settings.

Author response:
**W1&Q2** how our method generalizes to other new worlds

Thank you for the question. We admit that in the paper we only discussed the outworld of the Pokemon world and do not verify our findings in other new world settings. These are highly interesting to discover, and we will leave these new things for future work, thank you.

Reviewer comment (Theory):
The work may not delve deep enough into the specifics of why LLMs struggle with logical reasoning in new worlds.

Author response:
**W2&Q1** why LLMs face logical inconsistencies about the Pokemon world

As we have discussed in the paper, LLMs will encounter logical inconsistencies even facing general questions in the human world, e.g. hallucinations, even in simple Boolean logic [1]. When transferring to the Pokemon world, this issue is even amplified because:

* The vast majority of pokemon data on the web is like wikipedia, describing the basic attributes of each Pokemon, **while the available data that relates reasoning is very minimal.**

* **Most high-quality data is on videos**, which is hard to collect. We notice that there are methods that allow the model to learn from videos, which is an interesting future work for us.

Reviewer comment (Theory):
Could the authors give some intuitions on why LLMs face logical inconsistencies when reasoning about the Pokemon world?

Author response:
**W2&Q1** why LLMs face logical inconsistencies about the Pokemon world

As we have discussed in the paper, LLMs will encounter logical inconsistencies even facing general questions in the human world, e.g. hallucinations, even in simple Boolean logic [1]. When transferring to the Pokemon world, this issue is even amplified because:

* The vast majority of pokemon data on the web is like wikipedia, describing the basic attributes of each Pokemon, **while the available data that relates reasoning is very minimal.**

* **Most high-quality data is on videos**, which is hard to collect. We notice that there are methods that allow the model to learn from videos, which is an interesting future work for us.

Reviewer comment (Evaluation):
Does the model's self-competition lead to any form of model destabilization or other unexpected behaviors?

Author response:
**Q3** does self-competition lead to any unexpected behaviors

Thank you for your question. We have not found any unexpected outcomes of the model after self-training.

Reviewer comment (Writing):
I suggest the authors to give a definition or formulation on "outworld knowledge". And if this field has been previously researched before, the author should cite it. This can enable readers to more accurately get the problem the author wants to address, rather than having a vague concept.

Author response:
Thank you for your suggestion. We have defined “outworld knowledge” in the second paragraph of Section 1.
We notice that there may be a lack of discussion of the relevance of our defined outworld knowledge to previous studies. We will work on this point carefully in the new version. Thank you.

**However, in the context of emerging LLMs, we are the first to discuss the model adaption to new world settings, to the best of our knowledge.**

Reviewer comment (Novelty):
The method part seems to lack some innovativeness. Similar approaches might have been proposed in prior work, such as AlphaStar. Furthermore, it appears to share some similarities with RLHF or DPO, except that the annotator has transitioned from a human to the environment. However, I don't deny that this method can be helpful for transferring LLM to a new environment.

Author response:
First, we notice that our method is simple yet effective, and there are similar variants proposed in previous papers, but for different purposes, as suggested by Reviewer 5UvA.

**However, from a high-level perspective, the concept of environment feedback we present in the paper is more general than human feedback,** which is embodied in the human environment. This bridges the gap to transfer the model to other new worlds.

Another merit is that **our method does not necessitate a reward model** since the environment can be used all the time.

Reviewer comment (Experiments):
Can the authors add a baseline "playing against humans"? This is a much stronger baseline than Random and MaxDamage.

Author response:
Yes, we agree that playing against humans is a stronger baseline than MaxDamage. However, please allow us to clarify that **the goal of self-training is to learn a better initialization to adapt to a new world**, which means to perform better on new world downstream tasks.

Contrarily, to compete with an experienced human player (e.g. some tricks to confuse the opponent), this outworld knowledge is far from enough. **We argue that the performance against MaxDamage player is a more intuitive reflection of outworld knowledge acquisition.**

Hence, we respectfully argue that including human players is not a mandatory reference to prove the effectiveness of our method. Sure, we are willing to report it in our new version, thank you.

Reviewer comment (Novelty):
I believe that the main contribution of this paper is not the introduction of a learning approach to achieve a higher winning rate, but rather the demonstration of some form of transferability within LLM (because given the current context, game theory or search may potentially achieve better results than LLM.) Could the author discuss this issue?

Author response:
Yes, we agree with your opinion. As we clarify above, **forging a sophisticated LLM agent to compete against humans is neither the centric nor contribution of our paper.** From our empirical experience, LLMs even GPT4 cannot achieve human-like performances in playing games, though there is a chance if training a policy on them.

In our paper, we first unveil the powerlessness of LLMs on outworld knowledge. We then propose to utilize the nature of competition to generate strong data to transfer the model. The performance against MaxDamage player and on two downstream tasks demonstrate the better adaption of the trained model.

Reviewer comment (Novelty):
While the proposed self-supervised learning method to address the new outworld adaption problem is new, in the context of LLMs and outworld adaptation, it is actually very similar to Supervised Self-Play (S2P) in its 'scheduled' variant from [1], which was proposed in the context of Emergent Communication; which can be thought of as an outworld adaptation where the outworld is the natural language and the pre-trained world is that of the emergent language. Scheduled S2P is stage-wise identical to the algorithm proposed here.

Author response:
Thank you very much for suggesting a relevant paper. After reading this paper, we agree that there are similarities between our proposed self-training and self-play (S2P). We both have two models to play with each other, while the purposes are different.
Our focus is self-competition during the process of self-play, **which in our opinion is the most general way to screen out for better data or individuals,** in nature. This also means our method is totally unsupervised.
Instead, the purpose of S2P is self-cooperation, which largely relies on the supervised data, as discussed in the paper.

Thank you for your suggestion. We will carefully discuss the relationships in our new version.

Reviewer comment (Experiments):
In Section 4.1, the following claim is made without evidence: `The choice of ε affects the convergence rate of the algorithm, and a larger one will make it slower.' The reviewer requests citation of a similar phenomenon in previous literature or supporting evidence such as a table or graph showing either the final performance or the learning curves of different runs with different hyperparameter settings of ε.

Author response:
Thank you for your question. The idea of choosing $\epsilon$ is the same as \epsilon-greedy. According to our empirical experiments, it is useful. We show the results against Random player before Gen 30 below:

|              | Gen 1 | Gen 2 | Gen 3 | Gen 5 | Gen 10 | Gen 20 | Gen 30 |
|--------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
|$ \epsilon=0$   | 51.5  | 52.7  | 56.2  | 59.5  | 68.8   | 62.0   | 68.5   |
|$ \epsilon=0.2$ | 52.2  | 53.6  | 58.5  | 61.2  | 75.5   | 75.5   | 80.8   |
| $\epsilon=0.5$ | 51.9  | 51.4  | 52.3  | 52.1  | 55.9   | 58.8   | 62.6   |

We see that $\epsilon$ plays an important role for the eventual performance, while the $\epsilon=0.5$ case converges much slower, since the randomness is too high. 

Due to our multi-aspect experiments in the paper, we cannot discuss every setting in the paper with limited space. We will include these numbers in the appendix. Thank you.

Reviewer comment (Experiments):
In Tables 3 and 5, it seems that only one random-seeded run has been performed, since there is no mention of how many random seeds have been run. The reviewer expects at least 3 differently-seeded runs to be reported, if possible.

Author response:
Thank you for your reminder. In the paper, to reduce the variance, **we perform each battle play against Random and MaxDamage players with 100 times, which we have mentioned in the paper.**

In addition, we clarify that **all experiments are ran under 3 random seeds (42, 0, 2023), and we report the average numbers.** Since each result are averaged from 300 battles, **the variance is very low** even in the early generations of self-training.

The self-training process (for 50 generations) is only performed for once because of the great training cost.

For the fine-tuning experiments on two downstream tasks, **we ran only for once in the paper,** and this is a drawback. **Hence, we rerun the experiments over 3 seeds** (42, 0, 2023). We report the new results as well as the standard deviation below:

|        | Boolean QA  | Inference   |
|--------|:-----------:|:-----------:|
| 0      | 77.4 (1.14) | 46.0 (1.63) |
| Gen 5  | 80.2 (0.86) | 46.0 (0.00) |
| Gen 20 | 80.7 (0.90) | 49.7 (1.25) |
| Gen 30 | 86.0 (1.56) | 55.0 (0.82) |
| Gen 50 | 86.6 (1.03) | 60.0 (2.90) |

We will specify this setting in our new version. Thank you.

Reviewer comment (Reproducibility):
In Section 4.3, the details about the different random seeds are missing. If multiple seeds have been used, the omission should be addressed.

Author response:
Thank you for your reminder. In the paper, to reduce the variance, **we perform each battle play against Random and MaxDamage players with 100 times, which we have mentioned in the paper.**

In addition, we clarify that **all experiments are ran under 3 random seeds (42, 0, 2023), and we report the average numbers.** Since each result are averaged from 300 battles, **the variance is very low** even in the early generations of self-training.

The self-training process (for 50 generations) is only performed for once because of the great training cost.

For the fine-tuning experiments on two downstream tasks, **we ran only for once in the paper,** and this is a drawback. **Hence, we rerun the experiments over 3 seeds** (42, 0, 2023). We report the new results as well as the standard deviation below:

|        | Boolean QA  | Inference   |
|--------|:-----------:|:-----------:|
| 0      | 77.4 (1.14) | 46.0 (1.63) |
| Gen 5  | 80.2 (0.86) | 46.0 (0.00) |
| Gen 20 | 80.7 (0.90) | 49.7 (1.25) |
| Gen 30 | 86.0 (1.56) | 55.0 (0.82) |
| Gen 50 | 86.6 (1.03) | 60.0 (2.90) |

We will specify this setting in our new version. Thank you.

Reviewer comment (Writing):
Section 4.1 details the proposed self-supervised learning method, but the reviewer finds it difficult to follow. They suggest that adding formal notation to refer to each model and the training operations they go through would improve clarity, or, if space permits, providing an actual algorithm environment to refer to, in addition to the information in Figure 3.

Author response:
Thank you for your advice. We will further illustrate an algorithm of our training method to increase its clarity. Thank you.

Reviewer comment (Experiments):
In Section 4.3, batch size and learning rate hyperparameter values are highlighted, but it is unclear how those are used. The reviewer asks whether there have been multiple runs of the experiment with each combination of the hyperparameter values.

Author response:
For each model, we train it with two learning rates 3e-4 and 8e-4. The selection of the batch size (2 and 4) is purely based on the sentence length for different tasks, to maximize the utilization of chip memories, which do not impact the eventual training performance in our experiments.

Reviewer comment (Novelty):
In its current state, both (i) the gap in related work comparison and (ii) the missing statistics to truly evaluate the significance of the results make the significance of the paper difficult to evaluate. The reviewer can only vouch for minimal significance for now, but hopes to increase their appreciation as the review process continues.

Author response:
Thank you for your reminder. In the paper, to reduce the variance, **we perform each battle play against Random and MaxDamage players with 100 times, which we have mentioned in the paper.**

In addition, we clarify that **all experiments are ran under 3 random seeds (42, 0, 2023), and we report the average numbers.** Since each result are averaged from 300 battles, **the variance is very low** even in the early generations of self-training.

The self-training process (for 50 generations) is only performed for once because of the great training cost.

For the fine-tuning experiments on two downstream tasks, **we ran only for once in the paper,** and this is a drawback. **Hence, we rerun the experiments over 3 seeds** (42, 0, 2023). We report the new results as well as the standard deviation below:

|        | Boolean QA  | Inference   |
|--------|:-----------:|:-----------:|
| 0      | 77.4 (1.14) | 46.0 (1.63) |
| Gen 5  | 80.2 (0.86) | 46.0 (0.00) |
| Gen 20 | 80.7 (0.90) | 49.7 (1.25) |
| Gen 30 | 86.0 (1.56) | 55.0 (0.82) |
| Gen 50 | 86.6 (1.03) | 60.0 (2.90) |

We will specify this setting in our new version. Thank you.

Reviewer comment (Experiments):
The motivation for choosing Pokemon as a case study of outworld is ambiguous and unclear. Even though I agree that \textsc{Pokémon} has unique and interesting worldviews, I think it is not adequate to examine LLMs' outworld knowledge through a single strategy game.

Author response:
**W1&Q1** why choosing the Pokemon world as a case study

**Objectively, Pokemon is well-known enough and has a complete world setting. One can find a complete database from the web to verify our statements in the paper.** In addition, it is a strategy game with clear rule settings, which suits for crafting reasoning samples.

Subjectively, the research is from personal interest.

We notice that it is the limitation that we cannot totally generalize the pokemon knowledge to other outworld knowledge. It is also an interesting future direction to perform a more thorough evaluation of outworld knowledge in LLMs. However, the contribution of our paper is not solely an evaluation of outworld knowledge. Thank you.

Reviewer comment (Experiments):
Why is \textsc{Pokémon} considered as a good case study for outworld?

Author response:
**W1&Q1** why choosing the Pokemon world as a case study

**Objectively, Pokemon is well-known enough and has a complete world setting. One can find a complete database from the web to verify our statements in the paper.** In addition, it is a strategy game with clear rule settings, which suits for crafting reasoning samples.

Subjectively, the research is from personal interest.

We notice that it is the limitation that we cannot totally generalize the pokemon knowledge to other outworld knowledge. It is also an interesting future direction to perform a more thorough evaluation of outworld knowledge in LLMs. However, the contribution of our paper is not solely an evaluation of outworld knowledge. Thank you.

Reviewer comment (Evaluation):
From the analysis of outworld knowledge in the pokemon world, what kind of behaviors can we infer that LLMs would exhibit in other new worlds in general?

Author response:
**Q2** what we can infer LLMs in other new worlds

From the results in our paper, we highly conjecture that LLMs cannot tackle simple reasoning tasks in new worlds effectively, and suffer from hallucinations that are even more severe than those in the real world.