Paper: The Entity-Deduction Arena: A playground for probing the conversational reasoning and planning capabilities of LLMs (ICLR-2024)

Reviewer comment (Evaluation):
The major concern is that entity-deducing game is one small problem in terms of conversational reasoning and planning capabilities of language models. I am not convinced that a single game is a full reflection of the LLM reasoning ability

Author response:
**Entity-deducing game is one small problem**: We completely agree that the benchmark we used does not fully reflect the reasoning ability of the LLM. Nevertheless we thinkg this problems adds to the well needed suite of test cases for measuring these characteristics of LLMs. We will revise the paper to make this point clearer. Please refer to our general response.

Reviewer comment (Experiments):
The experimental scale in the paper is limited. The conclusion in the paper should be drawn from at least thousands of samples.

Author response:
**Experimental Scale**: We thank the reviewer for the suggestion. Please refer to the general questions for more discussion.

Reviewer comment (Evaluation):
There needs to be precise, quantifiable evaluation of the judge's performance. Without this evaluation it's impossible for a reader to understand how much of an influence its errors has on the game.

Author response:
**Evaluating the Judge**: We acknowledge the need for better quantification of the errors made by Judge. Since obtaining the ground truth is challenging, we decided to conduct a human evaluation of Judge (GPT3.5-turbo-0613) responses. This evaluation involved non-paid volunteers from our institute who assessed the correctness of Judge's responses to questions asked given the oracle entity or celebrity name. Due to time constraints during the rebuttal, based on our current annotation progress, we were only able to evaluate 223 Judge responses on Things and 237 Judge responses on Celebrities. The human evaluators identified 7 out of 223 (3.13%) and 7 out of 237 (2.95%) responses as problematic or incorrect. A more comprehensive summary of these results will be included in our revision.

Reviewer comment (Evaluation):
The evaluation set is really small! It's hard to conclude general performance on asking clarification questions when the set of target entities is so small. Ideally, we would be able to analyze performance across a number of entity features: how rare it is, how prototypical it is, how precise of an entity it is (e.g., specific breed of dog vs. dog in general), whether it's abstract or concrete, etc. With just 30 entities, it's hard to make any conclusions along these axes of difficulty. Figure 2 starts to get at this, and I think there are hints at interesting findings here (even just looking at human performance, to be honest) -- for example, umbrellas and cookie cutters are non-prototypical instances of their hypernyms in wordnet (respectively "canopy" and "kitchen utensil"). But with only 30 evaluation examples I don't think any strong conclusions could be made.

Author response:
**The evaluation set is small**: We agree that having a larger evaluation dataset would enable more comprehensive and detailed analysis at a finer granularity. To this end, we are currently working on expanding the evaluation by incorporating 300 examples for each dataset, across all the models being compared. We will incorporate the results of this evaluation in our general response.

Reviewer comment (Reproducibility):
Details on recruiting and managing crowdsourcing for the human baseline needs to be included in the main paper. There are also no details I could find in the appendix, e.g., on pay or the crowdsourcing platform used. There appears to be a lot of noise in the human dialogues (e.g., questions in the bottom game of table 6 appear to be obviously non-optimal) and I am wondering how workers were incentivized to actually try at the game. I also think analysis comparing the human strategies and LLM strategies should be in the main paper.

Author response:
**Human evaluation details**: We have discussed some details of creating the human baseline in the Appendix D. We agreed that some contents needed to be moved to the main text, while some information needed to be supplemented. In this study, we did not hire any paid crowd-source workers. As discussed in Section 3, because the complexity of the multi-turn game setup, we developed an internal demo server (shown in figure 4) and relied solely on enthusiastic internal volunteers who were interested in our study. Following a 30-day trial period, 108 volunteers contributed to this study, and we successfully collected a total of 216 human game play sessions for Things and Celebrities.

To ensure the quality of the data, we manually inspected human game plays to filter out low quality data. The authors reviewed all human game play sessions, and removed sessions that do not complete the entity deduction task, contain irrelavant chit-chat, or contain nonsensical repeated questions, leaving only sensible game plays. In the end 201 sessions were selected as valid game play.

We appreciate the suggestion and will incorporate a thorough discussion comparing the strategies employed by humans and LLMs into the main text.

Reviewer comment (Experiments):
I would have liked to see evaluation of human players on the reasoning task given a dialogue history generated by a model (and vice versa); essentially the experiment in Table 4 but replacing Vicuna 7B with a human. Since humans appear to perform worse, is this because they are worse at planning or reasoning (as defined in this paper)?

Author response:
**Evaluation of Human players on reasoning tasks**:
 
We thank the reviewer for the valuable suggestion. While we are still awaiting the full results, we would like to share our current findings below:


| Approach | Things | Celeb |
|--------------------------|---------------|------------|
| GPT-4 | 0.40±0.05 | 0.48±0.03 |
| Vicuna 7B | 0.10±0.04 | 0.04±0.03 |
| Human | 0.20±0.04 | 0.25±0.03 |
| Vicuna 7B → Human | 0.1 | |
| GPT-4 → Human | 0.14 | |
| Human → Vicuna 7B | 0.11±0.04 | 0.03±0.01 |
| Human → GPT-4 | 0.15±0.01 | 0.18±0.06 |

Note that these results are based on the evaluation set in the submission for consistency. In our next update, we plan to extend the study to include a new dataset with 300 items/names. Following the setup outlined in the paper, only the last turn for each dialog trajectory is replaced. We are currently relying on non-paid volunteers for the annotation. 

The findings indicate that when GPT-4 is employed for the last step reasoning in human trajectories, there is a decrease in performance. One possible explanation for this is that while GPT-4 may excel in comprehending model-generated trajectories, it may struggle to fully interpret the planning trajectory of humans based on the observed turns. This suggests that models may be more compatible with model-generated trajectories rather than human trajectories. For instance:

| Turn | Human Question            | Judge Answer | GPT-4 Question   |
|------|---------------------------|--------------|------------------|
| 1    | Is it a VR Headset        | No.          |                  |
| 2    | Is it made of metal       | Yes.         |                  |
| 3    | Is it used at home?       | Yes.         |                  |
| 4    | Is it a chair?            | No.          |                  |
| 5    | Is it a table?            | No.          |                  |
| 6    | Is it a faucet            | No.          |                  |
| 7    | Is it a fridge?           | No.          |                  |
| 8    | Is it a bed frame?        | No.          |                  |
| 9    | Is it a window frame?     | No.          |                  |
| 10   | Is it a stationary        | Yes.         |                  |
| 11   | Is it a pen               | No.          |                  |
| 12   | Is it a pencil box        | No.          |                  |
| 13   | Is it a scisor            | No.          |                  |
| 14   | Is it a letter opener?    | No.          |                  |
| 15   | Does it write?            | Yes.         |                  |
| 16   | Is it a metal pen         | No.          |                  |
| 17   | Is it a mechanical pencil | No.          |                  |
| 18   | Is it a printer           | Bingo!       | Is it a Stapler? |

Even though it was previously established that the entity can write, GPT-4 still asked if the entity is a Stapler or not.

Reviewer comment (Novelty):
I think the finding that these models have consistent strategies in question asking is cool, but it's not very surprising, because these are (mostly) deterministic models after all.

Author response:
**Deterministic models**: In our study, we compute statistics for each game by performing 5 repetitions for each game. We apply a sampling approach and use a temperature of 0.8 for the guesser models throughout our study. We observed that the gameplay can vary significantly, resulting in completely different outcomes. For example, the failure results presented in Table 2, 8, 12 are derived from the same model, entity, and settings.

Reviewer comment (Experiments):
How were the entities chosen? The paper mentions they are manually curated, but from where?

Author response:
**How were the entities chosen**: We manually created several categories and then asked GPT-3.5 to generate additional categories. Next, we asked it to generate unique entities for these categories without repetition. We will provide the prompts for creating these entities and celebrities' names in our revision.

Reviewer comment (Evaluation):
Why are the responses different for the celebrity dataset? ("dunno" vs. "maybe")

Author response:
**Why the Celebrities dataset use "Dunno"**: The Things dataset contains entities that has many different forms, such as if asking "is this made by metal?" for the entity "Chopstick". The answer to this question could be "Maybe" because sometimes they are made by metal and sometimes are not. On the other hand, in the Celebrities dataset, most questions can typically be clearly answered with a yes or no. These questions often refer to gender, birth year, or other specific information about a person. However, there is a possibility that the model might not know the answer to certain questions, such as "Does this celebrity smoke?". In these cases, the answer could be "Dunno" as it is considered very personal information. As a general observation, we have found that less than 1% of the responses in the Celebrities dataset are "Dunno", while around 20% of the responses in the Things dataset are "Maybe". Note that the percentage of "Dunno" or "Maybe" responses can vary depending on the specific Guesser model being used.

Reviewer comment (Writing):
What does "retrospective perspective" mean in Table 3 caption?

Author response:
**Meaning of Retrospective perspective**: We used strikethrough lines to indicate the guesses that were inconsistent with the Judge's answer, based on our knowledge and evaluation. This visual representation illustrates the authors' thought process to identify the irrelevant guesses after obtaining the Judge's response. Note that we observed a strong correspondence between our assessment of irrelevant guesses and the model's top predictions. This suggests that the model may be capable of similar reasoning to eliminate certain guesses. However, this observation has not been quantitatively verified yet. We will clarify these points in our paper.

Reviewer comment (Evaluation):
How do you measure uncertainty in the model's top-N predictions as mentioned in the discussion of RQ1?

Author response:
**Measurement of uncertainty**: We asked the Guesser model, "On a scale of 1-5, how confident are you in your top guesses based on the current conversation history?" We observed some qualitative correlation between the model's predicted uncertainty and significant shifts in its top guesses. However, this pattern was mainly observed in more powerful models such as GPT-4. In the case of weaker models like Vicuna, the top guesses or confidence scores generated by the model were not very sensible. We will include these specific details in the paper to enhance its clarity.

Reviewer comment (Experiments):
The paper lacks novelty and robustness to pass the bar of ICLR. The experiments designed don't show much improvement compared with referenced prototype. The datasets leveraged in the experiments, both in terms of range (two categories entities and things) and quantity, have certain limitations. However, the paper lacks discussion on this aspect. Moreover, given the inherent randomness in GPT's responses, it raises the question of whether multiple repeated experiments were conducted to ensure the stability of the results.

Author response:
**Not enough improvement over referenced LLM**: Perhaps we misunderstood the reviewer, but we feel that our methods show strong improvement over our baselines in Table 1 -- both BC trained models (V-FT* models in Table 1 -- e.g. V-FT 7B) and RL trained models (V-RLGP* models in Table 1 -- e.g. V-RLGP 7B) show more than 100% improvement over the base model (Vicuna 7B). Perhaps the reviewer is suggesting that we compare our trained models against GPT-4 ? In that case it would be unfair since the GPT-4 model is much larger than the models we trained.  

**Not enough diversity in the datasets and no discussion**: In section 3.1 and Appendix A, we discussed that we aim to create rich and diverse datasets. In fact, our datasets are diverse. The **Things** dataset encompasses a wide range of categories, such as objects, animals, foods, plants, vehicles, clothing, professions, materials, instruments, places, birds, sports, buildings, furniture, celestial bodies, mythical creatures, games, body parts, beverages, weather phenomena, groups, gemstones, toys, tools, patterns, appliances, and microorganisms, among others.

The **celebrities** dataset meanwhile includes names from different nationalities, eras of life, and various occupations, including entertainment, sports, politics, entrepreneurship, science, philanthropy, writing, music, military, engineering, and more. We used 980 entities for the Things dataset and 133 entities for celebrities. Additionally, we are in the process of collecting and evaluating 300 additional examples for each dataset to further expand its size. The results of this effort will be shared in our general response.

**No repeated experiment**: We apologize if we didn't emphasize this enough in the paper, but we did in fact repeat our experiments. In Table 1, our main results were obtained by conducting 5 repetitions for each gameplay, which provided the standard deviation as shown in the paper. This was further discussed in Section 4.

Reviewer comment (Experiments):
The observation and conclusion from experiments seem rather preliminary. The "entity deduction game" and the so-called "path planning" requires a more strict formulation for deeper analysis. This is especially true for the criterion of what can be a good path.

Author response:
**Lack path planning analysis**: In fact, we provide a comprehensive qualitative analysis of what constitutes a good or bad path for the game. In Table 2 and Tables 8, 9, and 12, we have highlighted how the models are different in their planning ability and illustrated and explained the characteristics of good "path planning" and identify the common failure modes that lead to inadequate planning. The criteria for a good path indeed depends on the domain and may be hard to generalize at this stage of our understanding of LLMs. However, we discussed some general observations. In Table 3 and 13, we present the criteria for successful planning, which include 1) prioritizing high-level questions before addressing specific details and enumerations, 2)being aware of the current state and asking questions to effectively bi-partition the search space, and 3) being able to recognize when the current path is incorrect and occasionally backtracking to consider previously overlooked options. All of these aspects are discussed in Section 4 of our paper, and we will make them clearer.

Reviewer comment (Experiments):
The ability of path planning can easily be influenced by domain knowledge. I suggest that if we want to rigorously examine planning capability, perhaps we could consider a much narrower and common sense domain, for example, having GPT guess numbers within a certain range. PS: I did try this number-guessing experiment on GPT-4. GPT-4 doesn't necessarily choose the binary search as the optimal path; it often opts for a narrower range enumeration.

Author response:
**Comments regarding domain knowledge**: We commend the reviewer's diligence in exploring the range guessing game. We think that it is a much simpler setup which may not adequately address some of the questions we are raising about the state of the art LLMs. As far as domain knowledge goes, even in the range guessing game each model may have a varying level of understanding of arithmetic, such as determining which number is larger or smaller and thus disentangling knowledge from planning would be a problem even in that set up. Domain knowledge is indeed a complicating factor but we believe that it would be difficult to entirely disentangle the knowledge and planning and remove the knowledge factor in any set up. In fact, the goal of our set up is to test a model's ability to **handle** the ambiguity in its domain-knowledge with better planning. We chose a real world game on common items because most humans are able to handle this problem intuitively without any specialized training in a domain like mathematics and wanted to explore how LLMs perform comparatively on this.

At the same time, we did indeed make an attempt to explore how domain knowledge plays a role in this set up. In section 4 and Appendix E, we discuss how stronger models possess a higher level of domain knowledge compared to humans. It is important to note that our focus is on the Guesser role, which aim to guess rather than provide accurate answers. However, it is undeniable that a stronger model will have a more comprehensive taxonomy representation of entities in its intrinsic knowledge store. We considered this an essential aspect of the planning ability, as we believe that the ability to plan is closely linked to a deep comprehension, digestion and representation of knowledge.