C hat GPT ’s Information Seeking Strategy: Insights from the 20-Questions Game (2023)

Abstract: Large Language Models, and ChatGPT in particular, have recently grabbed the attention of the community and the media. Having reached high language proficiency, attention has been shifting toward its reasoning capabilities. In this paper, our main aim is to evaluate ChatGPT’s question generation in a task where language production should be driven by an implicit reasoning process. To this end, we employ the 20-Questions game, traditionally used within the Cognitive Science community to inspect the information seeking-strategy’s development. This task requires a series of interconnected skills: asking informative questions, stepwise updating the hypothesis space, and stopping asking questions when enough information has been collected. We build hierarchical hypothesis spaces, exploiting feature norms collected from humans vs. ChatGPT itself, and we inspect the efficiency and informativeness of ChatGPT’s strategy. Our results show that ChatGPT’s performance gets closer to an optimal agent only when prompted to explicitly list the updated space stepwise.

## introduction chatgpt's impressive ability to solve numerous natural language tasks has put it in the spotlight of academia and media attention @xcite . the success on a variety of tasks has brought people to even claim that gpt-4 "could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (agi)" @xcite . others are more cautious, showing the weakness of the model's reasoning abilities, (e.g., @xcite . a core aspect of human intelligence is the implicit connection between the reasoning process and language production. this connection strongly drives the generation of questions in informationseeking scenarios which, therefore, have been largely studied in cognitive science. after the questioner answerer sp 1. is it a weapon? no 4 2. is it sweet? yes 2 3. is it the watermelon? no 1 4. is it the mulberry? yes pioneering work by @xcite , the 20-questions game has been employed to observe children's cognitive developmental trajectory: a player thinks of an entity, the second player is given a set of candidates (e.g., cat, dog, bird) and has to identify the target entity among the possible candidates by making yes-or-no questions. this and following experiments have shown that through the developmental trajectory, children learn to recognize object-general features, cluster similar objects into categories and use such categorization to ask context dependent informative questions: they shift from hypothesis scanning questions ("is it a dog?") to constraint-seeking questions ("does it has four legs?"). such a shift let the elder children be more efficient in their information seeking process. moreover, pre-scholar children tend to continue asking questions when enough information has been collected (i.e., the space has reduced to one candidate). they do not know when to stop @xcite , a core skill of infor-mation search and decision-making @xcite . interestingly, @xcite uses a hierarchical version of the 20-q game, in which candidates are organized into three category levels based on shared features; by providing children with the object-related features needed to halfsplit the space, children were able to target such higher category levels, reaching the solution more efficiently. inspired by this literature, we use a hierarchical 20-q game to evaluate whether chatgpt is able to generate questions driven by its reasoning over the hypothesis space (hypsp). we leverage on the widely used feature norms elicitated from human annotators, @xcite to build the hierarchical hypothesis spaces. such norms reflect humans' knowledge representation which could differ from chatgpt's knowledge. to mitigate this potential difference, we build also hypothesis spaces using norms elicitated from chatgpt itself, (gptnorms @xcite ). figure 1 reports an example of an 8 candidate symmetric nested space based on gpt-norms. we prompt chatgpt foot_0 to play the 20-q game, both in the role of the questioner and of the answerer. 2 we aim to understand whether the (a) questioner is able to identify the high-level property that clusters the space and hence asks whether the target has that property; (b) it also knows whether all the other candidates have or do not have that property, and is able to use such information to update the hypsp stepwise; and finally (c) it understands when to stop asking questions, i.e., the hypsp is reduced to a singleton. figure 1 includes a dialogue an optimal agent could ask, if driven by an half-split search. our results show that chatgpt's performance is far from an optimal agent when having to update the space internally and it is closer to it when prompted to explicitly list the updated space stepwise. ## related work our work put together two research lines: the current effort of the ai community to evaluate chat-gpt language and reasoning skills, and the cognitive science literature focusing on the developmental trajectory of information search strategies in humans. @xcite run a deep and broad evaluation of chatgpt on a variety of well recognized benchmarks in the natural language processing community. chatgpt results to be state-of-the-art in zero-shot setting for most natural language understanding tasks. though it is more suitable for open-domain dialogue tasks, it performs well also in task-oriented dialogues, and it is able to keep track of information given in previous turns, when answering follow up questions. moreover, @xcite evaluate chatgpt reasoning skills: though it lacks inducting reasoning skills, it performs well on deductive reasoning in clean settings. however, as other llms @xcite , chatgpt as well encounters problems with complex deductive reasoning involving multi-hops, viz. a combination of facts spread in different passages of a corpus. @xcite challenged chatgpt on the visual dialogue task, originally proposed by @xcite . the informativeness of the question is measured on the quality of the caption it summarises out of the dialogue. as far as we know, this is the first work to evaluate the information seeking strategy of a llm using the 20 questions game. our research question is whether and to what extent the language generation of a llm is tied to reasoning. ## agent roles below we describe how we employ chatgpt as game players to generate the dialogues and as diagnostic agents to evaluate the questioner's information seeking ability. ## experimental setup we are interested in understanding whether chat-gpt's language generation is driven by its reasoning process. to answer this question, we propose a number of measures aimed to shed light on the reasoning processes that are implicit in the game: identify the high-level property shared by several items, update the space stepwise, and efficiently arrive to a space with just one possible candidate and realize that it is time to stop asking questions. not having the possibility to run an ablation study of the model, we simulate it by comparing chatgpt-q, simply prompted to play the game, with chatgpt-q-stepwise which is explicitly told to update the space turn by turn. information seeking strategy following the method used in the cognitive science literature to evaluate children's developmental phases, we evaluated the information seeking strategy used by chatgpt-q by observing the type of questions it asks and their informativeness. first of all, we compute the percentage of questions that are hypothesis-scanning (hs) and constraint-seeking (cs). a question is considered hs iff it explicitly mentions one of the candidates in the hypothesis space. all the other questions are considered cs. we compute the percentage of hs and cs questions within a game and by the position of the turn within the dialogue. 6 following @xcite , meder et al. (2019) and testoni (2023) , we compute the expected information gain (eig) of each question and report the average eig per turn. 7 as clearly explained in @xcite , the information gain of a question is the entropy in the space (given by the number of items and the associated probability) at turn t i before asking the question minus the expected entropy after asking it (t i+1 ): as in @xcite , in our case, we consider all items in the space to be equally likely to be the target. hence, what defines entropy is the number of items within the subsets answered with yes vs. no, based on the external oracle annotation. a model that asks fewer hs, especially in the earlier turns is closer to the more efficient strategy used by adults. its question eig is expected to be very high in the early turns and to decrease in the later ones. hypothesis space update a core skill of the questioner playing the 20-q game is the ability to mentally keep the space of the hypothesis updated stepwise. we evaluate whether chatgpt-q is able to update at each turn the hypothesis space based on the given dialogue history. again, we consider the yes/no-annotation obtained from chatgpt-oracle as the ground truth and compute the hypothesis space at turn hypsp t i by filtering out from hypsp t i-1 the items which do not have the property required at t i . we compare the ground truth hypothesis space with a) the one generated by the external guesser, chatgpt-guesser, and b) the one generated by the questioner itself when 6 a third type of questions are the pseudoconstraintseeking (pcs) which ask about a property but actually refer to only one item among the candidates. for the sake of simplicity, we do not consider them in our analysis, but see the supplementary material for statistics on them. 7 we computed the eig adapting the code by @xcite . for chatgpt-guesser, a high difference would mean that the model has difficulty integrating the information collected through the dialogue history. while for chatgpt-q-stepwise it would signal a difficulty in integrating the answer with the question turn by turn. ## results in this section, we show the results we obtained following the experimental setup defined in section 5. we start by evaluating chatgpt's performance on the games with 8 candidates selected with the mcrae-norms (8-mcrae), and we then move to compare these results with those obtained by the model when challenged with spaces containing an higher number of candidates (16-mcrae), or whose nested structure reflects the model's knowledge representation (8-gpt). finally, we move to evaluate chatgpt on the 8-mcrae games when asked to play the game (chatgpt-q) and when asked to explicitly update the hypothesis space stepwise while playing the game (chatgpt-q-stepwise). ## qualitative analysis in this section, we further dive into the quality of the dialogues generated by chatgpt. first of all, we inspect whether it asks questions that do not reduce the space at all (trivial questions, tq) or refer to candidates that have already been excluded in previous turns (contradictory questions, cq). as we can see from the statistics reported in ## conclusion our work shows that chatgpt is able to identify superordinate features shared by items and ask questions that efficiently reduce the hypothesis space. at the first turn, it is close to an optimal agent using a half-split search. in later turns, however, it has difficulties making questions with respect to the updated space of the hypothesis. this weakness might be behind the high number of games in which it keeps on asking questions even though the dialogue history had led to identifying a possible target. we conjecture this behavior is not due to the lack of knowledge required by the game since it is displayed not only within the games based on mcrae norms but also on those built out of gpt feature norms. our conjecture is reinforced by the increased performance reached by the model when prompted to explicitly update the space before asking the next question. in this setting, the dialogue becomes shorter with fewer contradictory questions. yet, even in such scenario, it does not notice when the answerer reveals the target accidentally. our results call for attention to modeling the human ability to keep a mental scoreboard, echoing what stated in @xcite ; @xcite ; @xcite . finally, our work relates to the chain-of-thought @xcite and similar prompting strategies, which we plan to investigate in the future.