Is It Smaller Than a Tennis Ball? Language Models Play the Game of Twenty Questions (2022)

Abstract: Researchers often use games to analyze the abilities of Artificial Intelligence models. In this work, we use the game of Twenty Questions to study the world knowledge of language models. Despite its simplicity for humans, this game requires a broad knowledge of the world to answer yes/no questions. We evaluate several language models on this task and find that only the largest model has enough world knowledge to play it well, although it still has difficulties with the shape and size of objects. We also present a new method to improve the knowledge of smaller models by leveraging external information from the web. Finally, we release our dataset and Twentle, a website to interactively test the knowledge of language models by playing Twenty Questions.

## introduction generative language models achieve strong performance on multiple nlp tasks by using an unsupervised training objective: predicting the next token in a string of text @xcite @xcite . despite the simple training objective, these models capture a significant amount of world knowledge @xcite @xcite . however, we can quickly uncover some limitations by asking simple questions. for example, gpt-3 @xcite is more likely to complete the following sentence question: is a kettle smaller than a tennis ball? answer: with yes than no. while trivial for a human, gpt-3 has trouble comparing the size of a kettle and a tennis ball. we can use the let's think step by step method to look into the chain of reasoning of gpt-3 @xcite : question: is a kettle smaller than a tennis ball? answer: let's think step by step. [...] a tennis ball is about 6 inches in diameter [...] a typical kettle is about 8-10 inches tall and has a figure 1 : example twenty questions game: a human must discover the hidden entity (a keyboard) by asking yes/no questions to the language model. in this case, the model needs to know about the shape, composition, and purpose of a keyboard to correctly answer all questions. while trivial for humans, our results show that this is not the case for most language models, except for gpt-3, which displays fantastic world knowledge on all questions except size-related questions. in this work, we try to analyze the world knowledge of language models through the game of twenty questions. we collected a dataset of 2000+ questions and tried to understand the strength and weaknesses of language models by classifying questions into nine categories of knowledge (usage, size & shape, appearance). our results show that gpt-3, a 175 billion parameters language model, can play twenty questions thanks to a consistent world knowledge on all categories identified, except for size & shape questions (e.g., is it bigger than a foot). unfortunately, we also show that smaller models do not display the same consistency. however, leveraging the web improved the knowledgeability of t0 by 10% and brought it to a level competitive with gpt-3, despite having 16 times fewer parameters. our contributions are the following: we publicly release our dataset on huggingface @xcite . 1 we also present twentle, a website to interactively test the world knowledge of language model by playing the game of twenty questions. ## related work although analyzing the capabilities of language models through the game of twenty questions is new, researching the amount of general knowledge and common sense of language models is not. unfortunately, the knowledge stored by language models is not symbolic. therefore, we cannot look into the model and inspect its knowledge. instead, previous work relied on multiple proxy tasks. one option is to use regular reading comprehension datasets in a closed-book format. @xcite follow this approach. they evaluate how much knowledge can be stored inside the weights of a text-to-text t5 model @xcite . the authors repurposed three reading comprehension datasets to closed-book question answering: web questions @xcite , trivia qa @xcite and natural questions @xcite . they concluded that t5 performs on par with specialized machine comprehension models. gpt-3 @xcite was also evaluated on the same closed-book question-answering datasets. the largest model (175b parameters) achieved state-of-the-art results on triviaqa despite not being trained for the task. unfortunately, it has been demonstrated later by @xcite that the datasets used by @xcite and @xcite suffer from a considerable overlap between the training and test set, invalidating the authors' conclusion based on these datasets. furthermore, when the overlap between the training and test set is removed, the performance of bart @xcite diminishes from 26.7% to 0.8% on triviaqa @xcite , suggesting that the model is unable to generalize to previously unseen questions. to overcome the previously mentioned overlap problem, @xcite repurposed squad @xcite , a popular reading comprehension dataset, as a closed-book question answering dataset. they evaluated the performance of bart on this new dataset and concluded that it was still challenging for generative models to perform closed-book question answering. another approach is to look at how a language model fills in blanks (i.e., masking). one can estimate what the language model knows by carefully analyzing the model's suggestion. this is the approach followed by @xcite . the authors introduce a new dataset lama to test the factual and commonsense knowledge in language models. it provides a set of cloze tasks, e.g., ravens can the olmpic games @xcite ) tests the symbolic reasoning of language models through eight synthetic tasks. while very similar to our work, the dataset uses masking to probe the language model. mask tokens are only applicable to encoder language models, while we are interested in generative language models. previous studies have shown that providing generative language models with background information improves their performance. @xcite @xcite @xcite similar to @xcite , we find that including external knowledge improves the language model's performance, however, we obtain better results by restricting the source of knowledge to wikipedia instead of the entire internet. to summarize, we are the first to analyze the world knowledge of generative language models through the game of twenty questions. we depart from the work of @xcite and @xcite in several ways. first, we only have yes/no answers, which simplifies the evaluation and removes the surface-form problem @xcite . second, using generic questions allows disentangling the understanding of the object and the question. we make the distinction between generic and regular questions. generic questions refer to the entity as "it" (e.g. does it [a rake] have a seat). generic questions are asked multiple times over different entities (on average 3). we use this unique feature to disentangle the understanding of the question and the entity. ## data this section presents our dataset based on the twenty questions game -the first boolean closedbook question answering dataset regarding world and commonsense knowledge. we start this section by introducing the twenty questions game. we then explain our data collection process. finally, we analyze the type of knowledge required to perform well on this dataset. ## language models in this section, we review the subjects of this work: generative language models. language models come in all forms and shapes. however, we focus on two types: encoder-decoder and decoder-only models. ## experiments in this section, we report on our experiments using our dataset of twenty questions. we experimented with three setups: zero-shot, few-shot, and zeroshot with knowledge augmentation. we use these results in the section to understand the scale of the world knowledge stored by language models. ## world knowledge analysis we now use the results of the previous section to analyze the world knowledge of the three best models: gpt-3, t0, and t0 knowledge grounded (kg). ## twentle we present an interactive website to let anyone test the world knowledge of t0-kg by playing the game of twenty questions. inspired by wordle, we named our website twentle, available at twentle.com. ## future work reducing the world to yes/no questions is not an easy task. our human agreement section demonstrates that humans do not agree on all answers. future work is needed to compare the agreement of humans and language models by category of question. in this study, we limited ourselves to the study of the answerer. however, gpt-3 could potentially also play the role of the questioner. future work is needed to study the knowledgeability of language models on lesser-known objects. in this case, we anticipate that large models will also need to leverage the web for information. ## conclusion in this work, we analyzed the world knowledge of language models through the game of twenty questions. our analysis reveals that most language models do not have the world knowledge required to play this game. gpt-3 is a notable exception. it displays impressive world knowledge on all categories of questions identified, except for shape & size questions -is it smaller than a tennis ball. furthermore, we showed how grounding smaller models on information from the web improves their knowledgeability. through this work, we demonstrated the need for more clarity on which model architecture and pre-training method best captures world knowledge.