Measuring and Narrowing the Compositionality Gap in Language Models (2023)

Abstract: We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems. We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap. We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining. In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease. This surprising result suggests that while more powerful models memorize and recall more factual knowledge, they show no corresponding improvement in their ability to perform this kind of compositional reasoning. We then demonstrate how elicitive prompting (such as chain of thought) narrows the compositionality gap by reasoning explicitly instead of implicitly. We present a new method, self-ask, that further improves on chain of thought. In our method, the model explicitly asks itself (and then answers) follow-up questions before answering the initial question. We finally show that self-ask’s structured prompting lets us easily plug in a search engine to answer the follow-up questions, which additionally improves accuracy.

## introduction compositional reasoning lets models go beyond rote memorization of directly observed facts to deduce previously unseen knowledge. for example, a model should be able to answer "how long was queen elizabeth's reign?" even if the answer did not explicitly appear in the training data, by recalling her and her father's death dates and rea1 no models were trained or finetuned in the making of this paper. code and data at: https://github.com/ofirpress/ self-ask soning over these facts. while language models (lms) have shown strong question answering performance, it remains unclear how much is due to memorization of huge corpora vs. how much is due to reasoning. first, we quantify the reasoning abilities of lms using multi-hop question answering. we present a new, automatically generated dataset, compositional celebrities (cc), of 8.6k 2-hop questions; it combines frequently stated facts in improbable ways (e.g., "who won the master's tournament the year justin bieber was born?"), allowing us to disentangle memorization and reasoning. intuitively, the reasoning required to answer such questions seems trivial given access to the relevant facts. we introduce the term compositionality gap to describe the fraction of compositional questions that the model answers incorrectly out of all the compositional questions for which the model answers the sub-questions correctly. unsurprisingly, we find that performance on both single-and multihop question answering improves monotonically with the scale of the pretrained model. intriguingly, however, we find that the compositionality gap remains at a roughly constant 40% between different model sizes and training techniques, with no apparent improvement from scale (fig. 1 ). 2 this result is especially surprising given the straightforward reasoning steps required for such questions, and it suggests that larger scale pretraining is highly effective at teaching models to memorize facts but not how to compose them. we also find a positive correlation between lm confidence about a certain fact and whether it can compose it with other facts to answer compositional questions about it. we next narrow the compositionality gap by using what we call elicitive prompts. compositional questions require more computation and knowledge retrieval than 1-hop ones; however, by using naive prompting (which expects the answer to be output immediately after the question), we always give the model approximately the same number of steps to answer questions. we show that elicitive prompting, such as chain of thought @xcite , lets the model "think things through" before it outputs a final answer, which markedly boosts performance. we then show that our self-ask prompt, where the prompt has the lm decompose complex questions into easier sub-questions that it answers before answering the main question, can improve performance even more. beyond cc, we also apply elicitive prompts to two existing, automatically generated datasets, (2wikimultihopqa, @xcite , and on a third dataset of 125 questions, bamboogle, that we manually constructed. bamboogle is a dataset with 2-hop questions written by the authors, where all questions are sufficiently difficult to be unanswerable by a popular internet search engine, but where both supporting pieces of evidence can be found in wikipedia (and hence were probably included in the pretraining set of any lm). the two datasets we constructed-bamboogle and the previously mentioned compositional celebrities-are complementary and serve different research purposes. bamboogle is a small, handcrafted dataset that covers many different types of questions on different areas written in unique ways, whereas cc (similar to musique and 2wiki-multihopqa) is a large, automatically generated dataset where each question fits into one of the 17 templates we made (i.e., much lower variation than bamboogle). compositional celebrities is designed to estimate the compositionality gap on a large set of questions, and bamboogle is designed to measure the extent to which a question answering system can answer varied compositional questions, albeit with less statistical power. finally, we show that the structure of self-ask combines easily with an internet search engine to further improve results on compositional questions. in summary, we systematically reveal that although lms can sometimes compose two facts they observed separately during pretraining, they fail to do so in a large fraction of cases, even when they demonstrate knowledge of the constituent facts individually. we call this ratio the compositionality gap and show that it does not shrink with scale. we show that elicitive prompts, such as chain of thought and our self-ask prompting, narrow and sometimes close this gap and improve the ability of lms to solve complex compositional questions. finally, self-ask can be easily combined with a search engine to further improve performance. ## systematically measuring the compositionality gap as lms grow in size, they contain increasing amounts of knowledge about the world @xcite . but how does their compositional ability scale? we investigate this with a new method that shows how to formally quantify the compositional ability of an lm. our method is based on 2-hop questions that are grammatical but unlikely to have been previously uttered, e.g., "what is the calling code of the birthplace of frida kahlo?" we generate them by crawling a list of celebrities and where/when they were born. we then retrieve facts about each birth country (capital, currency, calling code, ...) and each birth year (winner of the master's tournament or nobel prize in literature that year ...), and we generate 2-hop questions by combining pairs of facts. appendix cc, which we design to measure the compositionality gap, intentionally contains direct and unambiguous questions, where (1) each fact has likely appeared many times in the training dataset, but (2) the combination of both facts is sufficiently unnatural that it likely never appeared in the training set or on the internet at all. 3 this question format has many advantages: almost all questions have a single correct answer, they are easily decomposable into sub-questions (letting us verify if the lm knows the background facts), and for most questions the answer domain is very large (unlike yes/no or multiple choice questions). hence, the chances of randomly guessing the correct answer are low. we evaluate gpt-3 on cc, using a 2-shot prompt, separately for the 2-and 1-hop questions. we use a category-specific prompt for each question that contains 2 randomly chosen examples that have been removed from the dataset; see appendix tables 4 and 5 . gpt-3 (davinci-002) correctly answers 45.4% of the 2-hop questions. in some categories, such as birthplace/domain name, the accuracy reaches 84.6% even though the model most likely did not explicitly see the vast majority of these questions during training. the model even answers some extreme questions correctly, such as "what is the top-level domain of the birthplace of plato?" on the hardest categories, such as birth year/literature nobel prize winner, gpt-3 answers only 1.2% of the questions correctly. however, it answers 80% of the sub-questions on this dataset correctly, showing that it has seen many of the individual facts required to answer these questions but lacks sufficient compositional capabilities to answer the question correctly. appendix we refer to the fraction of questions for which the model correctly answers individual subquestions but not the compositional question as the compositionality gap. figure 1 shows that the compositionality gap surprisingly does not shrink as we increase gpt-3 model size for both the instructgpt and non-instruct families of models. this shows that as the gpt-3 model size grows, it knows more about the world, but its ability to compose this knowledge increases at a slower rate. we have shown that gpt-3 can occasionally store a fact well enough to recall it when it appears alone (e.g., "when was justin bieber born?" (1994) and "who was the champion of the master's tournament in 1994?" (josé maría olazábal)). however, it cannot compose these two facts ("who was the champion of the master's tournament in the year that justin bieber was born?"). how can we determine which facts gpt-3 will and will not be able to compose? appendix figure 5 shows that as the perplexities assigned to correct sub-question answers decrease (i.e., the model becomes more confident about the correct answers), the probability of answering the compositional question correctly increases. for example, when the maximal perplexity assigned to the correct sub-question answer (i.e., assigned to the correct answer that the model is less confident about of the two sub-questions) is between 1.232 and 6.738, the model answers 42.6% of the compositional question correctly. however, when the maximal perplexity is between 1.000 and 1.002, the model answers 81.1% of the compositional question correctly. we observed a similar pattern when sorting sub-question pairs by average instead of worse perplexity. being able to answer a 1-hop question correctly in a prompt containing similar 1-hop questions does not mean that the model has fully "learned" the given fact. our results indicate that a model can compose facts at a much higher rate when it can recall these facts more confidently, i.e., with a low perplexity assigned to the correct answer for the 1hop question. this suggests that additionally reporting the perplexity assigned to the correct answer might be a better way to evaluate lms on downstream question answering tasks as opposed to just reporting accuracy. concurrent papers @xcite recently advocate for this, but from a different perspective. they show evidence that as model size scales, cross-entropy loss can indicate an improvement on downstream task performance even when actual task metrics (i.e., accuracy) do not improve. ## elicitive prompts narrow the compositionality gap and improve question answering performance we next show that using our elicitive prompt (selfask, figure 2 ) that first decomposes multi-hop questions into simpler sub-questions, then answers the sub-questions, and finally answers the main question allows the lm to answer compositional questions with a much higher success rate. we show this on four datasets: compositional celebrities, two previously existing datasets, and a new dataset, bamboogle; we design this new dataset by manually creating compositional questions that are simple enough for a popular internet search engine to propose an answer to but hard enough so the answer it proposes is incorrect. this also provides evidence that the question did not previously appear on the web and so would probably not be contained in the lm's training set. in addition, all questions are 2-hop, where both "hops" are sourced from wikipedia and so should be in the training set of any lm. as opposed to cc (and similarly musique and 2wikimultihopqa), in which each of the 17 categories of questions conform to the same template, bamboogle questions are almost all independent of one another. we therefore consider it a fitting complement to the other datasets for measuring lm compositional abilities. the two existing datasets we chose are 2wikimultihopqa @xcite and musique @xcite . we use them in an open-domain setting (these are not multiple choice datasets) and, like @xcite and brown et al. (2020), we use only the question-answer pairs from these datasets, not any passages of relevant text that they contain. these datasets both contain 2-hop compositional questions sourced from facts that appear in wikipedia articles. we can divide prompting approaches for question answering into two broad categories. the first category, direct-answer prompting, is the naïve prompting method used by @xcite , where each question is directly followed by its answer; see appendix figures 4 and 5 for examples. the second category, elicitive prompts, lets the model "talk things through" before answering. chain of thought and scratchpad @xcite are examples of prompts in this category. if we expect an immediate answer to a question, the amount of computation the lm employs to answer it is always approximately the same assuming a relatively similar question length that is not necessarily longer when the question is tougher. to let the model apply more computation to harder problems, we can allow it to "talk things through." we find that elicitive prompts lead to much higher accuracy on compositional questions. this can be observed in appendix figure 6 , which shows that on cc, elicitive prompts can occasionally answer even more compositional questions than direct prompts correctly answer sub-questions for, separately. this might be because elicitive prompts contain more information than direct ones. note that the rest of this section shows that elicitive prompts improve performance but does not show that they narrow the compositionality gap since we lack subquestions for datasets other than cc. "let's think step by step" from @xcite is also an elicitive method, but in our experiments it obtained 45.7%/1.1% accuracy with instructgpt-davinci-002/davinci whereas self-ask obtains 79.6%/54.2% accuracy with those models on cc. this is consistent with results from @xcite showing that their method is not as strong as chain of thought and that using non-instruct models further degrades performance. we therefore do not run more experiments with this method here. ## related work prior to the chain-of-thought and scratchpad methods, @xcite showed that sequence-tosequence models for solving arithmetic problems benefit from generating a rationale before outputting the final answer. allowing models to use more computation to solve harder problems has been previously explored in both language @xcite @xcite @xcite @xcite and vision @xcite @xcite . previous papers have explored decomposing a complex task into a set of simpler sub-problems. iyyer et al. (2017 @xcite trained supervised models to decompose compositional questions into sub-questions. these papers did not use a causal pretrained lm. @xcite shows that manually decomposing instructional prompts into simpler sub-steps improves performance. @xcite ; @xcite ; @xcite use a pretrained causal lm to generate context to improve performance on multiple choice commonsense tasks. @xcite use chain of thought to answer compositional questions that are presented with a context paragraph. here we do not input any context when presenting a question to the lm. @xcite tested masked lms on 2hop questions. cc contains 2-hop questions with an additional constraint, namely, that the 2 facts being composed are likely to have been observed separately many times but unlikely to have appeared together during pretraining. @xcite showed that manually breaking down complex questions into sub-questions improves performance. our self-ask method does this automati6 the experiments our self-ask + search engine model is partially inspired by previous works that have combined neural lms with retrieval models @xcite @xcite @xcite . we are the first to propose the use of 2-hop questions as a means of evaluating the compositional abilities of large lms. lake and baroni (2017) and @xcite previously tested the ability of sequence-to-sequence models trained on an artificial dataset to integrate known parts into novel combinations. @xcite generate compositional questions to test small encoder-decoder models, but they do not test pretrained lms. in parallel to our work @xcite and @xcite present methods similar to self-ask, but they do not present our findings on the compositionality gap, integrate their prompt with a web search engine or present new datasets as in our compositional celebrities and bamboogle. ## conclusion we present the compositionality gap. we then show that our self-ask improves over chain of thought by having the lm explicitly state and answer followup questions. finally, we improve self-ask by using while we showed here that the compositionality gap stays at around 40% for models of size between 1 billion to 175 billion parameters, we do not have access to, and so did not experiment with, models that are larger than 175 billion parameters. while this pattern is consistent across different variations of gpt-3 (both vanilla and instructgpt), the behavior of models that are larger than 175 billion parameters could potentially be different. the experiments in this paper focus on 2-hop question answering datasets in english, since we believe that these datasets are a powerful tool for an exploration into the reasoning abilities of language models and because we believe that focusing on these datasets will lead to better systems for applications such as user-facing question answering systems like siri. our limited manual experiments on other datasets, such as ones for semantic parsing or for arithmetic problems or logical puzzles, showed that self-ask also works on those very different problem sets. but a more thorough empirical evaluation of those could reveal different results. examples of bamboogle questions and answers appear in figure 4 shows an example of the popular internet search engine returning an incorrect "featured snippet" to a query containing a 2-hop question. to construct bamboogle we manually wrote 2-hop questions and then checked whether they lead the search engine to return featured snippets that contain the wrong answer. this provides evidence that the given question does not appear on the web, and so would likely not appear in the training set of any lm. figure 4 : wrong answer in a featured snippet. the question was "who was the head of nasa during apollo 11?". the nasa administrator during apollo was thomas o. paine. as can be observed from the snippet, eugene kranz was chief flight director and never a nasa administrator.