Comprehension Based Question Answering using Bloom’s Taxonomy (2021)

Abstract: Current pre-trained language models have lots of knowledge, but a more limited ability to use that knowledge. Bloom’s Taxonomy helps educators teach children how to use knowledge by categorizing comprehension skills, so we use it to analyze and improve the comprehension skills of large pre-trained language models. Our experiments focus on zero-shot question answering, using the taxonomy to provide proximal context that helps the model answer questions by being relevant to those questions. We show targeting context in this manner improves performance across 4 popular common sense question answer datasets.

## introduction recent large language models such as gpt-3 @xcite have made a giant leap forward in knowledge acquisition and even generalize this knowledge to a new tasks. but when less narrow tasks are considered they fail to understand as much as these benchmarks suggest. they turn out to be "stochastic parrots" @xcite or "smart/super parrots." @xcite ) that just memorize without all of the comprehension we want from a natural language understanding system. we focus on a particular kind of failure mode where the model knows (has memorized) the information it needs, but is not able to apply that information correctly, and we do so in a zero-shot fashion to control for what the model knows. for example, in fig. 1 the model is asked if a mixture of grape juice and cranberry juice is safe to drink @xcite . gpt-3 declares that it is a deadly poison, even though it appears to "know" that grape juice and cranberry juice are safe to drink by themselves (fig. 1 , level 1, dark purple). it even knows that cranberry juice with grape juice is not poisonous, but it still thinks the result is death (fig. 1 , level 2, light blue). the model has * these two authors contributed equally. memorized the necessary information from large amounts of text, but does not use its knowledge appropriately. following @xcite , we extract this knowledge as explicit language then feed it back as additional context during inference, forcing the model to use what it already knows but in our case targeting specifically useful knowledge. to formalize this distinction we drew inspiration from elementary school classrooms, where teachers @xcite have a schema based approach in which they teach children to demonstrate multiple levels of comprehension, making complex inferences and direct recall from memory. they use a hierarchy of comprehension skills called bloom's taxonomy @xcite (c.f . fig. 1 ) with memorization is at the bottom (requiring children to recall facts) followed by understanding (requiring children to grasp semantics) application (requiring children to solve problems), and more complex skills. for us, these comprehension skills describe ways our language model might fail to use its knowledge. in this paper we address our failure mode by relying on commonly understood relationships between the skills of bloom's taxonomy which we term proximal context. in order to understand whether the cranberry grape mixture is poisonous the model needs to remember whether grape juice is poisonous. in order to apply its knowledge to figure out what will happen next it needs to understand whether the cranberry grape mixture is poisonous or not. in general, the proximal context for a particular task t at level l is given by those tasks implicitly required by t , which are mostly at level l -1 of the taxonomy. we guide our language to answer questions more accurately by providing it not just any context, but proximal context foot_0 . in performing zero-shot question answering our language model asks itself additional clarification questions, choosing those most likely to result in proximal context. our contributions in this paper are: bloom's taxonomy. the original work @xcite ) defined taxonomies for learning in the cognitive (intellectual), affective (interests, attitudes, values), and psychomotor domains, though the cognitive domain is what we usually refer to today. almost half a century later the cognitive domain taxonomy was revised @xcite to reflect more active thinking and improve usability by adding verbs to describe levels. teachers use this taxonomy, for example in computer science education @xcite @xcite , and our inspiration is from this revised version of the cognitive taxonomy. machine learning has been applied to automatically classify questions @xcite @xcite into bloom's taxonomy levels, but the taxonomy has not been applied to analyze or improve machine learning models themselves. we use it to help our model think about what it knows. ## approach our approach builds on the zero-shot question answering approach of @xcite to answer questions (section 3.1) by adding clarifications with self-talk (section 3.2). we describe this approach then we use bloom's taxonomy to select better clarifications (section 3.2). ## conclusion large pre-trained language models sometimes have the right information, but they just do not know how to use it. we used bloom's taxonomy to pick questions with the right amount of proximal context. this helped the language models use their knowledge to more effectively answer questions. in the future we would like to extend our work on tasks that present a wide range of questions that fall under different levels of the taxonomy. similarly, we also would like to study and improve upon the current limited set of prefix questions used.