Perhaps PTLM s Should Go to School – A Task to Assess Open Book and Closed Book QA (2021)

Abstract: Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college textbook or a manual. PTLMs have shown great success in many question-answering tasks, given significant supervised training, but much less so in zero-shot settings. We propose a new task that includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History), hundreds of true/false statements based on review questions written by the textbook authors, validation/development tests based on the first eight chapters of the textbooks, blind tests based on the remaining textbook chapters, and baseline results given state-of-the-art PTLMs. Since the questions are balanced, random performance should be ~50%. T5, fine-tuned with BoolQ achieves the same performance, suggesting that the textbook’s content is not pre-represented in the PTLM. Taking the exam closed book, but having read the textbook (i.e., adding the textbook to T5’s pre-training), yields at best minor improvement (56%), suggesting that the PTLM may not have “understood” the textbook (or perhaps misunderstood the questions). Performance is better (~60%) when the exam is taken open-book (i.e., allowing the machine to automatically retrieve a paragraph and use it to answer the question).

## introduction question answering (qa) is a yardstick for measuring machine understanding performance @xcite . qa's popularity as an evaluation technique has led to several sub-categories: tasks can require a model to answer questions from either its background knowledge or from a short passage (e.g., squad, @xcite or with information retrieval to allow the model to search for the answer in a large corpus (e.g., arc, @xcite . answering can take the form of true/false classification @xcite , multiplechoice, span selection (squad, @xcite , or text generation @xcite . transformer architectures optimized for specific qa formulations have driven recent progress in question answering. for example, some models target ir-oriented qa @xcite while others optimize their learning strategy to specific question types (e.g., by optimizing for expected answers to factoid questions, @xcite . while specialization improves performance, it limits generalization. unifiedqa @xcite takes a step forward by generalizing the architecture and training over multiple data sets with different qa formulations. most research assumes that the information necessary to answer questions is either included with the query (e.g., boolq, squad 1.1) or that the information was already stored in language models during initial pre-training or a task-specific second pre-training. 1 however, this assumption limits language models relying on massive corpora @xcite to learning oftrepeated facts @xcite . valuable, domain-specific information seldom is repeated often enough to be captured by language models. an evaluation of domain-specific knowledge without access to a relevant text is even more challenging as simple strategies like identifying the answer by information retrieval are ineffective. even reasoning tasks such as arc @xcite only target general scientific knowledge and offer large text corpora to aid qa systems. we propose learning from textbooks (left), a new task to classify domain-specific statements drawn from a textbook's review questions as true or false using three evaluation configurations. the first configuration tests the ability to answer questions without any domain-specific material (e.g., applying a ptlm with no access to domain-specific knowledge). this setting is equivalent to a person taking the test before taking the class. in the second configuration, a model has access to the textbook's content and may encode the information in the textbook but may not access the textbook during the test; we call this closed book. the second configuration tests a model's ability to learn by reading. in the third configuration, which we call open book, models can access the textbook during the test. thus, left supports contrasting qa formulations and reading methods to explore the strengths and weaknesses of various qa approaches. the left data and leaderboard are available at https:// leftleaderboard.isi.edu . ## task description learning from textbooks (left) contains two machine-readable college-level introductory textbooks and a set of true/false statements manually derived from review questions written by the textbook authors. the task requires that systems based on language models classify the statements before and after reading the given textbook material to separate what was learned from the book from what was known before reading. "reading" is any algorithm method that learns from the domain text without storing a copy of the text. to support comparisons with existing qa approaches, left also supports the open-book setting, where a system can use a textbook paragraph when answering. our goal is to support testing pre-trained language models, e.g., t5 @xcite , and also those approaches that extract and store triples during reading (e.g., <u.s. declaration of independence; signed; aug 2, 1776>). while learning corpora appear in other question answering tasks (e.g., arc, 14m words, @xcite , the text included in left is small and corresponds to the textbook chapters relevant to each question set. the largest text in left contains only 300k words (for details, see left includes two openly licensed 2 collegelevel introductory textbooks, american government 2e @xcite and u.s. history @xcite , and true/false statements derived from each book's review questions. we manually rewrote each textbook's multiple-choice review questions into a balanced set of true and false statements. 3,4 we intentionally wrote the statements such that each true and false pair has high word overlap to deter classification strategies that rely on word overlap with the textbook. we include five sample statements from left in appendix a and discuss statement correctness in appendix c. we measure task performance by accuracy. since the two textbooks are used in teaching college students, we do not release the correct labels (see the ethical considerations section). we split each textbook into a dev set consisting of the first eight chapters and a test set consisting of the remaining chapters (see left has three evaluation configurations: (1) prior-knowledge; (2) closed-book, after reading; and (3) open-book. prior-knowledge tests the ability to answer questions without any domainspecific material. language models must rely solely on the knowledge learned from their large pre-training corpora. in the second configuration, closed-book, after reading, models may access the textbook's content and may encode the information in the textbook but may not access the textbook during the test. for each set, models may read the set's corresponding textbook chapters, the entire textbook, or both textbooks. we require that all model submissions to this evaluation configuration also submit to prior-knowledge. predictions before reading (prior-knowledge) quantify 2 both textbooks are licensed under the creative commons attribution license v4.0 license. 3 we construct one true and one false statement for each question to obtain a balanced data set. for example, the question when was the u.s. declaration of independence signed? (a)(correct) august 2, 1776 (b) december 2, 1776, (c) august 2, 1746, (d) august 22, 1976 could become the u.s. declaration of independence was signed on august 2, 1776 (true) and the u.s. declaration of independence was signed on august 2, 1746 (false). 4 for u.s. history's dev set, we also process questions written by a community of instructors. the information included in each model through initial pre-training. the change in performance from prior-knowledge to closed-book, after reading illustrates each model's reading effectiveness. in the third configuration, open-book, models can access the textbook or relevant chapter during the test. to support research on open-book question answering, with each statement, we include the textbook paragraph that provides the information necessary to classify the statement. in our experiments, we call this goldir. thus, left supports contrasting qa formulations and reading methods to explore the strengths and weaknesses of various qa approaches. ## results we illustrate baseline performance on left using two state-of-the-art language models: t5 @xcite and gpt-neo (a gpt-3 architecture, brown et al., 2020, trained on the open pile corpus, @xcite . we fine-tune the two language models using boolq @xcite . ## conclusions & future work there are several natural directions in which we can extend and improve left. we are extending u.s. history's test set as we did with the dev set by including statements based on questions written by a community of instructors. we are also collecting relevant paragraphs for the extra statements. lastly, we are categorizing the kind of knowledge required to classify each statement to better understand what kinds of knowledge pose the most difficulties. we draw several conclusions from this work. foremost, learning from textbooks (left) represents a new type of challenge task for ptlms, contrasted with the much-studied challenges of (1) common sense qa based on prior knowledge, (2) reading comprehension given a paragraph, and (3) qa using large domain-specific corpora, e.g., science at the elementary-or middle-school level. the task is intended to stimulate research on the following dimensions: our baseline studies show that t5 and gpt-neo thus far are challenged to show improvement after reading the relevant textbook, that open-book evaluation is easier than closed-book (as it is for humans), and that the gating factor in left is understanding the textbook and/or the question rather than paragraph retrieval. the baseline results show there is much room for improvement.