ISAAQ - Mastering Textbook Questions with Pre-trained Transformers and Bottom-Up and Top-Down Attention (2020)

Abstract: Textbook Question Answering is a complex task in the intersection of Machine Comprehension and Visual Question Answering that requires reasoning with multimodal information from text and diagrams. For the first time, this paper taps on the potential of transformer language models and bottom-up and top-down attention to tackle the language and visual understanding challenges this task entails. Rather than training a language-visual transformer from scratch we rely on pre-trained transformers, fine-tuning and ensembling. We add bottom-up and top-down attention to identify regions of interest corresponding to diagram constituents and their relationships, improving the selection of relevant visual information for each question and answer options. Our system ISAAQ reports unprecedented success in all TQA question types, with accuracies of 81.36%, 71.11% and 55.12% on true/false, text-only and diagram multiple choice questions. ISAAQ also demonstrates its broad applicability, obtaining state-of-the-art results in other demanding datasets.

## introduction within nlp, machine understanding of textbooks is one of the grand ai challenges. as originally put by @xcite : "reading a chapter in a college freshman text (say physics or accounting) and answering the questions at the end of the chapter is a hard (ai) problem that requires advances in vision, language, problem-solving, and learning theory.". towards such goals, the textbook question answering (tqa) dataset presented in @xcite offers an excellent testbed. drawn from middle school science curricula, it describes fairly complex phenomena through a combination of text and diagrams. answering questions may therefore involve text, diagrams or both, and require information from multiple sentences and/or diagrams in long textbook lessons. another characteristic of the tqa dataset that makes it rather unique and challenging is that questions often involve reasoning beyond methods based on co-occurrence analysis or simple look-up. tqa requires parsing information from different sentences, dealing with qualitative and quantitative information (high frequency vs. 20,000hz), and relating text or visual information with the question. solving the tqa task also requires dealing with language about negation, conjunction, polarity or commonsense. on the visual side, tqa is rich with diagrams that describe potentially complex concepts, such as photosynthesis, the trophic chain, and the cycle of water, which are hard to represent as a single natural image. quite on the contrary, diagrams contain simpler constituents and relationships between them, whose semantics needs to be captured in order to answer the questions. despite recent work, overall progress in the tqa dataset has been rather limited until now, suggesting that language and diagram understanding challenges like the ones above-mentioned are still far from solved. in this paper we address such limitations by building on the success of two recent developments in natural language processing and visionand-language reasoning: large-scale, pre-trained language models and bottom-up and top-down (butd) attention. we demonstrate that, compared to previous approaches, transformer-based language models like bert and roberta can significantly contribute to increase the language understanding and reasoning capabilities required to answer tqa questions. we also show that butd attention, originally proposed for tasks like image captioning and visual question answering with natural images, can be effectively adapted to propose regions of interest in the diagram that are relevant for the question in hand, enabling the identification of diagram constituents and their relationships. the main contributions of this paper are the following: the remainder of the paper is structured as follows. section 2 describes related work. section 3 introduces the notation that will be used throughout the paper. we present our method in section 4, including an overview of the overall model, its main building blocks (background information retrievers, solvers, ensemble), and their interplay. we focus on each tqa question type, i.e. true/false and text and diagram multiple-choice (mc) questions, as sub-tasks of the main tqa task and propose specific solvers for each case, based on pre-trained transformers, fine-tuning and butd attention. in section 5, we present our experimental results, including an ablation study focused on understanding the specific contribution of each solver and their components. we also analyze the robustness of our model, its ability to generalize to other datasets, and its reasoning abilities. finally, section 6 illustrates the impact of the different techniques used in isaaq to address diagram mc questions in tqa. ## related work in @xcite several tqa baselines were proposed that were based on machine comprehension (mc) models like bidaf @xcite and memorynet @xcite , as well as visual question answering (vqa) @xcite and diagram parsing algorithms like dsdp-net @xcite . their results were rather modest (50.4, 32.9, and 31.3 in true/false, text and diagram mc questions), suggesting that existing mc/vqa methods would not suffice for the tqa dataset. indeed, diagram questions entail greater complexity than dealing with natural images, as shown in (gomez-perez and ortega, 2019), where we beat the tqa baselines using visual and language information extracted from the correspondence between figures and captions in scientific literature enriched with lexicosemantic information from a knowledge graph @xcite . by contrast, @xcite focused on finding contradictions between the candidate answers and their corresponding context while @xcite applied graph convolutional networks on text and diagrams to represent relevant question background information as a unified graph. the field of nlp has advanced substantially with the advent of large-scale language models such as elmo @xcite , ulmfit @xcite , gpt @xcite , bert @xcite , and roberta @xcite . using large amounts of text, e.g. bert was trained on wikipedia plus the google book corpus of 10,000 books @xcite , they are trained to learn various language prediction tasks such as guessing a missing word or the next sentence. language models and particularly transformers have been used in question answering, as illustrated by the success of the aristo system @xcite in standard science tests. transformers have also proved their worth as soft reasoners @xcite , exhibiting capabilities for natural language inference. furthermore, whilst learning linguistic information, transformers have shown to capture semantic knowledge and general understanding of the world from the training text @xcite , including a notion of commonsense that can be useful in question answering. our approach is the first to leverage the language understanding and reasoning capabilities of existing transformer language models for tqa. focused on natural images, some vision-andlanguage reasoning systems are also adopting transformer architectures at their backbone. vl-bert @xcite and lxmert @xcite pre-train large-scale transformers that capture both visual concepts and language semantics, as well as cross-modal information. pretraining is done via several tasks, like masked language modeling, masked object prediction, crossmodality matching, and image question answering, on large-scale text and visual datasets, like @xcite , google book corpus, ms coco @xcite or visual genome @xcite , requiring considerable compute. by contrast, our approach is much more frugal. we fine-tune pre-trained existing transformers like bert and roberta and count on a much more limited variety of datasets focused on diagrams, like ai2d @xcite . like lxmert, we extend region-of-interest (roi) features with object positional embeddings. finally, we apply bottom-up and top-down attention @xcite to focus on the most relevant diagram regions for each question. ## proposed method figure 1 shows our two-stage process to answer text and diagram mc questions. first, for each question we propose different retrievers to extract relevant language and visual background knowledge from the textbook. note that we consider both approaches based on conventional information retrieval techniques and approaches that leverage transformers pre-trained on specific tasks. during training, the retrieved background is provided along with the question and candidate answers to our solvers. also during execution, providing the potential to "read" such knowledge and apply it to the question. we ensemble different solvers resulting from fine-tuning one or several transformers on a multiple choice classification task, which can be combined with others based e.g. on information retrieval. for text and diagram mc questions, each transformer-based solver results from training the mcc task on the text passages produced by one of the text retrieval methods. since each text retriever produces a different but complementary dataset of background text passages, the resulting solvers also complement each other, motivating their combination as an ensemble. in addition, for the visual part we apply butd attention as shown in figure 2 . for true/false questions we follow an analogous two-stage process, in this case fine-tuning our transformers on a text entailment task. ## conclusion this paper reports on isaaq, the first system to achieve accuracies above 80%, 70% and 55% on tqa true/false, text and diagram mc questions. isaaq demonstrates that it is possible to master the grand ai challenge of machine textbook understanding based on modern methods for language and visual understanding, with modest infrastructure requirements. key to this success are transformers, butd attention, pre-training on related datasets, and the selection of complementary background information to train and ensemble different solvers. our approach allowed overcoming critical challenges like the complexity and relatively small size of the tqa dataset or the scarcity of large diagram datasets. still, further research is necessary to keep pushing the boundaries of textbook understanding, e.g. by charting and expanding the reasoning skills of transformers, making model outcomes more interpretable by humans, and further exploiting diagrams. additional effort will also be needed in activities like the development of large diagram datasets, including the semantic annotation of diagram constituents and connectors, and annotating diagram questions with the reasoning and knowledge types required to answer them.