Original: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Revised: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Original: Anonymous submission ID:

Original: Abstract
Revised: Abstract

Original: When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT).
Revised: When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT).

Original: Thisprocess is normally a black box in the case of deep learning models like large-scalelanguage models.
Revised: This process is normally a black box in the case of deep learning models like large-scale language models.

Original: Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system.
Revised: Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system.

Original: However,existing datasets fail to provide annotations for the answers, or are restricted tothe textual-only modality, small scales, and limited domain diversity.
Revised: However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity.

Original: To this end,we present Science Question Answering ( SQA ), a new benchmark that consists of⇠ 21k multimodal multiple choice questions witha diverseset of science topics and annotations of their answers with corresponding lectures and explanations.
Revised: To this end, we present Science Question Answering (S CIENCE QA), a new benchmark that consists of ∼ 21k multimodal multiple choice questions with diverse science topics and annotations of their answers with corresponding lectures and explanations.

Original: We further design language models to learn to generate lectures and explanations as thechain of thought (CoT) to mimic the multi-hop reasoning process when answering
Revised: We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering

Original: SQA questions.
Revised: S CIENCE QA questions.

Original: SQA demonstrates the utility of CoT in language models, as CoTimproves the question answering performance by 1.20% in few-shot GPT-3 and3.99% in ﬁne-tuned UniﬁedQA.
Revised: S CIENCE QA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in fewshot GPT-3 and 3.99% in ﬁne-tuned UniﬁedQA.

Original: We also explore the upper bound for models toleverage explanations by feeding those in the input; we observe that it improvesthe few-shot performance of GPT-3 by 18.96%.
Revised: We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%.

Original: Our analysis further shows thatlanguage models, similar to humans, beneﬁt from explanations to learn from fewerdata and achieve the same performance with just 40% of the data.
Revised: Our analysis further shows that language models, similar to humans, beneﬁt from explanations to learn from fewer data and achieve the same performance with just 40% of the data.

Original: (cid:52)(cid:88)(cid:72)(cid:86)(cid:87)(cid:76)(cid:82)(cid:81) (cid:29)(cid:3)(cid:58)(cid:75)(cid:76)(cid:70)(cid:75)(cid:3)(cid:87)(cid:92)(cid:83)(cid:72)(cid:3)(cid:82)(cid:73)(cid:3)(cid:73)(cid:82)(cid:85)(cid:70)(cid:72)(cid:3)(cid:73)(cid:85)(cid:82)(cid:80)(cid:3)(cid:87)(cid:75)(cid:72)(cid:3)

Original: [Table]

Original: dataset where a data example consists of multimodal question answering information and the grounded lecture and explanation.

Original: We study if models can generate a reasonable explanation to reveal the chain-of-thought reasoning when answering an SQA question

Original:  Introduction
Revised:  Introduction

Original: A long-standing goal of AI systems is to act reliably and learn complex tasks efﬁciently like humanbeings.
Revised: A long-standing goal of AI systems is to act reliably and learn complex tasks efﬁciently like human beings.

Original: In the process of reliable decision making, humans follow an explicit chain-of-thought (CoT)reasoning process that is typically expressed as an explanation.
Revised: In the process of reliable decision making, humans follow an explicit chain-of-thought (CoT) reasoning process that is typically expressed as an explanation.

Original: However, machine learning modelsare trained mostly using a large number of input-output examples to perform a speciﬁc task.
Revised: However, machine learning models are trained mostly using a large number of input-output examples to perform a speciﬁc task.

Original: Theseblack-box models only generate the ﬁnal decision without reliably revealing the underlying reasoning process.
Revised: These black-box models only generate the ﬁnal decision without reliably revealing the underlying reasoning process.

Original: Not surprisingly, it is unclear if they understand the task and can generalize even thoughthey perform well on the benchmark.
Revised: Not surprisingly, it is unclear if they understand the task and can generalize even though they perform well on the benchmark.

Original: On the other hand, humans are able to learn from instructionsor explanations from past experience and generalize them to novel and unseen problems.
Revised: On the other hand, humans are able to learn from instructions or explanations from past experience and generalize them to novel and unseen problems.

Original: This helpsthem learn more quickly with fewer data.
Revised: This helps them learn more quickly with fewer data.

Original: In this work, we explore if machines can be endowed withsuch reasoning abilities in the context of science-based question answering.
Revised: In this work, we explore if machines can be endowed with such reasoning abilities in the context of science-based question answering.

Original: Recently, science problem solving benchmarks [17] have been used to diagnose the multi-hopreasoning ability and interpretability of AI systems.
Revised: Recently, science problem solving benchmarks [18] have been used to diagnose the multi-hop reasoning ability and interpretability of AI systems.

Original: To answer science questions, a model needs tonot only understand multimodal contents but also extract external knowledge to arrive at the correctanswer.
Revised: To answer science questions, a model needs to [Table] (cid:37)(cid:40)(cid:38)(cid:36)(cid:56)(cid:54)(cid:40)(cid:29) not only understand multimodal contents but also extract external knowledge to arrive at the correct answer.

Original: Since these tasks require domain-speciﬁc knowledge and explicit multi-hop reasoning, amodel would be not interpretable if it fails to provide explanations to reveal the reasoning process.
Revised: Since these tasks require domain-speciﬁc knowledge and explicit multi-hop reasoning, a model would be not interpretable if it fails to provide explanations to reveal the reasoning process.

Original: However, current science question datasets [17, 16, 46] mostly lack annotated explanations for theanswers.
Revised: However, current science question datasets [18, 17, 52] mostly lack annotated explanations for the answers.

Original: To address this issue, other science datasets annotate the explanations, but they are restrictedto the textual only modality and limited to small data scales [12, 7, 33] or a small set of topics [19, 13].
Revised: To address this issue, other science datasets annotate the explanations, but they are restricted to the textual only modality and limited to small data scales [13, 7, 37] or a small set of topics [20, 14].

Original: Therefore, we collect Science Question Answering ( SQA ), a large-scale multi-choice dataset thatcontains multimodal science questions with explanations and features rich domain diversity.
Revised: Therefore, we collect Science Question Answering (S CIENCE QA), a large-scale multi-choice dataset that contains multimodal science questions with explanations and features rich domain diversity.

Original: SQA is collected from elementary and high school science curricula, and contains 21,208 examplesalong with lectures and explanations.
Revised: S CIENCE QA is collected from elementary and high school science curricula, and contains 21, examples along with lectures and explanations.

Original: Different from existing datasets [16, 17, 46], SQA has richerdomain diversity from three different subjects: natural science, social science, and language science.
Revised: Different from existing datasets [17, 18, 52], S CIENCE QA has richer domain diversity from three different subjects: natural science, social science, and language science.

Original: A typicalSQA example consists of a question, multiple choices, visual and textual contexts, a correctanswer, as well as a lecture and an explanation.
Revised: A typical example consists of a question, multiple choices, multimodal contexts, a correct answer, as well as a lecture and an explanation.

Original: The lecture and explanation provide general externalknowledge and speciﬁc reasons, respectively, for arriving at the correct answer.
Revised: The lecture and explanation provide general external knowledge and speciﬁc reasons, respectively, for arriving at the correct answer.

Original: Consider the thoughts one person might have when answering the question in Figure 1.
Revised: Consider the thoughts one person might have when answering the question in Figure 1.

Original: One ﬁrstrecalls the knowledge regarding the deﬁnition of a force learned from textbooks: “ A force is a push ora pull that ...
Revised: One ﬁrst recalls the knowledge regarding the deﬁnition of a force learned from textbooks: “ A force is a push or a pull that ...

Original: The direction of a push is ...
Revised: The direction of a push is ...

Original: The direction of a pull is ... ”, then forms a line of reasoning:“ The baby’s hand applies a force to the cabinet door .!
Revised: The direction of a pull is ... ”, then forms a line of reasoning: “ The baby’s hand applies a force to the cabinet door .

Original:  This force causes the door to open .!
Revised: → This force causes the door to open .

Original: Thedirection of this force is toward the baby’s hand . ”, and ﬁnally arrives at the correct answer: “ Thisforce is a pull . ”.
Revised: → The direction of this force is toward the baby’s hand . ”, and ﬁnally arrives at the correct answer: “ This force is a pull . ”.

Original: Following [36], we formulate theSQA task to output a natural explanation alongsidethe predicted answer.
Revised: Following [41], we formulate the task to output a natural explanation alongside the predicted answer.

Original: In this paper, we train language models to generate lectures and explanations asthe chain of thought (CoT) to mimic the multi-hop reasoning process to answer SQA questions.
Revised: In this paper, we train language models to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process to answer S CIENCE QA questions.

Original: Our experiments show that current multimodal methods [49, 1, 20, 9, 24, 31] fail to achieve satisfactory performance on SQA and do not generate correct explanations.
Revised: Our experiments show that current multimodal methods [55, 1, 21, 9, 26, 35] fail to achieve satisfactory performance on S CIENCE QA and do not generate correct explanations.

Original: However, we ﬁnd that CoT canhelp large language models not only in the few-shot learning setting but also in the ﬁne-tuning setting.
Revised: Instead, we ﬁnd that CoT can help large language models not only in the few-shot learning setting but also in the ﬁne-tuning setting.

Original: When combined with CoT to generate the lecture and explanation, the ﬁne-tuned UniﬁedQA [18]achieves an improvement of 3.99% as opposed to not using CoT in the ﬁne-tuning stage.
Revised: When combined with CoT to generate the lecture and explanation, the ﬁne-tuned UniﬁedQA [19] achieves an improvement of 3.99% as opposed to not using CoT in the ﬁne-tuning stage.

Original: The few-shot GPT-3 model [5] via chain-of-thought prompting can obtain 75.17% on SQA with an improvementof 1.20% compared to the few-shot GPT-3 without CoT. Prompted with CoT, GPT-3 can generatereasonable explanations as evaluated by automated metrics, and promisingly, 65.2% of explanationsmeet the gold standard of human evaluations.
Revised: The few-shot GPT-3 model [5] via chain-of-thought prompting can obtain 75.17% on S CIENCE QA with an improvement of 1.20% compared to the few-shot GPT-3 without CoT. Prompted with CoT, GPTcan generate reasonable explanations as evaluated by automated metrics, and promisingly, 65.2% of explanations meet the gold standard of human evaluations.

Original: We also investigate the upper bound for models toharness explanations by including them in the input.
Revised: We also investigate the upper bound for models to harness explanations by including them in the input.

Original: We ﬁnd that doing so improves GPT-3’s few-shotperformance by 18.96%, suggesting that explanations do aid models and are currently underutilizedin the CoT framework.
Revised: We ﬁnd that doing so improves GPT-3’s few-shot performance by 18.96%, suggesting that explanations do aid models and are currently underutilized in the CoT framework.

Original: Further analysis shows that, like humans, language models beneﬁt fromexplanations to learn with less data: UniﬁedQA with CoT obtains the same results as UniﬁedQAwithout CoT with only 40% of the training data.
Revised: Further analysis shows that, like humans, language models beneﬁt from explanations to learn with less data: UniﬁedQA with CoT obtains the same results as UniﬁedQA without CoT with only 40% of the training data.

Original: To sum up, our contributions are three-fold: (a) To bridge the gap in existing datasets in the scientiﬁcdomain, we build Science Question Answering ( SQA ), a new dataset containing 21,208 multimodalscience questions with rich domain diversity.
Revised: To sum up, our contributions are three-fold: (a) To bridge the gap in existing datasets in the scientiﬁc domain, we build Science Question Answering (S CIENCE QA), a new dataset containing 21, multimodal science questions with rich domain diversity.

Original: To the best of our knowledge, SQA is the ﬁrst largescale multimodal dataset that annotates lectures and explanations for the answers.
Revised: To the best of our knowledge, S CIENCE QA is the ﬁrst large-scale multimodal dataset that annotates lectures and explanations for the answers.

Original: (b) We show that CoT beneﬁts large language models in both few-shot and ﬁnetuning learning by improving modelperformance and reliability via generating explanations (c)
Revised: (b) We show that CoT beneﬁts large language models in both few-shot and ﬁne-tuning learning by improving model performance and reliability via generating explanations.

Original:  We further explore the upper bound of GPT-3 and show that CoT helps language models learn from fewer data.
Revised: (c) We further explore the upper bound of GPT-3 and show that CoT helps language models learn from fewer data.

Original:  Related Work
Revised:  Related Work

Original: Visual question answering.
Revised: Visual question answering.

Original: Since the task of visual question answering (VQA) was ﬁrst proposedin [2], there have been plenty of VQA datasets [50, 52, 22, 10, 14, 11] conducted to facilitate theresearch work.
Revised: Since the task of visual question answering (VQA) was ﬁrst proposed in [2], there have been plenty of VQA datasets [56, 58, 23, 11, 15, 12] conducted to facilitate the research work.

Original: Although our SQA dataset shares some features with VQA, there are several maindifferences between them.
Revised: Although our S CIENCE QA dataset shares some features with VQA, there are several main differences between them.

Original: First, SQA is more challenging than existing VQA datasets because it contains multimodal contexts and diverse topics in the scientiﬁc domain.
Revised: First, S CIENCE QA is more challenging than existing VQA datasets because it contains multimodal contexts and diverse topics in the scientiﬁc domain.

Original: In addition, most answersare annotated with lectures and explanations, which makes SQA a suitable dataset for multi-modalquestion answering and multi-hop reasoning for AI systems.
Revised: In addition, most answers are annotated with lectures and explanations, which makes S CIENCE QA a suitable dataset for multi-modal question answering and multi-hop reasoning for AI systems.

Original: Inspired by the recent remarkableperformance achieved for VQA [9, 24, 8], in this paper, we further extensively benchmark SQA witha wide range of attention-based [1, 30, 20, 9] and Transformer-based [28, 24, 25, 8] methods.
Revised: Inspired by the recent remarkable performance achieved for VQA [33, 32, 10, 9, 26, 8], in this paper, we further extensively benchmark S CIENCE QA with a wide range of attention-based [1, 33, 21, 9] and Transformer-based [30, 26, 27, 8] methods.

Original: Datasets for science problems.
Revised: Datasets for science problems.

Original: Science problem solving is a challenging task that requires an AIsystem not only to understand the multimodal information from the science curriculum but also toreason about how to answer the domain-speciﬁc questions.
Revised: Science problem solving is a challenging task that requires an AI system not only to understand the multimodal information from the science curriculum but also to reason about how to answer the domain-speciﬁc questions.

Original: Current science problem datasets suchas AI2D [16], DVQA [15], VLQA [46], and FOODWEDS [23] have contributed to multimodalreasoning in the scientiﬁc domain. 
Revised: Current science problem datasets such as AI2D [17], DVQA [16], VLQA [52], and FOODWEDS [24] have contributed to multimodal reasoning in the scientiﬁc domain. For example, a portion of VLQA contains multimodal questions on science subjects.

Original: These datasets, however, lack annotated explanations for theanswers to reveal the reasoning steps.
Revised: These datasets, however, lack annotated explanations for the answers to reveal the reasoning steps.

Original: Some other datasets annotate the answers in the formsof supporting facts [33, 19], entailment trees [7], explanation graphs [12], reasoning chains [13].
Revised: Some other datasets annotate the answers in the forms of supporting facts [37, 20], entailment trees [7], explanation graphs [13], reasoning chains [14].

Original: However, these datasets are restricted to the single text modality with small data scales and limitedtopics.
Revised: However, these datasets are restricted to the single text modality with small data scales and limited topics.

Original: Instead, our SQA annotates the answers with grounded lectures and explanations.
Revised: Instead, our S CIENCE QA annotates the answers with grounded lectures and explanations.

Original: Besides, SQA features a richer domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills.
Revised: Besides, S CIENCE QA features a richer domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills.

Original: Learning from explanations and few-shot Learning.
Revised: Learning from explanations and few-shot learning.

Original: Explanations help humans understand a taskbetter, and there have been several attempts to show the same for models.
Revised: Explanations help humans understand a task better, and there have been several attempts to show the same for models.

Original: For examples, the learningfrom instruction paradigm [35, 38, 47, 34] where the task level explanation is provided in the form ofinstruction improves model performance signiﬁcantly.
Revised: For example, the learning from instruction paradigm [40, 43, 53, 39, 45, 25], where the task level explanation is provided in the form of instruction, improves model performance signiﬁcantly.

Original: An example of learning from explanationsin the scientiﬁc domain is proposed in [45] where the model interprets demonstrative solutions tosolve geometry problems.
Revised: An example of learning from explanations in the scientiﬁc domain is proposed in [51] where the model interprets demonstrative solutions to solve geometry problems.

Original: Recently, there has been a surge of interest in few-shot learning, wherelanguage models learn a speciﬁc task from a few examples [40, 3].
Revised: Recently, there has been a surge of interest in few-shot learning, where language models learn a speciﬁc task from a few examples [46, 3].

Original: For instance, [37, 48] ﬁnd thatexplanations in the format of the chain of thought can improve the reasoning abilityof languagemodels in few-shot learning.
Revised: For instance, [42, 54, 34] ﬁnd that explanations in the format of the chain of thought can improve language models’ reasoning ability in few-shot learning.

Original: In this paper, we show that the chain of thought boosts the performanceof large language models like UniﬁedQA [18] if the models generate explanations along with theanswer in a ﬁne-tuned way.
Revised: In this paper, we show that the chain of thought boosts the performance of large language models like UniﬁedQA [19] if the models generate explanations along with the answer in a ﬁne-tuning way.

Original: Furthermore, a few-shot GPT-3 model via chain-of-thought prompting isable to improve the reasoning performance on SQA and generate reasonable explanations.
Revised: Furthermore, a few-shot GPT-3 model via chain-of-thought prompting is able to improve the reasoning performance on S CIENCE QA and generate reasonable explanations.

Original:  Dataset
Revised:  Dataset

Original: We collect SQA , which is a multimodal multiple-choice science question dataset containing 21,208examples.
Revised: We collect S CIENCE QA, which is a multimodal multiple-choice science question dataset containing 21,208 examples.

Original: An example in SQA is shown in Figure 1.
Revised: An example in S CIENCE QA is shown in Figure 1.

Original: Given the science question and multimodalcontexts, the task is to select the correct answer from multiple options.
Revised: Given the science question and multimodal contexts, the task is to select the correct answer from multiple options.

Original: Different from existingdatasets [44, 16, 46, 29, 23], SQA covers diverse topics across three subjects: natural science, socialscience, and language science.
Revised: Different from existing datasets [50, 17, 52, 31, 24], S CIENCE QA covers diverse topics across three subjects: natural science, social science, and language science.

Original: Moreover, most questions are annotated with grounded lecturesand detailed explanations.
Revised: Moreover, most questions are annotated with grounded lectures and detailed explanations.

Original: The lecture provides general knowledge that introduces the backgroundinformation for solving problems of a similar class.
Revised: The lecture provides general knowledge that introduces the background information for solving problems of a similar class.

Original: The explanation reveals a speciﬁc reason forthe answer.
Revised: The explanation reveals a speciﬁc reason for the answer.

Original: To effectively answer the questions, a model often needs to be able to understand themultimodal content in the input and extract external knowledge, similar to how humans do.
Revised: To effectively answer the questions, a model often needs to be able to understand the multimodal content in the input and extract external knowledge, similar to how humans do.

Original: Moreimportantly, the goal of SQA is to aid development of a reliable model that is capable of generatinga coherent chain of thought when arriving at the correct answer to reveal the multi-step reasoningprocess.
Revised: More importantly, the goal of S CIENCE QA is to aid development of a reliable model that is capable of generating a coherent chain of thought when arriving at the correct answer to reveal the multi-step reasoning process.

Original: For data collection details, see Appendix A.1.
Revised: For data collection details, see Appendix A.1.

Original: [Table]
Revised: [Table]

Original: Statistic
Revised: f o ll o w i n g s t a t e s t h e t h e s e a t h e s e d o s p ee c h o f

Revised: y o u w o u l d h

Original: [Table]
Revised: i g h li g h t e d i s i s f a ll a c y

Original: 3.1 Comparisons with Existing Datasets
Revised: a h a s t h e d o e s t h e u s e s h i g h li g h t

Revised: e d i s t

Revised: h e s h

Revised: o w s h a s o f

Revised: o f

Original: Table 1 shows a comparison of SQA and other science problem datasets.
Revised: i s p r o p e r t y f i g u r e w o r d c o un t r y l

Revised: o g i c a l s o l u t i o n a n i m a l '

Revised: s s e n t e n c e t e n s e t e x t c

Original: As shown in the table,
Revised: o n t i n e n t c o rr e c t l y o b j e c t t y p e s t a t e m e n t c l o s i n g c a p i t a l n a m e s o u r c e t

Revised: h e t h e s u pp o r t s t w o t

Original: SQA is much larger than most other datasets.
Revised: h e s e t h e s e n t e n c e o f i s i n f o r m a t

Original: SQA also has the largest set of images, spans acrossall 12 grades, contains the longest questions, and has the most diverse input sources.
Revised: i o n d o d o e s c a n k i n d i n o r g a n i s m o n e t h e t r a i t s t a t e m e n t f o ll o w i n g t h e t h e m a g n e t i c t h e a b o u t k i n e t i c

Revised: o

Original: As opposed tolimiting the subject to only natural science, SQA also includes social science and language science,largely adding to the domain diversity of the dataset.
Revised: f a v e r a g e m o t i o n t h e a i s l o n g d e s c r i

Original: Furthermore, most of the questions in SQA areannotated with textual lectures (83.9%) and explanations (90.5%), which reveal the reasoning path tothe correct answer.
Revised: b e p a ss a g e t h i s t h e f i n d y o u W h

Original: To the best of our knowledge, SQA is the ﬁrst large-scale multimodal sciencequestion dataset that annotates the answers with detailed lectures and explanations.
Revised: i c h W h a t S e l e c t I s C o m

Revised: p l e t e T h

Revised: i n k C

Revised: o m p a r e H

Revised: o w D o e s

Original: 3.2 Data Analysis
Revised: W o u l d 3.1 Data Analysis

Original: Key statistics.
Revised: Key statistics.

Original: We randomly split the dataset into training, validation, and test splits with a ratio of60:20:20.
Revised: We randomly split the dataset into training, validation, and test splits with a ratio of 60:20:20.

Original: Each split has 12,726, 4,241, and 4,241 examples, respectively.
Revised: Each split has 12,726, 4,241, and 4,241 examples, respectively.

Original: Table 2 shows the mainstatistics of SQA .
Revised: Table 1 shows the main statistics of S CIENCE QA.

Original: SQA has a large set of different questions, totaling up to 9,122.
Revised: S CIENCE QA has a large set of different questions, totaling up to 9,122.

Original: Out of the 21,208questions in SQA , 10,332 (48.7%) have an image context, 10,220 (48.2%) have a text context, and6,532 (30.8%) have both.
Revised: Out of the 21,208 questions in S CIENCE QA, 10,332 (48.7%) have an image context, 10,220 (48.2%) have a text context, and 6,532 (30.8%) have both.

Original: 83.9% of the questions are annotated with a lecture, while 91.3% of thequestions feature an explanation.
Revised: 83.9% of the questions are annotated with a lecture, while 91.3% of the questions feature an explanation.

Original: The cross-combination of these information sources diversiﬁes theproblem scenario: sometimes the model is given a lot of information from multiple sources, while atother times, the only source of information is the question itself.
Revised: The cross-combination of these information sources diversiﬁes the problem scenario: sometimes the model is given a lot of information from multiple sources, while at other times, the only source of information is the question itself.

Original: This level of complexity is verycommon in grade-level science exams.
Revised: This level of complexity is very common in grade-level science exams.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: Question analysis.
Revised: Question analysis.

Original: SQA has a diverse set of science questions.
Revised: S CIENCE QA has a diverse set of science questions.

Original: Figure 2 shows a distribution ofthe ﬁrst four words in the question text.
Revised: Figure 2 shows a distribution of the ﬁrst four words in the question text.

Original: A large number of question lengths and formats highlightthe diversity of SQA .
Revised: A large number of question lengths and formats highlight the diversity of S CIENCE QA.

Original: The question lengths range from 3 words to 141 words, and the questions in SQA have an average length of 12.11 words.
Revised: The question lengths range from 3 words to 141 words, and the questions in S CIENCE QA have an average length of 12.11 words.

Original: The question length distribution is visualized againstother VQA datasets in Figure 3 (a).
Revised: The question length distribution is visualized against other VQA datasets in Figure 3 (a).

Original: As shown in the diagram, SQA ’s distribution is ﬂatter than otherdatasets, spanning more evenly across different question lengths.
Revised: As shown in the diagram, S CIENCE QA’s distribution is ﬂatter than other datasets, spanning more evenly across different question lengths.

Original: Context analysis.
Revised: Context analysis.

Original: Figure 3 (b) shows the number and percentage of questions with either an imagecontext, a text context, or both.
Revised: Figure 3 (b) shows the number and percentage of questions with either an image context, a text context, or both.

Original: There are a total of 7,803 unique image contexts and 4,651 unique textcontexts.
Revised: There are a total of 7,803 unique image contexts and 4,651 unique text [Table]

Original:  66.11% of the questions have at least one type of context information.
Revised: contexts. 66.11% of the questions have at least one type of context information.

Original: The image contextis in the format of diagrams or natural images, which visualize the critical scenario necessary forquestion answering or simply illustrate the question for better understanding.
Revised: The image context is in the format of diagrams or natural images, which visualize the critical scenario necessary for question answering or simply illustrate the question for better understanding.

Original: Similarly, the textualcontext can provide either semantically rich information or a simple hint to the question.
Revised: Similarly, the textual context can provide either semantically rich information or a simple hint to the question.

Original: Therefore,models need to be ﬂexible and general to understand these diverse types of contexts.
Revised: Therefore, models need to be ﬂexible and general to understand these diverse types of contexts.

Original: Domain analysis.
Revised: Domain diversity.

Original: Each SQA question belongs to one of the three subjects: natural science, languagescience, and social science.
Revised: Each S CIENCE QA question belongs to one of the three subjects: natural science, language science, and social science.

Original: With each subject, they are categorized ﬁrst by the topic ( Biology , Physics , Chemistry , etc.), then by the category ( Plants , Cells , Animals , etc.), and ﬁnally by the speciﬁc skill( Classify fruits and vegetables as plant parts , Identify countries of Africa , etc.).
Revised: With each subject, they are categorized ﬁrst by the topic ( Biology , Physics , Chemistry , etc.), then by the category ( Plants , Cells , Animals , etc.), and ﬁnally by the speciﬁc skill ( Classify fruits and vegetables as plant parts , Identify countries of Africa , etc.).

Original: SQA has a total of 26topics, 127 categories, and 379 skills.
Revised: S CIENCE QA has a total of 26 topics, 127 categories, and 379 skills.

Original: The treemap in Figure 9 visualizes the different subjects, topics,and categories and shows that SQA questions are very diverse, spanning a wide range of domains. 
Revised: The treemap in Figurevisualizes the different subjects, topics, and categories and shows that S CIENCE QA questions are very diverse, spanning a wide range of domains. 3.2 Comparisons with Existing Datasets Table 2 shows a comparison of S CIENCE QA and other science problem datasets. As shown in the table, S CIENCE QA is much larger than most other datasets.

Original: Baselines and Chain-of-Thought Models
Revised: S CIENCE QA also has the largest set of images, spans across all 12 grades, contains the longest questions, and has the most diverse input sources. As opposed to limiting the subject to only natural science, S CIENCE QA also includes social science and language science, largely adding to the domain diversity of the dataset. Furthermore, most of the questions in S CIENCE QA are annotated with textual lectures (83.9%) and explanations (90.5%), which reveal the reasoning path to the correct answer. To the best of our knowledge,

Revised: S CIENCE QA is the ﬁrst large-scale multimodal science question dataset that annotates the answers with detailed lectures and explanations

Original:  In this section, we establishvarious baselines and develop two chain-of-thought models on SQA .
Revised: [Table]  Baselines and Chain-of-Thought Models In this section, we establish baselines and develop two chain-of-thought models on S CIENCE QA.

Original: 4.1 Baselines
Revised: 4.1 Baselines

Original: Heuristic baselines.
Revised: Heuristic baselines.

Original: The ﬁrst heuristic baseline is random chance : we randomly select one from themultiple options.
Revised: The ﬁrst heuristic baseline is random chance : we randomly select one from the multiple options.

Original: Each trial is completed on the whole test set, and we take three different trials foran average result.
Revised: Each trial is completed on the whole test set, and we take three different trials for an average result.

Original: The second heuristic baseline is human performance .
Revised: The second heuristic baseline is human performance .

Original: We post the task to Amazon
Revised: We post the task to Amazon

Original: Mechanical Turk and ask workers to answer SQA questions.
Revised: Mechanical Turk and ask workers to answer S CIENCE QA questions.

Original: Only workers who obtain a high schoolor higher degree and pass the qualiﬁcation examples are qualiﬁed for the study.
Revised: Only workers who obtain a high school or higher degree and pass the qualiﬁcation examples are qualiﬁed for the study.

Original: Each worker needs toanswer a set of 10 test questions, and each question is answered by three different workers.
Revised: Each worker needs to answer a set of 10 test questions, and each question is answered by three different workers.

Original: For moredetails of the human performance study, see Appendix B.2.
Revised: For more details of the human performance study, see Appendix B.2.

Original: Zero-shot and few-shot baselines.
Revised: Zero-shot and few-shot baselines.

Original: We establish the zero-shot baselines on top of UniﬁedQA[18] and GPT-3 [5].
Revised: We establish the zero-shot baselines on top of UniﬁedQA [19] and GPT-3 [5].

Original: The zero-shot setup follows the format of QCM!
Revised: The zero-shot setup follows the format of QCM

Original:  A where the input is theconcatenation of tokens of the question text (Q), the context text (C), and multiple options (M), whilethe output is to predict the answer (A) from the option set.
Revised: → A where the input is the concatenation of tokens of the question text (Q), the context text (C), and multiple options (M), while the output is to predict the answer (A) from the option set.

Original: We extract the caption from the captioningmodel based on ViT [8] and GPT-2 [41] for the image as the visual context.
Revised: We extract the caption from the captioning model based on ViT [8] and GPT-2 [47] for the image as the visual context.

Original: In the few-shot setting, wefollow the standard prompting [4] where in-context examples from the training set are concatenatedbefore the test instance.
Revised: In the few-shot setting, we follow the standard prompting [4] where in-context examples from the training set are concatenated before the test instance.

Original: These in-context examples serve as an instruction for the language model toadjust to the speciﬁc task in SQA .
Revised: These in-context examples serve as an instruction for the language model to adjust to the speciﬁc task in S CIENCE QA.

Original: Fine-tuning baselines.
Revised: Fine-tuning baselines.

Original: We ﬁrst consider the ﬁne-tuning baselines from VQA models [1, 20, 49, 9,21, 31, 24] proposed in recent years.
Revised: We ﬁrst consider the ﬁne-tuning baselines from VQA models [1, 21, 55, 9, 22, 35, 26] proposed in recent years.

Original: These VQA baselines take the question, the context, and choicesas the textual input, take the image as the visual input, and predict the score distribution over choicecandidates via a linear classiﬁer.
Revised: These VQA baselines take the question, the context, and choices as the textual input, take the image as the visual input, and predict the score distribution over choice candidates via a linear classiﬁer.

Original: In addition, we build the ﬁne-tuning baseline on top of the largelanguage model UniﬁedQA [18].
Revised: In addition, we build the ﬁne-tuning baseline on top of the large language model UniﬁedQA [19].

Original: UniﬁedQA takes the textual information as the input and outputsthe answer option.
Revised: UniﬁedQA takes the textual information as the input and outputs the answer option.

Original: Similarly, the image is converted into a caption that provides the visual semanticsfor the language model.
Revised: Similarly, the image is converted into a caption that provides the visual semantics for the language model.

Original: 4.2 Language Models with the Chain of Thought
Revised: 4.2 Language Models with the Chain of Thought

Original: A chain of thought refers to a coherent ﬂow of sentences that reveals the premises and conclusion ofa reasoning problem [48].
Revised: A chain of thought refers to a coherent ﬂow of sentences that reveals the premises and conclusion of a reasoning problem [54].

Original: A chain of thought clearly decomposes a multi-hop reasoning task intointermediate steps instead of solving the task in a black-box way.
Revised: A chain of thought clearly decomposes a multi-hop reasoning task into intermediate steps instead of solving the task in a black-box way.

Original: The chain of thought can be thestep-by-step thought process [48] before arriving at the ﬁnal answer or explanations [36] that comeafter the answer.
Revised: The chain of thought can be the step-by-step thought process [54] before arriving at the ﬁnal answer or explanations [41] that come after the answer.

Original: The annotated lectures and explanations in SQA serve as demonstrations of the chain of thought that mimics the multi-step reasoning steps of human beings.
Revised: The annotated lectures and explanations in S CIENCE QA serve as demonstrations of the chain of thought that mimics the multi-step reasoning steps of human beings.

Original: In this paper, we studyif large language models can generate reasonable explanations as the chain of thought to reveal thethought process when answering SQA questions.
Revised: In this paper, we study if large language models can generate reasonable explanations as the chain of thought to reveal the thought process when answering S CIENCE QA questions.

Original: Further, we explore how the chain of thought canimprove the reasoning ability of language models on SQA in both few-shot and ﬁne-tuning learning.
Revised: Further, we explore how the chain of thought can improve the reasoning ability of language models on S CIENCE QA in both few-shot and ﬁne-tuning learning.

Original: UniﬁedQA with the chain of thought.
Revised: UniﬁedQA with the chain of thought.

Original: UniﬁedQA [18] is a state of the art model for multi-optionquestion answering.
Revised: UniﬁedQA [19] is a state of the art model for multi-option question answering.

Original: The original architecture of UniﬁedQA takes the question and options as theinput and outputs a short phrase as the ﬁnal answer.
Revised: The original architecture of UniﬁedQA takes the question and options as the input and outputs a short phrase as the ﬁnal answer.

Original: We make a format modiﬁcation to develop
Revised: We make a format modiﬁcation to develop

Original: UniﬁedQA with the chain of thought (CoT) i.e. UniﬁedQA is ﬁne-tuned to generate a long sequenceof text which consists of the answer followed by the lecture and explanation.
Revised: UniﬁedQA with the chain of thought (CoT), i.e., UniﬁedQA is ﬁne-tuned to generate a long sequence of text which consists of the answer followed by the lecture and explanation.

Original: GPT-3 via chain-of-thought prompting.
Revised: GPT-3 via chain-of-thought prompting.

Original: Recent research work [5] has shown that GPT-3 [5] canperform various tasks when provided in-context examples in a standard prompt.
Revised: Recent research work [5, 38, 34] has shown that GPT3 [5] can perform various tasks when provided in-context examples in a standard prompt.

Original: Take multi-optionquestion answering as an example, the standard prompt [32, 51, 27] builds instructions using incontext examples with components of the question text, options, and the correct answer text.
Revised: Take multi-option question answering as an example, the standard prompt [36, 57, 29] builds instructions using in-context examples with components of the question text, options, and the correct answer text.

Original: This styleof few-shot learning enables the GPT-3 model to answer speciﬁc questions without parameter updates.
Revised: This style of few-shot learning enables the GPT-3 model to answer speciﬁc questions without parameter updates.

Original: Different from standard prompting, we build GPT-3 via chain-of-thought (CoT) prompting, as shownin Figure 4.
Revised: Different from standard prompting, we build GPT-3 via chain-of-thought (CoT) prompting, as shown in Figure 5.

Original: To be speciﬁc, for each test problem t , we map the prompt instruction I : { I i } n , I t intoa textual format where { I i } n refers to the instruction set of n -shot in-context examples from thetraining set, while I t denotes the test instruction.
Revised: To be speciﬁc, for each test problem t , we map the prompt instruction I : { I i } n , I t into a textual format where { I i } n refers to the instruction set of n -shot in-context examples from the training set, while I t denotes the test instruction.

Original: Instead of the way where the explanation comesbefore the answer [48], we feed the instruction I into the encoder-decoder model GPT-3 to generatethe answer a followed by the lecture lect and explanation exp : M : { I i } n , I t ! a, lect, exp . Question: question : I
Revised: Instead of the way where the explanation comes before the answer [54], we feed the instruction I into the encoder-

Original: [Figure]
Revised: [Figure]

Original: Experiments
Revised: decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp :

Original: [Figure]
Revised: M : { I i } n , I t → a, lect, exp .

Original:   5.1 Experimental Setup
Revised:  Experiments 5.1 Experimental Setup

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: The heuristics and VQA baselines treat our SQA task as a multi-class classiﬁcation problem with multiple options and are evaluated with the accuracy metrics.UniﬁedQA and
Revised: The heuristics and VQA baselines treat our S CIENCE QA task as a multi-class classiﬁcation problem with multiple options and are evaluated with the accuracy metrics.

Original:  GPT-3 treat SQA as a text generation problem.
Revised: UniﬁedQA and GPT-3 treat S CIENCE QA as a text generation problem.

Original: So the most similar option is selected as the ﬁnalprediction to evaluate the question answering accuracy.
Revised: So the most similar option is selected as the ﬁnal prediction to evaluate the question answering accuracy.

Original: The generated lectures and explanations areevaluated by automatic metrics [39, 26, 43] and human scores by annotators.
Revised: The generated lectures and explanations are evaluated by automatic metrics [44, 28, 49] and human scores by annotators.

Original: Implementation details.
Revised: Implementation details.

Original: The VQA baselines are trained for a maximum number of 50 epochs with alearning rate of 5 e  5 .
Revised: The VQA baselines are trained for a maximum number of 50 epochs with a learning rate of 5 e − 5 .

Original: We ﬁne-tune the UniﬁedQA for 50 k iterations and evaluate every 1 k iteration.
Revised: We ﬁne-tune the UniﬁedQA for 50 k iterations and evaluate every 1 k iteration.

Original: The training process is stopped following the early stopping strategy with a patience period of threeevaluations.
Revised: The training process is stopped following the early stopping strategy with a patience period of three evaluations.

Original: For GPT-3, we use the text-davinci-002 engine, which is the most capable modelversion suggested in the ofﬁcial documentation.
Revised: For GPT-3, we use the text-davinci-002 engine, which is the most capable model version suggested in the ofﬁcial documentation.

Original: More details can be found in Appendix B.1.
Revised: More details can be found in Appendix B.1.

Original: 5.2 Results for Question Answering
Revised: 5.2 Results for Question Answering

Original: Table 3 demonstrates the empirical results for Science Question Answering.
Revised: Table 3 demonstrates the empirical results for Science Question Answering.

Original: VQA baselines.
Revised: VQA baselines.

Original: We feed the VQA baseline models with the input of QCM format to predict answers
Revised: We feed the VQA baseline models with the input of QCM format to predict answers

Original: A. Out of all the VQA models we benchmarked, VisualBERT [24, 25] performs the best on average(61.87%).
Revised: A. Out of all the VQA models we benchmarked, VisualBERT [26, 27] performs the best on average (61.87%).

Original: Interestingly, Patch-TRM [31] beats VisualBERT in natural science (NAT) and languagescience (LAN), and it also performs better in higher-grade questions (67.50% v.s. 59.92%).
Revised: Interestingly, Patch-TRM [35] beats VisualBERT in natural science (NAT) and language science (LAN), and it also performs better in higher-grade questions (67.50% v.s. 59.92%).

Original: However,in the subject of social science (SOC), VisualBERT outperforms Patch-TRM by a large margin[Table] (+22.39%).
Revised: However, in the subject of social science (SOC), VisualBERT outperforms Patch-TRM by a large margin (+22.39%).

Original: Such drastic changes in performance might imply that current VQA models are notgeneralized to process the challenging questions in SQA .
Revised: Such drastic changes in performance might imply that current VQA models are not generalized to process the challenging questions in S CIENCE QA.

Original: Language models.
Revised: Language models.

Original: We evaluate whether large-scale pretraining on text can help language modelslearn scientiﬁc knowledge and thus perform better on the SQA task.
Revised: We evaluate whether large-scale pretraining on text can help language models learn scientiﬁc knowledge and thus perform better on the S CIENCE QA task.

Original: For this purpose, we have triedtwo of the state-of-the-art pre-trained language models: UniﬁedQA and GPT-3.(i)
Revised: For this purpose, we have tried two of the state-of-the-art pre-trained language models: UniﬁedQA and GPT-3. i)

Original: UniﬁedQA.
Revised: UniﬁedQA

Original: The results show that without any supervised ﬁne-tuning (zero-shot), UniﬁedQAcannot beat any VQA baseline model, while the pretraining does help the model obtain some scientiﬁcknowledge to outperform the random baseline.
Revised: The results show that without any supervised ﬁne-tuning (zero-shot), UniﬁedQA cannot beat any VQA baseline model, while the pretraining does help the model obtain some scientiﬁc knowledge to outperform the random baseline.

Original: By training theT5 model (UniﬁedQA backbone) withthe answer labels in SQA , UniﬁedQA BASE reports an accuracy of 70.12% on average.
Revised: When ﬁne-tuned with the answer labels in S CIENCE QA, UniﬁedQA BASE reports an accuracy of 70.12% on average.

Original: By furtherteaching the model to generate the answer along with lecture and explanation, the developed languagemodel with chain-of-thought (UniﬁedQA BASE (CoT)) brings additional improvements of +3.21%(QCM ! AE) and +3.99% (QCM ! ALE).
Revised: By further teaching the model to generate the answer along with lecture and explanation, the developed language model with chain-of-thought (UniﬁedQA BASE (CoT)) brings additional improvements of +3.21% (QCM → AE) and +3.99% (QCM → ALE).

Original: These results show that generating the chain of thoughtalong with the answer beneﬁts the reasoning ability of language models.(ii) GPT-3.
Revised: These results show that generating the chain of thought along with the answer beneﬁts the reasoning ability of language models. (ii) GPT-

Original: The positive effect of pretraining is also proved by the surprisingly good results from
Revised: The positive effect of pretraining is also proved by the surprisingly good results from

Revised: GPT-

Revised: in the

Original: GPT-3 in the same zero-shot setting as UniﬁedQA.
Revised: same zero -shot setting as UniﬁedQA

Revised: Without any

Original: Without any ﬁne-tuning, GPT-3 already reachesalmost the best performance we can get.
Revised: ﬁne -tuning, GPT- 3 already reaches almost the best performance we can get.

Original: Interestingly, prompting the GPT-3 with two trainingexamples with just answers results in a negligible difference.
Revised: Interestingly, prompting the GPT-3 with two training examples with only answers results in a negligible difference.

Original: However, if we prompt GPT-3 withchain-of-thought prompting (QCM ! ALE), we obtain the state-of-the-art results so far (75.17%). 
Revised: However, if we prompt GPT-3 with chain of-thought prompting (QCM → ALE), we obtain the state-of-the-art result so far (75.17%). [Table] [Table]

Revised: -3 (CoT).

Original: Human performance.
Revised: Human performance.

Original: Humans outperform all benchmarks consistently across question classes,context types, and grades, e.g., a 20.07% gap for questions with the image context (IMG) betweenhumans and our best performing model.
Revised: Humans outperform all benchmarks consistently across question classes, context types, and grades, e.g., a 20.07% gap for questions with the image context (IMG) between humans and our best performing model.

Original: The gap is to be ﬁlled by future research on multimodalreasoning for scientiﬁc question answering.
Revised: The gap is to be ﬁlled by future research on multimodal reasoning for scientiﬁc question answering.

Original: [Table]

Original: 5.3 Results for Generated Explanations
Revised: 5.3 Results for Generated Explanations

Original: One prediction example of GPT-3 (CoT) is visualized in Figure 5.
Revised: One prediction example of GPT-3 (CoT) is visualized in Figure 6.

Original: We can see that GPT-3 (CoT)predicts the correct answer and generates a reasonable lecture and explanation to mimic the humanthought process.
Revised: We can see that GPT-3 (CoT) predicts the correct answer and generates a reasonable lecture and explanation to mimic the human thought process.

Original: We further report automatic metrics (BLEU-1/4 [39], ROUGE-L [39], and (sentence)
Revised: We further report automatic metrics (BLEU-1/4 [44], ROUGE-L [44], and (sentence)

Original: Similarity [43]) to evaluate the generated lectures and explanations, as shown in Table 4.The
Revised: Similarity [49] to evaluate the generated lectures and explanations, as shown in Table 4.

Original:  Similarity metric computes the cosine-similarity of semantic embeddings between two sentencesbased on the Sentence-BERT network [43].
Revised: The Similarity metric computes the cosine-similarity of semantic embeddings between two sentences based on the Sentence-BERT network [49].

Original: The results show that UniﬁedQA BASE (CoT) generates themost similar explanations to the given ones.
Revised: The results show that UniﬁedQA BASE (CoT) generates the most similar explanations to the given ones.

Original: However, it’s commonly agreed that automatic evaluationof generated texts only provides a partial view and has to be complemented by a human study.
Revised: However, it’s commonly agreed that automatic evaluation of generated texts only provides a partial view and has to be complemented by a human study.

Original: Byasking annotators to rate the relevance, correctness, and completeness of generated explanations, weﬁnd that the explanations generated by GPT-3 (CoT) conform best to human judgment.[Figure]
Revised: By asking annotators to rate the relevance, correctness, and completeness of generated explanations, we ﬁnd that the explanations generated by GPT-3 (CoT) conform best to human judgment.

Original: [Table]
Revised: [Table]

Original: 5.4 Analysis
Revised: 5.4 Analysis

Original: Blind studies.
Revised: Blind studies.

Original: Blind studies are conducted on top of the modiﬁcation of the full model, Top-Down[1].
Revised: Blind studies are conducted on top of the modiﬁcation of the full model, Top-Down [1].

Original: The results achieved in blind studies of Q only and C I only are close to random chance, showingthat the SQA dataset is robust and reliable in distribution.
Revised: The results achieved in blind studies of Q only and C I only are close to random chance, showing that the S CIENCE QA dataset is robust and reliable in distribution.

Original: The performance drops in Q+M only,
Revised: The performance drops in Q+M only,

Original: Q+C T +M only, and Q+C I +M only indicate that all input components provide critical information foranswering SQA questions.
Revised: Q+C T +M only, and Q+C I +M only indicate that all input components provide critical information for answering S CIENCE QA questions.

Original: Prompt types.
Revised: Prompt types.

Original: We study the effect of prompt types and visualize the comparison in Figure 6 (a).
Revised: We study the effect of prompt types and visualize the comparison in Figure 7 (a).

Original: It shows that prompting the GPT-3 model with both lecture and explanation (QCM ! ALE) resultsin the highest accuracy on average and the smallest variance.
Revised: It shows that prompting the GPT-3 model with both lectures and explanations (QCM → ALE) results in the highest accuracy on average and the smallest variance.

Original: In contrast, prompting with only theexplanation (QCM ! AE) gives the largest variance, resulting in a less stable model.
Revised: In contrast, prompting with only explanations (QCM → AE) gives the largest variance, resulting in a less stable model.

Original: [Figure] 
Revised: [Figure] [Figure]

Original: Number of in-context examples.
Revised: Number of in-context examples.

Original: In Figure 6 (b), we further investigate how different numbers oftraining examples encoded in prompts can affect the prediction accuracy.
Revised: In Figure 7 (b), we further investigate how different numbers of training examples encoded in prompts can affect the prediction accuracy.

Original: The QCM ! ALE prompt type outperforms or performs comparably the QCM!
Revised: The QCM → ALE prompt type outperforms or performs comparably the QCM

Original:  A type with all numbers of examples.
Revised: → A type with all numbers of examples.

Original: And weobserve the peak performance of QCM ! ALE with 2 training examples being prompted.
Revised: And we observe the peak performance of QCM → ALE with 2 training examples being prompted.

Original: After that,the accuracy goes down as more training examples are added to the model.
Revised: After that, the accuracy goes down as more training examples are added to the model.

Original: [Table]
Revised: [Table]

Original: Dynamic sampling.
Revised: Dynamic sampling.

Original: In Table 5, instead of random sampling, we try to dynamically select thein-context examples to prompt with the sameclass as the test sample.
Revised: In Table 5, instead of random sampling, we try to dynamically select the in-context examples to prompt with the same class as the test sample.

Original: However, slight differences in prediction accuracy are observed whencomparing them to simple random sampling.
Revised: However, slight differences in prediction accuracy are observed when comparing them to simple random sampling.

Original: Upper bound.
Revised: Upper bound.

Original: We search the upper bound of the GPT-3 accuracy by feeding the gold lecture andexplanation in the test prompt.
Revised: We search the upper bound of the GPT-3 accuracy by feeding the gold lecture and explanation in the test prompt.

Original: As reported in Table 6, QCME* !

Original:  A outperforms the QCM ! ALEbaseline by 18.86% and QCMLE* ! A outperforms QCM ! ALE by 18.96%, indicating a potentialimprovement direction by generating correct explanations before answering science questions.
Revised: As reported in Table 6, QCME* → A outperforms the QCM → ALE baseline by 18.86% and QCMLE* → A outperforms QCM → ALE by 18.96%, indicating a potential improvement direction by generating correct explanations before answering science questions.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Positions of lectures and explanations.
Revised: Positions of lectures and explanations.

Original: We study the performance of GPT-3 (CoT) in terms ofdifferent positions of lectures and explanations on 1,000 test examples.
Revised: We study the performance of GPT-3 (CoT) in terms of different positions of lectures and explanations on 1,000 test examples.

Original: Results  in Table 7 there couldbe huge accuracy decreases if GPT-3 (CoT) predictsthe lectures and explanations before answers.
Revised: Results are shown in Table 7. There could be huge accuracy decreases if GPT-3 (CoT) predicts lectures and explanations before answers.

Original: Itis mainly because if GPT-3 (CoT) is formalized to generate the long lecture and explanation ﬁrst,there is a larger chance that it stops generating the prediction early or use up the maximum tokenlimits before obtaining the required answer. 
Revised: It is mainly because if GPT-3 (CoT) is formalized to generate the long lecture and explanation ﬁrst, there is a larger chance that it stops generating the prediction early or use up the maximum token limits before obtaining the required answer. [Figure]

Original: CoT learns with fewer data.
Revised: CoT learns with fewer data.

Original: To study if the chain ofthought helps language models learn more efﬁciently, wereport the accuracies of UniﬁedQA and UniﬁedQA (CoT)ﬁne-tuned on different sizes of the training set in Figure 7.
Revised: To study if the chain of thought helps language models learn more efﬁciently, we report the accuracies of UniﬁedQA and UniﬁedQA (CoT) ﬁne-tuned on different sizes of the training set in Figure 8.

Original: UniﬁedQA (CoT) beneﬁtsthe language models by learningthe coherent reasoning path when answering SQA questions,resulting in similar accuracy with fewer training examples.
Revised: UniﬁedQA (CoT) beneﬁts language models by learning the coherent reasoning path when answering questions, resulting in similar accuracy with fewer training examples.

Original: Error analysis.
Revised: Error analysis.

Original: GPT-3 via chain-of-chain prompting obtains promising results but still fails to answer a wide rangeof challenging questions in SQA .
Revised: GPT-3 via chain-of-chain prompting obtains promising results but still fails to answer a wide range of challenging questions in S CIENCE QA.

Original: See examples of failurecases in Appendix B.4.
Revised: See examples of failure cases in Appendix B.4.

Original: The failure cases can be classiﬁedinto two types: (a) the model fails to understand the multimodal inputs and lacks domain-speciﬁc knowledge to arrive at the correct answer; (b) the model generates the wrong chain of thought with irrelevant, incorrect, or incomplete information.
Revised: The failure cases can be classiﬁed into two types: (a) the model fails to understand the multimodal inputs and lacks domain-speciﬁc knowledge to arrive at the correct answer; (b) the model generates the wrong chain of thought with irrelevant, incorrect, or incomplete information.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: In this paper, we propose Science Question Answering SQA , a dataset that features 21,208 multioption questions with multimodal contexts from the science curriculum.
Revised: In this paper, we propose S CIENCE QA, a dataset that features 21,208 multi-option questions with multimodal contexts from the science curriculum.

Original: To the best of our knowledge, SQA is the ﬁrst large-scale multimodal science dataset where most questions are annotated withcorresponding lectures and explanations.
Revised: To the best of our knowledge, S CIENCE QA is the ﬁrst large-scale multimodal science dataset where most questions are annotated with corresponding lectures and explanations.

Original: We establish various baselines, including recent VQAmodels and large language models on SQA .
Revised: We establish various baselines, including recent VQA models and large language models on S CIENCE QA.

Original: We further study if language models can generatereasonable explanations and then beneﬁt the reasoning ability.
Revised: We further study if language models can generate reasonable explanations and then beneﬁt the reasoning ability.

Original: Experiments show that the UniﬁedQAwith the chain of thought can achieve an improvement of 3.99% and few-shot GPT-3 via chainof-thought (CoT) prompting 
Revised: Experiments show that UniﬁedQA with the chain of thought can achieve an improvement of 3.99% and few-shot GPT-3 via chain-of-thought (CoT) prompting can obtain a satisfactory accuracy of 75.17% on S CIENCE QA. 65.2% of the generated explanations from GPT-3 (CoT) meet the gold standard by human evaluations.  Acknowledgment

Revised: We would like to thank the anonymous reviewers for their valuable comments and suggestions.

Revised: We would also like to thank Xiaodan Liang for insightful discussions on dataset collection.

Revised: We thank our colleagues at The Allen Institute of AI (AI2), Jiasen Lu and Jungo Kasai for helpful discussions.

Revised: The work does not relate to Liang Qiu’s position at Amazon Alexa.

Revised: Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Revised: [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] Yes, we did the error analysis in Section 5.4 and discussed the limitations of the work in Appendix B.4.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] We discussed the broader impacts in Appendix B.5.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Revised: [N/A] 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] We included 100 data examples and the data visualizer tool in the supplemental material.

Revised: The whole dataset and code will be available at https://scienceqa.github.io .

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] See Section 5.1 and Appendix B.1 for experimental details.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] We reported the error bars for GPT-3 (CoT) experiments in Figure 7, where each experiment was repeated four times.

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] We discussed compute resources in Appendix B.1.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] We collected the S CIENCE QA dataset from https://www.ixl.com/ .

Revised: The copyright belongs to IXL.

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] S CIENCE QA is under the CC BY-NCSA 4.0 license and is used for non-commercial research purposes.

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes] We included data examples and a visualizer tool in the supplemental material.

Revised: The dataset will be available at https://scienceqa.github.io .

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Revised: [Yes] The collected data does not contain personally identiﬁable information or offensive content.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [Yes] We included screenshots of the instructions in Appendix B.2 and B.3.

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [Yes] We included the monetary compensation details in Appendix B.2 and B.3.

Revised: A Dataset Analysis A.1 Data Collection

Revised: Questions in the S CIENCE QA dataset are sourced from open resources managed by IXL Learning, an online learning platform curated by experts in the ﬁeld of K-12 education.

Revised: The dataset includes problems that align with California Common Core Content Standards .

Revised: To construct S CIENCE QA, we downloaded the original science problems and then extracted individual components (e.g. questions, hints, images, options, answers, lectures, and solutions) from them based on heuristic rules.

Revised: We manually removed invalid questions, such as questions that have only one choice, questions that contain faulty data, and questions that are duplicated, to comply with fair use and transformative use of the law.

Revised: If there were multiple correct answers that applied, we kept only one correct answer.

Revised: Also, we shufﬂed the answer options of each question to ensure the choices do not follow any speciﬁc pattern.

Revised: To make the dataset easy to use, we then used semi-automated scripts to reformat the lectures and solutions.

Revised: Therefore, special structures in the texts, such as tables and lists, are easily distinguishable from simple text passages.

Revised: Similar to ImageNet, ReClor, and PMR datasets,

Revised: S CIENCE QA is available for non-commercial research purposes only and the copyright belongs to the original authors.

Revised: To ensure data quality, we developed a data exploration tool to review examples in the collected dataset, and incorrect annotations were further manually revised by experts.

Revised: The tool can be accessed at https://scienceqa.github.io/explore.html .

Revised: A.2 Question Statistics

Revised: Figure 9 (a) is a word cloud showing the most frequently appeared words in the question texts.

Revised: Stopping words that do not contain any semantic meaning, such as “ what ” or “ and ”, are removed to give us a clearer view of the semantic range of S CIENCE QA.

Revised: The diagram shows that S CIENCE QA covers a wide range of topics, with words from different topics showing up across the cloud.

Revised: Figures 9 (b) (c) (d) show the word clouds for each of the three subjects.

Revised: We can observe from the word clouds that the words are well-matched to the subject themes.

Revised: In natural science questions, words such as “ trait ”, “ magnet ”, and “ force ” appear frequently.

Revised: Words such as “ capital ” and “state” show up frequently in social science questions, whereas words such as “ dictionary ” and “ page ” are common in language science questions.

Revised: [Figure]

Revised: A.3 Choice Statistics

Revised: [Table]

Revised: Table 8 shows the number of questions with each number of different choices.

Revised: Questions have a minimum of two options and a maximum of ﬁve options.

Revised: Figure 10 shows the distribution of choice length in S CIENCE QA.

Revised: Most choices are short, containing up to ﬁve words.

Revised: However, the distribution has a long tail where about 5% of the choices contain more thanwords.

Revised: Hence, it requires models to have a high level of text understanding to address diversely distributed choices.

Revised: A.4 Subject Statistics

Revised: Figure 11 shows the question length distribution of each subject.

Revised: The three subjects all feature long-tail distributions in terms of the number of question words.

Revised: On average, social science questions are the shortest, while language science questions are the longest.

Revised: Language science questions are distributed more evenly than other questions across different numbers of words.

Revised: These features imply that the S CIENCE QA dataset is rich in compositional diversity.

Revised: [Figure]

Revised: [Figure]

Revised: A.5 Grade Statistics

Revised: The grade distribution is shown in Figure 12.

Revised: The majority of questions come from the middle level curriculum (i

Revised: [Figure]

Revised: e., from grade 3 to grade 8) while around 10% are taken from the high school curriculum (i.e., from grade 9 to grade 12).

Revised: These high school level questions are close to or at the difﬁculty level of the U.S. standardized tests for college admissions.

Revised: Machine algorithms need to master a large amount of scientiﬁc knowledge and perform complex reasoning in order to perform well on S CIENCE

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: B Experiments B.1 Experimental Details

Revised: Below are details on the experiments:

Revised: • Fine-tuning on the dataset.

Revised: Fine-tuning baselines (VQA baselines and UniﬁedQA) are trained on the training set, developed on the validation set, and evaluated on the test set.

Revised: • Input sizes: For VQA baselines, we set the maximum number of input words or tokens as 100.

Revised: • Batch sizes.

Revised: We use batches of 64 and 4 for VQA baselines and ﬁne-tuned UniﬁedQA, respectively.

Revised: • Newline character.

Revised: For language models, the newline separators (\ n ) in the text are replaced with \\ n when encoding the inputs because \ n is normally used as a stop symbol, following the original works [5, 19].

Revised: • Captioning model.

Revised: We use the tool 2 to generate captions for the images in the dataset.

Revised: The maximum length of generated captions is 16, the number of beams is 4, and the maximum number of output tokens is 512.

Revised: • Compute resources.

Revised: We use two GeForce RTX 3090 GPUs for ﬁne-tuning VQA baselines and UniﬁedQA on the dataset.

Revised: • Questions without any context.

Revised: For questions without any context, the context text is replaced with an empty string.

Revised: • GPT-3:

Revised: Following the default settings, we choose temperature, frequency penalty and presence penalty as 0.0, and top probability as 1.0.

Revised: All experiments of GPT-3 are run via online API.

Revised: Experiments in

Revised: Figure

Revised: are repeated four times with example candidates listed in

Revised: Table 9.

Revised: Experiments in Table 3, 5, and 6 are conducted using in-context examples with the trial ID of 1.

Revised: [Table]

Revised: B.2 Human Performance Study

Revised: In order to understand how humans perform on S CIENCE QA questions, we used Amazon Mechanical

Revised: Turk (AMT) to crowd source answers to the test set.

Revised: The interfaces of instructions and one example of a test question are shown in Figure 13.

Revised: A total of 4,241 test questions were shufﬂed and split into 425 batches, with each batch having 10 questions (excluding the last one).

Revised: For each batch, we also randomly added ﬁve training questions as exam examples.

Revised: Each set of 15 questions was then assigned to 3 AMT workers.

Revised: Only workers who correctly answer 4 out of the 5 exam examples or more are qualiﬁed for the human performance study.

Revised: In other words, workers who failed to pass the qualiﬁed exam were eliminated from the analysis.

Revised: For each set of 15 questions, we provided the worker with $0.5 per Hit task.

Revised: At the rate of 3 questions per minute, this amounts to $6.0 per hour.

Revised: B.3 Human Evaluation of Generated Explanations

Revised: We also evaluated the quality of predictions from GPT-3 (CoT) and UniﬁedQA (CoT) by asking

Revised: AMT workers to rate the model-generated explanations.

Revised: The interface is shown in Figure 14.

Revised: Each sample’s question text, contexts, choices, and answers were presented, along with the corresponding explanation generated by language models.

Revised: The workers were asked to decide whether the proposed explanation is relevant (is related to the question), correct (gives a correct answer and explanation),

Revised: [Figure]

Revised: and complete (fully explains the answer).

Revised: Prediction outputs that contain textual explanations were grouped into batches of 10, each assigned to 3 workers for evaluation.

Revised: For each batch, we provided the workers with a monetary compensation of $0.3.

Revised: Finally, the human scores for each explanation were determined by taking a majority vote.

Revised: B.4 Case Study and Limitations

Revised: Figure 15 shows three examples with a correct answer and a gold explanation predicted by GPTvia chain-of-thought prompting (CoT).

Revised: We can see that GPT-3 (CoT) not only predicts the correct answers but also generates reasonable explanations, which follow the multi-hop reasoning process of human beings.

Revised: This suggests that large language models like GPT-3 have great promise for implementing high-level reasoning abilities.

Revised: Figure 16 visualizes three more examples with predictions from GPT-3 (CoT).

Revised: In these examples,

Revised: GPT-3 (CoT) is able to predict the correct answers but fails to generate gold explanations.

Revised: For example, GPT-3 (CoT) generates an irrelevant explanation because the context text does not include ﬁne-grained visual information in the image (Figure 16a).

Revised: In the example shown in Figure 16b,

Revised: GPT-3 (CoT) fails to predict the coherent thought chains, where there are an incorrect example and

Revised: [Table]

Revised: [Figure]

Revised: an incorrect statement for a chemical change .

Revised: The third example is given in Figure 16c, where the generated explanation is just a repetition of the input question and the output answer, instead of following the complete thought chain to arrive at the ﬁnal answer.

Revised: Four failure examples with wrong predicted answers are listed in Figure 17.

Revised: We extract the image captions and feed them to the large language model as the visual content input.

Revised: However, these captions lack ﬁne-grained semantics and usually do not work well for diagrams, which results in two failure cases shown in Figure 17a and 17b.

Revised: Moreover, there exist challenges for large language models to reason about the questions that require them to understand complex and uncommon domain knowledge.

Revised: For example, GPT-3 (CoT) cannot understand accurately the terminology of personiﬁcation in language science (Figure 17c) and a series of complex chemical changes happen in the formation process of dinosaur fossils (Figure 17d).

Revised: B.5 Broader Impacts

Revised: Societal impact.

Revised: The S CIENCE QA dataset collects science questions sourced from textbooks and is proposed to diagnose the multimodal understanding and multi-hop reasoning abilities of AI systems.

Revised: Due to the nature of data sources, S CIENCE QA does not contain any user usage data or personally sensitive information such as gender and race.

Revised: After careful examination of our dataset, to our best knowledge, we have not found any improper content, such as pornographic information, racial remarks, or harmful social bias.

Revised: We adhere to the goal of AI for the common good, and any antisocial data points will be removed from the dataset based on feedback.

Revised: [Table]

Revised: [Figure]

Revised: Potential usage.

Revised: The proposed dataset S CIENCE QA and designed methods in this paper are beneﬁcial to both follow-up research work and real-world applications.

Revised: S CIENCE QA provides a useful benchmark for multi-modal learning, multi-hop reasoning, and general artiﬁcial intelligence.

Revised: Besides,

Revised: S CIENCE QA will contribute to the development of K-12 education applications such as tutoring systems.

Revised: Furthermore, the designed methods with the chain of thought investigate the ability of large language models to mimic the human mind process while reasoning about a challenging task.

Revised: [Table]

Revised: (cid:58)(cid:85)(cid:82)(cid:81)(cid:74)(cid:3)(cid:36)(cid:81)(cid:86)(cid:90)(cid:72)(cid:85) (cid:20)(cid:26)(cid:26)(cid:24)(cid:22)