Original: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Revised: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Original: Abstract
Revised: Abstract

Original: When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT).
Revised: When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT).

Original: This process is normally a black box in the case of deep learning models like large-scale language models.
Revised: This process is normally a black box in the case of deep learning models like large-scale language models.

Original: Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system.
Revised: Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system.

Original: However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity.
Revised: However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity.

Original: To this end, we present Science Question Answering (S CIENCE QA), a new benchmark that consists of ∼ 21k multimodal multiple choice questions with diverse science topics and annotations of their answers with corresponding lectures and explanations.
Revised: To this end, we present Science Question Answering (S CIENCE QA), a new benchmark that consists of ∼ 21k multimodal multiple choice questions with diverse science topics and annotations of their answers with corresponding lectures and explanations.

Original: We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering
Revised: We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering

Original: S CIENCE QA questions.
Revised: S CIENCE QA questions.

Original: S CIENCE QA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in fewshot GPT-3 and 3.99% in ﬁne-tuned UniﬁedQA.
Revised: S CIENCE QA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in fewshot GPT-3 and 3.99% in ﬁne-tuned UniﬁedQA.

Original: We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%.
Revised: We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%.

Original: Our analysis further shows that language models, similar to humans, beneﬁt from explanations to learn from fewer data and achieve the same performance with just 40% of the data.
Revised: Our analysis further shows that language models, similar to humans, beneﬁt from explanations to learn from fewer data and achieve the same performance with just 40% of the data.

Original:  Introduction
Revised:  Introduction

Original: A long-standing goal of AI systems is to act reliably and learn complex tasks efﬁciently like human beings.
Revised: A long-standing goal of AI systems is to act reliably and learn complex tasks efﬁciently like human beings.

Original: In the process of reliable decision making, humans follow an explicit chain-of-thought (CoT) reasoning process that is typically expressed as an explanation.
Revised: In the process of reliable decision making, humans follow an explicit chain-of-thought (CoT) reasoning process that is typically expressed as an explanation.

Original: However, machine learning models are trained mostly using a large number of input-output examples to perform a speciﬁc task.
Revised: However, machine learning models are trained mostly using a large number of input-output examples to perform a speciﬁc task.

Original: These black-box models only generate the ﬁnal decision without reliably revealing the underlying reasoning process.
Revised: These black-box models only generate the ﬁnal decision without reliably revealing the underlying reasoning process.

Original: Not surprisingly, it is unclear if they understand the task and can generalize even though they perform well on the benchmark.
Revised: Not surprisingly, it is unclear if they understand the task and can generalize even though they perform well on the benchmark.

Original: On the other hand, humans are able to learn from instructions or explanations from past experience and generalize them to novel and unseen problems.
Revised: On the other hand, humans are able to learn from instructions or explanations from past experience and generalize them to novel and unseen problems.

Original: This helps them learn more quickly with fewer data.
Revised: This helps them learn more quickly with fewer data.

Original: In this work, we explore if machines can be endowed with such reasoning abilities in the context of science-based question answering.
Revised: In this work, we explore if machines can be endowed with such reasoning abilities in the context of science-based question answering.

Original: Recently, science problem solving benchmarks [18] have been used to diagnose the multi-hop reasoning ability and interpretability of AI systems.
Revised: Recently, science problem solving benchmarks [18] have been used to diagnose the multi-hop reasoning ability and interpretability of AI systems.

Original: To answer science questions, a model needs to
Revised: To answer science questions, a model needs to

Original: [Table]
Revised: [Table]

Original: (cid:37)(cid:40)(cid:38)(cid:36)(cid:56)(cid:54)(cid:40)(cid:29) not only understand multimodal contents but also extract external knowledge to arrive at the correct answer.
Revised: (cid:37)(cid:40)(cid:38)(cid:36)(cid:56)(cid:54)(cid:40)(cid:29) not only understand multimodal contents but also extract external knowledge to arrive at the correct answer.

Original: Since these tasks require domain-speciﬁc knowledge and explicit multi-hop reasoning, a model would be not interpretable if it fails to provide explanations to reveal the reasoning process.
Revised: Since these tasks require domain-speciﬁc knowledge and explicit multi-hop reasoning, a model would be not interpretable if it fails to provide explanations to reveal the reasoning process.

Original: However, current science question datasets [18, 17, 52] mostly lack annotated explanations for the answers.
Revised: However, current science question datasets [18, 17, 52] mostly lack annotated explanations for the answers.

Original: To address this issue, other science datasets annotate the explanations, but they are restricted to the textual only modality and limited to small data scales [13, 7, 37] or a small set of topics [20, 14].
Revised: To address this issue, other science datasets annotate the explanations, but they are restricted to the textual only modality and limited to small data scales [13, 7, 37] or a small set of topics [20, 14].

Original: Therefore, we collect Science Question Answering (S CIENCE QA), a large-scale multi-choice dataset that contains multimodal science questions with explanations and features rich domain diversity.
Revised: Therefore, we collect Science Question Answering (S CIENCE QA), a large-scale multi-choice dataset that contains multimodal science questions with explanations and features rich domain diversity.

Original: S CIENCE QA is collected from elementary and high school science curricula, and contains 21, examples along with lectures and explanations.
Revised: S CIENCE QA is collected from elementary and high school science curricula, and contains 21, examples along with lectures and explanations.

Original: Different from existing datasets [17, 18, 52], S CIENCE QA has richer domain diversity from three different subjects: natural science, social science, and language science.
Revised: Different from existing datasets [17, 18, 52], S CIENCE QA has richer domain diversity from three different subjects: natural science, social science, and language science.

Original: A typical example consists of a question, multiple choices, multimodal contexts, a correct answer, as well as a lecture and an explanation.
Revised: A typical example consists of a question, multiple choices, multimodal contexts, a correct answer, as well as a lecture and an explanation.

Original: The lecture and explanation provide general external knowledge and speciﬁc reasons, respectively, for arriving at the correct answer.
Revised: The lecture and explanation provide general external knowledge and speciﬁc reasons, respectively, for arriving at the correct answer.

Original: Consider the thoughts one person might have when answering the question in Figure 1.
Revised: Consider the thoughts one person might have when answering the question in Figure 1.

Original: One ﬁrst recalls the knowledge regarding the deﬁnition of a force learned from textbooks: “ A force is a push or a pull that ...
Revised: One ﬁrst recalls the knowledge regarding the deﬁnition of a force learned from textbooks: “ A force is a push or a pull that ...

Original: The direction of a push is ...
Revised: The direction of a push is ...

Original: The direction of a pull is ... ”, then forms a line of reasoning: “ The baby’s hand applies a force to the cabinet door .
Revised: The direction of a pull is ... ”, then forms a line of reasoning: “ The baby’s hand applies a force to the cabinet door .

Original: → This force causes the door to open .
Revised: → This force causes the door to open .

Original: → The direction of this force is toward the baby’s hand . ”, and ﬁnally arrives at the correct answer: “ This force is a pull . ”.
Revised: → The direction of this force is toward the baby’s hand . ”, and ﬁnally arrives at the correct answer: “ This force is a pull . ”.

Original: Following [41], we formulate the task to output a natural explanation alongside the predicted answer.
Revised: Following [41], we formulate the task to output a natural explanation alongside the predicted answer.

Original: In this paper, we train language models to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process to answer S CIENCE QA questions.
Revised: In this paper, we train language models to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process to answer S CIENCE QA questions.

Original: Our experiments show that current multimodal methods [55, 1, 21, 9, 26, 35] fail to achieve satisfactory performance on S CIENCE QA and do not generate correct explanations.
Revised: Our experiments show that current multimodal methods [55, 1, 21, 9, 26, 35] fail to achieve satisfactory performance on S CIENCE QA and do not generate correct explanations.

Original: Instead, we ﬁnd that CoT can help large language models not only in the few-shot learning setting but also in the ﬁne-tuning setting.
Revised: Instead, we ﬁnd that CoT can help large language models not only in the few-shot learning setting but also in the ﬁne-tuning setting.

Original: When combined with CoT to generate the lecture and explanation, the ﬁne-tuned UniﬁedQA [19] achieves an improvement of 3.99% as opposed to not using CoT in the ﬁne-tuning stage.
Revised: When combined with CoT to generate the lecture and explanation, the ﬁne-tuned UniﬁedQA [19] achieves an improvement of 3.99% as opposed to not using CoT in the ﬁne-tuning stage.

Original: The few-shot GPT-3 model [5] via chain-of-thought prompting can obtain 75.17% on S CIENCE QA with an improvement of 1.20% compared to the few-shot GPT-3 without CoT. Prompted with CoT, GPTcan generate reasonable explanations as evaluated by automated metrics, and promisingly, 65.2% of explanations meet the gold standard of human evaluations.
Revised: The few-shot GPT-3 model [5] via chain-of-thought prompting can obtain 75.17% on S CIENCE QA with an improvement of 1.20% compared to the few-shot GPT-3 without CoT. Prompted with CoT, GPTcan generate reasonable explanations as evaluated by automated metrics, and promisingly, 65.2% of explanations meet the gold standard of human evaluations.

Original: We also investigate the upper bound for models to harness explanations by including them in the input.
Revised: We also investigate the upper bound for models to harness explanations by including them in the input.

Original: We ﬁnd that doing so improves
Revised: We ﬁnd that doing so improves

Original: GPT-3’s few-shot performance by 18.96%, suggesting that explanations do aid models and are currently underutilized in the CoT framework.
Revised: GPT-3’s few-shot performance by 18.96%, suggesting that explanations do aid models and are currently underutilized in the CoT framework.

Original: Further analysis shows that, like humans, language models beneﬁt from explanations to learn with less data: UniﬁedQA with CoT obtains the same results as UniﬁedQA without CoT with only 40% of the training data.
Revised: Further analysis shows that, like humans, language models beneﬁt from explanations to learn with less data: UniﬁedQA with CoT obtains the same results as UniﬁedQA without CoT with only 40% of the training data.

Original: To sum up, our contributions are three-fold: (a) To bridge the gap in existing datasets in the scientiﬁc domain, we build Science Question Answering (S CIENCE QA), a new dataset containing 21, multimodal science questions with rich domain diversity.
Revised: To sum up, our contributions are three-fold: (a) To bridge the gap in existing datasets in the scientiﬁc domain, we build Science Question Answering (S CIENCE QA), a new dataset containing 21, multimodal science questions with rich domain diversity.

Original: To the best of our knowledge, S CIENCE QA is the ﬁrst large-scale multimodal dataset that annotates lectures and explanations for the answers.
Revised: To the best of our knowledge, S CIENCE QA is the ﬁrst large-scale multimodal dataset that annotates lectures and explanations for the answers.

Original: (b) We show that CoT beneﬁts large language models in both few-shot and ﬁne-tuning learning by improving model performance and reliability via generating explanations.
Revised: (b) We show that CoT beneﬁts large language models in both few-shot and ﬁne-tuning learning by improving model performance and reliability via generating explanations.

Original: (c) We further explore the upper bound of GPT-3 and show that CoT helps language models learn from fewer data.
Revised: (c) We further explore the upper bound of GPT-3 and show that CoT helps language models learn from fewer data.

Original:  Related Work
Revised:  Related Work

Original: Visual question answering.
Revised: Visual question answering.

Original: Since the task of visual question answering (VQA) was ﬁrst proposed in [2], there have been plenty of VQA datasets [56, 58, 23, 11, 15, 12] conducted to facilitate the research work.
Revised: Since the task of visual question answering (VQA) was ﬁrst proposed in [2], there have been plenty of VQA datasets [56, 58, 23, 11, 15, 12] conducted to facilitate the research work.

Original: Although our S CIENCE QA dataset shares some features with VQA, there are several main differences between them.
Revised: Although our S CIENCE QA dataset shares some features with VQA, there are several main differences between them.

Original: First, S CIENCE QA is more challenging than existing VQA datasets because it contains multimodal contexts and diverse topics in the scientiﬁc domain.
Revised: First, S CIENCE QA is more challenging than existing VQA datasets because it contains multimodal contexts and diverse topics in the scientiﬁc domain.

Original: In addition, most answers are annotated with lectures and explanations, which makes S CIENCE QA a suitable dataset for multi-modal question answering and multi-hop reasoning for AI systems.
Revised: In addition, most answers are annotated with lectures and explanations, which makes S CIENCE QA a suitable dataset for multi-modal question answering and multi-hop reasoning for AI systems.

Original: Inspired by the recent remarkable performance achieved for VQA [33, 32, 10, 9, 26, 8], in this paper, we further extensively benchmark S CIENCE QA with a wide range of attention-based [1, 33, 21, 9] and Transformer-based [30, 26, 27, 8] methods.
Revised: Inspired by the recent remarkable performance achieved for VQA [33, 32, 10, 9, 26, 8], in this paper, we further extensively benchmark S CIENCE QA with a wide range of attention-based [1, 33, 21, 9] and Transformer-based [30, 26, 27, 8] methods.

Original: Datasets for science problems.
Revised: Datasets for science problems.

Original: Science problem solving is a challenging task that requires an AI system not only to understand the multimodal information from the science curriculum but also to reason about how to answer the domain-speciﬁc questions.
Revised: Science problem solving is a challenging task that requires an AI system not only to understand the multimodal information from the science curriculum but also to reason about how to answer the domain-speciﬁc questions.

Original: Current science problem datasets such as AI2D [17], DVQA [16], VLQA [52], and FOODWEDS [24] have contributed to multimodal reasoning in the scientiﬁc domain.
Revised: Current science problem datasets such as AI2D [17], DVQA [16], VLQA [52], and FOODWEDS [24] have contributed to multimodal reasoning in the scientiﬁc domain.

Original: For example, a portion of VLQA contains multimodal questions on science subjects.
Revised: For example, a portion of VLQA contains multimodal questions on science subjects.

Original: These datasets, however, lack annotated explanations for the answers to reveal the reasoning steps.
Revised: These datasets, however, lack annotated explanations for the answers to reveal the reasoning steps.

Original: Some other datasets annotate the answers in the forms of supporting facts [37, 20], entailment trees [7], explanation graphs [13], reasoning chains [14].
Revised: Some other datasets annotate the answers in the forms of supporting facts [37, 20], entailment trees [7], explanation graphs [13], reasoning chains [14].

Original: However, these datasets are restricted to the single text modality with small data scales and limited topics.
Revised: However, these datasets are restricted to the single text modality with small data scales and limited topics.

Original: Instead, our
Revised: Instead, our

Original: S CIENCE QA annotates the answers with grounded lectures and explanations.
Revised: S CIENCE QA annotates the answers with grounded lectures and explanations.

Original: Besides, S CIENCE QA features a richer domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills.
Revised: Besides, S CIENCE QA features a richer domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills.

Original: Learning from explanations and few-shot learning.
Revised: Learning from explanations and few-shot learning.

Original: Explanations help humans understand a task better, and there have been several attempts to show the same for models.
Revised: Explanations help humans understand a task better, and there have been several attempts to show the same for models.

Original: For example, the learning from instruction paradigm [40, 43, 53, 39, 45, 25], where the task level explanation is provided in the form of instruction, improves model performance signiﬁcantly.
Revised: For example, the learning from instruction paradigm [40, 43, 53, 39, 45, 25], where the task level explanation is provided in the form of instruction, improves model performance signiﬁcantly.

Original: An example of learning from explanations in the scientiﬁc domain is proposed in [51] where the model interprets demonstrative solutions to solve geometry problems.
Revised: An example of learning from explanations in the scientiﬁc domain is proposed in [51] where the model interprets demonstrative solutions to solve geometry problems.

Original: Recently, there has been a surge of interest in few-shot learning, where language models learn a speciﬁc task from a few examples [46, 3].
Revised: Recently, there has been a surge of interest in few-shot learning, where language models learn a speciﬁc task from a few examples [46, 3].

Original: For instance, [42, 54, 34] ﬁnd that explanations in the format of the chain of thought can improve language models’ reasoning ability in few-shot learning.
Revised: For instance, [42, 54, 34] ﬁnd that explanations in the format of the chain of thought can improve language models’ reasoning ability in few-shot learning.

Original: In this paper, we show that the chain of thought boosts the performance of large language models like UniﬁedQA [19] if the models generate explanations along with the answer in a ﬁne-tuning way.
Revised: In this paper, we show that the chain of thought boosts the performance of large language models like UniﬁedQA [19] if the models generate explanations along with the answer in a ﬁne-tuning way.

Original: Furthermore, a few-shot GPT-3 model via chain-of-thought prompting is able to improve the reasoning performance on S CIENCE QA and generate reasonable explanations.
Revised: Furthermore, a few-shot GPT-3 model via chain-of-thought prompting is able to improve the reasoning performance on S CIENCE QA and generate reasonable explanations.

Original:  Dataset
Revised:  Dataset

Original: We collect S CIENCE QA, which is a multimodal multiple-choice science question dataset containing 21,208 examples.
Revised: We collect S CIENCE QA, which is a multimodal multiple-choice science question dataset containing 21,208 examples.

Original: An example in S CIENCE QA is shown in Figure 1.
Revised: An example in S CIENCE QA is shown in Figure 1.

Original: Given the science question and multimodal contexts, the task is to select the correct answer from multiple options.
Revised: Given the science question and multimodal contexts, the task is to select the correct answer from multiple options.

Original: Different from existing datasets [50, 17, 52, 31, 24], S CIENCE QA covers diverse topics across three subjects: natural science, social science, and language science.
Revised: Different from existing datasets [50, 17, 52, 31, 24], S CIENCE QA covers diverse topics across three subjects: natural science, social science, and language science.

Original: Moreover, most questions are annotated with grounded lectures and detailed explanations.
Revised: Moreover, most questions are annotated with grounded lectures and detailed explanations.

Original: The lecture provides general knowledge that introduces the background information for solving problems of a similar class.
Revised: The lecture provides general knowledge that introduces the background information for solving problems of a similar class.

Original: The explanation reveals a speciﬁc reason for the answer.
Revised: The explanation reveals a speciﬁc reason for the answer.

Original: To effectively answer the questions, a model often needs to be able to understand the multimodal content in the input and extract external knowledge, similar to how humans do.
Revised: To effectively answer the questions, a model often needs to be able to understand the multimodal content in the input and extract external knowledge, similar to how humans do.

Original: More importantly, the goal of S CIENCE QA is to aid development of a reliable model that is capable of generating a coherent chain of thought when arriving at the correct answer to reveal the multi-step reasoning process.
Revised: More importantly, the goal of S CIENCE QA is to aid development of a reliable model that is capable of generating a coherent chain of thought when arriving at the correct answer to reveal the multi-step reasoning process.

Original: For data collection details, see Appendix A.1.
Revised: For data collection details, see Appendix A.1.

Original: [Table]
Revised: [Table]

Original: f o ll o w i n g s t a t e s t h e t h e s e a t
Revised: f o ll o w i n g s t a t e s t h e t h e s e a t

Original: h e s e d o s p ee c
Revised: h e s e d o s p ee c

Original: h
Revised: h

Original: o f
Revised: o f

Original: y o u w o u l d h
Revised: y o u w o u l d h

Original: i g h li g h t
Revised: i g h li g h t

Original: e d i s i s f a ll
Revised: e d i s i s f a ll

Original: a c
Revised: a c

Original: y
Revised: y

Original: a h a s t
Revised: a h a s t

Original: h e d o e s t
Revised: h e d o e s t

Original: h e u s e s h
Revised: h e u s e s h

Original: i g h li g h t
Revised: i g h li g h t

Original: e d i s t
Revised: e d i s t

Original: h e s h
Revised: h e s h

Original: o w s h a s o f
Revised: o w s h a s o f

Original: o f
Revised: o f

Original: i s p
Revised: i s p

Original: r
Revised: r

Original: o p e r t y f
Revised: o p e r t y f

Original: i g u r e w o r d c o un t r y l
Revised: i g u r e w o r d c o un t r y l

Original: o g i c a l s o l u t i o n a n i m a l '
Revised: o g i c a l s o l u t i o n a n i m a l '

Original: s s e n t e n c e t e n s e t e x t c
Revised: s s e n t e n c e t e n s e t e x t c

Original: o n t i n e n t c o rr e c t l y o b
Revised: o n t i n e n t c o rr e c t l y o b

Original: j e c t t y
Revised: j e c t t y

Original: p e s t a t e m e n t c
Revised: p e s t a t e m e n t c

Original: l o s i n g c a p i t a l n a m e s o u r c e t
Revised: l o s i n g c a p i t a l n a m e s o u r c e t

Original: h e t h e s u pp o r t s t w o t
Revised: h e t h e s u pp o r t s t w o t

Original: h e s e t
Revised: h e s e t

Original: h e s e n t e n c
Revised: h e s e n t e n c

Original: e
Revised: e

Original: o f i s i n f o r m a t
Revised: o f i s i n f o r m a t

Original: i o n d o d o e s c a n k i n d i n o r g a n i s m o n e t h e t r a i t s t a t e m e n t f o ll o w i n g t h e t
Revised: i o n d o d o e s c a n k i n d i n o r g a n i s m o n e t h e t r a i t s t a t e m e n t f o ll o w i n g t h e t

Original: h e m a g n e t i c t
Revised: h e m a g n e t i c t

Original: h e a b o u t k
Revised: h e a b o u t k

Original: i n e t i c
Revised: i n e t i c

Original: o
Revised: o

Original: f a v e r a g e m o t i o n t h e a
Revised: f a v e r a g e m o t i o n t h e a

Original: i s l o n g d e s c
Revised: i s l o n g d e s c

Original: r
Revised: r

Original: i
Revised: i

Original: b
Revised: b

Original: e p a ss a g e t h i s t
Revised: e p a ss a g e t h i s t

Original: h e f i n d y o u
Revised: h e f i n d y o u

Original: W h
Revised: W h

Original: i c h
Revised: i c h

Original: W h a t S e l e c t
Revised: W h a t S e l e c t

Original: I s C
Revised: I s C

Original: o m
Revised: o m

Original: p l e t e T h
Revised: p l e t e T h

Original: i n k C
Revised: i n k C

Original: o m p a r e H
Revised: o m p a r e H

Original: o w D o e s
Revised: o w D o e s

Original: W o u l d
Revised: W o u l d

Original: 3.1 Data Analysis
Revised: 3.1 Data Analysis

Original: Key statistics.
Revised: Key statistics.

Original: We randomly split the dataset into training, validation, and test splits with a ratio of 60:20:20.
Revised: We randomly split the dataset into training, validation, and test splits with a ratio of 60:20:20.

Original: Each split has 12,726, 4,241, and 4,241 examples, respectively.
Revised: Each split has 12,726, 4,241, and 4,241 examples, respectively.

Original: Table 1 shows the main statistics of S CIENCE QA.
Revised: Table 1 shows the main statistics of S CIENCE QA.

Original: S CIENCE QA has a large set of different questions, totaling up to 9,122.
Revised: S CIENCE QA has a large set of different questions, totaling up to 9,122.

Original: Out of the 21,208 questions in S CIENCE QA, 10,332 (48.7%) have an image context, 10,220 (48.2%) have a text context, and 6,532 (30.8%) have both.
Revised: Out of the 21,208 questions in S CIENCE QA, 10,332 (48.7%) have an image context, 10,220 (48.2%) have a text context, and 6,532 (30.8%) have both.

Original: 83.9% of the questions are annotated with a lecture, while 91.3% of the questions feature an explanation.
Revised: 83.9% of the questions are annotated with a lecture, while 91.3% of the questions feature an explanation.

Original: The cross-combination of these information sources diversiﬁes the problem scenario: sometimes the model is given a lot of information from multiple sources, while at other times, the only source of information is the question itself.
Revised: The cross-combination of these information sources diversiﬁes the problem scenario: sometimes the model is given a lot of information from multiple sources, while at other times, the only source of information is the question itself.

Original: This level of complexity is very common in grade-level science exams.
Revised: This level of complexity is very common in grade-level science exams.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Question analysis.
Revised: Question analysis.

Original: S CIENCE QA has a diverse set of science questions.
Revised: S CIENCE QA has a diverse set of science questions.

Original: Figure 2 shows a distribution of the ﬁrst four words in the question text.
Revised: Figure 2 shows a distribution of the ﬁrst four words in the question text.

Original: A large number of question lengths and formats highlight the diversity of S CIENCE QA.
Revised: A large number of question lengths and formats highlight the diversity of S CIENCE QA.

Original: The question lengths range from 3 words to 141 words, and the questions in S CIENCE QA have an average length of 12.11 words.
Revised: The question lengths range from 3 words to 141 words, and the questions in S CIENCE QA have an average length of 12.11 words.

Original: The question length distribution is visualized against other VQA datasets in Figure 3 (a).
Revised: The question length distribution is visualized against other VQA datasets in Figure 3 (a).

Original: As shown in the diagram, S CIENCE QA’s distribution is ﬂatter than other datasets, spanning more evenly across different question lengths.
Revised: As shown in the diagram, S CIENCE QA’s distribution is ﬂatter than other datasets, spanning more evenly across different question lengths.

Original: Context analysis.
Revised: Context analysis.

Original: Figure 3 (b) shows the number and percentage of questions with either an image context, a text context, or both.
Revised: Figure 3 (b) shows the number and percentage of questions with either an image context, a text context, or both.

Original: There are a total of 7,803 unique image contexts and 4,651 unique text
Revised: There are a total of 7,803 unique image contexts and 4,651 unique text

Original: [Table]
Revised: [Table]

Original: contexts.
Revised: contexts.

Original: 66.11% of the questions have at least one type of context information.
Revised: 66.11% of the questions have at least one type of context information.

Original: The image context is in the format of diagrams or natural images, which visualize the critical scenario necessary for question answering or simply illustrate the question for better understanding.
Revised: The image context is in the format of diagrams or natural images, which visualize the critical scenario necessary for question answering or simply illustrate the question for better understanding.

Original: Similarly, the textual context can provide either semantically rich information or a simple hint to the question.
Revised: Similarly, the textual context can provide either semantically rich information or a simple hint to the question.

Original: Therefore, models need to be ﬂexible and general to understand these diverse types of contexts.
Revised: Therefore, models need to be ﬂexible and general to understand these diverse types of contexts.

Original: Domain diversity.
Revised: Domain diversity.

Original: Each S CIENCE QA question belongs to one of the three subjects: natural science, language science, and social science.
Revised: Each S CIENCE QA question belongs to one of the three subjects: natural science, language science, and social science.

Original: With each subject, questions are categorized ﬁrst by the topic ( Biology , Physics , Chemistry , etc.), then by the category ( Plants , Cells , Animals , etc.), and ﬁnally by the speciﬁc skill ( Classify fruits and vegetables as plant parts , Identify countries of Africa , etc.).
Revised: With each subject, questions are categorized ﬁrst by the topic ( Biology , Physics , Chemistry , etc.), then by the category ( Plants , Cells , Animals , etc.), and ﬁnally by the speciﬁc skill ( Classify fruits and vegetables as plant parts , Identify countries of Africa , etc.).

Original: S CIENCE QA has a total of 26 topics, 127 categories, and 379 skills.
Revised: S CIENCE QA has a total of 26 topics, 127 categories, and 379 skills.

Original: The treemap in Figurevisualizes the different subjects, topics, and categories and shows that S CIENCE QA questions are very diverse, spanning a wide range of domains.
Revised: The treemap in Figurevisualizes the different subjects, topics, and categories and shows that S CIENCE QA questions are very diverse, spanning a wide range of domains.

Original: 3.2 Comparisons with Existing Datasets
Revised: 3.2 Comparisons with Existing Datasets

Original: Table 2 shows a comparison of S CIENCE QA and other science problem datasets.
Revised: Table 2 shows a comparison of S CIENCE QA and other science problem datasets.

Original: As shown in the table, S CIENCE QA is much larger than most other datasets.
Revised: As shown in the table, S CIENCE QA is much larger than most other datasets.

Original: S CIENCE QA also has the largest set of images, spans across all 12 grades, contains the longest questions, and has the most diverse input sources.
Revised: S CIENCE QA also has the largest set of images, spans across all 12 grades, contains the longest questions, and has the most diverse input sources.

Original: As opposed to limiting the subject to only natural science, S CIENCE QA also includes social science and language science, largely adding to the domain diversity of the dataset.
Revised: As opposed to limiting the subject to only natural science, S CIENCE QA also includes social science and language science, largely adding to the domain diversity of the dataset.

Original: Furthermore, most of the questions in S CIENCE QA are annotated with textual lectures (83.9%) and explanations (90.5%), which reveal the reasoning path to the correct answer.
Revised: Furthermore, most of the questions in S CIENCE QA are annotated with textual lectures (83.9%) and explanations (90.5%), which reveal the reasoning path to the correct answer.

Original: To the best of our knowledge,
Revised: To the best of our knowledge,

Original: S CIENCE QA is the ﬁrst large-scale multimodal science question dataset that annotates the answers with detailed lectures and explanations
Revised: S CIENCE QA is the ﬁrst large-scale multimodal science question dataset that annotates the answers with detailed lectures and explanations

Original: [Table]
Revised: [Table]

Original:  Baselines and Chain-of-Thought Models
Revised:  Baselines and Chain-of-Thought Models

Original: In this section, we establish baselines and develop two chain-of-thought models on S CIENCE QA.
Revised: In this section, we establish baselines and develop two chain-of-thought models on S CIENCE QA.

Original: 4.1 Baselines
Revised: 4.1 Baselines

Original: Heuristic baselines.
Revised: Heuristic baselines.

Original: The ﬁrst heuristic baseline is random chance : we randomly select one from the multiple options.
Revised: The ﬁrst heuristic baseline is random chance : we randomly select one from the multiple options.

Original: Each trial is completed on the whole test set, and we take three different trials for an average result.
Revised: Each trial is completed on the whole test set, and we take three different trials for an average result.

Original: The second heuristic baseline is human performance .
Revised: The second heuristic baseline is human performance .

Original: We post the task to Amazon
Revised: We post the task to Amazon

Original: Mechanical Turk and ask workers to answer S CIENCE QA questions.
Revised: Mechanical Turk and ask workers to answer S CIENCE QA questions.

Original: Only workers who obtain a high school or higher degree and pass the qualiﬁcation examples are qualiﬁed for the study.
Revised: Only workers who obtain a high school or higher degree and pass the qualiﬁcation examples are qualiﬁed for the study.

Original: Each worker needs to answer a set of 10 test questions, and each question is answered by three different workers.
Revised: Each worker needs to answer a set of 10 test questions, and each question is answered by three different workers.

Original: For more details of the human performance study, see Appendix B.2.
Revised: For more details of the human performance study, see Appendix B.2.

Original: Zero-shot and few-shot baselines.
Revised: Zero-shot and few-shot baselines.

Original: We establish the zero-shot baselines on top of UniﬁedQA [19] and GPT-3 [5].
Revised: We establish the zero-shot baselines on top of UniﬁedQA [19] and GPT-3 [5].

Original: The zero-shot setup follows the format of QCM
Revised: The zero-shot setup follows the format of QCM

Original: → A where the input is the concatenation of tokens of the question text (Q), the context text (C), and multiple options (M), while the output is to predict the answer (A) from the option set.
Revised: → A where the input is the concatenation of tokens of the question text (Q), the context text (C), and multiple options (M), while the output is to predict the answer (A) from the option set.

Original: We extract the caption from the captioning model based on ViT [8] and GPT-2 [47] for the image as the visual context.
Revised: We extract the caption from the captioning model based on ViT [8] and GPT-2 [47] for the image as the visual context.

Original: In the few-shot setting, we follow the standard prompting [4] where in-context examples from the training set are concatenated before the test instance.
Revised: In the few-shot setting, we follow the standard prompting [4] where in-context examples from the training set are concatenated before the test instance.

Original: These in-context examples serve as an instruction for the language model to adjust to the speciﬁc task in S CIENCE QA.
Revised: These in-context examples serve as an instruction for the language model to adjust to the speciﬁc task in S CIENCE QA.

Original: Fine-tuning baselines.
Revised: Fine-tuning baselines.

Original: We ﬁrst consider the ﬁne-tuning baselines from VQA models [1, 21, 55, 9, 22, 35, 26] proposed in recent years.
Revised: We ﬁrst consider the ﬁne-tuning baselines from VQA models [1, 21, 55, 9, 22, 35, 26] proposed in recent years.

Original: These VQA baselines take the question, the context, and choices as the textual input, take the image as the visual input, and predict the score distribution over choice candidates via a linear classiﬁer.
Revised: These VQA baselines take the question, the context, and choices as the textual input, take the image as the visual input, and predict the score distribution over choice candidates via a linear classiﬁer.

Original: In addition, we build the ﬁne-tuning baseline on top of the large language model UniﬁedQA [19].
Revised: In addition, we build the ﬁne-tuning baseline on top of the large language model UniﬁedQA [19].

Original: UniﬁedQA takes the textual information as the input and outputs the answer option.
Revised: UniﬁedQA takes the textual information as the input and outputs the answer option.

Original: Similarly, the image is converted into a caption that provides the visual semantics for the language model.
Revised: Similarly, the image is converted into a caption that provides the visual semantics for the language model.

Original: 4.2 Language Models with the Chain of Thought
Revised: 4.2 Language Models with the Chain of Thought

Original: A chain of thought refers to a coherent ﬂow of sentences that reveals the premises and conclusion of a reasoning problem [54].
Revised: A chain of thought refers to a coherent ﬂow of sentences that reveals the premises and conclusion of a reasoning problem [54].

Original: A chain of thought clearly decomposes a multi-hop reasoning task into intermediate steps instead of solving the task in a black-box way.
Revised: A chain of thought clearly decomposes a multi-hop reasoning task into intermediate steps instead of solving the task in a black-box way.

Original: The chain of thought can be the step-by-step thought process [54] before arriving at the ﬁnal answer or explanations [41] that come after the answer.
Revised: The chain of thought can be the step-by-step thought process [54] before arriving at the ﬁnal answer or explanations [41] that come after the answer.

Original: The annotated lectures and explanations in S CIENCE QA serve as demonstrations of the chain of thought that mimics the multi-step reasoning steps of human beings.
Revised: The annotated lectures and explanations in S CIENCE QA serve as demonstrations of the chain of thought that mimics the multi-step reasoning steps of human beings.

Original: In this paper, we study if large language models can generate reasonable explanations as the chain of thought to reveal the thought process when answering S CIENCE QA questions.
Revised: In this paper, we study if large language models can generate reasonable explanations as the chain of thought to reveal the thought process when answering S CIENCE QA questions.

Original: Further, we explore how the chain of thought can improve the reasoning ability of language models on S CIENCE QA in both few-shot and ﬁne-tuning learning.
Revised: Further, we explore how the chain of thought can improve the reasoning ability of language models on S CIENCE QA in both few-shot and ﬁne-tuning learning.

Original: UniﬁedQA with the chain of thought.
Revised: UniﬁedQA with the chain of thought.

Original: UniﬁedQA [19] is a state of the art model for multi-option question answering.
Revised: UniﬁedQA [19] is a state of the art model for multi-option question answering.

Original: The original architecture of UniﬁedQA takes the question and options as the input and outputs a short phrase as the ﬁnal answer.
Revised: The original architecture of UniﬁedQA takes the question and options as the input and outputs a short phrase as the ﬁnal answer.

Original: We make a format modiﬁcation to develop
Revised: We make a format modiﬁcation to develop

Original: UniﬁedQA with the chain of thought (CoT), i.e., UniﬁedQA is ﬁne-tuned to generate a long sequence of text which consists of the answer followed by the lecture and explanation.
Revised: UniﬁedQA with the chain of thought (CoT), i.e., UniﬁedQA is ﬁne-tuned to generate a long sequence of text which consists of the answer followed by the lecture and explanation.

Original: GPT-3 via chain-of-thought prompting.
Revised: GPT-3 via chain-of-thought prompting.

Original: Recent research work [5, 38, 34] has shown that GPT3 [5] can perform various tasks when provided in-context examples in a standard prompt.
Revised: Recent research work [5, 38, 34] has shown that GPT3 [5] can perform various tasks when provided in-context examples in a standard prompt.

Original: Take multi-option question answering as an example, the standard prompt [36, 57, 29] builds instructions using in-context examples with components of the question text, options, and the correct answer text.
Revised: Take multi-option question answering as an example, the standard prompt [36, 57, 29] builds instructions using in-context examples with components of the question text, options, and the correct answer text.

Original: This style of few-shot learning enables the GPT-3 model to answer speciﬁc questions without parameter updates.
Revised: This style of few-shot learning enables the GPT-3 model to answer speciﬁc questions without parameter updates.

Original: Different from standard prompting, we build GPT-3 via chain-of-thought (CoT) prompting, as shown in Figure 5.
Revised: Different from standard prompting, we build GPT-3 via chain-of-thought (CoT) prompting, as shown in Figure 5.

Original: To be speciﬁc, for each test problem t , we map the prompt instruction I : { I i } n , I t into a textual format where { I i } n refers to the instruction set of n -shot in-context examples from the training set, while I t denotes the test instruction.
Revised: To be speciﬁc, for each test problem t , we map the prompt instruction I : { I i } n , I t into a textual format where { I i } n refers to the instruction set of n -shot in-context examples from the training set, while I t denotes the test instruction.

Original: Instead of the way where the explanation comes before the answer [54], we feed the instruction I into the encoder-
Revised: Instead of the way where the explanation comes before the answer [54], we feed the instruction I into the encoder-

Original: [Figure]
Revised: [Figure]

Original: decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp :
Revised: decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp :

Original: M : { I i } n , I t
Revised: M : { I i } n , I t

Original: → a, lect, exp .
Revised: → a, lect, exp .

Original:  Experiments 5.1 Experimental Setup
Revised:  Experiments 5.1 Experimental Setup

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: The heuristics and VQA baselines treat our S CIENCE QA task as a multi-class classiﬁcation problem with multiple options and are evaluated with the accuracy metrics.
Revised: The heuristics and VQA baselines treat our S CIENCE QA task as a multi-class classiﬁcation problem with multiple options and are evaluated with the accuracy metrics.

Original: UniﬁedQA and GPT-3 treat S CIENCE QA as a text generation problem.
Revised: UniﬁedQA and GPT-3 treat S CIENCE QA as a text generation problem.

Original: So the most similar option is selected as the ﬁnal prediction to evaluate the question answering accuracy.
Revised: So the most similar option is selected as the ﬁnal prediction to evaluate the question answering accuracy.

Original: The generated lectures and explanations are evaluated by automatic metrics [44, 28, 49] and human scores by annotators.
Revised: The generated lectures and explanations are evaluated by automatic metrics [44, 28, 49] and human scores by annotators.

Original: Implementation details.
Revised: Implementation details.

Original: The VQA baselines are trained for a maximum number of 50 epochs with a learning rate of 5 e − 5 .
Revised: The VQA baselines are trained for a maximum number of 50 epochs with a learning rate of 5 e − 5 .

Original: We ﬁne-tune the UniﬁedQA for 50 k iterations and evaluate every 1 k iteration.
Revised: We ﬁne-tune the UniﬁedQA for 50 k iterations and evaluate every 1 k iteration.

Original: The training process is stopped following the early stopping strategy with a patience period of three evaluations.
Revised: The training process is stopped following the early stopping strategy with a patience period of three evaluations.

Original: For GPT-3, we use the text-davinci-002 engine, which is the most capable model version suggested in the ofﬁcial documentation.
Revised: For GPT-3, we use the text-davinci-002 engine, which is the most capable model version suggested in the ofﬁcial documentation.

Original: More details can be found in Appendix B.1.
Revised: More details can be found in Appendix B.1.

Original: 5.2 Results for Question Answering
Revised: 5.2 Results for Question Answering

Original: Table 3 demonstrates the empirical results for Science Question Answering.
Revised: Table 3 demonstrates the empirical results for Science Question Answering.

Original: VQA baselines.
Revised: VQA baselines.

Original: We feed the VQA baseline models with the input of QCM format to predict answers
Revised: We feed the VQA baseline models with the input of QCM format to predict answers

Original: A. Out of all the VQA models we benchmarked, VisualBERT [26, 27] performs the best on average (61.87%).
Revised: A. Out of all the VQA models we benchmarked, VisualBERT [26, 27] performs the best on average (61.87%).

Original: Interestingly, Patch-TRM [35] beats VisualBERT in natural science (NAT) and language science (LAN), and it also performs better in higher-grade questions (67.50% v.s. 59.92%).
Revised: Interestingly, Patch-TRM [35] beats VisualBERT in natural science (NAT) and language science (LAN), and it also performs better in higher-grade questions (67.50% v.s. 59.92%).

Original: However, in the subject of social science (SOC), VisualBERT outperforms Patch-TRM by a large margin (+22.39%).
Revised: However, in the subject of social science (SOC), VisualBERT outperforms Patch-TRM by a large margin (+22.39%).

Original: Such drastic changes in performance might imply that current VQA models are not generalized to process the challenging questions in S CIENCE QA.
Revised: Such drastic changes in performance might imply that current VQA models are not generalized to process the challenging questions in S CIENCE QA.

Original: Language models.
Revised: Language models.

Original: We evaluate whether large-scale pretraining on text can help language models learn scientiﬁc knowledge and thus perform better on the S CIENCE QA task.
Revised: We evaluate whether large-scale pretraining on text can help language models learn scientiﬁc knowledge and thus perform better on the S CIENCE QA task.

Original: For this purpose, we have tried two of the state-of-the-art pre-trained language models: UniﬁedQA and GPT-3.
Revised: For this purpose, we have tried two of the state-of-the-art pre-trained language models: UniﬁedQA and GPT-3.

Original: i)
Revised: i)

Original: UniﬁedQA
Revised: UniﬁedQA

Original: The results show that without any supervised ﬁne-tuning (zero-shot), UniﬁedQA cannot beat any VQA baseline model, while the pretraining does help the model obtain some scientiﬁc knowledge to outperform the random baseline.
Revised: The results show that without any supervised ﬁne-tuning (zero-shot), UniﬁedQA cannot beat any VQA baseline model, while the pretraining does help the model obtain some scientiﬁc knowledge to outperform the random baseline.

Original: When ﬁne-tuned with the answer labels in
Revised: When ﬁne-tuned with the answer labels in

Original: S CIENCE QA, UniﬁedQA BASE reports an accuracy of 70.12% on average.
Revised: S CIENCE QA, UniﬁedQA BASE reports an accuracy of 70.12% on average.

Original: By further teaching the model to generate the answer along with lecture and explanation, the developed language model with chain-of-thought (UniﬁedQA BASE (CoT)) brings additional improvements of +3.21% (QCM → AE) and +3.99% (QCM → ALE).
Revised: By further teaching the model to generate the answer along with lecture and explanation, the developed language model with chain-of-thought (UniﬁedQA BASE (CoT)) brings additional improvements of +3.21% (QCM → AE) and +3.99% (QCM → ALE).

Original: These results show that generating the chain of thought along with the answer beneﬁts the reasoning ability of language models.
Revised: These results show that generating the chain of thought along with the answer beneﬁts the reasoning ability of language models.

Original: (ii) GPT-
Revised: (ii) GPT-

Original: The positive effect
Revised: The positive effect

Original: of pretraining
Revised: of pretraining

Original: is also proved by the surprisingly good results from
Revised: is also proved by the surprisingly good results from

Original: GPT-
Revised: GPT-

Original: in the
Revised: in the

Original: same
Revised: same

Original: zero
Revised: zero

Original: -shot
Revised: -shot

Original: setting as UniﬁedQA
Revised: setting as UniﬁedQA

Original: Without any
Revised: Without any

Original: ﬁne
Revised: ﬁne

Original: -tuning,
Revised: -tuning,

Original: GPT-
Revised: GPT-

Original: 3 already reaches almost the best performance we can get.
Revised: 3 already reaches almost the best performance we can get.

Original: Interestingly, prompting the GPT-3 with two training examples with only answers results in a negligible difference.
Revised: Interestingly, prompting the GPT-3 with two training examples with only answers results in a negligible difference.

Original: However, if we prompt GPT-3 with chain of-thought prompting (QCM → ALE), we obtain the state-of-the-art result so far (75.17%).
Revised: However, if we prompt GPT-3 with chain of-thought prompting (QCM → ALE), we obtain the state-of-the-art result so far (75.17%).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: -3 (CoT).
Revised: -3 (CoT).

Original: Human performance.
Revised: Human performance.

Original: Humans outperform all benchmarks consistently across question classes, context types, and grades, e.g., a 20.07% gap for questions with the image context (IMG) between humans and our best performing model.
Revised: Humans outperform all benchmarks consistently across question classes, context types, and grades, e.g., a 20.07% gap for questions with the image context (IMG) between humans and our best performing model.

Original: The gap is to be ﬁlled by future research on multimodal reasoning for scientiﬁc question answering.
Revised: The gap is to be ﬁlled by future research on multimodal reasoning for scientiﬁc question answering.

Original: 5.3 Results for Generated Explanations
Revised: 5.3 Results for Generated Explanations

Original: One prediction example of GPT-3 (CoT) is visualized in Figure 6.
Revised: One prediction example of GPT-3 (CoT) is visualized in Figure 6.

Original: We can see that GPT-3 (CoT) predicts the correct answer and generates a reasonable lecture and explanation to mimic the human thought process.
Revised: We can see that GPT-3 (CoT) predicts the correct answer and generates a reasonable lecture and explanation to mimic the human thought process.

Original: We further report automatic metrics (BLEU-1/4 [44], ROUGE-L [44], and (sentence)
Revised: We further report automatic metrics (BLEU-1/4 [44], ROUGE-L [44], and (sentence)

Original: Similarity [49] to evaluate the generated lectures and explanations, as shown in Table 4.
Revised: Similarity [49] to evaluate the generated lectures and explanations, as shown in Table 4.

Original: The Similarity metric computes the cosine-similarity of semantic embeddings between two sentences based on the Sentence-BERT network [49].
Revised: The Similarity metric computes the cosine-similarity of semantic embeddings between two sentences based on the Sentence-BERT network [49].

Original: The results show that UniﬁedQA BASE (CoT) generates the most similar explanations to the given ones.
Revised: The results show that UniﬁedQA BASE (CoT) generates the most similar explanations to the given ones.

Original: However, it’s commonly agreed that automatic evaluation of generated texts only provides a partial view and has to be complemented by a human study.
Revised: However, it’s commonly agreed that automatic evaluation of generated texts only provides a partial view and has to be complemented by a human study.

Original: By asking annotators to rate the relevance, correctness, and completeness of generated explanations, we ﬁnd that the explanations generated by GPT-3 (CoT) conform best to human judgment.
Revised: By asking annotators to rate the relevance, correctness, and completeness of generated explanations, we ﬁnd that the explanations generated by GPT-3 (CoT) conform best to human judgment.

Original: [Table]
Revised: [Table]

Original: 5.4 Analysis
Revised: 5.4 Analysis

Original: Blind studies.
Revised: Blind studies.

Original: Blind studies are conducted on top of the modiﬁcation of the full model, Top-Down [1].
Revised: Blind studies are conducted on top of the modiﬁcation of the full model, Top-Down [1].

Original: The results achieved in blind studies of Q only and C I only are close to random chance, showing that the S CIENCE QA dataset is robust and reliable in distribution.
Revised: The results achieved in blind studies of Q only and C I only are close to random chance, showing that the S CIENCE QA dataset is robust and reliable in distribution.

Original: The performance drops in Q+M only,
Revised: The performance drops in Q+M only,

Original: Q+C T +M only, and Q+C I +M only indicate that all input components provide critical information for answering S CIENCE QA questions.
Revised: Q+C T +M only, and Q+C I +M only indicate that all input components provide critical information for answering S CIENCE QA questions.

Original: Prompt types.
Revised: Prompt types.

Original: We study the effect of prompt types and visualize the comparison in Figure 7 (a).
Revised: We study the effect of prompt types and visualize the comparison in Figure 7 (a).

Original: It shows that prompting the GPT-3 model with both lectures and explanations (QCM → ALE) results in the highest accuracy on average and the smallest variance.
Revised: It shows that prompting the GPT-3 model with both lectures and explanations (QCM → ALE) results in the highest accuracy on average and the smallest variance.

Original: In contrast, prompting with only explanations (QCM → AE) gives the largest variance, resulting in a less stable model.
Revised: In contrast, prompting with only explanations (QCM → AE) gives the largest variance, resulting in a less stable model.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Number of in-context examples.
Revised: Number of in-context examples.

Original: In Figure 7 (b), we further investigate how different numbers of training examples encoded in prompts can affect the prediction accuracy.
Revised: In Figure 7 (b), we further investigate how different numbers of training examples encoded in prompts can affect the prediction accuracy.

Original: The QCM → ALE prompt type outperforms or performs comparably the QCM
Revised: The QCM → ALE prompt type outperforms or performs comparably the QCM

Original: → A type with all numbers of examples.
Revised: → A type with all numbers of examples.

Original: And we observe the peak performance of QCM → ALE with 2 training examples being prompted.
Revised: And we observe the peak performance of QCM → ALE with 2 training examples being prompted.

Original: After that, the accuracy goes down as more training examples are added to the model.
Revised: After that, the accuracy goes down as more training examples are added to the model.

Original: [Table]
Revised: [Table]

Original: Dynamic sampling.
Revised: Dynamic sampling.

Original: In Table 5, instead of random sampling, we try to dynamically select the in-context examples to prompt with the same class as the test sample.
Revised: In Table 5, instead of random sampling, we try to dynamically select the in-context examples to prompt with the same class as the test sample.

Original: However, slight differences in prediction accuracy are observed when comparing them to simple random sampling.
Revised: However, slight differences in prediction accuracy are observed when comparing them to simple random sampling.

Original: Upper bound.
Revised: Upper bound.

Original: We search the upper bound of the GPT-3 accuracy by feeding the gold lecture and explanation in the test prompt.
Revised: We search the upper bound of the GPT-3 accuracy by feeding the gold lecture and explanation in the test prompt.

Original: As reported in Table 6, QCME* → A outperforms the QCM → ALE baseline by 18.86% and QCMLE* → A outperforms QCM → ALE by 18.96%, indicating a potential improvement direction by generating correct explanations before answering science questions.
Revised: As reported in Table 6, QCME* → A outperforms the QCM → ALE baseline by 18.86% and QCMLE* → A outperforms QCM → ALE by 18.96%, indicating a potential improvement direction by generating correct explanations before answering science questions.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Positions of lectures and explanations.
Revised: Positions of lectures and explanations.

Original: We study the performance of GPT-3 (CoT) in terms of different positions of lectures and explanations on 1,000 test examples.
Revised: We study the performance of GPT-3 (CoT) in terms of different positions of lectures and explanations on 1,000 test examples.

Original: Results are shown in Table 7.
Revised: The results are shown in Table 7.

Original: There could be huge accuracy decreases if GPT-3 (CoT) predicts lectures and explanations before answers.
Revised: There could be huge accuracy decreases if GPT-3 (CoT) predicts lectures and explanations before answers.

Original: It is mainly because if GPT-3 (CoT) is formalized to generate the long lecture and explanation ﬁrst, there is a larger chance that it stops generating the prediction early or use up the maximum token limits before obtaining the required answer.
Revised: It is mainly because if GPT-3 (CoT) is formalized to generate the long lecture and explanation ﬁrst, there is a greater chance that it will stop generating the prediction early or use up the maximum token limits before obtaining the required answer.

Original: [Figure]
Revised: [Figure]

Original: CoT learns with fewer data.
Revised: CoT learns with fewer data.

Original: To study if the chain of thought helps language models learn more efﬁciently, we report the accuracies of UniﬁedQA and UniﬁedQA (CoT) ﬁne-tuned on different sizes of the training set in Figure 8.
Revised: To study if the chain of thought helps language models learn more efﬁciently, we report the accuracies of UniﬁedQA and UniﬁedQA (CoT) ﬁne-tuned on different sizes of the training set in Figure 8.

Original: UniﬁedQA (CoT) beneﬁts language models by learning the coherent reasoning path when answering questions, resulting in similar accuracy with fewer training examples.
Revised: UniﬁedQA (CoT) beneﬁts language models by learning the coherent reasoning path when answering questions, resulting in similar accuracy with fewer training examples.

Original: Error analysis.
Revised: Error analysis.

Original: GPT-3 via chain-of-chain prompting obtains promising results but still fails to answer a wide range of challenging questions in S CIENCE QA.
Revised: GPT-3 via chain-of-chain prompting obtains promising results but still fails to answer a wide range of challenging questions in S CIENCE QA.

Original: See examples of failure cases in Appendix B.4.
Revised: See examples of failure cases in Appendix B.4.

Original: The failure cases can be classiﬁed into two types: (a) the model fails to understand the multimodal inputs and lacks domain-speciﬁc knowledge to arrive at the correct answer; (b) the model generates the wrong chain of thought with irrelevant, incorrect, or incomplete information.
Revised: The failure cases can be classiﬁed into two types: (a) the model fails to understand the multimodal inputs and lacks domain-speciﬁc knowledge to arrive at the correct answer; (b) the model generates the wrong chain of thought with irrelevant, incorrect, or incomplete information.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: In this paper, we propose S CIENCE QA, a dataset that features 21,208 multi-option questions with multimodal contexts from the science curriculum.
Revised: In this paper, we propose S CIENCE QA, a dataset that features 21,208 multi-option questions with multimodal contexts from the science curriculum.

Original: To the best of our knowledge, S CIENCE QA is the ﬁrst large-scale multimodal science dataset where most questions are annotated with corresponding lectures and explanations.
Revised: To the best of our knowledge, S CIENCE QA is the ﬁrst large-scale multimodal science dataset where most questions are annotated with corresponding lectures and explanations.

Original: We establish various baselines, including recent VQA models and large language models on S CIENCE QA.
Revised: We establish various baselines, including recent VQA models and large language models on S CIENCE QA.

Original: We further study if language models can generate reasonable explanations and then beneﬁt the reasoning ability.
Revised: We further study if language models can generate reasonable explanations and then beneﬁt the reasoning ability.

Original: Experiments show that UniﬁedQA with the chain of thought can achieve an improvement of 3.99% and few-shot GPT-3 via chain-of-thought (CoT) prompting can obtain a satisfactory accuracy of 75.17% on S CIENCE QA.
Revised: Experiments show that UniﬁedQA with the chain of thought can achieve an improvement of 3.99% and few-shot GPT-3 via chain-of-thought (CoT) prompting can obtain a satisfactory accuracy of 75.17% on S CIENCE QA.

Original: 65.2% of the generated explanations from GPT-3 (CoT) meet the gold standard by human evaluations.
Revised: 65.2% of the generated explanations from GPT-3 (CoT) meet the gold standard by human evaluations.

Original:  Acknowledgment
Revised:  Acknowledgment

Original: We would like to thank the anonymous reviewers for their valuable comments and suggestions.
Revised: We would like to thank the anonymous reviewers for their valuable comments and suggestions.

Original: We would also like to thank Xiaodan Liang for insightful discussions on dataset collection.
Revised: We would also like to thank Xiaodan Liang for insightful discussions on dataset collection.

Original: We thank our colleagues at The Allen Institute of AI (AI2), Jiasen Lu and Jungo Kasai for helpful discussions.
Revised: We thank our colleagues at The Allen Institute of AI (AI2), Jiasen Lu and Jungo Kasai for helpful discussions.

Original: The work does not relate to Liang Qiu’s position at Amazon Alexa.
Revised: The work does not relate to Liang Qiu’s position at Amazon Alexa.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Yes, we did the error analysis in Section 5.4 and discussed the limitations of the work in Appendix B.4.
Revised: [Yes] Yes, we did the error analysis in Section 5.4 and discussed the limitations of the work in Appendix B.4.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We discussed the broader impacts in Appendix B.5.
Revised: [Yes] We discussed the broader impacts in Appendix B.5.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We included 100 data examples and the data visualizer tool in the supplemental material.
Revised: [Yes] We included 100 data examples and the data visualizer tool in the supplemental material.

Original: The whole dataset and code will be available at https://scienceqa.github.io .
Revised: The whole dataset and code will be available at https://scienceqa.github.io .

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5.1 and Appendix B.1 for experimental details.
Revised: [Yes] See Section 5.1 and Appendix B.1 for experimental details.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We reported the error bars for GPT-3 (CoT) experiments in Figure 7, where each experiment was repeated four times.
Revised: [Yes] We reported the error bars for GPT-3 (CoT) experiments in Figure 7, where each experiment was repeated four times.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We discussed compute resources in Appendix B.1.
Revised: [Yes] We discussed compute resources in Appendix B.1.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We collected the S CIENCE QA dataset from https://www.ixl.com/ .
Revised: [Yes] We collected the S CIENCE QA dataset from https://www.ixl.com/ .

Original: The copyright belongs to IXL.
Revised: The copyright belongs to IXL.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] S CIENCE QA is under the CC BY-NCSA 4.0 license and is used for non-commercial research purposes.
Revised: [Yes] S CIENCE QA is under the CC BY-NCSA 4.0 license and is used for non-commercial research purposes.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We included data examples and a visualizer tool in the supplemental material.
Revised: [Yes] We included data examples and a visualizer tool in the supplemental material.

Original: The dataset will be available at https://scienceqa.github.io .
Revised: The dataset will be available at https://scienceqa.github.io .

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [Yes] The collected data does not contain personally identiﬁable information or offensive content.
Revised: [Yes] The collected data does not contain personally identiﬁable information or offensive content.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [Yes] We included screenshots of the instructions in Appendix B.2 and B.3.
Revised: [Yes] We included screenshots of the instructions in Appendix B.2 and B.3.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [Yes] We included the monetary compensation details in Appendix B.2 and B.3.
Revised: [Yes] We included the monetary compensation details in Appendix B.2 and B.3.

Original: A Dataset Analysis A.1 Data Collection
Revised: A Dataset Analysis A.1 Data Collection

Original: Questions in the S CIENCE QA dataset are sourced from open resources managed by IXL Learning, an online learning platform curated by experts in the ﬁeld of K-12 education.
Revised: Questions in the S CIENCE QA dataset are sourced from open resources managed by IXL Learning, an online learning platform curated by experts in the ﬁeld of K-12 education.

Original: The dataset includes problems that align with California Common Core Content Standards .
Revised: The dataset includes problems that align with California Common Core Content Standards .

Original: To construct S CIENCE QA, we downloaded the original science problems and then extracted individual components (e.g. questions, hints, images, options, answers, lectures, and solutions) from them based on heuristic rules.
Revised: To construct S CIENCE QA, we downloaded the original science problems and then extracted individual components (e.g. questions, hints, images, options, answers, lectures, and solutions) from them based on heuristic rules.

Original: We manually removed invalid questions, such as questions that have only one choice, questions that contain faulty data, and questions that are duplicated, to comply with fair use and transformative use of the law.
Revised: We manually removed invalid questions, such as questions that have only one choice, questions that contain faulty data, and questions that are duplicated, to comply with fair use and transformative use of the law.

Original: If there were multiple correct answers that applied, we kept only one correct answer.
Revised: If there were multiple correct answers that applied, we kept only one correct answer.

Original: Also, we shufﬂed the answer options of each question to ensure the choices do not follow any speciﬁc pattern.
Revised: Also, we shufﬂed the answer options of each question to ensure the choices do not follow any speciﬁc pattern.

Original: To make the dataset easy to use, we then used semi-automated scripts to reformat the lectures and solutions.
Revised: To make the dataset easy to use, we then used semi-automated scripts to reformat the lectures and solutions.

Original: Therefore, special structures in the texts, such as tables and lists, are easily distinguishable from simple text passages.
Revised: Therefore, special structures in the texts, such as tables and lists, are easily distinguishable from simple text passages.

Original: Similar to ImageNet, ReClor, and PMR datasets,
Revised: Similar to ImageNet, ReClor, and PMR datasets,

Original: S CIENCE QA is available for non-commercial research purposes only and the copyright belongs to the original authors.
Revised: S CIENCE QA is available for non-commercial research purposes only and the copyright belongs to the original authors.

Original: To ensure data quality, we developed a data exploration tool to review examples in the collected dataset, and incorrect annotations were further manually revised by experts.
Revised: To ensure data quality, we developed a data exploration tool to review examples in the collected dataset, and incorrect annotations were further manually revised by experts.

Original: The tool can be accessed at https://scienceqa.github.io/explore.html .
Revised: The tool can be accessed at https://scienceqa.github.io/explore.html .

Original: A.2 Question Statistics
Revised: A.2 Question Statistics

Original: Figure 9 (a) is a word cloud showing the most frequently appeared words in the question texts.
Revised: Figure 9 (a) is a word cloud showing the most frequently appeared words in the question texts.

Original: Stopping words that do not contain any semantic meaning, such as “ what ” or “ and ”, are removed to give us a clearer view of the semantic range of S CIENCE QA.
Revised: Stopping words that do not contain any semantic meaning, such as “ what ” or “ and ”, are removed to give us a clearer view of the semantic range of S CIENCE QA.

Original: The diagram shows that S CIENCE QA covers a wide range of topics, with words from different topics showing up across the cloud.
Revised: The diagram shows that S CIENCE QA covers a wide range of topics, with words from different topics showing up across the cloud.

Original: Figures 9 (b) (c) (d) show the word clouds for each of the three subjects.
Revised: Figures 9 (b) (c) (d) show the word clouds for each of the three subjects.

Original: We can observe from the word clouds that the words are well-matched to the subject themes.
Revised: We can observe from the word clouds that the words are well-matched to the subject themes.

Original: In natural science questions, words such as “ trait ”, “ magnet ”, and “ force ” appear frequently.
Revised: In natural science questions, words such as “ trait ”, “ magnet ”, and “ force ” appear frequently.

Original: Words such as “ capital ” and “state” show up frequently in social science questions, whereas words such as “ dictionary ” and “ page ” are common in language science questions.
Revised: Words such as “ capital ” and “state” show up frequently in social science questions, whereas words such as “ dictionary ” and “ page ” are common in language science questions.

Original: [Figure]
Revised: [Figure]

Original: A.3 Choice Statistics
Revised: A.3 Choice Statistics

Original: [Table]
Revised: [Table]

Original: Table 8 shows the number of questions with each number of different choices.
Revised: Table 8 shows the number of questions with each number of different choices.

Original: Questions have a minimum of two options and a maximum of ﬁve options.
Revised: Questions have a minimum of two options and a maximum of ﬁve options.

Original: Figure 10 shows the distribution of choice length in S CIENCE QA.
Revised: Figure 10 shows the distribution of choice length in S CIENCE QA.

Original: Most choices are short, containing up to ﬁve words.
Revised: Most choices are short, containing up to ﬁve words.

Original: However, the distribution has a long tail where about 5% of the choices contain more thanwords.
Revised: However, the distribution has a long tail where about 5% of the choices contain more thanwords.

Original: Hence, it requires models to have a high level of text understanding to address diversely distributed choices.
Revised: Hence, it requires models to have a high level of text understanding to address diversely distributed choices.

Original: A.4 Subject Statistics
Revised: A.4 Subject Statistics

Original: Figure 11 shows the question length distribution of each subject.
Revised: Figure 11 shows the question length distribution of each subject.

Original: The three subjects all feature long-tail distributions in terms of the number of question words.
Revised: The three subjects all feature long-tail distributions in terms of the number of question words.

Original: On average, social science questions are the shortest, while language science questions are the longest.
Revised: On average, social science questions are the shortest, while language science questions are the longest.

Original: Language science questions are distributed more evenly than other questions across different numbers of words.
Revised: Language science questions are distributed more evenly than other questions across different numbers of words.

Original: These features imply that the S CIENCE QA dataset is rich in compositional diversity.
Revised: These features imply that the S CIENCE QA dataset is rich in compositional diversity.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A.5 Grade Statistics
Revised: A.5 Grade Statistics

Original: The grade distribution is shown in Figure 12.
Revised: The grade distribution is shown in Figure 12.

Original: The majority of questions come from the middle level curriculum (i
Revised: The majority of questions come from the middle level curriculum (i

Original: [Figure]
Revised: [Figure]

Original: e., from grade 3 to grade 8) while around 10% are taken from the high school curriculum (i.e., from grade 9 to grade 12).
Revised: e., from grade 3 to grade 8) while around 10% are taken from the high school curriculum (i.e., from grade 9 to grade 12).

Original: These high school level questions are close to or at the difﬁculty level of the U.S. standardized tests for college admissions.
Revised: These high school level questions are close to or at the difﬁculty level of the U.S. standardized tests for college admissions.

Original: Machine algorithms need to master a large amount of scientiﬁc knowledge and perform complex reasoning in order to perform well on S CIENCE
Revised: Machine algorithms need to master a large amount of scientiﬁc knowledge and perform complex reasoning in order to perform well on S CIENCE

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: B Experiments B.1 Experimental Details
Revised: B Experiments B.1 Experimental Details

Original: Below are details on the experiments:
Revised: Below are details on the experiments:

Original: • Fine-tuning on the dataset.
Revised: • Fine-tuning on the dataset.

Original: Fine-tuning baselines (VQA baselines and UniﬁedQA) are trained on the training set, developed on the validation set, and evaluated on the test set.
Revised: Fine-tuning baselines (VQA baselines and UniﬁedQA) are trained on the training set, developed on the validation set, and evaluated on the test set.

Original: • Input sizes: For VQA baselines, we set the maximum number of input words or tokens as 100.
Revised: • Input sizes: For VQA baselines, we set the maximum number of input words or tokens as 100.

Original: • Batch sizes.
Revised: • Batch sizes.

Original: We use batches of 64 and 4 for VQA baselines and ﬁne-tuned UniﬁedQA, respectively.
Revised: We use batches of 64 and 4 for VQA baselines and ﬁne-tuned UniﬁedQA, respectively.

Original: • Newline character.
Revised: • Newline character.

Original: For language models, the newline separators (\ n ) in the text are replaced with \\ n when encoding the inputs because \ n is normally used as a stop symbol, following the original works [5, 19].
Revised: For language models, the newline separators (\ n ) in the text are replaced with \\ n when encoding the inputs because \ n is normally used as a stop symbol, following the original works [5, 19].

Original: • Captioning model.
Revised: • Captioning model.

Original: We use the tool 2 to generate captions for the images in the dataset.
Revised: We use the tool 2 to generate captions for the images in the dataset.

Original: The maximum length of generated captions is 16, the number of beams is 4, and the maximum number of output tokens is 512.
Revised: The maximum length of generated captions is 16, the number of beams is 4, and the maximum number of output tokens is 512.

Original: • Compute resources.
Revised: • Compute resources.

Original: We use two GeForce RTX 3090 GPUs for ﬁne-tuning VQA baselines and UniﬁedQA on the dataset.
Revised: We use two GeForce RTX 3090 GPUs for ﬁne-tuning VQA baselines and UniﬁedQA on the dataset.

Original: • Questions without any context.
Revised: • Questions without any context.

Original: For questions without any context, the context text is replaced with an empty string.
Revised: For questions without any context, the context text is replaced with an empty string.

Original: • GPT-3:
Revised: • GPT-3:

Original: Following default settings, we choose temperature, frequency penalty and presence penalty as 0.0, and top probability as 1.0.
Revised: Following default settings, we choose temperature, frequency penalty and presence penalty as 0.0, and top probability as 1.0.

Original: All experiments for GPT-3 are run via the online API.
Revised: All experiments for GPT-3 are run via the online API.

Original: Experiments in Figure 7 are repeated four times with in-context examples listed in Table 9.
Revised: Experiments in Figure 7 are repeated four times with in-context examples listed in Table 9.

Original: Experiments in Table 3, 5, 6, and 7 are conducted using examples with the trial ID of 1.
Revised: Experiments in Table 3, 5, 6, and 7 are conducted using examples with the trial ID of 1.

Original: [Table]
Revised: [Table]

Original: B.2 Human Performance Study
Revised: B.2 Human Performance Study

Original: In order to understand how humans perform on S CIENCE QA questions, we used Amazon Mechanical
Revised: In order to understand how humans perform on S CIENCE QA questions, we used Amazon Mechanical

Original: Turk (AMT) to crowd source answers to the test set.
Revised: Turk (AMT) to crowd source answers to the test set.

Original: The interface of instructions and one example of a test question are shown in Figure 13.
Revised: The interface of instructions and one example of a test question are shown in Figure 13.

Original: A total of 4,241 test questions were shufﬂed and split into 425 batches, with each batch having 10 questions (excluding the last one).
Revised: A total of 4,241 test questions were shufﬂed and split into 425 batches, with each batch having 10 questions (excluding the last one).

Original: For each batch, we also randomly added ﬁve training questions as exam examples.
Revised: For each batch, we also randomly added ﬁve training questions as exam examples.

Original: Each set of 15 questions was then assigned to 3 AMT workers.
Revised: Each set of 15 questions was then assigned to 3 AMT workers.

Original: Only workers who correctly answer 4 out of the 5 exam examples or more are qualiﬁed for the human performance study.
Revised: Only workers who correctly answer 4 out of the 5 exam examples or more are qualiﬁed for the human performance study.

Original: In other words, workers who failed to pass the qualiﬁed exam were eliminated from the analysis.
Revised: In other words, workers who failed to pass the qualiﬁed exam were eliminated from the analysis.

Original: For each set of 15 questions, we provided the worker with $0.5 per HIT task.
Revised: For each set of 15 questions, we provided the worker with $0.5 per HIT task.

Original: At the rate of 3 questions per minute, this amounts to $6.0 per hour.
Revised: At the rate of 3 questions per minute, this amounts to $6.0 per hour.

Original: B.3 Human Evaluation of Generated Explanations
Revised: B.3 Human Evaluation of Generated Explanations

Original: We also evaluated the quality of predictions from GPT-3 (CoT) and UniﬁedQA (CoT) by asking
Revised: We also evaluated the quality of predictions from GPT-3 (CoT) and UniﬁedQA (CoT) by asking

Original: AMT workers to rate the model-generated explanations.
Revised: AMT workers to rate the model-generated explanations.

Original: The interface is shown in Figure 14.
Revised: The interface is shown in Figure 14.

Original: Each sample’s question text, contexts, choices, and answers were presented, along with the corresponding explanation generated by language models.
Revised: Each sample’s question text, contexts, choices, and answers were presented, along with the corresponding explanation generated by language models.

Original: The workers were asked to decide whether the proposed explanation is relevant (is related to the question), correct (gives a correct answer and explanation),
Revised: The workers were asked to decide whether the proposed explanation is relevant (is related to the question), correct (gives a correct answer and explanation),

Original: [Figure]
Revised: [Figure]

Original: and complete (fully explains the answer).
Revised: and complete (fully explains the answer).

Original: Prediction outputs that contain textual explanations were grouped into batches of 10, each assigned to 3 workers for evaluation.
Revised: Prediction outputs that contain textual explanations were grouped into batches of 10, each assigned to 3 workers for evaluation.

Original: For each batch, we provided the workers with a monetary compensation of $0.3.
Revised: For each batch, we provided the workers with a monetary compensation of $0.3.

Original: Finally, the human scores for each explanation were determined by taking a majority vote.
Revised: Finally, the human scores for each explanation were determined by taking a majority vote.

Original: B.4 Case Study and Limitations
Revised: B.4 Case Study and Limitations

Original: Figure 15 shows three examples with correct answers and gold explanations predicted by GPTvia chain-of-thought prompting (CoT).
Revised: Figure 15 shows three examples with correct answers and gold explanations predicted by GPTvia chain-of-thought prompting (CoT).

Original: We can see that GPT-3 (CoT) not only predicts the correct answers but also generates reasonable explanations, which follow the multi-hop reasoning process of human beings.
Revised: We can see that GPT-3 (CoT) not only predicts the correct answers but also generates reasonable explanations, which follow the multi-hop reasoning process of human beings.

Original: This suggests that large language models like GPT-3 have great promise for implementing high-level reasoning abilities.
Revised: This suggests that large language models like GPT-3 have great promise for implementing high-level reasoning abilities.

Original: Figure 16 visualizes three more examples with predictions from GPT-3 (CoT).
Revised: Figure 16 visualizes three more examples with predictions from GPT-3 (CoT).

Original: In these examples,
Revised: In these examples,

Original: GPT-3 (CoT) is able to predict the correct answers but fails to generate gold explanations.
Revised: GPT-3 (CoT) is able to predict the correct answers but fails to generate gold explanations.

Original: For example, GPT-3 (CoT) generates an irrelevant explanation because the context text does not include ﬁne-grained visual information in the image (Figure 16a).
Revised: For example, GPT-3 (CoT) generates an irrelevant explanation because the context text does not include ﬁne-grained visual information in the image (Figure 16a).

Original: In the example shown in Figure 16b,
Revised: In the example shown in Figure 16b,

Original: GPT-3 (CoT) fails to predict the coherent thought chains, where there are an incorrect example and
Revised: GPT-3 (CoT) fails to predict the coherent thought chains, where there are an incorrect example and

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: an incorrect statement for a chemical change .
Revised: an incorrect statement for a chemical change .

Original: The third example is given in Figure 16c, where the generated explanation is just a repetition of the input question and the output answer, instead of following the complete thought chain to arrive at the ﬁnal answer.
Revised: The third example is given in Figure 16c, where the generated explanation is just a repetition of the input question and the output answer, instead of following the complete thought chain to arrive at the ﬁnal answer.

Original: Four failure examples with wrong predicted answers are listed in Figure 17.
Revised: Four failure examples with wrong predicted answers are listed in Figure 17.

Original: We extract the image captions and feed them to the large language model as the visual content input.
Revised: We extract the image captions and feed them to the large language model as the visual content input.

Original: However, these captions lack ﬁne-grained semantics and usually do not work well for diagrams, which results in two failure cases shown in Figure 17a and 17b.
Revised: However, these captions lack ﬁne-grained semantics and usually do not work well for diagrams, which results in two failure cases shown in Figure 17a and 17b.

Original: Moreover, there exist challenges for large language models to reason about the questions that require them to understand complex and uncommon domain knowledge.
Revised: Moreover, there exist challenges for large language models to reason about the questions that require them to understand complex and uncommon domain knowledge.

Original: For example, GPT-3 (CoT) cannot understand accurately the terminology of personiﬁcation in language science (Figure 17c) and a series of complex chemical changes happen in the formation process of dinosaur fossils (Figure 17d).
Revised: For example, GPT-3 (CoT) cannot understand accurately the terminology of personiﬁcation in language science (Figure 17c) and a series of complex chemical changes happen in the formation process of dinosaur fossils (Figure 17d).

Original: B.5 Broader Impacts
Revised: B.5 Broader Impacts

Original: Societal impact.
Revised: Societal impact.

Original: The S CIENCE QA dataset collects science questions sourced from textbooks and is proposed to diagnose the multimodal understanding and multi-hop reasoning abilities of AI systems.
Revised: The S CIENCE QA dataset collects science questions sourced from textbooks and is proposed to diagnose the multimodal understanding and multi-hop reasoning abilities of AI systems.

Original: Due to the nature of data sources, S CIENCE QA does not contain any user usage data or personally sensitive information such as gender and race.
Revised: Due to the nature of data sources, S CIENCE QA does not contain any user usage data or personally sensitive information such as gender and race.

Original: After careful examination of our dataset, to our best knowledge, we have not found any improper content, such as pornographic information, racial remarks, or harmful social bias.
Revised: After careful examination of our dataset, to our best knowledge, we have not found any improper content, such as pornographic information, racial remarks, or harmful social bias.

Original: We adhere to the goal of AI for the common good, and any antisocial data points will be removed from the dataset based on feedback.
Revised: We adhere to the goal of AI for the common good, and any antisocial data points will be removed from the dataset based on feedback.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Potential usage.
Revised: Potential usage.

Original: The proposed S CIENCE QA dataset and designed methods in this paper are beneﬁcial to both follow-up research work and real-world applications.
Revised: The proposed S CIENCE QA dataset and designed methods in this paper are beneﬁcial to both follow-up research work and real-world applications.

Original: S CIENCE QA provides a useful benchmark for multi-modal learning, multi-hop reasoning, and general artiﬁcial intelligence.
Revised: S CIENCE QA provides a useful benchmark for multi-modal learning, multi-hop reasoning, and general artiﬁcial intelligence.

Original: Besides,
Revised: Besides,

Original: S CIENCE QA will contribute to the development of K-12 education applications such as tutoring systems.
Revised: S CIENCE QA will contribute to the development of K-12 education applications such as tutoring systems.

Original: Furthermore, the designed methods with the chain of thought investigate the ability of large language models to mimic the human mind process when reasoning about a challenging task.
Revised: Furthermore, the designed methods with the chain of thought investigate the ability of large language models to mimic the human mind process when reasoning about a challenging task.

Original: [Table]
Revised: [Table]

Original: (cid:58)(cid:85)(cid:82)(cid:81)(cid:74)(cid:3)(cid:36)(cid:81)(cid:86)(cid:90)(cid:72)(cid:85) (cid:20)(cid:26)(cid:26)(cid:24)(cid:22)
Revised: (cid:58)(cid:85)(cid:82)(cid:81)(cid:74)(cid:3)(cid:36)(cid:81)(cid:86)(cid:90)(cid:72)(cid:85) (cid:20)(cid:26)(cid:26)(cid:24)(cid:22)