Original: PINTO: F AITHFUL L ANGUAGE R EASONING U SING P ROMPTED -G ENERATED R ATIONALES
Revised: PINTO: F AITHFUL L ANGUAGE R EASONING U SING P ROMPT -G ENERATED R ATIONALES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural language models (LMs) have achieved impressive results on various language-based reasoning tasks by utilizing latent knowledge encoded in their own pretrained parameters.
Revised: Neural language models (LMs) have achieved impressive results on various language-based reasoning tasks by utilizing latent knowledge encoded in their own pretrained parameters.

Original: To make this reasoning process more explicit, recent works retrieve a rationalizing LM’s internal knowledge by training/prompting it to generate free-text rationales, which can be used to guide task predictions made by either the same LM or a separate reasoning LM.
Revised: To make this reasoning process more explicit, recent works retrieve a rationalizing LM’s internal knowledge by training or prompting it to generate free-text rationales, which can be used to guide task predictions made by either the same LM or a separate reasoning LM.

Original: However, rationalizing LMs require expensive rationale annotation, without any assurance that the generated rationales improve LM task performance or faithfully reflect LM decision-making.
Revised: However, rationalizing LMs require expensive rationale annotation and/or computation, without any assurance that their generated rationales improve LM task performance or faithfully reflect LM decision-making.

Original: In this paper, we propose PINTO, an LM pipeline that rationalizes via promptbased learning, and learns to faithfully reason over rationales via counterfactual regularization.
Revised: In this paper, we propose PINTO, an LM pipeline that rationalizes via prompt-based learning, and learns to faithfully reason over rationales via counterfactual regularization.

Original: First, PINTO maps out a suitable reasoning process for the task input by prompting a frozen rationalizing LM to generate a free-text rationale.
Revised: First, PINTO maps out a suitable reasoning process for the task input by prompting a frozen rationalizing LM to generate a free-text rationale.

Original: Second, PINTO’s reasoning LM is fine-tuned to solve the task using the generated rationale as context, while regularized to output less confident predictions when the rationale is perturbed.
Revised: Second, PINTO’s reasoning LM is fine-tuned to solve the task using the generated rationale as context, while regularized to output less confident predictions when the rationale is perturbed.

Original: Across four datasets, we show that PINTO significantly improves the generalization ability of the reasoning LM, yielding higher performance on both in-distribution and out-of-distribution test sets.
Revised: Across four datasets, we show that PINTO significantly improves the generalization ability of the reasoning LM, yielding higher performance on both in-distribution and out-of-distribution test sets.

Original: Also, PINTO leverages the rationales  more faithfully than  competitive baselines do.
Revised: Also, we find that PINTO’s rationales are more faithful to its task predictions than those generated by competitive baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many language-based reasoning tasks require retrieving and reasoning over knowledge beyond the task input— e.g., commonsense reasoning and closed-book QA (Fig. 1, left) (Talmor et al., 2018;
Revised: Many language-based reasoning tasks require retrieving and reasoning over knowledge beyond the task input— e g., commonsense reasoning and closed-book QA (Fig. 1, left) (Talmor et al., 2018;

Original: Mihaylov et al., 2018).
Revised: Mihaylov et al., 2018).

Original: Neural language models (LMs) have achieved impressive results on such tasks by utilizing latent knowledge encoded in their pretrained parameters (Raffel et al., 2020b;
Revised: Neural language models (LMs) have achieved impressive results on such tasks by utilizing latent knowledge encoded in their pretrained parameters (Raffel et al., 2020b;

Original: Brown et al., 2020).
Revised: Brown et al., 2020).

Original: Still, given LMs’ black-box nature, it is unclear whether this knowledge is being used properly (Doshi-Velez & Kim, 2017; Lipton, 2018).
Revised: Still, given LMs’ black-box nature, it is unclear whether this knowledge is being used properly (Doshi-Velez & Kim, 2017; Lipton, 2018).

Original: Previous studies have shown that
Revised: Previous studies have shown that

Original: LMs often learn spurious correlations from artifacts in downstream training data, thus limiting their generalizability (Branco et al., 2021;
Revised: LMs often learn spurious correlations from artifacts in downstream training data, thus limiting their generalizability (Branco et al., 2021;

Original: Geirhos et al., 2020; D’Amour et al., 2020).
Revised: Geirhos et al., 2020; D’Amour et al., 2020).

Original: With this in mind, a number of prior works aim to make LMs’ reasoning processes more explicit by generating free-text rationales, which use LMs’ internal knowledge to describe a reasoning process in natural language (Narang et al., 2020; Wei et al., 2022b; Marasovi´c et al., 2022; Zelikman et al., 2022).
Revised: With this in mind, a number of prior works aim to make LMs’ reasoning processes more explicit by generating free-text rationales, which use LMs’ internal knowledge to describe a reasoning process in natural language (Narang et al., 2020; Wei et al., 2022b; Marasovi´c et al., 2022; Zelikman et al., 2022).

Original: In the fine-tuned self-rationalizing paradigm, a single LM is fine-tuned to jointly generate the task output and rationale (Narang et al., 2020; Marasovi´c et al., 2022; Zelikman et al., 2022).
Revised: In the fine-tuned self-rationalizing paradigm, a single LM is fine-tuned to jointly generate the task output and rationale (Narang et al., 2020; Marasovi´c et al., 2022; Zelikman et al., 2022).

Original: In the prompted self-rationalizing paradigm, a single LM is instead frozen and prompted to jointly generate the task output and rationale, with the prompt consisting of a few input-output-rationale demonstrations (Wei et al., 2022b).
Revised: In the prompted self-rationalizing paradigm, a single LM is instead frozen and prompted to jointly generate the task output and rationale, with the prompt consisting of a few input-output-rationale demonstrations (Wei et al., 2022b).

Original: In the pipeline-rationalizing paradigm, a fine-tuned  LM first generates the rationale, which is then used as input for a separate fine-tuned reasoning LM to generate the output (Kumar & Talukdar, 2020; Rajani et al., 2019).
Revised: In the pipeline-rationalizing paradigm, a fine-tuned rationalizing LM first generates the rationale, which is then used as input for a separate fine-tuned reasoning LM to generate the output (Kumar & Talukdar, 2020; Rajani et al., 2019).

Original: However, when considering generalization performance, reliability, and deployment cost, these existing paradigms all have key limitations.

Original: Fine-tuned self-rationalizing LMs often perform worse than non-rationalizing LMs, since their parameters are learned using two relatively dissimilar objectives, while also requiring expensive rationale annotations (Wiegreffe et al., 2020; Narang et al.,

Original: Q
Revised: Q

Revised: [Figure]

Revised: R

Revised: A

Original: [Figure]
Revised: [Figure]

Original: Q
Revised: Q

Original: [Figure]
Revised: [Figure]

Original: Q
Revised: Q

Original: [Figure]
Revised: [Figure]

Original: Q
Revised: Q

Original: [Figure]
Revised: [Figure]

Original: A
Revised: A

Original: Q
Revised: Q

Original: [Figure]
Revised: [Figure]

Revised: R

Original: [Figure] 
Revised: [Figure] [Figure]

Original: Self
Revised: Self

Original: [Figure]
Revised: [Figure]

Original:  3) Pipeline [Figure]
Revised: Rationalization 3) Pipeline Rationalization 4)

Original: PINTO
Revised: PINTO

Original: StrategyQA:
Revised: StrategyQA:

Original: Question
Revised: Question

Original: Did Aristotle [Figure]
Revised: Did Aristotle use a laptop?

Original: Answer
Revised: Answer

Original: No.
Revised: No.

Original: Rationale Aristotle was alive until … while the first laptop was invented in …
Revised: Rationale Aristotle was alive until … while the first laptop was invented in …

Original: Question
Revised: Question

Original: Google Maps and other highway and street GPS services have replaced what?
Revised: Google Maps and other highway and street GPS services have replaced what?

Original: A) Mexico B) countryside C) atlas D) oceans …
Revised: A) Mexico B) countryside C) atlas D) oceans …

Original: Answer
Revised: Answer

Original: Atlas
Revised: Atlas

Original: Rationale
Revised: Rationale

Original: Atlas are replaced by more precise Google maps, other highway and street GPS services.
Revised: Atlas are replaced by more precise Google maps, other highway and street GPS services.

Original: CommonsenseQA:
Revised: CommonsenseQA:

Original: Q
Revised: Q

Original: [Figure]
Revised: [Figure]

Original: A
Revised: A

Original: [Figure]
Revised: R

Original:  +
Revised: A +

Original: R A + or R A 
Revised: R A + or R A + or

Original: [Figure]
Revised: However, when considering generalization performance, reliability, and deployment costs, these existing paradigms all have key limitations.

Original: or
Revised: Fine-tuned self-rationalizing LMs often perform worse than non-rationalizing LMs, since their parameters are learned using two relatively dissimilar objectives, while also requiring expensive rationale annotations (Wiegreffe et al., 2020; Narang et al., 2020).

Original: Prompted self-rationalizing LMs yield strong task performance and only need a few rationale demonstrations for the prompt, but are computationally prohibitive since they generally require very large-scale ( i.e., over 100B parameters) LMs to work effectively (Wei et al., 2022a;b).
Revised: Prompted self-rationalizing LMs yield strong task performance and only need a few rationale demonstrations for the prompt, but are computationally prohibitive since they generally require very large-scale ( i.e., over 100B parameters) LMs to work effectively (Wei et al., 2022a;b).

Original: Besides requiring expensive rationale annotations, pipeline-rationalizing LMs’ generated rationale forms a non-differentiable bottleneck between the two modules, which complicates end-to-end training and can hurt task performance (Wiegreffe et al., 2020; Hase et al., 2020).
Revised: Besides requiring expensive rationale annotations, pipeline-rationalizing LMs’ generated rationale forms a non-differentiable bottleneck between the two modules, which complicates end-to-end training and can hurt task performance (Wiegreffe et al., 2020; Hase et al., 2020).

Original: Moreover, none of these paradigms has a mechanism for regularizing the rationale generation to faithfully reflect the reasoning process of the LM, without hurting task performance.
Revised: Moreover, none of these paradigms has a mechanism for regularizing the rationale generation to faithfully reflect the reasoning process of the LM, without hurting task performance.

Original: In this paper, we propose P rompted
Revised: In this paper, we propose P rompted

Original: Rat I onalizing with Cou NT erfactual Reas O ning (
Revised: Rat I onalizing with Cou NT erfactual Reas O ning (

Original: PINTO ), an LM pipeline that rationalizes via prompt-based learning, then reasons over the task input and rationale via counterfactual regularization.
Revised: PINTO ), an LM pipeline that rationalizes via prompt-based learning, then reasons over the task input and rationale via counterfactual regularization.

Original: PINTO’s rationalizing module is a medium-scale ( i.e., 20B parameters) LM that contains vast latent knowledge obtained via pretraining (Black et al., 2022).
Revised: PINTO’s rationalizing module is a medium-scale ( i.e., 20B parameters) LM that contains vast latent knowledge obtained via pretraining (Black et al., 2022).

Original: Though prohibitive to fine-tune, it is affordable for prompt-based learning.
Revised: Though prohibitive to fine-tune, it is affordable for prompt-based learning.

Original: Given the task input and a minimal input-output demonstration prompt, the rationalizing module uses its internal knowledge to map out a suitable reasoning process for the task input by generating a free-text rationale.
Revised: Given the task input and a minimal input-output-rationale demonstration prompt, the rationalizing module uses its internal knowledge to map out a suitable reasoning process for the task input by generating a free-text rationale.

Original: The rationalizing module is frozen during fine-tuning, which drastically reduces training costs and prevents it from exploiting spurious shortcuts in the downstream training data.
Revised: The rationalizing module is frozen during fine-tuning, which drastically reduces training costs and prevents it from exploiting spurious shortcuts in the downstream training data.

Original: PINTO’s reasoning module is a small-scale ( i.e., under 1B parameters) LM to which knowledge is transferred from themuch larger rationalizing module.
Revised: PINTO’s reasoning module is a small-scale ( i.e., under 1B parameters) LM to which knowledge is transferred from the rationalizing module.

Original: The reasoning module is fine-tuned to solve the downstream reasoning task by using the generated rationale as context for the task input.
Revised: The reasoning module is fine-tuned to solve the downstream reasoning task by using the generated rationale as context for the task input.

Original: Crucially, to help ensure that the rationale dictates the behavior of the reasoning module, the reasoning module is regularized to output less confident predictions when the rationale is  perturbed.
Revised: Crucially, to help ensure that the reasoning module’s behavior is dictated by the rationale (instead of by spurious shortcuts), the reasoning module is regularized to output less confident predictions when the rationale is noisily perturbed.

Original: To simulate shortcut reasoning with rationales, we consider two rationale perturbation strategies: token masking ( i.e., rationale is ignored) and token replacement ( i.e., rationale is misused).
Revised: To simulate shortcut reasoning, we consider two rationale perturbation strategies: token masking ( i.e., rationale is ignored) and token replacement ( i.e., rationale is misused).

Original: Across four question answering datasets (CSQA, StrategyQA, OpenBookQA, QASC), we show that
Revised: Across four question answering datasets (CSQA, StrategyQA, OpenBookQA, QASC), we show that

Original: PINTO significantly improves the reasoning LM’s generalization, yielding higher performance on both in-distribution (ID) and out-of-distribution (OOD) test sets.
Revised: PINTO significantly improves the reasoning LM’s generalization, yielding higher performance on both in-distribution (ID) and out-of-distribution (OOD) test sets.

Original: Also, we find that rationales are leveraged more faithfully by PINTO than by other methods, leading to better performance in lowresource settings.
Revised: Also, we find that rationales are utilized more faithfully by PINTO than by other methods, leading to better performance in lowresource settings.

Original: Furthermore, we show that PINTO’s counterfactual regularization improves the reasoning module’s robustness to noise in the rationalizing module’s generated rationales.
Revised: Furthermore, we show that PINTO’s counterfactual regularization improves the reasoning module’s robustness to noise in the rationalizing module’s generated rationales.

Original:  R ATIONALE -B ASED L ANGUAGE R EASONING
Revised:  R ATIONALE -B ASED L ANGUAGE R EASONING

Original: In this work, we study LMs’ ability on language-based reasoning using implicit knowledge.
Revised: In this work, we study LMs’ ability to reason about language using implicit knowledge.

Original: We consider a specific type of multi-choice question answering (QA) tasks where the required knowledge foranswering the question is not explicitly provided in the input and needs to be inferred from the LM’s parameters (Talmor et al., 2019; Khot et al., 2020): Given a question q and a set of answer
Revised: We consider a specific type of multi-choice question answering (QA) tasks where the required knowledge for

Original: Question:
Revised: Question:

Original: What do people use to absorb extra ink from a fountain pen?
Revised: What do people use to absorb extra ink from a fountain pen?

Original: The answer is Calligrapher's hand.
Revised: The answer is Calligrapher's hand.

Original: Calligrapher's hand is used to …
Revised: Calligrapher's hand is used to …

Original: The answer is Inkwell.
Revised: The answer is Inkwell.

Original: Inkwell is used to absorb extra ink from a …
Revised: Inkwell is used to absorb extra ink from a …

Original: The answer is blotter.
Revised: The answer is blotter.

Original: Blotting paper absorbs liquids like ink well.
Revised: Blotting paper absorbs liquids like ink well.

Original: [Figure] [Figure]
Revised: a) Rationalizing Module : frozen, prompted medium-scale LM b) Reasoning Module : fine-tuned small-scale LM

Original:  Rationales
Revised: Choice-Specific Rationales

Original: [Figure]
Revised: DemonstrationQ1, A1, R Q2, A2,

Original: [Figure]
Revised: [Figure]

Original: QA
Revised: Q3,

Original: [Figure] Prompt
Revised: [Figure] [Figure]

Original: Q
Revised: Q

Original: R
Revised: R

Original: A
Revised: A

Original: Q
Revised: Q

Original: R
Revised: R

Original: A
Revised: A

Original: Training with
Revised: Training with

Original: Counterfactual Regularization
Revised: Counterfactual Regularization

Original: Inference 
Revised: Inference answering the question is not explicitly provided in the input and needs to be inferred from the LM’s parameters (Talmor et al., 2019; Khot et al., 2020):

Original:  choices
Revised: Given a question q and a set of answer choices

Original: A = { a i } , the model’s goal is to predict a plausibility score ρ ( q, a i ) for each ( q, a i ) pair, so that the predicted answer ˆ a = arg max a i ∈
Revised: A = { a i } , the model’s goal is to predict a plausibility score ρ ( q, a i ) for each ( q, a i ) pair, so that the predicted answer ˆ a = arg max a i ∈

Original: A ρ ( q, a i ) matches the correct answer choice a ∗ ∈ A .
Revised: A ρ ( q, a i ) matches the correct answer choice a ∗ ∈ A .

Original: Motivated by the LM tendency to exploit reasoning shortcuts for solving tasks (Branco et al., 2021), we focus on methods that explicitly generate free-text rationales to explain their predictions.
Revised: Motivated by LMs’ common tendency to exploit reasoning shortcuts when solving tasks (Branco et al., 2021), we focus on methods that explicitly generate free-text rationales to explain their predictions.

Original: Whereas extractive rationales are limited to input token scoring (Denil et al., 2014; Sundararajan et al., 2017; Chan et al., 2022), free-text rationales use natural language to describe a reasoning process ( e.g., things beyond the task input) (Narang et al., 2020; Wei et al., 2022b).
Revised: Whereas extractive rationales are limited to input token scoring (Denil et al., 2014; Sundararajan et al., 2017; Chan et al., 2022), free-text rationales use natural language to describe a reasoning process ( e.g., with knowledge beyond the task input) (Narang et al., 2020; Wei et al., 2022b).

Original: Below, we discuss several paradigms (see also Fig. 1) for rationale-based language reasoning.
Revised: Below, we discuss several paradigms (see also Fig. 1) for rationale-based language reasoning.

Original: Fine-Tuned Self-Rationalization
Revised: Fine-Tuned Self-Rationalization

Original: In this paradigm, an LM is fine-tuned to autogregressively generate the task output and rationale as a single sequence (Narang et al., 2020; Liu et al., 2018).
Revised: In this paradigm, an LM is fine-tuned to autogregressively generate the task output and rationale as a single sequence (Narang et al., 2020; Liu et al., 2018).

Original: If the rationale is generated after the task output, then the rationale is conditioned on the task output, and vice versa.
Revised: If the rationale is generated after the task output, then the rationale is conditioned on the task output, and vice versa.

Original: Since the LM parameters are shared across two relatively dissimilar objectives, they often perform worse than non-rationalizing LMs (Wiegreffe et al., 2020; Narang et al., 2020).
Revised: Since the LM parameters are shared across two relatively dissimilar objectives, they often perform worse than non-rationalizing LMs (Wiegreffe et al., 2020; Narang et al., 2020).

Original: Notably, this paradigm requires expensive rationale annotations for all training instances.
Revised: Notably, this paradigm requires expensive rationale annotations for all training instances.

Original: Prompted Self-Rationalization
Revised: Prompted Self-Rationalization

Original: In this paradigm, a pretrained LM is frozen and prompted to autogregressively generate the task output and rationale as a single sequence, with the prompt consisting of a few input-output-rationale demonstrations (Lampinen et al., 2022; Wei et al., 2022b).
Revised: In this paradigm, a pretrained LM is frozen and prompted to autogregressively generate the task output and rationale as a single sequence, with the prompt consisting of a few input-output-rationale demonstrations (Lampinen et al., 2022; Wei et al., 2022b).

Original: If the rationale is generated after the task output, then the rationale is conditioned on the task output, and vice versa.
Revised: If the rationale is generated after the task output, then the rationale is conditioned on the task output, and vice versa.

Original: This paradigm performs well andit only needs a few rationale annotations for the prompt, but it is computationally prohibitive since it generally requires very large-scale ( i.e., over 100B parameters) LMs to work effectively (Lampinen et al., 2022; Wei et al., 2022b).
Revised: This paradigm performs well and only needs a few rationale annotations for the prompt, but it is computationally prohibitive since it generally requires very large-scale ( i.e., over 100B parameters) LMs to work effectively (Lampinen et al., 2022; Wei et al., 2022b).

Original: Pipeline Rationalization Here, a fine-tuned rationalizing LM first generates the rationale, which is then used as input for a separate fine-tuned reasoning LM to predict the task output (Kumar & Talukdar, 2020; Rajani et al., 2019).
Revised: Pipeline Rationalization In this paradigm, a fine-tuned rationalizing LM first generates the rationale, which is then used as input for a separate fine-tuned reasoning LM to predict the task output (Kumar & Talukdar, 2020; Rajani et al., 2019).

Original: This paradigm’s generated rationale forms a discrete ( i.e., non-differentiable) bottleneck between the two modules, which complicates end-to-end training and can hurt task performance (Wiegreffe et al., 2020; Hase et al., 2020).
Revised: Here, the generated rationale forms a discrete ( i.e., non-differentiable) bottleneck between the two modules, which complicates end-to-end training and can hurt task performance (Wiegreffe et al., 2020; Hase et al., 2020).

Original: Additionally, the dedicated rationalizing LM requires extra rationale annotation/computation costs.
Revised: Additionally, the dedicated rationalizing LM requires extra rationale annotation/computation costs.

Original:  PINTO: F AITHFUL L ANGUAGE R EASONINGU SING P ROMPTED -G ENERATED R ATIONALES
Revised:  PINTO: F AITHFUL L ANGUAGE R EASONING

Original: PINTO is a two-stage, rationalize-then-reason pipeline, designed to address the limitations of existing paradigms for rationale-based language reasoning (§2).
Revised: PINTO is a two-stage, rationalize-then-reason pipeline, designed to address the limitations of existing paradigms for rationale-based language reasoning (§2).

Original: Like the pipeline rationalization paradigm, PINTO has separate modules for rationalizing and reasoning.
Revised: Like the pipeline rationalization paradigm, PINTO has separate modules for rationalizing and reasoning (Fig. 2).

Original: However,  PINTO’s rationalizing module is prompted instead of fine-tuned.
Revised: However, [Table] PINTO’s rationalizing module is prompted instead of fine-tuned.

Original: As a result, PINTOdoes not suffer from the non-differentiable bottleneck issue and has lower rationale annotation/computation costs.[Table]
Revised: Thus, PINTO does not suffer from the non-differentiable bottleneck issue and has lower rationale annotation/computation costs.

Original: Following prior works, PINTO is based on choice-specific rationales (Kumar & Talukdar, 2020;
Revised: Following prior works, PINTO is based on choice-specific rationales (Kumar & Talukdar, 2020;

Original: Hase et al., 2020).
Revised: Hase et al., 2020).

Original: First, given q and A , the rationalizing module generates a set of choice-specific rationales R = { r i } , where each r i explains a reasoning process that supports answer choice a i ∈ A (§3.1), as opposed to generating one rationale per question.
Revised: First, given q and A , the rationalizing module generates a set of choice-specific rationales R = { r i } , where each r i explains a reasoning process that supports answer choice a i ∈ A (§3.1), as opposed to generating one rationale per question.

Original: We opt for this design choice because rationales are often answer-leaking (Sun et al., 2022), i.e., the rationale itself is already sufficiently predictive of one of the answer choices.
Revised: We opt for this design choice because rationales are often answer-leaking (Sun et al., 2022), i.e., the rationale itself is already sufficiently predictive of one of the answer choices.

Original: If the rationalizing module only generates one rationale per question, then it is forced to make an “early decision”, and the reasoning module would only be left to recover the answer from the rationale (Kumar & Talukdar, 2020).
Revised: If the rationalizing module only generates one rationale per question, then it is forced to make an “early decision” on the predicted answer, such that the reasoning module would only be left to recover the answer from the rationale (Kumar & Talukdar, 2020).

Original: While prior works require expensive rationale annotations to train/prompt the rationalizing module (Kumar & Talukdar, 2020; Hase et al., 2020), PINTO’s rationalizing module is a frozen pretrained LM that uses only a few question-answer demonstrations as a prompt (§3.1).
Revised: While prior works require expensive rationale annotations to train/prompt the rationalizing module (Kumar & Talukdar, 2020; Hase et al., 2020), PINTO’s rationalizing module is a frozen pretrained LM that uses only a few question-answer-rationale demonstrations as a prompt (§3.1).

Original: Second, given q , a i ∈ A , and r i ∈ R , the reasoning module outputs plausibility score ρ ( q, a i , r i ) (§3.2).
Revised: Second, given q , a i ∈ A , and r i ∈ R , the reasoning module outputs plausibility score ρ ( q, a i , r i ) (§3.2).

Original: We also design a regularization objective that encourages the reasoning module to properly use the rationales to predict the answer (§3.3).
Revised: We also design a regularization objective that encourages the reasoning module to properly use the rationales to predict the answer (§3.3).

Original: We describe each module in more detail below.
Revised: We describe each module in more detail below.

Original: 3. R ATIONALIZING M ODULE
Revised: 3. R ATIONALIZING M ODULE

Original: Prior works mainly rely on human-annotated rationales for teaching a model to rationalize (Kumar & Talukdar, 2020; Hase et al., 2020; Sun et al., 2022).
Revised: Prior works mainly rely on human-annotated rationales for teaching a model to rationalize (Kumar & Talukdar, 2020; Hase et al., 2020; Sun et al., 2022).

Original: However, such rationale annotations are expensive and frequently of low quality (Aggarwal et al., 2021; Sun et al., 2022; Rajani et al., 2019), e.g., not providing sufficientcommonsense knowledge to support a given answer.
Revised: However, such rationale annotations are expensive and frequently of low quality (Aggarwal et al., 2021; Sun et al., 2022; Rajani et al., 2019), e.g., not providing sufficient knowledge to support a given answer.

Original: Meanwhile, a recent study shows that rationales automatically generated by pretrained LMs are often preferred over human-annotated rationales (Wiegreffe et al., 2021).
Revised: Meanwhile, a recent study shows that rationales automatically generated by pretrained LMs are often preferable over human-annotated rationales (Wiegreffe et al., 2021).

Original: Therefore, for PINTO’s rationalizing module, we propose using a pretrained LM to generate rationales via in-context learning, which prompts the frozen LM to retrieve knowledge from its parameters (Wei et al., 2022b).
Revised: Therefore, for PINTO’s rationalizing module, we propose using a pretrained LM to generate rationales via in-context learning, which prompts the frozen LM to retrieve knowledge from its parameters (Wei et al., 2022b).

Original: The prompt consists of a fixed set of question-answer demonstrations that are randomly selected from the training set.
Revised: The prompt consists of a fixed set of question-answer-rationale demonstrations that are randomly selected from the training set.

Original: Each demonstration consists of a question q , answer choices A , 2 gold answer a ∗ ∈ A , and a human-annotated free-text rationale r ∗ ∈ R for a ∗ (Table 1).
Revised: Each demonstration consists of a question q , answer choices A , 2 gold answer a ∗ ∈ A , and a human-annotated free-text rationale r ∗ ∈ R for a ∗ (Table 1).

Original: With this prompt p , we use the LM to generate rationales for every instance from the dataset.
Revised: With this prompt p , we use the LM to generate rationales for every instance from the dataset.

Original: Specifically, for each a i ∈ A of some instance ( q , A ), the rationalizing LM’s input is constructed as [ p , q , A , a i ].
Revised: Specifically, for each a i ∈ A of some instance ( q , A ), the rationalizing LM’s input is constructed as [ p , q , A , a i ].

Original: Then, we use greedy decoding of the LM output to obtain rationale r i for a i .
Revised: Then, we use greedy decoding of the LM output to obtain rationale r i for a i .

Original: Note that the LM input does not have any information about the gold answer a ∗ .
Revised: Note that the LM input does not have any information about the gold answer a ∗ .

Original: Ourdesign of the rationalizing module assumes that r i will be aligned with common sense if and only if a i = a ∗ , since it should intuitively be difficult to retrieve correctcommonsense knowledge that supports an incorrect answer choice.
Revised: Our rationalizing module’s design assumes that r i will be aligned with accurate knowledge if and only if a i = a ∗ , since it should intuitively be difficult to retrieve correct knowledge that supports an incorrect answer choice (see Table 11 in the appendix for examples of the generation).

Original: The reasoning module then predicts the correct answer by reasoning over the rationales for each answer choice.
Revised: The reasoning module then predicts the correct answer by reasoning over the rationales for each answer choice.

Original: 3. R EASONING M ODULE
Revised: 3. R EASONING M ODULE

Original: Given a question q , the answer choices A , answer candidate a i ∈ A , and rationale r i , the reasoning module learns to output plausibility score ρ i =
Revised: Given a question q , the answer choices A , answer candidate a i ∈ A , and rationale r i , the reasoning module learns to output plausibility score ρ i =

Original: ρ ( q, A, a i , r i ) .
Revised: ρ ( q, A, a i , r i ) .

Original: Following prior works, we use a text-to-text Transformer LM as the backbone of our reasoning module (Wiegreffe et al., 2020;
Revised: Following prior works, we use a text-to-text Transformer LM as the backbone of our reasoning module (Wiegreffe et al., 2020;

Original: Hase et al., 2020).
Revised: Hase et al., 2020).

Original: For each a i , the reasoning module’s input is defined as the token sequence s =
Revised: For each a i , the reasoning module’s input is defined as the token sequence s =

Original: [ q ⊕ a 1 ⊕ ...
Revised: [ q ⊕ a 1 ⊕ ...

Original: ⊕ a | A | ⊕ r i ] , where ⊕ denotes concatenation.
Revised: ⊕ a | A | ⊕ r i ] , where ⊕ denotes concatenation.

Original: Meanwhile, the reasoning module’s

Original: As opposed to full human annotation, we only need a few (usually < 8) examples for one dataset.

Original:  output is obtained by sequentially teacher-forcing a i ’s tokens t i =
Revised: Meanwhile, the reasoning module’s output is obtained by sequentially teacher-forcing a i ’s tokens t i =

Original: [ ti 1  , ti 2  , ..., ti | a i |  ] into the decoder, rather than via greedy decoding.
Revised: [ t 1 i , t 2 i , ..., t | a i | i ] into the decoder, rather than via greedy decoding.

Original: This way, we can compute the reasoning module’s output token probabilities for arbitrary answer choices a i .
Revised: This way, we can compute the reasoning module’s output token probabilities for arbitrary answer choices a i .

Original: Following Shwartz et al.
Revised: Following Shwartz et al.

Original: , we compute a i ’s plausibility score ρ i by aggregating t i ’s token probabilities  as:
Revised: , we compute a i ’s plausibility score ρ i by aggregating the probabilities P of tokens t ji as:

Original: [Equation]
Revised: [Equation]

Original: Next, we use the softmax function to normalize ρ i as probability P ( a i | q, A, R ) =
Revised: Next, we use the softmax function to normalize ρ i as probability P ( a i | q, A, R ) =

Original: e ρ i / (cid:80)
Revised: e ρ i / (cid:80)

Original: | A | j =1 e ρ j .
Revised: | A | j =1 e ρ j .

Original: During inference , the rationalizing module firstly generate R = { r i } given a new question and answer choices, and then the  predicted answer choiceis computed by the reasoning module as ˆ a = arg max a i ∈ A P ( a i | q, A, R ) .
Revised: During inference, given question q and answer choices A , the rationalizing module first generates rationales R = { r i } , then the reasoning module computes the predicted answer choice as ˆ a = arg max a i ∈ A P ( a i | q, A, R ) .

Original: 3. T RAINING
Revised: 3. T RAINING

Original: Fora multi-choice QA, the standard training objective is to maximize the likelihood of the correct answer choice using cross-entropy loss, computed as: 
Revised: For multi-choice QA, the standard training objective is to maximize the likelihood of the correct answer choice using cross-entropy loss, computed as: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Q ( a i | q, A ) is 

Revised: where Q ( a i | q, A ) is

Original: if a i = a ∗ and 0 otherwise.
Revised: if a i = a ∗ and 0 otherwise.

Original: Let Q ( A | q, A ) be the one-hot target distribution over all a i ∈ A .
Revised: Let Q ( A | q, A ) be the one-hot target distribution over all a i ∈ A .

Original: There can be spurious correlations between q and A (Branco et al., 2021), so the reasoning module may take undesirable shortcuts instead of properly using the rationale to predict the answer (Gururangan et al., 2018; McCoy et al., 2019).
Revised: There can be spurious correlations between q and A (Branco et al., 2021), so the reasoning module may take undesirable shortcuts instead of properly using the rationale to predict the answer (Gururangan et al., 2018; McCoy et al., 2019).

Original: In this case, the rationales would be unfaithful in explaining the model’s behavior and useless for model debugging
Revised: In this case, the rationales would be unfaithful in explaining the model’s behavior and useless for model debugging.

Original: [Figure]
Revised: [Figure]

Original: To address this, we introduce a counterfactual regularization objective in which the reasoning module is regularized to output less confident predictions when the rationale is not leveraged properly ( i.e., shortcuts are used).
Revised: To address this, we introduce a counterfactual regularization objective in which the reasoning module is regularized to output less confident predictions when the rationale is not utilized properly ( i.e., shortcuts are used).

Original: This is implemented using label smoothing (Szegedy et al., 2016), which softens the target distribution Q ( A | q, A ) by linearly combining it with a noisy distribution U ( A | q, A ) , often set as the uniform distribution.
Revised: This is implemented using label smoothing (Szegedy et al., 2016), which softens the target distribution Q ( A | q, A ) by linearly combining it with a noisy distribution U ( A | q, A ) , often set as the uniform distribution.

Original: Therefore, given tunable label smoothing factor 0 < ϵ < 1 , we compute the label-smoothed target distribution as: Q ′ ( A | q, A ) =
Revised: Therefore, given tunable label smoothing factor 0 < ϵ < 1 , we compute the label-smoothed target distribution as: Q ′ ( A | q, A ) =

Original: (1 − ϵ ) Q ( A | q, A ) + ϵU
Revised: (1 − ϵ ) Q ( A | q, A ) + ϵU

Original: ( A | q, A ) .
Revised: ( A | q, A ) .

Original: In order to simulate shortcut reasoning, we considerthe two strategies for perturbing the generated rationales r i .
Revised: In order to simulate shortcut reasoning, we consider two strategies for perturbing the generated rationales r i .

Original: Token Masking addresses the case where the reasoning module ignores the rationale and instead exploits spurious cues in the rest of the input.
Revised: Token Masking addresses the case where the reasoning module ignores the rationale and instead exploits spurious cues in the rest of the input.

Original: To simulate this, we mask  the rationales from the input.
Revised: To simulate this, we mask out the rationales in the input.

Original: Recall that the backbone of the reasoning module is a Transformer LM, which uses a self-attention mechanism to aggregate information across tokens.
Revised: Recall that the backbone of the reasoning module is a Transformer LM, which uses a self-attention mechanism to aggregate information across tokens.

Original: Hence, we implement rationale masking by zeroing the attention mask for rationale tokens.
Revised: Hence, we implement rationale masking by zeroing the attention mask for rationale tokens

Original: Token Replacement addresses the scenario in which the reasoning module misunderstands the rationales.
Revised: the case where the reasoning module misunderstands the rationales’ meaning and thus uses them improperly.

Original: It randomly replaces k % of the rationale tokens with other tokens randomly sampled from the entire  vocabulary.
Revised: To simulate this, we randomly replace k % of the rationale tokens with other tokens uniformly sampled from the entire language vocabulary.

Original: At each fine-tuning step, we randomly select one of the strategies for obtaining perturbed rationales R ′ =
Revised: At each fine-tuning step, we randomly select one of the strategies for obtaining perturbed rationales R ′ =

Original: { r ′ i } , which helps keep the LM from overfitting to any particular strategy.
Revised: { r ′ i } , which helps keep the LM from overfitting to any particular strategy.

Original: Then, the counterfactual regularization loss is computed as:
Revised: Then, the counterfactual regularization loss is computed as:

Original: [Equation]
Revised: [Equation]

Original: This counterfactual regularization teaches the reasoning module to be less confident when the rationales are either absent or problematic, so that it can learn to make sounder use of the rationales.
Revised: This counterfactual regularization teaches the reasoning module to be less confident when the rationales are either absent or problematic, so that it can learn to make sounder use of the rationales.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Questions and hypotheses We design experiments to answer the following questions: (1) What is the impact of our PINTO pipeline on faithfulness and end-task performance?
Revised: Questions and hypotheses We design experiments to answer the following questions: (1) What is the impact of our PINTO pipeline on faithfulness and end-task performance?

Original: We expect our pipeline with counterfactual training technique to obtain improvements in both aspects.
Revised: We expect our pipeline with counterfactual training technique to obtain improvements in both aspects.

Original: (2) How does the quality of rationales affect the end-task performance of PINTO?
Revised: (2) How does the quality of rationales affect the end-task performance of PINTO?

Original: We hypothesize that improving the quality of the rationales of PINTO improves its accuracy.
Revised: We hypothesize that improving the quality of the rationales of PINTO improves its accuracy.

Original: (3) Does faithful reasoning based on rationales lead to better generalization?
Revised: (3) Does faithful reasoning based on rationales lead to better generalization?

Original: We expect that a method like PINTO that learns to rely on rationales can better generalize to a low resource setting and out-of-distribution (OOD) datasets.
Revised: We expect that a method like PINTO that learns to rely on rationales can better generalize to a low resource setting and out-of-distribution (OOD) datasets.

Original: Datasets We experiment with several CSR benchmarks.
Revised: Datasets We experiment with several CSR benchmarks.

Original: (1) CommonsenseQA
Revised: (1) CommonsenseQA

Original: (Talmor et al., 2018) is a 5-choice QA dataset testing general commonsense reasoning about the concepts from
Revised: (Talmor et al., 2018) is a 5-choice QA dataset testing general commonsense reasoning about the concepts from

Original: ConceptNet (Speer et al., 2017).
Revised: ConceptNet (Speer et al., 2017).

Original: (2) StrategyQA
Revised: (2) StrategyQA

Original: (Geva et al., 2021) is a binary (yes/no) QA dataset that requires models to infer the reasoning strategy.
Revised: (Geva et al., 2021) is a binary (yes/no) QA dataset that requires models to infer the reasoning strategy.

Original: (3) OpenBookQA (Mihaylov et al., 2018) is a 4-choice QA dataset that requests reasoning based on open book as well as broad commonsense knowledge.
Revised: (3) OpenBookQA (Mihaylov et al., 2018) is a 4-choice QA dataset that requests reasoning based on open book as well as broad commonsense knowledge.

Original: (4) QASC (Khot et al., 2020) is an 8-choice QA dataset that requires a system to answer a question with a valid composition of basic facts using common sense.
Revised: (4) QASC (Khot et al., 2020) is an 8-choice QA dataset that requires a system to answer a question with a valid composition of basic facts using common sense.

Original: Since the gold labels for the testing sets of these datasets are not publicly available, we treat the official development set as our test set, and separate the training data into our own training set and development set.
Revised: Since the gold labels for the testing sets of these datasets are not publicly available, we treat the official development set as our test set, and separate the training data into our own training set and development set.

Revised: Evaluation Metrics To evaluate the reasoning model’s task performance , we use the accuracy metric and consider both ID and OOD test sets in our experiments.

Revised: ID/OOD test sets are taken from the same/different dataset as the training set.

Revised: To evaluate the faithfulness of the generated rationale to the reasoning model’s predicted label, we adopt the LAS metric (Hase et al., 2020).

Revised: LAS measures rationale-label consistency as how well the rationale helps a simulator model predict the reasoning model’s predicted label.

Original: Implementation Details For the rationalizing module, we use GPT-neox (Black et al., 2022), a pretrained, autoregressive LM with 20B parameters.
Revised: Following Hase et al. , we implement the simulator as a fine-tuned T5-Base LM (Raffel et al., 2020a). To aggregate accuracy and LAS as a single metric, we use Normalized Relative Gain (NRG) metric (Chan et al., 2022). Across all compared methods,

Revised: NRG first normalizes each of the two constituent metrics’ scores as values in [0 , 1] , then obtains the aggregate score by taking the mean of the two normalized scores.

Revised: Implementation Details For the rationalizing module, we use GPT-neox (Black et al., 2022), a pretrained, autoregressive LM with 20B parameters.

Original: We manually annotate 7 examples to set up the prompt for each task dataset.
Revised: We manually annotate 7 examples to set up the prompt for each task dataset.

Original: For the reasoning module, we adopt T5-base (Raffel et al., 2020a) with only 220 million parameters, which is around two orders of magnitude smaller than the rationalizing module.
Revised: For the reasoning module, we adopt T5-base (Raffel et al., 2020a) with only 220 million parameters, which is around two orders of magnitude smaller than the rationalizing module.

Original: During fine-tuning, the standard training loss (Eq. 1) and our counterfactual training loss (Eq. 2) are directly combined as the overall training loss.
Revised: During fine-tuning, the standard training loss (Eq. 1) and our counterfactual training loss (Eq. 2) are directly combined as the overall training loss.

Original: For perturbing rationales, we randomly choose the token masking or token replacement strategy with a equal chance in each training batch.
Revised: For perturbing rationales, we randomly choose the token masking or token replacement strategy with a equal chance in each training batch.

Original: The replacing rate for token replacement is empirically set to 30% .
Revised: The replacing rate for token replacement is empirically set to 30% .

Original: We run all the experiments on the compared methods 4 times using a fixed set of random seeds and report the average results.
Revised: We run all the experiments on the compared methods 4 times using a fixed set of random seeds and report the average results.

Original: Baselines (1) Without Rationales is a T5-based model fine-tuned with the task dataset without using any rationales as additional input.
Revised: Baselines (1) Without Rationales is a T5-based model fine-tuned on the task dataset without using any rationales as additional input.

Original: (2) Prompted Self-Rationalization is a frozen LM which learns from a few examples in the prompt to firstly generate a few short sentences as the rationale and then predict the answer.
Revised: (2) Prompted Self-Rationalization is a GPT-NeoX LM that learns from a few examples in the prompt to firstly generate a few short sentences as the rationale and then predict the answer.

Original: We experiment with GPT-neox (20B) and also report the results of GPT- (175B) from Wei et al.
Revised: Here, we use the chain-of-thought prompting configuration from Wei et al.

Original: (2022b).
Revised: (2022b).

Original: (3) Distilled Self-Rationalization is a small LM (T5-base) trained on the rationales generated by the Prompted Self-Rationalization model.
Revised: (3) Distilled Self-Rationalization is a small LM (T5-base) trained on the rationales generated by the Prompted Self-Rationalization model.

Original: We implement two variants of the distillation model: a) Rationalize-First, which firstly generates the rationale and then predicts the answer, and b) Predict-First, which firstly predicts the answer and then generates the rationale.
Revised: We implement two variants of the distillation model: a) Rationalize-First, which firstly generates the rationale and then predicts the answer, and b) Predict-First, which firstly predicts the answer and then generates the rationale.

Revised: (4) NILE (Kumar &

Revised: Talukdar, 2020) trains a rationalization module by fine-tuning a T5-3B model (Raffel et al., 2020a) with the rationales annotated by humans, then trains a reasoning module by fine-tuning a T5-Base model with the task dataset as in our method.

Revised: We only apply NILE on the CSQA and StrategyQA datasets, since they provide human-annotated gold rationales.

Original: (4) Standard Training adopts the same rationalizing and reasoning pipeline as our method, but the reasoning module is not fine-tuned with the counterfactual training loss.
Revised: (5) Standard Training uses the same rationalize-then-reason pipeline as our method, except the reasoning module is not fine-tuned with [Table]

Original: (5) Dropout Context is the same as the Standard Training baseline exceptthat during the fine-tuning of the reasoning module, the question is randomly dropped out from the input, which is a strategy adopted in prior work(Hase et al., 2020) to encourage the reasoning module to make good use of the input rationales.
Revised: [Table] the counterfactual training loss. (6) Dropout Context is the same as the Standard Training baseline, except the question is randomly dropped out from the input while fine-tuning the reasoning module. This is a strategy used in prior work to encourage the reasoning module to make good use of the input rationales (Hase et al., 2020).

Original: Further, we also consider two variants of PINTO, namely Token Masking Only and Token Replacement
Revised: Further [Table] we also consider two variants of PINTO, namely Token Masking Only and Token Replacement

Original: Only as baselines.
Revised: Only as baselines.

Original: These baselines only adopt token masking or token replacement for perturbing rationale tokens, respectively.
Revised: These baselines only adopt token masking or token replacement for perturbing rationale tokens, respectively.

Original: Evaluation Metrics To evaluate task performance , we use accuracy as the main metric and consider both ID and OOD evaluation sets in our experiments.

Original: To evaluate the faithfulness of the generated rationale to the model’s prediction, we adopt LAS metric (Hase et al., 2020), which measures how well rationales help a simulator predict a model’s output.

Original: We implement the simulator with a finetuned T5-base model as recommended by Hase et al.

Original:  E XPERIMENTS 5. M AIN R ESULTS
Revised:  E XPERIMENTS 5. M AIN R ESULTS

Original:  Performance  on ID data Table 2 shows the task performancein accuracy of all the compared methods on the four CSR datasets we consider, from which we make two observations.
Revised: In-Distribution (ID) Performance We first evaluate all methods on ID test sets. Table 2 shows the task performance of these methods, with fine-tuning methods using T5-Base as the reasoning module.

Original: First, the 
Revised: We have the following two observations.

Original:  Prompted Self-Rationalization baseline with GPT-neox (20B) generally does not outperform the[Table] [Table] fine-tuning methods while the GPT-3 version is reported to achieve 73 .
Revised: First, the Prompted Self-Rationalization baseline (using the 20B-parameter GPT-NeoX) generally does not outperform the fine-tuning methods while the GPT-3 version is reported to achieve 73 .

Original: 50 and 66 .
Revised: 50 and 66 .

Original: 53 in accuracy on CSQA and StrategyQA respectively (Wei et al., 2022b).
Revised: 53 in accuracy on CSQA and StrategyQA, respectively (Wei et al., 2022b).

Original: This validates that Prompted SelfRationalization requires very large LMs to work effectively (Wei et al., 2022a).
Revised: This validates that Prompted Self-Rationalization requires very large LMs to work effectively (Wei et al., 2022a).

Original: Second, simply augmenting the reasoning module with rationales (as in Standard Training) does not always lead to better results compared with the Without Rationales baseline since the rationales may not be properly utilized.
Revised: Second, simply augmenting the reasoning module with rationales (as in Standard Training) does not always lead to better results compared with the Without Rationales baseline since the rationales may not be properly utilized.

Original: The Dropout Context baseline helps to address this issue in some, but not all cases, while PINTO consistently yields the best accuracy in most of the cases
Revised: The Dropout Context baseline helps to address this issue in some, but not all cases, while PINTO consistently yields the best accuracy in most of the cases.

Original: [Table]
Revised: We have similar observations from results using RoBERTa-Large as the reasoning module (Table 5 of §A.1). This demonstrates the effectiveness of our counterfactual regularization method in improving ID generalization.

Original: Generalizability to OOD data To  demonstrate the generalizability brought by faithful reasoning over rationales, we further investigate the performance of our method on out-of-distribution (OOD) data.
Revised: Out-of-Distribution (OOD) Performance To further demonstrate the generalizability brought by faithful reasoning over rationales, we also investigate the performance of our method on OOD test [Figure] [Figure]

Original:  The intuition is that by utilizing rationales faithfully rather than fitting only the in-distribution training data, our model achieves better OOD generalization without any fine-tuning.
Revised: sets. The intuition is that by utilizing rationales faithfully rather than fitting only the ID training data, our model achieves better OOD generalization without any fine-tuning.

Original: Table 3 shows the OOD performance of all the fine-tuning methods.
Revised: Table 3 shows the OOD performance of all the fine-tuning methods using T5-Base.

Original: We conclude that rationales are helpful in improving the generalizability of the model to a dataset unseen during fine-tuning.
Revised: We conclude that rationales are helpful in improving the generalizability of the model to a dataset unseen during fine-tuning.

Original: Among all the methods utilizing rationales, our method yields the best OOD performance, which confirms the benefit of faithful reasoning. 
Revised: Among all the methods utilizing rationales, our method yields the best OOD performance, which confirms the benefit of faithful reasoning. A consistent conclusion can be made from the results based on

Revised: RoBERTa-Large (Table 6 of §A.1).

Original: Rationale-Label Association Table 2 also reports the faithfulness of all the methods involving rationalization measured by LAS.
Revised: Rationale-Label Association Table 2 also reports the faithfulness of all the methods involving rationalization measured by LAS.

Original: We observe that PINTO achieves a much higher score compared with the baselines except on OpenBookQA.
Revised: We observe that PINTO achieves a much higher score compared with the baselines except on OpenBookQA.

Original: This demonstrates thatwith counterfactual regularization, the reasoning modulecan make predictions more faithfully with regard to the rationales.
Revised: This demonstrates that counterfactual regularization helps the reasoning module make predictions more faithfully with respect to the rationales.

Original: 5. P ERFORMANCE A NALYSIS
Revised: 5. P ERFORMANCE A NALYSIS

Original: How do different perturbation strategies contribute to the overall performance?
Revised: How do different perturbation strategies contribute to the overall performance?

Original: Table 2 shows the results of the ablation study where we only conduct Token Masking or Token Replacement when perturbing the rationale tokens.
Revised: Table 2 shows the results of the ablation study where we only conduct Token Masking or Token Replacement when perturbing the rationale tokens.

Original: From more cases, we note that Token Replacement leads to both better accuracy and faithfulness compared with Token Masking.
Revised: From more cases, we note that Token Replacement leads to both better accuracy and faithfulness compared with Token Masking.

Original: This is because Token Replacement perturbs the semantics of the rationales more severely, thus further forcing the reasoning module[Figure] [Figure] to properly make use of the rationales.
Revised: This is because Token Replacement perturbs the semantics of the rationales more severely, thus further forcing the reasoning module to properly make use of the rationales.

Original: Our method yields the best results when both types of perturbation are conducted, which validates that these two strategies consider comprehensively the different ways in which a reasoning module could use the rationales improperly.
Revised: Our method yields the best results when both types of perturbation are conducted, which validates that these two strategies consider comprehensively the different ways in which a reasoning module could use the rationales improperly.

Original: How does the smoothing factor ϵ affect accuracy and faithfulness?

Original: Intuitively, the smoothing factor ϵ in §3.3 controls how much we want to enforce the reasoning module to consider the rationales.

Original: To understand its role in affecting task accuracy and rationales faithfulness, we conduct an analysis where we vary the ϵ from 0 .

Original: [-1 to 1 . 
-]
Revised: [-1 to 1 . 
-]

Original: when fine-tuning the reasoning module.

Original: Figure 4 shows accuracy on the left axis and faithfulness on the right axis.

Original: We can see that with a small ϵ ( 0 . 1 ), our method strikes a balance between obtaining high accuracy and ensuring faithfulness.

Original: When ϵ = 0 .

Original: 8 , our method yields the highest faithfulness measure at the cost of accuracy.

Original: Note that there is no clear trend in the results plot due to the randomness brought by the training dynamics

Original: [Figure]

Original: notated rationales vs. models using only generated rationales on CSQA.

Original: Can faithful rationales lead to better lowresource performance?
Revised: Can faithful rationales lead to better low-resource performance?

Original: We also investigate whether, with counterfactual training, the reasoning module can be fine-tuned with less training data.
Revised: We also investigate whether, with counterfactual training, the reasoning module can be fine-tuned with less training data.

Original: Figure 5 shows the accuracy of all the fine-tuning methods.
Revised: Figureshows the accuracy of all the fine-tuning methods.

Original: We can observe that our method consistently outperforms the baselines at different percentages of training data.
Revised: We can observe that our method consistently outperforms the baselines at different percentages of training data.

Original: The observed larger performance gap is larger when less training data is used, demonstrating the data efficiency of our method.
Revised: The observed larger performance gap is larger when less training data is used, demonstrating the data efficiency of our method.

Original: Can we refine the reasoning behavior via rationales?
Revised: Can we refine the reasoning behavior via rationales?

Original: One important application of faithful reasoning is that rationales provide a way to refine the behavior of a model, i.e., we can correct reasoning mistakes by providing a better rationale.
Revised: One important application of faithful reasoning is that rationales provide a way to refine the behavior of a model, i.e., we can correct reasoning mistakes by providing a better rationale.

Original: To verify this, we make use of ECQA (Aggarwal et al., 2021) which augments CSQA with human-annotated rationales.
Revised: To verify this, we make use of ECQA (Aggarwal et al., 2021) which augments CSQA with human-annotated rationales.

Original: We directly provide the human-annotated rationales to the fine-tuned reasoning modules to obtain its oracle results, shown in Figure 6.
Revised: We directly provide the human-annotated rationales to the fine-tuned reasoning modules to obtain its oracle results, shown in Figure 5.

Original: We see that human-annotated rationales generally lead to performance gain for all fine-tuning methods whereof the gain of our method is the largest.
Revised: We see that human-annotated rationales generally lead to performance gain for all finetuning methods whereof the gain of our method is the largest.

Original: This again showcases the merits of ensuring the faithful reasoning on rationales in refining a system.
Revised: This again showcases the merits of ensuring the faithful reasoning on rationales in refining a system.

Original: [Table]

Original: Is our method more sensitive to perturbed rationales?
Revised: Is our method more sensitive to perturbed rationales?

Original: Intuitively, higher rationale faithfulness (i.e., stronger connection between the rationale the and reasoning module’s behavior) should yield greater sensitivity to noisily perturbed rationales.
Revised: Intuitively, higher rationale faithfulness (i.e., stronger connection between the rationale the and reasoning module’s behavior) should yield greater sensitivity to noisily perturbed rationales.

Original: In other words, higher performance drop (sensitivity) signals higher faithfulness.
Revised: In other words, higher performance drop (sensitivity) signals higher faithfulness.

Original: To verify this, we conduct a stress test.
Revised: To verify this, we conduct a stress test.

Original: We choose CSQA and OpenBookQA and replace each question in the testing set with a randomly sampled question but still keep the original answer choices.
Revised: We choose CSQA and OpenBookQA and replace each question in the testing set with a randomly sampled question but still keep the original answer choices.

Original: We then prompt our rationalizing module with the replaced question and the original choices to obtain a set of perturbed rationales.
Revised: We then prompt our rationalizing module with the replaced question and the original choices to obtain a set of perturbed rationales.

Original: We finally provide the perturbed rationales to the reasoning module.
Revised: We finally provide the perturbed rationales to the reasoning module.

Original: Our results in Table 4 show that PINTO achieves a[Table] significantly higher performance drop than the other two methods (esp.
Revised: Our results in Table 4 show that PINTO achieves a significantly higher performance drop than the other two methods (esp.

Original: on OBQA), indicating that counterfactual regularization is effective in improving rationale faithfulness.
Revised: on OBQA), indicating that counterfactual regularization is effective in improving rationale faithfulness.

Original: 5. C ASE S TUDY

Original: We provide concrete examples in Table 5 to showcase how our prompted LM rationalizes for correct and incorrect choices and how PINTO reasons more faithfully compared with the Standard baseline.

Original: In the question (second row) from CSQA, we can see that for incorrect choices, the generated rationales do not support them to be the answer while the one for the correct choice refrigerator does.

Original: In the question (third row) from StrategyQA, the rationale for the correct choice yes is sound and reasonable while the rationale for the incorrect choice

Original: no is factually correct but does not answer the question directly ( died in a plane crash vs. died in the space journey ).

Original: For both questions, PINTO properly leverages the rationales and make the correct predictions while the Standard baseline fails.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Revised: [Table]

Original: Extensive work have been done on solving implicit reasoning tasks by augmenting the models with external knowledge.
Revised: Extensive work has been done on solving implicit reasoning tasks by augmenting reasoning LMs with external knowledge beyond the task input.

Original: They retrieve such knowledge from (1) knowledge graphs (Lin et al., 2019;
Revised: Prior works have explored retrieving implicit knowledge from: (1) knowledge graphs (Lin et al., 2019;

Original: Feng et al., 2020), (2) web corpus (Lv et al., 2020) or (3) pre-trained language models (Shwartz et al., 2020; Liu et al., 2021).
Revised: Feng et al., 2020; Wang et al., 2020; Yan et al., 2021; Chan et al., 2021; Raman et al., 2021), (2) web corpora (Lv et al., 2020; Chen et al., 2017; Yang et al., 2015; Ryu et al., 2014), or (3) pretrained LMs (Shwartz et al., 2020; Liu et al., 2021; Bosselut et al., 2019; Shin et al., 2020).

Original: Althoughthe retrieved knowledge is shown to be helpful in boosting  task performance, it may not necessarily explain the decisions made by their models.
Revised: Although knowledge retrieval has shown to be helpful in boosting reasoning LMs’ task performance, it may not necessarily explain the decisions made by the LM.

Revised: Given the lack of transparency in neural LMs’ complex behavior (Rudin, 2019; Caruana, 2019), model explainability is important for promoting human trust in NLP systems for high-stakes decision-making (Doshi-Velez & Kim, 2017; Lipton, 2018; Bender et al., 2021).

Original: We focus on rationales in this work as a way to both improve a model’s performance and support its decisions.
Revised: We focus on rationale generation in this work as a way to both improve an LM’s task performance and provide justification for its predictions.

Original: Traditional works in explaining NLP models study extractive rationales such as attention scores (Bahdanau et al., 2014) or input gradients (Sundararajan et al., 2017), which highlight the tokens that are more important for models’ prediction. Extractive rationales fall short in implicit reasoning tasks since the required knowledge is not even part of the input.
Revised: Prior works on free-text rationale generation can be grouped into three paradigms. In the fine-tuned self-rationalizing paradigm, a single LM is fine-tuned to jointly generate the task output and rationale (Narang et al., 2020; Marasovi´c et al., 2022; Zelikman et al., 2022; Li et al., 2022). Since the LM parameters are shared across two relatively dissimilar objectives, they often perform worse than non-rationalizing LMs (Wiegreffe et al., 2020; Narang et al., 2020).

Original:  In contrast, free-text rationales provide a more flexible and human-accessible way of revealing models’ behavior.
Revised: Notably, this paradigm requires expensive rationale annotations for all training instances. In the prompted self-rationalizing paradigm, a single LM is instead frozen and prompted to jointly generate the task output and rationale, with the prompt consisting of a few input-output-rationale demonstrations (Wei et al., 2022b). This paradigm performs well and only needs a few rationale annotations for the prompt, but it is computationally prohibitive since it generally requires very large-scale LMs to work effectively (Lampinen et al., 2022; Wei et al., 2022b).

Original: Prior work in generating free-text rationales can be grouped into two paradigms. The first one is selfrationalizing, which adopts one model to generate rationales and predictions together.
Revised: In the pipeline-rationalizing paradigm, a fine-tuned rationalizing LM first generates the rationale, which is then used as input for a separate fine-tuned reasoning LM to generate the output (Kumar & Talukdar, 2020; Rajani et al., 2019). Here, the generated rationale forms a discrete ( i.e., non-differentiable) bottleneck between the two modules, which complicates end-to-end training and can hurt task performance (Wiegreffe et al., 2020; Hase et al., 2020).

Original: The second paradigm is pipeline-rationalizing, where the  rationalizing and reasoning modules are two separate models.
Revised: Additionally, the dedicated rationalizing LM requires extra rationale annotation/computation costs. Moreover, none of these paradigms has a mechanism for regularizing the rationale generation to faithfully reflect the reasoning process of the LM, without hurting task performance.

Original: Existing works (Kumar & Talukdar, 2020; Hase et al., 2020; Sun et al., 2022) use humanannotated rationales to fine-tune the rationalizing module while our PINTO uses a prompted LM to obtain those rationales without expensive human labor.
Revised: PINTO avoids these limitations by rationalizing via prompt-based learning (using a frozen medium-scale LM), then reasoning over the task input and rationale via counterfactual regularization (using a finetuned small-scale LM).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper presents PINTO, an LM pipeline that rationalizes with prompt-based learning and reasons via counterfactual regularization.
Revised: This paper presents PINTO, an LM pipeline that rationalizes with prompt-based learning and reasons via counterfactual regularization.

Original: Through prompting, we remove the need for expensive human annotation and leverage the massive knowledge encoded in a medium-sized LM to perform rationalization.
Revised: Through prompting, we remove the need for expensive human annotation and leverage the massive knowledge encoded in a medium-sized LM to perform rationalization.

Original: With counterfactual regularization in addition to standard training objective, our reasoning module learns to reason over the generated rationales more faithfully.
Revised: With counterfactual regularization in addition to standard training objective, our reasoning module learns to reason over the generated rationales more faithfully.

Original: Experiments show that our method outperforms baselines on both in-distribution and out-of-distribution datasets in accuracy, while providing higher faithfulness.
Revised: Experiments show that our method outperforms baselines on both in-distribution and out-of-distribution datasets in accuracy, while providing higher faithfulness.

Original: Our analysis also shows that we can further improve task performance with a more faithful reasoning module and refined rationales.
Revised: Our analysis also shows that we can further improve task performance with a more faithful reasoning module and refined rationales A CKNOWLEDGEMENT

Revised: We thank the anonymous reviewers and all the collaborators in USC INK research lab for their valuable feedback.

Revised: This material is based upon work sponsored by the DARPA MCS program under

Revised: Contract No. N660011924033 with the United States Office Of Naval Research

Original:  A A PPENDIX A. E XPERIMENTS WITH R O BERT A AS THE REASONING MODULE
Revised: . A A PPENDIX A. E XPERIMENTS WITH R O BERT A AS THE REASONING MODULE

Original: Table 6-7 show both the ID and OOD results based on RoBERTa-Large as the reasoning module.
Revised: Table 5-6 show both the ID and OOD results based on RoBERTa-Large as the reasoning module.

Original: The observations are consistent with Table 2-3 where we fine-tune T5-Base as the reasoning module. [Table] [Table]
Revised: The observations are consistent with Table 2-3 where we fine-tune T5-Base as the reasoning module

Original:  A BLATION ON THE LM SIZE FOR THE R ATIONALIZING M ODULE
Revised: . A. A BLATION ON THE LM SIZE FOR THE R ATIONALIZING M ODULE

Original: Table 8 shows the results of the Pipeline approaches using LMs with different model sizes as the rationalizing module. [Table] [Table]
Revised: Table 7 shows the results of the Pipeline approaches using LMs with different model sizes as the rationalizing module

Original:  V ARIANCE STATISTICS OF ALL THE FINE - TUNED MODELS
Revised: . A. V ARIANCE STATISTICS OF ALL THE FINE - TUNED MODELS

Original: Table 9 and Table 10 show the variance statistics (standard deviation) along with ID and OOD task performance (accuracy) of the fine-tuned methods from Table 2 and Table 3. 
Revised: Table 8 and Table 9 show the variance statistics (standard deviation) along with ID and OOD task performance (accuracy) of the fine-tuned methods from Table 2 and Table 3. [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A. H UMAN EVALUATION ON THE RATIONALES
Revised: A. H UMAN EVALUATION ON THE RATIONALES

Original: We conducted a human evaluation of 100 generated rationales from the CSQA dataset.
Revised: We conducted a human evaluation of 100 generated rationales from the CSQA dataset.

Original: The evaluation is a head-to-head comparison between the human-annotated rationales and the machinegenerated rationales.
Revised: The evaluation is a head-to-head comparison between the human-annotated rationales and the machinegenerated rationales.

Original: Annotators were asked to judge for 5 dimensions on a 3-point Likert scale following a prior work (Wiegreffe et al., 2021): 1) Factuality (How factual is this Explanation?) 2)
Revised: Annotators were asked to judge for 5 dimensions on a 3-point Likert scale following a prior work (Wiegreffe et al., 2021): 1) Factuality (How factual is this Explanation?) 2)

Original: Grammaticality (Is this Explanation grammatical?) 3) New Info (Does the Explanation provide new facts, information, or reasoning not stated in the Question and Answer?) 4)
Revised: Grammaticality (Is this Explanation grammatical?) 3) New Info (Does the Explanation provide new [Table] [Table] [Table]

Revised: Q: If you have leftover cake, where would you put it?

Revised: [Table]

Original: Supports Answer (Is the Explanation relevant to justifying the Answer?)
Revised: refrigerator ✓ (a) quandry: Quandry is a place where you put things that you don’t know ... (b) refrigerator: Refrigerator is used to store food. (c) oven: Oven is used to bake cakes.

Revised: Q: Has every astronaut survived their space journey?

Revised: yes ✗ no ✓ yes: Astronauts have died in space.

Revised: For example, the Challenger astronauts died in the Challenger space shuttle explosion.

Revised: no: The first astronaut to die was Yuri Gagarin.

Original:  5) Completeness (Does the Explanation provide enough information to jusify the answer?)
Revised: Gagarin died in a plane crash. facts, information, or reasoning not stated in the Question and Answer?) Supports Answer (Is the Explanation relevant to justifying the Answer?) 5) Completeness (Does the Explanation provide enough information to jusify the answer?)

Original: We obtain a fair level of agreement measured by Fleiss Kappa (k=0.34) for the evaluation.
Revised: We obtain a fair level of agreement measured by Fleiss Kappa (k=0.34) for the evaluation.

Original: The results in Table 11 show that machine-generated results are competitive with human annotation on most of the evaluating dimensions.
Revised: The results in Table 10 show that machine-generated results are competitive with human annotation on most of the evaluating dimensions.

Original: Generated rationales are even judged to be more grammatical than human annotations.
Revised: Generated rationales are even judged to be more grammatical than human annotations.

Original: As for completeness, generated rationales are slightly worse than human annotations.
Revised: As for completeness, generated rationales are slightly worse than human annotations.

Original: We think this is because the human annotators were explicitly encouraged to provide more comprehensive rationales when annotating the CSQA dataset (Aggarwal et al., 2021). 
Revised: We think this is because the human annotators were explicitly encouraged to provide more comprehensive rationales when annotating the CSQA dataset (Aggarwal et al., 2021). A. C ASE S TUDY

Revised: We provide concrete examples in Table 11 to showcase how our prompted LM rationalizes for correct and incorrect choices and how PINTO reasons more faithfully compared with the Standard baseline.

Revised: In the question (second row) from CSQA, we can see that for incorrect choices, the generated rationales do not support them to be the answer while the one for the correct choice refrigerator does.

Revised: In the question (third row) from StrategyQA, the rationale for the correct choice yes is sound and reasonable while the rationale for the incorrect choice

Original: [Table]
Revised: no is factually correct but does not answer the question directly ( died in a plane crash vs. died in the space journey ). For both questions, PINTO properly leverages the rationales and make the correct predictions while the Standard baseline fails. A. P ROMPTS FOR RATIONALIZATION

Revised: Table 12- 15 show the complete prompts we use to obtain rationales from LM for CSQA, StrategyQA, OpenBookQA and QASC datasets.

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]