Original: G ENERATING S EQUENCES BY L EARNING TO [S ELF -]C ORRECT
Revised: G ENERATING S EQUENCES BY L EARNING TO [S ELF -]C ORRECT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content.
Revised: Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content.

Original: Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack a mechanism to iteratively revise their outputs.
Revised: Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack a mechanism to iteratively revise their outputs.

Original: Moreover, some powerful language models are of extreme scale or inaccessible, making it inefficient, if not infeasible, to update their parameters for task-specific adaptation.
Revised: Moreover, some powerful language models are of extreme scale or inaccessible, making it inefficient, if not infeasible, to update their parameters for task-specific adaptation.

Original: We present SELF - CORRECTION , an approach that decouples an imperfect base generator (an off-the-shelf language model or supervised sequence-to-sequence model) from a separate corrector that learns to iteratively correct imperfect generations.
Revised: We present SELF - CORRECTION , an approach that decouples an imperfect base generator (an off-the-shelf language model or supervised sequence-to-sequence model) from a separate corrector that learns to iteratively correct imperfect generations.

Original: To train the corrector, we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations.
Revised: To train the corrector, we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations.

Original: We show that SELFCORRECTION improves upon the base generator in three diverse generation tasks– mathematical program synthesis, lexically-constrained generation, and toxicity control– even when the corrector is much smaller than the base generator.
Revised: We show that SELFCORRECTION improves upon the base generator in three diverse generation tasks– mathematical program synthesis, lexically-constrained generation, and toxicity control– even when the corrector is much smaller than the base generator.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The standard practice for natural language generation tasks is inherently single-pass: applying a decoding procedure to either a few-shot prompted language model or one tuned for a given task, then considering the generation as “finished” (e.g. Radford et al.
Revised: The standard practice for natural language generation tasks is inherently single-pass: applying a decoding procedure to either a few-shot prompted language model or one tuned for a given task, then considering the generation as “finished” (e.g. Radford et al.

Original: ; Brown et al.
Revised: ; Brown et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: Powerful generation models often meet most of the task requirements, yet miss a few (e.g., omitting a subset of keywords), or generate incorrect hypotheses that nevertheless provide useful structure (e.g., a correct problem solving strategy with a missing step).
Revised: Powerful generation models often meet most of the task requirements, yet miss a few (e.g., omitting a subset of keywords), or generate incorrect hypotheses that nevertheless provide useful structure (e.g., a correct problem solving strategy with a missing step).

Original: However, after generating even a slightly sub-optimal sequence, the single-pass paradigm requires models to “start from scratch”, effectively discarding work already done.
Revised: However, after generating even a slightly sub-optimal sequence, the single-pass paradigm requires models to “start from scratch”, effectively discarding work already done.

Original: A more natural, intuitive approach is leveraging the generation as a useful starting point to refine into a higher quality output.
Revised: A more natural, intuitive approach is leveraging the generation as a useful starting point to refine into a higher quality output.

Original: To formalize this intuition, we introduce Self-Correction for Sequence Generation .
Revised: To formalize this intuition, we introduce Self-Correction for Sequence Generation .

Original: Figuredemonstrates its central principle: a generation model is re-framed as a base generator , which produces a reasonable initial hypothesis but does not need to solve the task in one pass, and a second module–the corrector –trained to make up the difference between the hypothesis and an optimal solution.
Revised: Figuredemonstrates its central principle: a generation model is re-framed as a base generator , which produces a reasonable initial hypothesis but does not need to solve the task in one pass, and a second module–the corrector –trained to make up the difference between the hypothesis and an optimal solution.

Original: Neither the generator nor the corrector must solve the full task in one pass, and the corrector can be applied multiple times to iteratively improve the output (§3.6).
Revised: Neither the generator nor the corrector must solve the full task in one pass, and the corrector can be applied multiple times to iteratively improve the output (§3.6).

Original: We propose a simple, general procedure for training the corrector (Figure 2) by pairing generator outputs with carefully selected targets.
Revised: We propose a simple, general procedure for training the corrector (Figure 2) by pairing generator outputs with carefully selected targets.

Original: The result is a system which self-corrects, producing outputs through multiple generation passes and breaking the task into steps that can be solved by dedicated and efficient sub-systems.
Revised: The result is a system which self-corrects, producing outputs through multiple generation passes and breaking the task into steps that can be solved by dedicated and efficient sub-systems.

Original: Self-Correction builds on past work for correction in the code and text (e.g. Yasunaga et al.
Revised: Self-Correction builds on past work for correction in the code and text (e.g. Yasunaga et al.

Original: Faltings et al.
Revised: Faltings et al.

Original: (2021)) domains, but provides a unified formalism with minimal assumptions about data and feedback, which applies generally to diverse tasks.
Revised: (2021)) domains, but provides a unified formalism with minimal assumptions about data and feedback, which applies generally to diverse tasks.

Original: A corrector model improves the base generator on 3 such tasks in our experiments: mathematical program synthesis (§3.1), lexically constrained generation (§3.2), and toxicity reduction (§3.3).
Revised: A corrector model improves the base generator on 3 such tasks in our experiments: mathematical program synthesis (§3.1), lexically constrained generation (§3.2), and toxicity reduction (§3.3).

Original: The trained corrector model even transfers to a larger generator with similar performance to training from scratch (§3.4).
Revised: The trained corrector model even transfers to a larger generator with similar performance to training from scratch (§3.4).

Original: Finally, we explore introducing a third module to the Self-Correction system (§3.5)–explicitly using natural language feedback to guide corrections–with promising results.
Revised: Finally, we explore introducing a third module to the Self-Correction system (§3.5)–explicitly using natural language feedback to guide corrections–with promising results.

Original: Self-Correction is an exciting path to build on the generations of strong models, with efficient, effective, and transferable corrector networks.
Revised: Self-Correction is an exciting path to build on the generations of strong models, with efficient, effective, and transferable corrector networks.

Original:  S ELF - CORRECTING SEQUENCE GENERATORS
Revised:  S ELF - CORRECTING SEQUENCE GENERATORS

Original: A typical autoregressive text generator (e.g. GPT-3 (Brown et al., 2020)) maps an input prompt to a distribution over outputs using a single parameterized module (e.g. a large transformer), p 

Revised: A typical autoregressive text generator (e.g. GPT-3 (Brown et al., 2020)) maps an input prompt to a distribution over outputs using a single parameterized module (e.g. a large transformer), p

Original: ( y | x ) .
Revised: ( y | x ) .

Original: We explore an alternative that decomposes into two modules, a base generator , and a corrector ,
Revised: We explore an alternative that decomposes into two modules, a base generator , and a corrector ,

Original: [Equation]
Revised: [Equation]

Original: where the generator provides an initial hypothesis that is refined by the corrector.
Revised: where the generator provides an initial hypothesis that is refined by the corrector.

Original: In practice, the corrector can be applied multiple times, p
Revised: In practice, the corrector can be applied multiple times, p

Original: [Equation]
Revised: [Equation]

Original: Since a model of this form can both generate and correct its generations, we call it a Self-Corrector .
Revised: Since a model of this form can both generate and correct its generations, we call it a Self-Corrector .

Original: Self-correctors have several unique properties compared to typical generators.
Revised: Self-correctors have several unique properties compared to typical generators.

Original: First, a self-corrector decouples generation and correction, allowing us to freely parameterize each module – for instance, by prompting a single language model or using two different language models.
Revised: First, a self-corrector decouples generation and correction, allowing us to freely parameterize each module – for instance, by prompting a single language model or using two different language models.

Original: In this paper, we develop a framework to train a separate corrector model (§2.1).
Revised: In this paper, we develop a framework to train a separate corrector model (§2.1).

Original: We find that the resulting selfcorrector improves upon the generator alone (§3), even when the corrector is much smaller (§3.4).
Revised: We find that the resulting selfcorrector improves upon the generator alone (§3), even when the corrector is much smaller (§3.4).

Original: Second, since the generator and the corrector are separated, we can keep the generator as a generalpurpose language model and train the corrector with different objectives for different task requirements.
Revised: Second, since the generator and the corrector are separated, we can keep the generator as a generalpurpose language model and train the corrector with different objectives for different task requirements.

Original: In §2.1, we propose a training algorithm for the corrector that is dedicated to improving generations, where the improvement can be in any aspect, measured by scalar values.
Revised: In §2.1, we propose a training algorithm for the corrector that is dedicated to improving generations, where the improvement can be in any aspect, measured by scalar values.

Original: Third, the corrector can receive explicit feedback about intermediate generations to guide subsequent generations.
Revised: Third, the corrector can receive explicit feedback about intermediate generations to guide subsequent generations.

Original: Formally, p ( y | x ) =
Revised: Formally, p ( y | x ) =

Original: (cid:80)
Revised: (cid:80)

Original: y
Revised: y

Original: 0 p 0 ( y 0 | x ) p θ ( y | y 0 , x, f ( y 0 )) , where f is the feedback.
Revised: 0 p 0 ( y 0 | x ) p θ ( y | y 0 , x, f ( y 0 )) , where f is the feedback.

Original: The feedback can be of many forms, e.g. a sentence, a compiler trace, etc.
Revised: The feedback can be of many forms, e.g. a sentence, a compiler trace, etc.

Original: In contrast, a typical generator that generates in a single pass does not leverage feedback on its own generation.
Revised: In contrast, a typical generator that generates in a single pass does not leverage feedback on its own generation.

Original: In this paper, we show that the corrector can learn to exploit explicit natural language feedback to achieve better performance (§3.5).
Revised: In this paper, we show that the corrector can learn to exploit explicit natural language feedback to achieve better performance (§3.5).

Original: Next, we describe our training framework of the corrector.
Revised: Next, we describe our training framework of the corrector.

Original: 2. L EARNING A C ORRECTOR
Revised: 2. L EARNING A C ORRECTOR

Original: Our goal is to have the generator generate an initial hypothesis, then improve the hypothesis with the corrector (Eq. 1).
Revised: Our goal is to have the generator generate an initial hypothesis, then improve the hypothesis with the corrector (Eq. 1).

Original: We train the corrector to improve the quality of a hypothesis, while staying as close as possible to the original hypothesis.
Revised: We train the corrector to improve the quality of a hypothesis, while staying as close as possible to the original hypothesis.

Original: Here, quality is measured with a scalar value function v ( y ) which is accessible at training time (e.g. 0/1 indicator of program correctness, a toxicity score).
Revised: Here, quality is measured with a scalar value function v ( y ) which is accessible at training time (e.g. 0/1 indicator of program correctness, a toxicity score).

Original: Since direct supervision on how to improve hypotheses is not available, we design a new algorithm to train the corrector, which we refer to as self-corrective learning.
Revised: Since direct supervision on how to improve hypotheses is not available, we design a new algorithm to train the corrector, which we refer to as self-corrective learning.

Original: The algorithm collects a pool of generations, pairs them and selects pairs of generation that increase in value and are nearby, then updates the corrector on these pairs.
Revised: The algorithm collects a pool of generations, pairs them and selects pairs of generation that increase in value and are nearby, then updates the corrector on these pairs.

Original: As training progresses, more generations are added to the pool using the current corrector.
Revised: As training progresses, more generations are added to the pool using the current corrector.

Original: Algorithm 1 summarizes self-corrective learning, detailed below.
Revised: Algorithm 1 summarizes self-corrective learning, detailed below.

Original: Initialization.
Revised: Initialization.

Original: Self-corrective learning begins with a generator p 

Revised: Self-corrective learning begins with a generator p

Original: ( y 0 | x ) , a corrector p θ ( y ′ | y, x ) , a set of training prompts X , and a value function v : Y → R .
Revised: ( y 0 | x ) , a corrector p θ ( y ′ | y, x ) , a set of training prompts X , and a value function v : Y → R .

Original: Optionally, we can use additional feedback f : Y → F and learn p θ ( y ′ | y, x, f ( y )) , where F is arbitrary.
Revised: Optionally, we can use additional feedback f : Y → F and learn p θ ( y ′ | y, x, f ( y )) , where F is arbitrary.

Original: input p θ ,
Revised: input p θ ,

Original: X v
Revised: X v

Original: · ) , feedback f
Revised: · ) , feedback f

Original: The algorithm initializes a datapool of (input, output, value, feedback) examples by using the generator to generate multiple outputs for each input.
Revised: The algorithm initializes a datapool of (input, output, value, feedback) examples by using the generator to generate multiple outputs for each input.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: all y ∈
Revised: all y ∈

Original: [Equation]
Revised: [Equation]

Original: where y 1: N denotes N outputs generated with decoding algorithm q (e.g. temperature sampling).
Revised: where y 1: N denotes N outputs generated with decoding algorithm q (e.g. temperature sampling).

Original: When available, ( x, y, v ( y ) , f ( y )) examples from another source (e.g. a dataset) can also be added.
Revised: When available, ( x, y, v ( y ) , f ( y )) examples from another source (e.g. a dataset) can also be added.

Original: Pairing.
Revised: Pairing.

Original: Next, self-corrective learning forms value-improving pairs : examples of mapping a hypothesis to a higher-valued correction.
Revised: Next, self-corrective learning forms value-improving pairs : examples of mapping a hypothesis to a higher-valued correction.

Original: We use the datapool D to form a set of (input, hypothesis, correction) pairs.
Revised: We use the datapool D to form a set of (input, hypothesis, correction) pairs.

Original: A pair is formed when an output has a higher value than another 1 :
Revised: A pair is formed when an output has a higher value than another 1 :

Original: [Equation]
Revised: [Equation]

Original: Learning.
Revised: Learning.

Original: Next, self-corrective learning selects (input, hypothesis, correction) pairs to update the corrector with.
Revised: Next, self-corrective learning selects (input, hypothesis, correction) pairs to update the corrector with.

Original: We sample an input, x ∼ U
Revised: We sample an input, x ∼ U

Original: ( X ) , then sample a ( x, y, y ′ ) pair proportional to its improvement in value as well as the proximity between the hypothesis y and the correction y ′ :,
Revised: ( X ) , then sample a ( x, y, y ′ ) pair proportional to its improvement in value as well as the proximity between the hypothesis y and the correction y ′ :,

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: where s ( y, y ′ ) is a similarity function and Z ( y ) normalizes over the available corrections for y in
Revised: where s ( y, y ′ ) is a similarity function and Z ( y ) normalizes over the available corrections for y in

Original: P x .
Revised: P x .

Original: Increasing the hyperparameter α ∈ R ≥ 0 puts more weight on targets that add more value, while increasing β ∈ R ≥ 0 retains more similar targets.
Revised: Increasing the hyperparameter α ∈ R ≥ 0 puts more weight on targets that add more value, while increasing β ∈ R ≥ 0 retains more similar targets.

Original: We update the corrector using the cross-entropy loss L ( θ ) =
Revised: We update the corrector using the cross-entropy loss L ( θ ) =

Original: − log p θ ( y ′ | y, x, f ( y )) on batches sampled in this way.
Revised: − log p θ ( y ′ | y, x, f ( y )) on batches sampled in this way.

Original: Exploration.
Revised: Exploration.

Original: During exploration, self-corrective learning adds new generations to the datapool by generating from the current corrector:
Revised: During exploration, self-corrective learning adds new generations to the datapool by generating from the current corrector:

Original: [Equation]
Revised: [Equation]

Original: | for
Revised: | for

Original: [Equation]
Revised: [Equation]

Original: and updating the datapool D ← D ∪ D ′ .
Revised: and updating the datapool D ← D ∪ D ′ .

Original: The hypotheses y to correct can come from any source, e.g. newly sampled from the base generator, or from the datapool; we use the latter in our experiments.
Revised: The hypotheses y to correct can come from any source, e.g. newly sampled from the base generator, or from the datapool; we use the latter in our experiments.

Original: Inference.
Revised: Inference.

Original: We use the trained corrector along with a generator to generate a trajectory y 0 , y 1 , . . .
Revised: We use the trained corrector along with a generator to generate a trajectory y 0 , y 1 , . . .

Original: , y T , and consider y T the final output.
Revised: , y T , and consider y T the final output.

Original: Since marginalizing over the intermediate generations in Eq.
Revised: Since marginalizing over the intermediate generations in Eq.

Original: 1 is intractable, we approximate each summation with a single sequence generated with a decoding algorithm q ( · ) .
Revised: 1 is intractable, we approximate each summation with a single sequence generated with a decoding algorithm q ( · ) .

Original: That is, we decode from the generator, then repeatedly from the corrector:
Revised: That is, we decode from the generator, then repeatedly from the corrector:

Original: • Generation: y 0 ∼ q ( p 0 ( y 0 | x )) ; • Correction: y t
Revised: • Generation: y 0 ∼ q ( p 0 ( y 0 | x )) ; • Correction: y t

Original: +1 ∼ q ( p θ ( y t +1 | y t , x, f ( y t ))) , t = 0 , 1 , . . .
Revised: +1 ∼ q ( p θ ( y t +1 | y t , x, f ( y t ))) , t = 0 , 1 , . . .

Original: , T −
Revised: , T −

Original: [Equation]
Revised: [Equation]

Original: The stopping time T is either fixed, or when a target value is obtained (if v ( y ) is available).
Revised: The stopping time T is either fixed, or when a target value is obtained (if v ( y ) is available).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate SELF - CORRECTION on a diversity of tasks: mathematical program synthesis , in which generations are strictly correct or incorrect, and generators typically have low performance; lexically-constrained generation , which allows for partial credit, and generators usually give partially-correct solutions (e.g. matching 3 out of 5 constraints); and toxicity control , where ‘correctness’ is more loosely defined, and the output space is much more open-ended.
Revised: We evaluate SELF - CORRECTION on a diversity of tasks: mathematical program synthesis , in which generations are strictly correct or incorrect, and generators typically have low performance; lexically-constrained generation , which allows for partial credit, and generators usually give partially-correct solutions (e.g. matching 3 out of 5 constraints); and toxicity control , where ‘correctness’ is more loosely defined, and the output space is much more open-ended.

Original: Our experiments are organized to study three settings:
Revised: Our experiments are organized to study three settings:

Original: Using self-correctors to improve upon generators (§3.1,3.2,3.3).
Revised: Using self-correctors to improve upon generators (§3.1,3.2,3.3).

Original: Correcting generators that are much larger than the corrector (§3.4).
Revised: Correcting generators that are much larger than the corrector (§3.4).

Original: Leveraging explicit feedback during training and inference (§3.5).
Revised: Leveraging explicit feedback during training and inference (§3.5).

Original: Next, we describe the self-correction setup and baselines for each task, along with their results.
Revised: Next, we describe the self-correction setup and baselines for each task, along with their results.

Original: 3. M ATHEMATICAL P ROGRAM S YNTHESIS
Revised: 3. M ATHEMATICAL P ROGRAM S YNTHESIS

Original: First, we consider mathematical program synthesis (Austin et al., 2021;
Revised: First, we consider mathematical program synthesis (Austin et al., 2021;

Original: Mishra et al., 2022).
Revised: Mishra et al., 2022).

Original: Given a natural language problem specification x , the task is to generate a program y that upon execution returns the correct answer to x .
Revised: Given a natural language problem specification x , the task is to generate a program y that upon execution returns the correct answer to x .

Original: The task is challenging as it draws on language understanding, multiple-step mathematical problem solving (e.g. identifying a solution strategy, decomposing a problem), and leveraging symbolic tools (e.g. built-in operations, variables).
Revised: The task is challenging as it draws on language understanding, multiple-step mathematical problem solving (e.g. identifying a solution strategy, decomposing a problem), and leveraging symbolic tools (e.g. built-in operations, variables).

Original: Furthermore, the task demands a high level of precision, e.g. a single misplaced operation makes the program incorrect.
Revised: Furthermore, the task demands a high level of precision, e.g. a single misplaced operation makes the program incorrect.

Original: Experimental setup.
Revised: Experimental setup.

Original: As the corrector we use GPT-Neo 1.3B (Black et al., 2021), an open-source autoregressive language model.
Revised: As the corrector we use GPT-Neo 1.3B (Black et al., 2021), an open-source autoregressive language model.

Original: GPT-Neo is pre-trained on language and code (Gao et al., 2021), and hence is widely used for code-related generation (e.g. Chen et al.
Revised: GPT-Neo is pre-trained on language and code (Gao et al., 2021), and hence is widely used for code-related generation (e.g. Chen et al.

Original: ; Ni et al.
Revised: ; Ni et al.

Original: Mishra et al.
Revised: Mishra et al.

Original: We consider two settings for the initial generator: (1) a separate fine-tuned instance of GPT-Neo 1.3B, and (2) few-shot prompted GPT-3 (Brown et al., 2020).
Revised: We consider two settings for the initial generator: (1) a separate fine-tuned instance of GPT-Neo 1.3B, and (2) few-shot prompted GPT-3 (Brown et al., 2020).

Original: For GPT-3, we evaluate the davinci and text-davinci-002 engines, representative of large ( ≈ 175 B 3 ) generators that are state-of-the-art in related tasks (Wei et al., 2022).
Revised: For GPT-3, we evaluate the davinci and text-davinci-002 engines, representative of large ( ≈ 175 B 3 ) generators that are state-of-the-art in related tasks (Wei et al., 2022).

Original: See the Appendix for additional details.
Revised: See the Appendix for additional details.

Original: Self-correction setup.
Revised: Self-correction setup.

Original: As the value function we use correctness, which is 1 when the program y executes and outputs the ground-truth answer and 0 otherwise.
Revised: As the value function we use correctness, which is 1 when the program y executes and outputs the ground-truth answer and 0 otherwise.

Original: Our main experiments do not use explicit feedback, i.e. f ( y )
Revised: Our main experiments do not use explicit feedback, i.e. f ( y )

Original: = ∅ .
Revised: = ∅ .

Original: At inference time, we study two settings for the corrector: (1) applying k corrections and selecting the final generation, (2) an oracle setting that only corrects a draft if the draft is incorrect.
Revised: At inference time, we study two settings for the corrector: (1) applying k corrections and selecting the final generation, (2) an oracle setting that only corrects a draft if the draft is incorrect.

Original: We use greedy decoding for the generator and corrector, and k = 1 .
Revised: We use greedy decoding for the generator and corrector, and k = 1 .

Original: Datasets.
Revised: Datasets.

Original: We evaluate on problems from 5 problem solving datasets:
Revised: We evaluate on problems from 5 problem solving datasets:

Original: MultiArith (Roy et al., 2015),
Revised: MultiArith (Roy et al., 2015),

Original: AddSub (Hosseini et al., 2014), SingleOp (Roy et al., 2015), SVAMP (Patel et al., 2021), and
Revised: AddSub (Hosseini et al., 2014), SingleOp (Roy et al., 2015), SVAMP (Patel et al., 2021), and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Problem:
Revised: Problem:

Original: It takes Jennifer 20 minutes to groom each of her 2 long hair dachschunds.
Revised: It takes Jennifer 20 minutes to groom each of her 2 long hair dachschunds.

Original: If she grooms her dogs every day, how many hours does she spend grooming her dogs in 30 days?
Revised: If she grooms her dogs every day, how many hours does she spend grooming her dogs in 30 days?

Original: Generator: a=20*2b=a* answer=
Revised: Generator: a=20*2b=a* answer=

Original: b print (answer)
Revised: b print (answer)

Original: Corrector: a=20*2b=a* c=
Revised: Corrector: a=20*2b=a* c=

Original: b/60 #fix answer=
Revised: b/60 #fix answer=

Original: c print (answer)
Revised: c print (answer)

Original: Problem
Revised: Problem

Original: [Table]
Revised: [Table]

Original: b
Revised: b

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: c=
Revised: c=

Original: [Equation]
Revised: [Equation]

Original: +b
Revised: +b

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: print (answer)
Revised: print (answer)

Original: [Table]
Revised: [Table]

Original: print
Revised: print

Original: [Figure]
Revised: [Figure]

Original: GSM8k (Cobbe et al., 2021).
Revised: GSM8k (Cobbe et al., 2021).

Original: As in prior work (Austin et al., 2021; Ni et al., 2022; Mishra et al., 2022), we frame these as program synthesis by converting their solutions to Python programs.
Revised: As in prior work (Austin et al., 2021; Ni et al., 2022; Mishra et al., 2022), we frame these as program synthesis by converting their solutions to Python programs.

Original: We separate our experiments into three increasingly difficult settings:
Revised: We separate our experiments into three increasingly difficult settings:

Original: MultiArith , using problems from the MultiArith arithmetic word problem dataset.
Revised: MultiArith , using problems from the MultiArith arithmetic word problem dataset.

Original: Multitask , using problems from 4 arithmetic datasets (MultiArith, AddSub, SingleOp, SVAMP).
Revised: Multitask , using problems from 4 arithmetic datasets (MultiArith, AddSub, SingleOp, SVAMP).

Original: GSM , using problems from the challenging GSM8k dataset.
Revised: GSM , using problems from the challenging GSM8k dataset.

Original: For the MultiArith and Multitask settings, we make train/valid/test splits using 60/20/20% of the respective datasets.
Revised: For the MultiArith and Multitask settings, we make train/valid/test splits using 60/20/20% of the respective datasets.

Original: Similar to Ni et al.
Revised: Similar to Ni et al.

Original: (2022), for the GSM setting we use the official GSM8k test split, and create a validation split using 20% of the training set.
Revised: (2022), for the GSM setting we use the official GSM8k test split, and create a validation split using 20% of the training set.

Original: Note that the problems and answers in all datasets are the same as those from the original non-program datasets.
Revised: Note that the problems and answers in all datasets are the same as those from the original non-program datasets.

Original: Baselines.
Revised: Baselines.

Original: We compare SELF - CORRECT with its fine-tuned baseline generator (GPT-Neo 1.3B) in all three settings.
Revised: We compare SELF - CORRECT with its fine-tuned baseline generator (GPT-Neo 1.3B) in all three settings.

Original: For the GSM setting, we compare with existing work that uses models within the same magnitude of scale, including NEO FCP+PCP (Ni et al., 2022), which tunes GPT-NEO 2.7B with additional self-sampled programs, and their fine-tuned GPT-NEO 2.7B baseline.
Revised: For the GSM setting, we compare with existing work that uses models within the same magnitude of scale, including NEO FCP+PCP (Ni et al., 2022), which tunes GPT-NEO 2.7B with additional self-sampled programs, and their fine-tuned GPT-NEO 2.7B baseline.

Original: We also report 3B and 6B fine-tuned GPT3-like language models from Cobbe et al.
Revised: We also report 3B and 6B fine-tuned GPT3-like language models from Cobbe et al.

Original: (2021), which were trained on the non-program version of GSM8k.
Revised: (2021), which were trained on the non-program version of GSM8k.

Original: We evaluate larger models later in (§3.4).
Revised: We evaluate larger models later in (§3.4).

Original: Results.
Revised: Results.

Original: As seen in Table 1, the self-corrector improves upon the generator in all three settings, using either inference strategy: always correcting ( SELF - CORRECT ), or only correcting incorrect solutions ( SELF - CORRECT ∗ ).
Revised: As seen in Table 1, the self-corrector improves upon the generator in all three settings, using either inference strategy: always correcting ( SELF - CORRECT ), or only correcting incorrect solutions ( SELF - CORRECT ∗ ).

Original: The self-corrector’s performance on Multiarith is very high after correction (9899%), a 38 point improvement over the generator, with a similar gain in the Multitask arithmetic setting.
Revised: The self-corrector’s performance on Multiarith is very high after correction (9899%), a 38 point improvement over the generator, with a similar gain in the Multitask arithmetic setting.

Original: On the challenging GSM dataset, the self-corrector achieves 21%, and 24% with only correcting incorrect solutions, up from 8.57% for the generator.
Revised: On the challenging GSM dataset, the self-corrector achieves 21%, and 24% with only correcting incorrect solutions, up from 8.57% for the generator.

Original: Notably, this is higher than the larger 2.7B GPT-Neo (also larger than generator+corrector), or larger models tuned on the language
Revised: Notably, this is higher than the larger 2.7B GPT-Neo (also larger than generator+corrector), or larger models tuned on the language

Original: [Table]
Revised: [Table]

Original: version of GSM.
Revised: version of GSM.

Original: The results show that self-corrective learning can improve task performance via training a corrector.
Revised: The results show that self-corrective learning can improve task performance via training a corrector.

Original: Qualitatively, the self-corrector can correct values in a correctly structured solution, fix the order of operations within a multistep solution, adjust unit conversions, and make larger multipart revisions (see Figures 3,7,8).
Revised: Qualitatively, the self-corrector can correct values in a correctly structured solution, fix the order of operations within a multistep solution, adjust unit conversions, and make larger multipart revisions (see Figures 3,7,8).

Original: Notably, these are learned automatically.
Revised: Notably, these are learned automatically.

Original: 3. L EXICALLY C ONSTRAINED G ENERATION
Revised: 3. L EXICALLY C ONSTRAINED G ENERATION

Original: Next, we consider lexically constrained generation.
Revised: Next, we consider lexically constrained generation.

Original: Given a set of constraint words x , the task is to generate a sentence y that includes all the given constraints.
Revised: Given a set of constraint words x , the task is to generate a sentence y that includes all the given constraints.

Original: Faithful constraint satisfaction is crucial for many downstream tasks, e.g., those that require converting information to text (McKeown, 1985).
Revised: Faithful constraint satisfaction is crucial for many downstream tasks, e.g., those that require converting information to text (McKeown, 1985).

Original: Datasets and Metrics.
Revised: Datasets and Metrics.

Original: We experiment on C OMMON G EN (Lin et al., 2020) and E2E (Novikova et al., 2017).
Revised: We experiment on C OMMON G EN (Lin et al., 2020) and E2E (Novikova et al., 2017).

Original: C OMMON G EN is a benchmark for generative commonsense reasoning where the task is to generate a coherent sentence given a set of words (e.g., dog, catch).
Revised: C OMMON G EN is a benchmark for generative commonsense reasoning where the task is to generate a coherent sentence given a set of words (e.g., dog, catch).

Original: E2E involves converting structured inputs into natural language.
Revised: E2E involves converting structured inputs into natural language.

Original: For both tasks, we report standard metrics including human/automatic measures of fluency (BLEU, CIDER, etc.)
Revised: For both tasks, we report standard metrics including human/automatic measures of fluency (BLEU, CIDER, etc.)

Original: as well as constraint coverage.
Revised: as well as constraint coverage.

Original: We collect human measures of fluency on Amazon Mechanical Turk; see the Appendix for details.
Revised: We collect human measures of fluency on Amazon Mechanical Turk; see the Appendix for details.

Original: Setup.
Revised: Setup.

Original: We parameterize the base generator with GPT-2 Radford et al.
Revised: We parameterize the base generator with GPT-2 Radford et al.

Original: (large-size for C OMMON G EN and medium-size for E2E).
Revised: (large-size for C OMMON G EN and medium-size for E2E).

Original: We fine-tuned the generator for each task.
Revised: We fine-tuned the generator for each task.

Original: As the value function for self-corrective learning we use coverage, i.e. the percentage of constraints that are present in the output.
Revised: As the value function for self-corrective learning we use coverage, i.e. the percentage of constraints that are present in the output.

Original: For inference, we use beam search with the generator, then do up to 3 corrections using beam search, stopping early if all constraints are met.
Revised: For inference, we use beam search with the generator, then do up to 3 corrections using beam search, stopping early if all constraints are met.

Original: See the Appendix for additional details.
Revised: See the Appendix for additional details.

Original: Results.
Revised: Results.

Original: Table 2 shows the evaluation results.
Revised: Table 2 shows the evaluation results.

Original: The self-corrector substantially improves constraint coverage over its GPT-2 generator for both tasks, while maintaining or improving its language quality.
Revised: The self-corrector substantially improves constraint coverage over its GPT-2 generator for both tasks, while maintaining or improving its language quality.

Original: On the C OMMON G EN benchmark, the self-corrector paired with the NeuroLogic constrained decoding algorithm (Lu et al., 2021) achieves the best results, outperforming the more sophisticated NeuroLogic-A* decoding algorithm, while being an order of magnitude faster.
Revised: On the C OMMON G EN benchmark, the self-corrector paired with the NeuroLogic constrained decoding algorithm (Lu et al., 2021) achieves the best results, outperforming the more sophisticated NeuroLogic-A* decoding algorithm, while being an order of magnitude faster.

Original: Notably, on
Revised: Notably, on

Original: E2E, self-correction outperforms Neurologic-A* decoding, despite only using standard beam search.
Revised: E2E, self-correction outperforms Neurologic-A* decoding, despite only using standard beam search.

Original: This suggests that a corrector can be viewed as an alternative to using a more sophisticated decoding procedure (A*) for improving performance without modifying the underlying model.
Revised: This suggests that a corrector can be viewed as an alternative to using a more sophisticated decoding procedure (A*) for improving performance without modifying the underlying model.

Original: See Figurefor qualitative examples.
Revised: See Figurefor qualitative examples.

Original: 3. T OXICITY R EDUCTION
Revised: 3. T OXICITY R EDUCTION

Original: Next, we consider the task of toxicity reduction (Gehman et al., 2020; Liu et al., 2021).
Revised: Next, we consider the task of toxicity reduction (Gehman et al., 2020; Liu et al., 2021).

Original: Given a prompt x , the task is to generate a fluent continuation y while avoiding offensive content.
Revised: Given a prompt x , the task is to generate a fluent continuation y while avoiding offensive content.

Original: This task is important for ensuring safe language model deployment, yet challenging: due to misaligned pretraining objectives (i.e. modeling internet text vs. non-toxic text), language models are susceptible to generating toxic completions, even when prompted with seemingly innocuous text (Gehman et al., 2020).
Revised: This task is important for ensuring safe language model deployment, yet challenging: due to misaligned pretraining objectives (i.e. modeling internet text vs. non-toxic text), language models are susceptible to generating toxic completions, even when prompted with seemingly innocuous text (Gehman et al., 2020).

Original: Along with its practical importance, the task tests whether (self-)correctors can be an effective mechanism for controlling the outputs of language models in an open-ended setting.
Revised: Along with its practical importance, the task tests whether (self-)correctors can be an effective mechanism for controlling the outputs of language models in an open-ended setting.

Original: Datasets and Metrics.
Revised: Datasets and Metrics.

Original: We use the R EAL T OXICITY P ROMPTS benchmark (Gehman et al., 2020) which contains 100k prompts designed to elicit toxic generations.
Revised: We use the R EAL T OXICITY P ROMPTS benchmark (Gehman et al., 2020) which contains 100k prompts designed to elicit toxic generations.

Original: Following the experimental setup of Liu et al.
Revised: Following the experimental setup of Liu et al.

Original: (2021), during training we use 85K prompts from the training set, and for evaluation
Revised: (2021), during training we use 85K prompts from the training set, and for evaluation

Original: we
Revised: we

Original: [Table]
Revised: [Table]

Original: use the same 10K non-toxic prompts from test set as Liu et al.
Revised: use the same 10K non-toxic prompts from test set as Liu et al.

Original: We use Perspective API to measure maximum toxicity , defined as the average maximum toxicity over 25 sampled generations, and the (empirical) toxicity probability of at least 1 out of 25 generations being toxic.
Revised: We use Perspective API to measure maximum toxicity , defined as the average maximum toxicity over 25 sampled generations, and the (empirical) toxicity probability of at least 1 out of 25 generations being toxic.

Original: Baselines.
Revised: Baselines.

Original: We compare S ELF -C ORRECT with its generator (GPT-2) and previously reported baselines from Lu et al.
Revised: We compare S ELF -C ORRECT with its generator (GPT-2) and previously reported baselines from Lu et al.

Original: (2022a), including PPLM (Dathathri et al., 2020), GeDi (Krause et al., 2021),
Revised: (2022a), including PPLM (Dathathri et al., 2020), GeDi (Krause et al., 2021),

Original: DExpert (Liu et al., 2020), DAPT (Gururangan et al., 2020), PPO (Lu et al., 2022a), and Quark (Lu et al., 2022a).
Revised: DExpert (Liu et al., 2020), DAPT (Gururangan et al., 2020), PPO (Lu et al., 2022a), and Quark (Lu et al., 2022a).

Original: The latter two – Proximal Policy Optimization (PPO) and Quantized Reward Konditioning (Quark) – represent strong, state-of-the art approaches based on reinforcement learning.
Revised: The latter two – Proximal Policy Optimization (PPO) and Quantized Reward Konditioning (Quark) – represent strong, state-of-the art approaches based on reinforcement learning.

Original: Setup.
Revised: Setup.

Original: We use the off-the-shelf GPT-

Revised: We use the off-the-shelf GPT-

Original: Large as the generator, and finetune another GPT-

Revised: Large as the generator, and finetune another GPT-

Original: Large as the corrector.
Revised: Large as the corrector.

Original: During inference, we use nucleus sampling with p = 0 .
Revised: During inference, we use nucleus sampling with p = 0 .

Original: 9 to generate 25 samples for all baselines.
Revised: 9 to generate 25 samples for all baselines.

Original: As the value function, we use the Perspective API score, v ( y ) ∈ [0 , 1] , which measures the toxicity of the completed sequence.
Revised: As the value function, we use the Perspective API score, v ( y ) ∈ [0 , 1] , which measures the toxicity of the completed sequence.

Original: We do up to three corrections with the corrector model.
Revised: We do up to three corrections with the corrector model.

Original: Results.
Revised: Results.

Original: Table 3 shows that S ELF -C ORRECT reduces the rate of toxic generations substantially, while also maintaining fluency and diversity.
Revised: Table 3 shows that S ELF -C ORRECT reduces the rate of toxic generations substantially, while also maintaining fluency and diversity.

Original: S ELF -C ORRECT outperforms all baselines.
Revised: S ELF -C ORRECT outperforms all baselines.

Original: This includes inference-time algorithms (PPLM, GeDi, DExpert), which do not modify the generator but degrade fluency and yield higher toxicity compared to S ELF -C ORRECT , as well as reinforcement learning methods (PPO, Quark) that adjust the generator using toxicity as a (negative) reward.
Revised: This includes inference-time algorithms (PPLM, GeDi, DExpert), which do not modify the generator but degrade fluency and yield higher toxicity compared to S ELF -C ORRECT , as well as reinforcement learning methods (PPO, Quark) that adjust the generator using toxicity as a (negative) reward.

Original: The strong baselines use equal or more parameters: PPO and Quark use 3 and 2 model copies.
Revised: The strong baselines use equal or more parameters: PPO and Quark use 3 and 2 model copies.

Original: The results show that S ELF -C ORRECT is effective for detoxification, without modifying the 
Revised: The results show that S ELF -C ORRECT is effective for detoxification, without modifying the generator.

Original: 3. C HANGING M ODULES – C ORRECTING GPT- 
Revised: 3. C HANGING M ODULES – C ORRECTING GPT-

Original: Next, we show that a self-corrector can improve the outputs of a generator that is much larger than the corrector.
Revised: Next, we show that a self-corrector can improve the outputs of a generator that is much larger than the corrector.

Original: We consider two cases: (1) training with a small generator, then swapping in the larger generator at test time; (2) training with the larger generator, i.e. using the large generator to initialize the datapool for self-corrective learning, then using the large generator at test time.
Revised: We consider two cases: (1) training with a small generator, then swapping in the larger generator at test time; (2) training with the larger generator, i.e. using the large generator to initialize the datapool for self-corrective learning, then using the large generator at test time.

Original: Toxicity.
Revised: Toxicity.

Original: We evaluate case (1) for reducing the toxicity of a large generator (GPT-2 XL, GPT-3).
Revised: We evaluate case (1) for reducing the toxicity of a large generator (GPT-2 XL, GPT-3).

Original: We generate an initial sequence using the large generator, then refine it with our corrector trained in the previous experiments (§3.3).
Revised: We generate an initial sequence using the large generator, then refine it with our corrector trained in the previous experiments (§3.3).

Original: Table 4 shows that the resulting self-corrector (large generator + corrector) has substantially reduced toxicity compared to the large generator.
Revised: Table 4 shows that the resulting self-corrector (large generator + corrector) has substantially reduced toxicity compared to the large generator.

Original: This shows the promise of using (self-)correctors for controlling the outputs of large language models.
Revised: This shows the promise of using (self-)correctors for controlling the outputs of large language models.

Original: Math program synthesis.
Revised: Math program synthesis.

Original: Table 4 shows results for math.
Revised: Table 4 shows results for math.

Original: Analogous to toxicity, the corrector is able to correct larger generators swapped in at test-time.
Revised: Analogous to toxicity, the corrector is able to correct larger generators swapped in at test-time.

Original: For instance, the GPT-3 Instruct generator has quite high performance (84.90 Multitask, 36.80 GSM), which improves to 90.90 and 45.00, respectively, by adding in a corrector.
Revised: For instance, the GPT-3 Instruct generator has quite high performance (84.90 Multitask, 36.80 GSM), which improves to 90.90 and 45.00, respectively, by adding in a corrector.

Original: The self-corrector (large generator + corrector) improves further by training with the GPT-3 Instruct generator, to 92.75 and 45.92, respectively.
Revised: The self-corrector (large generator + corrector) improves further by training with the GPT-3 Instruct generator, to 92.75 and 45.92, respectively.

Original: 3. L EVERAGING E XPLICIT F EEDBACK
Revised: 3. L EVERAGING E XPLICIT F EEDBACK

Original: Next, we demonstrate S ELF -C ORRECT ’s capacity to incorporate explicit natural language feedback.
Revised: Next, we demonstrate S ELF -C ORRECT ’s capacity to incorporate explicit natural language feedback.

Original: This amounts to defining a feedback function f , then using the same self-corrective learning and
Revised: This amounts to defining a feedback function f , then using the same self-corrective learning and

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: inference algorithms (§2.1) as in our preceding experiments (in those experiments, f returned ∅ ).
Revised: inference algorithms (§2.1) as in our preceding experiments (in those experiments, f returned ∅ ).

Original: We show that correctors learn to use the feedback, as evidenced by higher performance.
Revised: We show that correctors learn to use the feedback, as evidenced by higher performance.

Original: Toxicity.
Revised: Toxicity.

Original: We use additional fine-grained information from the toxicity API as natural language feedback.
Revised: We use additional fine-grained information from the toxicity API as natural language feedback.

Original: Specifically, besides the overall toxicity score, Perspective API also provides scores for fine-grained attributes of toxicity (e.g. identity attack, profanity, flirtation, etc.).
Revised: Specifically, besides the overall toxicity score, Perspective API also provides scores for fine-grained attributes of toxicity (e.g. identity attack, profanity, flirtation, etc.).

Original: At training time, we compare the attribute scores from a hypothesis and its selected correction, and use the attribute with the largest decrease as natural language feedback (e.g. "decrease toxicity in profanity ").
Revised: At training time, we compare the attribute scores from a hypothesis and its selected correction, and use the attribute with the largest decrease as natural language feedback (e.g. "decrease toxicity in profanity ").

Original: At inference time, we call the API on the current hypothesis, and use the attribute with the highest score.
Revised: At inference time, we call the API on the current hypothesis, and use the attribute with the highest score.

Original: Here we use the API at inference time, which is not required in our previous experiments.
Revised: Here we use the API at inference time, which is not required in our previous experiments.

Original: Lexical constraints.
Revised: Lexical constraints.

Original: In training time, we generate natural language feedback for every example pair ( x, y, y ′ ) by elaborating the extra lexical constraints satisfied by y ′ but not y .
Revised: In training time, we generate natural language feedback for every example pair ( x, y, y ′ ) by elaborating the extra lexical constraints satisfied by y ′ but not y .

Original: e.g. “adding constraint word: read” .
Revised: e.g. “adding constraint word: read” .

Original: At inference time, we elaborate all missing constraints in the current hypothesis.
Revised: At inference time, we elaborate all missing constraints in the current hypothesis.

Original: Math program synthesis.
Revised: Math program synthesis.

Original: Math program synthesis contains a variety of problem types and errors, without an automated means for identifying the errors (e.g. an API).
Revised: Math program synthesis contains a variety of problem types and errors, without an automated means for identifying the errors (e.g. an API).

Original: We explore obtaining natural language feedback about the current program by prompting a large language model.
Revised: We explore obtaining natural language feedback about the current program by prompting a large language model.

Original: We prompt the model with a problem, hypothesis program, a gold solution, and few-shot demonstrations that show feedback on one part of the program; e.g. In the initial guess, 3 should be subtracted.
Revised: We prompt the model with a problem, hypothesis program, a gold solution, and few-shot demonstrations that show feedback on one part of the program; e.g. In the initial guess, 3 should be subtracted.

Original: When the program is correct, the feedback is Correct.
Revised: When the program is correct, the feedback is Correct.

Original: At inference time, we also use feedback from the language model.
Revised: At inference time, we also use feedback from the language model.

Original: We allow the feedback model access to a gold solution, which we expect makes the feedback higher quality, with the risk of solution leakage at inference-time.
Revised: We allow the feedback model access to a gold solution, which we expect makes the feedback higher quality, with the risk of solution leakage at inference-time.

Original: Our results in this task are thus used only to study the feasibility of explicit feedback for math program synthesis.
Revised: Our results in this task are thus used only to study the feasibility of explicit feedback for math program synthesis.

Original: Setup.
Revised: Setup.

Original: For toxicity, lexical constraints, and math we use R EAL T OXICITY P ROMPTS , C OMMON G EN , and the M ULTITASK arithmetic setting, respectively.
Revised: For toxicity, lexical constraints, and math we use R EAL T OXICITY P ROMPTS , C OMMON G EN , and the M ULTITASK arithmetic setting, respectively.

Original: We follow the setup of each task’s previous experiments (§3.3,§3.2,§3.1), except for math we use 5 correction iterations (previously 1).
Revised: We follow the setup of each task’s previous experiments (§3.3,§3.2,§3.1), except for math we use 5 correction iterations (previously 1).

Original: For math, we use GPT-3 ( as the model.
Revised: For math, we use GPT-3 ( as the model.

Original: Results.
Revised: Results.

Original: Table 5 shows that explicit natural language feedback improves performance in all three tasks.
Revised: Table 5 shows that explicit natural language feedback improves performance in all three tasks.

Original: For toxicity, this means that providing fine-grained attributes (e.g. identity attack, profanity, etc.)
Revised: For toxicity, this means that providing fine-grained attributes (e.g. identity attack, profanity, etc.)

Original: during learning and inference improves upon using only the scalar toxicity score.
Revised: during learning and inference improves upon using only the scalar toxicity score.

Original: Intuitively, feedback may help the model to focus on a useful correction; e.g., see Figure 6.
Revised: Intuitively, feedback may help the model to focus on a useful correction; e.g., see Figure 6.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3. A DDITIONAL A BLATIONS AND A NALYSIS
Revised: 3. A DDITIONAL A BLATIONS AND A NALYSIS

Original: Effect of multiple corrections.
Revised: Effect of multiple corrections.

Original: Previously, Figure 4 showed that multiple corrections led to better toxicity reduction.
Revised: Previously, Figure 4 showed that multiple corrections led to better toxicity reduction.

Original: On math (Multitask setting), Figure 5 shows that performance improves with more than one correction, and that multiple corrections are more beneficial with feedback.
Revised: On math (Multitask setting), Figure 5 shows that performance improves with more than one correction, and that multiple corrections are more beneficial with feedback.

Original: Intuitively, in this math task, after 2-3 corrections the model needs additional guidance.
Revised: Intuitively, in this math task, after 2-3 corrections the model needs additional guidance.

Original: Effect of pairing and proportional sampling.
Revised: Effect of pairing and proportional sampling.

Original: Self-corrective learning (i) samples pairs for learning proportional to Equation 4, (ii) only pairs sequences that improve value.
Revised: Self-corrective learning (i) samples pairs for learning proportional to Equation 4, (ii) only pairs sequences that improve value.

Original: We ablate these features by training on Multitask using a data pool that samples a pair for learning uniformly (rather than
Revised: We ablate these features by training on Multitask using a data pool that samples a pair for learning uniformly (rather than

Original: Equation 4), and a data pool without value pairing.
Revised: Equation 4), and a data pool without value pairing.

Original: Table 6 shows that both improve performance.
Revised: Table 6 shows that both improve performance.

Original: Effect of exploration.
Revised: Effect of exploration.

Original: To ablate the effect of exploration, we train a baseline only on correction pairs induced from the base generator.
Revised: To ablate the effect of exploration, we train a baseline only on correction pairs induced from the base generator.

Original: Table 7 shows results on the three math datasets, indicating that exploration improves performance.
Revised: Table 7 shows results on the three math datasets, indicating that exploration improves performance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Self-Correction relates to work modeling text edits including supervised Wikipedia edits (Reid &
Revised: Self-Correction relates to work modeling text edits including supervised Wikipedia edits (Reid &

Original: Neubig, 2022; Faltings et al., 2021; Schick et al., 2022), unsupervised perturbations (e.g. insertion, deletion)(Miao et al., 2019; Liu et al., 2020), training on human-written critiques (Saunders et al., 2022), or iterating on continuous variables (Lee et al., 2020; Li et al., 2022; Qin et al., 2022).
Revised: Neubig, 2022; Faltings et al., 2021; Schick et al., 2022), unsupervised perturbations (e.g. insertion, deletion)(Miao et al., 2019; Liu et al., 2020), training on human-written critiques (Saunders et al., 2022), or iterating on continuous variables (Lee et al., 2020; Li et al., 2022; Qin et al., 2022).

Original: In contrast, Self-Correction learns a text corrector online to improve a quality measure without supervised edits or critiques.
Revised: In contrast, Self-Correction learns a text corrector online to improve a quality measure without supervised edits or critiques.

Original: Denoising sequences is a common pretraining objective (Devlin et al., 2019;
Revised: Denoising sequences is a common pretraining objective (Devlin et al., 2019;

Original: Lewis et al., 2020; Raffel et al., 2020), while self-correction ‘denoises’ generations to improve a scalar quality measure.
Revised: Lewis et al., 2020; Raffel et al., 2020), while self-correction ‘denoises’ generations to improve a scalar quality measure.

Original: Reinforcement learning (RL) is often used to improve scalar measures in a generator (Ziegler et al., 2019; Stiennon et al., 2020; Lu et al., 2022a), yet is infeasible for many models (e.g. those accessed by API), and uses only scalar feedback.
Revised: Reinforcement learning (RL) is often used to improve scalar measures in a generator (Ziegler et al., 2019; Stiennon et al., 2020; Lu et al., 2022a), yet is infeasible for many models (e.g. those accessed by API), and uses only scalar feedback.

Original: Moreover, Self-Correction learns a delta between generator and solution, and thus RL-tuned generators can be used with SelfCorrection .
Revised: Moreover, Self-Correction learns a delta between generator and solution, and thus RL-tuned generators can be used with SelfCorrection .

Original: Self-Correction decomposes generation into multiple steps similar to methods that generate rationales before a response (Wei et al., 2022; Dohan et al., 2022), yet intermediate steps of the same form as the output, allowing iterative application.
Revised: Self-Correction decomposes generation into multiple steps similar to methods that generate rationales before a response (Wei et al., 2022; Dohan et al., 2022), yet intermediate steps of the same form as the output, allowing iterative application.

Original: Self-Correction relates to work on program synthesis (Fu et al., 2019;
Revised: Self-Correction relates to work on program synthesis (Fu et al., 2019;

Original: Balog et al., 2020; Gupta et al., 2020; Le et al., 2022) and repair (Gupta et al., 2020; Yasunaga & Liang, 2020; 2021).
Revised: Balog et al., 2020; Gupta et al., 2020; Le et al., 2022) and repair (Gupta et al., 2020; Yasunaga & Liang, 2020; 2021).

Original: Yasunaga & Liang (2021) is closest in methodology, but Self-Correction uses a domain-agnostic formulation; see Appendix for a discussion.
Revised: Yasunaga & Liang (2021) is closest in methodology, but Self-Correction uses a domain-agnostic formulation; see Appendix for a discussion.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced self-correctors, a class of models that decompose generation into initial generation and correction steps.
Revised: We introduced self-correctors, a class of models that decompose generation into initial generation and correction steps.

Original: We study self-correctors with a fixed base generator along with a corrector trained to improve outputs according to a scalar measure of quality.
Revised: We study self-correctors with a fixed base generator along with a corrector trained to improve outputs according to a scalar measure of quality.

Original: We presented a simple, general procedure for training the corrector, and find that self-correction is applicable and effective for improving performance, and controlling the outputs of both small and large generators.
Revised: We presented a simple, general procedure for training the corrector, and find that self-correction is applicable and effective for improving performance, and controlling the outputs of both small and large generators.

Original: Moreover, we found that self-correction along with our learning framework provides a promising mechanism for using natural language feedback to improve generation.
Revised: Moreover, we found that self-correction along with our learning framework provides a promising mechanism for using natural language feedback to improve generation.

Original: These findings, along with exploring alternative self-correctors, open up many avenues that we leave for future work.
Revised: These findings, along with exploring alternative self-correctors, open up many avenues that we leave for future work.

Original: A PPENDIX A R ELATED W ORK
Revised: A PPENDIX A R ELATED W ORK

Original: Self-correction provides a flexible framework for improving the performance of off-the-shelf and fine-tuned language models on a wide range of tasks by decomposing generation into a base generator and a corrector.
Revised: Self-correction provides a flexible framework for improving the performance of off-the-shelf and fine-tuned language models on a wide range of tasks by decomposing generation into a base generator and a corrector.

Original: Our framework’s minimal assumptions on the form of the corrector, value function, and data used to train the corrector, as well as its wide applicability differ from prior work.
Revised: Our framework’s minimal assumptions on the form of the corrector, value function, and data used to train the corrector, as well as its wide applicability differ from prior work.

Original: Learning to fix code.
Revised: Learning to fix code.

Original: Our work relates to two streams of research in the code domain.
Revised: Our work relates to two streams of research in the code domain.

Original: One stream deals with program synthesis , in which a corrector model corrects code from a base synthesizer until it meets a given specification (Fu et al., 2019; Balog et al., 2020; Gupta et al., 2020; Le et al., 2022), while another stream deals with program repair : correcting code that is provided as input (Gupta et al., 2020; Yasunaga & Liang, 2020; 2021).
Revised: One stream deals with program synthesis , in which a corrector model corrects code from a base synthesizer until it meets a given specification (Fu et al., 2019; Balog et al., 2020; Gupta et al., 2020; Le et al., 2022), while another stream deals with program repair : correcting code that is provided as input (Gupta et al., 2020; Yasunaga & Liang, 2020; 2021).

Original: Recently, Le et al.
Revised: Recently, Le et al.

Original: (2022) developed a modular program synthesis approach that involves a correction module trained on ground-truth outputs.
Revised: (2022) developed a modular program synthesis approach that involves a correction module trained on ground-truth outputs.

Original: In contrast, self-corrective learning supports cases without ground-truth outputs, e.g. toxicity.
Revised: In contrast, self-corrective learning supports cases without ground-truth outputs, e.g. toxicity.

Original: Closest to our methodology is Yasunaga & Liang (2021).
Revised: Closest to our methodology is Yasunaga & Liang (2021).

Original: Unlike Yasunaga & Liang (2021), selfcorrection does not assume a mechanism for generating synthetic negatives, a dataset of negatives, or a separate model that generates negatives.
Revised: Unlike Yasunaga & Liang (2021), selfcorrection does not assume a mechanism for generating synthetic negatives, a dataset of negatives, or a separate model that generates negatives.

Original: This is important because engineering these components for each new task can be prohibitive.
Revised: This is important because engineering these components for each new task can be prohibitive.

Original: Second, Yasunaga & Liang (2021) assume a 0/1 value function, while self-correction supports general scalar value functions.
Revised: Second, Yasunaga & Liang (2021) assume a 0/1 value function, while self-correction supports general scalar value functions.

Original: This is important for tasks such as toxicity that do not have a strict notion of correctness.
Revised: This is important for tasks such as toxicity that do not have a strict notion of correctness.

Original: Finally, we propose new pairing and proportional sampling mechanisms found to be important (Table 6).
Revised: Finally, we propose new pairing and proportional sampling mechanisms found to be important (Table 6).

Original: Iterative text edits.
Revised: Iterative text edits.

Original: Self-correction relates to recent works on editing text, including modeling
Revised: Self-correction relates to recent works on editing text, including modeling

Original: Wikipedia edits (Reid & Neubig, 2022; Faltings et al., 2021; Schick et al., 2022), which relies on supervised edits, unsupervised methods (Miao et al., 2019; Liu et al., 2020) that perturb sequences with simple operations (e.g. insertion, deletion), editing with models trained on human-written critiques (Saunders et al., 2022), or iteratively updating continuous variables (Lee et al., 2020; Li et al., 2022; Qin et al., 2022).
Revised: Wikipedia edits (Reid & Neubig, 2022; Faltings et al., 2021; Schick et al., 2022), which relies on supervised edits, unsupervised methods (Miao et al., 2019; Liu et al., 2020) that perturb sequences with simple operations (e.g. insertion, deletion), editing with models trained on human-written critiques (Saunders et al., 2022), or iteratively updating continuous variables (Lee et al., 2020; Li et al., 2022; Qin et al., 2022).

Original: In contrast to these, self-correction learns an expressive text-totext corrector that is trained online to improve a quality measure, without requiring a supervised dataset of edits or critiques.
Revised: In contrast to these, self-correction learns an expressive text-totext corrector that is trained online to improve a quality measure, without requiring a supervised dataset of edits or critiques.

Original: Finally, correcting text is inherent to the task of grammatical error correction (e.g. Lichtarge et al.
Revised: Finally, correcting text is inherent to the task of grammatical error correction (e.g. Lichtarge et al.

Original: ; Yasunaga et al.
Revised: ; Yasunaga et al.

Original: (2021); our work differs in that we correct a module within a generation system, and provide a framework for addressing a variety of tasks.
Revised: (2021); our work differs in that we correct a module within a generation system, and provide a framework for addressing a variety of tasks.

Original: Denoising and reinforcement learning.
Revised: Denoising and reinforcement learning.

Original: Separately, denoising ground-truth sequences is a common pretraining objective (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020), while selfcorrection ‘denoises’ generations to improve a scalar quality measure.
Revised: Separately, denoising ground-truth sequences is a common pretraining objective (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020), while selfcorrection ‘denoises’ generations to improve a scalar quality measure.

Original: Scalar measures are often improved with reinforcement learning (RL) on a base generator (Ziegler et al., 2019; Stiennon et al., 2020; Lu et al., 2022a), which is infeasible for improving many language models (e.g. those accessed through an API), and uses only scalar feedback.
Revised: Scalar measures are often improved with reinforcement learning (RL) on a base generator (Ziegler et al., 2019; Stiennon et al., 2020; Lu et al., 2022a), which is infeasible for improving many language models (e.g. those accessed through an API), and uses only scalar feedback.

Original: Moreover, self-correction learns a delta between a generation and solution, and is complementary to RL-tuned generators, which can be used within a self-corrector.
Revised: Moreover, self-correction learns a delta between a generation and solution, and is complementary to RL-tuned generators, which can be used within a self-corrector.

Original: Finally, RL can be used as an alternative learning algorithm for training a corrector, which is an interesting direction for future work.
Revised: Finally, RL can be used as an alternative learning algorithm for training a corrector, which is an interesting direction for future work.

Original: Modular generation.
Revised: Modular generation.

Original: Self-correction decomposes generation into multiple steps, and is thus part of the general class of methods that decompose generation into a ‘cascade’ of modules (Dohan et al., 2022).
Revised: Self-correction decomposes generation into multiple steps, and is thus part of the general class of methods that decompose generation into a ‘cascade’ of modules (Dohan et al., 2022).

Original: Examples include using separate knowledge generation modules (Shwartz et al., 2020;
Revised: Examples include using separate knowledge generation modules (Shwartz et al., 2020;

Original: Liu et al., 2022), or generating rationales before a response (Wei et al., 2022).
Revised: Liu et al., 2022), or generating rationales before a response (Wei et al., 2022).

Original: Self-correction also produces a chain of intermediate steps, but each step is of the same form as the output, allowing for re-using previous generations.
Revised: Self-correction also produces a chain of intermediate steps, but each step is of the same form as the output, allowing for re-using previous generations.

Original: B A DDITIONAL E XPERIMENTAL D ETAILS B. C ROSS -E XPERIMENT D ETAILS
Revised: B A DDITIONAL E XPERIMENTAL D ETAILS B. C ROSS -E XPERIMENT D ETAILS

Original: In all of our experiments we use an off-the-shelf embedding similarity function from SentenceTransformers (Reimers & Gurevych, 2019): sentence-transformers/all-MiniLM-L6-v2 .
Revised: In all of our experiments we use an off-the-shelf embedding similarity function from SentenceTransformers (Reimers & Gurevych, 2019): sentence-transformers/all-MiniLM-L6-v2 .

Original: B. M ATHEMATICAL P ROGRAM S YNTHESIS
Revised: B. M ATHEMATICAL P ROGRAM S YNTHESIS

Original: We fine-tune a separate instance of GPT-Neo 1.3B as an initial generator, using the Huggingface library with default hyperparameters, except for evaluation steps, which we set to a small number to ensure a strong checkpoint is selected for each dataset.
Revised: We fine-tune a separate instance of GPT-Neo 1.3B as an initial generator, using the Huggingface library with default hyperparameters, except for evaluation steps, which we set to a small number to ensure a strong checkpoint is selected for each dataset.

Original: We use the finetuned initial generator as initialization for the corrector, and tune the corrector on sequences [SC]x[CURR]yi[START]yj[END] , where x is a problem, y i and y j form a residual pair, and [ · ] are special tokens.
Revised: We use the finetuned initial generator as initialization for the corrector, and tune the corrector on sequences [SC]x[CURR]yi[START]yj[END] , where x is a problem, y i and y j form a residual pair, and [ · ] are special tokens.

Original: The loss is on tokens after [START] .
Revised: The loss is on tokens after [START] .

Original: Feedback.
Revised: Feedback.

Original: We write 6 demonstrations using training problems and generations from our GPTNeo base generator, and use GPT-3 (text-davinci-002) as a feedback model.
Revised: We write 6 demonstrations using training problems and generations from our GPTNeo base generator, and use GPT-3 (text-davinci-002) as a feedback model.

Original: We use the same training procedure and hyperparameters, except that the sequences now include feedback, [SC]x[CURR]yi[FEEDBACK]F(x,yi)[START]yj[END] , where x is a problem, y i and y j form a residual pair, and F ( x, y i ) is feedback.
Revised: We use the same training procedure and hyperparameters, except that the sequences now include feedback, [SC]x[CURR]yi[FEEDBACK]F(x,yi)[START]yj[END] , where x is a problem, y i and y j form a residual pair, and F ( x, y i ) is feedback.

Original: We include loss on tokens after [FEEDBACK] .
Revised: We include loss on tokens after [FEEDBACK] .

Original: B. L EXICALLY - CONSTRAINED G ENERATION
Revised: B. L EXICALLY - CONSTRAINED G ENERATION

Original: Hyper-parameters.
Revised: Hyper-parameters.

Original: Table 8 and Table 9 show hyperparameters for CommonGen and E2E.
Revised: Table 8 and Table 9 show hyperparameters for CommonGen and E2E.

Original: Human
Revised: Human

Original: Evaluation
Revised: Evaluation

Original: We evaluate fluency of generations in E2E task using human annotators on Amazon Mechanical Turk (AMT).
Revised: We evaluate fluency of generations in E2E task using human annotators on Amazon Mechanical Turk (AMT).

Original: We randomly sampled 100 instances, along with generations of different baselines and self-corrections.
Revised: We randomly sampled 100 instances, along with generations of different baselines and self-corrections.

Original: For each instance, we ask 3 annotators to evaluate the fluency of generations on a 3-point Likert scale.
Revised: For each instance, we ask 3 annotators to evaluate the fluency of generations on a 3-point Likert scale.

Original: We aggregate annotations from 3 annotators using majority vote.
Revised: We aggregate annotations from 3 annotators using majority vote.

Original: We restricted the pool of annotators to those who are located in US or CA, and had 98% approval rate for at least 5,000 previous annotations
Revised: We restricted the pool of annotators to those who are located in US or CA, and had 98% approval rate for at least 5,000 previous annotations

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C A DDITIONAL R ESULTS
Revised: C A DDITIONAL R ESULTS

Original: [Table]
Revised: [Table]

Original: D Q UALITATIVE E XAMPLES
Revised: D Q UALITATIVE E XAMPLES

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: Feedback (
Revised: Feedback (

Original: [Equation]
Revised: [Equation]

Original: 3): # In the initial guess, # 2 tickets are not included.
Revised: 3): # In the initial guess, # 2 tickets are not included.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Problem:
Revised: Problem:

Original: Ralph watches TV for 4 hours a day from Monday to Friday, and 6 hours a day on Saturday and Sunday.
Revised: Ralph watches TV for 4 hours a day from Monday to Friday, and 6 hours a day on Saturday and Sunday.

Original: How many hours does Ralph spend watching TV in one week?
Revised: How many hours does Ralph spend watching TV in one week?

Original: Generator:
Revised: Generator:

Original: [Figure]
Revised: [Figure]

Original: print (answer)
Revised: print (answer)

Original: Corrector:
Revised: Corrector:

Original: [Figure]
Revised: [Figure]

Original: print (answer)
Revised: print (answer)

Original: Problem:
Revised: Problem:

Original: The pirates plan to explore 4 islands.
Revised: The pirates plan to explore 4 islands.

Original: Two is- lands require walking 20 miles per day while the other two islands require 25 miles per day.
Revised: Two is- lands require walking 20 miles per day while the other two islands require 25 miles per day.

Original: How many miles will they have to walk if it takes 1.5 days to explore each island?
Revised: How many miles will they have to walk if it takes 1.5 days to explore each island?

Original: Generator:
Revised: Generator:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Corrector:
Revised: Corrector:

Original: [Figure]
Revised: [Figure]

Original: Problem:
Revised: Problem:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: print (answer)
Revised: print (answer)

Original: Corrector:
Revised: Corrector:

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure] 
Revised: [Figure] [Equation] [Figure] [Equation]

Original:  [Equation] [Figure]
Revised: [Figure] [Equation] [Figure]

Original: [Equation][Figure] [Equation] [Figure]
Revised: [Equation]

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: print (answer)
Revised: print (answer)

Original: Problem:
Revised: Problem:

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]