Original: G ENERATING S EQUENCES BY L EARNING TO [S ELF -]C ORRECT
Revised: G ENERATING S EQUENCES BY L EARNING TO [S ELF -]C ORRECT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content.
Revised: Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content.

Original: Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack a mechanism to iteratively revise their outputs.
Revised: Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack a mechanism to iteratively revise their outputs.

Original: Moreover, some powerful language models are of extreme scale or inaccessible, making it inefficient, if not infeasible, to update their parameters for task-specific adaptation.
Revised: Moreover, some powerful language models are of extreme scale or inaccessible, making it inefficient, if not infeasible, to update their parameters for task-specific adaptation.

Original: We present SELF - CORRECTION , an approach that decouples an imperfect base generator (an off-the-shelf language model or supervised sequence-to-sequence model) from a separate corrector that learns to iteratively correct imperfect generations.
Revised: We present SELF - CORRECTION , an approach that decouples an imperfect base generator (an off-the-shelf language model or supervised sequence-to-sequence model) from a separate corrector that learns to iteratively correct imperfect generations.

Original: To train the corrector, we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations.
Revised: To train the corrector, we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations.

Original: We show that SELFCORRECTION improves upon the base generator in three diverse generation tasks– mathematical program synthesis, lexically-constrained generation, and toxicity control– even when the corrector is much smaller than the base generator.
Revised: We show that SELFCORRECTION improves upon the base generator in three diverse generation tasks– mathematical program synthesis, lexically-constrained generation, and toxicity control– even when the corrector is much smaller than the base generator.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The standard practice for natural language generation tasks is inherently single-pass: applying a decoding procedure to either a few-shot prompted language model or one tuned for a given task, then considering the generation as “finished” (e.g. Radford et al.
Revised: The standard practice for natural language generation tasks is inherently single-pass: applying a decoding procedure to either a few-shot prompted language model or one tuned for a given task, then considering the generation as “finished” (e.g. Radford et al.

Original: ; Brown et al.
Revised: ; Brown et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: Powerful generation models often meet most of the task requirements, yet miss a few (e.g., omitting a subset of keywords), or generate incorrect hypotheses that nevertheless provide useful structure (e.g., a correct problem solving strategy with a missing step).
Revised: Powerful generation models often meet most of the task requirements, yet miss a few (e.g., omitting a subset of keywords), or generate incorrect hypotheses that nevertheless provide useful structure (e.g., a correct problem solving strategy with a missing step).

Original: However, after generating even a slightly sub-optimal sequence, the single-pass paradigm requires models to “start from scratch”, effectively discarding work already done.
Revised: However, after generating even a slightly sub-optimal sequence, the single-pass paradigm requires models to “start from scratch”, effectively discarding work already done.

Original: A more natural, intuitive approach is leveraging the generation as a useful starting point to refine into a higher quality output.
Revised: A more natural, intuitive approach is leveraging the generation as a useful starting point to refine into a higher quality output.

Original: To formalize this intuition, we introduce Self-Correction for Sequence Generation .
Revised: To formalize this intuition, we introduce Self-Correction for Sequence Generation .

Original: Figuredemonstrates its central principle: a generation model is re-framed as a base generator , which produces a reasonable initial hypothesis but does not need to solve the task in one pass, and a second module–the corrector –trained to make up the difference between the hypothesis and an optimal solution.
Revised: Figuredemonstrates its central principle: a generation model is re-framed as a base generator , which produces a reasonable initial hypothesis but does not need to solve the task in one pass, and a second module–the corrector –trained to make up the difference between the hypothesis and an optimal solution.

Original: Neither the generator nor the corrector must solve the full task in one pass, and the corrector can be applied multiple times to iteratively improve the output (§3.6).
Revised: Neither the generator nor the corrector must solve the full task in one pass, and the corrector can be applied multiple times to iteratively improve the output (§3.6).

Original: We propose a simple, general procedure for training the corrector (Figure 2) by pairing generator outputs with carefully selected targets.
Revised: We propose a simple, general procedure for training the corrector (Figure 2) by pairing generator outputs with carefully selected targets.

Original: The result is a system which self-corrects, producing outputs through multiple generation passes and breaking the task into steps that can be solved by dedicated and efficient sub-systems.
Revised: The result is a system which self-corrects, producing outputs through multiple generation passes and breaking the task into steps that can be solved by dedicated and efficient sub-systems.

Original: We find that Self-Correction is broadly applicable.
Revised: Self-Correction builds on past work for correction in the code and text (e.g. Yasunaga et al. Faltings et al. (2021)) domains, but provides a unified formalism with minimal assumptions about data and feedback, which applies generally to diverse tasks.

Original: Training a corrector model improves the base generator on 3 diverse tasks: mathematical program synthesis (§3.1), lexically constrained generation (§3.2), and toxicity reduction (§3.3).
Revised: A corrector model improves the base generator on 3 such tasks in our experiments: mathematical program synthesis (§3.1), lexically constrained generation (§3.2), and toxicity reduction (§3.3).

Original: The trained corrector modelcan even be applied to a larger generator with similar performance to training a new corrector (§3.4), showing that the subtask of correction is transferable, even to stronger generators.
Revised: The trained corrector model even transfers to a larger generator with similar performance to training from scratch (§3.4).

Original: Finally, we explorethe prospect of introducing a third module to the Self-Correction system (§3.5)–explicitly using natural language feedback to guide corrections–with promising results.
Revised: Finally, we explore introducing a third module to the Self-Correction system (§3.5)–explicitly using natural language feedback to guide corrections–with promising results.

Original: Self-Correction offers an exciting opportunity to build onexisting generation models and the sequences they generate, with efficient, effective, and transferable corrector networks.
Revised: Self-Correction is an exciting path to build on the generations of strong models, with efficient, effective, and transferable corrector networks.

Original:  S ELF - CORRECTING SEQUENCE GENERATORS
Revised:  S ELF - CORRECTING SEQUENCE GENERATORS

Original: A typical autoregressive text generator (e.g. GPT-3 (Brown et al., 2020)) maps an input prompt to a distribution over outputs using a single parameterized module (e.g. a large transformer), p 

Revised: A typical autoregressive text generator (e.g. GPT-3 (Brown et al., 2020)) maps an input prompt to a distribution over outputs using a single parameterized module (e.g. a large transformer), p

Original: ( y | x ) .
Revised: ( y | x ) .

Original: We explore an alternative that decomposes into two modules, a base generator , and a corrector ,
Revised: We explore an alternative that decomposes into two modules, a base generator , and a corrector ,

Original: [Equation]
Revised: [Equation]

Original: where the generator provides an initial hypothesis that is refined by the corrector.
Revised: where the generator provides an initial hypothesis that is refined by the corrector.

Original: In practice, the corrector can be applied multiple times, p
Revised: In practice, the corrector can be applied multiple times, p

Original: [Equation]
Revised: [Equation]

Original: Since a model of this form can both generate and correct its generations, we call it a Self-Corrector .
Revised: Since a model of this form can both generate and correct its generations, we call it a Self-Corrector .

Original: Self-correctors have several unique properties compared to typical generators.
Revised: Self-correctors have several unique properties compared to typical generators.

Original: First, a self-corrector decouples generation and correction, allowing us to freely parameterize each module – for instance, by prompting a single language model or using two different language models.
Revised: First, a self-corrector decouples generation and correction, allowing us to freely parameterize each module – for instance, by prompting a single language model or using two different language models.

Original: In this paper, we develop a framework to train a separate corrector model (§2.1).
Revised: In this paper, we develop a framework to train a separate corrector model (§2.1).

Original: We find that the resulting selfcorrector improves upon the generator alone (§3), even when the corrector is much smaller (§3.4).
Revised: We find that the resulting selfcorrector improves upon the generator alone (§3), even when the corrector is much smaller (§3.4).

Original: Second, since the generator and the corrector are separated, we can keep the generator as a generalpurpose language model and train the corrector with different objectives for different task requirements.
Revised: Second, since the generator and the corrector are separated, we can keep the generator as a generalpurpose language model and train the corrector with different objectives for different task requirements.

Original: In §2.1, we propose a training algorithm for the corrector that is dedicated to improving generations, where the improvement can be in any aspect, measured by scalar values.
Revised: In §2.1, we propose a training algorithm for the corrector that is dedicated to improving generations, where the improvement can be in any aspect, measured by scalar values.

Original: Third, the corrector can receive explicit feedback about intermediate generations to guide subsequent generations.
Revised: Third, the corrector can receive explicit feedback about intermediate generations to guide subsequent generations.

Original: Formally, p ( y | x ) =
Revised: Formally, p ( y | x ) =

Original: (cid:80)
Revised: (cid:80)

Original: y
Revised: y

Original: 0 p 0 ( y 0 | x ) p θ ( y | y 0 , x, f ( y 0 )) , where f is the feedback.
Revised: 0 p 0 ( y 0 | x ) p θ ( y | y 0 , x, f ( y 0 )) , where f is the feedback.

Original: The feedback can be of many forms, e.g. a sentence, a compiler trace, etc.
Revised: The feedback can be of many forms, e.g. a sentence, a compiler trace, etc.

Original: In contrast, a typical generator that generates in a single pass does not leverage feedback on its own generation.
Revised: In contrast, a typical generator that generates in a single pass does not leverage feedback on its own generation.

Original: In this paper, we show that the corrector can learn to exploit explicit natural language feedback to achieve better performance (§3.5).
Revised: In this paper, we show that the corrector can learn to exploit explicit natural language feedback to achieve better performance (§3.5).

Original: Next, we describe our training framework of the corrector.
Revised: Next, we describe our training framework of the corrector.

Original: 2. L EARNING A C ORRECTOR
Revised: 2. L EARNING A C ORRECTOR

Original: Our goal is to have the generator generate an initial hypothesis, then improve the hypothesis with the corrector (Eq. 1).
Revised: Our goal is to have the generator generate an initial hypothesis, then improve the hypothesis with the corrector (Eq. 1).

Original: We train the corrector to improve the quality of a hypothesis, while staying as close as possible to the original hypothesis.
Revised: We train the corrector to improve the quality of a hypothesis, while staying as close as possible to the original hypothesis.

Original: Here, quality is measured with a scalar value function v ( y ) whichwe assume is accessible at training time (e.g.  a classifier).
Revised: Here, quality is measured with a scalar value function v ( y ) which is accessible at training time (e.g. 0/1 indicator of program correctness, a toxicity score).

Original: Since direct supervision on how to improve hypotheses is not available, we design a new algorithm to train the corrector, which we refer to as self-corrective learning.
Revised: Since direct supervision on how to improve hypotheses is not available, we design a new algorithm to train the corrector, which we refer to as self-corrective learning.

Original: The algorithm collects a pool of generations, groups them and selects pairs of generation that increase in value and are nearby, then updates the corrector on these pairs.
Revised: The algorithm collects a pool of generations, pairs them and selects pairs of generation that increase in value and are nearby, then updates the corrector on these pairs.

Original: As training progresses, more generations are added to the pool using the current corrector.
Revised: As training progresses, more generations are added to the pool using the current corrector.

Original: Algorithm 1 summarizes self-corrective learning, detailed below.
Revised: Algorithm 1 summarizes self-corrective learning, detailed below.

Original: Initialization.
Revised: Initialization.

Original: Self-corrective learning begins with a generator p 

Revised: Self-corrective learning begins with a generator p

Original: ( y 0 | x ) , a corrector p θ ( y ′ | y, x ) , a set of training prompts X , and a value function v : Y → R .
Revised: ( y 0 | x ) , a corrector p θ ( y ′ | y, x ) , a set of training prompts X , and a value function v : Y → R .

Original: Optionally, we can use additional feedback f : Y → F and learn p θ ( y ′ | y, x, f ( y )) , where F is arbitrary. 
Revised: Optionally, we can use additional feedback f : Y → F and learn p θ ( y ′ | y, x, f ( y )) , where F is arbitrary. input p θ , X v

Original: ▷ Learning
Revised: · ) , feedback f

Original: The algorithm initializes a datapool of (input, output, value, feedback) examples by using the generator to generate multiple outputs for each input.
Revised: The algorithm initializes a datapool of (input, output, value, feedback) examples by using the generator to generate multiple outputs for each input.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original:  where y 1: N denotes N outputs generated with decoding algorithm q (e.g. temperature sampling).
Revised: all y ∈ [Equation] where y 1: N denotes N outputs generated with decoding algorithm q (e.g. temperature sampling).

Original: When available, ( x, y, v ( y ) , f ( y )) examples from another source (e.g. a dataset) can also be added.
Revised: When available, ( x, y, v ( y ) , f ( y )) examples from another source (e.g. a dataset) can also be added.

Original: Pairing.
Revised: Pairing.

Original: Next, self-corrective learning forms value-improving pairs : examples of mapping a hypothesis to a higher-valued correction.
Revised: Next, self-corrective learning forms value-improving pairs : examples of mapping a hypothesis to a higher-valued correction.

Original: We use the datapool D to form a set of (input, hypothesis, correction) pairs.
Revised: We use the datapool D to form a set of (input, hypothesis, correction) pairs.

Original: A pair is formed when an output has a higher value than another 1 :
Revised: A pair is formed when an output has a higher value than another 1 :

Original: [Equation]
Revised: [Equation]

Original: Learning.
Revised: Learning.

Original: Next, self-corrective learning selects (input, hypothesis, correction) pairs to update the corrector with.
Revised: Next, self-corrective learning selects (input, hypothesis, correction) pairs to update the corrector with.

Original: We sample  a ( x, y, y ′ ) pair proportional to its improvement in value as well as the proximity between the hypothesis y and the correction y ′ :
Revised: We sample an input, x ∼ U ( X ) , then sample a ( x, y, y ′ ) pair proportional to its improvement in value as well as the proximity between the hypothesis y and the correction y ′ :, P

Original: [Equation]
Revised: [Equation]

Original: where s ( y, y ′ ) is a similarity function and Z ( y ) normalizes over the available corrections for y in
Revised: where s ( y, y ′ ) is a similarity function and Z ( y ) normalizes over the available corrections for y in

Original: P x .
Revised: P x .

Original: Increasing the hyperparameter α ∈ R ≥ 0 puts more weight on targets that add more value, while increasing β ∈ R ≥ 0 retains more similar targets.
Revised: Increasing the hyperparameter α ∈ R ≥ 0 puts more weight on targets that add more value, while increasing β ∈ R ≥ 0 retains more similar targets.

Original: We update the corrector using the cross-entropy loss L ( θ ) =
Revised: We update the corrector using the cross-entropy loss L ( θ ) =

Original: − log p θ ( y ′ | y, x, f ( y )) on batches sampled in this way.
Revised: − log p θ ( y ′ | y, x, f ( y )) on batches sampled in this way.

Original: Exploration.
Revised: Exploration.

Original: During exploration, self-corrective learning adds new generations to the datapool by generating from the current corrector:
Revised: During exploration, self-corrective learning adds new generations to the datapool by generating from the current corrector:

Original: [Equation]
Revised: [Equation]

Original: | for
Revised: | for

Original: [Equation]
Revised: [Equation]

Original: and updating the datapool D ← D ∪ D ′ .
Revised: and updating the datapool D ← D ∪ D ′ .

Original: The hypotheses y to correct can come from any source, e.g. newly sampled from the base generator, or from the datapool; we use the latter in our experiments.
Revised: The hypotheses y to correct can come from any source, e.g. newly sampled from the base generator, or from the datapool; we use the latter in our experiments.

Original: Inference.
Revised: Inference.

Original: We use the trained corrector along with a generator to generate a trajectory y 0 , y 1 , . . .
Revised: We use the trained corrector along with a generator to generate a trajectory y 0 , y 1 , . . .

Original: , y T , and consider y T the final output.
Revised: , y T , and consider y T the final output.

Original: Since marginalizing over the intermediate generations in Eq.
Revised: Since marginalizing over the intermediate generations in Eq.

Original: 1 is intractable, we approximate each summation with a single sequence generated with a decoding algorithm q ( · ) .
Revised: 1 is intractable, we approximate each summation with a single sequence generated with a decoding algorithm q ( · ) .

Original: That is, we decode from the generator, then repeatedly from the corrector:
Revised: That is, we decode from the generator, then repeatedly from the corrector:

Original: • Generation: y 0 ∼ q ( p 0 ( y 0 | x )) ; • Correction: y t
Revised: • Generation: y 0 ∼ q ( p 0 ( y 0 | x )) ; • Correction: y t

Original: +1 ∼ q ( p θ ( y t +1 | y t , x, f ( y t ))) , t = 0 , 1 , . . .
Revised: +1 ∼ q ( p θ ( y t +1 | y t , x, f ( y t ))) , t = 0 , 1 , . . .

Original: , T −
Revised: , T −

Original: [Equation]
Revised: [Equation]

Original: The stopping time T is either fixed, or when a target value is obtained (if v ( y ) is available).
Revised: The stopping time T is either fixed, or when a target value is obtained (if v ( y ) is available).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate SELF - CORRECTION on a diversity of tasks: mathematical program synthesis , in which generations are strictly correct or incorrect, and generators typically have low performance; lexically-constrained generation , which allows for partial credit, and generators usually give partially-correct solutions (e.g. matching 3 out of 5 constraints); and toxicity control , where ‘correctness’ is more loosely defined, and the output space is much more open-ended.
Revised: We evaluate SELF - CORRECTION on a diversity of tasks: mathematical program synthesis , in which generations are strictly correct or incorrect, and generators typically have low performance; lexically-constrained generation , which allows for partial credit, and generators usually give partially-correct solutions (e.g. matching 3 out of 5 constraints); and toxicity control , where ‘correctness’ is more loosely defined, and the output space is much more open-ended.

Original: Our experiments are organized to study three settings:
Revised: Our experiments are organized to study three settings:

Original: Using self-correctors to improve upon generators (§3.1,3.2,3.3).
Revised: Using self-correctors to improve upon generators (§3.1,3.2,3.3).

Original: Correcting generators that are much larger than the corrector (§3.4).
Revised: Correcting generators that are much larger than the corrector (§3.4).

Original: Leveraging explicit feedback during training and inference (§3.5).
Revised: Leveraging explicit feedback during training and inference (§3.5).

Original: Next, we describe the self-correction setup and baselines for each task, along with their results.
Revised: Next, we describe the self-correction setup and baselines for each task, along with their results.

Original: 3. M ATHEMATICAL P ROGRAM S YNTHESIS
Revised: 3. M ATHEMATICAL P ROGRAM S YNTHESIS

Original: First, we consider mathematical program synthesis (Austin et al., 2021;
Revised: First, we consider mathematical program synthesis (Austin et al., 2021;

Original: Mishra et al., 2022).
Revised: Mishra et al., 2022).

Original: Given a natural language problem specification x , the task is to generate a program y that upon execution returns the correct answer to x .
Revised: Given a natural language problem specification x , the task is to generate a program y that upon execution returns the correct answer to x .

Original: The task is challenging as it draws on language understanding, multiple-step mathematical problem solving (e.g. identifying a solution strategy, decomposing a problem), and leveraging symbolic tools (e.g. built-in operations, variables).
Revised: The task is challenging as it draws on language understanding, multiple-step mathematical problem solving (e.g. identifying a solution strategy, decomposing a problem), and leveraging symbolic tools (e.g. built-in operations, variables).

Original: Furthermore, the task demands a high level of precision, e.g. a single misplaced operation makes the program incorrect.
Revised: Furthermore, the task demands a high level of precision, e.g. a single misplaced operation makes the program incorrect.

Original: Experimental setup.
Revised: Experimental setup.

Original: As the corrector we use GPT-Neo 1.3B (Black et al., 2021), an open-source autoregressive language model.
Revised: As the corrector we use GPT-Neo 1.3B (Black et al., 2021), an open-source autoregressive language model.

Original: GPT-Neo is pre-trained on language and code (Gao et al., 2021), and hence is widely used for code-related generation (e.g. Chen et al.
Revised: GPT-Neo is pre-trained on language and code (Gao et al., 2021), and hence is widely used for code-related generation (e.g. Chen et al.

Original: ; Ni et al.
Revised: ; Ni et al.

Original: Mishra et al.
Revised: Mishra et al.

Original: We consider two settings for the initial generator: (1) a separate fine-tuned instance of GPT-Neo 1.3B, and (2) few-shot prompted GPT-3 (Brown et al., 2020).
Revised: We consider two settings for the initial generator: (1) a separate fine-tuned instance of GPT-Neo 1.3B, and (2) few-shot prompted GPT-3 (Brown et al., 2020).

Original: For GPT-3, we evaluate the davinci and text-davinci-002 engines, representative of large ( ≈ 175 B 3 ) generators that are state-of-the-art in related tasks (Wei et al., 2022).
Revised: For GPT-3, we evaluate the davinci and text-davinci-002 engines, representative of large ( ≈ 175 B 3 ) generators that are state-of-the-art in related tasks (Wei et al., 2022).

Original: See the Appendix for additional details.
Revised: See the Appendix for additional details.

Original: Self-correction setup.
Revised: Self-correction setup.

Original: As the value function we use correctness, which is 1 when the program y executes and outputs the ground-truth answer and 0 otherwise.
Revised: As the value function we use correctness, which is 1 when the program y executes and outputs the ground-truth answer and 0 otherwise.

Original: Our main experiments do not use explicit feedback, i.e. f ( y )
Revised: Our main experiments do not use explicit feedback, i.e. f ( y )

Original: = ∅ .
Revised: = ∅ .

Original: At inference time, we study two settings for the corrector: (1) applying k corrections and selecting the final generation, (2) an oracle setting that only corrects a draft if the draft is incorrect.
Revised: At inference time, we study two settings for the corrector: (1) applying k corrections and selecting the final generation, (2) an oracle setting that only corrects a draft if the draft is incorrect.

Original: We use greedy decoding for the generator and corrector, and k = 1 .
Revised: We use greedy decoding for the generator and corrector, and k = 1 .

Original: Datasets.
Revised: Datasets.

Original: We evaluate on problems from 5 problem solving datasets:
Revised: We evaluate on problems from 5 problem solving datasets:

Original: MultiArith (Roy et al., 2015),
Revised: MultiArith (Roy et al., 2015),

Original: AddSub (Hosseini et al., 2014), SingleOp (Roy et al., 2015), SVAMP (Patel et al., 2021), and
Revised: AddSub (Hosseini et al., 2014), SingleOp (Roy et al., 2015), SVAMP (Patel et al., 2021), and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Problem:
Revised: Problem:

Original: It takes Jennifer 20 minutes to groom each of her 2 long hair dachschunds.
Revised: It takes Jennifer 20 minutes to groom each of her 2 long hair dachschunds.

Original: If she grooms her dogs every day, how many hours does she spend grooming her dogs in 30 days?
Revised: If she grooms her dogs every day, how many hours does she spend grooming her dogs in 30 days?

Original: Generator: a=20*2b=a* answer=
Revised: Generator: a=20*2b=a* answer=

Original: b print (answer)
Revised: b print (answer)

Original: Corrector: a=20*2b=a* c=
Revised: Corrector: a=20*2b=a* c=

Original: b/60 #fix answer=
Revised: b/60 #fix answer=

Original: c print (answer)
Revised: c print (answer)

Original: Problem
Revised: Problem

Original: [Table]
Revised: [Table]

Original: b=
Revised: b

Original:  [Equation] [Table]
Revised: [Table] [Equation] [Table]

Original: c=
Revised: c=

Original: [Equation]
Revised: [Equation]

Original: +banswer=
Revised: +b [Table] = [Table]

Original: c print (answer)
Revised: print (answer)

Original: [Table]
Revised: [Table]

Original: print
Revised: print

Original: [Figure]
Revised: [Figure]

Original: GSM8k (Cobbe et al., 2021).
Revised: GSM8k (Cobbe et al., 2021).

Original: As in prior work (Austin et al., 2021; Ni et al., 2022; Mishra et al., 2022), we frame these as program synthesis by converting their solutions to Python programs.
Revised: As in prior work (Austin et al., 2021; Ni et al., 2022; Mishra et al., 2022), we frame these as program synthesis by converting their solutions to Python programs.

Original: We separate our experiments into three increasingly difficult settings:
Revised: We separate our experiments into three increasingly difficult settings:

Original: MultiArith , using problems from the MultiArith arithmetic word problem dataset.
Revised: MultiArith , using problems from the MultiArith arithmetic word problem dataset.

Original: Multitask , using problems from 4 arithmetic datasets (MultiArith, AddSub, SingleOp, SVAMP).
Revised: Multitask , using problems from 4 arithmetic datasets (MultiArith, AddSub, SingleOp, SVAMP).

Original: GSM , using problems from the challenging GSM8k dataset.
Revised: GSM , using problems from the challenging GSM8k dataset.

Original: For the MultiArith and Multitask settings, we make train/valid/test splits using 60/20/20% of the respective datasets.
Revised: For the MultiArith and Multitask settings, we make train/valid/test splits using 60/20/20% of the respective datasets.

Original: Similar to Ni et al.
Revised: Similar to Ni et al.

Original: (2022), for the GSM setting we use the official GSM8k test split, and create a validation split using 20% of the training set.
Revised: (2022), for the GSM setting we use the official GSM8k test split, and create a validation split using 20% of the training set.

Original: Note that the problems and answers in all datasets are the same as those from the original non-program datasets.
Revised: Note that the problems and answers in all datasets are the same as those from the original non-program datasets.

Original: Baselines.
Revised: Baselines.

Original: We compare SELF - CORRECT with its  baseline generator (GPT-Neo 1.3B) in all three settings.
Revised: We compare SELF - CORRECT with its fine-tuned baseline generator (GPT-Neo 1.3B) in all three settings.

Original: For the GSM setting, we compare with existing work that uses models within the same magnitude of scale, including NEO FCP+PCP (Ni et al., 2022), which tunes GPT-NEO 2.7B with additional self-sampled programs, and their fine-tuned GPT-NEO 2.7B baseline.
Revised: For the GSM setting, we compare with existing work that uses models within the same magnitude of scale, including NEO FCP+PCP (Ni et al., 2022), which tunes GPT-NEO 2.7B with additional self-sampled programs, and their fine-tuned GPT-NEO 2.7B baseline.

Original: We also report 3B and 6B fine-tuned GPT3-like language models from Cobbe et al.
Revised: We also report 3B and 6B fine-tuned GPT3-like language models from Cobbe et al.

Original: (2021), which were trained on the non-program version of GSM8k.
Revised: (2021), which were trained on the non-program version of GSM8k.

Original: We evaluate larger models later in (§3.4).
Revised: We evaluate larger models later in (§3.4).

Original: Results.
Revised: Results.

Original: As seen in Table 1, the self-corrector improves upon the generator in all three settings, using either inference strategy: always correcting ( SELF - CORRECT ), or only correcting incorrect solutions ( SELF - CORRECT ∗ ).
Revised: As seen in Table 1, the self-corrector improves upon the generator in all three settings, using either inference strategy: always correcting ( SELF - CORRECT ), or only correcting incorrect solutions ( SELF - CORRECT ∗ ).

Original: The self-corrector’s performance on Multiarith is very high after correction (9899%), a 38 point improvement over the generator, with a similar gain in the Multitask arithmetic setting.
Revised: The self-corrector’s performance on Multiarith is very high after correction (9899%), a 38 point improvement over the generator, with a similar gain in the Multitask arithmetic setting.

Original: On the challenging GSM dataset, the self-corrector achieves 21%, and 24% with only correcting incorrect solutions, up from 8.57% for the generator.
Revised: On the challenging GSM dataset, the self-corrector achieves 21%, and 24% with only correcting incorrect solutions, up from 8.57% for the generator.

Original: Notably, this is higher thanprevious work based on the larger 2.7B GPT-Neo, or larger models tuned on the languageversion of GSM.
Revised: Notably, this is higher than the larger 2.7B GPT-Neo (also larger than generator+corrector), or larger models tuned on the language

Original: [Table]
Revised: [Table]

Original:  The results show that self-corrective learning can improve task performance via training a corrector.
Revised: version of GSM. The results show that self-corrective learning can improve task performance via training a corrector.

Original: Qualitatively, the self-corrector can correct values in a correctly structured solution, fix the order of operations within a multistep solution, adjust unit conversions, and make larger multipart revisions (see Figures 3,7,8).
Revised: Qualitatively, the self-corrector can correct values in a correctly structured solution, fix the order of operations within a multistep solution, adjust unit conversions, and make larger multipart revisions (see Figures 3,7,8).

Original: Notably, these are learned automatically through self-corrective learning.
Revised: Notably, these are learned automatically.

Original: 3. L EXICALLY C ONSTRAINED G ENERATION
Revised: 3. L EXICALLY C ONSTRAINED G ENERATION

Original: Next, we consider lexically constrained generation.
Revised: Next, we consider lexically constrained generation.

Original: Given a set of constraint words x , the task is to generate a sentence y that includes all the given constraints.
Revised: Given a set of constraint words x , the task is to generate a sentence y that includes all the given constraints.

Original: Faithful constraint satisfaction is crucial for many downstream tasks, e.g., those that require converting information to text (McKeown, 1985).
Revised: Faithful constraint satisfaction is crucial for many downstream tasks, e.g., those that require converting information to text (McKeown, 1985).

Original: Datasets and Metrics.
Revised: Datasets and Metrics.

Original: We experiment on C OMMON G EN (Lin et al., 2020) and E2E (Novikova et al., 2017).
Revised: We experiment on C OMMON G EN (Lin et al., 2020) and E2E (Novikova et al., 2017).

Original: C OMMON G EN is a benchmark for generative commonsense reasoning where the task is to generate a coherent sentence given a set of words (e.g., dog, catch).
Revised: C OMMON G EN is a benchmark for generative commonsense reasoning where the task is to generate a coherent sentence given a set of words (e.g., dog, catch).

Original: E2E involves converting structured inputs into natural language.
Revised: E2E involves converting structured inputs into natural language.

Original: For both tasks, we report standard metrics including human/automatic measures of fluency (BLEU, CIDER, etc.)
Revised: For both tasks, we report standard metrics including human/automatic measures of fluency (BLEU, CIDER, etc.)

Original: as well as constraint coverage.
Revised: as well as constraint coverage.

Original: We collect human measures of fluency on Amazon Mechanical Turk; see the Appendix for details.
Revised: We collect human measures of fluency on Amazon Mechanical Turk; see the Appendix for details.

Original: Setup.
Revised: Setup.

Original: We parameterize the base generator with GPT-2 Radford et al.
Revised: We parameterize the base generator with GPT-2 Radford et al.

Original: (large-size for C OMMON G EN and medium-size for E2E).
Revised: (large-size for C OMMON G EN and medium-size for E2E).

Original: We fine-tuned the generator for each task.
Revised: We fine-tuned the generator for each task.

Original: As the value function for self-corrective learning we use coverage, i.e. the percentage of constraints that are present in the output.
Revised: As the value function for self-corrective learning we use coverage, i.e. the percentage of constraints that are present in the output.

Original: For inference, we use beam search with the generator, then do up to 3 corrections using beam search, stopping early if all constraints are met.
Revised: For inference, we use beam search with the generator, then do up to 3 corrections using beam search, stopping early if all constraints are met.

Original: See the Appendix for additional details.
Revised: See the Appendix for additional details.

Original: Results.
Revised: Results.

Original: Table 2 shows the evaluation results.
Revised: Table 2 shows the evaluation results.

Original: The self-corrector substantially improves constraint coverage over its GPT-2 generator for both tasks, while maintaining or improving its language quality.
Revised: The self-corrector substantially improves constraint coverage over its GPT-2 generator for both tasks, while maintaining or improving its language quality.

Original: On the C OMMON G EN benchmark, the self-corrector paired with the NeuroLogic constrained decoding algorithm (Lu et al., 2021) achieves the best results, outperforming the more sophisticated NeuroLogic-A* decoding algorithm, while being an order of magnitude faster.
Revised: On the C OMMON G EN benchmark, the self-corrector paired with the NeuroLogic constrained decoding algorithm (Lu et al., 2021) achieves the best results, outperforming the more sophisticated NeuroLogic-A* decoding algorithm, while being an order of magnitude faster.

Original: Notably, on
Revised: Notably, on

Original: E2E, self-correction outperforms Neurologic-A* decoding, despite only using standard beam search.
Revised: E2E, self-correction outperforms Neurologic-A* decoding, despite only using standard beam search.

Original: This suggests that a corrector can be viewed as an alternative to using a more sophisticated decoding procedure (A*) for improving performance without modifying the underlying model.
Revised: This suggests that a corrector can be viewed as an alternative to using a more sophisticated decoding procedure (A*) for improving performance without modifying the underlying model.

Original: See Figurefor qualitative examples.
Revised: See Figurefor qualitative examples.

Original: 3. T OXICITY R EDUCTION
Revised: 3. T OXICITY R EDUCTION

Original: Next, we consider the task of toxicity reduction (Gehman et al., 2020; Liu et al., 2021).
Revised: Next, we consider the task of toxicity reduction (Gehman et al., 2020; Liu et al., 2021).

Original: Given a prompt x , the task is to generate a fluent continuation y while avoiding offensive content.
Revised: Given a prompt x , the task is to generate a fluent continuation y while avoiding offensive content.

Original: This task is important for ensuring safe language model deployment, yet challenging: due to misaligned pretraining objectives (i.e. modeling internet text vs. non-toxic text), language models are susceptible to generating toxic completions, even when prompted with seemingly innocuous text (Gehman et al., 2020).
Revised: This task is important for ensuring safe language model deployment, yet challenging: due to misaligned pretraining objectives (i.e. modeling internet text vs. non-toxic text), language models are susceptible to generating toxic completions, even when prompted with seemingly innocuous text (Gehman et al., 2020).

Original: Along with its practical importance, the task tests whether (self-)correctors can be an effective mechanism for controlling the outputs of language models in an open-ended setting.
Revised: Along with its practical importance, the task tests whether (self-)correctors can be an effective mechanism for controlling the outputs of language models in an open-ended setting.

Original: Datasets and Metrics.
Revised: Datasets and Metrics.

Original: We use the R EAL T OXICITY P ROMPTS benchmark (Gehman et al., 2020) which contains 100k prompts designed to elicit toxic generations.
Revised: We use the R EAL T OXICITY P ROMPTS benchmark (Gehman et al., 2020) which contains 100k prompts designed to elicit toxic generations.

Original: Following the experimental setup of Liu et al.
Revised: Following the experimental setup of Liu et al.

Original: (2021), during training we use 85K prompts from the training set, and for evaluation
Revised: (2021), during training we use 85K prompts from the training set, and for evaluation

Original: we
Revised: we

Original: [Table]
Revised: [Table]

Original: use the same 10K non-toxic prompts from test set as Liu et al.
Revised: use the same 10K non-toxic prompts from test set as Liu et al.

Original: We use Perspective API to measure maximum toxicity , defined as the average maximum toxicity over 25 sampled generations, and the (empirical) toxicity probability of at least 1 out of 25 generations being toxic.
Revised: We use Perspective API to measure maximum toxicity , defined as the average maximum toxicity over 25 sampled generations, and the (empirical) toxicity probability of at least 1 out of 25 generations being toxic.

Original: Baselines.
Revised: Baselines.

Original: We compare S ELF -C ORRECT with its generator (GPT-2) and previously reported baselines from Lu et al.
Revised: We compare S ELF -C ORRECT with its generator (GPT-2) and previously reported baselines from Lu et al.

Original: (2022a), including PPLM (Dathathri et al., 2020), GeDi (Krause et al., 2021),
Revised: (2022a), including PPLM (Dathathri et al., 2020), GeDi (Krause et al., 2021),

Original: DExpert (Liu et al., 2020), DAPT (Gururangan et al., 2020), PPO (Lu et al., 2022a), and Quark (Lu et al., 2022a).
Revised: DExpert (Liu et al., 2020), DAPT (Gururangan et al., 2020), PPO (Lu et al., 2022a), and Quark (Lu et al., 2022a).

Original: The latter two – Proximal Policy Optimization (PPO) and Quantized Reward Konditioning (Quark) – represent strong, state-of-the art approaches based on reinforcement learning.
Revised: The latter two – Proximal Policy Optimization (PPO) and Quantized Reward Konditioning (Quark) – represent strong, state-of-the art approaches based on reinforcement learning.

Original: Setup.
Revised: Setup.

Original: We use the off-the-shelf GPT-

Revised: We use the off-the-shelf GPT-

Original: Large as the generator, and finetune another GPT-

Revised: Large as the generator, and finetune another GPT-

Original: Large as the corrector.
Revised: Large as the corrector.

Original: During inference, we use nucleus sampling with p = 0 .
Revised: During inference, we use nucleus sampling with p = 0 .

Original: 9 to generate 25 samples for all baselines.
Revised: 9 to generate 25 samples for all baselines.

Original: As the value function, we use the Perspective API score, v ( y ) ∈ [0 , 1] , which measures the toxicity of the completed sequence.
Revised: As the value function, we use the Perspective API score, v ( y ) ∈ [0 , 1] , which measures the toxicity of the completed sequence.

Original: We do up to three corrections with the corrector model.
Revised: We do up to three corrections with the corrector model.

Original: Results.
Revised: Results.

Original: Table 3 shows that S ELF -C ORRECT reduces the rate of toxic generations substantially, while also maintaining fluency and diversity.
Revised: Table 3 shows that S ELF -C ORRECT reduces the rate of toxic generations substantially, while also maintaining fluency and diversity.

Original: S ELF -C ORRECT outperforms all baselines.
Revised: S ELF -C ORRECT outperforms all baselines.

Original: This includes inference-time algorithms (PPLM, GeDi, DExpert), which do not modify the generator but degrade fluency and yield higher toxicity compared to S ELF -C ORRECT , as well as reinforcement learning methods (PPO, Quark) that adjust the generator using toxicity as a (negative) reward. 
Revised: This includes inference-time algorithms (PPLM, GeDi, DExpert), which do not modify the generator but degrade fluency and yield higher toxicity compared to S ELF -C ORRECT , as well as reinforcement learning methods (PPO, Quark) that adjust the generator using toxicity as a (negative) reward. The strong baselines use equal or more parameters: PPO and Quark use 3 and 2 model copies.

Original: The results show that S ELF -C ORRECT is effective for detoxification, without having to modify theunderlying generator.
Revised: The results show that S ELF -C ORRECT is effective for detoxification, without modifying the

Original: We study implications of this latter property further in the next section.

Original: 3. C HANGING M ODULES – C ORRECTING GPT- 
Revised: 3. C HANGING M ODULES – C ORRECTING GPT-

Original: Next, we show that a self-corrector can improve the outputs of a generator that is much larger than the corrector.
Revised: Next, we show that a self-corrector can improve the outputs of a generator that is much larger than the corrector.

Original: We consider two cases: (1) training with a small generator, then swapping in the larger generator at test time; (2) training with the larger generator, i.e. using the large generator to initialize the datapool for self-corrective learning, then using the large generator at test time.
Revised: We consider two cases: (1) training with a small generator, then swapping in the larger generator at test time; (2) training with the larger generator, i.e. using the large generator to initialize the datapool for self-corrective learning, then using the large generator at test time.

Original: Toxicity.
Revised: Toxicity.

Original: We evaluate case (1) for reducing the toxicity of a large generator (GPT-2 XL, GPT-3).
Revised: We evaluate case (1) for reducing the toxicity of a large generator (GPT-2 XL, GPT-3).

Original: We generate an initial sequence using the large generator, then refine it with our corrector trained in the previous experiments (§3.3).
Revised: We generate an initial sequence using the large generator, then refine it with our corrector trained in the previous experiments (§3.3).

Original: Table 4 shows that the resulting self-corrector (large generator + corrector) has substantially reduced toxicity compared to the large generator.
Revised: Table 4 shows that the resulting self-corrector (large generator + corrector) has substantially reduced toxicity compared to the large generator.

Original: This shows the promise of using (self-)correctors for controlling the outputs of large language models.
Revised: This shows the promise of using (self-)correctors for controlling the outputs of large language models.

Original: Math program synthesis.
Revised: Math program synthesis.

Original: Table 4 shows results for math.
Revised: Table 4 shows results for math.

Original: Analogous to toxicity, the corrector is able to correct larger generators swapped in at test-time.
Revised: Analogous to toxicity, the corrector is able to correct larger generators swapped in at test-time.

Original: For instance, the GPT-3 Instruct generator has quite high performance (84.90 Multitask, 36.80 GSM), which improves to 90.90 and 45.00, respectively, by adding in a corrector.
Revised: For instance, the GPT-3 Instruct generator has quite high performance (84.90 Multitask, 36.80 GSM), which improves to 90.90 and 45.00, respectively, by adding in a corrector.

Original: The self-corrector (large generator + corrector) improves further by training with the GPT-3 Instruct generator, to 92.75 and 45.92, respectively.
Revised: The self-corrector (large generator + corrector) improves further by training with the GPT-3 Instruct generator, to 92.75 and 45.92, respectively.

Original: 3. L EVERAGING E XPLICIT F EEDBACK
Revised: 3. L EVERAGING E XPLICIT F EEDBACK

Original: Next, we demonstrate S ELF -C ORRECT ’s capacity to incorporate explicit natural language feedback.
Revised: Next, we demonstrate S ELF -C ORRECT ’s capacity to incorporate explicit natural language feedback.

Original: This amounts to defining a feedback function f , then using the same self-corrective learning and
Revised: This amounts to defining a feedback function f , then using the same self-corrective learning and

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: inference algorithms (§2.1) as in our preceding experiments (in those experiments, f returned ∅ ).
Revised: inference algorithms (§2.1) as in our preceding experiments (in those experiments, f returned ∅ ).

Original: We show that correctors learn to use the feedback, as evidenced by higher performance.
Revised: We show that correctors learn to use the feedback, as evidenced by higher performance.

Original: Toxicity.
Revised: Toxicity.

Original: We use additional fine-grained information from the toxicity API as natural language feedback.
Revised: We use additional fine-grained information from the toxicity API as natural language feedback.

Original: Specifically, besides the overall toxicity score, Perspective API also provides scores for fine-grained attributes of toxicity (e.g. identity attack, profanity, flirtation, etc.).
Revised: Specifically, besides the overall toxicity score, Perspective API also provides scores for fine-grained attributes of toxicity (e.g. identity attack, profanity, flirtation, etc.).

Original: At training time, we compare the attribute scores from a hypothesis and its selected correction, and use the attribute with the largest decrease as natural language feedback (e.g. "decrease toxicity in profanity ").
Revised: At training time, we compare the attribute scores from a hypothesis and its selected correction, and use the attribute with the largest decrease as natural language feedback (e.g. "decrease toxicity in profanity ").

Original: At inference time, we call the API on the current hypothesis, and use the attribute with the highest score.
Revised: At inference time, we call the API on the current hypothesis, and use the attribute with the highest score.

Original: Here we use the API at inference time, which is not required in our previous experiments.
Revised: Here we use the API at inference time, which is not required in our previous experiments.

Original: Lexical constraints.
Revised: Lexical constraints.

Original: In training time, we generate natural language feedback for every example pair ( x, y, y ′ ) by elaborating the extra lexical constraints satisfied by y ′ but not y .
Revised: In training time, we generate natural language feedback for every example pair ( x, y, y ′ ) by elaborating the extra lexical constraints satisfied by y ′ but not y .

Original: e.g. “adding constraint word: read” .
Revised: e.g. “adding constraint word: read” .

Original: At inference time, we elaborate all missing constraints in the current hypothesis.
Revised: At inference time, we elaborate all missing constraints in the current hypothesis.

Original: Math program synthesis.
Revised: Math program synthesis.

Original: Math program synthesis contains a variety of problem types and errors, without an automated means for identifying the errors (e.g. an API).
Revised: Math program synthesis contains a variety of problem types and errors, without an automated means for identifying the errors (e.g. an API).

Original: We explore obtaining natural language feedback about the current program by prompting a large language model.
Revised: We explore obtaining natural language feedback about the current program by prompting a large language model.

Original: We prompt the model with a problem, hypothesis program, a gold solution, and few-shot demonstrations that show feedback on one part of the program; e.g. In the initial guess, 3 should be subtracted.
Revised: We prompt the model with a problem, hypothesis program, a gold solution, and few-shot demonstrations that show feedback on one part of the program; e.g. In the initial guess, 3 should be subtracted.

Original: When the program is correct, the feedback is Correct.
Revised: When the program is correct, the feedback is Correct.

Original: At inference time, we also use feedback from the language model.
Revised: At inference time, we also use feedback from the language model.

Original: We allow the feedback model access to a gold solution, which we expect makes the feedback higher quality, with the risk of solution leakage at inference-time.
Revised: We allow the feedback model access to a gold solution, which we expect makes the feedback higher quality, with the risk of solution leakage at inference-time.

Original: Our results in this task are thus used only to study the feasibility of explicit feedback for math program synthesis.
Revised: Our results in this task are thus used only to study the feasibility of explicit feedback for math program synthesis.

Original: Setup.
Revised: Setup.

Original: For toxicity, lexical constraints, and math we use R EAL T OXICITY P ROMPTS , C OMMON G EN , and the M ULTITASK arithmetic setting, respectively.
Revised: For toxicity, lexical constraints, and math we use R EAL T OXICITY P ROMPTS , C OMMON G EN , and the M ULTITASK arithmetic setting, respectively.

Original: We follow the setup of each task’s previous experiments (§3.3,§3.2,§3.1), except for math we use 5 correction iterations (previously 1).
Revised: We follow the setup of each task’s previous experiments (§3.3,§3.2,§3.1), except for math we use 5 correction iterations (previously 1).

Original: For math, we use GPT-3 ( as the model.
Revised: For math, we use GPT-3 ( as the model.

Original: Results.
Revised: Results.

Original: Table 5 shows that explicit natural language feedback improves performance in all three tasks.
Revised: Table 5 shows that explicit natural language feedback improves performance in all three tasks.

Original: For toxicity, this means that providing fine-grained attributes (e.g. identity attack, profanity, etc.)
Revised: For toxicity, this means that providing fine-grained attributes (e.g. identity attack, profanity, etc.)

Original: during learning and inference improves upon using only the scalar toxicity score.
Revised: during learning and inference improves upon using only the scalar toxicity score.

Original: Intuitively, feedback may help the model to focus on a useful correction; e.g., see Figure 6.
Revised: Intuitively, feedback may help the model to focus on a useful correction; e.g., see Figure 6.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3. A DDITIONAL A BLATIONS AND A NALYSIS
Revised: 3. A DDITIONAL A BLATIONS AND A NALYSIS

Original: Effect of multiple corrections.
Revised: Effect of multiple corrections.

Original: Previously, Figure 4 showed that multiple corrections led to better toxicity reduction.
Revised: Previously, Figure 4 showed that multiple corrections led to better toxicity reduction.

Original: On math (Multitask setting), Figure 5 shows that performance improves with more than one correction, and that multiple corrections are more beneficial with feedback.
Revised: On math (Multitask setting), Figure 5 shows that performance improves with more than one correction, and that multiple corrections are more beneficial with feedback.

Original: Intuitively, in this math task, after 2-3 corrections the model needs additional guidance.
Revised: Intuitively, in this math task, after 2-3 corrections the model needs additional guidance.

Original: Effect of pairing and proportional sampling.
Revised: Effect of pairing and proportional sampling.

Original: Self-corrective learning (i) samples pairs for learning proportional to Equation 4, (ii) only pairs sequences that improve value.
Revised: Self-corrective learning (i) samples pairs for learning proportional to Equation 4, (ii) only pairs sequences that improve value.

Original: We ablate these features by training on Multitask using a data pool that samples a pair for learning uniformly (rather than
Revised: We ablate these features by training on Multitask using a data pool that samples a pair for learning uniformly (rather than

Original: Equation 4), and a data pool without value pairing.
Revised: Equation 4), and a data pool without value pairing.

Original: Table 6 shows that both improve performance.
Revised: Table 6 shows that both improve performance.

Original: Effect of exploration.
Revised: Effect of exploration.

Original: To ablate the effect of exploration, we train a baseline only on correction pairs induced from the base generator.
Revised: To ablate the effect of exploration, we train a baseline only on correction pairs induced from the base generator.

Original: Table 7 shows results on the three math datasets, indicating that exploration improves performance.
Revised: Table 7 shows results on the three math datasets, indicating that exploration improves performance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Self-correction relates to recent works on editing text, including modeling  Wikipedia edits (Reid &
Revised: Self-Correction relates to work modeling text edits including supervised Wikipedia edits (Reid &

Original: Neubig, 2022; Faltings et al., 2021; Schick et al., 2022),which relies on supervised edits, unsupervised methods (Miao et al., 2019; Liu et al., 2020) that perturb sequences with simple operations (e.g. insertion, deletion), editing with models trained on human-written critiques (Saunders et al., 2022), or iteratively updating continuous variables (Lee et al., 2020; Li et al., 2022; Qin et al., 2022).
Revised: Neubig, 2022; Faltings et al., 2021; Schick et al., 2022), unsupervised perturbations (e.g. insertion, deletion)(Miao et al., 2019; Liu et al., 2020), training on human-written critiques (Saunders et al., 2022), or iterating on continuous variables (Lee et al., 2020; Li et al., 2022; Qin et al., 2022).

Original: In contrast to these, self-correction learns an expressive text-to-text correctorthat is trained online to improve a quality measure, withoutrequiring a superviseddataset of edits or critiques.
Revised: In contrast, Self-Correction learns a text corrector online to improve a quality measure without supervised edits or critiques.

Original: Separately, denoising ground-truth sequences is a common pretraining objective (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020), while self-correction ‘denoises’ generations to improve a scalar quality measure.
Revised: Denoising sequences is a common pretraining objective (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020), while self-correction ‘denoises’ generations to improve a scalar quality measure.

Original: Scalar measures are often improved with reinforcement learning (RL) on abase generator (Ziegler et al., 2019; Stiennon et al., 2020; Lu et al., 2022a), which is infeasible forimproving manylanguage models (e.g. those accessed through an API), and uses only scalar feedback.
Revised: Reinforcement learning (RL) is often used to improve scalar measures in a generator (Ziegler et al., 2019; Stiennon et al., 2020; Lu et al., 2022a), yet is infeasible for many models (e.g. those accessed by API), and uses only scalar feedback.

Original: Moreover, self-correction learns a delta between a generation and solution, and is complementary to RL-tuned generators, which can be used within a self-corrector.
Revised: Moreover, Self-Correction learns a delta between generator and solution, and thus RL-tuned generators can be used with SelfCorrection .

Original: Finally, self-correction decomposes generation into multiple steps, which relates to methods that generate rationales before a response (Wei et al., 2022; Dohan et al., 2022). Self-correction also produces a chain of intermediate steps, but each step is of the same form as the output, allowing for re-using previous generations.
Revised: Self-Correction decomposes generation into multiple steps similar to methods that generate rationales before a response (Wei et al., 2022; Dohan et al., 2022), yet intermediate steps of the same form as the output, allowing iterative application. Self-Correction relates to work on program synthesis (Fu et al., 2019; Balog et al., 2020; Gupta et al., 2020; Le et al., 2022) and repair (Gupta et al., 2020; Yasunaga & Liang, 2020; 2021).

Revised: Yasunaga & Liang (2021) is closest in methodology, but Self-Correction uses a domain-agnostic formulation; see Appendix for a discussion.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced self-correctors, a class of models that decompose generation into initial generation and correction steps.
Revised: We introduced self-correctors, a class of models that decompose generation into initial generation and correction steps.

Original: We study self-correctors with a fixed base generator along with a corrector trained to improve outputs according to a scalar measure of quality.
Revised: We study self-correctors with a fixed base generator along with a corrector trained to improve outputs according to a scalar measure of quality.

Original: We presented a simple, general procedure for training the corrector, and find that self-correction is applicable and effective for improving performance, and controlling the outputs of both small and large generators.
Revised: We presented a simple, general procedure for training the corrector, and find that self-correction is applicable and effective for improving performance, and controlling the outputs of both small and large generators.

Original: Moreover, we found that self-correction along with our learning framework provides a promising mechanism for using natural language feedback to improve generation.
Revised: Moreover, we found that self-correction along with our learning framework provides a promising mechanism for using natural language feedback to improve generation.

Original: These findings, along with exploring alternative self-correctors, open up many avenues that we leave for future work.
Revised: These findings, along with exploring alternative self-correctors, open up many avenues that we leave for future work.

Original: A PPENDIX A A DDITIONAL E XPERIMENTAL D ETAILS A. M ATHEMATICAL P ROGRAM S YNTHESIS
Revised: A PPENDIX A R ELATED W ORK

Revised: Self-correction provides a flexible framework for improving the performance of off-the-shelf and fine-tuned language models on a wide range of tasks by decomposing generation into a base generator and a corrector. Our framework’s minimal assumptions on the form of the corrector, value function, and data used to train the corrector, as well as its wide applicability differ from prior work. Learning to fix code.

Revised: Our work relates to two streams of research in the code domain.

Revised: One stream deals with program synthesis , in which a corrector model corrects code from a base synthesizer until it meets a given specification (Fu et al., 2019; Balog et al., 2020; Gupta et al., 2020; Le et al., 2022), while another stream deals with program repair : correcting code that is provided as input (Gupta et al., 2020; Yasunaga & Liang, 2020; 2021).

Revised: Recently, Le et al.

Revised: (2022) developed a modular program synthesis approach that involves a correction module trained on ground-truth outputs.

Revised: In contrast, self-corrective learning supports cases without ground-truth outputs, e.g. toxicity.

Revised: Closest to our methodology is Yasunaga & Liang (2021).

Revised: Unlike Yasunaga & Liang (2021), selfcorrection does not assume a mechanism for generating synthetic negatives, a dataset of negatives, or a separate model that generates negatives.

Revised: This is important because engineering these components for each new task can be prohibitive.

Revised: Second, Yasunaga & Liang (2021) assume a 0/1 value function, while self-correction supports general scalar value functions.

Revised: This is important for tasks such as toxicity that do not have a strict notion of correctness.

Revised: Finally, we propose new pairing and proportional sampling mechanisms found to be important (Table 6).

Revised: Iterative text edits.

Revised: Self-correction relates to recent works on editing text, including modeling

Revised: Wikipedia edits (Reid & Neubig, 2022; Faltings et al., 2021; Schick et al., 2022), which relies on supervised edits, unsupervised methods (Miao et al., 2019; Liu et al., 2020) that perturb sequences with simple operations (e.g. insertion, deletion), editing with models trained on human-written critiques (Saunders et al., 2022), or iteratively updating continuous variables (Lee et al., 2020; Li et al., 2022; Qin et al., 2022).

Revised: In contrast to these, self-correction learns an expressive text-totext corrector that is trained online to improve a quality measure, without requiring a supervised dataset of edits or critiques.

Revised: Finally, correcting text is inherent to the task of grammatical error correction (e.g. Lichtarge et al.

Revised: ; Yasunaga et al.

Revised: (2021); our work differs in that we correct a module within a generation system, and provide a framework for addressing a variety of tasks.

Revised: Denoising and reinforcement learning.

Revised: Separately, denoising ground-truth sequences is a common pretraining objective (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020), while selfcorrection ‘denoises’ generations to improve a scalar quality measure.

Revised: Scalar measures are often improved with reinforcement learning (RL) on a base generator (Ziegler et al., 2019; Stiennon et al., 2020; Lu et al., 2022a), which is infeasible for improving many language models (e.g. those accessed through an API), and uses only scalar feedback.

Revised: Moreover, self-correction learns a delta between a generation and solution, and is complementary to RL-tuned generators, which can be used within a self-corrector.

Revised: Finally, RL can be used as an alternative learning algorithm for training a corrector, which is an interesting direction for future work.

Revised: Modular generation.

Revised: Self-correction decomposes generation into multiple steps, and is thus part of the general class of methods that decompose generation into a ‘cascade’ of modules (Dohan et al., 2022).

Revised: Examples include using separate knowledge generation modules (Shwartz et al., 2020;

Revised: Liu et al., 2022), or generating rationales before a response (Wei et al., 2022).

Revised: Self-correction also produces a chain of intermediate steps, but each step is of the same form as the output, allowing for re-using previous generations.

Revised: B A DDITIONAL E XPERIMENTAL D ETAILS B. C ROSS -E XPERIMENT D ETAILS

Revised: In all of our experiments we use an off-the-shelf embedding similarity function from SentenceTransformers (Reimers & Gurevych, 2019): sentence-transformers/all-MiniLM-L6-v2 .

Original: We fine-tune a separate instance of GPT-Neo 1.3B as an initial generator, using the Huggingface library with default hyperparameters, except for evaluation steps, which we set to a small number to ensure a strong checkpoint is selected for each dataset.
Revised: We fine-tune a separate instance of GPT-Neo 1.3B as an initial generator, using the Huggingface library with default hyperparameters, except for evaluation steps, which we set to a small number to ensure a strong checkpoint is selected for each dataset.

Original: We use the finetuned initial generator as initialization for the corrector, and tune the corrector on sequences [SC]x[CURR]yi[START]yj[END] , where x is a problem, y i and y j form a residual pair, and [ · ] are special tokens.
Revised: We use the finetuned initial generator as initialization for the corrector, and tune the corrector on sequences [SC]x[CURR]yi[START]yj[END] , where x is a problem, y i and y j form a residual pair, and [ · ] are special tokens.

Original: The loss is on tokens after [START] .
Revised: The loss is on tokens after [START] .

Original: Feedback.
Revised: Feedback.

Original: We write 6 demonstrations using training problems and generations from our GPTNeo base generator, and use GPT-3 (text-davinci-002) as a feedback model.
Revised: We write 6 demonstrations using training problems and generations from our GPTNeo base generator, and use GPT-3 (text-davinci-002) as a feedback model.

Original: We use the same training procedure and hyperparameters, except that the sequences now include feedback, [SC]x[CURR]yi[FEEDBACK]F(x,yi)[START]yj[END] , where x is a problem, y i and y j form a residual pair, and F ( x, y i ) is feedback.
Revised: We use the same training procedure and hyperparameters, except that the sequences now include feedback, [SC]x[CURR]yi[FEEDBACK]F(x,yi)[START]yj[END] , where x is a problem, y i and y j form a residual pair, and F ( x, y i ) is feedback.

Original: We include loss on tokens after [FEEDBACK] .
Revised: We include loss on tokens after [FEEDBACK] .

Original: A. L EXICALLY - CONSTRAINED G ENERATION
Revised: B. L EXICALLY - CONSTRAINED G ENERATION

Original: Hyper-parameters.
Revised: Hyper-parameters.

Original: Table 8 and Table 9 show hyperparameters for CommonGen and E2E.
Revised: Table 8 and Table 9 show hyperparameters for CommonGen and E2E.

Original: HumanEvaluation.
Revised: Human

Revised: Evaluation

Original: We evaluate fluency of generations in E2E task using human annotators on Amazon Mechanical Turk (AMT).
Revised: We evaluate fluency of generations in E2E task using human annotators on Amazon Mechanical Turk (AMT).

Original: We randomly sampled 100 instances, along with generations of different baselines and self-corrections.
Revised: We randomly sampled 100 instances, along with generations of different baselines and self-corrections.

Original: For each instance, we ask 3 annotators to evaluate the fluency of generations on a 3-point Likert scale.
Revised: For each instance, we ask 3 annotators to evaluate the fluency of generations on a 3-point Likert scale.

Original: We aggregate annotations from 3 annotators using majority vote.
Revised: We aggregate annotations from 3 annotators using majority vote.

Original: We restricted the pool of annotators to those who are located in US or CA, and had 98% approval rate for at least 5,000 previous annotations. [Figure]
Revised: We restricted the pool of annotators to those who are located in US or CA, and had 98% approval rate for at least 5,000 previous annotations

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B A DDITIONAL R ESULTS
Revised: C A DDITIONAL R ESULTS

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: C Q UALITATIVE E XAMPLES
Revised: D Q UALITATIVE E XAMPLES

Original: [Table]
Revised: [Table]

Original: 19 dimes in her bank

Original: [Table]
Revised: [Table]

Original: Her dad gave her 39 dimes and her mother gave herdimes

Original: [Table]
Revised: [Table]

Original: How many dimes does Melanie have now?

Original: Generator (GPT-Neo)

Original: [Table]
Revised: [Figure]

Original: # In the initial guess, # 39 is not included. Corrector (GPT [Table] [Table]
Revised: Feedback (

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Equation] [Figure] [Equation]
Revised: 3): # In the initial guess, # 2 tickets are not included.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Problem:
Revised: Problem:

Original: Ralph watches TV for 4 hours a day from Monday to Friday, and 6 hours a day on Saturday and Sunday.
Revised: Ralph watches TV for 4 hours a day from Monday to Friday, and 6 hours a day on Saturday and Sunday.

Original: How many hours does Ralph spend watching TV in one week?
Revised: How many hours does Ralph spend watching TV in one week?

Original: Generator:
Revised: Generator:

Original: [Figure]
Revised: [Figure]

Original: print (answer)
Revised: print (answer)

Original: Corrector:
Revised: Corrector:

Original: [Figure]
Revised: [Figure]

Original: print (answer)
Revised: print (answer)

Original: Problem:
Revised: Problem:

Original: The pirates plan to explore 4 islands.
Revised: The pirates plan to explore 4 islands.

Original: Two is- lands require walking 20 miles per day while the other two islands require 25 miles per day.
Revised: Two is- lands require walking 20 miles per day while the other two islands require 25 miles per day.

Original: How many miles will they have to walk if it takes 1.5 days to explore each island?
Revised: How many miles will they have to walk if it takes 1.5 days to explore each island?

Original: Generator:
Revised: Generator:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Corrector:
Revised: Corrector:

Original: [Figure]
Revised: [Figure]

Original: Problem:
Revised: Problem:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: print (answer)
Revised: print (answer)

Original: Corrector:
Revised: Corrector:

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure] 
Revised: [Figure] [Equation] [Figure] [Equation]

Original:  [Equation] [Figure]
Revised: [Figure] [Equation] [Figure]

Original: [Equation][Figure] [Equation] [Figure]
Revised: [Equation]

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: print (answer)
Revised: print (answer)

Original: Problem:
Revised: Problem:

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Figure]