S IMPLIFYING
M ODELS
WITH
U NLABELED
O
UTPUT
D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
We focus on prediction problems with high-dimensional outputs that are subject to output validity constraints, e.g. a pseudocode-to-code translation task where the code must compile.
For these problems, labeled input-output pairs are expensive to obtain, but “unlabeled” outputs, i.e. outputs without corresponding inputs, are freely available and provide information about output validity (e.g. code on
GitHub).
In this paper, we present predict-and-denoise , a framework that can leverage unlabeled outputs.
Speciﬁcally, we ﬁrst train a denoiser to map possibly invalid outputs to valid outputs using synthetic perturbations of the unlabeled outputs.
Second, we train a predictor composed with this ﬁxed denoiser.
We show theoretically that for a family of functions with a high-dimensional discrete valid output space, composing with a denoiser reduces the complexity of a 2-layer
ReLU network needed to represent the function and that this complexity gap can be arbitrarily large.
We evaluate the framework empirically on several datasets, including image generation from attributes and pseudocode-to-code translation.
On the SP O C pseudocode-to-code dataset, our framework improves the proportion of code outputs that pass all test cases by 3-5% over a baseline Transformer.
I NTRODUCTION
We study problems whose outputs have validity constraints.
For example, in pseudocode-to-code translation, the output code must compile.
Other examples include natural language translation and molecule generation, where outputs should be grammatically correct or chemically valid, respectively.
State-of-the-art models typically learn the input-output mapping from expensively-obtained labeled data Kulal et al.
; Vaswani et al.
(2017); Méndez-Lucio et al.
(2020); Senior et al.
(2020), which may not contain enough examples to learn a complex validity structure on high-dimensional output spaces.
However, there are often lots of “unlabeled” outputs—outputs without a corresponding input (e.g., GitHub has over 40 million public code repositories).
How do we leverage these with a much smaller amount of labeled input-output pairs to improve accuracy and validity?
In this paper, we present predict-and-denoise , a framework in which we compose a base predictor , which maps an input to a possibly invalid output, with a denoiser , which maps the possibly invalid output to a valid output.
We ﬁrst train the denoiser on synthetic perturbations of unlabeled outputs.
Second, we train the base predictor composed with the ﬁxed denoiser on the labeled data (Figureleft).
By factorizing into two modules, base predictor and denoiser, the framework allows the base predictor to be simpler by ofﬂoading the complexity of modeling the output validity structure to the denoiser, which has the beneﬁt of being trained on much more data.
We aim to lay down a principled framework for using unlabeled outputs with theoretical justiﬁcation for improving sample efﬁciency by reducing the complexity of the learned base predictor.
Figure(middle,right) shows a pictorial example of a staircase function where valid outputs are integers and requires a complex spline to represent.
When composed with a denoiser (which rounds to the nearest integer), a simple linear base predictor can represent the staircase function.
We theoretically show that our framework reduces the complexity of a 2-layer ReLU network needed to represent a family of functions on a discrete valid output set in high-dimensions.
This complexity gap can be arbitrarily large depending on the stability of the target function being learned.
We expect such a lower complexity function to be learnable with fewer samples, improving generalization.
Empirically, we show on image generation and two pseudocode-to-code datasets (synthetic and
SP O C Kulal et al.
that predict-and-denoise improves test performance across continuous and discrete output data modalities.
In image generation, our framework improves the clarity and styling of font images by learning a low-complexity base predictor to generate an abstract image while the denoiser sharpens the image.
For pseudocode-to-code, we consider the more difﬁcult full-program
[Figure]
translation task rather than line-by-line translation (with compiler side information) studied by previous work Kulal et al.
; Yasunaga and Liang (2020).
We ﬁrst study a synthetic pseudocode-to-code dataset where the denoiser simpliﬁes the base predictor by helping with global type inference.
On
SP O C, a recent pseudocode-to-code dataset on programming competition problems, we improve the proportion of correct programs by 3-5% points over a baseline Transformer.
S ETUP
We consider prediction problems from an input space X (e.g., pseudocode) to an output space Y (e.g., code) where there is an unknown subset of valid outputs V ⊆Y (e.g., code that compiles), where the true output is always valid (in V ).
We have a labeled dataset ( x 1 ,y 1 ) ,..., ( x n ,y n ) where x i ∈ X and y i ∈V and access to many unlabeled outputs (˜ y 1 ,..., ˜ y m ) from V .
We do not assume access to any black box function for testing validity (whether y ∈V or not), allowing for general problems (e.g. language generation) where output validity is imprecisely characterized.
A predictor f : X → Y from a chosen hypothesis class H maps from inputs to the ambient output space.
Our goal is to improve the predictor by leveraging information about the valid space V from the unlabeled examples { ˜ y i } mi =1 .
We leverage a denoiser Π: Y →V , which projects a possibly invalid output in Y and to the valid set V .
We can use unlabeled outputs to learn an approximate denoiser.
Base, composed, and direct predictors.
Let (cid:107)·(cid:107) be a norm on H .
Let Π ◦ f base be a composed predictor that is supposed to represent the target function f (cid:63) (that is, Π ◦ f base = f (cid:63) on X ).
In the context of a composed predictor, we call f base the base predictor .
We compare against f direct ∈ argmin f ∈H {(cid:107) f (cid:107) : f
( x )=
f (cid:63) ( x ) ,x ∈X} , a minimum norm direct predictor which represents f (cid:63) .
D ENOISERS CAN REDUCE MODEL COMPLEXITY
In this section, we study direct and composed predictors from an approximation standpoint and use complexity measures on predictors as surrogates for sample complexity.
We aim to represent a target function f (cid:63) :
X → V .
We assume access to a denoiser Π : Y → V which projects to the nearest valid output for an appropriate metric on the output space (breaking ties arbitrarily).
In Section 3.1, we give a simple example for when composing with a denoiser ( Π ◦ f base ) can drastically reduce the complexity of the learned predictor.
Since f base becomes easier to approximate, we may expect better generalization Bartlett et al.
; Neyshabur et al.
Wei and Ma (2020; 2019).
In Section 3.2, we theoretically show for two-layer ReLU networks that the complexity required to directly represent f (cid:63) can be arbitrarily larger than representing with a composed predictor depending on the stability of f (cid:63) .
3. M OTIVATING EXAMPLE
Figure 1 shows a staircase function f (cid:63) that requires a complex direct predictor f direct but the minimum norm base predictor f ∗ base has low complexity.
For 0 < δ < 1 , let the input space
X = (cid:93) N i =
[ i − (1 − δ ) / 2 ,i +(1 − δ ) / 2] be a union of N disjoint intervals and the valid outputs V
= Z be the integers, a subset of the ambient output space Y =
R .
The staircase function is f (cid:63) ( x ) = (cid:98) x (cid:101) deﬁned on X , which rounds a linear function onto the integers.
Following Savarese et al.
(2019), we deﬁne the norm of a univariate function f : R → R as
[Equation]
This norm measures the (lack of) stability of f .
Complex functions will have a higher norm.
Consider representing f (cid:63) with linear splines, a family of piecewise linear functions.
In linear splines, the norm in Equation (1) becomes roughly the sum of absolute changes in slope between piecewise segments.
If we represent f (cid:63) directly with a linear spline f direct , the norm of f direct has to be large due to the large number of slope changes: (cid:107) f direct (cid:107) =(
N − 1) /δ
(Figure 1 left).
Suppose we have access to a denoiser Π( y )
= (cid:98) y (cid:101) , which projects onto V
= Z .
Then a linear function f ∗ base composed with Π can represent the staircase on X , reducing the norm to 1 (Figure 1 right).
By not requiring f ∗ base to represent the local complexity and discreteness in f (cid:63) , the base predictor f ∗ base better captures the underlying globally linear structure of f (cid:63) .
3. A NALYSIS FOR 2- LAYER R E LU NETWORKS
We extend to more general hypothesis classes and high dimensional outputs.
Our setting is motivated by the task of generating images of font characters from attributes, which we study empirically in
Section 5.1.
In font image generation, there is a discrete set of valid font images in the continuous ambient output space.
Formally, we take the valid set V =
{ y ∗ 1 ,...,y ∗ N } to be a discrete set over N output values in R k and f (cid:63) is a piecewise constant function deﬁned on N disjoint intervals
X = (cid:93) Ni =
[ x li ,x ui ] (in ascending order), where there is a δ> 0 gap between each interval and the next.
The target function f (cid:63) is deﬁned such that if x ∈ [ x li ,x ui ] , then f (cid:63) ( x )=
y ∗ i .
We study 2-layer ReLU networks, often studied as a ﬁrst step towards understanding the expressivity of neural networks Neyshabur et al.
(2014); Savarese et al.
; Eldan and Shamir (2016).
Following Savarese et al.
(2019), we deﬁne f θ ∈H as
[Equation]
on x ∈ R d , where we will take d =1 throughout.
Here, [ x ] + =max( x, 0) is the element-wise ReLU nonlinearity.
The parameters θ contain the hidden unit size h ∈ N and all weights and biases.
We let
W (1) ∈
R h ×
d denote the matrix with w (1) l ∈ R d as rows and let b (1) ,b (2) ,w (2) ∈
R h be vectors with b (1) l ,b (2) l ,w (2) l ∈ R as elements respectively.
We let Θ denote this parameter space.
Measure of complexity.
Following Savarese et al.
(2019), the complexity of a network is associated with the squared Euclidean norm of the weights
[Equation]
The norm of f ∈H is the minimum norm required to represent f :
[Equation]
Savarese et al.
(2019) showed that this norm is equivalent to Equation 1 for univariate networks.
Since these complexity measures typically appear in generalization bounds Bartlett et al.
; Neyshabur et al.
(2017), we expect to improve generalization error by reducing these complexity measures.
Minimum complexity reduces with a denoiser.
Given Π( y ) ∈ argmin y ∗ ∈V (cid:107) y ∗ − y (cid:107) 2 which is projection onto V (breaking ties arbitrarily), we want to compare the norms of f direct that represents f (cid:63) directly and the minimum norm base predictor that represents f (cid:63) :
[Equation]
Note that (cid:107) f ∗ base (cid:107)≤(cid:107) f direct (cid:107) since f direct is a feasible solution.
Thus composing cannot increase the norm.
Adjacent intervals measure stability.
Our result depends crucially on the number of non-adjacent pairs of intervals in f (cid:63) .
Suppose the output dimension is k = 1 .
We deﬁne a pair of interval indices ( i,i +1) as adjacent if there is no valid output value y ∈ V such that either y ∗ i < y <
y ∗
i +
or y ∗
i +1 <
y<y ∗ i hold.
The number of non-adjacent interval pairs characterizes the instability of f (cid:63) .
Let | J | be the number of non-adjacent pairs and | I | be the number of adjacent pairs, where | I | + | J | = N − 1 .
Our bound also depends on L =min
i | y ∗
i − y ∗
i +1 |
and U =max i | y ∗
i − y ∗
i +1 | , the min and max separation between valid points.
For higher output dimensions ( k> 1 ), let y ∗ i,j be the j -th output coordinate of the i -th valid point and let | J j | , | I j | ,L j ,U j be the analogous quantities for each output coordinate j ∈ [ k ] .
[Figure]
Theorem 1.
Let the valid output space V ⊂ R k be a set over N multivariate output values { y ∗ 1 ,...,y ∗ N } in V .
Let f (cid:63) :
R → R k be a piecewise constant function deﬁned on X = (cid:93) Ni
=1 [ x li ,x ui ] where f (cid:63) ( x )=
y ∗ i if x ∈ [ x li ,x ui ] .
Let ∆ x be the length of the smallest interval in X .
Then
[Equation]
See Appendix A for a proof.
If | J j | are sublinear in N and valid points are evenly spaced, then the gap is Ω(1 /δ ) which can be arbitrarily large for a ﬁxed output dimension as δ → 0 and N → ∞ .
If any | J j | is linear in N (many non-adjacent intervals), then there is only a constant factor gap in the worst case.
Overall, if f (cid:63) is stable with respect to its discrete output space, we can learn a simpler base predictor that still represents f (cid:63) when composed with the denoiser.
Note that in practice, we need to regularize the base predictor to ﬁnd this low complexity solution.
P REDICT - AND - DENOISE FRAMEWORK
In Section 3, we assumed access to a denoiser Π that maps output y ∈ Y to a valid output Π( y ) ∈ V , allowing the min-norm base predictor f ∗ base to have much lower complexity.
In this section, we are not given a denoiser but instead have access to a large number of unlabeled outputs ˜
y 1 ,..., ˜ y m ∈ V .
We present predict-and-denoise , a framework for utilizing unlabeled output examples to simplify models.
In this framework, we ﬁrst use self-supervised learning on the unlabeled outputs to learn an approximate denoiser Π , and then use Π (which is now ﬁxed) to learn a composed predictor Π ◦ f θ .
Here, f θ is the learned base predictor with parameters θ .
Figure 2 (left-middle) gives an example input-output pair in a pseudocode-to-code task.
Using the predict-and-denoise framework, the model could learn to make code translations on a mostly local, line-by-line basis (a simpler solution) while relying on the denoiser to correct types globally.
Learning the denoiser.
Assume that as domain knowledge, we have a noising distribution q (˜ y (cid:48) | ˜ y ) over outputs given a valid output ˜ y .
Figure 2 (middle) gives an example of an output program in a pseudocode-to-code translation task.
Here, a noising distribution may make random semantic and syntactic corruptions such as changing types or removing semicolons and parentheses (Figure 2 right).
The denoising objective here is to recover the original code from corrupted code.
More generally, given the noising distribution, we train a probabilistic model p β (˜ y | ˜
y (cid:48) )
on output pairs (˜ y (cid:48) , ˜ y ) where ˜ y (cid:48) ∼ q ( ·| ˜ y ) .
We train the probabilistic model by maximizing the log-likelihood
[Equation]
using unlabeled output samples.
The denoiser Π β (˜ y (cid:48) )
= argmax ˜
y p β
(˜ y | ˜
y (cid:48) ) is deﬁned via the probabilistic model.
Learning the composed predictor.
In this step, we ﬁx the learned denoiser Π β and learn the composed predictor Π β ◦ f θ on labeled examples.
We train a probabilistic model p θ for the base predictor by optimizing
[Equation] [Equation]
The ﬁrst term maximizes a lower bound on the log-likelihood of the composed predictor via p β and p θ (see Appendix D).
We optimize a lower bound since optimizing the log-likelihood directly requires computing an intractable partition function over the high-dimensional output space.
The second term is the log-likelihood of only p θ .
We deﬁne the base predictor f θ ( x )=
argmax y p θ ( y | x ) .
[Figure]
Since the learned Π β is imperfect, the hyperparameter λ in the objective trades off between ﬁtting the composition Π β ◦ f θ and ﬁtting f θ directly to the data.
For discrete output spaces, the ﬁrst term in this objective involves an expectation over a discrete space of outputs.
Depending on the model and the task, optimizing this objective may require REINFORCE Williams (1992) or a Gumbel-softmax reparameterization Jang et al.
(2017); Maddison et al.
The direct predictor is only trained with the second of
E x,y [log p θ ( y
Choice of noising distribution.
Learning the base predictor composed with the denoiser allows for some distribution mismatch between the errors of the base predictor and the noising distribution the denoiser is trained on.
By learning in a composed manner, the base predictor can adapt to the choice of noising distribution.
In our experiments in Appendix B, we ﬁnd that predict-and-denoise gives gains across a variety of noising distributions.
E XPERIMENTS
We evaluate predict-and-denoise on image generation from given attributes and full-program psuedocode-to-code translation, showing its beneﬁts on both continuous and discrete output spaces.
In image generation, composed models generate clearer images with less artifacts with few labeled examples.
For full-program pseudocode-to-code translation in SP O C Kulal et al.
(2019), a recent pseudocode-code dataset, our framework improves the proportion of correctly generated programs by 3-5% points over a baseline Transformer and achieves comparable or better results to a line-by-line translation model from previous work Kulal et al.
5. I MAGE GENERATION FROM ATTRIBUTES
We evaluate predict-and-denoise on font image generation, where the ambient output space is continuous.
This task closely mirrors the theoretical setup, where the input is low-dimensional (index of the font and character type) to a high-dimensional output (image).
We also validate the theory from
Section 3, which suggested that regularization is required to realize the complexity reduction of the minimum-norm base predictor f ∗ base .
Qualitatively, image samples from our composed predictor are clearer and has less artifacts.
Prediction task and denoising objective.
We mapping two one-hot vectors corresponding to the character identity (out of 62 possible) and the font of the character to generate (out of 100 fonts) 32 ×grayscale font images.
Here, valid font images have cleanly deﬁned lines and adhere to the font styling.
We train using the pixel-wise squared error loss for all models and tune L2 regularization strength on a validation set.
To train the composed predictor, we set λ = 0 in (6), using only the composed loss.
The denoising objective is to sharpen unlabeled font images distorted by a Gaussian blur ﬁlter with randomly sampled radii in [0 , 2] .
We also report gains with other noising functions (embossing, contrast perturbations) in Appendix B.
Data.
We use a dataset of 56k fonts originally scraped from the Internet Bernhardsson (2016).
Out of the 6200 labeled examples (62 characters × 100 fonts), we split randomly into 2500 training examples, 100 validation examples, and 3600 test examples.
The training examples contain a random subset of the characters for each font.
The models must generate the unseen characters of each font with the correct font styling at test-time.
The denoiser uses additional unlabeled images for ∼ 50k other fonts.
[Figure]
Models and metrics.
The base predictor f θ and the direct predictor f direct are both 7-layer fully-connected networks (see Appendix
The denoiser Π β is a 3-layer U-Net Ronneberger et al.
We test image sample quality directly by computing the pixel-wise squared error with respect to ground truth test images.
Results.
For regularization strength tuned on the validation set (Figure 4 left), the composed predictor achieves an 11% reduction in test MSE compared to the best direct predictor test error.
The direct predictor test MSE increases when its outputs are processed by the denoiser at test time.
We visualize the predicted images for some randomly-selected fonts for comparison (Figure 3).
The base predictor trained with L2 regularization outputs noisy gray images, suggesting that it has learned a lower complexity model.
In contrast, L2 regularization does not improve the direct predictor (Figure 4 right) since directly outputting clearly deﬁned lines and transitions between black and white pixels requires a relatively high complexity model.
Note that we study L2 regularization as motivated by theory, but we expect any reasonable regularization method to help.
Indeed, we ﬁnd that adding dropout to the composed model improves the MSE further to 0.165.
Additional results on varying labeled and unlabeled data size are in Appendix B, where the performance of the Composed model improves upon Direct on all instances.
5. P SEUDOCODE - TO - CODE
We evaluate predict-and-denoise on pseudocode-to-code translation, where the ambient output space is discrete.
We evaluate on two pseudocode-to-code datasets (synthetic in Section 5.2.1 and SP O C in
Section 5.2.2.
On SP O C, our framework improves the proportion of programs that pass all test cases by 3-5% points over a baseline Transformer and has competitive or better results to line-by-line models Kulal et al.
Prediction task and denoising objective.
We consider full-program pseudocode-to-code translation, where inputs X are human-generated pseudocode.
The ambient output space Y is all possible strings and the set of valid outputs V are strings that compile with the g++ compiler.
In contrast to previous works which decompose the problem into line-by-line translation and use information from the compiler Kulal et al.
; Yasunaga and Liang (2020), we translate the entire program at once without compiler access.
Following Yasunaga and Liang (2020), the denoising objective for both pseudocode-to-code datasets consists of repairing random semantic and syntactic corruptions of unlabeled code examples (see Appendix E).
Models and regularization.
We use a Transformer Vaswani et al.
(2017) for both the base predictor and the denoiser.
In all models, we use a combination of weight decay, dropout, attention dropout, and
ReLU dropout as regularization.
To train the composed predictor, we use λ =1 to balance between the ﬁtting the composed and direct objectives.
During inference, we use a greedy decoding for simplicity (without beam search).
Problem-speciﬁc optimizations such as beam search and querying a compiler during inference can improve the results further.
Pretraining models.
In machine translation, a standard way to incorporate unlabeled outputs is to pretrain the encoder/decoder on monolingual data Ramachandran et al.
; Skorokhodov et al.
; Devlin et al.
We consider a pretrained predictor which is pretrained with the denoising objective on unlabeled code and then trained on labeled examples, utilizing a shared encoder/decoder vocabulary.
We employ predict-and-denoise on top by initializing from the pretrained model ( Pretrained+Composed ), which provides complementary beneﬁts beyond pretraining.
Back-translation models.
Back-translation methods use an output to input model (learned on the labeled data) applied on unlabeled outputs to generate additional synthetic inputs Sennrich et al.
(2016b).
We employ predict-and-denoise on top by initializing from a back-translation model ( Back-translation + Composed ), showing complementary beneﬁts.
[Table] [Table]
[Table] [Table]
[Figure]
Metrics.
A generated program has three possible outcomes: compilation error, execution error, or correct.
A program is correct if, executed on a set of input test cases, its outputs match the set of gold outputs.
We measure the proportion of programs that fall into these outcomes.
5.2. S YNTHETIC DATASET
Pseudocode speciﬁes local information but there are global consistency constraints to enforce (Figure 2).
Modeling everything directly requires a complex model.
With predict-and-denoise, the base predictor f θ can do local translation while the denoiser Π enforces global constraints such as type correctness.
To test this intuition, we generate a synthetic pseudocode-to-code dataset where the pseudocode speciﬁes all but the declaration types (see Figure 2).
Dataset generation.
The synthetic programs involve 1-4 variables (bools, ints, and strings) drawn from 10 possible variable names, which are ﬁrst initialized (by reading stdin) and then processed by up to 1-5 random operations, including 3 unary operations per type and 2 binary operations on ints.
There are 100 possible integer values and 10 possible string values.
We generate 1000 labeled examples and 20000 unlabeled code examples.
Results.
Table 1 shows the results for all models.
The Pretrained+Composed predictor improves the proportion of correct programs over direct training by 51% and over pretraining by 9.6%.
We can also apply the learned denoiser to the outputs of the direct and pretrained predictors at test time, which reduces the improvement to 29.6% and 7.6% respectively.
Similarly, Backtranslation+Composed improves upon direct training by 55.2% and over the strong backtranslation baseline by 2.6%.
Results on varying unlabeled and labeled data sizes, where the composed model improves over the baselines in all instances, are in Appendix C. Figure 5 gives an example input with the output of the base and composed predictors.
With the denoiser, the base predictor does not have to output all the correct variable types.
Here, the denoiser correctly instantiates var_5 and corrects the type of temp .
5.2. SP O C
Finally, we evaluate on the challenging SP O C pseudocode-to-code dataset Kulal et al.
(2019), which contains code scraped from codeforces.com and pseudocode written by crowdsourced workers. [Table]
Since we consider the full-program translation task instead of line-by-line as in previous works Kulal et al.
; Yasunaga and Liang (2020), we ﬁlter out training examples where the code is longer than 1000 tokens after pre-processing, retaining over 95% (11355/11925) of the training examples.
We use the two given SP O C test sets, T EST P and T EST W. T EST P tests for generalization to unseen problems, while T EST W tests for generalization to pseudocode written by different workers.
We report results on the full (unﬁltered) test sets.
Denoising objective.
We use random syntactic and semantic corruptions of additional ∼ 280k unlabeled code examples from codeforces.com as in Yasunaga and Liang (2020).
Previous program repair works Yasunaga and Liang (2020) utilize compiler error messages to guide the repair model.
We only use code as input, and thus the task is relatively difﬁcult.
We deﬁne p β in two parts.
First, we train a binary classiﬁer g γ :
Y →{ 0 , 1 } which detects if a program has an error (error is label 1), trained using the denoising dataset.
For an output y (cid:48) ,
if g γ ( y (cid:48) )=
then we deﬁne p β ( y | y (cid:48) )
=
δ ( y (cid:48) ) to be a delta distribution on y (cid:48) .
Otherwise, if g γ ( y (cid:48) )
= 1 , then p β ( y | y (cid:48) )
=
p ν ( y | y (cid:48) ) , where p ν is a
Transformer.
The Transformer p ν is ﬁrst pretrained using a linewise code repair dataset generated from unlabeled examples, then trained on full-program repair where the input program has one random corrupted line with probability 0.75.
Thus, taking β =( γ,ν ) , we have Π β ( y (cid:48) )
=
y (cid:48) if g γ ( y (cid:48) )
=
and Π β ( y (cid:48) )
=argmax y p
ν
( y | y (cid:48) ) otherwise.
Results.
On both test sets, predict-and-denoise models improve the proportion of correct code over the direct predictor by 3-5%, and applying predict-and-denoise to pretrained and backtranslation models improve them by about 1-2% (Table 2).
Applying the denoiser to the direct and pretrained models during test time did not improve their performance.
Backtranslation+Composed matches the top-1 performance of a line-by-line LSTM with attention-based copying Kulal et al.
on T EST P and improve upon it by 11.4% on T EST W despite considering the more difﬁcult full-program generation task.
5. C OMPARISONS TO SCALED - UP BASELINES .
Although the composed model does not optimize the denoiser during joint training with the base predictor, the ﬁnal composed model consists of roughly double the number of layers as the baselines.
Thus, we also present baseline results with the same number of layers as the composed model.
We note that these scaled-up baselines have roughly double the amount of trainable parameters as the composed model, since the denoiser is ﬁxed when training on labeled data.
Table 3 shows the results of scaled-up direct, pretrained, and backtranslation baselines on the synthetic code task.
While the scaled direct predictor improves with respect to the unscaled direct predictor, scaling up worsens the pretrained and backtranslation baselines. Intruigingly, the pretraining becomes dramatically less effective with a very large model, possibly due to the relatively large size of the model in comparison to the unlabeled data (20k examples). Table 2 gives results for a scaled-up direct predictor for the SP O C dataset. We ﬁnd that while the compilation error rate decreases, scaling up slightly degrades the correct rate of its output programs.
5. C OMPARISONS TO BASELINES WITH A DENOISER .
We also give comparisons to the baselines when their outputs are post-processed by the same denoiser used by the composed model.
This results in baselines with the exact same architecture and same number of trainable parameters.
The only difference is that the Composed model trains the base predictor jointly with the ﬁxed denoiser.
Table 3 shows the results of the baselines with a denoiser.
While the improvement between the composed models and the baselines decreases, predict-and-denoise gives gains of 29.6%, 7.6%, and 3.2% above direct+denoiser, pretrained+denoiser, and backtranslation+denoiser respectively.
We ﬁnd that using the denoiser does not improve SP O C baseline results in general.
These experiments highlight the importance of training jointly with the ﬁxed denoiser.
R ELATED WORK
Semi-supervised learning.
Like semi-supervised learning, predict-and-denoise leverages large amounts of unlabeled data.
However, semi-supervised learning works typically use unlabeled input data Tarvainen and Valpola (2017);
Miyato et al.
Shu et al.
; Berthelot et al.
(2019), whereas we have “unlabeled” outputs.
In classiﬁcation, unlabeled outputs can help with handling label shift Lipton et al.
; Azizzadenesheli et al.
, but otherwise there is very little output structure.
If both unlabeled inputs and outputs are available, our method is complementary with semi-supervised methods.
Denoising autoencoding.
Denoising autoencoders (DAE) are classical building blocks for unsupervised deep representation learning Vincent et al.
Recently, DAEs have been considered on the input side to combat adversarial robustness by attempting to clean the adversarial example
ﬁrst using invariances learned from unlabeled data Gu and Rigazio (2015)
; Wong and Kolter (2020).
We consider DAEs for learning invariances and structure in the output space instead of inputs.
Machine translation.
Machine translation methods use monolingual data in both the source and target languages to improve their models Sennrich et al.
(2016b)
; Cheng et al.
Pretraining methods use language modeling on monolingual data to initialize the encoder and decoder Ramachandran et al.
; Skorokhodov et al.
; Devlin et al.
Back-translation methods generate additional synthetic parallel examples by training on the backwards (target to source) problem Sennrich et al.
(2016b).
Predict-and-denoise gives complementary gains on top of pretraining and back-translation.
Semantic parsing and structured prediction.
Some recent semantic parsing works have explicitly provided output constraints using abstract syntax trees (AST) and enforcing type constraints Yin and
Neubig (2017); Krishnamurthy et al.
; Xiao et al.
; Dong and Lapata (2016).
Krishnamurthy et al.
(2017) note that enforcing type constraints during training not only prevents invalid outputs but also improves generalization, supporting our results.
While these methods are useful when the validity structure is known and well-deﬁned, we focus on extracting unknown structure from unlabeled outputs.
Structured prediction spans applications including speech Zhang and Wu (2013), vision Mueller (2013), and medical diagnosis Jagannatha and Yu (2016).
Many approaches use graphical models (on top of neural models) for enforcing validity, e.g. HMMs and CRFs in OCR and sequence tagging Kassel (1995);
Huang et al.
These approaches typically require carefully engineering the graphical model to integrate with a neural component and do not consider the simplicity beneﬁts of composition.
C ONCLUSION
Many tasks in machine learning are no longer classiﬁcation or regression but require generating outputs with rich structure (images, text, music, proteins, etc.), for which unpaired outputs are very common.
We introduce the predict-and-denoise framework, in which we compose a predictor with a denoiser trained on unpaired outputs.
Open questions include whether we can train in a more differentiable way for discrete output spaces and how to choose the best denoising objective for a given prediction task.  A CKNOWLEDGEMENTS
We thank Michi Yasunaga, Robin Jia, Albert Gu, and Karan Goel for helpful discussions and comments.
SMX is supported by an NDSEG Graduate Fellowship.
The work is partially supported by a PECASE award, SDSI, and SAIL at Stanford University.