SUNMASK: Mask Enhanced Control in Step Unrolled Denoising Autoencoders
Anonymous Author(s)
AffiliationAddress email
Abstract
This paper introduces SUNMASK, an approach for generative sequence modelingbased on masked unrolled denoising autoencoders.
By explicitly incorporating aconditional masking variable, as well as using this mask information to modulatelosses during training based on expected exemplar difficulty, SUNMASK modelsdiscrete sequences without direct ordering assumptions.
The addition of maskingterms allows for fine-grained control during generation, starting from randomtokens and a mask over subset variables, then predicting tokens which are againcombined with a subset mask for subsequent repetitions.
This iterative processgradually improves token sequences toward a structured output, while guided byproposal masks.
The broad framework for unrolled denoising autoencoders islargely independent of model type, and we utilize both transformer and convolutionbased architectures in this work.
We demonstrate the efficacy of this approach bothqualitatively and quantitatively, applying SUNMASK to generative modeling ofsymbolic polyphonic music, and language modeling for English at both the wordlevel and the character level.
Introduction
Generative modeling is a widespread methodology, used across various domains for automatic contentgeneration, dataset understanding, compression, feature learning, and expert aid for domain analysis. Due to improvements in quality, diversity, and control for generative models, these tools have begunto see commercial application and adoption in an ever growing set of real-world settings. Generativemodels as a whole can be further stratified into different factorizations of the modeling approach,with one of the most prominent categories being autoregressive models (AR).
1.1 Autoregressive Models
AR modeling with deep neural networks has been a dominant approach to generative modelingand feature learning [35, 66, 69, 36, 72, 70] which has many crucial advantages in both trainingand inference.
One key concern is the necessity of defining a "dependency chain" in the form ofa (typically) directed acyclic graph (DAG).
By defining this chain of variables, sampling duringinference can be accomplished in a straightforward manner using ancestral sampling - samplingfrom the first variable or variables in the DAG, using those to conditionally estimate a probabilitydistribution for subsequent variables.
Repeatedly applying this process to walk the chain willeventually result in a sampled estimate of the data, under the assumptions of the AR factorizationused.
Many applications have straightforward orderings in which to define this chain of variables, basedon domain knowledge.
For example following the flow of time for timeseries modeling is often a logical choice, allowing models to make predictions into the future from the past.
However inmany other domains, for example images, language, or music, the process of defining a dependencychain over input variables (e.g. pixels, characters, words, or notes) is far from straightforward, asfor any arbitrary ordering there can frequently be examples where this ordering creates long-termdependencies, or otherwise makes satisfaction of dependencies during training and evaluation moredifficult than another alternative ordering.
This divide becomes further compounded in many creative applications to these domains, as creatorstypically iterate repeatedly: forming a concept, applying an initial sequence of steps to create theframing of the concept, and seeing where the creative flow may lead to alterations in the originalconcept, thus altering future steps.
Though the resulting output may be perceived in a time-orderedfashion (for example, reading a book or listening to a song), the initial creation was performedglobally and holistically.
This global view is often critical to creating elements such as foreshadowingand tension which make the resulting output interesting or enjoyable.
This iterative process is directlyat odds with a strict AR factorization, and requires well trained AR models to cope with a high degreeof uncertainty and multi-modality for long range dependencies, which can lead to logical mistakes orother errors.
1.2 Non-Autoregressive Models
An alternative methodology for generative modeling is non-autoregression (NAR), broadly covering alarge number of different modeling approaches which attempt to remove assumptions about variableordering, instead either hand-defining per-exemplar orderings, or modeling variables jointly withoutresorting to chain rule factorization.
One way to define an ordering over variables is via maskingof inputs or intermediate network representations [20, 67, 68, 73, 69, 53], and indeed modern ARapproaches such as transformers [71] use an autoregressive mask internally to define the chain ofvariables order.
These masks can either be constant over all training (as in standard AR transformersand PixelCNN [69]) or dynamic per example (as in MADE [20]).
When masks are dynamic perexample, we begin to see the relationship between enforcing AR via masking and NAR methods, asalthough some ordering is assumed this ordering is no longer constant, and it becomes possible touse the same trained model to evaluate the probability of a particular output variable under multiplepossible orderings.
Closely linked to masking methods are so called diffusion models , which relax the variable orderingproblem through noise prediction [63, 65, 27].
Rather than predicting a new variable or variablesgiven previous ones in an arbitrarily chosen DAG, diffusion models focus on predicting a lessnoisy version of many variables jointly, given a set of noisy input variables.
Iteratively applyingthis learned denoising improvement operator should eventually result in predicting a fully cleanoutput estimate, given either a noisy version of the target domain, or even starting from pure noise.
Given this framing it is clear that diffusion models are closely linked to denoising methods ingeneral, specifically denoising autoencoders, as well as modern density modeling approaches suchas generative adversarial networks (GAN [21]), variational autoencoders (VAE [38]), flow-basedmodels (NICE [13], RealNVP [14], Normalizing Flows [57], IAF [39], MAF [53]), iterative canvassampling (DRAW [23]), and noise contrastive estimation (NCE [25]).
Particular applications ofthis denoising philosophy such as BERT [12], WaveGrad [8], and GLIDE [52], have resulted inlarge quality improvements for feature learning and data generation for text, images, and audio[43, 62, 56, 26].
1.3 Trade-offs
The choice between AR and NAR methods is not clear-cut.
For many domains, high-quality modelsexist using both approaches but we can define some crucial parameters.
Some NAR methods such as
GAN or VAE are capable of generating output in only one inference step, however they are typicallyhard to train on certain data modalities (e.g. text data) comparing to the AR counterparts.
Other NARmethods such as diffusion models typically allow for choosing a diffusion length during inference,which is independent of that used at training.
Choosing a low diffusion length can frequently lead topoor sample quality, and tuning this setting (among many others) is critical to high quality generation.
However if the tuned diffusion length for a given sequence (of length T ) is shorter than the lengthof those sequences, the NAR method has a computational advantage over the equivalent AR model(which would require T steps for a T length sequence).
In addition, the ability to tune this diffusion length can be useful in interactive applications, or when a variety of output is desirable.
Thissetting can also be a curse, as even well-trained models perform poorly with improper diffusionsettings.
Several branches of current research are focused on improving guarantees [32, 64, 37] andconvergence speed for diffusion models [40, 41, 33, 74].
Method
The relationship between discrete diffusion and denoising autoencoders has been explored in previouswork [28, 61, 3, 29]. We build upon this foundation, combined with many insights from priororderless modeling approaches, crucially Orderless NADE [68], Coconet [30], and SUNDAE [61].
We approach generative modeling in the following manner: given a target data representation, learn amodel which can map from a noisy version of input data to a corrected form of the input.
In this work,we use multinomial noise - namely entries are corrupted to 1 of P possible values (for a given setsize P ), with the number of noised entries defining the relative noise level for the training example.
This is similar to many diffusion approaches at a high level, and shown to be an effective tool in
SUNDAE [61].
In addition to the use of multinomial noise, we also form a mask representing wherethe data was noised, feeding this mask alongside the input data to form a conditional probabilitydistribution.
Like SUNDAE, our method is built around a process x t ∼ f θ ( ·| x t − 1 ; m ) on a space X
={ 1 , . .
. , v } N of arrays of categorical variables.
However in our case the parametric transition functionf θ takes an additional argument m ∈ 0 , 1 N .
During training, m indicates variables that were notcorrupted, and as a consequence we can use it during inference to tell f θ which variables to trust.
Asin SUNDAE, the transition f θ models the variables as conditionally independent of one another.
Given a sequence of masks m 0 , . . .
, m T − 1 , the generating distribution of our model derives from aprior p 0 (typically uniform noise) and repeated application of f θ :
[Equation]
In practice, p 0 is typically elementwise iid uniform noise, and the masks m 0 , . . .
, m T − 1 are drawnaccording to a schedule and may be held constant for several steps.
To train f θ , we take a training example x ∼ p data and draw a mask m .
We apply the corruptionprocedure x 0 ∼ q ( ·| x ; m ) to obtain x 0 which equals x where the mask m is true and uniform randomvalues elsewhere.
Then we iterate x t ∼ f θ ( ·| x t − 1 ; m ) with the aim of reconstructing x .
Similar to SUNDAE, our objective is to minimize 12 ( L (1) + L (2) )
where
L (
[Equation]
[Equation]
=
[Equation]
E m
[Equation]
m
[Equation]
∼
[Equation]
q [Equation] [Equation] f θ
[Equation]
[Equation]
f
[Equation]
[Equation] x [Equation] log f
[Equation] (cid:80)
[Equation] [Equation] (cid:35) is the reconstruction loss for the elements of x that were corrupted.
As in Orderless NADE, we weigheach term according to the size of the mask, to ensure that the overall weight on each conditional f ( i ) θis uniform across i . In practice we choose m 0 = · · · = m t − 1 – since we only go to t = 2 , keepingthe mask constant is a close enough approximation to the masking schedule used in inference.
Coupled with multi-step unrolling, this training scheme encourages learning complex relationshipsbetween the mask and the data, allowing the potential for multi-level trust over the input data:variables with a mask value of 1 which appear correct (given context), variables of mask value 1which appear incorrect, variables of mask value 0 which appear correct, and variables of mask value0 which appear incorrect.
Denoising only methods (such as SUNDAE [61]) would need to forman internal, non-controllable mask in order to disentangle these states, and 0 mask models (such as
Coconet [30]) have controllable input masks but combine both masked states.
This approach allows for direct control at inference using both proposal masks and noising ofvariables, combining elements of both SUNDAE and Coconet, which we name SUNMASK.
Weshow a high level example of the unrolled training scheme, mask proposals, and input data processingin Figure 1.
This outer unrolled loop is largely independent of architecture choice, and as long as the internalarchitecture does not make any ordering assumption over the input data we can incorporate it into SUNMASK.
We use two primary archetypes for the internal model in this paper: Attentional U-Netand Relative Transformer.
Detailed description of the respective architectures can be seen in the
Appendix.
2.1 Training
During training, the internal architecture is combined with a step unrolled training procedure, ashighlighted by SUNDAE [61].
Rather than directly randomizing positions, we re-write this as amasking scheme, first sampling a mask (with 0 randomize, 1 keep, which we denote as 0 -activeformat) then performing randomization to one of P possibilities, for the masked subset of K variables.
This random masking procedure is equivalent to the approach from SUNDAE, but using a maskallows us to further combine the mask information with the input data, in order to form a conditionalprobability estimate.
In addition, this 0 -active masking scheme makes direct comparison to maskingschemes with absorbing states (such as OrderlessNADE [68], Coconet [30], and OA-ARDM [28])simpler, as the mask can be directly multiplied with the data in a 0 -active format.
Convolutional SUNMASK incorporates mask information with a one-hot data representation byconcatenation along the channel axis.
Transformer SUNMASK uses a slightly different settingassuming input transformer data ( T , B ), with T timesteps, B batch elements, and vocabulary size Pis transformed to a ( T, B, L ) dimensional embedding, we repeat the mask along the embedding axis
L times, downweighting the values in the mask by 1 √ L for numerical reasons.
Concatenating thisreduced mask with the input embedding along the last axis is sufficient to form the desired conditionalprobability distribution.
This stretched and reduced mask format provides more stable training thanother schemes such as separately embedding the mask, then concatenating or summing with the inputdata embedding.
Each training batch is randomly sampled from the training dataset, and a corresponding noise valuedrawn from rand ( N ) for N examples in the minibatch.
This per-example noise value is then usedto derive a per-step mask over T timesteps, by comparing noise rand ( N ) < rand ( N, T ) .
Duringtraining, a variable difficulty mask is drawn for each minibatch, with some examples having a highper-example noise value (e.g. . 99 ), and thus many values masked and noised in the training, whileother examples may have a low noise value (e.g. . 01 ) drawn instead.
This means that, combined witha training loss which learns to denoise the input, the overall model will learn a chain to go from morenoisy data to less noisy step-wise, resulting in a learned improvement operator.
This improvement operator, once learned, can be applied to noisy data or pure noise, and iteratetoward a predictive sample from the training distribution.
See Multinomial Diffusion [29] and SUNDAE [61] for more detail on this proof.
In SUNMASK, we combine the mask used to noisethe input with the input data itself, while modifying the loss to predict only masked variables .
Inaddition, we downweight the loss by 1 1+ (cid:80) 1 − mask for each batch element, meaning that losses forheavily masked entries are downweighted compared to losses on examples with little masking, in aform of curriculum weighting based on expected estimation difficulty.
While a one step denoising schemes can be sufficient for learning the data manifold [3, 1], unrollingthis denoising scheme into a multi-step process can have performance benefits.
SUNMASK directlyuses the unrolled loop scheme described in [61], using a step value of 2.
For a detailed description ofthe step unrolled training scheme, see Appendix or the overview description from SUNDAE [61].
The masked and unrolled training can be seen as a container for any internal model which doesnot make ordering assumptions, and we utilize both convolutional U-Net (a variant of GLIDE [52]
U-Net) and Relative Transformer [11, 31, 55] models for various experiments, shown in Section 4.
2.2 Inference
Inference with the learned model can be an involved process, allowing many possible tasks. Because training was done with different masks, a well-trained model should be applicable to contentgeneration, as well as a variety of partially conditional generative tasks such as infilling and humanin-the-loop creation.
Basic sampling involves creating a set of variables, with all variables randomlyset to 1 of P values in the domain (or partial randomization in the case of infilling) along with anaccompanying mask, which is initially all 0 or full generation, or mixed 1 s and 0 s for partial generation tasks.
Given this data and mask as input, trained model then predicts a probability distributionover all possible P values, for all variables.
Despite the use of masked losses in training, we samplethese prediction distributions for all variables.
These predictions are then accepted or rejected fromthe original set, resulting in a new variable set.
We then sample a new mask (based on a predefinedschedule) and combine it with the initial mask, then iterate this overall process, updating at leastsome of the variables at each step.
During inference we use several key techniques to improve generative quality.
We use typicalitysampling [48] on the output probability distribution and a variable number of diffusion steps, onthe order of 100 to 2000 .
Masks are randomly sampled using the schedule defined in [30] whichlinearly decreases the number of masked variables over time according to α n = max( α min , α max −nηN ( α max − α min )) with α min = .
001 , α max = .
999 , and η = 3 / 4 as in [30], along with a triangularlinear ramp-up and ramp-down schedule for the probability of accepting predictions from the modelinto the current variable set at each step, as shown in [61].
Acceptance can be further driven toexplicitly incorporate the update count by increasing the probability of updating variables which havebeen updated less often, rather than relying on random exploration which will eventually result ineven coverage but may have limitations on small diffusion steps or early in the chain.
Variables canalso be re-noised at any step, randomly resetting any variable with a 0 value for the mask at thatdiffusion step.
Some problems (primarily symbolic music modeling) showed increased variabilityfrom re-noising, but the usefulness of this option is task dependent.
We caution that tuning these hyperparameters for inference is critical to success, as improper settingscan drastically lower the performance of SUNMASK, see Section 4 for variance over various inferencesettings in different tasks.
For human-in-the-loop applications, the existence of these controls canallow a number of fine-grained workflows to emerge, driven by expert users to create and curateinteresting output [16, 10], demonstrated in Figure 5.
Related Work
We state here some key related approaches, as well as how our method differentiates from theseprevious settings.
A number of recent publications on diffusion models and feature learning haveincorporated masks as part of their overall training scheme [28, 7, 26], however these papers to usemasks for blanking, rather than as indicators over stochastic variables.
Many infilling models [15,34, 12, 45, 9], and masked image models [26] feature conditional modeling with a mask (blank)token, predicting the variables masked from the input for feature learning or generative modeling.
XLNet [73] combines the infilling and autoregressive paradigms, learning arbitrary permuted orders over masked out variables, using blank-out masking and randomly generated autoregressive orderingsimilar to OrderlessNADE and Coconet.
Conditional diffusion generators [49, 59, 60] and GANgenerators [18] have the combination of mask indicators as well as preserving stochasticity of themasked variables.
However these methods do not use an unrolled training scheme, and generallytarget image related tasks, with the notable exception of maskGAN.
Many models use a concept of aworking canvas, and do repeated inference steps for generation or correction of data [23, 4, 42, 19,51], SUNMASK differs from these models due to architecture choices, training scheme, and lossweighting, as well as application domain [50, 54, 58, 24, 52, 56].
3.1 Convolutional SUNMASK
SUNMASK is most closely related to coconet [30] and SUNDAE [61].
Coconet (as an instance of
OrderlessNADE)
, trains by sampling a random mask per training example, using this mask to setpart of the input (in one hot format) to zero.
The mask is further concatenated to the zeroed dataalong the channel axis, and this combined batch is passed through a deep convolutional network withsmall 3 × 3 kernels.
Coconet is trained to reconstruct the input in whole (categorical loss per variable,over all variables), downweighting losses based on the overall number of active variables in the mask. However, alternate versions of Coconet also experimented with partial prediction using loss only overmissing variables. Convolutional SUNMASK has largely the same input structure as Coconet, anduses a downweighted loss over only variables masked in the input.
However, SUNMASK additionallyuses the unrolled training scheme, as well as a different inference procedure due to preserving thevalues of masked out variables during training and sampling.
Our best performing convolutional SUNMASK architecture takes hints from recent image transformerand vector quantized generators, exchanging the small kernels used in Coconet for extremely largekernels of shape 4 × P over the time and feature dimensions, somewhat analogous to input patches,removing the model’s translation invariance over the feature axis by setting kernel dimension equalto the total feature size.
However this makes the number of parameters per convolutional layerextremely large.
Convolutional SUNMASK adopts an attentional U-Net structure which reduces onlyacross the time axis, modified from GLIDE [52], rather than the deep residual convolution networkused by Coconet.
Combined with the addition of step unrolled training, we are only able to trainconvolutional SUNMASK with a batch size of 1 (expanded to effective batch size 2 due to stepunrolling) on commodity GPU hardware with 16 GB VRAM.
Due to the design choice of extremely large kernel sizes which depend on the size of the domain, weonly use convolutional SUNMASK for polyphonic music experiments, see Section 4 for more details.
Exact specification of the convolutional U-Net architecture can be seen in the Appendix.
3.2 Transformer SUNMASK
Transformer SUNMASK relates closely to the transformer used in SUNDAE.
The architecture uses arelative multi-head attention [11, 31] and has no autoregressive masking.
SUNMASK transformeralso uses larger batch sizes, typically 20 or larger. Though this is far smaller than the batch sizesseen in the experiments of SUNDAE.
Sequence length and data iterator strategy were both a criticalaspect for training transformer SUNMASK.
We found short sequences (from 32 to 128 ) worked best,along with iteration strategies that were example based.
In the language experiments, padding eachexample to some max length resulted in more stable training than the typical language modelingapproach of treating the corpus as one long sequence and slicing into even sized chunks, or iteratingsequentially.
The stability gap between padded sequences and non-overlapping chunking becameespecially apparent at sequence lengths above 128 with transformer SUNMASK.
We list the hyperparameters for the transformer SUNMASK models in the Appendix.
Transformer
SUNMASK was trained on every dataset used in this paper, and we show performance in Section 4,as well as comparisons to convolutional SUNMASK on symbolic polyphonic music modeling.
Bothconvolutional and transformer based SUNMASK use the Adam optimizer, with gradient clipping byvalue at 3 .
Inference hyperparameter types and general sampling strategies used are the same withboth models, though specific hyperparameter values may change between datasets.
[Table]  Experiments4.1 Quantitative
We demonstrate the use of SUNMASK for polyphonic symbolic music modeling on the JSB dataset [2,5].
The JSB dataset consists of 382 four-part chorales, originally written by Johann Sebastian Bach.
These chorales are quantized at the 16 th note interval, cut into non-overlapping chunks of length 128 ,skipping chunks which would cross the end of a piece.
This processing results in a training datasetof 4956 examples, with each example being size (4 , 128) .
We train convolutional and transformerversions of both SUNMASK and SUNDAE for comparison, as well as the pretrained Coconet [30].
For polyphonic music, the quantized data was rasterized in soprano, alto, tenor, bass (SATB) order, asin Music Transformer [31] and BachBot [44], then chunked into non-overlapping training examples.
Results are shown in Table 1.
The EMNLP 2017 News dataset is a common benchmark for word-level language modeling [6],containing a large number of news article sentences [46].
Preprocessing steps remove sentences froman initial corpus down to those containing the most common 5700 words, resulting in a training setof 200 k sentences with a test set of 10 k .
The overall maximum sentence length is 51 .
Commonprocessing for this dataset includes padding all sentences up to this maximum length, which issomewhat different than the standard long sequence chunking commonly used in other languagemodeling tasks.
We show the results of several SUNMASK models, compared to benchmarks usingthe Negative BLEU/Self-BLEU in generated corpora of 1000 sentences in Figure 2.
The text8 dataset [47] is derived from 1GB of Wikipedia text, cleaned to a 27 character alphabet (a-zand spaces), spelling out numbers as their letter equivalent. This cleaned equivalent is then furthertruncated to 100 MB. This results in small dataset used extensively for testing compression, and morerecently generative modeling of text [22]. Although the data doesn’t fully match natural language, itis sufficient to test many properties of generative models for text.
We train character level languagemodels, using transformer SUNMASK and SUNDAE variants to compare learning on small datasets. Generated samples from SUNMASK can be seen in Table 2.
4.2 Masking For General Task Control
Given the flexibility of masking at inference, we perform a number of qualitative queries to inspecthow the model adapts based on noise and mask value.
Figures 3 and 5 demonstrate the use of
SUNMASK for musical inpainting, holding the top voice (soprano) either fully or partially fixed tothe well-known melody "Ode to Joy", by Ludwig van Beethoven.
The task of infilling support partsaround a central theme is broadly known as harmonization in music composition, and we can specify
N Text
with a team means voices of word decembers organizes she is one ae capital missions with favour to a place events of a human dol s office and basefully called cases may be inmixed ideated piece in germanic nine borders canada aulained in canadarian german re factures since
of
the us large human north origin some houses of
short houses also remained and as a
mathematic
province concern
attempts
despite
death
six
five
one k
one that six
one one
nine ed status like any higher language growth ratio battle is down a ts and simply does and a new western and provinces after after s that is often for distinct for a money and control legality the a mask in order to perform harmonization of this melody.
The melodic style of "Ode to Joy" does notfully match the training data, since Beethoven composed in a later era than J.S. Bach.
However wesee the trained model is more than capable of inpainting based on a pre-defined mask.
We test masks which hold the whole soprano fixed, masks which cover only parts of the soprano butdo not allow randomization away from those notes, and masks which cover parts of the soprano butallow rewriting of the non-masked parts of the soprano, as well as rewriting all other voices.
The useof masks to focus on subsets of variables while preserving undferlying intermediate predictions isunique to SUNMASK, as SUNDAE does not have an explicitly controllable input mask and Coconetdoes not have the ability to mask without also blanking the underlying variable.
The high level taskspecification of imputing parts given the rest is shared across all three methods, with both Coconet and
SUNDAE demonstrating effective task performance for infilling and completion in their respectivepublications.
All random seeds in this task control experiment were held identical for each task,meaning the underlying changes in the output content are directly driven by changes in the specifiedmask.
Conclusion
We introduce SUNMASK, a method for masked unrolled denoising modeling of structured data.
SUNMASK separates the role of masking and correction by conditioning predictions on the mask, allowing for fine-grained control at inference.
When applied to character level and word level text aswell as symbolic polyphonic music, SUNMASK is competitive with strong baselines, outperformingreference baselines on music modeling.
Leveraging the separation of mask and noise allows for subtlecontrol at inference, paving the way for a variety of domain specific applications and generativepipelines for human-in-the-loop creation.
A : SUNMASK harmonization (bass, tenor, alto) of an existing melody (soprano), based on pre-defined mask which only highlights the first beat of each measure (0 through 4 of every 16 on the x-axis), but preserves all notes of the melody by overriding predictions in that voice each step.
B : SUNMASK harmonization (bass, tenor, alto) of an existing melody (soprano), based on pre-defined mask which only highlights the first half of the melody ( 0 through 64 on the x-axis), but preserves all notes of the melody by overriding predictions in that voice at each diffusion step.
C : SUNMASK harmonization (bass, tenor, alto) of an existing melody (soprano), based on predefined mask which only highlights the last half of the melody ( 64 through 128 on the x-axis), but preserves all notes of the melody by overriding predictions in that voice at each diffusion step.
D : SUNMASK harmonization and infilling of an existing melody (soprano), based on pre-defined mask on the first beat of each measure ( 0 to 4 of every 16 on the x-axis).
E : SUNMASK harmonization and infilling of an existing melody (soprano), based on pre-defined mask which only preserves the first half of the melody ( 0 to 64 on the x-axis).
F : SUNMASK harmonization and infilling of an existing melody (soprano), based on pre-defined mask which only preserves the last half of the melody ( 64 through 128 on the x-axis).
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[TODO]
[Yes]
(b) Did you describe the limitations of your work?
[TODO]
[Yes] Throughout the paper
we list downsides to including the mask in the input, as well as the complexity of our
inference pipeline
(c) Did you discuss any potential negative societal impacts of your work?
[TODO]
[Yes]
We plan to address this in the supplementary material and appendix due to space
limitations
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[TODO]
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[TODO]
[Yes]
Our
theorems state the relevant conditions
(b) Did you include complete proofs of all theoretical results?
[TODO]
[Yes] We also citerelevant prior work on said theorems
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[TODO]
[No]
We will
include a (likely non-runnable) version of the experimental code in the supplementalmaterial, and work toward a more generally usable release during the review cycle
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[TODO]
[No] To fully disclose this, requires full experimental code
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[TODO]
[Yes] Variance in the scoring table
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[TODO]
[No] We discuss the GPU type,but do
not do a full accounting of resources used.
All experiments were trained on
commodity hardware (P100/V100/A using single GPUs, for less than
24 hours
an
experiment run
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [TODO] [Yes] (b) Did you mention the license of the assets?
[TODO]
[No] All material is extensively
used in prior work
(c) Did you include any new assets either in the supplemental material or as a URL? [TODO] [No] No new assets (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[TODO] [No] No new assets, and assets used are public domain (e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[TODO]
[No] If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[TODO] [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [TODO] [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[TODO] [N/A]