I NS N ET : An Efﬁcient, Flexible, and Performant Insertion-based Text Generation Model
Abstract
We propose I NS N ET , an expressive insertion-based text generator with efﬁcienttraining and ﬂexible decoding (parallel or sequential).
Unlike most existinginsertion-based text generation works that require re-encoding of the context aftereach insertion operation and thus are inefﬁcient to train, I NS N ET only requiresone pass of context encoding for the entire insertion sequence during training byintroducing a novel insertion-oriented position encoding to enable computationsharing.
Furthermore, I NS N ET provides a controllable switch between paralleland sequential decoding, making it ﬂexible to handle more parallelizable tasks suchas machine translation to support efﬁcient decoding, or less parallelizable taskssuch as lexically constrained open-domain text generation to guarantee high-qualityoutputs.
Experiments on two unsupervised lexically constrained text generationdatasets and three machine translation datasets demonstrate I NS N ET ’s advantagesover previous insertion-based methods in terms of training speed, inference efﬁciency, and generation quality.
Introduction
Insertion-based text generation that formulates the generation process as a sequence of token insertionoperations has received increasing attention in the research community in recent years.
There aretwo major advantages of insertion-based generation over the prevalent left-to-right auto-regressiveparadigm: 1) It can reduce the decoding cost to sub-linear w r t. the sequence length with paralleldecoding (Stern et al., 2019; Gu et al., 2019b) and 2) these non-monotonic text generators with ﬂexibledecoding orders may better recover/utilize the underlying linguistic structures of languages (Wellecket al., 2019; Gu et al., 2019a).
However, this new paradigm of text generation brings unique challenges, mostly in the trainingefﬁciency.
Unlike left-to-right auto-regressive decoders which monotonically expand the context,the insertion operations complicate the position information of each token as the context expands.
Concretely, as is shown in Figure 1, the absolute position of a token in a sequence constantly changesalong with the insertion operations.
As a result, a naive implementation of insertion-based models(e.g., Stern et al.
; Gu et al.
(2019b)) needs to re-encode the context with updated positionalinformation for each token as the insertions proceed, yielding inefﬁcient training with O ( n ) times ofcontext re-encoding (with n indicating the sequence length).
To tackle this problem, previous insertion-based generation models such as Insertion Transformer(InsT) (Stern et al., 2019) and Levenshtein Transformer (LevT) (Gu et al., 2019b) propose paralleltoken insertion to reduce the insertion/re-encoding steps from O ( n ) to Θ(log n ) for both trainingand inference.
However, while it works well for machine translation, such parallel insertion falls
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
short on high-entropy generation tasks such as open-domain dialogue systems(Li et al.
, 2017a),creative generation like story (Yao et al., 2019), poetry (Manurung et al., 2000), and humor generation(Hempelmann, 2008), where many output tokens are highly dependent on each other and thus cannotbe accurately predicted in parallel.
Another thread of work seek to maintain sequential insertions butsimplify the position encoding for the tokens so that they do not change as the insertion operationsproceed (Gu et al., 2019a) to save computation.
However, this sacriﬁces model capability and limitsits applications to complex sequences.
In this paper, we propose I NS N ET , which addresses the training efﬁciency issue of insertion-basedtext generators by enabling computation sharing similar as that in vanilla decoder transformers.
Our ﬁrst contribution is an insertion-oriented, relative positional encoding named offset that allows
I NS N ET to achieve computation sharing without compromising model capacity.
At each insertion step,previously computed position encodings, i.e. offsets, remain unchanged while accurately recordingthe updated pairwise token spatial relations in the pair-wise level.
A corresponding offset can beefﬁciently computed for any given insertion order with a novel process named offset compression .
In this way, we avoid expensive re-encoding of the context, and based on the encoded context, oursecond contribution is the design of an effective aggregation process that allows us to parallellygenerate expressive slot representations for every slot in the partially generated sequence, ﬂexiblysupport both sequential and parallel decoding.
Finally , inspired by the layerization idea of Dinic’salgorithm(Dinitz, 2006), we propose an algorithm for I NS N ET to better determine the parallelizedinsertion operations in order to mitigate the likelihood discrepancy before and after the parallelization.
With all the components, I NS N ET is a novel framework that enables efﬁcient training, ﬂexibledecoding, and expressive positional encoding.
Table 1 shows a comparison between I NS N ET and allprior insertion-based models.
Related Works and Background
Auto-regressive Language Models minimize the negative-log-likelihood of a sequence of n tokenss <n = [ x 0 , x 1 , ..., x n − 1 ] with a left-to-right factorization.
With the transformer architecture (Vaswaniet al., 2017), each step of likelihood estimation can be calculated in parallel while sharing thepreﬁx context encoding calculations.
This makes it possible to build powerful and efﬁcient textgeneration models, like the GPT family (Radford et al., 2018, 2019; Brown et al., 2020).
A lot ofsuccessful applications are based on this paradigm of models, such as automatic story generation (Yaoet al., 2019; Tan et al., 2020), image captioning (Vinyals et al., 2015;
Xu et al., 2015), machinetranslation (Bahdanau et al., 2015; Liu et al., 2020), and dialogue system (Li et al., 2017b,a).
Insertion-based Models with Parallel Decoding Insertion transformer (Stern et al., 2019)
(InsT)proposes a design for insertion-based text generation.
In each step, a bi-directional encoder transformer is performed on the expanded sequence to compute the representation for each candidate slotbetween every two consecutive positions.
After such, an optimization process for the joint distributionof position-token is performed to actually insert one or more token(s).
Variants of InsT share the common atomic objective that models the step log-likelihood .
On step twhere a token x i ↓ i +1 is inserted in slot l i ↓ i +1 between position i and i + 1 of context s ot , the steplog-likelihood can be written as: log p ( x i ↓ i +1 , l i ↓ i +1 | s ot )
= log p position ( i + 1 | s ot ) + log p token ( x i ↓
i +1 | e ( s ot )
i ⊕ e
( s ot ) i +1 ) , where e ( · ) i stands for the i-th position of bi-directional encoding of the sequence and ⊕ stands forvector concatenation.
InsT adopts the original absolute positional encoding of transformers, therepresentation of the generated sequence has to be completely re-encoded after each step of contextexpansion to match the position changes of tokens.
The expectation of the negative log step likelihoodover all permitted context-insertion pairs at each step is computed as the step loss .
The step lossesfrom the ﬁrst step to the last one are summed up as the sequence loss .
Beneﬁting from the partiallyparallelized prediction of tokens, InsT can reduce the number of re-encoding steps to Θ(log n ) .
Levenshtein Transformer (Gu et al., 2019b) (LevT) contains two phases during generation: 1) Insertionphase: it ﬁrst uses a similar strategy as InsT, but only inserts placeholders; an MLM is applied tofulﬁll the placeholders.
2) Deletion phase: the model is trained as a token-wise discriminator todetermine where to delete, under the evaluation of the Levenshtein distance by dynamic programming.
In practice, this would result in a slower process compared to InsT, but better generation quality as italleviates the incoherence caused by parallel insertions.
Sequential Insertion-based Model Previous exploration in insertion-based models mostly assumedthe usage of parallel decoding in each decoding step, resulting in a partially auto-regressive procedure.
For those are trained to only generate one token per decoding step, a vanilla implementation wouldresult in an O ( n ) factor in training time complexity.
NMSTG (Non-monotonic Sequential Text
Generation, Welleck et al.
(2019)) is one of the ﬁrst attempts at modeling a non-monotonic sequentialinsertion-based generation process.
It constrains the dependencies of each inserted token to pseudoMarkovian on an expansion tree.
InDIGO(Gu et al., 2019a) is proposed as a sequentially-decodedinsertion-based model with the transformer architecture.
It supports the efﬁcient likelihood estimationby working around the aforementioned volatility problem (see Figure 1) at a cost of omitting thedistance information in its position encoding, and is thus able to adopt the conventional computationsharing trick to boost the multiplicative factor in training time complexity for each sequence to as fastas O (1) .
However, InDIGO uses a encoding of all previously inserted tokens to predict the next tokenregardless of their tentative position.
It is only after the next token is predicted, the inserted positionfor it is predicted.
Thus, there’s no trivial solution to use InDIGO as a (partially) non-autoregressiveinsertion-based generator.
Absolute vs. Relative Position Embedding The original transformer (Vaswani et al., 2017) usessinusoidal, absolute position embeddings.
Relative position embedding in transformers (Shaw et al.
,2018; Dai et al., 2019) was originally proposed to make the modeling of spatial relation invariantto position translation, and to improve the long-term dependency performance of the model.
Inreplacement of absolute positions, which are tied to each token in the sequence, relative positionstry to encode the spatial layout of the tokens with a matrix that records a directed distance from thecolumn token to the row token.
We further develop the idea of relative position embedding as thekey component to overcome the issue of volatile position information of absolute positions in aninsertion-based generation process.
A noticeable fact is that InDIGO’s implementation of positionencoding can also be regarded as a direction-only relative position embedding system.
The I NS N ET Model
There are three major components of I NS N ET : 1) An insertion context encoder based on thetransformer architecture (Vaswani et al., 2017) that uses a novel way to compute relative position tobetter suit the insertion-based nature of the generation process and enable computation sharing; 2) amodule to compose expressive slot representation for predicting tokens to insert in different slotssimultaneously; and 3) an algorithm that adaptively determines the parallelization of the insertionoperations to minimize conﬂicts.
We introduce each component as follows.
3.1 Context Encoder: A Transformer with Insertion-Oriented Relative Position
In the introduction, we discussed the challenges in (efﬁcient) computation sharing caused by thevolatile position information of the context tokens as shown in Figure 1.
We address this problemby empowering the transformer-based context encoder with a distance-aware, insertion-orientedpairwise relative position encoding.
The proposed insertion-oriented relative position encoding shares
3- 

havepen I a .
Step 0:
Step 1:
Step 2:
Step 3:
Step 4:
Step 5:
Step 6: (a) Absolute Positions 6 2 4 1 3
-3 -1 -4 -23 -1 1 -2 0 -  

havepen I a .
have pen
I a .
(b) Vanilla Relative Positions - - - - -- 0 - - - - - - 1 0 - - - - - 1 -1 0 - - - - 3 1 2 0 - - -
-22 -1 1 -2 0 - - 

havepen I a .
have pen
I a .
(c) Our Relative Positions - - - - -- 0 1 - 2 - -  - 0 - - - -  -2 -1 0 1 - -  - -1 - 0 - -  - - - - - -  -4 -3 -2 -1 0 -
I have a pen .
</s
>
I have a pen .
(d) Ours, Order-Restored oriented relative positions, 2(c) the proposed offset matrix presented in the insertion order, and 2(d) the same offset matrix permuted to show how it looks like if we restore the actual (partial-)sequence.
important designs with previous relative position embeddings(Dai et al., 2019; Yang et al., 2019;
Shih et al., 2019), but differs in how it accurately depicts the spatial layout (i.e., the actual sequentialorder) of the inserted tokens in an insertion-based generation process.
Offset: Insertion-Oriented Relative Position Encoding To illustrate our relative position encoding, considering the partial context (to be completed by further insertions) of “I have pen” with aninsertion order of “have pen I” .
To insert an “a” in between “have” and “pen” , the directed distancevector for token “a” against the rest of the context inserted so far should be [(“have”, -1), (“pen”,+1), (“I”, -2)], simpliﬁed as [-1, +1, -2].
This relative position encoding clearly deﬁnes where theinsertion happens by only describing the pairwise spatial relationship between the incoming tokenand the existing context tokens.
We can pack the relative position vectors for each insertion step toget a matrix that reﬂects the relative spatial relation along the trajectory of insertions, with each rowcorresponding to an insertion step.
We name it the offset matrix.
As a concrete example, to generate the sentence “I have a pen.”
with an insertion order“ 〈 BOS 〉 〈 EOS 〉 have pen I a .”, the complete offset matrix is shown in Figure 2(c).
The effective elements of this matrix are all in the lower triangular region, making the application of thecomputation sharing trick for the decoder transformers trivial.
This reduces the number of context(re-)encoding steps from n in naive insertion-based models (Stern et al., 2019) to 1 during training.
Discussion: Comparisons With Other Positional Encoding Strategies
To illustrate how theproposed insertion-oriented relative position encoding is different from existing position encodingstrategies, Figure 2 shows 2(a) the volatile, un-reusable absolute positions; 2(b) the traditionalnon-insertion-oriented relative positions; 2(c) the proposed offset matrix in the insertion order; and2(d) the same offset matrix permuted to show the restored actual (partial-)sequence.
We cansee that the absolute and the traditional relative positional encoding strategies are not applicablefor efﬁcient insertion-based generators.
The former requires constant updates of the positionalencoding of the existing tokens as the insertion progresses, disabling computation sharing.
The latterrequires knowledge about the ﬁnal sequence length at the beginning of the insertion process, which isunrealistic and destroys the ﬂexibility of insertion-based models.
Our relative position encoding, onthe other hand, reﬂects the order of the original sequence.
Tokens that are on the left of the currentone in the original sequence have a negative relative position to the current one while preserving theinsertion order.
As is illustrated in Figure 2(d), all the “later inserted tokens” are masked out andexcluded from the previously inserted token’s distance computation.
Efﬁcient Computation of Offset Matrix During training, given an insertion order to generate asequence, we can pre-compute the offset matrix.
However, naively constructing the offset matrix withelement-wise operations takes O ( n 2 ) complexity for each sequence, limiting the training efﬁciency.
Therefore we design an efﬁcient algorithm to construct the offset matrix.
Assume the insertion orderis described in the form of absolute position permutation indices (i.e., in the previous example, [0,6, 2, 4, 1, 3, 5]), we design a matrix algorithm called offset compression for our purpose.
Figure 3illustrates
the algorithm with our exemplar sentence.
Speciﬁcally, we ﬁrst convert the absoluteposition vector into a matrix by duplication.
Then, the upper triangular elements are masked by“inﬁnity” to remove their impact in relative position computation because the inserting token should
Permutation indices 0 6 2 4 1 30 6 2 4 1 30 6 2 4 1 30 6 2 4 1 30 6 2 4 1 30 6 2 4 1 30 6 2 4 1 30 inf inf inf inf inf inf 0 6 inf inf inf inf inf 0 6 2 inf inf inf inf 0 6 2 4 inf inf inf 0 6 2 4 1 inf inf 0 6 2 4 1 3 inf 0 6 2 4 1 30 - - - - - - 0 1 - - - - - 0 2 1 - - - - 0 3 1 2 - - - 0 4 2 3 1 - - 0 5 2 4 1 30 6 2 4 1 30 - - - - - - -1 0 - - - - - -1 1 0 - - - - -2 1 -1 0 - - - -1 3 1 2 0 - - -3 2 -1 1 -2 0 -
Step 2: set upper triagular elements to infinity
Step 4: subtract diagonal element from each row
Step 1: tile
[Figure]
an
[Figure]
-d matrix
[Figure]
not attend to future to-be-inserted tokens.
In the third step, each element is replaced by its in-rowranking i.e. its absolute position skipping the masked positions.
In the ﬁnal step, each row is baselinedby the diagonal element to reﬂect that the relative position is between the last inserted token (thediagonal element) and previously inserted ones.
The algorithm can be efﬁciently executed as a seriesof fast matrix operations.
An I NS N ET Layer
We hereby describe how to incorporate the offset matrix into each transformerlayer in I NS N ET for efﬁcient context encoding.
In general, we inherit most designs from previousones with relative position encoding (Dai et al., 2019; Yang et al., 2019).
For layer number i = 1 ...N ,given the output embedding E i − 1 from the last layer and the offset matrix R , the formulation of atransformer layer in an N -layer
I NS N ET can be written as:
[Equation]
where Q i , K i , V i , P i are query, key, value, position matrices, respectively, for layer
i .u
i , v i , W iq , W ik,E , W iv , W ik,R and θ i are learnable model parameters of each I NS N ET layer.
Wedenote the input word embeddings as E 0 for notation consistency.
The last layer of transformer in
I NS N ET outputs the step-wise context-aware token embeddings.
They will be used to compute slotrepresentations for the insertion steps.
3.2 Slot Representation and Insertion Prediction
With the context efﬁciently encoded, the subsequent step of an insertion-based generator is to predictthe next position and token to insert.
Thus, a representation for each potential insertion slot should becomputed.
We hereby show how slot representations can be aggregated from I NS N ET outputs andthe potential challenges during this aggregation process.
A naive design of the slot representation is to simply concatenate the representation vectors fromthe left-neighbor and right-neighbor (in the natural observation order) of the slot as is done in priorworks (Stern et al., 2019).
This slot representation can efﬁciently compute the slot representationfor all possible slots in parallel for each time step.
However, the context encoding we obtainedis insertion-order sensitive and unaware of the tokens inserted later in I NS N ET .
This naive slotrepresentation thus falls short of capturing the global sequence information.
As a remedy, we proposeto also include the last insertion token’s representation to compose the slot representation e ( t ) i ↓ j .
Speciﬁcally, given the representations of the left-side neighbor e i − , the right-side neighbor e + j andthe last inserted token e t , the slot representation can be computed as:
[Equation]
[Figure]
where f l and f r are linear projections for left-point and right-point representation vectors and ⊕stands for vector concatenation.
Figure 4 illustrates the process and how the resulting representationchanges in response to a new step.
The slot representation can then be converted into a probabilistic distribution over the vocabulary topredict the log-likelihood of inserted tokens using a log-linear transformation i.e. log p ( x ( t ) i ↓
j | e ( t ) i ↓
j )
=log-softmax ( W p e ( t )
i ↓ j + b ) x
( t ) i ↓ j .
To decide which slot to insert next, a position logit α ( t ) k
= w (cid:62) o
e ( t ) i k ↓
j kis computed for each slot k with a linear layer.
Then a (log-)soft-max operation is applied on top of theposition logits to obtain the log-probability to insert to each candidate slot.
i.e. log o ( x ( t ) i k ↓
j k |{ e ( t ) ↓ } )
=log-softmax ([ α ( t ) 0 , α ( t ) 1 , ... ]) k .
A binary classiﬁcation head q (0 / 1 |· ) is applied to the last inserted token’s representation e t to decidethe termination of the generation in each step.
During training, only the ﬁnal step q (0 / 1 | e n − 1 ) istrained to predict 1; all intermediate steps would be trained to predict 0.
In the following section, wewill continue to discuss how to formulate the ﬁnal objective of our method using these step-wisedistributions.
3.3 Adaptive Parallelization of Insertions
In addition to our efforts in better training efﬁciency, we also propose an algorithm to adaptivelyparallelize the generation process of I NS N ET to speed up the inference while preserving the generationquality.
Precisely, the idea mimics the graph layerization process in the Dinic’s algorithm (Dinitz,2006).
Assume that we partition the tokens into different layers , such that the tokens within the samelayer are assumed to be safe to insert in parallel without corrupting the coherence of context.
Thealgorithm aims to determine the partition such that the log-likelihood estimation of the resultingparallel insertion-based model is as close to the sequential insertion-based models as possible.
Theintuition is that the sequential insertion process maximally captures the inter-dependencies betweenthe output tokens, but it is inefﬁcient.
By staying close to the sequential insertion-based modelregarding likelihood prediction ability, we speed up the inference without severely sacriﬁcing thelikelihood estimation quality.
We initialize the layerization with a sequential insertion order, which is equivalent to single-nodelayerization, as is illustrated in Figure 5(a).
Starting from this initialization, we perform I NS N ETDinic to gradually evolve this fully-sequential layerization into a non-autoregressive, parallelizedlayerization, as shown in Figure 5(b).
We promote a token x ← from its respective slot ( i l , j l ) inlayer l to ( i l − 1 , j l − 1 ) in the previous layer l − 1 if two conditions are satisﬁed: 1)
In layer l − 1 , slot( i l − 1 , j l − 1 ) does not yet have an assigned insertion; 2) log p ( x ← | e ( l ) i l ↓ j l ) − log p ( x ← | e ( l − 1)
i l − 1 ↓ j l − 1 ) ≤ τ .
[Figure]
[Figure]
POINTERLevTInDIGOAutoregressive
[Figure]
[Figure]
Here τ is a hyper-parameter that controls how much to parallel.
Intuitively, larger τ usually indicatesmore tolerance for token likelihood loss caused by parallel decoding, i.e. more parallelization ofinsertion operations with tradeoffs on likelihood estimation accuracy.
In practice, we ﬁnd settingτ = 2 .
effective enough for most machine translation tasks.
Note that the promotion of tokens only affects the slot representation computation and the insertionlikelihood calculation of the respective insertion.
When encoding the context (as is discussed in
Section 3.1), we always serialize the layers and encode the context as if these insertion operations areperformed sequentially.
For a target sequence s <n = [ x 0 , x 1 , ..., x n − 1 ] , after the layerization algorithm, if token x i ↓ j islayerized in the l -th layer and to be inserted in the slot ( i, j ) , we denote it as x li ↓ j .
Suppose we have meffective layers in total.
Denote the corresponding representation for slot ( i, j ) given available contextin layer l as e ( l ) i ↓
j and the transformer’s encoding of the partial context until the end of layer l ase ( l ) − 1 .
We denote the token likelihood, position likelihood and termination likelihood of the token x li ↓ jto be p ( x li ↓
j | e ( l ) i ↓ j ) , o ( x li ↓ j |{ e ( l ) ↓ } ) and q (0 / 1 | e ( l ) − 1 ) , respectively.
The general objective of I NS N ETparameterized by φ can be formulated as:
[Equation]
As is discussed in previous sections, to compute the objective function for each sequence, ourformulation only requires one pass of transformer encoding computation.
Experiments
We demonstrate the efﬁciency, ﬂexibility, and model capability of I NS N ET with two sets of experiments.
To show I NS N ET ’s performance and the ﬂexibility to switch between sequential and parallel
[Table]
decoding on datasets with high inter-token dependency (i.e., less suitable for parallel decoding).
We follow the setup in Zhang et al.
and address the unsupervised lexically constrained textgeneration problem on two datasets Yelp Review and News. 2)
To further verify the effectiveness of
I NS N ET -Dinic, we evaluate I NS N ET as a (partially) non-autoregressive machine translation modelwith parallel decoding on three classical datasets: WMT Ro-En, WMT En-De and WAT En-Ja.
The general setup about how hyper-parameters are determined can be found in appendixA.
4.1 Sequential-Parallel Trade-off
Experimental Setup Yelp Review dataset consists of 160K training sequences, 10K sequences forvalidation and 1k test sequences.
News dataset consists of 268586 sentences in total, of which 10kare randomly selected as validation set, 1k for testing.
YAKE 2 is performed to the test split of eachdataset to extract lexical constraints.
We vary the hyperparameter τ in the range of { 10 . 0 , 3 . 0 , 1 . 0 , 0 . 3 , −∞ ( fully-sequential ) } and collectthe results on both datasets.
For position prediction, we are inserting into slots with positions lying inthe top 70% of the position distribution mass.
For token prediction, we are doing top- { 1 , 1 , 3 , 3 , 5 }sampling over the vocabulary distribution.
The results are shown in Table 2, Figure 6(a) and 6(b)
Discussion 1)
I NS N ET is able to signiﬁcantly outperform most previous works on the quality-latencytrade-off spectrum.
With more sequential ﬂavor ( i.e. smaller τ ), it is generally to achieve the newstate-of-the-art in generation quality.
Compared to a Plan-And-Write style autoregressive baseline,
I NS N ET guarantees the perfect incorporation of keywords due to its insertion-based nature.
Also, itsimproved performance is consistent on both datasets.
2) When setting τ larger to encourage moreparallelization of decoding, I NS N ET is mostly able to parallelize the insertions with good preservationof the generative quality.
Compared to the vanilla uniformly-parallel setup (I NS N ET -uniform),
I NS N ET -Dinic signiﬁcantly improved the effectiveness of parallel decoding.
4.2 Non-autoregressive Machine Translation
We train I NS N ET -Dinic models with τ = 2 .
for extended investigation of the performance onmachine learning problems.
The results in Table 3 show that, I NS N ET -Dinic is able to achievecomparable or even better performance compared to previous state-of-the-art non-autoregressive(insertion-based) machine translation models in terms of generation quality and latency.
[Table] [Table]
4.3 Training Time Analysis
To show the training efﬁciency of the proposed model, we hereby do an empirical and theoreticalanalysis on the candidate models’ training procedure.
See Table 4.
For empirical results, the statisticsare collected on the Yelp Review unsupervised lexically constrained text generation problem.
Allresults are collected on a single NVIDIA RTX3090 GPU.
The transformer Seq2seq baseline is trainedas a Plan-And-Write model.
POINTER’s training time per epoch is calculated with the ofﬁcialimplementation with mixed precision supported by the NVIDIA APEX library 3 .
Discussion
The results in Table 4 conﬁrm the signiﬁcant improvement in training efﬁciency of
I NS N ET over most previous baselines.
In addition, this concretely shows that it is not practicallyaffordable to obtain a sequential insertion-based generator directly with InsT/LevT. Note that SAO of
InDIGO is an ofﬂine, inference-only algorithm that does not require any computation of gradients.
Thus its constant factor is signiﬁcantly smaller than other O ( n ) gradient-propagating procedures.
According to its original paper and our previous experiments, InDIGO practically needs this processto achieve reasonable performance.
Conclusion & Future Work
We propose I NS N ET , an efﬁcient, ﬂexible and performant insertion-based generator that supportssequential and parallel decoding.
I NS N ET provides a controllable and smooth trade-off betweenthe two paradigms, allowing it to best adapt to different application scenarios.
Experiments on twounsupervised lexically constrained text generation datasets and three machine translation datasetsshow the advantages of I NS N ET over previous methods.
As for future work, it is interesting to explore obtaining a large-scale pre-trained version of I NS N ETfor further ﬁne-tuning under different downstream scenarios, expecting such insertion-based modelscan better generalize on compositionally or recursively varied samples.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] Most theoretical
results are self-explanatory.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] They will bereleased upon camera ready.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A AppendixA.1 General Setup of Experiments
For all language generation tasks, based on whether the data is pre-processed upon release or not,either a BPE-based or a classical tokenizer is applied.
Each of the evaluated transformer models, ifnot otherwise stated, is implemented as a base -sized transformer model, which has 12-layers with 12attention head and 768 hidden dimensions.
The batch size is set to 256.
In cases where the modelsize exceeds the device capacity, the cumulative gradient trick is applied to support an equivalentoptimization effect.
The learning rate is selected from {5e-5, 1e-4, 2e-4}.
The dropout rate is selectedfrom {0.1, 0.2}.
The weight decay rate is set to 0.02.
All the models are trained with 1000 warm-upiterations and a maximum of 200000 iterations of training.
A linear-decay learning rate scheduleris applied for ﬁne-grained training of the model.
If presented with a development set, the trainingwill be early-stopped when the model stops improving for 5 consecutive epochs.
For I NS N ET -Dinic,unless otherwise stated, the parallelization ﬁne-tuning lasts for 5 epochs typically.
The setups of MT tasks follow previous works, including, but not limited to InsT/LevT and InDIGO(Stern et al., 2019; Gu et al., 2019b,a).
Models are implemented based on libraries like PyTorch(Paszke et al., 2019)
(for the generalconstruction of deep learning models, License: BSD) and huggingface’s Transformers(Wolf et al.,2020) (for speciﬁc implementation of transformer-based models, License: Apache-2.0).
Some ofour baselines also used NVIDIA APEX for efﬁcient mix-precision training 4 .
We thank them fortheir contributions to the community.
With the best of our effort we use datasets that contain eitheruser-identity agnostic contents or those that could contain publicly reported information about famous,well-known individuals (for the News dataset).
All results that could contain randomness, if not otherwise stated, is collected from averagingthe results of 3 individual runs of sampling.
The variances of the results won’t affect/reverse theconclusion.
A.2 Ablation Study on the Effectiveness of Components
Although different modules of InsNet are closely entangled with each other and thus hard to beablated into some inferior variants, we hereby conduct an ablation study to show that:
• The proposed offset matrix is indeed a more powerful insertion-oriented position encodingwith signiﬁcantly better capacity.
We show that this in particular helps the model learn to
terminate the sequence in correct/better timings.
To study this, we truncate the offset matrix
to range [ − 1 . 0 , 1 . 0] (InsNet-truncated) so that it is somehow similar to that of InDIGO.
• The introduction of the current-step, global representation when computing the slot represen-
tation is necessary.
It also in general helps to improve the sequence termination performance,
and leads to (however slower) convergence to better local optimals.
We report the results
from an ablated version of InsNet without it (InsNet-noglobal).
Due to the limited time in the rebuttal period, we are only able to collect some primitive results
onthe yelp review dataset.
Most hyperparameters of the three models are kept the same.
The results areas follows.
For quantitative results, we report the termination NLL (see Sec 3.3) of each model.
We also showsome randomly picked samples that shows the performance instability of ablated variants.
InsNet
InsNet-truncated
InsNet-noglobal
Terminating NLL ↓ 2.
BLEU-2/4 ↑ 19.17/5.
[Table]
intriguing meals and traditional .
fun atmosphere .
great prices .
great service .
[Table]
... very intriguing .
great service , great staff , and the whole meals are amazing .
i love the place .
love the food here here . ..
[Table]
( non-terminating )
[Table]
very intriguing meals .
great atmosphere .
and service was great .
food was amazing .
service was delicious .
traditional italian cuisine , very fun .
atmosphere is great .
prices are good and very reasonable . ! .
[Table]
i have been here several times .
great atmosphere and the service is always great .
love the food , and the best steak salad .
it is fantastic !
[Table]
i ve been coming here multiple times in years now .
great food , good food and delicious .
also , the service .
great atmosphere and atmosphere .
service and food was great .
great experience here .
love food and salad , and the staff is very friendly ! . ! .
[Table]
... everything is very delicious .
the whole menu was awesome , especially .
we could have ever eaten since we found this place .
we have had everything on the food , kids menu , , and large portions and food , great service .
great place is awesome .
great service , clean , fresh , tasty , tasty , hot , , , good food , service , ...
( non-terminating )
[Table]
this place has been a long time to take out of my favorite spot .
service is friendly and fast , the food here are excellent .
fantastic sandwiches .
this is my favorite spot.
[Table]
... we love it here , and it s always so delicious .
the service is amazing .
the owner owner is always nice and has been super friendly service , and one of my favorite lunch spot .
the food is always fantastic .
oh and delicious .
and there is the best value .
the food is so fresh and the sandwiches are great .
is my favorite .
, it is fantastic .
i .. ...
( non-terminating )
[Table]
delicious food and service .
great customer service and food .
good food , clean atmosphere .
nice staff .
good breakfast .
great service , fresh food and atmosphere .
will come here everyday ! .
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]