I NS N ET : An Efficient, Flexible, and Performant Insertion-based Text Generation Model
Abstract
We propose I NS N ET , an expressive insertion-based text generator with efficient training and flexible decoding (parallel or sequential).
Unlike most existing insertion-based text generation works that require re-encoding of the context after each insertion operation and thus are inefficient to train, I NS N ET only requires one pass of context encoding for the entire sequence during training by introducing a novel insertion-oriented position encoding and a light-weighted slot representation strategy to enable computation sharing.
Furthermore, we propose an algorithm
I NS N ET -Dinic to better determine the parallelization of insertion operations that provides a controllable switch between parallel and sequential decoding, making it flexible to handle more parallelizable tasks such as machine translation with efficient decoding, or less parallelizable tasks such as open-domain text generation to guarantee high-quality outputs.
Experiments on two lexically constrained text generation datasets and three machine translation datasets demonstrate I NS N ET ’s advantages over previous insertion-based methods in terms of training speed, inference efficiency, and generation quality.
Introduction
Insertion-based text generation that formulates the generation process as a sequence of token insertion operations has received increasing attention in recent years.
There are two major advantages of insertion-based generation over the prevalent left-to-right auto-regressive paradigm: 1) It reduces the decoding cost to sub-linear w.r.t.
the sequence length with parallel decoding (Stern et al., 2019; Gu et al., 2019b), and 2) the flexible insertion orders may better recover/utilize the underlying linguistic structures of languages (Welleck et al., 2019; Gu et al., 2019a).
However, this new paradigm of text generation brings unique challenges, mostly in the training efficiency.
Unlike left-to-right auto-regressive decoders which monotonically expand the context, the insertion operations complicate the position information of each token as the context expands.
Concretely, as is shown in Figure 1, the absolute position of a token in a sequence constantly changes along with the insertion operations.
As a result, a naive implementation of insertion-based models (e.g., Stern et al.
; Gu et al.
(2019b)) needs to re-encode the context with updated positional information for each token as the insertions proceed, yielding inefficient training with O ( n ) times of context re-encoding (with n indicating the sequence length).
To tackle this problem, previous insertion-based generation models such as Insertion Transformer (InsT) (Stern et al., 2019) and Levenshtein Transformer (LevT) (Gu et al., 2019b) propose parallel token insertion to reduce the insertion/re-encoding steps from O ( n ) to Θ(log n ) for both training and inference.
However, while it works well for machine translation, such parallel insertion falls short on high-entropy generation tasks such as open-domain dialogue systems(Li et al., 2017a), creative
[Figure] [Figure]
#Re-Enc.
#Re-Enc.
PosInfo w/ ParaDec w/ SeqDec
Ins.
Trans. Θ(log n )
O ( n )
Absolute Lev. Trans. Θ(log n ) Ω ( n ) Absolute NMSTG
O ( n )
O ( n ) Markovian/Absolute InDIGO N/A O (1) Direction
[Figure]
only I NS N ET (Ours)
O (1)
O (1)
Relative generation such as stories (Yao et al., 2019; Goldfarb-Tarrant et al., 2020; Han et al., 2022), poetry (Manurung et al., 2000; Tian and Peng, 2022), and humor generation (Hempelmann, 2008; He et al., 2019; Mittal et al., 2022; Tian et al., 2022; Sun et al., 2022), where the output is open-ended and the tokens are usually highly dependent on each other, thus cannot be inserted in parallel.
Another thread of work seeks to maintain sequential insertions but simplify the position encoding for the tokens so that they do not change as the insertion operations proceed (Gu et al., 2019a) to save computation.
However, this sacrifices model capability and limits its applications to complex sequences.
In this paper we propose I NS N ET , which addresses the training efficiency issue of insertion-based text generators by enabling computation sharing similar as that in vanilla decoder transformers.
Our first contribution is an insertion-oriented, relative positional encoding coined offset that allows
I NS N ET to achieve computation sharing without compromising model capacity.
At each insertion step, previously computed position encodings of the existing tokens remain unchanged, while the position encodings of newly inserted tokens accurately recording the updated pairwise token spatial relations of all the inserted tokens.
A corresponding offset can be efficiently computed for any given insertion order with a novel process named offset compression .
In this way, we avoid expensive re-encoding of the context, and based on the encoded context, our second contribution is the design of an effective aggregation strategy that allows us to parallelly generate expressive slot representations for every slot in the partially generated sequence, flexibly support both sequential and parallel decoding.
Finally , inspired by the layerization idea of Dinic’s algorithm(Dinitz, 2006), we propose an algorithm for I NS N ET to better determine the parallelization of insertion operations in order to reduce the likelihood discrepancy before and after the parallelization.
With all the components, I NS N ET is a novel framework that enables efficient training, flexible decoding, and expressive positional encoding.
Table 1 shows a comparison between I NS N ET and all prior insertion-based models.
Related Works and Background Auto-regressive Language Models
minimize the negative-log-likelihood of a sequence of n tokens s <n = [ x 0 , x 1 , ..., x n − 1 ] with a left-to-right factorization.
With the transformer architecture (Vaswani et al., 2017), each step of likelihood estimation can be calculated in parallel while sharing the prefix context encoding calculations.
This makes it possible to build powerful and efficient text generation models, like the GPT family (Radford et al., 2018, 2019; Brown et al., 2020).
A lot of successful applications are based on this paradigm of models, such as automatic story generation (Yao et al., 2019; Tan et al., 2020), image captioning (Vinyals et al., 2015;
Xu et al., 2015), machine translation (Bahdanau et al., 2015; Liu et al., 2020), and dialogue system (Li et al., 2017b,a).
Insertion-based Models with Parallel Decoding Insertion transformer (Stern et al., 2019)
(InsT) proposes a design for insertion-based text generation.
In each step, a bi-directional encoder transformer is performed on the partial sequence to compute the representation for each candidate slot between every two consecutive positions.
Then, a model for the joint distribution of position-token is built to insert one or more token(s).
Variants of InsT share the common atomic objective that models the step log-likelihood .
On step t where a token x i ↓ i +1 is inserted in slot l i ↓ i +1 between position i and i + 1 of context
s ot , the step log-likelihood can be written as:
[Equation]
where e ( · ) i stands for the i-th position of bi-directional encoding of the sequence and ⊕ stands for vector concatenation.
InsT adopts the original absolute positional encoding of transformers, the representation of the generated sequence has to be completely re-encoded after each step of context expansion to match the position changes of tokens.
The expectation of the negative log step likelihood over all permitted context-insertion pairs at each step is computed as the step loss .
The step losses from the first step to the last one are summed up as the sequence loss .
Benefiting from the partially parallelized prediction of tokens, InsT can reduce the number of re-encoding steps to Θ(log n ) .
Levenshtein Transformer (Gu et al., 2019b) (LevT) contains two phases during generation: 1) Insertion phase: it first uses a similar strategy as InsT, but only inserts placeholders; an MLM is applied to fulfill the placeholders.
2) Deletion phase: the model is trained as a token-wise discriminator to determine where to delete, under the evaluation of the Levenshtein distance by dynamic programming.
In practice, this would result in a slower process compared to InsT, but better generation quality as it alleviates the incoherence caused by parallel insertions.
Sequential Insertion-based Model Previous exploration in insertion-based models mostly assumed the usage of parallel decoding in each decoding step, resulting in a partially auto-regressive procedure.
For those are trained to only generate one token per decoding step, a vanilla implementation would result in an O ( n ) factor in training time complexity.
NMSTG (Non-monotonic Sequential Text
Generation, Welleck et al.
(2019)) is one of the first attempts at modeling a non-monotonic sequential insertion-based generation process.
It constrains the dependencies of each inserted token to pseudoMarkovian on an expansion tree.
InDIGO(Gu et al., 2019a) is proposed as a sequentially-decoded insertion-based model with the transformer architecture.
It supports the efficient likelihood estimation by working around the aforementioned volatility problem (see Figure 1) at a cost of omitting the distance information in its position encoding, and is thus able to adopt the conventional computation sharing trick to boost the multiplicative factor complexity to as O (1) .
However, InDIGO uses a encoding of all previously inserted tokens to predict the next token regardless of their tentative position.
It is only after the next token is predicted, the inserted position for it is predicted.
Thus, there’s no trivial solution to use InDIGO as a (partially) non-autoregressive insertion-based generator.
Absolute vs. Relative Position Embedding The original transformer (Vaswani et al., 2017) uses sinusoidal, absolute position embeddings.
Relative position embedding in transformers (Shaw et al., 2018; Dai et al., 2019) was originally proposed to make the modeling of spatial relation invariant to position translation, and to improve the long-term dependency performance of the model.
In replacement of absolute positions, which are tied to each token in the sequence, relative positions try to encode the spatial layout of the tokens with a matrix that records a directed distance from the column token to the row token.
We further develop the idea of relative position embedding as the key component to overcome the issue of volatile position information of absolute positions in an insertion-based generation process.
A noticeable fact is that InDIGO’s implementation of position encoding can also be regarded as a direction-only relative position embedding system.
The I NS N ET Model
There are three major components of I NS N ET : 1) An context encoder based on the transformer architecture (Vaswani et al., 2017) that uses a novel way to compute insertion-oriented relative position encoding to better suit the insertion-based nature of the generation process and enable computation sharing (Section 3.1); 2) a module to compose expressive slot representation for predicting tokens to insert in different slots simultaneously (Section 3.2); and 3) an algorithm that adaptively determines the parallelization of the insertion operations to minimize conflicts (Section 3.3).
Figure 2 illustrate the components of the full model.
3.1 Context Encoder: A Transformer with Insertion-Oriented Relative Position
In the introduction, we discussed the challenges in (efficient) computation sharing caused by the volatile position information of the context tokens as shown in Figure 1.
We address this problem by empowering the transformer-based context encoder with a distance-aware, insertion-oriented pairwise relative position encoding.
The proposed insertion-oriented relative position encoding shares important designs with previous relative position embeddings(Dai et al., 2019; Yang et al., 2019;
Encoder
Inputs
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Shih et al., 2019), but differs in how it accurately depicts the spatial layout (i.e., the actual sequential order) of the inserted tokens in an insertion-based generation process.
3.1.1 Offset: Insertion-Oriented Relative Position Encoding
To illustrate our relative position encoding, considering the partial context (to be completed by further insertions) of “I have pen” with an insertion order of “have pen I” .
To insert an “a” in between “have” and “pen” , the distance vector for token “a” against the rest of the context inserted so far should be [(“have”, -1), (“pen”, +1), (“I”, -2)], simplified as [-1, +1, -2].
This relative position encoding clearly defines where the insertion happens by only describing the pairwise spatial relationship between the incoming token and the existing context tokens.
We can pack the relative position vectors for each - - - - -- 1 - - - -- 12 1 - - -- 23 1 2 - -- 34 12 23 1 -- 56 2 4 1 3 45 2 34
3- 

havepen I a .
Step 0:
Step 1:
Step 2:
Step 3:
Step 4:
Step 5:
Step 6: (a) Absolute Positions 6 2 4 1 3
-3 -1 -4 -23 -1 1 -2 0 -  

havepen I a .
have pen
I a .
(b) Vanilla Relative Positions - - - - -- 0 - - - - - - 1 0 - - - - - 1 -1 0 - - - - 3 1 2 0 - - -
-22 -1 1 -2 0 - - 

havepen I a .
have pen
I a .
(c) Our Relative Positions - - - - -- 0 1 - 2 - -  - 0 - - - -  -2 -1 0 1 - -  - -1 - 0 - -  - - - - - -  -4 -3 -2 -1 0 -
I have a pen .
</s
>
I have a pen .
(d) Ours, Order-Restored
Permutation indices 0 6 2 4 1 30 6 2 4 1 30 6 2 4 1 30 6 2 4 1 30 6 2 4 1 30 6 2 4 1 30 6 2 4 1 30 inf inf inf inf inf inf 0 6 inf inf inf inf inf 0 6 2 inf inf inf inf 0 6 2 4 inf inf inf 0 6 2 4 1 inf inf 0 6 2 4 1 3 inf 0 6 2 4 1 30 - - - - - - 0 1 - - - - - 0 2 1 - - - - 0 3 1 2 - - - 0 4 2 3 1 - - 0 5 2 4 1 30 6 2 4 1 30 - - - - - - -1 0 - - - - - -1 1 0 - - - - -2 1 -1 0 - - - -1 3 1 2 0 - - -3 2 -1 1 -2 0 -
Step 2: set upper triagular elements to infinity
Step 4: subtract diagonal element [Figure] each row
[Figure]
insertion step to get a matrix that reflects the relative spatial relation along the trajectory of insertions, with each row corresponding to an insertion step.
We name it the offset matrix.
As a concrete example, to generate the sentence “I have a pen.”
with an insertion order “ 〈 BOS 〉 〈 EOS 〉 have pen I a .”, the complete offset matrix is shown in Figure 3(c).
The effective elements of this matrix are all in the lower triangular region, making the application of the computation sharing trick for the decoder transformers trivial.
This reduces the number of context (re-)encoding steps from n in naive insertion-based models (Stern et al., 2019) to 1 during training.
Discussion: Comparisons With Other Positional Encoding Strategies To illustrate how the proposed insertion-oriented relative position encoding is different from existing position encoding strategies, Figure 3 shows the volatile, un-reusable absolute positions in 3(a), the traditional noninsertion-oriented relative positions as used in models such as XLNet Yang et al. (2019) in 3(b), the proposed offset matrix in the insertion order in 3(c), and the same offset matrix permuted to show the restored actual (partial-)sequence in 3(d).
We can see that the absolute and the traditional relative positional encoding strategies are not applicable for efficient insertion-based generators.
The former requires constant updates of the positional encoding of the existing tokens as the insertion progresses, disabling computation sharing.
The latter requires knowledge about the final sequence length at the beginning of the insertion process, which is unrealistic and destroys the flexibility of insertion-based models.
Our relative position encoding, on the other hand, reflects the order of the original sequence.
Tokens that are on the left of the current one in the original sequence have a negative relative position to the current one while preserving the insertion order.
As is illustrated in Figure 3(d), all the “later inserted tokens” are masked out and excluded from the previously inserted token’s computation.
3.1.2 Efficient Computation of Offset Matrix
During training, given an insertion order to generate a sequence, we can pre-compute the offset matrix.
However, naively constructing the offset matrix with element-wise operations takes O ( n 2 ) complexity for each sequence, limiting the training efficiency.
Therefore we design an efficient algorithm to construct the offset matrix.
Assume the insertion order is described in the form of absolute position permutation indices (i.e., in the previous example, [0, 6, 2, 4, 1, 3, 5]), we design a matrix algorithm called offset compression for our purpose.
Figure 4 illustrates the algorithm with our exemplar sentence.
Specifically, we first convert the absolute position vector into a matrix by duplication.
Then, the upper triangular elements are masked by “infinity” to remove their impact in relative position computation because the inserting token should not attend to future to-be-inserted tokens.
In the third step, each element is replaced by its in-row ranking i.e. its absolute position skipping the masked positions.
In the final step, each row is baselined by the diagonal element to reflect that the relative position is between the last inserted token (the diagonal element) and previously inserted ones.
The algorithm can be efficiently executed as a series of fast matrix operations.
3.1.3 An I NS N ET Layer
We hereby describe how to incorporate the offset matrix into each transformer layer in I NS N ET for efficient context encoding.
In general, we inherit most designs from previous ones with relative position encoding (Dai et al., 2019; Yang et al., 2019).
The encoder panel in Figure 2 (the upper left panel in purple) illustrate the process.
Specifically, for layer number i = 1 ...N , given the output embedding E i − 1 from the last layer and the offset matrix R , the formulation of a transformer layer in an N -layer
I NS N ET can be written as:
Q i , K i , V i , P i =
W iq E i − 1 , W ik,E E
i − 1 , W iv E i − 1 , W ik,R R A i = Q i ⊤ K i + Q
i ⊤ P
i + u i ⊤ K
i + v i ⊤ P
i
V i reduced =
Masked-Softmax ( A i ) V i
V i skipconn =
V i reduced + E i − E i = Feed-Forward ( V i skipconn ; θ i ) , where Q i , K i , V i , P i are query, key, value, position matrices, respectively, for layer i .
u i , v i , W iq , W ik,E , W iv , W ik,R and θ i are learnable model parameters of each I NS N ET layer.
We denote the input word embeddings as E 0 for notation consistency.
The last layer of transformer in
I NS N ET outputs the step-wise context-aware token embeddings.
They will be used to compute slot representations for the insertion steps.
3.2 Slot Representation and Insertion Prediction
With the context efficiently encoded, the subsequent step of an insertion-based generator is to predict the next position and token to insert.
Thus, a representation for each potential insertion slot should be computed.
We hereby show how slot representations can be aggregated from I NS N ET outputs and the potential challenges during this aggregation process.
A naive design of the slot representation is to simply concatenate the representation vectors from the left-neighbor and right-neighbor (in the natural observation order) of the slot as is done in prior works (Stern et al., 2019).
This slot representation can efficiently compute the slot representation for all possible slots in parallel for each time step.
However, the context encoding we obtained is insertion-order sensitive and unaware of the tokens inserted later in I NS N ET .
This naive slot representation thus falls short of capturing the global sequence information.
As a remedy, we propose to also include the last insertion token’s representation to compose the slot representation e ( t ) i ↓
j .
Specifically, given the representations of the left-side neighbor e i − , the right-side neighbor e + j and the last inserted token e t , the slot representation can be computed as:
[Equation]
where f l and f r are linear projections for left-point and right-point representation vectors and ⊕ stands for vector concatenation.
The slot representation panel in Figure 2 (the bottom panel in yellow) illustrates the process.
Note that comparing step 3 and step 4, both compute the slot representation for
B ↓ 
, but the resulting slot representation changes in response to a new step due to the introduction of the global vector.
The slot representation can then be converted into a probabilistic distribution over the vocabulary to predict the log-likelihood of inserted tokens using a log-linear transformation i.e. log p ( x ( t ) i ↓
j | e ( t ) i ↓
j )
= log-softmax ( W p e ( t )
i ↓ j + b ) x
( t ) i ↓ j .
To decide which slot to insert next, a position logit α ( t ) k
= w ⊤ o e ( t ) i k ↓
j k is computed for each slot k with a linear layer.
Then a (log-)soft-max operation is applied on top of the position logits to obtain the log-probability to insert to each candidate slot.
i.e. log o ( x ( t ) i k ↓
j k |{ e ( t ) ↓ } )
= log-softmax ([ α ( t ) 0 , α ( t ) 1 , ... ]) k .
We apply a binary classification q (0 / 1 |· ) on the last inserted token’s representation e t to decide the termination of the generation at each step.
During training, only the final step q (0 / 1 | e n − 1 ) is trained to predict 1; all intermediate steps are trained to predict 0.
In the following section, we will continue to discuss how to formulate the final objective function of I NS N ET using these step-wise distributions.
3.3 Adaptive Parallelization of Insertions
In addition to our efforts for better training efficiency, we also propose an algorithm to adaptively parallelize the generation process of I NS N ET to speed up the decoding while preserving the generation quality.
The idea mimics the graph layerization process in the Dinic’s algorithm (Dinitz, 2006). Specifically, assume that we partition the tokens into different layers , such that the tokens within the Source: Trebuie doar s?
transform?m cuvintele în fapte.
Target:
It is just a matter of translating words into action.
a translating matter just it layer layer layer layer ...
(a) Initial Layerization
Source: Trebuie doar s?
transform?m cuvintele în fapte.
Target:
It is just a matter of translating words into action.
translating matter action it layer layer layer ...
a into (b) Merged Layerization same layer are safe to insert in parallel without affecting the coherence of context.
The algorithm aims to determine the partition such that the log-likelihood estimation of the resulting parallel insertionbased model is as close to the sequential insertion-based models as possible.
The intuition is that the sequential insertion process maximally captures the inter-dependencies between the output tokens.
By staying close to the sequential insertion-based model regarding likelihood prediction ability, we speed up the inference without severely sacrificing the likelihood estimation quality.
We initialize the layerization with a sequential insertion order, which is equivalent to single-node layerization, as is illustrated in Figure 5(a).
Starting from this initialization, we perform I NS N ETDinic to gradually evolve this fully-sequential layerization into a non-autoregressive, parallelized layerization, as shown in Figure 5(b).
We promote a token x ← from its respective slot ( i l , j l ) in layer l to ( i l − 1 , j l − 1 ) in the previous layer l − 1 if two conditions are satisfied: 1)
In layer l − 1 , slot ( i l − 1 , j l − 1 ) does not yet have an assigned insertion; 2) log p ( x ← | e ( l ) i l ↓ j l ) − log p ( x
[Equation]
Here τ is a hyper-parameter that controls how much to parallel.
Intuitively, larger τ usually indicates more tolerance for token likelihood loss caused by parallel decoding, i.e. more parallelization with tradeoff on likelihood estimation accuracy.
The choice of τ is highly task/data-dependent as it is a value associated with the likelihood.
[-In practice, we find setting τ = 2 . 
-]
Note that the promotion of tokens only affects the slot representation computation and the insertion likelihood calculation of the respective insertion.
When encoding the context (as is discussed in Section 3.1), we always treat it as if these insertion operations are performed sequentially.
For a target sequence s <n = [ x 0 , x 1 , ..., x n − 1 ] , after the layerization algorithm, if token x i ↓ j is layerized in the l -th layer and to be inserted in the slot ( i, j ) , we denote it as x li ↓ j .
Suppose we have m effective layers in total.
Denote the corresponding representation for slot ( i, j ) given available context in layer l as e ( l ) i ↓
j and the transformer’s encoding of the partial context until the end of layer l as e ( l ) − 1 .
We denote the token likelihood, position likelihood and termination likelihood of the token x l i ↓ j to be p ( x li ↓
j | e ( l ) i ↓ j ) , o ( x li ↓ j |{ e ( l ) ↓ } ) and q (0 / 1 | e ( l ) − 1 ) , respectively.
The general objective of I NS N ET parameterized by ϕ can be formulated as:
[Equation]
Experiments
We demonstrate the efficiency, flexibility, and model capability of I NS N ET with two sets of experiments.
To show I NS N ET ’s performance and the flexibility to switch between sequential and parallel decoding on datasets with high inter-token dependency (i.e., less suitable for parallel decoding).
We follow the setup in Zhang et al.
and address the unsupervised lexically constrained text generation problem on two datasets Yelp Review and News. 2)
To further verify the effectiveness of
I NS N ET -Dinic, we evaluate I NS N ET as a (partially) non-autoregressive machine translation model [Table] with parallel decoding on three classical datasets: WMT Ro-En, WMT En-De and WAT En-Ja.
The general setup about how hyper-parameters are determined can be found in appendixA.
4.1 Lexically Constrained Generation
Experimental Setup Yelp Review dataset consists of 160K training sequences, 10K sequences for validation and 1k test sequences.
News dataset consists of 268586 sentences in total, of which 10k are randomly selected as validation set, 1k for testing.
YAKE 2 is performed to the test split of each dataset to extract lexical constraints.
We vary the hyperparameter τ in the range of { 10 . 0 , 3 . 0 , 1 . 0 , 0 . 3 , −∞ ( fully-sequential ) } and collect the results on both datasets.
For position prediction, we are inserting into slots with positions lying in the top 70% of the position distribution mass.
For token prediction, we are doing top- { 1 , 1 , 3 , 3 , 5 } sampling over the vocabulary distribution.
The results are shown in Table 2, Figure 6(a) and 6(b) [Table]
B L E USequential Ratio
InsNet
POINTER
LevT
InDIGO Autoregressive 𝜏 = 10.0 𝜏 = 3.0 𝜏 = 1.0𝜏 = 0. 𝜏 = −∞ ( [Figure]
Yelp Review 3.75 4.03 4.
B LE
USequential Ratio
InsNet POINTER
LevT InDIGO
Autoregressive 𝜏 = 10.0 𝜏 = 3.0𝜏 = 1.
𝜏 = 0.
𝜏 = −∞ (
[Figure] News Discussion 1) I NS N ET is able to significantly outperform most previous works on the quality-latency trade-off spectrum.
With more sequential ( i.e. smaller τ ) it is generally to achieve the new state-of- generation
[Table]
I NS N ET guarantees the perfect incorporation of keywords due to its insertion-based nature.
Also, its improved performance is consistent on both datasets.
2) When setting τ larger to encourage more parallelization of decoding, I NS N ET is mostly able to parallelize the insertions with good preservation of the generative quality.
Compared to the vanilla uniformly-parallel setup (I NS N ET -uniform)
[Table]
I NS N ET -Dinic significantly improved the effectiveness of parallel decoding.
Generation sample We also show two samples of the generation process in Table 3.
It is interesting to observe that the model tends to insert punctuation and function words (e
[Table]
g., Articles and
Prepositions) first, and then more concrete content words.
4.2 Non-autoregressive Machine Translation
We train I NS N ET -Dinic models with τ = 2 .
for extended investigation of the performance on machine learning problems.
The results in Table 4 show that, I NS N ET -Dinic is able to achieve comparable or even better performance compared to previous state-of-the-art non-autoregressive (insertion-based) machine translation models in terms of generation quality and latency
[Table]
4.3 Training Time Analysis
To show the training efficiency of the proposed model [Table] we hereby do an empirical and theoretical analysis on the candidate models’ training procedure.
See Table 5.
For empirical results, the statistics are collected on the Yelp Review unsupervised lexically constrained text generation problem.
All results are collected on a single NVIDIA RTX3090 GPU.
The transformer Seq2seq baseline is trained [Table] + KD 33.91 28.
Θ(log n ) ≈ 16 .103ms
[Table]
as a Plan-And-Write model.
POINTER’s training time per epoch is calculated with the official implementation with mixed precision supported by the NVIDIA APEX library 3 .
Discussion
The results in Table 5 confirm the significant improvement in training efficiency of
I NS N ET over most previous baselines.
In addition, this concretely shows that it is not practically affordable to obtain a sequential insertion-based generator directly with InsT/LevT. Note that SAO of
InDIGO is an offline, inference-only algorithm that does not require any computation of gradients.
Thus its constant factor is significantly smaller than other O ( n ) gradient-propagating procedures.
According to its original paper and our previous experiments, InDIGO practically needs this process to achieve reasonable performance.
4.4 Ablation Study on the Effectiveness of Components
Since the model components of InsNet are closely entangled, it is nontrivial to do ablation study. We design experiments to show that 1).
The proposed offset matrix is a powerful insertion-oriented position encoding with significantly better capacity.
We show that this particularly helps the model learn to terminate the sequence in correct timings.
To study this, we truncate the offset matrix to range [ − 1 . 0 , 1 . 0]
(InsNet-truncated) so that it is similar to InDIGO’s position encoding, and 2) the global representation is necessary for computing the slot representation in I NS N ET .
It in general helps to improve the sequence termination performance, and leads to convergence to better local optimum.
We report the results of an ablated version of InsNet without global representation (InsNet-noglobal).
The termination NLL (see Sec 3.3 for its definition) and BLEU scores for each model can be found in appendix Table 6.
We also show some random samples from the ablated variants and the failure cases can be find in appendix Table 7.
Conclusion & Future Work
We propose I NS N ET , an efficient and performant insertion-based generator that supports sequential and parallel decoding.
Experiments on two unsupervised lexically constrained text generation datasets and three machine translation datasets show the advantages of I NS N ET over previous methods.
Future work can explore obtaining a large-scale pre-trained version of I NS N ET for further fine-tuning under different downstream scenarios. We anticipate such insertion-based models to have better compositional generalizability and controllability.
Acknowledgement
We thank I-Hung Hsu, Dr. Rujun "RJ" Han, Te-Lin Wu, Kai-Wei Chang, Sarik Ghazarian, Alexander
Spangher, Yining Hong, Mingyu Derek Ma and all other members from PlusLabNLP/UCLANLP group for their participation in initial discussions and comments on paper writing.
We would like to thank Huggingface for their great work of the Transformers project.
The work is partially supported by a Meta SRA and an Amazon research gift.
A Appendix A.1 General Setup of Experiments
For all language generation tasks, based on whether the data is pre-processed upon release or not, either a BPE-based or a classical tokenizer is applied.
Each of the evaluated transformer models, if not otherwise stated, is implemented as a base -sized transformer model, which has 12-layers withattention head and 768 hidden dimensions.
The batch size is set to 256.
In cases where the model size exceeds the device capacity, the cumulative gradient trick is applied to support an equivalent optimization effect.
The learning rate is selected from {5e-5, 1e-4, 2e-4}.
The dropout rate is selected from {0.1, 0.2}.
The weight decay rate is set to 0.02.
All the models are trained with 1000 warm-up iterations and a maximum of 200000 iterations of training.
A linear-decay learning rate scheduler is applied for fine-grained training of the model.
If presented with a development set, the training will be early-stopped when the model stops improving for 5 consecutive epochs.
For I NS N ET -Dinic, unless otherwise stated, the parallelization fine-tuning lasts for 5 epochs typically.
The setups of MT tasks follow previous works, including, but not limited to InsT/LevT and InDIGO (Stern et al., 2019; Gu et al., 2019b,a).
Models are implemented based on libraries like PyTorch(Paszke et al., 2019)
(for the general construction of deep learning models, License: BSD) and huggingface’s Transformers(Wolf et al., 2020) (for specific implementation of transformer-based models, License: Apache-2.0).
Some of our baselines also used NVIDIA APEX for efficient mix-precision training 4 .
We thank them for their contributions to the community.
With the best of our effort we use datasets that contain either user-identity agnostic contents or those that could contain publicly reported information about famous, well-known individuals (for the News dataset).
All results that could contain randomness, if not otherwise stated, is collected from averaging the results of 3 individual runs of sampling.
The variances of the results won’t affect/reverse the conclusion.
A.2 Extended Study and Qualitative Examples
Due to the page limit, we could only manage to include some qualitative examples here in the appendix.
Table 6 contains some basic quantitative results for the ablated variants of
InsNet. Table 7 contains a few failure cases of the ablated variants of InsNet.
Tablecontains some randomly sampled trajectories on the News dataset, showing how the context is gradually enriched by InsNet in sequentially-decoded insertion-based text generation. [Table]
[Table]
intriguing meals and traditional .
fun atmosphere .
great prices .
great service .
[Table]
... very intriguing .
great service , great staff , and the whole meals are amazing .
i love the place .
love the food here here . ...
( non-terminating )
[Table]
very intriguing meals .
great atmosphere .
and service was great .
food was amazing .
service was delicious .
traditional italian cuisine , very fun .
atmosphere is great .
prices are good and very reasonable . ! .
[Table]
i have been here several times .
great atmosphere and the service is always great .
love the food , and the best steak salad .
it is fantastic !
[Table]
i ve been coming here multiple times in years now .
great food , good food and delicious .
also , the service .
great atmosphere and atmosphere .
service and food was great .
great experience here .
love food and salad , and the staff is very friendly ! . ! .
[Table]
... everything is very delicious .
the whole menu was awesome , especially .
we could have ever eaten since we found this place .
we have had everything on the food , kids menu , , and large portions and food , great service .
great place is awesome .
great service , clean , fresh , tasty , tasty , hot , , , good food , service , ...
( non-terminating )
[Table]
this place has been a long time to take out of my favorite spot .
service is friendly and fast , the food here are excellent .
fantastic sandwiches .
this is my favorite spot.
[Table]
... we love it here , and it s always so delicious .
the service is amazing .
the owner owner is always nice and has been super friendly service , and one of my favorite lunch spot .
the food is always fantastic .
oh and delicious .
and there is the best value .
the food is so fresh and the sandwiches are great .
is my favorite .
, it is fantastic .
i .. ...
( non-terminating )
[Table]
delicious food and service .
great customer service and food .
good food , clean atmosphere .
nice staff .
good breakfast .
great service , fresh food and atmosphere .
will come here everyday ! .
[Table] [Table]
thought
[Table]
i ’ ve had a good run and if this is the way i go , bring it on
[Table] [Table]
but he
[Table]
played in his
[Table]
three years completed .
but he
[Table]
played in his
[Table]
three years
[Table]
but he has played in his team three years before completed the .
but he has played in
[Table]
the
[Table]
but he has played in his
[Table]
for three years before being completed the .
but he
[Table]
played in
[Table]
the but he has played in his
[Table]
for three years
[Table]
year
[Table]
in the
but he
[Table]
in
[Table]
years
[Table]
year
[Table]
he in
[Table]
but
[Table]
he
[Table] [Table]
local
[Table]
rule out
[Table]
[Table]
day decided started focus on day decided started focus on .
the day decided started focus on .
the day decided to started focus on . the day , decided to started focus on .
the day , he decided to started focus on . the day , he decided to get started focus on .
on the day , he decided to get started focus on . on the day , he decided to get started focus on the .
on the day , he decided to get started focus on the court . but on the day , he decided to get started focus on the court .
but on the day , he decided to get started to focus on the court . but on the next day , he decided to get started to focus on the court .
but , on the next day , he decided to get started to focus on the court . but , on the next day , he decided to get started to focus on the court for .
but , on the next day , he decided to get started to focus
[Table] [Table]