Efﬁcient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Effective training of today’s large language models (LLMs) depends on largebatches and long sequences for throughput and accuracy.
To handle variable-lengthsequences on hardware accelerators, it is common practice to introduce paddingtokens, so that all sequences in a batch have the same length.
We show in this paperthat the variation in sequence lengths in common NLP datasets is such that up to50% of all tokens can be padding.
In less common, but not extreme, cases (e.g.
GLUE-cola with sequence length 128), the ratio is up to 89%.
Existing methodsto address the resulting inefﬁciency are complicated by the need to avoid ‘crosscontamination’ in self-attention, by a reduction in accuracy when sequence orderinginformation is lost, or by customized kernel implementations only valid for speciﬁcaccelerators.
This paper introduces a new formalization of sequence packing inthe context of the well-studied bin packing problem, and presents new algorithmsbased on this formulation which, for example, confer a 2x speedup for phase 2pre-training in BERT.
We show how existing models can be adapted to ensuremathematical equivalence between the original and packed models, meaning thatpacked models can be trained with existing pre-training and ﬁne-tuning practices.
Introduction
Many language datasets, including the de-facto pre-training dataset for BERT—Wikipedia, havea skewed distribution of sequence lengths (see Figure 1).
However, typical machine learningaccelerators and their corresponding libraries, exhibit poor performance when processing variablelength workloads.
A simple mitigation is to set a maximum sequence length and to pad shortersequences with padding tokens.
This naive batching is widely used and provided in the vanilla BERTimplementation as well as the Hugging Face framework [32].
Its effect is enhanced by the ofﬂinedataset generation process which, in BERT, attempts to “pack” together sentences so as to ﬁll thesequence length as completely as possible [8].
We improve this process at a whole-dataset level.
We show that, even after this pre-processing, padding tokens represent 50% of all tokens of the
Wikipedia pre-training dataset at sequence length 512 .
Thus, by avoiding processing the paddingtokens one can get a 2x speed-up for phase 2.
Overall, the lengths range between 5 tokens up to 512 .
Samples of length 512 represent only 23 .
5% of the dataset,
Beyond the simple batching, other solutions have been addressed in the literature, and in open-sourcesoftware implementations.
When processing sequences, most libraries and algorithms mentionpacking as reference to concatenating sentences from the same document (BERT) or from differentdocuments (BERT, T5 [24], GPT-3 [4], and RoBERTa [16]) as they arrive (GREEDY) from thesource dataset to generate the training dataset.
None of the respective papers addresses the packing efﬁciency, i.e., remaining fraction of padding.
To “separate” sequences from different documents, aseparator token is introduced.
However, this is not sufﬁcient and can have a signiﬁcant impact onperformance.
This is discussed only in the RoBERTa paper which shows that downstream F1 scoresget consistently reduced on average by 0 .
Alternative common approaches to overcome the largeamount of padding in many datasets are “un-padding” as in Effective Transformer [5] and sortedbatching (SORT) as in Faster Transformer [21], lingvo [28] fairseq [22], and RoBERTa.
However, forrunning efﬁciently on arbitrary accelerators, these approaches require substantial hardware-speciﬁclow-level code optimizations only available on GPUs. Further details are in Sections C [1] and 4.4.
Beyond language models, packing has been also present in other areas of machine learning, howeverwith little to no exploration in the literature and mostly hidden in some libraries without any furtherdiscussion.
For example, PyG (PyTorch Geometric) combines multiple small graphs in a batch toaccount for the large variation in size and to optimize the hardware usage when training a Graph
Neural Network (GNN).
Another example is the RNN implementation in PyTorch which introduces a“PackedSequence” object and states that “All RNN modules accept packed sequences as inputs” butdoes not address how sequences are packed efﬁciently and how the processing of packed sequencesis implemented in an efﬁcient manner while avoiding interaction between sequences.
Even thoughwe focus on BERT [6] and other transformers in this paper, the general principles can be transferredto many more machine learning algorithms with differently sized data samples.
In this paper, we formally frame the packing problem in transformer based models, and provide somesolutions, showing that sequences can be packed efﬁciently, separator tokens are not required, andcross-contamination can be avoided with little overhead.
In summary, the contributions of the paper are as follows.
In Section 2, we produce histograms of avariety of datasets showing the high percentage of padding tokens.
In Section 3.1, we present two newdeterministic and efﬁcient packing algorithms based on established solvers which efﬁciently packdatasets with millions of sequences in a matter of seconds (or less).
In Section 3.2 and Section 3.3, wedescribe ‘cross-contamination’ —the cause of the accuracy reduction which separator tokens do notmitigate— and show how the BERT model can be adjusted to show the same convergence behavioron packed and unpacked sequences.
We empirically show that the proposed packing algorithmsproduce a nearly-optimal packing scheme for Wikipedia pre-training dataset (Section 4.1) and morein the Appendix.
In Section 4.2, we demonstrate that the convergence of the BERT large model onthe packed dataset is equivalent to that on the un-packed dataset with 2x throughput increase on the
Wikipedia sequence length 512 pre-training dataset.
Further experiments underline the necessity andefﬁciency of our changes.
Sequence length distributions
BERT is pre-trained using masked-language modelling and next-sentence prediction on a largecorpus of Wikipedia articles.
Each sequence is composed of one < CLS > token followed by theﬁrst “segment” of sentences, followed by a <
SEP > token, and then ﬁnally the second “segment” ofsentences.
Because these “segments” are created in sentence-level increments there is no token-levelcontrol of sequence length.
Furthermore 10% (default value, [7]) of sequences are intentionallycut short.
This leads to signiﬁcant levels of padding, especially for longer maximum sequencelengths (see Figure 1 and Section J[1]).
At sequence length 128 (commonly used in phase 1 ofpre-training) the theoretical speed-up is around 1 . 2 , at sequence length 384 this increases to 1 . 7 , andﬁnally at sequence length 512 (commonly used for phase 2 of pre-training) it is 2 . 0 .
Despite thewidespread use of the Wikipedia dataset for pre-training BERT such histograms have, to the bestof our knowledge, not been published previously.
This has perhaps lead to the underestimation ofthe speed-up opportunity available.
To put things into perspective, the sequence length 512 datasetcontains 8 .
33 billion tokens, of which 4 .
17 billion are padding tokens.
Note that the skewed sequence length distributions are neither limited to Wikipedia, as shown with
GLUE [30, 31] from Section L[1] and SQuAD 1.1 [25] from Section K[1] ( 2 . 2 x speed up), to BERTtraining, as shown with LibiSpeech text distributions [23] from Section M[1], nor to text itself,given the LibriSpeech audio data distributions, and the QM9 molecular data [27, 26] ( 1 . 6 x speed-up,
Section Q[1]).
All distributions can be found in Figure 1.
Since LibriSpeech audio data is skewed tolonger sequences, only 1 . 3 x speed-up could be achieved despite the theoretical maximum of 1 .
6 x .
For all other cases, the algorithms presented in Section 3.1 lead to close to optimal packing.
Methods
Our approach consists of three distinct components.
Firstly, we pack the n data samples efﬁcientlyduring pre-processing to make full use of the maximum sequence length, s m (Sections 3.1 and F).
Secondly, we introduce a series of model changes in Section 3.2 that preserve the equivalence withthe original BERT implementation.
The changes include a self-attention mask to prevent the modelfrom attending between different sequences in the same pack (Section 3.2.2), and an adjustmentof the the positional embeddings (Section 3.2.1) to handle packs of sequences.
Other componentsof the model, such as the feed-forward layer [29], operate on a per-token basis and do not requiremodiﬁcation for pre-training.
In Section 3.2.3, we also demonstrate how to compute a per-sequenceloss and accuracy for NSP and downstream ﬁne-tuning tasks.
Thirdly, we provide suggestions forhyperparameter adjustment (Section 3.3) that lead to analogous convergence behavior between thepacked and un-packed BERT implementations.
Additional videos and animations are provided assupplemental material.
3.1 Packing algorithms
The widely studied and well established bin packing problem deals with the assignment of items intobins of a ﬁxed capacity such that the number of utilized bins is minimized.
It has been known fordecades if not centuries.
Since an exact solution is strongly NP-complete [14], numerous approximatesolutions have been proposed [12, 15, 13, 36].
Since most existing approximations have a highcomplexity of at least O ( n log n ) , we propose two new heuristic ofﬂine algorithms that are tailoredto the NLP setting applied to the whole dataset.
For a detailed introduction to packing see Section F.3.1.1 Shortest-pack-ﬁrst histogram-packing (SPFHP)
Shortest-pack-ﬁrst histogram-packing (SPFHP) works on the bins in the sequence length histogram(with bin size 1) rather than the individual samples.
The histogram is traversed in sorted order fromlongest to shortest sequences.
Then, to pack the data during the traversal, we apply the worst-ﬁtalgorithm [12, 36] such that the histogram bin being processed goes to the “pack” 1 that has the mostspace remaining (“shortest-pack-ﬁrst”).
If the histogram bin does not ﬁt completely, a new pack iscreated.
We also limit the packing depth , in other words the maximum number of sequences thatare allowed in a pack.
Therefore, an existing pack is only extended if it is not already at maximumpacking depth.
The detailed code for the algorithm is provided in Listing 3.
The time and spacecomplexity of the algorithm are O ( n + s 2 m ) and O ( s 2 m )
(Section G.2[1]).
3.1.2 Non-negative least squares histogram-packing (NNLSHP)
The proposed NNLSHP algorithm is based on re-stating the packing problem as a (weighted) nonnegative least squares problem (NNLS) [3] of the form wAx =
wb where x  0 .
The vector b is thehistogram containing the counts of all the sequence lengths in the dataset.
Next, we deﬁne the Amatrix (the “packing matrix“) by ﬁrst generating a list of all possible sequence length combinations(“strategies”) that add up exactly to the maximum sequence length.
We focus speciﬁcally on strategiesthat consist of at most 3 sequences per pack (independent of b ) and encode each strategy as a columnof the sparse matrix A .
For example, a strategy consisting of the sequence length 128 , 128 , and256 in represented a column vector that has the value 2 at the 128 th row, the value 1 at the 256 throw, and zero at all other rows.
The variable x describes the non-negative repetition count for eachstrategy.
So a 24 in the i th row of x means that the strategy represented by the i th column of A shouldrepeat 24 times.
Moreover, in the un-weighted setting, Ax = b states that we would like to “mix” thepre-deﬁned strategies (columns of A ) such that the number of samples matches the histogram b , andwhere each strategy is used x  0 times.
We use the residual weight w to control the penalizationof the Ax  b residual on different sequence lengths (different rows of b ).
Heuristically, we setthe weight of 0 . 09 for all sequences of length 8 or smaller because they are considered acceptablepadding sequences while all other sequence lengths get weight 1 .
We discuss this heuristic choice ofparameters in Section F.4.5 and F.5[1].
The overall efﬁciency of the packing is not greatly inﬂuencedby the weighing (less than 1% extra speed-up).
After solving wAx =
wb for x  0 using an off-the-shelf solver, we obtain a ﬂoating point solution,which means that the repetition counts are not necessarily integers.
Since we cannot use a non-naturalnumber of strategies, we round the solution ˆ x to the nearest integer.
The error introduced by thisrounding is found to be negligible (a few hundred sequences in the worst case) compared to the sizeof the dataset (millions of sequences).
The time complexity and space complexity of the algorithmare O ( n + s 5 m ) and O ( s 3 m ) .
Further details are provided in Section F.4.
3.2 packedBERT: model changes
This section describes how any vanilla BERT implementation should be modiﬁed for packed sequenceprocessing, such that the behavior of the model is the same as when processing unpacked sequences.
Preserving the mathematical equivalence is necessary to ensure existing BERT pre-training andﬁne-tuning practices remain valid, as well as being required by benchmarks such as MLPerf™ [17].
The presented approaches and principles apply to a variety of other models.
3.2.1 Adjust positional embeddings
The BERT model uses three types of embeddings: token, segment, and positional embeddings.
Thelatter is canonically implemented as a bias add operation, rather than a full embedding look-up.
Thisis possible because the positional indices increase linearly for every sequence.
However, when usingthe packed data format the position index needs to be reset with each new packed sequence.
Forinstance, when packing two sequences one of length 2 and one of length 3 , the positional embeddingindexes that need to be picked up are [0 , 1 , 0 , 1 , 2] .
To achieve this, the bias add needs to be replacedby an embedding look-up to extract the correct positional embedding for each token in the pack.
Thisalso requires keeping an extra input which speciﬁes the position of each token in its sequence.
Thisrequired adjustment has only a minor impact on absolute accuracy/loss (see Section 4.2 and 4.2.1).
3.2.2 Adjust attention masking
[Figure]
To maintain an implementation that is consistent with the un-packed version, tokens from differentsequences within a pack should not be able to attend to each other.
This is typically achieved inother implementations by unpacking the sequences using custom attention kernels and then doingthe attention per-sequence [5].
Instead, we propose directly masking the attention matrix with ablock-diagonal mask before the attention softmax.
This is straightforward to implement in modernframeworks (see Figure 2).
Naturally, there is a cost to both the mask construction and applyingit to the attention matrix.
However, it is required to keep the accuracy (see Table 1, Section 4.1,
Section 4.2).
See also the code of the deprecated tensor2tensor library and our own provided code.
3.2.3 Adjust per-sequence loss and accuracy
Canonical implementations of BERT compute the cross-entropy loss for the masked language modelon a per-token basis.
However other NLP tasks, such as SQuAD, compute the loss and accuracy ona per-sequence basis.
This section discusses how to handle such tasks when training with packedsequences.
Simply feeding packs of sequences to the same implementation of cross-entropy wouldresult in a per-pack weighted loss.
In other words, the overall loss on the micro-batch would sum-upthe losses on the individual packs, rather than individual sequences.
As a result, the model wouldconverge to a different optimum than when running with the un-packed implementation.
For instance,a pack of a single sequence would contribute to the loss with the same weight as a pack of threesequences.
To recover the per-sequence averaging behavior of the canonical un-packed BERT implementation,we effectively “unpack” the incoming logits and labels.
Once the sequences have been unpacked,we can compute the loss on each sequence separately as usual and then add up the losses.
However,rather than looping through the sequences index, we compute on all indexes in parallel (see Figure 2).
This minimizes the latency overhead of un-packing the loss calculation.
As an example, we show howper-sequence loss can be implemented for the pre-training task.
We use the “masked lm weight”
[7]input tensor to represent which sequence a given masked token belongs to (0, 1, 2 and so on).
Thisis consistent with the canonical BERT implementation where this input takes a value of either 1(belonging to the sequence) or 0 (belonging to padding).
The full methodology is detailed in Listing 5and can be applied to other classiﬁcation or pre-training tasks.
3.3 Adjust hyperparameters
In terms of convergence behavior, the primary consequence of packing is an increase in the effectivebatch size (with respect to number of sequences and real tokens) with some added variation overdifferent iterations.
If we look on the sentence level, the number of sentences in one batch increasesby the packing factor.
Similarly, the number of tokens in one batch increases.
Hence, hyperparametersthat are sensitive to these numbers need to be adjusted.
A direct solution is to reduce the computational batch size by the packing factor (average number ofsequences per pack) and keep all other hyperparameters the same.
For example, if the packing factoris 2, cutting the gradient accumulation count by half is sufﬁcient.
The advantage of this strategy is thatno ﬁne-tuning of hyperparameters is required and performance curves are comparable.
However, thisapproach might be not desirable as it might imply under-utilizing the memory/compute, especially ifthe micro batch size needs to be reduced.
Hence to preserve batch size and optimize hardware utilization, we additionally propose an approximate heuristic for updating the decay parameters of the LAMB optimizer [35] .
For a packed datasetwith a packing factor p , we update the decay parameters as:  1 :=
p 1 ,  2 :=
p 2 .
For p = 2 , thiscorresponds to the exact parameters for calculating momentum and velocity, when updating with thesame gradient twice (Section D).
A common approach is to scale the learning rate with the batch size.
However, our experiments in Section 4.2 show that this reduces convergence speed.
Since these adjustments are only heuristics the convergence of the model will be comparable but notidentical.
In particular, it is unlikely that simply adjusting the hyperparameters will fully undo theimpact of the increased batch size.
However, with these adjustments, researchers should be able tocontinue to use existing conﬁgurations.
Experiments
4.1 Bin packing algorithm comparison
We evaluate our algorithms using the following metrics: number of packs , number of all tokens ,number of padding tokens , solution time of the packing algorithm (after histogram and strategycreation), number of strategies used , packing efﬁciency (the fraction of non-padding tokens in thepacked dataset), the speed-up achieved compared to not packing (depth 1), and the average numberof sequences per sample ( packing factor ).
For SPFHP, we analyse different (maximum) packingdepth, since packing is less efﬁcient with smaller depth and we want to get a general understandingon how the packing depth inﬂuences the processing time.
For NNLSHP, we focus on packingdepth 3 because it packs the data sufﬁciently well.
For the speed-up analysis, we focus on theintelligence processing unit (IPU) [11] (IPU-M2000, 16 accelerator chips), BERT phase 2 pretrainingsetup as in Section 4.2.
A GPU dynamically loads the code into the accelerator; in contrast, the IPU works with a static pre-compiled engine that gets loaded onto the chip at the start of the run.
While other approaches result in excessive padding or continuous changes of the code, our approachcan work with the same code for the whole dataset.
So in this setting the IPU architecture wouldespecially beneﬁt from our approach since it avoids code changes.
Nevertheless, it can be appliedto any implementation on GPU or TPU.
For determining the speed-up, we take advantage of theprecompiled kernel.
Since time measurements are quite noisy, we can proﬁle the kernel and howmany cycles it takes for processing a batch.
That way, we can determine the overhead (in cycles)from processing the additional attention masking and for unpacking the loss.
Combining overheadand packing factor , we get the speed-up estimate.
No experiment repetitions are required since thealgorithms and measurements are deterministic.
[Table]
Packing depth describes the maximum number of packed sequences.
NONE is the baseline BERT implementation, whereas SORT corresponds to sorted batching, and GREEDY concatenates sequences as they arrive until they would exceed 512 tokens.
Setting no limit resulted in a maximum packing depth of 16 .
EFF iciency is the percentage of real tokens in the packed dataset.
The p acking factor describes the resulting potential speed-up compared to packing depth 1.
With overhead (OH) , we denote the percentage decrease in throughput due to changes to the model to enable packing (such as the masking scheme introduced in Section 3.2.2).
The realized speed-up is the combination of the speed-up due to packing (the packing factor ) and the decrease in throughput due to the overhead on the IPU.
It is used to measure the relative speed-up in throughput and the overhead from masking and loss adjustment.
SORT can be only efﬁcient on GPUs (see Section 4.4).
The main results for the performance metric evaluation are displayed in Table 1.
The processingtime for SPFHP on an
Intel(R) Xeon(R)
{+Gold 6138 CPU with 2 . 
+}
GHz, 80 nodes, and 472 G RAMwas around 0 .
03 s and independent from the packing depth.
Classical First-Fit-Decreasing requires87 - 120 s, a lot of memory, and scales almost linear with the number of samples.
We see that theoverhead slightly increases with packing depth but that the beneﬁts of packing outweigh the cost.
Thebest speed-up is obtained with NNLSHP at depth 3 which required 28 .
4 s on the CPU for processingand ran out of memory for larger depth.
With a value of 1 . 913 , it is close to the theoretical upperbound of 2 . 001 .
The results show that efﬁciency, packing factor, and speed-up can be viewed interchangeably.
The amount of time needed to process a sample (a pack of sequences) is barely changedrelative to the un-packed implementation.
The packing factor, or the improvement in efﬁciency, effectively provide an accurate estimate of the speed-up.
GREEDY packing as used in T5 showsto be quite inefﬁcient and sorted batching (SORT) is highly efﬁcient in avoiding padding but theresulting different computational graphs cause a major overhead on the IPU that exceeds the beneﬁtsof avoiding the padding.
Since we made our algorithm and code public available, results have beenreproduced with a different framework on the Habana Gaudi accelerator [10] and conﬁrmed that ourapproach is hardware and software independent giving it a huge advantage over existing approaches.
4.2 MLPerf™ phase 2 pretraining setup: learning curves and hyperparameter adjustment
For depth 1 (classic BERT) and NNLSHP with depth 3, we additionally evaluate on the MLPerf™ version 0 .
7 BERT pre-training benchmark [17].
Brieﬂy, this involves training from a standard checkpointto a masked-language model accuracy of 71 .
2% using 3 million sequences with a maximum length of512 tokens (refer to [19] for details).
Following this standardized benchmark supports reproduction ofresults even on other systems and makes sure that the reproduction effort is moderate and setup rulesare clearly documented.
We compare the resulting speed-up as well as the respective learning curvesby evaluating the data on a held-out validation dataset.
The objective of this additional evaluation isto analyse if convergence behavior is changed by the packing strategy and if the theoretical speed-upcan be achieved in practice.
With packing, we effectively increase the average batch size by the packing factor ( ⇡ 2 ).
However,with a different batch size, different hyperparameters are required (see Section 3.3) and there is nomapping that will generate exact matching of results but only heuristics.
In a ﬁrst comparison, weuse the same hyperparameters when comparing packed and unpacked training except for cutting theaccumulation count by half.
This way, we make sure that the batch size is constant on average andwe have the same amount of training steps.
In the second comparison, we evaluate our heuristics andhow they compensate the difference in batch size.
This setup is more desirable because it is beneﬁcialto use the hardware to its full potential and cutting the batch size by half usually reduces throughput.
In the third comparison, we compare two optimized setups.
In these two cases, packing takes half theamount of training steps.
The learning curves are displayed in Figure 3.
In the ﬁrst setup, we see the curves almost matchingperfectly when normalizing by the numbers of samples processed.
Differences can be explainedby the variation of the number of sequences in the packing batch, and general noise in the trainingprocess.
Especially after the initial phase, the curves show a near-identical match.
The second setupshows bigger differences since changing the batch size and hyperparameters changes the trainingdynamics.
We observe slower convergence early on in training due to the increased batch size.
Thisis expected.
The adjustment of the learning rate actually decreases performance probably because wecorrect for the increased number of sequences already in the modiﬁed loss.
With the adjustment ofthe decay parameter of LAMB, we see matching performance at the later training stages.
However,it is not feasible to completely recover the early convergence behavior of the smaller batch size byadjusting the hyperparameters.
For instance doubling the batch size of unpacked BERT to 3000and adjusting the LAMB decay parameters leads to more of a slow down in convergence thanwhen running packed BERT with a batch size of 1500 and a packing factor of 2 .
n practice, ourimplementations exceeds the estimated 1 . 913 maximum speed-up.
This estimate is based on thereduction in the computational work needed to process the dataset.
However, packing the data alsoreduces the latency of the transferring the data to the device.
Figure 3 shows that the realized totalspeed-up from packing exceeds 2 x .
4.2.1 Ablation study
So far, we have shown that with the introduced adjustments, we can match the accuracy of unpacked
BERT.
In the following, we analyze in how far the masking adjustment is required.
In Figure 4, wecan see that without our adjustments, training loss and accuracy worsen drastically and a longertraining time does not lead to a recovery.
When not adjusting the positional embedding, the loss andaccuracy almost match.
However, the accuracy stalls at 71 . 8% and does not reach the target accuracyof 72 .
So overall, both adjustments are crucial to avoid a reduction in performance.
When running packed BERT without the NSP loss but keeping everything else the same in a fulltraining setup, we observed that downstream performance on SQuAD reduced the F1 measure by1 .
31% and EM by 1 . 15% .
Hence, we do not consider removing NSP as done in approaches like
RoBERTa and T5 as discussed in Section I.
4.3 Full pretraining and SQuAD ﬁnetuning
Packing slightly violates the i.i.d.
assumption of data.
Thus, we have to check that downstreamperformance is not impacted by packing.
This is especially relevant in a full training setup withouta starting checkpoint.
To this aim, we show that the packed and unpacked SQuAD 1.1 scores arecomparable after a full-pretraining of BERT base and large plus ﬁne-tuning.
During pre-training,in order to avoid giving an advantage to packing by further hyperparameter tuning, we reduce thegradient accumulation count for the packed BERT training for phase 1 and phase 2 to match, onaverage, the total number of sequences that get processed before each weight update.
With thisapproach, we can use the same hyperparameters and number of training steps but process each batchfaster by avoiding the processing of padding.
This gives a slight disadvantage to the packed run interms of machine utilization, as explained in Section 3.3 and is different to the speedup analysis in
Section 4.2.
For Phase 2, we use sequence length 384 since longer range attention is not relevantfor SQuAD 1.1.
The respective speed-ups from packing for BERT base and large are shown in
Table 2: the realized speed-up, measured as the quotient of the throughputs between the packedand unpacked runs, is slightly lower to the theoretical throughput (i.e. the packing factor) due tothe packing overhead.
Further learning curves with the loss function and accuracy are provided in
Section P. For the ﬁne-tuning training on SQuAD 1.1, we do not use packing.
The scores, computedas the median of 10 different seeds, are displayed in Table 3.
They are comparable to the referenceones in [6]: for BERT base (resp.
large) the F1 score is reduced by 0 .
2% (resp. 0 .
and the EMscore increases by 0 .
3% (resp. 0 .
[Table] [Table]
4.4 Scaling analysis: Impact of accelerators count
A further advantage of packing over competing un-padding approaches is the inherent load balancingprovided by packing.
So called un-padding approaches rely on dynamically launching custom kernelsthat ignore padding.
A stated advantage of such implementations is the ability to avoid computingthe complete (512 x 512) attention matrix.
This provides additional computational savings comparedto packing, where the attention matrix is computed in its entirety and then masked.
Because ofthese additional savings, un-padding can exceed the theoretical upper bound for speed-up frompacking ( 2 . 013 on Wikipedia).
As a result of the dynamic nature of the approach, the processingtime with un-padding is different for each sequence in the batch, and the amount of time required toprocess a batch of sequences will be determined by the processing time of the longest sequence inthe batch (with the sequences being processed in parallel).
Furthermore, in the multiple acceleratorsetting the processing time on each device will vary depending on the sequences in the batch that itreceives.
Devices which ﬁnish early have to wait for the slowest device to ﬁnish before exchanginggradients.
This load-imbalance between the devices (and inside the batch) leads to a considerabledecrease in the speed-up from un-padding as the number of accelerators is increased (see Figure 5and Section E [1]).
In contrast, packing (our approach) is inherently load-balanced.
The processingtime on each accelerator is independent of the content inside the batch received by the device.
Anynumber of accelerators can therefore operate in unison without having to wait for the slowest batch toprocess (all per-device batches are equally fast).
Conclusion
Whereas packing is a well known concept, this paper sheds a new light onto it in multiple aspects.
First, we visualize the sequence length distributions of multiple datasets not just from languagedomains but also audio and molecular domains to emphasize that packing is beneﬁcial for a lot ofdatasets and that in many cases, more than 2x acceleration can be achieved by removing 50% ormore padding.
Second, we provide two new highly efﬁcient packing approaches based on establishedsolvers that leave almost no padding and that can tackle arbitrarily large datasets in a matter ofseconds, in contrast to existing approaches that are slow and suboptimal.
Third, we demonstrate thatwithout adjusting the sequence processing algorithm (e.g., BERT) to the packed sequences, predictiveperformance is reduced.
Thus, we propose several model adjustments that are all necessary to keeppredictive performance.
Last but not least, we prove that, thanks to such adjustments, predictiveperformance is preserved as if no packing was used — but speed signiﬁcantly increases, especiallysince the adjustments come with an overhead of less than 5% .
We prove in our experiments thatdownstream performance is not impacted by packing and that the anticipated 2x acceleration can beachieved.
In the future, an interesting direction is the packing of images of different sizes to help acceleratecomputer-vision applications.
This is especially relevant given the recent advances in the use oftransformer-based approaches in the computer vision domain, for example the visual transformer [33].
Note that many images come in different shapes and resolutions and packing them can be a newapproach to tackle this diversity instead of casting them all to the same resolution and shape.
Maskingout the self-attention within transformers is easier to implement than avoiding cross-contamination ofconvolutions applied to packed images.
Future work should explore improving the performance ofother models (RoBERTa, GPT-3, T5) by avoiding contamination between non-contiguous segmentsfrom different documents.
Even BERT itself might beneﬁt from avoiding contamination between thetwo concatenated segments.