Massive-scale Decoding for Text Generation using Lattices
Abstract
Conditional neural text generation models generate high-quality outputs, but often concentrate around a mode when what we really want is a diverse set of options.
We present a search algorithm to construct lattices encoding a massive number of generation options.
First, we restructure decoding as a best-ﬁrst search , which explores the space differently than beam search and improves efﬁciency by avoiding pruning paths.
Second, we revisit the idea of hypothesis recombination : we can identify pairs of similar generation candidates during search and merge them as an approximation.
On both summarization and machine translation, we show that our algorithm encodes thousands of diverse options that remain grammatical and high-quality into one lattice.
This algorithm provides a foundation for building downstream generation applications on top of massive-scale diverse outputs.
Introduction
Although pre-trained text generation models (Lewis et al., 2020; Raffel et al., 2020) have achieved impressive results across a range of tasks, these models do not always deliver what system developers want.
Machine generated text may be non-factual (Kryscinski et al., 2020;
Maynez et al., 2020; Goyal and Durrett, 2021) or toxic (Gehman et al., 2020).
We might patch these problems by applying discriminators over the output (Holtzman et al., 2018; Yang and Klein, 2021) to enforce these properties post-hoc; we could, for instance, apply a secondary model as a reranker over a small collection of outputs.
However, if the generator returns a homogeneous set of candidates, we may fail to ﬁnd any usable generation output.
What if generation models could return massive numbers of candidates rather than a few outputs with optimal score?
single hypothesis to proceed with lattice encodes 4 variations compactly Beam Search (high redundancy in outputs, hypotheses duplicated) A Cardiff recycling company has gone into bankruptcy. A Cardiff waste management company has gone into bankruptcy.
A Cardiff waste processing company has gone into bankruptcy. A Cardiff waste processing plant has gone into bankruptcy. A recycling company has waste management processing plant Cardiff gone into bankruptcy.
This Work: Search with Recombination Model With a large set of candidates, our secondary model could more easily ﬁnd an acceptable one without having to take more extreme steps like re-training the initial generation model.
Output diversity has separately been established as a useful goal for for applications such as dialogue and story generation (Li et al., 2016; Fan et al., 2019).
Standard approaches including beam search (BS) and sampling methods fall short of our goal.
Beam search uses signiﬁcant computational resources to explore similar hypotheses, and much of the computation in the search process is invested into paths that could be acceptable generation outputs, but are ultimately pruned.
Sampling approaches like nucleus sampling (Holtzman et al., 2020), although achieving better diversity than beam search, often re-discover seen hypotheses and can be harder to control for quality.
A central problem with both methods is that they do not handle very similar hypotheses efﬁciently.
In this paper, we present a decoding framework with two key components.
First, we argue that a modiﬁed best-ﬁrst search (B FS ) is the right way to explore the search space.
We augment standard best-ﬁrst search with a depth-ﬁrst path completion strategy: we eagerly expand each node until we reach an EOS token, thereby guaranteeing that each node is part of some completed path returned to the user.
This generation strategy avoids exploring large numbers of states which end up being pruned.
B FS is also more ﬂexible than static beam search and can prioritize exploration in more uncertain parts of the generation.
Second, our algorithm returns a massive number of generation options encoded in a lattice, with different hypotheses recombined in an approximate fashion.
Beam search preserves similar outputs such as “ A Cardiff recycling company has gone into ” and “ A Cardiff waste management company has gone into ” as different states.
However, these preﬁxes actually have very similar distributions of following words under the model; if we identify states like this, we can recombine them (Figure 2) and treat them as the same from the perspective of future continuations.
In Figure 1, we show an illustration of the lattice structure this recombination can form for document summarization.
We broaden a recombination method used previously in beam search for machine translation (Och et al., 2001; Zhang et al., 2018), enabling us to compactly encode large number of generation candidates and achieve dense lattices.
We show results for both document summarization and machine translation in three language pairs.
For each setting, we show that our lattice encodes a large number of high-quality candidates, including good matches with annotated reference generations.
We further show that a variant of our method can still achieve strong results with a lower number of nodes expanded than the baselines, suggesting that this can be a path towards saving computational resources.
We believe that computing thousands of high-quality generation candidates within a single compact data structure can provide a powerful starting point for various downstream purposes: diversity, factuality, customizability, and more.
Problem & Setup
We deﬁne our algorithm in the context of conditional text generation (Sutskever et al., 2014; Bahdanau et al., 2014).
Conditional text generation is formulated as sequence transformation from a source input x to target output y =
( y 1 , . . .
, y n ) via a neural text generation model parameterized by θ .
Each y i is a symbol in a vocabulary V .
The probability of a decoded sequence is given by p ( y | x ; θ ) =
(cid:81) nt =1 p ( y t | y <t , x ; θ ) .
Decoding text from a model can be framed as a search problem, where the search objective is to ﬁnd the Path to Merge Merge Target A Cardiff waste company has gone into bankruptcy recycling plant has gone into 𝒪 .pop
Apply recombination Check for possible recombinations Pop from pop the node with highest score from search frontier 𝒪 𝒪 . Y Expand a node N
After recombination: A Cardiff waste company has gone into bankruptcy recycling plant has gone into . deleted output sequence that maximizes the conditional probability under the model: arg max ˆ y p (ˆ y | x ; θ ) .
Because p (ˆ y t | ˆ y <t , x ; θ ) depends on the entire generated sequence so far, this decoding problem is intractable to solve exactly.
While typically the goal of decoding is to ﬁnd the hypothesis with the highest possible model score, we instead focus on ﬁnding a large set of “good enough” hypotheses.
That is, ﬁnding a set Y :
[Equation]
for some threshold (cid:15) .
(cid:15) emerges naturally by adjusting search hyperparameters to control the number of returned hypotheses.
Our goal in this paper is to design an algorithm that can efﬁciently ﬁnd Y .
Notation
We encode predicted generation candidates ˆ y in a lattice.
A lattice L =
( N, E ) is a directed graph where each node represents a token and paths deﬁned by directed edges encode candidates.
A path π in L from a unique startof-sequence node n sos to any node n represents a (partially) decoded string, consisting of the words in that path.
All completed paths start with n sos and end at (potentially different) end-of-sequence nodes n eos.
The search graph L is constructed iteratively through a search procedure.
We maintain the closed graph C with explored nodes and edges as well as a search frontier O , a set consisting of successors to nodes currently in the graph.
For each node, there are |V| possible successors.
We deﬁne the search budget as the number of nodes expanded from the search frontier.
Our ex- [Figure] periments will seek to compare different methods using the same search budget.
We will deﬁne this more precisely in Sec. 7.
Inadequacies of Beam Search
As we have alluded to, beam search is inadequate for our goal for several reasons.
We show experiments on these aspects in this section and
Appendix A.
Better Model Score (cid:59) Better Hypothesis
The most critical issue is that beam search is designed to efﬁciently approximate arg max ˆ y = p (ˆ y | x ; θ ) , but the optimal model score is neither our goal nor a guarantee of a good hypothesis.
In Figure 3, we compare the correlation of model score and
ROUGE under beam search for text summarization.
The Pearson correlation between these two variables is very weak.
Beyond ROUGE score, the example in Fig.
1 shows that the main differences between these summaries may be minor differences in surface realization that have little effect on our qualitative judgments of summary quality.
The hypothesis with the best score under beam search is not substantially better quality than hypotheses with slightly lower scores.
Lack of Diversity
in (Diverse) Beam
Search Are
the model outputs
from
BS
and
DBS di-
verse?
We use Self-BLEU ( S B L ) (Zhu et al., 2018) to measure the BLEU score for randomly sampled pairs from each algorithm’s output.
The lower the self-BLEU, the more dissimilar the pairs are.
On decoding summaries on XSum, the S B L for BS/DBS are 87/79 while a nucleus sampling method can achieve 57/50 depending on the conﬁguration.
Although DBS slightly improves the
[Figure]
[Figure]
[Table]
diversity compared to the original variant, the overlap of outputs from beam search based method is still very high, and the diversity remains a challenge.
Poor Efﬁciency from Pruning
One ﬁnal issue with beam search is that most of its computation is not even useful in producing ﬁnished hypotheses ; that is, the set Y of answers produced does not contain most of the nodes expanded in the typical course of operation.
We conduct an empirical pruning study on a summarization dataset and three translation datasets and show the results in
Table 1.
For all studied cases, beam search and diverse beam search prune over half of the expanded nodes.
Many pruned hypotheses are not truly ungrammatical or low quality, but are merely slightly less likely than other nodes.
How we can preserve more of the explored lattice and do so efﬁciently is addressed in next by our use of best-ﬁrst search.
Modiﬁed Best-ﬁrst Search
As established in the previous section, beam search prunes many paths that would potentially yield high-quality summaries and wastes computational resources expanding nodes that aren’t included in a
Input: Generation get checking and running path recombination. Output: P 1: O ← {( ∞ , n sos) , C ← ∅ , expanded ← 0 .
2: while expanded < b do 3: ˆ h ← O .pop
() 4: if isRecomb (ˆ h, C ) then 5: doRecomb (ˆ h, C ) 6: continue 7: end if 8:
if ˆ h (cid:54) =
EOS then 9: v greedy = arg max v ∈V p ( v | ˆ h, x ; θ ) 10: for v ∈ V do 11: score ← s (ˆ h (cid:76) v ) // concatenation 12: if v = v greedy then 13: score ← ∞ // depth-ﬁrst completion 14: end if 15:
O ← O ∪ ( score , n v ) 16: end for 17: expanded ← expanded +18: end if 19:
C ← C ∪ ˆ h 20: end while ﬁnal search graph.
We tackle this issue by changing from beam search to best-ﬁrst search (B FS ) (Hart et al., 1968; Pearl, 1984).
B FS prioritizes searching over nodes according to a scoring function, giving us more ﬂexibility in how we explore the space.
Our chief modiﬁcation of the base algorithm is a heuristic we call depth-ﬁrst completion.
Depth-ﬁrst Path Completion
Neural text generation is a search problem with large branching factor ( V ) and deep search depth (sequence length).
As a result, applying B FS with the scoring function being the model score of a state often leads to a broad search that rarely returns a valid path.
One solution to this problem is to incorporate a heuristic based on length.
Model score is monotonically decreasing as a sequence grows in length, so prior work (Wu et al., 2016; Zhang et al., 2018; Meister et al., 2020b) has used a length reward term to alleviate this issue.
We found that, even with a length heuristic, B FS will still have “dangling” nodes that are not part of any path to an EOS (goal) token, and it might return few or no valid hypotheses.
Recognizing our objective from Equation 1, we can take a simple step to ensure that every node ends up on some completed path: eagerly do a greedy search from each node until we reach n eos [Figure] or exceed a maximum length .
In Algorithm 1, we implement this by modifying the priority of the highest scored token with ∞ (line 12), so it will be explored immediately after the current time step.
In Figure 5, we illustrate depth-ﬁrst completion.
Search Algorithm
We describe B FS with depthﬁrst completion in Algorithm 1.
The algorithm is a modiﬁed best-ﬁrst search algorithm applied to text generation.
s ( · ) is a function to evaluate the value of a path.
Typically it is deﬁned as s ( y ) = (cid:80) log p ( y t | y <t ) .
b is the budget for total model calls to neural text generation model.
Note that isRecomb and doRecomb do not invoke the neural generation model, so they do not count towards the computation budget we deﬁned here.
In practice, we only consider top 5 expansions rather than the whole vocabulary V for line 10.
Path Recombination
Path recombination, also known as hypothesis recombination, was originally proposed and used in phrase-based machine translation (Och et al., 2001;
Koehn et al., 2003; Zhang et al., 2018).
The idea of path recombination is to combine similar paths if what the model predicts for them in the future is the same , reﬂecting a similar dynamic programming principle as the Viterbi algorithm.
We focus on ﬁnding hypotheses which approximately exhibit this property, and show that merging them can yield high-quality outputs.
Figure 2 shows an example of recombination.
The two hypotheses being merged here roughly convey the same intent, and it turns out that the shared sufﬁx “ has gone into ” is a strong indicator that the model will treat them similarly in the rest of the generation.
Prerequisites of Recombination
Theoretically, two search states should only be recombined if they yield the exact same distribution over future generation decisions (see strong equivalence in Ap- A
Cardiff waste company has gone into bankruptcy recycling plant has gone into .
[Figure]
pendix B).
However, this is intractable even to check approximately; we deﬁne a weaker criterion:
Deﬁnition 5.1 (Weak equivalence) .
Let a and b be two preﬁx strings starting with n sos.
a and b are weakly equivalent if greedy completions of these two strings are the same: arg max y P ( y | a )
= arg max y (cid:48) P ( y (cid:48) | b ) .
This criterion can be checked empirically, but it is not practical to do so during search.
To approximate equivalence, we deﬁne a similarity function isRecomb ( h , ˆh ) to determine if an expanded node ˆh should be merged with an existing expanded node h .
A similar recombination idea was explored in Zhang et al.
Following their work, we explore a family of rule-based heuristics for merging.
There are two rules: (1) two strings share a common n -gram sufﬁx, (2) the length difference of two strings is less than α .
Assume that the canonical paths for h and ˆh are lengths l and ˆ l , then isRecomb ( h , ˆh ) [-=
-]{+= 
+}
[ π ( h ) l − n +1 ,...,l = π ( ˆh ) ˆ l − n +1 ,..., ˆ l ∧
| l − ˆ l | < α ] where α and n are hyper-parameters.
For a large enough value of n , note that the shared sufﬁxes encourage hypotheses like this in Figure 6 that share large parts of the structure already.
Prior Work: BSZB EAM
Zhang et al.
(2018) use their merging criterion in the context of beam search for neural machine translation.
If the merging criteria hold, ˆh will be recombined with h .
However, ˆh will not be considered as a future merging candidate.
We call this merging strategy
ZB EAM .
We implement this model together with its merging criteria and denote it as BSZB EAM .
This strategy is tailored to beam search and explores a more limited set of merges than we do.
Canonical Paths
After recombination, a single node may represent multiple different possible sentence preﬁxes.
If an edge is created due to the extension of search graph via model’s prediction, we call it a G EN edge.
Otherwise, the edge is created due to path recombination, and we call it a
M RG edge.
We deﬁne the notion of a canonical path, which represents the single path used to score candidate expansions.
Deﬁnition 5.2 (Canonical Path) .
Let n be a node.
The canonical path to n is deﬁned as the unique path from n sos to n consisting only of G EN edges.
Theorem 5.1.
For any node n in the graph except n sos, there exists exactly one canonical path.
We present the proof in Appendix.
C. We deﬁne the path of a node n , π ( n ) , as returning the sequence of words corresponding to the canonical path of that node.
Expanding n computes P ( y | π ( n )) under the neural model.
Recombination Mechanism
We illustrate the two major recombination techniques we introduce, R CB and Z IP , in Figure 6.
These methods represent our two implementations of doRecomb in Algorithm 1.
R CB : Generalization of ZB EAM
ZB EAM has a major limitation: a limited set of merging candidates.
The potential merge candidates in ZB EAM are only nodes in the current beam hypotheses and their previous steps, so the method cannot merge with nodes from earlier timesteps.
For example, “ A waste plant has gone into ” cannot be merged with the hypothesis with ending in node 4 shown in Figure 6.
The proposed generalization, R CB , addresses this limitation.
We index all of the nodes in the lattice across all timesteps by their n -grams using a hash table, making it O (1) time to look up an n -gram pattern and retrieve potential merge candidates if they exist.
Z IP : Recombining More
If we take a closer look at R CB in Figure 6, we see that even in the merged structure, nodes 3 and 7 and nodes 2 andare preserved as separate.
They do not pass the recombination criterion themselves, but these nodes are part of the sufﬁx matched strings, still correspond to the same words, and have the same directly generated next word.
There is reason to believe that these might be equivalent as well.
Hence, we explore a variant called Z IP that propagates the merge backwards through the lattice.
This change relaxes the merging criterion and up to n pairs of nodes are combined when a merge is identiﬁed, leading to a more compact lattice.
We describe some of the details in Appendix D.
Our Methods
In this work, we combine the two proposed techniques, the modiﬁed best-ﬁrst search (B FS ) and recombination methods, together.
Hence, we name them as B FS R CB and B FS Z IP .
Evaluation
To evaluate the proposed methods, we conduct experiments on abstractive text summarization and machine translation.
Our evaluation focuses on two questions: (1) how large and diverse are our lattices? (2) are the candidates encoded in the lattices
high quality and grammatical ? 7. Datasets & Base Models
We obtain all the models and certain baseline decoding methods from the Transformers library (Wolf et al., 2020).
Since our methods are inference techniques without learned components, we do not re-train any models.
For summarization , we use
XSum (Narayan et al., 2018), a popular English news summarization dataset.
We sample 100 examples from the validation set.
The base model we use is BART-large-XSum (Lewis et al., 2020).
For machine translation , we study our models on the English-French (en-fr) pairs from WMT(Bojar et al., 2014) and Chinese-to-English (zhen) pair from WMT 2019 (Barrault et al., 2019).
We use mBART (Liu et al., 2020), a state-of-the-art neural machine translation model.
We set the max decoding length to be twice the input length, so it varies per example.
7. Search Budget
To fairly compare the resource usage of all methods, we deﬁne the search budget as the number of calls to the neural model, equivalent to the number of nodes expanded.
4 With beam size k and maxi- mum length T , beam search methods are given a theoretical budget of kT .
We could simply allow best-ﬁrst search and sampling methods to expand this number of nodes.
However, since hypotheses may terminate before they reach EOS, empirically there is a gap between effective length (the average generated hypothesis length) and max length for both beam search and sampling.
To balance the computation used across the different methods, we apply a correction factor so that the different methods are expanding the same number of nodes in aggregate.
We increase the beam size k by 50% for translation, from 8 to 12, and 25% for summarization, from 16 to 20, for our baseline methods: k to BS, DBS, N CLS , T EMP , and BSZB EAM .
This correction was empirically determined to balance the number of nodes expanded between our method and the baselines.
We emphasize that this correction improves the baseline performance relative to our methods.
7. Search Algorithms
We implemented G REEDY , BS, DBS, N CLS , and
T EMP as baseline methods.
N CLS 0 .
9 represents nucleus sampling method with p = 0 .
We refer to Appendix E for detailed descriptions.
We also experiment with basic B FS without path recombination, but including our depth-ﬁrst path completion technique to ensure that ﬁnished hypotheses are produced.
BSZB EAM is our implementation of Zhang et al.
We integrate R CB with nucleus sampling and best-ﬁrst search as N CLS R CB and B FS R CB .
We also test B FS with the Z IP strategy.
l B FS Z IP is a resource-efﬁcient version of
B FS Z IP where only 25% of the search budget is used, exploring what this method can achieve with a lower budget given its more aggressive merges.
7. Evaluation Metrics
We describe our metrics to evaluate both quality and diversity.
Several of our methods build on
ROUGE (including R1, R2, RL) (Lin, 2004) and BLEU (Papineni et al., 2002; Post, 2018) comparing the generated text to references.
Diversity-oriented Metrics
We evaluate the diversity of generated texts with the following metrics.
(1) | path | is the average number of unique paths in the produced lattice.
5 (2) Number of [Table] We use unique n -grams encoded in the lattice; this captures a different type of diversity than the number of paths, since there could be many paths reusing the same words.
N1 and N2 are average number of novel unigrams and bigrams in the graph.
(3) S B L is the average self-BLEU among m samples (Zhu et al., 2018).
The samples are drawn from a uniform random walk from n sos.
The range of S B L is [0 , 100] .
(4) ED is the average edit-distance among m samples.
We set m = 5 in our experiment.
Quality: Grammaticality
We adopt GECToR a neural grammatical error correction model (Omelianchuk et al., 2020) to automatically assess the grammaticality of generated texts.
We report
G RM E RR (%), the average number of grammar errors per token, for all English-output experiments.
Quality: Oracle Reference Match
Given the reference, we ﬁnd the path with highest ROUGE or BLEU over all found paths.
Oracle ROUGE is deﬁned as O R ( Y , y ∗ )
= max y ∈Y ( R2 ( y , y ∗ )) .
This metric captures both quality and diversity: the algorithm needs to ﬁnd something close to the reference, but a diverse lattice will have a higher chance of exhibiting a good candidate all else being equal.
Quality: Average Reference Match
Although our method focuses on deriving diverse text summaries or translations, we aim to guarantee that the generated text is highly relevant to the generation target and is of high quality in general.
We sample 1,000 paths from the lattice with replacement and evaluate the average ROUGE or BLEU compared to the reference.
We denote this metric as S P .
Results Text Summarization
We present the experimental results on the dev set of XSum in Table 2.
Full results are kept in Table 4 for reference.
Among non-recombination methods, BS and DBS are the least diverse methods.
Sampling based methods including T EMP are generally more diverse, but the oracle ROUGE is lower than that of B FS .
Given the sacriﬁced text quality (lower sample ROUGE and more grammar errors) of sampling based methods, we argue that modiﬁed best-ﬁrst search is a strong decoding strategy even without path recombination .
The bottom half shows all methods with path recombination techniques.
Recombination signiﬁcantly improves the diversity of generated outputs , with a much higher number of paths.
The S B L of the recombination variants are lower than their non-recombination counterparts.
In terms of search quality, the proposed B FS R CB and B FS Z IP methods obtain signiﬁcantly higher oracle ROUGE compared to all other methods.
We show these results later in Figure 9: our approach can ﬁnd much better oracle solutions, even compared with beam search method with quadruple the amount of computational resources.
The design of the oracle ROUGE metric is also motivated by a real use case: if you want a speciﬁc summary (e.g., a summary covering a speciﬁc entity or topic), does it exist in the search graph?
Higher oracle ROUGE indicates a closer match, meaning a strategy using some kind of reranking model could help ﬁnd the user’s preferred outputs.
Comparison: R CB & Z IP
The Z IP method yields even more diverse output at the cost of text quality.
There are a few reasons for this: 1) recombination of more nodes makes the lattice denser, increasing the number of paths but also potential errors; 2) elimination of unexplored children from merged branch reduces the waste of exploration which means Z IP can explore more hypotheses than R CB .
With the same amount of computational resources, Z IP explores a larger search space while
R CB explores a smaller collection more reliably. l Z IP exploits the efﬁciency of Z IP to achieve high diversity, and by searching through fewer states, it manages to achieve higher quality as well. [Table]
Machine Translation
We show the result on machine translation in Table 3 and 6.
Results on translation tasks show the consistent gains of diversity from path recombination models.
In Table 3, we show two translation tasks where the target language is English.
B FS R CB works better than B FS Z IP because it disables some aggressive and bad merges which explores bad hypotheses.
Compared to summarization, we found the search space in MT to be more constrained, so there was less room for aggressive merging and exploration to improve over R CB .
Our lower-resource method, l B FS Z IP approach, actually performs quite well on most metrics with only 25% of search budget.
It has better diversity performance than any nonrecombination methods, and comes with quality better than most of the recombination methods.
The usage of B FS and path recombination methods like B FS R CB and B FS Z IP is promising for being able to ﬁnd a better cost-diversity tradeoff in MT
. Validating the Merging Criterion
Our merging criterion is fundamentally an approximation of the equivalence criteria described in Section 5.
Our question is: what fraction of nodes merged by our merging criterion satisfy the weak equivalence assumption?
We conduct an experiment to verify this on XSum.
We compute the greedy completion for L timesteps and check whether the continuation of the base candidates would be the same.
In Figure 7, we show the fraction of merged pairs for which the generations match exactly un- [Figure] der three values of the recombination criterion.
For
B FS R CB , when using n = 4 the greedy continuation over 4 timesteps is the same 71.2% of the time.
For B FS Z IP it is the same 62.5% of the time.
Following the weak equivalence criterion is a strong indication that these hypotheses can admit many of the same continuations. R CB is more reliable than Z IP , but both methods show moderate adherence to the equivalence criterion. Error Analysis & Visualization
In Figure 8, we present two examples on XSum by l B FS Z IP .
The upper example has more word level recombination and paraphrasing while the bottom one has more ways of ending and more diverse content coverage.
We show more examples on both summarization , GPs and paramedics nurses paramedics nurses must be more aware of the warning sepsis of should consider the possibility should be looking for signs treating when when treating patients with an infection, the UK’s health watchdog has said.
as early as possible to help prevent thousands of deaths, health officials say.
patients with an infection, the UK’s health watchdog has said.
be looking in should in the UK hospital staff
England the UK for are of sepsis before look warned being signs told to urged to think twice before diagnosing sepsis, which can lead to death.
will begin their Euro 2016 Northern Ireland campaign for the first time , have qualified for Eurowill be hoping to emulate Leicester City’s Premier League success when they take on Italy in their opening Euro 2016 qualifier on Thursday.
’s players have reached their first major tournament since 1982, and are hoping to emulate Leicester City’s Premier League success by winning Euro 2016.
are preparing for their first major tournament since 1982, when they reached the European
Championship finals.
and one of the favorites to win Euro 2016, along with Leicester City, who are the book makers, favorites to win the tournament.
since 1982.
in their history.
are against Albania on Thursday, 5 June.
on Thursday against Albania, having qualified for the tournament by finishing second in their group.
and translation in Appendix.
H.
We manually examine the output and found a few common types of errors introduced by our algorithm.
(1) Factual errors at high entropy nodes.
Our approach assumes that high-scoring candidates under the model are good quality, but this assumption is violated in certain cases, like when the model attempts to hallucinate information.
For example, the preﬁx “ The company, founded in ” will cause the model to guess answers like “ 1989 ” or “ 1999 ”.
Encoding all of these in the lattice is incorrect.
However, we did not see signiﬁcant factual errors introduced by merging speciﬁcally.
(2) Aggressive bad merges.
In the upper example in Figure 8, the cluster of “ GPs ”, “ nurses ”, “ paramedics ” is an example case.
The lattice encodes paths like “ GPs, nurses and nurses should ... ”.
This could be ﬁxed by heuristics or rules in future work.
Related Work
The techniques used in this work partially reﬂect an outgrowth of a few lines of literature: understanding the behavior of text generation models (Xu et al., 2020; Xu and Durrett, 2021; Zhong et al., 2021), investigations into beam search (Stahlberg and Byrne, 2019; Meister et al., 2020a), and studies of diversity in generation.
In terms of search strategies, best-ﬁrst beam search (Meister et al., 2020b) is a method integrating best-ﬁrst search with beam search.
Some other variants of search have also been studied in previous work (Meister et al., 2021b,a).
Beam search has been critically examined in some recent work (Huang et al., 2017; Stahlberg and Byrne, 2019), but largely of focused on speciﬁc challenges in MT.
As for diverse generation, neural text degeneration has been discussed in Radford et al.
Holtzman et al.
Welleck et al.
(2020), which led to an interest in diverse generation models.
Diverse text generation has been studied in previous work (Yu et al., 2017), including in dialogue (Li et al., 2016), story generation (Fan et al., 2019), and particularly paraphrasing (Iyyer et al., 2018; Goyal and Durrett, 2020).
Our method can also diversify content coverage (Gehrmann et al., 2018) and word choice (Cao and Wang, 2021).
Discussion & Conclusion
We presented an algorithm for decoding in text generation with two main components: best-ﬁrst search to more efﬁciently structure exploration of the space and hypothesis recombination to encode summaries in a lattice structure.
We showed that across summarization and machine translation, these lattices successfully encode large numbers of high-quality generation options.
There are a few limitations of our method.
First, we currently benchmark these techniques using number of nodes expanded, not wall clock time.
There are strategies for parallelizing the B FS expansion (Shu and Nakayama, 2018), but it remains to be seen how this parallelism compares to the parallelism achieved by beam search.
Regardless, the dramatically larger number of hypotheses we return outweighs efﬁciency differences for now.
Second, we focus on auto-regressive methods in this paper.
However, we believe our framework could also be applied and adopted to non auto-regressive generation models (Song et al., 2021).
Acknowledgments
We would like to thank Shuyang Cao, Eunsol Choi,
Tanya Goyal, Jonathan Kummerfeld, Jessy Li, Yasumasa Onoe, Xi Ye, and Zhisong Zhang for input and feedback on this work.
This work was principally supported by a gift from Amazon, as well as
NSF Grant IIS-1814522 and a gift from Salesforce
Inc.
Thanks to the anonymous reviewers for their helpful feedback.
A Inadequacies of Beam Search: Poor Scaling Behavior
In spite of the issues with beam search that we describe in the main text, perhaps beam search could still be viable with larger beam sizes if more computational resources are available.
We experiment with beam sizes of 2 { 4 , 5 , 6 , 7 } and see how diversity scales with beam size.
In Figure 4, we found that an exponential increase of beam size does not lead to a strong increase of number of novel bigram in beam search.
In DBS, the diversity does ramp up, but the quality of the generated text decreases substantially.
For BS and DBS, increasing beam size is not an effective solution for better diversity.
We compare the oracle R2 of BS/DBS with larger beam size in Figure 9.
The oracle R2 increases slowly as k doubles, but our model B FS R CB with k =achieves 35.8, much higher than all BS/DBS cases.
B Strong Equivalence of Path recombination
In the strictest form, recombining two hypotheses assumes the following equivalence between them:
Deﬁnition B.1 (Strong equivalence) .
Let a and b be two preﬁx strings starting with n sos.
a and b are strongly equivalent if P ( y | a ) =
P ( y | b ) holds for all y .
Merging such states in the search tree is valid with no loss of information, as any expanded node will receive the same score under both preﬁxes.
However, this assumption is not realistic since seq2seq models condition on the entire sequence so far, and any tiny perturbation changes the predicted distribution.
To relax the assumption, we then propose the weak alternative.
C Proof of
Theorem 5.
Proof by induction.
Base case: we begin with just n sos in the lattice, which has exactly one canonical path consisting of itself.
Inductive case: assume every node in the lattice has exactly one canonical path.
We have to consider two cases when expanding a node in the lattice: (1) Expanding the node n as normal.
In this case, n is on the search frontier due to its parent node n (cid:48) being expanded, which establishes a G EN edge from n (cid:48) to n .
Since n (cid:48) has exactly one canonical path, n then has exactly one canonical path consisting of the canonical path to n (cid:48) extended to n .
(2) Applying recombination.
This operation only adds M RG edges and deletes nodes, neither of which have any impact on the canonical paths.
D Implementation Details: Z IP
We summarize the key differences of ZB EAM , R CB and Z IP in Table 5.
In Z IP , nodes that are already expanded might be removed from the lattice due to recombination.
For example, in Figure 6, nodeand 7 are removed in this fashion.
In general, we handle this by re-mapping the eliminated node to its surviving counterpart.
Any reference to nodeis routed to node 3, or whatever node 3 is mapped to.
This procedure is deﬁned and implemented as a union-ﬁnd data structure.
Deduplication of Unexplored Successors
After the Z IP procedure, we also remove the unexplored successors of the merged nodes, like node 6, 7, and 8 in Fig.
We show a more detailed example in
Figure 10.
In Z IP , we will merge node 3 and node 6.
If we take a closer look at the successors of these two nodes, the distributions could be similar and in fact are expected to be if the equivalence criteria hold.
We remove the unexplored direct successors of the merged node as part of the merging process, and the surviving node (node 3) captures these with similar probabilities regardless.
E Baselines G REEDY
is the deterministic greedy decoding method that always selects the highest probability token as prediction.
The equivalent beam size for this approach is 1 since we only run one pass.
Frontier 𝒪
{+Frontier 𝒪 2 love 3 this 4 show 
+}
I 5 like 6 this 7 show play musical performance p = 0.
play musical amazing …… …… 0.2 0.
p = 0. BS & DBS stand for beam search and its variant diverse beam search (Vijayakumar et al., 2018).
In our conﬁguration, we use Hamming distance as the diversity function and set the diversity strength to 1.5, following Vijayakumar et al.
N CLS is the nucleus sampling method proposed in Holtzman et al.
(2020), which encourages quality by truncating the distribution over the vocabulary with a parameter p before sampling.
We experiment it with p = 0 .
9 and p = 0 . 8 .
T EMP changes the temperature of softmax function to reshape the prediction distribution (Ficler and Goldberg, 2017).
We set the temperature parameter τ = 1 .
so the prediction picks more lowscored tokens than τ = 1 .
F Implementation Details: Beam Search
In our beam search implementation, the size of the search frontier O is up to the beam size k .
When a path is completed, we move it from the search frontier O to a completed set F to free up the beam for exploring unﬁnished hypotheses.
Naturally, ﬁnished hypotheses F in the end can be of variable length.
After reaching the max generation step T , we sort all hypotheses in F according to the model score.
Following common practice in libraries such as Transformers (Wolf et al., 2020), we return a number of completed hypotheses equal to the beam size.
G Results of WMT14 English to French
Table 6 shows an additional experiment on EnglishFrench.
We do not evaluate on grammaticality due to the GECToR model being specialized to English.
The results show broadly similar trends as those in
Table 3, discussed in the main text.
H Examples
We show three examples with visualization in Figure 11,12 and 13.
We use PyVis as the visualization tool.
6 More examples are available at anonymized .
I Computational Considerations Resources Used
All experiments were conducted on a server with 32GB RAM and Intel Xeon
E5-2630 CPU, using a single NVIDIA GTX1080.
The total computational budget in GPU hours is within 50 hours for experiments in text summarization and machine translation.
Memory and Runtime
Although the ﬁnal lattices returned encode large numbers of paths, they do not take large amounts of memory.
Because the number of nodes in a lattice is no larger than the number of node expansion operations during beam search, it is always less than the search budget and can be stored compactly.
Moreover, the wall clock time of our BFSRecomb strategy is manageable, on the order of between 1 and 10 seconds for summarization.
As mentioned in the Conclusion, additional parallelism can be combined with our BFS search to further improve the time and make it comparable to beam search.
However, even this version of the algorithm can be “embarrassingly” parallelized across examples to improve efﬁciency.
Descriptive Statistics
We randomly sampledata instances from the validation set for each dataset, and they are used by all methods.
When sampling is needed, we take 1,000 samples for each data instance, so all the metrics are reported on 100,000 translations/summaries for one dataset.
J Risks
By generating additional outputs from a generation model, we may cause a system to produce outputs that are biased, factually inaccurate, or contain hallucinations.
However, all of these risks are present in the raw text generation models as well.
Moreover, because we present many options, we believe our approach more appropriately reﬂects the model’s uncertainty over its output, and may have a part to play in mitigating such risks in systems of the future.
[Figure] [Figure]
[Table] [Table]
[Table]